Evaluating Real-World Robot Manipulation Policies in Simulation
Authors: Xuanlin Li
Kyle Hsu
Jiayuan Gu
Oier Mees
Karl Pertsch
Homer Rich Walke
Chuyuan Fu
Ishikaa Lunawat
Isabel Sieh
Sean Kirmani
Sergey Levine
Jiajun Wu
Chelsea Finn
Hao Su
Quan Vuong
Ted Xiao
Venue: CoRL 2024 (2024)
Keywords: real-to-sim
policy evaluation
robot manipulation
TL;DR: We introduce a suite of simulated environments for scalable evaluation of real-robot generalist policies, and show strong correlation to real robot evaluations.

Abstract: The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, issues that are likely to worsen as policies broaden the spectrum of tasks they can perform. In this work, we demonstrate that simulation-based evaluation can be a scalable, reproducible, and reliable proxy for real-world evaluation. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to painstakingly craft full-fidelity digital twins. We then employ these techniques to create SIMPLER, a collection of simulated environments for policy evaluation on common real robot manipulation setups. Through over 1500 paired sim-and-real evaluations of manipulation policies across two embodiments and eight task families, we demonstrate strong correlation between policy performance in SIMPLER environments and that in the real world. Beyond aggregated trends, we find that SIMPLER evaluations effectively reflect the real-world behaviors of individual policies, such as sensitivity to various distribution shifts. We are committed to open-sourcing all SIMPLER environments along with our workflow for creating new environments to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. Website: https://simpler-env.github.io/

Evaluating Real-World Robot Manipulation Policies in Simulation
Xuanlin Li
Kyle Hsu
Jiayuan Gu
Karl Pertsch 2
Oier Mees
Homer Rich Walke
Chuyuan Fu
Ishikaa Lunawat
Isabel Sieh
Sean Kirmani
Sergey Levine
Jiajun Wu²
Chelsea Finn? Hao Su
Quan Vuong!
Ted Xiao
UC San Diego, SStanfor University, 'UC Berkeley, 'Google Deepmind
"Equal contribution "Core contributors 'Co-advising
Abstract: The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, issues that are likely to worsen as policies broaden the spectrum of tasks they can perform. In this work, we demonstrate that simulation-based evaluation can be scalable, reproducible, and reliable proxy for real- world evaluation. We identify control and visual disparities between real and simulated environments key challenges for reliable simulated evaluation and propose approaches to mitigating these gaps without the need to painstakingly craft full-fidelity digital twins. We then employ these techniques to create SIMPLER, a collection of simulated environments for policy evaluation on common real robot manipulation setups. Through over 1,500 paired sim-and- real evaluations of manipulation policies across two embodiments and eight task families, demonstrate strong correlation between policy performance in SIM- PLER environments and that in the real world. Beyond aggregated trends, we find that SIMPLER evaluations effectively reflect the real-world behaviors of individ- ual policies, such as sensitivity to various distribution shifts. We are committed to open-sourcing all SIMPLER environments along with our workflow for creating new environments to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. Website: https://sipperr env. github.io Keywords: Policy Evaluation, Real-to-Sim, Robot Manipulation
Real robot evaluation (train real, evaluate in real)
RT-1
RT-1-X RT-2XX + Octo
1.0
Expensive and slow Dificuut reproduue
X
Our dirrlltedd succass rate
Simulated evaluation (train real, evaluate in sim)
Cheap and scai scalable
Fully reproducible
0.0
01.4
success
Figure 1: Characterizing generalist robot manipulation policies typically involves evaluating them on many tasks & scenarios, laborious undertaking in the real world (top left). In this work, we design an evaluation procedure where policies trained on real data are evaluated in purpose-built simulated environments (bottom left), Our approach yields strong correlation between real world and simulated performance (right) for various open-soure robot policies [1, 3] across two commonly used robot embodiments (Google Robot Wid- owX) and over 1500 evaluation episodes. These results highlight the potential of simulation-han approaches for evaluating generalist real-world robot manipulation policies in a scalable, reproducible, and reliable way.
Introduction
Remarkable progress has been made in recent years toward building generalist real-world robot manipulation policies [1, 2, 3, 4, 5], i.e., policies that perform wide range of tasks across many environments and even robot embodiments, However, evaluating these policies in scalable
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Google Robot
SIMPLER
BridgeData
Stmubneed Maniizu lutiioniin Pol Evellation
Roboot Rehh
simpler.
ere staster eny anleet "poogle_rdbor,
Pux Spoon
prkdoy teot_poloevt)
atN.rsset() erw.atepl polloy O
Open-Claze Drawer
Put Object in Dratver
Stack Cuzes
PLt Eggplant Bashot
We introduce SIMPLER, a suite of simulated evaluation environments for common real robot manip- ulation sctups, namely the Google Robot evaluations from the Robotics Transformer series of works [1, 2, 17), and environments from the BridgeData V2 dataset [S] (see Appendix for detailed environment descriptions). make using SIMPLER easy via exposing a standard Gym interface. Additionally, we commit open-source policy inference code for real-to-sim evaluation of common generalist robot policies (RT-1 [1], RT-1-X [2], and Octo [3]). and provide detailed guides for evaluating new policies and creating new evaluation environments.
Figure 2:
and reproducible way remains challenging, as real-world evaluation expensive and inefficient. Compared to the evaluation burden of works that study robot performance narrower settings, the scope of evaluations required for faithful performance estimates of generalist policies increases with the breadth of their abilities. This underlines a growing challenge in robot manipulation research: as scale the capabilities of robot policies, how do we correspondingly scale our ability accurately, reproducibly, and comprehensively evaluate them?
In this work, we propose simulated evaluation as a possible answer, in which manipulation policies trained on real data are evaluated in purpose-built simulated environments (Fig. 1). Such real-to-sim evaluation can serve as a scalable, reproducible, and informative tool to complement gold-standas real-world evaluations. Indeed, evaluation in simulation is common practice for testing autonomous driving policies across wide range of scenarios before real-world deployment [6, 7, 8]. However, performing simulated evaluations for robotic manipulation poses additional challenges due to the diverse interactions between the agent and the environment. At the same time, research on sim-to- real policy learning [9, 10] has demonstrated that considerable transfer between simulation and the real world is possible even for manipulation policies. While sim-to-real approaches typically train simulation and evaluate in the real world, we are interested in the opposite question: how can we build systems for evaluating manipulation policies trained on real data in simulated environments?
One option to build such simulated environments is to fully replicate an existing real-world en- viromment by creating a simulated "digital twin", an approach popular in navigation [11, 12] and autonomous driving [13]. However, for robot manipulation, reconstructing dynamic and interactive objects [14], along with realistic materials and lighting [15] in simulation remains an open research question. Furthermore, building full- full-fidelity digital twins demands extensive time and resources, typically requiring digital artists to manually craft object geometries and materials [11, 16). Captur- ing precise physical properties of objects for manipulation simulation, such the center of mass, inertia, and static and dynamic friction, further complicates scalability.
A key idea in this work is that we do not need to build exact replicas real-world environments. Instead, we aim for simulated environments that are merely realistic enough, such that the perfor- mance of policies evaluated in simulation correlates well with their real-world performance. This allows us design environment creation pipelines that are more scalable than creating exact digital twins. Through extensive experiments, examine the challenges of building effective simulated evaluation pipelines, specifically control and visual disparities between real and simulated environ- ments. We then propose and evaluate approaches for mitigating these differences based on offline system identification, "green-screening" simulation observations using real-world backgrounds, and object texture baking from real-world images.
Using these techniques, create SIMPLER, suite of simulated manipulation policy evaluation environments for commonly used real robot setups, namely the RT-1 [1] and BridgeData V2 [5]

---

evaluation setups (Fig. 2). For both setups, we perform extensive paired sim-and-real evaluations for multiple open-source manipulation policies such as RT-1-X [2] and Octo [3], and we demonstrate a strong correlation between policy performance as assessed by SIMPLER and the corresponding real environments (Fig. 1, right). In addition, we find that simulated policy evaluations in SIM- PLER environments accurately reflect policy behavior modes the real world, such as sensitivity to various distribution shifts. As such, SIMPLER is a first step toward using simulated evaluation as tool for reliable, scalable, and reproducible manipulation policy evaluation. We are committed to open-sourcing our workflow for constructing SIMPLER environments to facilitate research on general- purpose manipulation policies and simulated evaluation frameworks.
2 Related Work
Reproducible evaluation of real robot policies a long-standing challenge. Initiatives like YCB [18] and NIST [19] were introduced standardize evaluation objects, yet standardizing other variables such lighting, camera setups, and workspaces proves difficult. Other benchmarks like the Amazon Picking Challenge [20], DARPA Robotics Challenges [21]. TOTO [22], and RB2 [23] require on- going maintenance of real-world evaluation setups, representing significant long-term investment. As real-world robotic datasets [1, 2, 5] and generalist policies [24, 17, 3, 25] proliferate, the demand for reliable, scalable, and reproducible methods of evaluating these policies grows. The need is particularly acute given the difficulty faced by the research community in conducting evaluations without standardized hardware.
Simulation- algorithmic research offers an alternative to real-world evaluation. A wide range of simulation benchmarks [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38] have been established to facilitate scalable and reproducible evaluation. However, most previous work considers both training and evaluation in simulation, and the resulting policies might exhibit distinct behaviors when deployed on real robot hardware. In contrast, we aim to both measure and enhance simulated evaluations' ability to reflect policy's real-world performance and behaviors.
Can simulated evaluations reliably predict real-world policy performance and behavior modes? While several studies [12, 11, 39] have explored this question in navigation tasks, we focus on sim- ulation evaluation for real-world manipulation policies. Manipulation presents unique challenges due to the tight interaction loop between policy and environment, dynamic rather than static scenes, along with the intricate, precise, and complex action sequences where even slight variations can significantly impact task outcomes.
For robot manipulation, the inmtto-real setting has been extensively investigated, where one aims to train policies in simulation and deploy them in the real world. To tackle the gaps between simulation
and reality, prior work has adopted domain randomization [40, 41, 42, 43, 44, 10, 45, 46, 47] and domain adaptation [48, 49, 50, 51, 52] approaches. For instance, Generative Adversarial Networks (GANs) [48, 49, 50, 51] are trained to modify images generated in simulations so they resemble the style of real-world images. Alternatively, Du al. [52] aim to align the feature space of observa- tions between simulated and real-world environments, creating more consistent visual experience across these domains. In contrast to these methods of sim-to-real learning, we focus on the opposite question: building simulation systems that effectively and faithfully evaluate real-world robot ma- nipulation policies. To this end, we introduce approaches to addressing both real-to-sim visual and control gaps to enhance real-&-sim evaluation correlations.
3 Using Physics Simulators for Evaluation of Robot Manipulation Policies
Problem formulation. We study the problem of using physics simulators evaluate performance and examine the behavior modes of real robot manipulation policies. We emphasize that our goal is not to completely replace real-world evaluations or to perfectly replicate real-world policy behaviors in simulations, as there are always gaps between sim and real. Instead, we aim to achieve strong correlation between relative policy performances in the real world and in simulation. This gives practitioners readily available policy improvement signal to guide their research. Formally, for two policies na and no with real-world performance measures R. and RG such as their success rates across tasks aim to construct simulator S (with performance measures Rs,a and Rs,s) that achieves strong correlation between relative performances in simulation and in the real world.
3

---

Good Sim Eval
Bad Sim Eval
Limitations of Pearson Correlation
Shrong dim coms
Poor real pesl-and-sm oomsiation
Only Inser
Ovarit
Smn 3U00883 rate
Small rank
Sim SLACSOOd nate:
Sin BuDD890 rale
MMRV
Sin aa000ee 1000
MMRV MMRV
Large man
Real success rate
Peel success rate
Real success ratee
Real success ratte
Figure 3: Illustration of mean maximum rank violation (MMRV, range [0, 1], lower is better) and Pearson cor- relation coefficient (Pearson r, range [-1.1] 1I, higher is better) for assessing the policy performance correlation in real-world and simulation, as well as the overall quality of simulated evaluation pipelines. Each circle rep- resents policy. For the two leftmost pipelines, both metries yicld valuable insights, identifying one as poor and the other good. The two rightmost examples highlight the limitations of Pearson it can penalize simulation pipelines that fail linearly recover the real results despite recovering the correct ranking, and it overly sensitive to minor noise in evaluations when different policies perform similarly in the real world.
Metrics for real-to-sim evaluation pipelines. A standard approach for measuring the correlation between two variables is the Pearson correlation coefficient (Pearson r) [53]. However, Pearson correlation has two important limitations for judging simulated evaluation pipelines: (1) It only assesses the linear fit between real and simulated performance, which is not necessarily required so long as the simulated pipeline reflects real-workls performance improvements between different policies (Fig. 3. middle right); (2) It does not reflect the range of values is computed over. Thus, for policies performing similarly in the real world, r may change drastically based on the inherent noise in real-world evaluations (Fig. 3, far right grey vs. green).
To address the first drawback of the Pearson correlation, we can additionally report a ranking metric that measures whether the simulated evaluation ranks the policies correctly based on their real-world performance, independent of whether the relationship is linear. However, conventional ranking met- rics such as Spearman's 's rank correlation [54] still suffer from the second shortcoming: they operate purely on the rankings and disregard the underlying margins between real values. When choosing suitable ranking metric, the key point is that we need to take the magnitude of the rank violation into account, measured as the difference in real-world performance between the mis-ranked policies. This provides a clear signal whether rank violations are caused by small real-world performance dif- ferences that are often the result of inherent noise in real robot evaluations, as the far-left example of Fig. 3, or constitute clear failures of the evaluation pipeline, as the middle-left example of Fig. 3.
Thus, propose the mean maximum rank violation (MMRV) metric to better assess the consis- tency of the real-and-sim policy ranking. Given N policies TI...N and their respective performance measures (e.g., success rates) RI...N, RS.I... from real and simulated evaluations, we have:
1 max IR, RşI [(Rs,, < Rs.)) + (Ra < R2)
MMRV(R, Rs)
(1)
N
1SJSN
The key underlying quantity is the rank violation between two policies т; and πj. which weighs the significance of the simulator's incorrect ranking by the corresponding real-world performance mar- gin. MMRV then averages each policy's worst-case rank violation. We will report both MMRV and Pearson r as they provide complementary perspectives on the simulated evaluation's 's effectiveness.
4 SIMPLER: Simulated Manipulation Policy Evaluation for Real Robots
This section introduces our approach to designing a simulation evaluation pipeline for real robot manipulation policies. We take inspiration from the rich literature on sim-to-real policy learning [45, 40, 41, 10] and focus on control and visual gaps between simulation and the real world.
4.1 Mitigating the Real-to-Sim Control Gap via Offline System Identification
First, we need to ensure that the policy actions executed in simulation yield comparable ef- fects on the system state as those observed when executed on the real robot. Concretely, let ((x;, Ra) x; E R3, R. E 80(3)) be 6D end-effector pose trajectory recorded when rolling out action trajectory {a,)= on real robot. Let {(x{, Rt) x² Xo, R/ Sim(p, d, (a (T Xo, Ro) be the corresponding trajectory when unrolling the same sequence of actions in the simulation in an open-loop manner using stiffness and damping parameters (i.e., PD parameters) (p.d). Then, we define our system identification loss from translation and rotation
emrrs as Loysid (p. d) = + E'z (IIx; - -xlla + arcsin
2 R -((lE)

---

In practice, we use a small sample of 20 trajectories from an offline dataset D, e.g., real-world demonstration dataset, to retrieve action and end-effector pose trajectories and compute the system identification loss above. For all environments in this work, we use trajectories from existing open- source demonstration datasets [1. 5], and thus do not need to collect any new data. We also ensured that each trajectory comes from different task instructions to encourage trajectory diversity.
Next, we optimize the parameters of our controller: given initial PD parameters (Po. do) and search ranges [Plow.. Phigh.o]; [dlowo, dnighoo,, we normalize the range to [0, 1] and perform simulated an- nealing [55] to optimize Lsysid in a gradient-free manner. We then select the PD) parameters with the lowest Caysid as (Pi.di) and initialize another round of simulated annealing with a reduced parameter search range. In total, we perform rounds of simulated annealing.
On the right, qualitatively illustrate the ef- fects of our system identification for one of our simulated environments, the Google Robot [1]. We find that naively using PD parameter values from real controllers results in inaccurate track- ing of the real robot's end-effector movements, which culminates missed grasp on the Coke can. After system identification, the controller more accurately tracks the motion in simulation: the robot is able to grasp the object when replaying the demonstration' action sequence.
Resl Sim SysiD Sim w/ SyslD
4.2 Mitigating the Real-to-Sim Visual Gap via Visual Matching and Variant Aggregation
Visual discrepancies between real and simu- lated environments can cause distribution shifts that adversely impact policy behavior, making simulated evaluations less reliable [11]. While modern graphics pipelines are able to create highly realistic visuals, developing the underly- ing assets and determining the lighting parame- ters to accurately model existing environments involves significant manual labor. Our goal is to match the simulator visuals those of the real- world environment with only a modest amount of manual effort. For the scene background, we propose a "green screening" approach in which we overlay an image of the real-world environment onto the background of the simulated scene (see the wrapped figure). Concretely, we perform the following steps: (1) we remove the robot and foreground objects from the first frame of a real- world evaluation video Iraa using online image inpainting tools (e.g., https://cleanup. http://clcamuuppcpurs)). (2) we create a binary mask M isolating the foreground objects (robot arm and interactable objects, such as tabletop objects and articulated objects) in the simulation rendering Isim by querying ground truth segmentation masks in simulation; and (3) combine the real-world background with the simulation foreground: I' M c Isim + (1 M) Leat. producing the green-screened image.
Roal
Hew Sim
Sim Green Screen
Sim Green Screen Matching
pictures);
In practice, we find that many tested policies are also sensitive to changes in foreground object and robot textures, yet available simulation assets often exhibit appearance differences from real- world objects. Thus, we tune simulation asset textures for many objects more closely match their real-world counterparts. We term this approach "Visual Matching" (see above for an illustration). Concretely, we project the real texture onto the simulation object by (1) segmenting the object in a world image [56]; (2) aligning the simulated object pose to the real image; and (3) "unpro- jecting" the texture onto the object mesh in simulation. We provide step-by-step instructions in Appendix B.2, and we will release a convenient command-ling script for this process. Otherwise, for assets like robot visual meshes with texture maps already resembling their real-world counter- parts, we can instead selectively copy and paste color values from real to simulated texture maps. Finally, as robot arms may change colors during movement, we found helpful obtain multiple tuned robot arm colors that match the real-world textures from different phases of manipulation task. We then average their evaluation results to mitigate this confounding factor.
5

---

As an alternative to Visual Matching, we explore a mitigation strategy for real-to-sim visual gaps inspired by domain randomization: instead of minimizing the gap, we heavily randomize visual aspects of the scene to create environment variants. We then obtain an estimate of a policy's 's perfor- mance by aggregating evaluation results across multiple such variants, which we term as "Variant Aggregation" (see Appendix B.3 for more details and visualizations). Note that the Variant Aggre- gation setup does not seek to match the visual appearances between sim and real. As a result, it can utilize any scenes from public datasets like ReplicaCAD [16] and any objects (including the tabletop objects and the cabinets) from public datasets like Objaverse [57] for evaluation, whose appearances are out-of-distribution for the policies evaluated in this paper.
SIMPLER environments and other details. We instantiate SIMPLER on two commonly used real robot evaluation setups: the Google Robot from the RT series of work [1, 17, 2] and the WidowX BridgeData V2 setup [5]. For each setup, we provide simulations for multiple tasks spanning range of skills, interacted objects, object positions and orientations, backgrounds, and lighting con- ditions (see Fig. 2). The tasks are chosen to be representative of those in the corresponding training datasets, while also involving largely rigid body objects whose dynamics can be reasonably well- approximated by modern physics simulators. We also instantiate SIMPLER on top of the SAPIEN physics simulator [58], but later show that our contributions are independent of the simulator used and can be reproduced in Isaac Sim [59]. Creating most new environments takes an experienced user approximately one hour. See Appendix A and Appendix B for more details.
5 Experimental Results
In this section, we empirically test the performance correlation between real-world robot evaluations and simulated evaluations in SIMPLER environments for representative set of open-source gener- alist robot manipulation policies. Concretely, our experiments are designed to answer the following questions: (1) Do relative performances of different manipulation policies in SIMPLER strongly correlate with those observed in real evaluation (Section 5.2)? (2) For a given policy, can SIMPLER evaluations accurately reproduce real- wolld policy behavior modes, e.g. sensitivity to various visual distribution shifts (Section 5.3)? (3) How sensitive is the fidelity SIMPLER evaluations to (i) control and visual gaps, (ii) physical property gaps, and (iii) the choice of simulator (Section 5.4)?
5.1 Experimental Setup
To quantitatively evaluate correlations between real and simulation policy performance, perform paired sim-and-real experiments, i.c., with the same task instructions, object & robot poses, evalu- ation trials, and success criterions between the simulation and the real world (note that the paired experiments are not restricted to the tasks and setups in policies' training data). We use popular open-source generalist robot policies: RT-1-X [2] and Octo [3] (Octo-Base and Octo-Small). For evaluations in the Google Robot environments, we additionally use a number of RT-1 [1] check- points various stages of training: RT-1 trained to convergence (RT-1 (Converged)). RT-1 15% of training steps (RT-1 (15%)), and RT-1 the beginning of training (RT-1 (Begin)). We also report results on RT-2-X [17]. More details in Appendix A.
5.2 SIMPLER Environments Show Strong Performance Correlations with Real Evaluations
We summarize the results of our main paired real-world and simulation evaluations in Fig. 4 (detailed breakdown in Appendix Table 2). We observe a strong correlation between the relative performances in simulation and in the real world across most policy checkpoints and tasks we evaluate, as reflected in low values for the MMRV metric introduced in Section and high values for Pearson r.
In Table 1, we compare SIMPLER with using the action MSE in validation episodes for policy ranking, which is common for model selection within supervised learning settings (c.g., imitation learning). However, we find that validation MSE is not a good proxy for a policy's 's real-world per- formance, leading to high MMRV and low Pearson r. On the other hand, SIMPLER evaluations more accurately reflect relative policy performances in the real world, obtaining significantly lower MMRV and higher Pearson r. Additionally, we find that an alternative implementation of SIMPLER using Variant Aggregation (Section 4.2) performs worse: since it does not seek to match real-worth visuals (Fig. 5), policies more sensitive visual distribution shifts exhibit larger real-and per-

---

Google Robot
BridgeData V2
Evaluatioon Protocal
Pick Con Move Near Drawer Arg.
>DRRV -LONL -00297
MMRV = 0.0044 0=00000
SIMPL ER TRR-Maeettnn SUC0500 nite
MMRV...
SIMPLER SUPLLERRR VRRViokaaon suocess
0.5
0.8
SIMPLER- NarAAg
Validation MSE
0.084
0.408
0.111
0.305
0.235
0.375
0.143
SIMPLER NisMlanc
0
0.111
0,027
0.056
0.6
0.8
Peanson 1
0.44
0.4
Validatonn MSE
0.464
0.230
0.231
0.308
SIMPLER- -RrrER 0.960
0J587
0.486
0.778
0.5
SIMPI FR-AisMatch
0.976
0,855
0,942
0.924
0.5
1.0
11E
0.35
Table I: Comparison of manipulation policy evaluation protocols for ranking common open-source policy check- points (3 RT-1 checkpoints. RT-I-X, RT-2-X, Octo- Octo-Base) on Google Robot tasks, See Appendix Table 2 for a de- tailed breakdown of results per policy.
Real SUcDess rate
Reall SUCDOSS rate
33 RT1 Bogie
RIT (150))
T HT (Doregged
53 RF10
+ Octa Buse 4 Octy Bred
RTX
+
A Octy Sote
Dow Jaon
Gue Oree thes Agpa
Figure 4: Real vs. SIMPLER success rates on Google Robot & Bridge V2 tasks. SIMPLER evaluations ("Visual Matching" setup) show a strong correlation with real policy performance.
formance discrepancies. In summary, SIMPLER leads to strong correlation with real-world policy performance, and we recommend Visual Matching as the default evaluation approach.
5.3 SIMPLER Evaluations Effectively Model Policy Robustness to Distribution Shifts
Previously, we showed that SIMPLER evaluations strongly correlate with based on trial averages. Beyond comparing average policy performances, it would be beneficial to let practitioners gauge more fine-grained aspects of a policy's 's behavior, such its robustness to distribution shifts like lighting, background, and texture changes. We ask: do SIMPLER evaluations accurately reflect a policy's 's real-world behavior under such distribution shifts, and can they thus be used for more fine-grained policy evaluation beyond average performance?
performances
To test this, we use SIMPLER environments to perform controlled experiments along five dis- tribution shift axes inspired by Xie et al. [60]: background, lighting, distractors, table texture, and robot camera pose. We adopt the base environment setup and the two variations per axis
from our Variant Aggregation evaluation (see Appendix B.3), adding two more variations for the new camera pose axis. We evaluate two RT-1 check-
wio Data Augmentation
MXBV =00000 0.511
SIMPLER
points trained with and without data augmentation, which exhibit different robustness behaviors to distri- bution shifts. For simulated results and real-world re- sults, we report the difference in the success rate with and without each distribution shift. The wrapped figure shows the results (more details in App. Table 4). We find that SIMPLER evaluations effectively reflect the poli- cies' robustness to various distribution shifts in the real world. Notably, in both real and sim, changing robot camera poses and table textures has a significant impact on policy performance, while the impact of lighting and distractor changes are relatively minor.
0.47
Camera pose
Taide tedtune
-0.13
-0OS
0.00
-0.6
Lighting
0.03
0.00
Buckground
w/ Data Augmentation
MXRV = 016
0 170
rata
SIMPL rata
-0.2
-0.9
Camera
Tatlo texture
-017
Badkgiound
0.08
-000
Unimestons
-004
Lighting
Furthermore, we find that an even more fine-grained analysis is possible through simulated evalua- tion. For example, when varying real-world table textures, both policies are more robust to unseen solid table colors than unseen patterned table textures (4% vs. 25% avg. performance decrease). This behavior is well reflected SIMPLER: policy performance in simulation decreases by 2% on average when new colors of the solid table are introduced and by 24% for new patterned textures.
Testing novel distribution shifts. Based on these results, we put our simulated evaluations to the test and ask: can SIM- PLER evaluations be used to predict the robustness of policies to new distribution shifts in the real world? Throughout our simulated evaluations, we observe that Octo-Base is partieu- larly sensitive changes in the simulated robot arm textures. Specifically, under our "Visual Matching" evaluation setup, its success rate is 0% the "Pick Coke Can" task using the un-
Rook
In SisPL,2 arm becure suroi boly
Octo Base
HT-1-X
db
Susoosa ruto
7

---

tuned robot arm, but 29.3% using one of our tuned robot arms. On the other hand, RT-1-X, also trained on the same Open-X-Emboddment dataset [2], exhibits higher robustness to different simu- lated robot arm textures. To test whether this trend in simulation holds in real-world evaluations, we design a novel real-world distribution shift evaluation, where we change the real robot arm tex- ture by wrapping it using multiple gift wrapping papers. We report results in the wrapfigure and Appendix Table 6. The real-world evaluations support the simulated results: Octo-Base is more sen- sitive to changes in arm texture than RT-1-X. This indicates that simulated evaluations in SIMPLER environments can be predictive of real-world policy behaviors under novel distribution shifts.
5.4
Ablation Studies
Effect of system identification. To test the impact of system identi- fication (Section 4.1), we repeat the Google Robot's Visual Matching evaluations from Section 5.2, but perturb the stiffness and damping parameters of robot joints from the results of system identification. On the right, show that the noisy system identification parameters lead to worse MMRV, i.e.,
Control Params
Courrll Loes & MMRY
Setting
0.267
0.070
Setting 2
0.452
0.100
SIMPLER SysID
0,131
0,031
worse correlation between simulated and real-world evaluations. This underlines the importance of accurate system identification for simulated evaluation.
Effeet of visual matching. We ablate the impact of our ap- proaches in Section 4.2 for matching sim and real visual ap- pearances. We use the RT-1 (Converged), RT-1 (Begin), and RT-1-X checkpoints on the Google Robot's open/close drawer tasks, and we compare the effectiveness of different combi- nations of background "green-screening", object texture, and robot texture settings. Results are reported in the wrapfigure (more details in App. Table 7). We observe the lowest MMRV and real-to-sim performance gap when combining background ggreen-screening" with object texture tuning for both drawer and robot assets. Interestingly, only tuning the drawer texture or the robot texture (but not both) is insufficient in improving the real-and-sim correlation. We hypothesize that this causes appearance inconsisten- cies between different parts of a scene, resulting in larger real-and-sim performance disparities.
Green Screen Matching Matching
Driwer
Robot
MMRV
Real-Sin Sucpess Gap J
x
X
X
0L087
0.272
x
X
0.087
0.266
x
0.087
0.272
x
0.087
0.328
0.087
0.198
X
0,142
0,253
0.050
0.136
Sensitivity to physical property gap. When developing SIMPLER environments, we simplified the physical properties (e.g., the center of mass and friction coefficients) of objects and robots due to the complexity and time-consuming nature of precise modeling and system identification. In this section, we investigate whether our simulated evaluation is sensitive to such a real-to-sim physical property gap. We conduct 2 experiments:
(1) For the "pick coke can" task, we vary the mass of the empty coke can (by varying its density), along with the static friction of the gripper finger; (2) For the "open/close drawer" task, we vary the joint frictions of the articulated cabinet. We report the MMRV and the Pearson correlation results in Appendix Tab. 8. We find that our simulated evaluation remains effective across a spectrum of plausible physical property parameters, evidenced by the low MMRV and the high Pearson correlation, even though altering these parameters has moderate (5 15%) impact on the success rates of different policies.
X
SAFTEN
MSIIV D.NEL D-22
biaac Sim MBMUV on 0.038) 2203222
Sensitivity to the choice of physics simulator. To investigate whether our results are sensitive to the underlying physics simulator, we repro- duce the Google Robot evaluation in Isaac Sim [59]. The results on the right (details in Table 9) show that SIMPLER's results are repro-
SPMPLER-/WAA waa.s//-/rggpppao accesss mie
x
ducible in Isaac Sim. In particular, we also observe strong real-to-sim performance correlation across most checkpoints for SIMPLER- This suggests that the choice of physics simulator is not critical for the
0.5
tasks we tested, which mainly involve rigid body manipulations.
0.0
0.4
6 Conclusion
Reall succoss
As generalist robot manipulation policies advance, scalable evaluation approaches become essential for the rapid development of algorithms, models, and datasets. We introduce SIMPLER, a suite of real-to-sim evaluation environments that show strong correlation to real-world policy performance. We provide detailed discussion of our current work's 's limitations in Appendix Sec. G.
8

---

Acknowledgments
We sincerely thank Jeffery Bingham and Paul Wohlhart from Google DeepMind for clarifying some details of the RT-1 Robot controller. We also thank Justice Carbajal, Samuel Wan, Jornell Quiambao, Deeksha Manjunath, Jaspiar Singh, Sarah Nguyen, Jodilyn Peralta, and Grecia Salazar for conduct- ing real-world RT-1 Robot experiments. We thank Google DeepMind for the RT-2-X alpha access. Additionally, we thank Fanbo Xiang from UC San Diego for his help in matching the real and sim- ulation visual appearances of foreground objects. Kyle Hsu was supported by a Sequoia Capital Stanford Graduate Fellowship.
References
[1] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, Haus- man, A. Herzog, Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: robotics transformer for real-world control at scale. Proceedings of Robotics: Science and Systems (RSS), 2023.
[2] O. X.-E. Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, A. Raffin, A. Wahid, B. Burgess-Limerick, Kim, B. Schölkopf, B. Ichter, Lu, C. Xu, C. Finn, C. Xu, C. Chi, C. Huang, C. Chan, C. Pan, C. Fu,
C. Devin, D. Driess, D. Pathak, D. Shah, D. Büchler, D. Kalashnikov, D. Sadigh, E. Johns, F. Ceola, F. Xia, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Schiavi, H. Su, H.-S. Fang, H. Shi, H. B. Amor, H. I. Christensen, H. Furuta, H. Walke, H. Fang, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Kim, J. Schneider, J. Hsu, J. Bohg, J. Bingham, J. Wu, J. Wu, J. Luo, J. Gu, J. Tan, J. Oh, J. Malik, J. Tompson, J. Yang, J. J. Lim, J. Silvério, J. Han, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Zhang. K. Majd, K. Rana, K. Srinivasan, L. Y. Chen, L. Pinto, L. Tan, L. Out, L. Lee, M. Tomizuka, M. Du, M. Ahn, M. Zhang, M. Ding, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. D. Palo, N. M. M. Shafiullah, O. Mces, O. Kroemer, P. R. Sanketi, P. Wohlhart, P. Xu, P. Sermanet, P. Sundaresan, Q. Vuong. R. Rafailov, R. Tian, R. Doshi, R. Martín-Martin R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Moore, S. Bahl, S. Dass, S. Song. S. Xu, S. Haldar, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, Dasari, S. Belkhale, Osa, Harada, Matsushima, T. Xiao, Yu, T. Ding, T. Davehev, Z. Zhao, T. Armstrong, T. Darrell, V. Jain, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Wang, X. Zhu, X. Li, Y. Lu, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Xu, Y. Wang. Y. Bisk, Y. Cho, Y. Lee, Y. Cui, Y. hua Wu, Y. Tang. Y. Zhu, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Xu, and Z. J. Cui. Open X- X-Embodiment: Robotic learning datasets and RT-X models. 2023.
[3] Octo Model Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, C. Xu, J. Luo, T. Kreiman, Y. Tan, D. Sadigh, C. Finn, and S. Levine. Octo: An open-source generalist robot policy. https://octo- mooioooooooosooeeeee githuu..o. 2023.
[4] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, Rettinghouse, Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, M. Yan, and A. Zeng. Do as I can, not say: Grounding language in robotic affordances. In Conference Robot Learning (CoRL), 2022.

---

[5] H. Walke, K. Black, A. Lee, M. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruce Q- Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[6] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
An open urban driving
[7] S. Manivasagam, S. Wang, K. Wong, W. Zeng, M. Sazanovich, S. Tan, B. Yang, W.-C. Ma, and R. Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11167-11176, 2020.
[8] Z. Yang, Y. Chen, J. Wang, S. Manivasagam, W.-C. Ma, A. Yang, and R. Urtasun. Unisim: A neural closed- loop sensor simulator. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-1392 2023.
the
[9] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In Conference Robot Learning, 2022.
[10] Y. Zhang, L. Ke, A. Deshpande, A. Gupta, and S. Srinivasa. Cherry-picking with reinforce- ment learning. In Roboticx: Science and Systems, 2023.
[11] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, L.. Weihs, M. Yatskar, and A. Farhadi. Robothor: An open simulation-to-ret to-rell embodied ai platform. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition. pages 3164-3174, 2020.
[12] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra. Sim2real predictivity: Does evaluation in simulation predict real-worke performance? IEEE Robotics and Automation Letters, 5:6670-6677, 2019.
[13] G. E. Mullins, P. G. Stankiewicz, and K. Gupta. Automated generation of diverse and chal- lenging scenarios for test and evaluation of autonomous vehicles. In 2017 IEEE international conference on robotics and automation (ICRA), pages 1443-1450. IEEE, 2017.
[14] K. Yang. X. Zhang, Z. Huang, X. Chen, Z. Xu, and H. Su. Movingparts: Motion-based 3d part discovery dynamic radiance field. In The Twelfth International Conference on Learning Representations, 2024.
[15] H. Jin, I. Liu, P. Xu, X. Zhang, S. Han, S. Bi, X. Zhou, Z. Xu, and H. Su. Tensoir: Tenso- rial inverse rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[16] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings the IEEF/CVF international conference on computer vision, pages 9339-9347, 2019.
[17] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.W. E. Lee, Levine, Lu, H. Michalewski. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-2: Vision-languuag--ction models transfer web knowledge to robotic control. In Conference on Robot Learning (CoRL), 2023.
[18] B. Calli, A. Singh, A. Walsman, Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. In 2015 international conference advanced robotics (ICAR), pages 510-517. IEEE, 2015.
10

---

[19] K. Van Wyk, Falco, and E. Messina. Robotic grasping and manipulation competition: Future tasks to support the development of assembly robotics. In Robotic Grasping and Manipulation: First Robotic Grasping and Manipulation Challenge, RGMC 2016, Held in Conjunction with IROS 2016, Daejeon, South Korea, October 10-12, 2016, Revised Papers 1, pages 190-200. Springer, 2018.
[20] N. Correll, K. E. Bekris, D. Berenson, O. Brock, A. Causo, K. Hauser, K. Okada, A. Rodriguez, J. M. Romano, and P. R. Wurman. Analysis and observations from the first amazon picking challenge. IEEE Transactions on Automation Science and Engineering, 15(1):172-188, 2016.
[21] E. Krotkov, D. Hackett, L. Jackel, M. Perschbacher, J. Pippine, J. Strauss, G. Pratt, and C. Or- lowski. The darpa robotics challenge finals: Results and perspectives. The DARPA robotics challenge finals: Humanoid robots to the rescue, pages 1-26, 2018.
[22] G. Zhou, V. Dean, M. K. Srirama, A. Rajeswaran, J. Pari, K. Hatch, A. Jain, T. Yu, P. Abbeel,
I.. Pinto, C. Finn, and A. Gupta. Train offline, test online: A real robot learning benchmark. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May
29 - June 2, 2023, pages 9197-9203, IEEE, 2023. doi: 109111119//RAA4889 20..2.202.0000 10160944.
[23] S. Dasari, J. Wang, J. Hong, S. Bahl, Y. Lin, A. S. Wang. A. Thankaraj. K. Chahal, B. Çalli, S. Gupta, D. Held, L. Pinto, D. Pathak, V. Kumar, and A. Gupta. RB2: robotic manipulation benchmarking with twist. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
[24] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, A. Raju, A. Laurens, C. Fantacci, V. Dalibard, M. 7ambelli, M. Martins, R. Pevce- viciute, M. Blokzijl, M. Denil, N. Batchelor, T. Lampe, E. Parisotto, K. Zoina, S. Reed, S. G. Colmenarejo, J. Scholz, A. Abdolmaleki, O. Groth, J.-B. Regli, O. Sushkov, T. Rothörl, E. Chen, Y. Aytar, Barker, J. Ortiz, M. Riedmiller, T. Springenberg. Hadsell, F. Nori, and N. Heess. Robocat: self-improving foundation agent for robotic manipulation, 2023.
[25] S. E. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Ed- wards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022.
[26] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mot- taghi. Manipulathor: A framework for visual object manipulation. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4495 2021.
[27] R. Gong, J. Huang, Y. Zhao, H. Geng, X. Gao. Q. Wu, W. Ai, Z. Zhou, D. Terzopoulos, S.- C. Zhu, B. Jia, and S. Huang. Arood:: A benchmark for language-grounded task learning with continuous states in realistic 3d scenes. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 20426-20433 2023.
[28] J. Gu, F. Xiang, X. Li. Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao. X. Wei, Y. Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su. Maniskil12: A unified benchmark for generalizable manipulation skills. In The Eleventh International Conference on Learning Representations, 2023.
[29] S. James. Z. Ma. D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019-3026, 2020.
[30] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martin-Martín, C. Wang, G. Levine, M. Lingelbach, Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, K.-Y. Kim, A. Lou, C. R. Matthews, I. Villa-Renteria, J. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, C. K. Liu, J. Wu, and L. Fei-Fei. Behavior- 1k: A benchmark for embodied ai with
11

---

1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pages 80-93. PMLR, 2023.
[31] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin,
A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021.
[32] O. Mees, L. Hermann, E. Rosete-Beat Beate, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L). 7(3):7227-734, 2022.
[33] T. Mu, Z. Ling, F. Xiang, D. C. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su. Man- iskill: Generalizable manipulation skill benchmark with large-scale demonstrations. In Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[34] X. Puig. F. Undersander, A. Szot, M. D. Cote, T.-Y. Yang, Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Y. Min, al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint wwXiv:2310.13722 2023.
[35] Srivastava, C. Li, M. Lingelbach, R. Mart'in-Mart in, F. Xia, Vainio, Z. Lian, C. Gok- men, S. Buch, C. K. Liu, S. Savarese, H. Gweon, J. Wu, and L. Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Con- ference on Robot Learning, pages
PMLR, 2022.
[36] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, A. Gokaslan, V. Vondruš, S. Dharur, F. Meier, W. Galuba, A. X. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems, 34: 251-266, 2021.
[37] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094-1100. PMLR, 2020.
[38] Y. Zhu, J. Wong, A. Mandlekar, R. Martin-Martin, A. Joshi, S. Nasiriany, and Y. Zhu. ro- bosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009. 12293, 2020.
[39] Zhang, L. Tai, P. Yun, Y. Xiong, M. Liu, J. Boedecker, and W. Burgard. Vr-goggles for robots: Real-to-sim domain adaptation for visual control. IEEE Robotics and Automation Letters, 4(2):1148-1155, 2019.
[40] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[41] Tobin, R. Fong, A. Ray, Schneider, W. Zaremba, and Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IROS, pages 23-30 IEEE, 2017.
[42] Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26)caau5872, 2019.
[43] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
[44] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference Robotics and Automation (ICRA), pages 8973-8979. IEEE, 2019.
12

---

[45] O. M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz. B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray. J. Schneider, S. Sidor, J. Tobin, Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Jour- nal of Robotics Research, (9,):3220, 2020.
[46] H. Qi. A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. In Conference Robot Learning, pages 1722-1732. PMLR, 2023.
[47] Z. Yin, B. Huang, Y. Qin, Q. Chen, and X. Wang. Rotating without seeing: Towards in-hand dexterity through touch. In Robotics: Science and Systems, 2023.
[48] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S. Levine, and V. Vanhoucke. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE international conference on robotics and automation (ICRA), pages 4243 4250. IEEE, 2018.
[49] D. Ho, K. Rao. Z. Xu, E. Jang, M. Khansari, and Y. Bai. Retinagan: An object-aware ap- proach sim-to-real transfer. 2021 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 10920-109999 10926, 2020.
[50] S. James, P. Wohlhart, M. Kalakrishnan, D. Kalashnikov, A. Irpan, J. Ibarz, S. Levine, R. Hadsell, and K. Bousmalis. Sim-to-real via sim-to-sim: Data-efficient robotic grasping via wewpuniied-tt-ccnnniin adaptation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12627-12637, 2019.
[51] K. Rao. C. Harris, A. Irpan, S. Levine, J. Ibarz, and M. Khansari. Rl-cyclegan: Reinforce- ment learning aware simulation-to-real. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11154-11 1844-1163,, 2020.
[52] Y. Du, D. Ho, A. Alemi, E. Jang, and M. Khansari. Bayesian imitation learning for end-to-end mobile manipulation. In International Conference on Machine Learning, pages 5531-5546. PMLR, 2022.
[53] K. Pearson. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240-242, 1895. ISSN 03701662. URL http://www.jstor.org stable/11579.
[54] C. Spearman. Journal of Psychology, 100(3/4)444--71, 1987. ISSN 00029556. URL http://www.jstorrormg stable/4422689
The proof and measurement of association between two things. The American
[55] M. Pincus. monte carlo method for the approximate solution of certain types of constrained optimization problems. Operations Research, 1826) 1970. ISSN 0030364X, 15265463. URL http://www..joorroog//aaalllloo0
[56] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, Dollár, and R. B. Girshick. Segment anything. 2023 IEFF/CVF International Conference on Computer Vision (ICCV), pages 3992-4003, 2023.
[57] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13142-13153, 2023.
[58] Xiang, Y. Qin, K. Mo, Xia, H. Zhu, Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. Guibas, and H. Su. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11097-11107, 2020.
13

---

[59] NVIDIA. Nvidia isaac sim. https://eveoope..viiaa.omm///// http:///iiiiiiiiiiaiaeoiiiiaaaai sim, 2022.
[60] A. Xie, L. Lee, Xiao, and C. Finn. Decomposing the generalization gap imitation learning for visual robotic manipulation. arXiv preprini arXiv:2307.03665) 2023..
[61] M. Liu, R. Shi, L. Chen, Z. Zhang, C. Xu, X. Wei, H. Chen, C. Zeng, Gu, and H. Su. One-2-3-45++: Fast single image to 3d objects with consistent multi-view generation and 3d diffusion. arXiv preprint arXiv:2311.00 1.08855, 2023.
[62] X. Huang, I. Walker, and S. Birchfield. Occlusion-aware reconstruction and manipulation of 3d articulated objects. In 2012 IEEE international conference on robotics and automation, pages 1365-1371. IEEE, 2012.
[63] Z. Jiang, C.-C. Hsu, and Y. Zhu. Ditto: Building digital twins of articulated objects from interaction. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[64] X. Wei, M. Liu, Z. Ling, and H. Su. Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions Graphics (TOG), 41(4):1-18, 2022.
[65] R. Shi, Chen, Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su. Zero single image consistent multi-view diffusion base model. arXiv preprint arX1v:2310.15110. 2023.
[66] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
[67] A. Guédon and V. Lepetit. Sugar: Surface-alignel gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5354-5366 2024.
[68] C. Chen, C. Schissler, S. Garg, P. Kobernik, A. Clegg, P. Calamia, D. Batra, P. Robinson, and K. Grauman. Soundspaces 2.0: A simulation platform for in Neural Information Processing Systems, 35:8896-8911, 2022.
learning. Advances
[69] S. Wang, M. Lambeta, P.-W. Chou, and R. Calandra. Tacto: fast, flexible, and open-source simulator for high-resolution vision-based tactile sensors. IEEE Robotics and Automation Letters, 7(2):3930-39) 3937, 2022.
14

---

A Full Environment and Evaluation Protocol Details
In this section, we provide detailed deseriptions of our SIMPLER environments along with our simulation and real-world evaluation protocols.
For the RT-1 Robot, we adopt the following language-condittoned tasks:
"pick coke can". The robot is instructed to grasp the empty coke can the table and lift it up. In the default setting, no distractors are added to the scene. We place the coke can in 3 different orientations: horizontally laying, vertically laying, and standing. For each orientation, we place the coke can at 25 grid positions within a rectangle on the tabletop, yielding 25 trials per orientation and 75 trials in total.
"move {obj1} near {obj2}"" We place a triplet of objects on the tabletop in a triangle pattern. In each trial, one object serves as the source object, one serves as the target, and the other serves as the distractor (this creates trials for each triplet and each triangle pattern). We randomly choose triplets of objects among a total of 8 objects (blue plastic bottle, pepsi can, orange, 7up can, apple, sponge, coke can, redbull can), and adopt 2 triangle patterns (upright and inverted). This creates a total of x 2 x 6 60 trials. The 5 triplets chosen are:
blue plastic bottle, pepsi can, orange
7up can, apple, sponge
coke can, redbull can, apple
sponge, blue plastic bottle, 7up can
orange, pepsi can, redbull can
"(open / close) (top / middle / bottom) drawer". The robot is positioned in front of cabinet that contains 3 drawers and instructed open close a specific drawer, testing its ability to manipulate articulated objects. We place the robot at 9 grid positions within rectangle on the floor, yielding a total of 9 x 3 x 2 = 54 trials.
"open top drawer; place apple into top drawer". The robot opens the top drawer and places the apple from the cabinet top into the top drawer, testing its ability to perform longer-horizon tasks. We place the robot at different positions on the floor and the apple at 9 different positions within a grid on the cabinet top, yielding a total of 3 x 9 = 27 trials. Initially, the policies receive the "open top drawer" instruction. We switch to the "place apple into top drawer" instruction once the robot outputs the "terminate" token after half of the time limit has elapsed.
For the WidowX + Bridge (with WidowX-250 6DOF robot), we adopt the following tasks:
"put the spoon on the towel". We place the spoon on vertex of a square (with edge length 15cm) on the tabletop, and we place the towel on another vertex. The spoon's 's initial orientation switches between horizontal and vertical, requiring the robot to perform gripper reorientation. This creates a total of x 12 = 24 trials.
"put carrot on plate". We adopt a similar setup as "put the spoon on the towel", replacing the spoon with carrot and the towel with plate.
"stack the green block on the yellow block". We place a green block on a vertex of a square on the tabletop, and we position a yellow block on another vertex. The block dimensions are 3cm. We also adopt two differently-sized squares (edge length 10cm and 20em). This creates a total of 2 12 24 trials.
"put eggplant into yellow basket". We place an eggplant on the right basin of sink, and we place yellow basket on the left basin. The eggplant is dropped into the sink at random position and orientation, and we ensure that the eggplant is directly graspable (i.e., not too close to the edges of the sink basin). We perform total of 24 trials.
15

---

Algorithm RT-1 Robot Controller in Simulation
Require: (1) Current end-effector action (Xe; Ra), along with sensed arm joint positions and velocities
Jams Cami (2) Current gripper action Ja. along with sensed gripper joint position and velocity Quips Ugipi
(3) Simulation frequency Hum. (501 in implementation). action output frequency (control frequency)
Han our implementation following [1]): (4) Arm velocity, acceleration. and jerk limits Larm (equal to 1.5, 2.0, 50.0 respectively); (5) Gripper velocity, acceleration, and jerk limits Lugip (equal to 1.0, 7.0, 50.0 respectively); (6) Current action timestep 7 within an episode; (7) A planner that takes goal and initial joint positions and velocities input (along with velocity, acceleration, and jerk constraints), and outputs pime-parametrized trajectory.
1: # Arm motion planning
2: (x, It) = ForwardKinemation (Gams)
3: (Xgoul, Rgoa) = (Xa + x. Ra Ram
4: (Isoal, Fggoa)
fnnverseKinmmatiexxx..... (xood: Rgoal Qum), 0.0)
5: ArmPlan Planner(ggout V/goal Jamns Vrr, Lumm)
# Gripper motion planning
7: T 0) then
Di At the beginning of episode
8:
Thstplan.grips Vhapplan.rpp Ogrip: 0.0
gusstgool.grip 4gap
10: end if
11: [9ul < 0.01 then
Small action filtering
12:
Oaeal.grip Qasigool.grip
13: else
14:
Gaoal.grip Nasiplan.grip + Ja
15: end if
16: Usyallerip = 0.0
17: GripPlan Planner Planner(ggol.ggr Qgool.prpp Ugaal.grip.
qustplan.grips Phauplan.ssips Lyri))
18: Execute arm and gripper plans at cach simulation step
19: for each i = 1 1...
Ham do
Has
20:
Ham
21:
Qlastplan: ArmPlan(!)
22:
SctArmlointPosTargeee Quagee))
23: Qusplan.grip Uhssplan.ssip = GripPlan(t)
24: SetGripper TinnPosTaageett( qhipplan.grip)
25:
SetGripperJoing Verripperloin/VelTaaaaee ValTargeet( (Uhoplan.rrpp) end for each
26:
27:
Quangaal.grip
Qpoal.gop
28:
For Octo simulated evaluations, since the model involves a non-detemiiiistcc diffusion head, we average its success rates across three different random seeds to produce a lower-variance estimate of the policy's simulation performance. Additionally, for RT-1 Robot simulated evaluations, we aver- age results over four versions robot arm and gripper colors to account for changes arm texture during real robot rollouts (see Section 4.2). For the WidowX environments, given the consistent black color of the arm and gripper across videos, we skip this step.
The number of evaluation trials we present above pertain to the real-world evaluation setup. For our "Variant Aggregation" simulation evaluation setup, the number of trials is multiplied by the number simulation environment variants. For our "Visual Matching" simulation evaluation setup, the number of trials is multiplied by the number of tuned robot arm colors for the RT-1 Robot evaluation setup, along with the number of seeds for the Octo policies.
B More Implementation Details of Our Real-to-Sim Evaluation System
B.1 Robot Controllers
RT-1 Robot Given translation, rotation, and gripper action output from a model, we adopt Algo- rithm simulator to execute the action commands. The simulation frequency the algorithm refers to the number of simulation steps per second, while the control frequency refers the num-
16

---

Algorithm 2 WidowX Controller in Simulation
Require: (1) Current end-effector action (Xa, R.). along with sensed arm joint positions Gom: (2) Current
gripper action Sa. along with sensed gripper joint position Qpiipi (3) Simulation frequency Ham (500 in our implementation), action output frequency (control frequency) Hen (5 in implementation following); (4) Current action timestep I' within an episode; (5) A function S that maps a R position vector and 3x3 (3(3) rotation matrix to 4x4 SE(3) matrix.
1: if T O then
D At the beginning of episode
2:
Masigoal Jarm
3: end
(x, R) = (orvar(Kiemattiossssaaaaaa
5: (Xgeal, Rgus) = S (S(x, 1) (xa, Ra).
S(-x, 1) S(x, Rum)
6: Quaal
Inverse (ggual Rgcal; Ium)
QuoaLarip Be 8: SctArmJointPos Targeet((
9: SelGripperJointPo Tanlet((gool.ppp)
10: Masgaal fgool
11: T T
ber of control commands (policy action outputs) per second. We use the source library Ruckig' for time-optimal joint motion planning with velocity, acceleration, and jerk constraints. Note that the duration of planned trajectories may exceed the interval between two control commands.
WidowX We present WidowX controller implementation in Algorithm 2.
B.2 Robot and Object Assets
Robots For RT-1 Robot, we convert the publically-released MuJoCo .mjcf robot description to URDF robot description. We also refine the collision mesh of the robot base link from the original assets to prevent erroncous mesh penctrations. For WidowX, we directly export the URDF robot descriptions from the official Interbotix repository using ROS. To simulate the RT-1 Robot, we find that the Projected Gauss-Seidel solver in PhysX causes mesh penetration behaviors during the process of object grasping. Thus, enable the Temporal Gauss-Seidel solver in both SAPIEN and Isaac Sim's simulation backends to produce correct grasping behaviors.
The RT-1 Robot uses a customized egocentric camera mounted on the robot head, while the WidowX + Bridge V2 setup uses a Logitech C920 hhiddviiw camera. We use known robot camera intrinsics possible, and when they are unknown, we obtain them from real evaluation video frames using efficient interactive GUT tools such fSpy.
Objects We adopt the following procedure to obtain object assets. Except creating precise models for articulated objects like cabinets, the process is semi-automatic, does not require extensive manual effort, and typically takes less than 20 minutes.
Obtain 3D object models from public repositories (e.g.. Objaverse [57]), from 3D scanning of objects purchased from Amazon, from single-view 3D generation (e.g., One- 2-3-45++ [6]]], or from manual modeling based on precise measurements of real-world counterparts (we only used the last technique for articulated objects like cabinets since this requires the most human effort; we highlight the acceleration of articulated asset curation process through approaches like multi-view [62] or interactive [63] articulated object gen- eration as an avenue for future work).
Process 3D object models in Blender such that the dimensions of objects are similar to those used in the real world, and that the object meshes do not contain too many vertices (to limit the sizes of object meshes).
Optionally, use our Visual Matching approach (see below) improve the texture of 3D object models.
bbapss//gitubb.c//ppaaoo//reee
17

---

Base Environment
Lighting
A Background
Texture
Pick Ooke Can
Move Near
Open/ Close
Figure 5: Subset of enviromment variations under "Variant Aggregation" evaluation setup. visualized in SAPIEN from RT-1 Robot's egocentric view. The variations cover different lightings, backgrounds and table textures and are modified from ReplicaCAD [16] scenes.
Export visual mesh and collision mesh of objects. For collision mesh, further perform CoACD [64] obtain watertight and locally convex collision meshes. Optionally, simplify the resulting collision mesh and perform minor modifications using Blender (e.g., make the bottom of cans or bottles flat).
Set the object to have a simple uniform density by querying their common material density in GPT-4 or google search, or (for objects with non-uniform densities like empty coke can), querying their mass and dividing by their visual mesh volume.
To perform visual matching of object textures, we adopt the following steps: (1) Crop the target object in a real image using SAM [56]] (2) Perform a coarse estimation of object pose by importing it into the simulation and adjusting its position such that its simulation segment mask overlaps with the real one; (3) Employ differential rendering (using Nvdiffrast) optimize the simulation asset's pose such that precisely aligns with the real object's segmentation mask; (4) "Unproject" the real object's i RGB texture values onto the simulation object mesh; (5) Optionally, generate the remaining views of the object through a diffusion model (Zero 23++ [65]), and refine the poses of novel views using a rendering loss with the existing object view. Finally, unproject the novel view textures onto the simulation object mesh. This whole process semi-automatic, and can thus be completed efficiently. We commit to release convenient command line python script for this process.
B.3 SIMPLER-Variane Aggregation
A common approach for addressing visual gaps in sim-to-real policy training is domain random- ization. By performing training across range of randomized parameters, such as textures and lighting, prior works aim to obtain policies that are robust to visual distribution shifts in the real- world [40, 41]. Similarly, in real-to-sim evaluation, we can aggregate evaluation results across a range of visual simulator characteristies to obtain more faithful signal for the policy's perfor- mance. In practice, we implement this SIMPLER "Variant Aggregation" approach as an alternative to SIMPLER- Visual Matching, described in Section 4.2. Concretely, of our simulation environment and then creating "variants" of this enviromment along four axes of visual variation: background, lighting, distractors, and table texture. For each axis, we construct 2 variations of the base setup similar to [60], covering backgrounds from different rooms, lighter and darker lighting, fewer and more distractors, and solid color and complex table textures. We visual- ize an example of such simulator variations for various table-top tasks in Fig. 5. We average policy performance in simulation across all variants of an environment to obtain our final performance estimate.
create "base" version
18

---

Pick Coke Can
Move Near
Open Close Drawer
Open Top Drawer
and Place Apple
Exalation Setup
RT-1 Ruboo
Policy
Herizontal Loying
Vertiall Laying
Standing
Average
Average
Open
Close
Average
Average
KI-1 (Converged)
0,960
0,880
0,720
0.85.
0.633
0.815
0.926
0,870
0.185
KT-1 (15%)
1.000
0.960
0.800
0.920
0,583
0.704
0.889
0.796
0.185
RT: 1-X
0.880
0.560
0.840
0,760
Real Eval
KI-2-X
0.920
0.800
1.000
0.907
0.733
0.333
0.630
0.481
0,0774
Octo Base
0.440
0.200
0.230
0.293
0.350
0.148
0.519
0.333
0,000
DgenNLAA-78
0.640
0.280
0.360
0.130
0,000
RT-1 (Regin)
0.200
0.000
0.200
0.133
0.017
0.000
0.000
0.000
0,000
KT-1 (Converged)
0.969
0.760
0.964
0.898
0.500
0.270
0.376
0.323
0,026
RT-1 (15%)
0.920
0.704
0.813
0.813
0.446
0.212
0.323
0.267
0,021
RT 1-X
0,569
0.204
0.698
0,490
0.323
0,059
0.519
0.294
0.101
SIMPI FR
0.754
0.206
(Virinst Aggregation)
Octo Base
0.005
0.000
0.013
0.006
00011
0.000
0.021
0.011
0,000
OLe 7B
0.644
0,218
0.729
0,530
0.469
0.148
0,164
0.156
0,000
RT-1 (Begin)
0.022
0.013
0.031
0.022
0.040
0,005
0.132
0.069
0,000
MMRV
0.340
0.105
0.378
Pearson +
0.927
0.977
0.824
0.922
0.881
0.648
0.728
0.313
(Converged)
0.900
0.700
KI-1 (15%)
0.360
0.790
0.480
0.710
0.354
0,463
0.667
0.565
0.130
KF-1XX
0.820
0.330
0.550
0,567
0.317
0.296
0.891
0.597
0,213
SIMPLER
K1-2-X
0,740
0.740
0,800
0,787
0.343
0.250
0,015
(Visual Matching)
OpenVLAA-BB
Base
0.310
0.030
0.210
0.090
0.190
0.170
0.177
0.492
0.042
0.250
0.444
0.574
0.227
0:412
0.000
0,000
KI-1 (Begin)
0,050
0,000
0.030
0:027
0,050
0.000
0.278
0.139
0,000
MMRV
0.023
0.046
0.046
0.027
0L055
0.009
0.265
0,177
0,000
Pearson r1
0.963
0,951
0.948
0:969
0.864
0.914
0.674
0.825
0,973
Table 2: Real-world and SAPIEN evaluation results across different policies on RT-1 Robot tasks. We present success rates for the "Variant Aggregation" and "Visual Matching" approaches in Sec. 4.2, We calculate the Mean Maximum Rank Violation ("MMRV", lower is better) and the Pearson correlation coefficient ("Pearson r", higher is better) to assess the alignment between simulation and real-world relative performances across different policics.
Widow--Bridge
Policy
Spocn
Carror on
Yellow Black
Eggpllant
Evaltattoon Serap
Grasp Spoen
Success
Grasp Carror
Suceess
Grusp Block
Suecess
Graup Eggplent
RT-1-X
0.042
0.000
0.167
0,000
0,000
0.000
0.033
0.000
Octo
0,500
0,250
Octo Small
0,562
0.417
0.208
0.083
0,583
0.125
0,700
0.433
KF-1-X
D.167
0.000
0.942
SIMPLER Eval
Deto-Base
0.347
0.12
0.528
0.083
0,319
0.000
06677
0.31
(Visual Matching)
Deto-Small
0.778
0.472
0.278
0,097
0.403
0.042
0.755
0,569
MMRV
0.000
0.000
0.000
0,111
0.000
0.000
0.000
0.000
Pearson
0.778
0.827
0.995
0,575
0.964
1.000
0.995
0.990
Table 3: Real-world and SAPIEN simulation evaluation results for the WidowX Bridge sctup. We report both the final success rate ("Success") along with partial success (e.g., "Grasp Spoon").
C Full Results for Real-and-Sim Relative Policy Performance Correlation Experiments
In Table 2 and Table 3, we present full evaluation results for our experiments in Sec. 5.2, which demonstrate that SIMPLER environments show strong performance relationship correlations with real-world evaluations.
19

---

Pick Coke Can
Move Near
Avg.
Real TableTop [60)
Policy
Distribution Shift
1A Successi
MMRV
1
J4 Success
MMRVI
Success
MMRV
Success]
Background
RT-1
Aug
Tuble Texture
Lighting
Distractors
0,027
0,113
0,940
0,000
0,779
0.175
0.075
0.123
0,055
0.939
0,144
0.057
0,080
0,000
0,331
0.389
0.033
0.111
Camera
0.192
Background
0.002
0.167
Lighting
0.117
KF-1 +Aug
Distractons
0.033
0,04]
0.984
0.084
0.125
0.721
0.059
0.041
0.970
0.083
Table Testure
0,220
0.159
0.
0.167
Camera
Table 4: Impact of various distribution shifts the tabletop manipulation performance of RT-1 policies trained with and without image augmentation. SIMPLER evaluations accurately track the policies' robustness to dis- tribution shifts, exhibiting low Mean Maximum Rank Violation ("MMRV") and high Pearson correlation coef- ficient ("r") with the real world evaluations [60].
Policy
Robustness Factor
Pick Coke Can
Move Near
Base Sctup
0.920
0.467
Background
0.933/09007
0.533/05667
RT-1
Lighting
0.960/0 960
0,483/0.600
w/o Aug
Distractors
0.880/0.900
0.600"
Table Texture
0.867AU.747
0.550/02000
Camora Pose
0.053/0,280
0.117/0.433
Base Setup
0.800
0.383
Background
0.747/0.547
0.483/0.467
RT-1 +Aug
Lighting
0.760/0.773
0.517A0.483
Distractors
0,813/0,747
0,467
Tahle Texture
0.667/01499
0.45000133
Camera Pose
0,267/0.107
0,2000.217
"The base setup environment already contains dis- tractors, so we construct environment variants without
distractors.
Table 5: Success rates of different out-of-distributis generalization factors on the tabletop manipulation per- formance of RT-1 policies in the SAPIEN simulator. "a/b" denote results different environment variants (lighting: darker brighter; table texture: solid color contrastively patterned; camera pose: oriented lower / higher).
D Full Results for Real-and-Sim Policy Behavior Correlation Experiments under Environment Distribution Shifts
In Table 4, Table 5, and Table we present full evaluation results for our experiments in Sec. 5.3, which demonstrate that SIMPLER environments show strong policy behavior correlations with real- world evaluations under different environment distribution shifts.
E
Full Results for Main Paper Ablation Experiments
We present detailed results for main paper's ablations in Table 7, Table 8, and Table 9.
After running 2 real evaluation trials, robot operators decided that since this policy would potentially damage the robot on the Drawer tasks, the real evaluation was terminated. As real evaluation was terminated due to risk of damaging the robot, we expect the MMRV to be less than this number if real evaluation were continue.
Real Success
Policy
Sim Success Range
Onig Texture
OOD Texture
105507, 0.653]
0.760
Octo-Base
10.000, 0,293]
0.293
0.000
Table 6: Impact of arm textures on the success rates of "Pick Coke Can" task in the SAPIEN simulator (Visual Matching evaluation setup) and in the real-world. The ranges of simulation success rates across multiple (tuned and untuned) robot colors can predict policy sensitivity real-world OOD textures.
20

---

Components
Open Drawer
Closs Drower
Background
Drawer
Robot
RT-1 (Canvarged)
RT-1 (15%)
RT-I-X
MMRV
RT-1 (Comergol)
RT-1 (15%))
RT-1XX
MMRV]
Real
Real
Real
0.815
0.704
0.519
NIA
0.926
0.389
0.741
N/A
GreenSereen
Curated
Curated
0,703
0.556
0.333
0.000
0.889
0.667
0,331
0,899
GreenScreen
Curated
Odginal
0.444
0,406
0.259
0.111
0.741
0.630
0.173
GreenSereen
Original
Original
0.519
ReplicCCAD
Curated
Curated
0.4077
0.259
0.111
0.000
0.667
0.481
0.778
0.173
ReplicaCAD
Curated
Orignnal
0,600
0.407
0.075
0.000
0.630
0.593
0.067
0.173
ReplicoCAD
Curated
0.556
ReplicCCA
Original
Original
Table 7: Impact of real-to-sim visual gaps on real-and-sim performance correlations. We report the success rates of different policies tasks: Open Drawer and Close Drawer. The settings with the smallest MMRV and the smallest absolute performance gap with the real performance are highlighted. Using a combination of "green srreen-crrened" background and curated foreground object and robot assets provides the best correlation.
Gripper Friction Coefficient
Coke Can Mass
0.25
0.50
1.0
2.0
10
0.957
0.967
0.971
0.978
20
0.969
0.975
0.978
0.977
g
0.963
80
0.962
0.962
0.975
0.990
(a) Pearson r between real and SIMPLER evaluations on the Pick Coke Can task under different settings can mass and gripper friction coefficient. The MMRV is 0.031 in all cases. The use of empty coke cans follows the setup from the RT-1 Robot demonstration dataset and the RT-1 paper [1].
Cabinet Joint Friction
0.0125
0.025
0.05
0.10
0.15
0.20
MMRV] Pearson rt
0,055
0.055
0.055
0.055
0.105
0.055
0.930
0,941
0,915
0,923
0.903
0.928
(b) MMRV and Pearson between real and SIMPLER evaluations on the Open/Close Drawer tasks under different settings of cabinet joint friction.
Table 8: SIMPLER is robust to imprecisely estimated physical simulation parameters such as object mass and friction coefficients, indicated by the low MMRV and high Pearson 7 in both ablation studies. We use the 6 policies from RT-1 Robot experiments in these ablations.
Coke
Move Near
Evaluction Sctup
Policy
Horizontal Laying
Vertical Laying
Stunding
Avg.
Avg.
(Converged)
0.880
0.853
RT-1 (15%)
1,000
0.960
0.500
0.920
0.583
Real Eval
RT-1XX
0.380
0.560
0.340
0.760
0.450
Octo-Bar Base
0.440
0.200
0.240
0.293
0.350
(Begin)
0.133
0.017
RT-1 (Conserged)
0410
(10%)
0.428
0.441
SIMPLER Eval
KI-1-X
01340
0.182
0.618
0.380
0.125
(baac, Viriant Aggn:)
Dcto-Rase
0.055
0.020
0.010
0.055
0.020
RI-1 (Begin)
0.04
MMRV
0.096
0.112
0.016
0.044
0.053
0.961
Table 9: Real-world and Isaac Sim evaluation results for the RT-1 Robot setup. The findings on Isaac Sim are consistent with the findings on the SAPIEN simulator.
21

---

Task
Validation Action MSE Sim Eval (Visual Matching)
Pick Coke Can
0,412 / 00.64
0,031 /0.976
Move Neur
0.408 / 002230
0.111 /0.855
Open Cloose Drawer
0.346 00.264
0,055 /0.915
Open Drawer& & Place Apple
0.265 /0. 198
0.000/ /.966
Put Sporn Towel
0.359 -0.951
0.000/0 0.827
Put Carrot on Plate
0.194 -0.342
0.111 /0.575
Stack Block
0.125 -0.857
0.000/ 1,000
Put Eggplant Basket
0.366 /1.000
0.000/ /0.90
Table 10: MMRV Pearson correlation comparison between our Visual Matching simulation evaluation ap- proach and the simulation-free approach that assesses the MSE between predicted and ground-truth actions on validation trajectories. For the latter approach, we calculate the MMRV Pearson correlation between the negative MSE and the real policy performance. Our approach yields significantly better real-and-sim policy performance correlations.
Arg. Sim Snccess (Visual Matching)
Policy
Avg. Real Success
RT-1 (Convergod)
0.853
0.857
RT-1 (15%)
0.920
0.710
RI-1 (Single Task Policy)
0.680
0.403
RT-1-X
0.760
0.567
RT-2-X
0.907
0.787
Octo Base
0.293
0.170
RT-I (Regin)
0.135
0.027
MMRV
0.027
Pearson rf
0.959
Table 11: Real-world and simulated evaluation results on the Pick Coke Can task, after adding an RT-1 policy trained solely on the Pick Coke Can demonstrations. Our simulated evaluation remains effective, exhibiting low MMRV and high Pearson correlation coefficient with real evaluations.
F
More Experiment Results
F1 More Ablations
Simulated vs. simulation-free evaluation approaches: To evaluate and select real-world robot ma- nipulation policies, widely-adopted approach involves calculating the MSE loss between predicted and ground-truth actions a set of held-out validation demonstration trajectories. We are thus in- terested in the following question: Does simulated evaluation produce significantly better real-to- sim relative performance correlation than simulation-free approaches? We conduct an experiment where we calculate the action-pred prediccion MSE loss on the RT-1 Robot dataset and the Bridge dataset. For the Bridge dataset, we randomly select 25 trajectories from the validation demonstration split. For the RT-1 Robot dataset, a validation split is not publicly available, we randomly select 25 trajectories from the training demonstrations.
We report the results in Table 10. We find that SIMPLER evaluation produces significantly bet- ter correlations between real-and-sim performances across different policies, as highlighted by a substantially-lower MMRV and a substantially-higher Pearson correlation coefficient. Furthermore, as demonstrated in Sec. 5.3 of the main paper, SIMPLER evaluation reveals finegrained policy behavior modes, such as robustness visual distribution shifts, offering insights beyond policy performance comparisons, unlike simulation-free evaluations.
Is simulated evaluation still effective on single-task policies? Previously in the main paper, we focused our simulated evaluation on policies trained on multi-task datasets, such as the RT-1 Robot RT-1 dataset and the Open-X Open-X-Emboon Embodimmnnt dataset, which contain 280k demonstrations. In this section, we further ask the question: Is SIMPLER evaluation still effective on policies trained on smaller-scale data, which are potentially more sensitive to real-to-sim visual and control gaps? To this end, we conduct experiment where we train RT-1 only with the "pick coke can" demon- strations and evaluate its real and simulation performance. We also compare the MMRV and the
22

---

Coke Can
Move Near
Open / Close Drawer
Open Top Drawer and Place Apple
Evaluation Serap
RT-1 Robot
Metric
Horizontal Laying
Vertical
Laying
Standing
Arg. Success
Arg. Success
Open
Close
Arg. Success
Arg. Success
SIMPLER (VisMarch) Kruskal Kruskal-fFollcy p.00.00
Wiriwe+ Evalustion Sctup
Metric
Spoon
Put Carrot Plate
Stack Cieenn Blokk
Yellow
Pur Figplosr in Yellow Racket
Graup Spoon
Graup
Success Grasp
Grasp Eggplant
Success
SIMPLER (VisMatch) Kruskal-PPolicy p-00.0
(b)
For our Visual Matching evaluation approach, conduct Kruskal- Wallis test to assess whether simulation and real-world policy evaluations exhibit significant distribution shift, even though we not expect to obtain an exact reproduction of real-world performance.
Table 12:
Pearson correlation before and after incorporating this single-task policy into the RT-1 Robot exper- iments. Results are shown in Table 11. We find that our simulated evaluation effectively reflects the performance rankings of the newly-added single-task policy, with the MMRV remaining low and the Pearson Coefficient remaining high. This demonstrates SIMPLER evaluation's versatility across policies trained on diverse data scales.
F.2 Other Metrics: Kruskal Wallis
In our previous analysis, we primarily focused on metrics that measure real-to-sim relative per- formance alignment between policies. As match real-to-sim visual input appearance in Visual Matching evaluation approach, it also becomes meaningful to measure the simulation distri- bution shift of absolute performance from real-world evaluations, even though we do not expect the real-to-sim absolute performances to exactly match. Let the real-world evaluation results of N policies be r = 12,, .N J. where r; = (rig)) Nana is the indicator each trial's success in the worldworld. Let the corresponding simulation evaluation results be s = {ss, ....... SN where
s; = (str) We perform Kruskal- Wallis test for each individual policy (i.e., between each r, and 8,) to measure whether simulation evaluations exhibit significant distribution shift from real evalua- tions. We then report the number of policies with significant distribution shift (which we denote as "Kruskal-#Policy p<0.055")
We present the Kruskal- Wallis results in Tab. 12. We find that with the Visual Matching evalua- tion approach, the simulation trial success distribution not significantly different from the real results (p 0.05) across many tasks and policies, demonstrating the effectiveness of our simulation evaluation tool. We also note that MMRV and the Kruskal metrics complement each other's limitations, with the former providing real-to-sim relative performance alignment perspective, and the latter providing an absolute performance alignment perspective.
G Limitations of Our Current Work
Our current work has several limitations: (1) We focus our evaluations on rigid-object manipulation tasks, as they are most straightforward to simulate. Extending simulated evaluation to soft-object tasks is an exciting avenue for future work. (2) For our Visual Matching evaluation approach, the background "green- screening" technique limits evaluations to fixed cameras and does not accurately capture object shadows and other visual details (our Variant Aggregation evaluation does support variable camera poses). (3) Our current pipeline for generating simulated evaluation environments involves some manual effort in curating assets and assembling scenes. Enabling a fully automatic and more scalable pipeline for creating thousands of realistic simulated environments is an ambitious goal for future work. In particular, promising future direction is to automatically reconstruct real- world scenes and backgrounds with high-qualin, geometries, materials, and lighting, while enabling
23

---

fast simulation of these scenes. In the beginning of the project, we did preliminary investigations using state-of-the-art approaches in Gaussian Splatting [66, 67], but found the scene quality to be insufficient for real-to-sim manipulation evaluation (e.g., uneven table surface geometries, floaters while rendering from novel camera views). Additionally, importing Gaussian Splatting-based scenes into simulation environments would entail non-trite non-trivial changes to the existing simulators' rendering pipeline, which we leave for future work. (4) Our current simulated environments limited to RGB image and proprioceptive policy inputs. We believe that future simulated evaluation tools could incorporate other modalities such as LIDAR [6], acoustic [68], and tactile sensors [69] as research on their effective and accurate simulation continues to advance. (5) We adopt a simple system identification approach that we find to be adequate for scenarios like rigid object grasping and articulated furniture manipulation. It unlikely that system identification using open-loop contact- free trajectories will be sufficient accurately model robot control parameters for situations such as high-speed robot-object collisions and dexterous manipulation.
24


Reviews:
[{"review_id": "ZwxLAOKPJL", "reviewer": "Reviewer_SfVk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a framework called SIMPLER for validating manipulation policies in simulation that have been trained with real robot evaluation.", "summary_of_recommendation": "I would be ok to accept this paper if others fight for it. But my recommendation is rather borderline (which is not possible here).", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "There are no questions from my side.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6ykMJVxx2", "reviewer": "Reviewer_VgL2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a simulation framework, called SIMPLER, for evaluating real-robot manipulation policies at scale. The authors propose to close the sim-real control gap via system identification using a small sample of real trajectories and the sim-real appearance gap by matching real backgrounds and textures. The simulation performance is shown to strongly correlate with real robot performance, and is shown to be indicative of the relative sensitivity of real robot policies to different distribution shifts.", "summary_of_recommendation": "Proposing acceptance as the developed framework holds value for reproducible benchmarking of real robot manipulation policies.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Elaborated under weaknesses. Critically would like to know an estimate of manual hours spent preparing the environments, and cause of poor correlation in select cases.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OvLjnFzt3b", "reviewer": "Reviewer_mzQD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper explores the challenging problem of simulation-based evaluation of robot manipulation policies trained from real world data. The key insight is to develop realistic simulated environments that are realistic enough by overcoming control and visual disparities, so they can serve as a scalable and repeatable tool for evaluating real-world manipulation policies. The paper proposes different simulated environments and a ranking metric for policy evaluation showing strong correlation in policy performance between real-world and sim for open-source RT1-X and Octo manipulation policies.", "summary_of_recommendation": "I am recommending weak accept based on the points mentioned in weaknesses and questions above. I am willing to update my recommendation if authors sufficiently address them.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Q1. Sec 3: it is unclear how the ranking metric behaves if the simulated environment consistently overestimates the real-world performance of policies. In this case, even though we have a strong correlation, simulated evaluations could mislead us to believe that a bad real-world policy is good. How does the current ranking metric overcome this issue? \n\nQ2. Sec 4.1 :  The offline system identification approach does not clearly address the coupling between end-effector motion and object properties when in contact. Does the offline dataset used for system identification also span different objects being manipulated or other potential contact surfaces in the environment?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper appears to have been relatively well-received by the reviewers, with a number of notable strengths. 

-well written and well-structured
-clear in its motivation
-timely 
-algorithm and approach is well-defined and well-described

But with some weaknesses:

-apparent use of extensive manual labour in capturing textures etc. (reviewer VgL2)
-lack of thorough discussion on weaknesses and limitations
-uses only camera poses
-unclear sim2real transferability according to some of the reported results, which requires some investigation and explanation
-does not consider some SOTA techniques, e.g., NERFs.

The authors should clarify these issues
I encourage the authors to address the comments of the reviewers to increase chances of acceptance.


Updated:

The authors have been responsive to the reviewer's comments, run extra experiments, and tidied up the text.
Decision: Accept