Evaluating Real-World Robot Manipulation Policies in Simulation

Authors: xuanlin  li
kyle  hsu
jiayuan  gu
karl  pertsch
† oier mees
homer rich walke
chuyuan  fu
ishikaa  lunawat
isabel  sieh
sean  kirmani
sergey  levine
jiajun  wu
chelsea  finn
hao  su
quan  vuong
ted  xiao
u c berkeley
google  deepmind
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-05-14 05:23:19

Abstract: The field of robotics has made significant advances towards generalist robot manipulation policies. However, real-world evaluation of such policies is not scalable and faces reproducibility challenges, issues that are likely to worsen as policies broaden the spectrum of tasks they can perform. In this work, we demonstrate that simulation-based evaluation can be a scalable, reproducible, and reliable proxy for real-world evaluation. We identify control and visual disparities between real and simulated environments as key challenges for reliable simulated evaluation and propose approaches for mitigating these gaps without needing to painstakingly craft full-fidelity digital twins. We then employ these techniques to create SIMPLER, a collection of simulated environments for policy evaluation on common real robot manipulation setups. Through over 1500 paired sim-and-real evaluations of manipulation policies across two embodiments and eight task families, we demonstrate strong correlation between policy performance in SIMPLER environments and that in the real world. Beyond aggregated trends, we find that SIMPLER evaluations effectively reflect the real-world behaviors of individual policies, such as sensitivity to various distribution shifts. We are committed to open-sourcing all SIMPLER environments along with our workflow for creating new environments to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. Website: https://simpler-env.github.io/

Main paper: 
 introduction: 
 Remarkable progress has been made in recent years toward building generalist real-world robot manipulation policies [1, 2, 3, 4, 5] , i.e., policies that can perform a wide range of tasks across many environments and even robot embodiments. However, evaluating these policies in a scalable 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. import simpler_env env = simpler_env.make( "google_robot_pick_coke_can" ) policy = load_policy() env.reset() env.step( policy.sample_action() )Figure 2 : We introduce SIMPLER, a suite of simulated evaluation environments for common real robot manipulation setups, namely the Google Robot evaluations from the Robotics Transformer series of works [1, 2, 17] , and environments from the BridgeData V2 dataset [5] (see Appendix for detailed environment descriptions). We make using SIMPLER easy via exposing a standard Gym interface. Additionally, we commit to open-source policy inference code for real-to-sim evaluation of common generalist robot policies (RT-1 [1] , RT-1-X [2] , and Octo [3] ), and provide detailed guides for evaluating new policies and creating new evaluation environments. and reproducible way remains challenging, as real-world evaluation is expensive and inefficient. Compared to the evaluation burden of works that study robot performance in narrower settings, the scope of evaluations required for faithful performance estimates of generalist policies increases with the breadth of their abilities. This underlines a growing challenge in robot manipulation research: as we scale the capabilities of robot policies, how do we correspondingly scale our ability to accurately, reproducibly, and comprehensively evaluate them?In this work, we propose simulated evaluation as a possible answer, in which manipulation policies trained on real data are evaluated in purpose-built simulated environments (Fig. 1 ). Such real-to-sim evaluation can serve as a scalable, reproducible, and informative tool to complement gold-standard real-world evaluations. Indeed, evaluation in simulation is common practice for testing autonomous driving policies across a wide range of scenarios before real-world deployment [6, 7, 8] . However, performing simulated evaluations for robotic manipulation poses additional challenges due to the diverse interactions between the agent and the environment. At the same time, research on sim-toreal policy learning [9, 10] has demonstrated that considerable transfer between simulation and the real world is possible even for manipulation policies. While sim-to-real approaches typically train in simulation and evaluate in the real world, we are interested in the opposite question: how can we build systems for evaluating manipulation policies trained on real data in simulated environments? One option to build such simulated environments is to fully replicate an existing real-world environment by creating a simulated "digital twin", an approach popular in navigation [11, 12] and autonomous driving [13] . However, for robot manipulation, reconstructing dynamic and interactive objects [14] , along with realistic materials and lighting [15] in simulation remains an open research question. Furthermore, building full-fidelity digital twins demands extensive time and resources, typically requiring digital artists to manually craft object geometries and materials [11, 16] . Capturing precise physical properties of objects for manipulation simulation, such as the center of mass, inertia, and static and dynamic friction, further complicates scalability.A key idea in this work is that we do not need to build exact replicas of real-world environments. Instead, we aim for simulated environments that are merely realistic enough, such that the performance of policies evaluated in simulation correlates well with their real-world performance. This allows us to design environment creation pipelines that are more scalable than creating exact digital twins. Through extensive experiments, we examine the challenges of building effective simulated evaluation pipelines, specifically control and visual disparities between real and simulated environments. We then propose and evaluate approaches for mitigating these differences based on offline system identification, "green-screening" simulation observations using real-world backgrounds, and object texture baking from real-world images.Using these techniques, we create SIMPLER, a suite of simulated manipulation policy evaluation environments for commonly used real robot setups, namely the RT-1 [1] and BridgeData V2 [5] evaluation setups (Fig. 2 ). For both setups, we perform extensive paired sim-and-real evaluations for multiple open-source manipulation policies such as RT-1-X [2] and Octo [3] , and we demonstrate a strong correlation between policy performance as assessed by SIMPLER and the corresponding real environments (Fig. 1 , right). In addition, we find that simulated policy evaluations in SIM-PLER environments accurately reflect policy behavior modes in the real world, such as sensitivity to various distribution shifts. As such, SIMPLER is a first step toward using simulated evaluation as a tool for reliable, scalable, and reproducible manipulation policy evaluation. We are committed to open-sourcing our workflow for constructing SIMPLER environments to facilitate research on general-purpose manipulation policies and simulated evaluation frameworks. 
related work: 
 Reproducible evaluation of real robot policies is a long-standing challenge. Initiatives like YCB [18] and NIST [19] were introduced to standardize evaluation objects, yet standardizing other variables such as lighting, camera setups, and workspaces proves difficult. Other benchmarks like the Amazon Picking Challenge [20] , DARPA Robotics Challenges [21] , TOTO [22] , and RB2 [23] require ongoing maintenance of real-world evaluation setups, representing a significant long-term investment. As real-world robotic datasets [1, 2, 5] and generalist policies [24, 17, 3, 25] proliferate, the demand for reliable, scalable, and reproducible methods of evaluating these policies grows. The need is particularly acute given the difficulty faced by the research community in conducting evaluations without standardized hardware.Simulation-based algorithmic research offers an alternative to real-world evaluation. A wide range of simulation benchmarks [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38] have been established to facilitate scalable and reproducible evaluation. However, most previous work considers both training and evaluation in simulation, and the resulting policies might exhibit distinct behaviors when deployed on real robot hardware. In contrast, we aim to both measure and enhance simulated evaluations' ability to reflect a policy's real-world performance and behaviors.Can simulated evaluations reliably predict real-world policy performance and behavior modes? While several studies [12, 11, 39] have explored this question in navigation tasks, we focus on simulation evaluation for real-world manipulation policies. Manipulation presents unique challenges due to the tight interaction loop between policy and environment, dynamic rather than static scenes, along with the intricate, precise, and complex action sequences where even slight variations can significantly impact task outcomes.For robot manipulation, the sim-to-real setting has been extensively investigated, where one aims to train policies in simulation and deploy them in the real world. To tackle the gaps between simulation and reality, prior work has adopted domain randomization [40, 41, 42, 43, 44, 10, 45, 46, 47] and domain adaptation [48, 49, 50, 51, 52] approaches. For instance, Generative Adversarial Networks (GANs) [48, 49, 50, 51] are trained to modify images generated in simulations so they resemble the style of real-world images. Alternatively, Du et al. [52] aim to align the feature space of observations between simulated and real-world environments, creating a more consistent visual experience across these domains. In contrast to these methods of sim-to-real learning, we focus on the opposite question: building simulation systems that effectively and faithfully evaluate real-world robot manipulation policies. To this end, we introduce approaches to addressing both real-to-sim visual and control gaps to enhance real-&-sim evaluation correlations. 
using physics simulators for evaluation of robot manipulation policies: 
 Problem formulation. We study the problem of using physics simulators to evaluate performance and examine the behavior modes of real robot manipulation policies. We emphasize that our goal is not to completely replace real-world evaluations or to perfectly replicate real-world policy behaviors in simulations, as there are always gaps between sim and real. Instead, we aim to achieve strong correlation between relative policy performances in the real world and in simulation. This gives practitioners a readily available policy improvement signal to guide their research. Formally, for two policies π a and π b with real-world performance measures R a and R b -such as their success rates across tasks -we aim to construct a simulator S (with performance measures R S,a and R S,b ) that achieves strong correlation between relative performances in simulation and in the real world. r , range [-1, 1], higher is better) for assessing the policy performance correlation in real-world and simulation, as well as the overall quality of simulated evaluation pipelines. Each circle represents a policy. For the two leftmost pipelines, both metrics yield valuable insights, identifying one as poor and the other as good. The two rightmost examples highlight the limitations of Pearson r: it can penalize simulation pipelines that fail to linearly recover the real results despite recovering the correct ranking, and it is overly sensitive to minor noise in evaluations when different policies perform similarly in the real world.Metrics for real-to-sim evaluation pipelines. A standard approach for measuring the correlation between two variables is the Pearson correlation coefficient (Pearson r) [53] . However, Pearson correlation has two important limitations for judging simulated evaluation pipelines: (1) It only assesses the linear fit between real and simulated performance, which is not necessarily required so long as the simulated pipeline reflects real-world performance improvements between different policies (Fig. 3 , middle right); (2) It does not reflect the range of values it is computed over. Thus, for policies performing similarly in the real world, r may change drastically based on the inherent noise in real-world evaluations (Fig. 3 , far right grey vs. green).To address the first drawback of the Pearson correlation, we can additionally report a ranking metric that measures whether the simulated evaluation ranks the policies correctly based on their real-world performance, independent of whether the relationship is linear. However, conventional ranking metrics such as Spearman's rank correlation [54] still suffer from the second shortcoming: they operate purely on the rankings and disregard the underlying margins between real values. When choosing a suitable ranking metric, the key point is that we need to take the magnitude of the rank violation into account, measured as the difference in real-world performance between the mis-ranked policies. This provides a clear signal whether rank violations are caused by small real-world performance differences that are often the result of inherent noise in real robot evaluations, as the far-left example of Fig. 3 , or constitute clear failures of the evaluation pipeline, as the middle-left example of Fig. 3 . Thus, we propose the mean maximum rank violation (MMRV) metric to better assess the consistency of the real-and-sim policy ranking. Given N policies π 1...N and their respective performance measures (e.g., success rates) R 1...N , R S,1...N from real and simulated evaluations, we have:EQUATIONThe key underlying quantity is the rank violation between two policies π i and π j , which weighs the significance of the simulator's incorrect ranking by the corresponding real-world performance margin. MMRV then averages each policy's worst-case rank violation. We will report both MMRV and Pearson r as they provide complementary perspectives on the simulated evaluation's effectiveness. 
simpler: simulated manipulation policy evaluation for real robots: 
 This section introduces our approach to designing a simulation evaluation pipeline for real robot manipulation policies. We take inspiration from the rich literature on sim-to-real policy learning [45, 40, 41, 10] and focus on control and visual gaps between simulation and the real world.First, we need to ensure that the policy actions executed in simulation yield comparable effects on the system state as those observed when executed on the real robot. Concretely, let{(x i , R i ) : x i ∈ R 3 , R i ∈ SO(3)} T i=1be a 6D end-effector pose trajectory recorded when rolling out an action trajectory{a i } T i=1 on a real robot. Let {(x ′ i , R ′ i ) : x ′ 0 = x 0 , R ′ 0 = R 0 } T i=1 = Sim(p, d, {a i } T i=1 ,x 0 , R 0 ) be the corresponding trajectory when unrolling the same sequence of actions in the simulation in an open-loop manner using stiffness and damping parameters (i.e., PD parameters) (p, d). Then, we define our system identification loss from translation and rotation errors asL sysid (p, d) = 1 T T i=1 ||x i -x ′ i || 2 + arcsin 1 2 √ 2 ||R i -R ′ i || F .In practice, we use a small sample of 20 trajectories from an offline dataset D, e.g., a real-world demonstration dataset, to retrieve action and end-effector pose trajectories and compute the system identification loss above. For all environments in this work, we use trajectories from existing opensource demonstration datasets [1, 5] , and thus do not need to collect any new data. We also ensured that each trajectory comes from different task instructions to encourage trajectory diversity.Next, we optimize the parameters of our controller: given initial PD parameters (p 0 , d 0 ) and search ranges [p low,0 , p high,0 ], [d low,0 , d high,0 ], we normalize the range to [0, 1] and perform simulated annealing [55] to optimize L sysid in a gradient-free manner. We then select the PD parameters with the lowest L sysid as (p 1 , d 1 ), and initialize another round of simulated annealing with a reduced parameter search range. In total, we perform 3 rounds of simulated annealing.On the right, we qualitatively illustrate the effects of our system identification for one of our simulated environments, the Google Robot [1] . We find that naively using PD parameter values from real controllers results in inaccurate tracking of the real robot's end-effector movements, which culminates in a missed grasp on the Coke can.After system identification, the controller more accurately tracks the motion in simulation: the robot is able to grasp the object when replaying the demonstration's action sequence.Real Raw Sim Sim + Green Screen + Texture Matching Sim + Green Screen Visual discrepancies between real and simulated environments can cause distribution shifts that adversely impact policy behavior, making simulated evaluations less reliable [11] . While modern graphics pipelines are able to create highly realistic visuals, developing the underlying assets and determining the lighting parameters to accurately model existing environments involves significant manual labor. Our goal is to match the simulator visuals to those of the realworld environment with only a modest amount of manual effort. For the scene background, we propose a "green screening" approach in which we overlay an image of the real-world environment onto the background of the simulated scene (see the wrapped figure). Concretely, we perform the following steps: (1) we remove the robot and foreground objects from the first frame of a realworld evaluation video I real using online image inpainting tools (e.g., https://cleanup.pictures); (2) we create a binary mask M isolating the foreground objects (robot arm and interactable objects, such as tabletop objects and articulated objects) in the simulation rendering I sim by querying ground truth segmentation masks in simulation; and (3) we combine the real-world background with the simulation foreground:I ′ = M ⊙ I sim + (1 -M ) ⊙ I real , producing the green-screened image.In practice, we find that many tested policies are also sensitive to changes in foreground object and robot textures, yet available simulation assets often exhibit appearance differences from realworld objects. Thus, we tune simulation asset textures for many objects to more closely match their real-world counterparts. We term this approach "Visual Matching" (see above for an illustration). Concretely, we project the real texture onto the simulation object by (1) segmenting the object in a real-world image [56] ; (2) aligning the simulated object pose to the real image; and (3) "unprojecting" the texture onto the object mesh in simulation. We provide step-by-step instructions in Appendix B.2, and we will release a convenient command-line script for this process. Otherwise, for assets like robot visual meshes with texture maps already resembling their real-world counterparts, we can instead selectively copy and paste color values from real to simulated texture maps. Finally, as robot arms may change colors during movement, we found it helpful to obtain multiple tuned robot arm colors that match the real-world textures from different phases of a manipulation task. We then average their evaluation results to mitigate this confounding factor.As an alternative to Visual Matching, we explore a mitigation strategy for real-to-sim visual gaps inspired by domain randomization: instead of minimizing the gap, we heavily randomize visual aspects of the scene to create environment variants. We then obtain an estimate of a policy's performance by aggregating evaluation results across multiple such variants, which we term as "Variant Aggregation" (see Appendix B.3 for more details and visualizations). Note that the Variant Aggregation setup does not seek to match the visual appearances between sim and real. As a result, it can utilize any scenes from public datasets like ReplicaCAD [16] and any objects (including the tabletop objects and the cabinets) from public datasets like Objaverse [57] for evaluation, whose appearances are out-of-distribution for the policies evaluated in this paper.SIMPLER environments and other details. We instantiate SIMPLER on two commonly used real robot evaluation setups: the Google Robot from the RT series of work [1, 17, 2] and the WidowX BridgeData V2 setup [5] . For each setup, we provide simulations for multiple tasks spanning a range of skills, interacted objects, object positions and orientations, backgrounds, and lighting conditions (see Fig. 2 ). The tasks are chosen to be representative of those in the corresponding training datasets, while also involving largely rigid body objects whose dynamics can be reasonably wellapproximated by modern physics simulators. We also instantiate SIMPLER on top of the SAPIEN physics simulator [58] , but later show that our contributions are independent of the simulator used and can be reproduced in Isaac Sim [59] . Creating most new environments takes an experienced user approximately one hour. See Appendix A and Appendix B for more details. 
experimental results: 
 In this section, we empirically test the performance correlation between real-world robot evaluations and simulated evaluations in SIMPLER environments for a representative set of open-source generalist robot manipulation policies. Concretely, our experiments are designed to answer the following questions: (1) Do relative performances of different manipulation policies in SIMPLER strongly correlate with those observed in real evaluation (Section 5.2)? (2) For a given policy, can SIMPLER evaluations accurately reproduce real-world policy behavior modes, e.g. sensitivity to various visual distribution shifts (Section 5.3)? (3) How sensitive is the fidelity of SIMPLER evaluations to (i) control and visual gaps, (ii) physical property gaps, and (iii) the choice of simulator (Section 5.4)?To quantitatively evaluate correlations between real and simulation policy performance, we perform paired sim-and-real experiments, i.e., with the same task instructions, object & robot poses, evaluation trials, and success criterions between the simulation and the real world (note that the paired experiments are not restricted to the tasks and setups in policies' training data). We use popular open-source generalist robot policies: RT-1-X [2] and Octo [3] (Octo-Base and Octo-Small). For evaluations in the Google Robot environments, we additionally use a number of RT-1 [1] checkpoints at various stages of training: RT-1 trained to convergence (RT-1 (Converged)), RT-1 at 15% of training steps (RT-1 (15%)), and RT-1 at the beginning of training (RT-1 (Begin)). We also report results on RT-2-X [17] . More details in Appendix A.We summarize the results of our main paired real-world and simulation evaluations in Fig. 4 (detailed breakdown in Appendix Table 2 ). We observe a strong correlation between the relative performances in simulation and in the real world across most policy checkpoints and tasks we evaluate, as reflected in low values for the MMRV metric introduced in Section 3 and high values for Pearson r.In Table 1 , we compare SIMPLER with using the action MSE in validation episodes for policy ranking, which is common for model selection within supervised learning settings (e.g., imitation learning). However, we find that validation MSE is not a good proxy for a policy's real-world performance, leading to high MMRV and low Pearson r. On the other hand, SIMPLER evaluations more accurately reflect relative policy performances in the real world, obtaining significantly lower MMRV and higher Pearson r. Additionally, we find that an alternative implementation of SIMPLER using Variant Aggregation (Section 4.2) performs worse: since it does not seek to match real-world visuals (Fig. 5 ), policies more sensitive to visual distribution shifts exhibit larger real-and-sim per- formance discrepancies. In summary, SIMPLER leads to a strong correlation with real-world policy performance, and we recommend Visual Matching as the default evaluation approach.Previously, we showed that SIMPLER evaluations strongly correlate with real-world performances based on trial averages. Beyond comparing average policy performances, it would be beneficial to let practitioners gauge more fine-grained aspects of a policy's behavior, such as its robustness to distribution shifts like lighting, background, and texture changes. We ask: do SIMPLER evaluations accurately reflect a policy's real-world behavior under such distribution shifts, and can they thus be used for more fine-grained policy evaluation beyond average performance?To test this, we use SIMPLER environments to perform controlled experiments along five distribution shift axes inspired by Xie et al. [ w/o Data Augmentation MMRV = 0.000 ↓ r = 0.831 ↑ w/ Data Augmentation MMRV = 0.016 ↓ r = 0.970 ↑ new camera pose axis. We evaluate two RT-1 checkpoints trained with and without data augmentation, which exhibit different robustness behaviors to distribution shifts. For simulated results and real-world results, we report the difference in the success rate with and without each distribution shift. The wrapped figure shows the results (more details in App. Table 4 ). We find that SIMPLER evaluations effectively reflect the policies' robustness to various distribution shifts in the real world. Notably, in both real and sim, changing robot camera poses and table textures has a significant impact on policy performance, while the impact of lighting and distractor changes are relatively minor. Furthermore, we find that an even more fine-grained analysis is possible through simulated evaluation. For example, when varying real-world table textures, both policies are more robust to unseen solid table colors than unseen patterned table textures (4% vs. 25% avg. performance decrease). This behavior is well reflected in SIMPLER: policy performance in simulation decreases by 2% on average when new colors of the solid table are introduced and by 24% for new patterned textures.Real, new arm texture SIMPLER, arm texture sensitivity Success rateTesting novel distribution shifts. Based on these results, we put our simulated evaluations to the test and ask: can SIM-PLER evaluations be used to predict the robustness of policies to new distribution shifts in the real world? Throughout our simulated evaluations, we observe that Octo-Base is particularly sensitive to changes in the simulated robot arm textures. Specifically, under our "Visual Matching" evaluation setup, its success rate is 0% on the "Pick Coke Can" task using the un-tuned robot arm, but 29.3% using one of our tuned robot arms. On the other hand, RT-1-X, also trained on the same Open-X-Embodiment dataset [2] , exhibits higher robustness to different simulated robot arm textures. To test whether this trend in simulation holds in real-world evaluations, we design a novel real-world distribution shift evaluation, where we change the real robot arm texture by wrapping it using multiple gift wrapping papers. We report results in the wrapfigure and Appendix Table 6 . The real-world evaluations support the simulated results: Octo-Base is more sensitive to changes in arm texture than RT-1-X. This indicates that simulated evaluations in SIMPLER environments can be predictive of real-world policy behaviors under novel distribution shifts. Effect of system identification. To test the impact of system identification (Section 4.1), we repeat the Google Robot's Visual Matching evaluations from Section 5.2, but perturb the stiffness and damping parameters of robot joints from the results of system identification.On the right, we show that the noisy system identification parameters lead to worse MMRV, i.e., worse correlation between simulated and real-world evaluations. This underlines the importance of accurate system identification for simulated evaluation. Effect of visual matching. We ablate the impact of our approaches in Section 4.2 for matching sim and real visual appearances. We use the RT-1 (Converged), RT-1 (Begin), and RT-1-X checkpoints on the Google Robot's open/close drawer tasks, and we compare the effectiveness of different combinations of background "green-screening", object texture, and robot texture settings. Results are reported in the wrapfigure (more details in App. Table 7 ). We observe the lowest MMRV and real-to-sim performance gap when combining background "green-screening" with object texture tuning for both drawer and robot assets. Interestingly, only tuning the drawer texture or the robot texture (but not both) is insufficient in improving the real-and-sim correlation. We hypothesize that this causes appearance inconsistencies between different parts of a scene, resulting in larger real-and-sim performance disparities.Sensitivity to physical property gap. When developing SIMPLER environments, we simplified the physical properties (e.g., the center of mass and friction coefficients) of objects and robots due to the complexity and time-consuming nature of precise modeling and system identification. In this section, we investigate whether our simulated evaluation is sensitive to such a real-to-sim physical property gap. We conduct 2 experiments: (1) For the "pick coke can" task, we vary the mass of the empty coke can (by varying its density), along with the static friction of the gripper finger; (2) For the "open/close drawer" task, we vary the joint frictions of the articulated cabinet. We report the MMRV and the Pearson correlation results in Appendix Tab. 8. We find that our simulated evaluation remains effective across a spectrum of plausible physical property parameters, evidenced by the low MMRV and the high Pearson correlation, even though altering these parameters has a moderate (≤ 15%) impact on the success rates of different policies. Sensitivity to the choice of physics simulator. To investigate whether our results are sensitive to the underlying physics simulator, we reproduce the Google Robot evaluation in Isaac Sim [59] . The results on the right (details in Table 9 ) show that SIMPLER's results are reproducible in Isaac Sim. In particular, we also observe a strong real-to-sim performance correlation across most checkpoints for SIMPLER-Isaac. This suggests that the choice of physics simulator is not critical for the tasks we tested, which mainly involve rigid body manipulations. 
conclusion: 
 As generalist robot manipulation policies advance, scalable evaluation approaches become essential for the rapid development of algorithms, models, and datasets. We introduce SIMPLER, a suite of real-to-sim evaluation environments that show a strong correlation to real-world policy performance. We provide a detailed discussion of our current work's limitations in Appendix Sec. G. 
real robot evaluation (train on real, evaluate in real): 
 Simulated evaluation (train on real, evaluate in sim) Our simulated success rate RT-1 RT-1-X RT-2-X OctoFigure 1 : Characterizing generalist robot manipulation policies typically involves evaluating them on many tasks & scenarios, a laborious undertaking in the real world (top left). In this work, we design an evaluation procedure where policies trained on real data are evaluated in purpose-built simulated environments (bottom left). Our approach yields a strong correlation between real-world and simulated performance (right) for various open-source robot policies [1, 2, 3] across two commonly used robot embodiments (Google Robot and Wid-owX) and over 1500 evaluation episodes. These results highlight the potential of simulation-based approaches for evaluating generalist real-world robot manipulation policies in a scalable, reproducible, and reliable way. 
 
 Appendix: In this section, we provide detailed descriptions of our SIMPLER environments along with our simulation and real-world evaluation protocols.For the RT-1 Robot, we adopt the following language-conditioned tasks:• "pick coke can". The robot is instructed to grasp the empty coke can on the table and lift it up. In the default setting, no distractors are added to the scene. We place the coke can in 3 different orientations: horizontally laying, vertically laying, and standing. For each orientation, we place the coke can at 25 grid positions within a rectangle on the tabletop, yielding 25 trials per orientation and 75 trials in total.• "move {obj1} near {obj2}". We place a triplet of objects on the tabletop in a triangle pattern. In each trial, one object serves as the source object, one serves as the target, and the other serves as the distractor (this creates 6 trials for each triplet and each triangle pattern). We randomly choose 5 triplets of objects among a total of 8 objects (blue plastic bottle, pepsi can, orange, 7up can, apple, sponge, coke can, redbull can), and adopt 2 triangle patterns (upright and inverted). This creates a total of 5 × 2 × 6 = 60 trials. The 5 triplets chosen are:blue plastic bottle, pepsi can, orange -7up can, apple, sponge coke can, redbull can, apple sponge, blue plastic bottle, 7up can orange, pepsi can, redbull can• "(open / close) (top / middle / bottom) drawer". The robot is positioned in front of a cabinet that contains 3 drawers and instructed to open / close a specific drawer, testing its ability to manipulate articulated objects. We place the robot at 9 grid positions within a rectangle on the floor, yielding a total of 9 × 3 × 2 = 54 trials.• "open top drawer; place apple into top drawer". The robot opens the top drawer and places the apple from the cabinet top into the top drawer, testing its ability to perform longer-horizon tasks. We place the robot at 3 different positions on the floor and the apple at 9 different positions within a grid on the cabinet top, yielding a total of 3 × 9 = 27 trials. Initially, the policies receive the "open top drawer" instruction. We switch to the "place apple into top drawer" instruction once the robot outputs the "terminate" token or after half of the time limit has elapsed.For the WidowX + Bridge (with WidowX-250 6DOF robot), we adopt the following tasks:• "put the spoon on the towel". We place the spoon on a vertex of a square (with edge length 15cm) on the tabletop, and we place the towel on another vertex. The spoon's initial orientation switches between horizontal and vertical, requiring the robot to perform gripper reorientation. This creates a total of 2 × 12 = 24 trials.• "put carrot on plate". We adopt a similar setup as "put the spoon on the towel", replacing the spoon with carrot and the towel with plate.• "stack the green block on the yellow block". We place a green block on a vertex of a square on the tabletop, and we position a yellow block on another vertex. The block dimensions are 3cm. We also adopt two differently-sized squares (edge length 10cm and 20cm). This creates a total of 2 × 12 = 24 trials.• "put eggplant into yellow basket". We place an eggplant on the right basin of a sink, and we place a yellow basket on the left basin. The eggplant is dropped into the sink at a random position and orientation, and we ensure that the eggplant is directly graspable (i.e., not too close to the edges of the sink basin). We perform a total of 24 trials.Require: (1) Current end-effector action (xa, Ra), along with sensed arm joint positions and velocities qarm, varm; (2) Current gripper action ga, along with sensed gripper joint position and velocity qgrip, vgrip;(3) Simulation frequency Hsim (501 in our implementation), action output frequency (control frequency) Hctrl (3 in our implementation following [1] ); (4) Arm velocity, acceleration, and jerk limits Larm (equal to 1.5, 2.0, 50.0 respectively); (5) Gripper velocity, acceleration, and jerk limits Lgrip (equal to 1.0, 7.0, 50.0 respectively); (6) Current action timestep T within an episode; (7) A planner that takes goal and initial joint positions and velocities as input (along with velocity, acceleration, and jerk constraints), and outputs a time-parametrized trajectory. For Octo simulated evaluations, since the model involves a non-deterministic diffusion head, we average its success rates across three different random seeds to produce a lower-variance estimate of the policy's simulation performance. Additionally, for RT-1 Robot simulated evaluations, we average results over four versions of robot arm and gripper colors to account for changes in arm texture during real robot rollouts (see Section 4.2). For the WidowX environments, given the consistent black color of the arm and gripper across videos, we skip this step.The number of evaluation trials we present above pertain to the real-world evaluation setup. For our "Variant Aggregation" simulation evaluation setup, the number of trials is multiplied by the number of simulation environment variants. For our "Visual Matching" simulation evaluation setup, the number of trials is multiplied by the number of tuned robot arm colors for the RT-1 Robot evaluation setup, along with the number of seeds for the Octo policies. ber of control commands (policy action outputs) per second. We use the open-source library Ruckig1 for time-optimal joint motion planning with velocity, acceleration, and jerk constraints. Note that the duration of planned trajectories may exceed the interval between two control commands.WidowX We present our WidowX controller implementation in Algorithm 2.Robots For RT-1 Robot, we convert the publically-released MuJoCo .mjcf robot description to URDF robot description. We also refine the collision mesh of the robot base link from the original assets to prevent erroneous mesh penetrations. For WidowX, we directly export the URDF robot descriptions from the official Interbotix repository using ROS. To simulate the RT-1 Robot, we find that the Projected Gauss-Seidel solver in PhysX causes mesh penetration behaviors during the process of object grasping. Thus, we enable the Temporal Gauss-Seidel solver in both SAPIEN and Isaac Sim's simulation backends to produce correct grasping behaviors.The RT-1 Robot uses a customized egocentric camera mounted on the robot head, while the WidowX + Bridge V2 setup uses a Logitech C920 third-view camera. We use known robot camera intrinsics if possible, and when they are unknown, we obtain them from real evaluation video frames using efficient interactive GUI tools such as fSpy.Objects We adopt the following procedure to obtain object assets. Except creating precise models for articulated objects like cabinets, the process is semi-automatic, does not require extensive manual effort, and typically takes less than 20 minutes.• Obtain raw 3D object models from public repositories (e.g., Objaverse [57] ), from 3D scanning of objects purchased from Amazon, from single-view 3D generation (e.g., One-2-3-45++ [61] ), or from manual modeling based on precise measurements of real-world counterparts (we only used the last technique for articulated objects like cabinets since this requires the most human effort; we highlight the acceleration of articulated asset curation process through approaches like multi-view [62] or interactive [63] articulated object generation as an avenue for future work).• Process 3D object models in Blender such that the dimensions of objects are similar to those used in the real world, and that the object meshes do not contain too many vertices (to limit the sizes of object meshes).• Optionally, use our Visual Matching approach (see below) to improve the texture of 3D object models. • Export visual mesh and collision mesh of objects. For collision mesh, further perform CoACD [64] to obtain watertight and locally convex collision meshes. Optionally, simplify the resulting collision mesh and perform minor modifications using Blender (e.g., make the bottom of cans or bottles flat).• Set the object to have a simple uniform density by querying their common material density in GPT-4 or google search, or (for objects with non-uniform densities like empty coke can), querying their mass and dividing by their visual mesh volume.To perform visual matching of object textures, we adopt the following steps: (1) Crop the target object in a real image using SAM [56] ; (2) Perform a coarse estimation of object pose by importing it into the simulation and adjusting its position such that its simulation segment mask overlaps with the real one; (3) Employ differential rendering (using Nvdiffrast) to optimize the simulation asset's pose such that it precisely aligns with the real object's segmentation mask; (4) "Unproject" the real object's RGB texture values onto the simulation object mesh; (5) Optionally, generate the remaining views of the object through a diffusion model (Zero123++ [65] ), and refine the poses of novel views using a rendering loss with the existing object view. Finally, unproject the novel view textures onto the simulation object mesh. This whole process is semi-automatic, and can thus be completed efficiently. We commit to release a convenient command-line python script for this process.A common approach for addressing visual gaps in sim-to-real policy training is domain randomization. By performing training across a range of randomized parameters, such as textures and lighting, prior works aim to obtain policies that are robust to visual distribution shifts in the realworld [40, 41] . Similarly, in real-to-sim evaluation, we can aggregate evaluation results across a range of visual simulator characteristics to obtain a more faithful signal for the policy's performance. In practice, we implement this SIMPLER-"Variant Aggregation" approach as an alternative to SIMPLER-Visual Matching, described in Section 4.2. Concretely, we create a "base" version of our simulation environment and then creating "variants" of this environment along four axes of visual variation: background, lighting, distractors, and table texture. For each axis, we construct 2 variations of the base setup similar to [60] , covering backgrounds from different rooms, lighter and darker lighting, fewer and more distractors, and solid color and complex table textures. We visualize an example of such simulator variations for various table-top tasks in Fig. 5 . We average policy performance in simulation across all variants of an environment to obtain our final performance estimate. The base setup environment already contains distractors, so we construct environment variants without distractors. Table 5 : Success rates of different out-of-distribution generalization factors on the tabletop manipulation performance of RT-1 policies in the SAPIEN simulator. "a/b" denote results on different environment variants (lighting: darker / brighter; table texture: solid color / contrastively patterned; camera pose: oriented lower / higher).In Table 4 , Table 5 , and Table 6 , we present full evaluation results for our experiments in Sec. 5.3, which demonstrate that SIMPLER environments show strong policy behavior correlations with realworld evaluations under different environment distribution shifts.We present detailed results for our main paper's ablations in Table 7 , Table 8 , and Table 9 .fast simulation of these scenes. In the beginning of the project, we did preliminary investigations using state-of-the-art approaches in Gaussian Splatting [66, 67] , but found the scene quality to be insufficient for real-to-sim manipulation evaluation (e.g., uneven table surface geometries, floaters while rendering from novel camera views). Additionally, importing Gaussian Splatting-based scenes into simulation environments would entail non-trivial changes to the existing simulators' rendering pipeline, which we leave for future work. (4) Our current simulated environments are limited to RGB image and proprioceptive policy inputs. We believe that future simulated evaluation tools could incorporate other modalities such as LIDAR [6] , acoustic [68] , and tactile sensors [69] as research on their effective and accurate simulation continues to advance. (5) We adopt a simple system identification approach that we find to be adequate for scenarios like rigid object grasping and articulated furniture manipulation. It is unlikely that system identification using open-loop contactfree trajectories will be sufficient to accurately model robot control parameters for situations such as high-speed robot-object collisions and dexterous manipulation.https://github.com/pantor/ruckigAfter running 2 real evaluation trials, robot operators decided that since this policy would potentially damage the robot on the Drawer tasks, the real evaluation was terminated.As real evaluation was terminated due to risk of damaging the robot, we expect the MMRV to be less than this number if real evaluation were to continue. 


Decision: This paper appears to have been relatively well-received by the reviewers, with a number of notable strengths. 

-well written and well-structured
-clear in its motivation
-timely 
-algorithm and approach is well-defined and well-described

But with some weaknesses:

-apparent use of extensive manual labour in capturing textures etc. (reviewer VgL2)
-lack of thorough discussion on weaknesses and limitations
-uses only camera poses
-unclear sim2real transferability according to some of the reported results, which requires some investigation and explanation
-does not consider some SOTA techniques, e.g., NERFs.

The authors should clarify these issues
I encourage the authors to address the comments of the reviewers to increase chances of acceptance.


Updated:

The authors have been responsive to the reviewer's comments, run extra experiments, and tidied up the text.

Reviews: review: title: title: Review of submission 9
paper_summary: paper_summary: This paper presents a framework called SIMPLER for validating manipulation policies in simulation that have been trained with real robot evaluation.
main_review: main_review: Summary

This paper presents a framework called SIMPLER for validating policies in simulation that have been trained with real robot evaluation. The methodology in the paper proceeds by to estimating empirically the correlation between the simulation evolution and real-world evaluation of the policies. The key contributions are an algorithmic framework for assessing the performance of such policies in simulation that highly correlates with the assessment in evaluation on the real robotic platform. Mainly the authors propose two features that are essential for robotic manipulation tasks. First, importing textures from the real dataset to the simulator, and second, performing system identification for having a robot behavior that mimics the real one.

Review

= Contribution

The paper’s contribution is focused on robot manipulation tasks. It also assumes that the policies are optimized for a certain distribution. In this context the simulation scenarios, are designed by hand and replicate some of the available scenarios in the dataset. In contrast, the recent trend in autonomous driving and other robotic applications is to automate this step using data-driven simulation. In a nutshell, this means that the dataset is automatically used for generating novel scenario that are generalized using hybrid modeling. In my humble opinion, the approach of the paper is more incremental and probably performs better in some cases but requires a lot of manual design.

= Research Idea

== Quality

The paper is very well executed. The experiments are and the focus are timely. The contribution is clear but the situation with respect to state-of-the-art in policy validation is not clear. 

It would be nice to have a broader picture in terms of the limitations and tradeoffs of the approach. What is generally the challenge in simulation? Is it the model accuracy or the compute performance? Are there other challenges linked to contact simulation, which are specific to manipulation? Linked to model-based reinforcement learning? How difficult is it learn a closed-loop version of an open-loop dataset? What are the perspectives in the domain?

== clarity

The paper is quite clear and well written. The experiments clearly show that the aim of the paper are met and that the focus of SIMPLER (Visual matching and System Identification) are well chosen. In the limitation part of the paper the authors show that they are conscious of the need for more automation.

== originality 

The contribution is interesting in the scene that it clearly addresses the problem of validating real world policies in simulation and succeeds to that end. As mentioned above this type of approach might be very well suited for this problem. The limitation to camera poses may however be a showstopper for real world applications.

== significance

The significance of the work I would rate as low. It is a timely contribution because the community needs to work on full statistical assessment of these foundation models for robots. However, it is important to have the ability to assess out-of-distribution scenarios, and to modulate how far from distribution the validation takes place. The first thing is to be able to automatically generate simulation pipeline. Hence the research effort should be placed in designing fully automated frameworks.

= Strength and Weakness

Strengths
-	Addressing a timely problem
-	Very well written paper & clarity of argumentation
-	Interesting idea and theoretical considerations
-	Good baseline comparison

Weakness
-	Not sufficiently connection to state-of-the-art methods (i.e., NeRFs etc)
-	Not enough focus on tackling the difficult scientific issue linked to the problem
-	Limited to in distribution testing with very moderate out-of-distribution validation
questions: questions: There are no questions from my side.
limitations: limitations: 3
review_summary: review_summary: I would be ok to accept this paper if others fight for it. But my recommendation is rather borderline (which is not possible here).
title: title: Well written and valuable contributions
paper_summary: paper_summary: The authors propose a simulation framework, called SIMPLER, for evaluating real-robot manipulation policies at scale. The authors propose to close the sim-real control gap via system identification using a small sample of real trajectories and the sim-real appearance gap by matching real backgrounds and textures. The simulation performance is shown to strongly correlate with real robot performance, and is shown to be indicative of the relative sensitivity of real robot policies to different distribution shifts.
main_review: main_review: Strengths
1. The paper’s contributions are valuable: the proposed framework takes a step towards reproducible benchmarking of general purpose manipulation policies in simulation, demonstrating strong correlation with real robot performance.
2. The different choices are well analysed and it is interesting to see the simulation environments discovering novel distribution shifts: sensitivity to robot arm textures and patterned tables.
3. The paper is well written and the procedure for generating simulated environments is well detailed in the paper and the appendix.

Weaknesses
1. The real2sim conversion of assets and textures seem to require multiple steps of human effort. Raw estimates of manual hours needed for creating a single evaluation environment would be useful to include.
2. The diversity and scale of evaluation environments (backgrounds, textures, static camera angles) is limited to the evaluation environments present in the real world trajectories (due to paired creation). While some axes of further variation are explored (eg. arm trajectories), further scaling of unpaired simulated evaluation environments is unexplored.
3. Performance in simulation seems to be slightly uncorrelated with real robot performance in select scenarios (Close Drawer Table 2, Grasp Spoon Table 3, Augs not helping Move Near in Table 4).
questions: questions: Elaborated under weaknesses. Critically would like to know an estimate of manual hours spent preparing the environments, and cause of poor correlation in select cases.
limitations: limitations: 3
review_summary: review_summary: Proposing acceptance as the developed framework holds value for reproducible benchmarking of real robot manipulation policies.
title: title: Paper Review
paper_summary: paper_summary: This paper explores the challenging problem of simulation-based evaluation of robot manipulation policies trained from real world data. The key insight is to develop realistic simulated environments that are realistic enough by overcoming control and visual disparities, so they can serve as a scalable and repeatable tool for evaluating real-world manipulation policies. The paper proposes different simulated environments and a ranking metric for policy evaluation showing strong correlation in policy performance between real-world and sim for open-source RT1-X and Octo manipulation policies.
main_review: main_review: ### Strengths

1. The paper tackles an important and long-standing challenge of reproducible evaluation of robotic policies. In contrast to prior works, the paper does not assume a fixed set of evaluation objects or training in simulation, and is applicable to policies pre-trained on real world data.

2. I found the paper to be well motivated and enjoyable to read. The setup is well motivated and the approach is clearly presented and easy to follow.

3. The experimental section evaluates key aspects of the proposed evaluation environments - correlation with real world performance as well as testing finer grained details of policy behavior such as distribution shifts. 


### Weaknesses

1. In the related work section, the authors do not elaborate sufficiently on existing methods for real-to-sim.

2. It is unclear how the current approach will scale to real world policies with more complicated inputs like tactile sensors which can be hard to simulate at high level of accuracy. The authors should comment on how this could potentially be achieved or mention it in the limitations.
questions: questions: Q1. Sec 3: it is unclear how the ranking metric behaves if the simulated environment consistently overestimates the real-world performance of policies. In this case, even though we have a strong correlation, simulated evaluations could mislead us to believe that a bad real-world policy is good. How does the current ranking metric overcome this issue? 

Q2. Sec 4.1 :  The offline system identification approach does not clearly address the coupling between end-effector motion and object properties when in contact. Does the offline dataset used for system identification also span different objects being manipulated or other potential contact surfaces in the environment?
limitations: limitations: 2
review_summary: review_summary: I am recommending weak accept based on the points mentioned in weaknesses and questions above. I am willing to update my recommendation if authors sufficiently address them.