Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation
Authors: Joanne Truong
Max Rudolph
Naoki Harrison Yokoyama
Sonia Chernova
Dhruv Batra
Akshara Rai
Venue: CoRL 2022 Poster (2022)
Keywords: Sim2Real
Deep Reinforcement Learning
Visual-Based Navigation

Abstract: If we want to train robots in simulation before deploying them in reality, it seems natural and almost self-evident to presume that reducing the sim2real gap involves creating simulators of increasing fidelity (since reality is what it is). We challenge this assumption and present a contrary hypothesis -- sim2real transfer of robots may be improved with lower (not higher) fidelity simulation. We conduct a systematic large-scale evaluation of this hypothesis on the problem of visual navigation -- in the real world, and on 2 different simulators (Habitat and iGibson) using 3 different robots (A1, AlienGo, Spot). Our results show that, contrary to expectation, adding fidelity does not help with learning; performance is poor due to slow simulation speed (preventing large-scale learning) and overfitting to inaccuracies in simulation physics. Instead, building simple models of the robot motion using real-world data can improve learning and generalization.

Rethinking Sim2Real: Lower Fidelity Simulation Leads to Higher Sim2Real Transfer in Navigation
Joanne Truong', Max Rudolph', Naoki Yokoyama', Sonia Chernova', Dhruv Batra 1.2, Akshara Rai²
'Georgia Institute of Technology, M AI
[truong.) J, maxrudolph, nyokoyama, chernova, dbatra)@gatech.cm edu (akshararai] com
Abstract: If we want to train robots in simulation before deploying them in real- ity, it seems natural and almost self-evident to presume that reducing the sim2real gap involves creating simulators of increasing fidelity (since reality is what it is). We challenge this assumption and present contrary hypothesis sim2real trans- fer of robots may be improved with lower (not higher) fidelity simulation. We conduct a systematic large-scale evaluation of this hypothesis on the problem visual navigation in the real world, and on 2 different simulators (Hahitat and iGibson) using different robots (A1, AlienGo, Spot). Our results show that, con- trary expectation, adding fidelity does not help with learning: performance is poor due to slow simulation speed (preventing large-scale learning) and overfit- ting to inaccuracies in simulation physics. Instead, building simple models of the robot motion using real-world data can improve learning and generalization.
Keywords: Sim2Real, Deep Reinforcement Learning, Visual-Based Navigation
Introduction
The sim2real paradigm consists of training robots in simulation (potentially for billions of simula- tion steps corresponding to decades of experience [1]) before deploying them in reality. The last few years have seen significant investments the development of new simulators [2-12], curation and annotation of 3D scans and assets [13-15]. and development of techniques for overcoming the sim2real gap [16-19] resulting in number of successful demonstrations of sim2real transfer [20- 25]. However, no simulator is a perfect replica of reality and the main challenge in this paradigm is overcoming the sim2real gap, defined the drop in robot's performance in the real-world (com- pared to simulation). It seems natural and almost self-evident to presume that reducing this sim2real gap involves creating simulators of increasing physics fidelity, and this sometimes forms the default operating hypothesis of the field.
We challenge this convention and present counter-intuitive idea - sim2real transfer of robots may be improved not by increasing but by decreasing simulation fidelity. Specifically, we propose that instead of training robots entirely in simulation, we use classical ideas from hierarchical robot con- trol [26] to decompose the policy into a 'high-level policy' (that is trained solely in simulation) and 'low-level controller' (that is designed entirely on hardware and may even be a black-box troller shipped by a manufacturer). This decomposition means that the simulator does not need to model low-level dynamics, which can save both simulation time (since there is need to simulate expensive low-level controllers), and developer time spent building and designing these controllers.
We conduct systematic large-scale evaluation of our hypothesis on the task of PointGoal (visual) Navigation [27] unknown environments - using 2 simulators (Habitat and iGibson) and 3 different robots (A1, AlienGo, Spot). We train policies using two physics fidelities kinematic and dynamic. Kinematic simulation uses abstracted physics and 'teleports' the robot to the next state using Euler integration; kinematic policies command robot center-of-mass. (CoM) linear and angular velocities. Dynamic simulation consists rigid-bodi body mechanics and simulates contact dynamics (via Bullet [12]): dynamic policies command CoM linear and angular velocities, which are converted to robot joint-torques by a low-level controller operating at 240 Hz. We find that across all robots, kinemat- ically trained policy outperforms dynamic policies,
when evaluated using dynamic simulation 6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Kinematic Control
Simulation Trajectories
Dynamic Control
Real World Trajectories
Start
Unsuccessful
Kinematic Policy
Dynamic Policy
Figure 1: Left: We train visual navigation policies at two levels of fidelity kinematic and dynamic. In kinematic control (top). the robot 'teleported" the state using Euler integration. In dynamic control (bottom), the robot's velocity commands are converted leg joint-torques and rigid-body physics simulated 240Hz. Right: We evaluate the kinematic and dynamic trained policies simulation (top) and the real-world (bottom) across 5 identical episodes. The kinematic policy achieves 100% success rate in all 5 episodes, and the robot takes similar paths in both simulation and the real world. On the other hand, the dynamic policy achieves 20-60% success rate, and the trajectories taken in simulation and the real-world do not correlate, pointing towards larger sim2real gap.
and control. Additionally, we show that the trained kinematic policy can be transferred real Spot robot, which ships with manufacturer-provvided "black-box' low-level controllers that cannot be accurately simulated. In contrast, dynamic policies fail achieve efficient navigation behavior on Spot, due to the sim2real gap and less simulation experience.
The reasons for these improvements perhaps unsurprising in hindsight - learning beareing-bbsed methods overfit to simulators, and present-day physics simulators have approximations and imperfections that do not transfer the real-world. A second equally significant mechanism is also in play lower fidelity simulation is typically faster, enabling policies to be trained with more experience under fixed wall-clock budget. Even when the kinematic policies clock time than the dynamic policies (with the same compute), the kinematic policies were able to learn from 10x the amount of data. While results are presented legged locomotion and visual navigation, the underlying principle- of architecting hierarchical policies and only training the high- level policy in abstracted simulation rethink in how the research community pursues sim2real and in how we develop the simulators of tomorrow, Specifically, our findings suggest that instead of investing in higher-fidelity physies, the field should prioritize simulation speed for tasks that can be represented with abstract action spaces.
trained for 2.3x less wall-
broadly applicable. We hope that our work leads
2 Related Work
Visual Navigation. Recent works have shown that large-scale indoor environments and simulators like Habitat [2, 3] and iGibson [4] can enable end-to-end learning of navigation policies from large amounts agent- or expert-generated data [28-30] on simple, wheeled systems. This is in contrast to the typical mapping and planning paradigm used in classical robotics, which can suffer when the quality maps is low [31] or requires expensive equipment like LiDAR [32]. this work, we show that such end-to-end learning is also possible for complex, legged robots.

---

Sim2real for Legged Robots. Sim2real quadrupedal locomotion has been widely studied in the past several decades [22. 33-36], with most learning low-level skills in simulation and transferring them to hardware [37], or adapting them online to reduce the sim2real gap [38. However, these policies are typically blind, and use only proprioceptive sensors on the robot to determine actions [23, 25]. In contrast, an autonomous robot needs to respond to its environment, and take visual input into account. Some works have proposed learning visual policies in simulation and applying them to the real-world [22, 24, 40], and other works leverage expensive LiDAR sensors for external sensing [41]. These works use learned or hand-designed physically simulated low-level controllers; we show that physics simulations can be detrimental to learning perffrrefnn sim2real policies, even for complex legged robots. Work from [42] also utilize similar simulator simplifications to increase simulation speed in training navigation policies for 1 robot in single room, but does not discuss how this formulation affects sim2real transfer. In our work we present a rigorous (multi- robot, multi-simulator) study of the effect of different simulation fidelities on visual navigation.
Abstracted Task-space Learning. Abstracted (hierarchical, high-level) action spaces com- mon in robotics literature. Examples include task and motion planning for manipulation [43-45], legged locomotion [46, 47], navigation [20], etc. Several works reason over symbolic actions like pick and place, or hierarchical policies with discrete/continuous attributes [33-35, 48, 49), or even abstracted dynamies models [36]. While the ideas of abstracted/hieracchiaal policies are fairly com- mon, typically both the high- and low-level policies are learned in simulation and transferred to reality [33, 34, 36), often augmented with techniques like domain randomization [37] and real-word adaption [18]. Instead, we use an abstracted simulator, which does not model low-level physics, and learn high-level policies that are transferred the real-world in zero-shot manner.
3 Experimental Setup
Task: PointGoal Navigation. In the task of PointGoal Navigation [27], robot is initialized in an unknown environment and is tasked with navigating to a goal coordinate without access to a pre-built map of the environment. The goal is specified relative to the robot's starting location for the episode (i.e.. "go to Ax, Ay"). The robot has access to an egocentric depth sensor and an egomotion sensor (sometimes referred to GPS+Compass in this literature) from which the robot derives the goal location relative to its current pose. An episode is considered successful when the robot reaches the goal position within a success radius (typically half of the robot's body length). The robot operates within constraints of maximum number of steps per episode (150 for Spot) and velocity limits (+ 0.5 m/s for linear and = 0.3 rad/s for angular velocities Spot). We linearly scale the linear and angular velocity limits for Al and Aliengo to be proportional to the length of each robot's 's leg, and inversely scale the maximum number of steps allowed. In effect, smaller robots have a smaller maximum allowed velocity to improve stability during execution, but are allowed more steps to reach the goal. The exact parameters used for each robot is described in the appendix. For evaluation, success rate (SR), and Success inversely weighted by Path Length (SPL) [27], which measures the efficiency of the trajectory taken with respect to the ground-truth shortest path.
report the
Robot Platforms. We study visual navigation for 3 quadrupedal robots - Al and Aliengo from Unitree [50], and Spot from Boston Dynamics (BD) [51] in simulation. In the real-world, we show sim2real transfer of the learned navigation policies to Spot. To have a consistent camera setup across all the robots, we attach an Intel RealSense D435 camera to Spot in the real-world, and use this camera for visual inputs the policy. In our hardware experiments, we want to measure how often our sim2real policies lead to collisions without jeopardizing safety. We achieve this balance as follows: the BD collision-avoidane of 10m Next, we track the number of times the robot comes within 0.20m of any obstacle (as measured by any of the onboard depth cameras). This gap (between 0.20m and 0.10m) allows us to record possible collisions while preventing actual ones. While the BD API allows for high- level navigation without access to a map, cannot navigate around obstacles autonomously, without a map. In our work, we consider complex, long-range navigation paths (up to 30m) in cluttered environments with many obstacles; the goals are unreachable with the just BD navigation API.
capability is kept turned on, set to trigger at a tight threshold
Simulation Environments. We use two simulation platforms - Habitat [2. 3] and iGibson [4] for training and evaluation. Both simulators support rendering of photorealistic environments; Habitat uses low-level (C++) integration with the Bullet physies engine [12], while iGibson leverages Py- Bullet, the Python-base integration of Bullet. Thus, while the underlying physics engines between
3

---

A1
Aliengo
Spot
Spot-Real Real
Figure 2: Robots used for training and evaluation.
the two the same, Habitat runs 1200% faster than iGibson [3]. This allows train policies faster with Habitat than with iGibson even when using identical policies and compute.
Dataset. For training and evaluation, we use a combination of the Habitat- Habitat-Matterpon (HM3D) [13] and Gibson [52] 3D datasets. The two datasets combined consist of over 1000 high-resolution 3D scans of real-world indoors environments, and consists of realistic clutter. We generate training and evaluation episodes compatible with robots for the HM3D and Gibson scenes following the procedure described in [2]. Specifically, we restrict the geodesic distance from the start and positions to be between and 30m, and increase navigation complexity by rejecting paths that consist of near- straight lines, with few obstacles. As described in [2], both of these heuristics result in complex, but check for collisions along the sampled paths using the URDF of
navigable paths. Additionally, the largest robot (Spot) to ensure that all paths are navigable.
Real-World Test Environment. The real- world evaluation environment, LAB, is 325m² lobby in a commercial office building. The lobby contains furniture such a couches, cushions, bookshelves and tables. We specifyy set of 5 waypoints as the start and end locations for the navigation episodes in LAB with an av- erage episode length of 10m. We match the furniture layout to the position captured in the 3D sean (Figure 3) to run identical evaluation experiments in both simulation and the real- world. The sean LAB is not part of training.
Figure The real- world testing environment a part of a large commercial building and contains clutter from furniture such as tables, bookshelves, and couches.
4 Kinematic and Dynamic Control for Visual Navigation
As illustrated in Figure 4, our proposed approach is hierarchical, with (1) a high-level visual navi- gation policy that commands desired center of mass (CoM) motion at 1Hz, and (2) low-level con- troller that follows this desired motion. We consider controllers at two levels of abstraction - 'kine- matic" and 'dynamic'. The kinematic controller simply integrates the desired velocity and outputs CoM position at 1Hz; kinematic simulation then teleports the robot to the desired state. The dynamic controller uses low-level controller that commands joint torques 240Hz; dynamic simulation models rigid-body and contact dynamics via Bullet (with physics step-size of 1/240 sec). We pro- vide details of all three of these pieces (high-level policy, kinematic and dynamic controllers) next.
High-level Visual Navigation Policies. The high-level policy takes input an egocentric depth image, and the goal location relative to the robot's current pose. The output of the policy is a 3-dimensional vector, representing the desired CoM forward, lateral, and angular velocities (Vz, w). The neural network architecture consists of a ResNet-18 visual encoder and 2-layer LSTM policy. LolMyy Using recurrent policy allows the policy to learn temporal dependencies through the hidden state. The final layer of the policy parameterizes a Gaussian action distribution from which the action is sampled. The policy is trained using DD-PPO [1], a distributed reinforcement learning method, in both the Habitat and iGibson simulators. Our reward function is derived from [22], with an added penalty for backward velocities, which can lead to collisions and hurts performance.
Kinematic Control and Simulation. kinematic control, the final state of the robot calculated by integrating the desired CoM velocity commanded by the high-level navigation policy at HHz. The robot is directly moved to the desired pose, without running physics simulation. In both Habitat and iGibson, the robot is kept in place if being at the new desired state would result in collision.

---

High-level Policy 1
Low-level Controller
Robot
Depti observation
Kinematic control step
Distanco heading to goal
Spor Bari
Integrate Velocity
Goal state
VP Ky
OR Dynamic control 240 steps
CNN
CoM linear angular velocity
Footstep Generator
Fazs
Policy
Inverse Kinematics
angles
Joint
Depth observation, CoM state
Figure 4: Our architecture for PointGoal Navigation on a legged robot. A high-level visual navigation policy predicts CoM linear and angular velocities. The velocities are passed into either kinematic dynamic low- level controller to step the robot in simulation. In the real-world, we directly send the velacity commands from the high-level policy to the robot, and uses the low-level controller from Boston Dynamics for movement.
The objective of the kinematic control is to abstract away the low-level physics interactions between the robot and its environment. This has two advantages: (1) it avoids the need to accurately model low-level controllers, especially for closed-source robots like Spot; (2) enables faster simulation speed by avoiding high-frequency physics integration, conducive to model-free RI. that requires large amounts of experience. On the other hand, teleporting the robot to the desired state might remove necessary dynamics, such poor tracking of low-level controllers. In Section 5, we propose how to incorporate such low-level characteristics into a kinematic simulation using real-world data.
Dynamic Control in Simulation and Hardware. We experiment with two different low-level dy- namic controllers for quadruped robots. The first is an expert-designed Raibert-style controller from footstep generator and an inverse kinematic solver that commands desired joint angles from CoM velocities. The joint angles are converted to joint torques using linear feed- back controller, and applied to the simulation. This controller was shown to achieve sim2real transfer for A1 [22]. However, other robots in experiments, it shows relatively poor tracking of high- level commands. Thus, we also experiment with another predictte control (MPC) dynamic controller from [38]. which commands joint torques directly. This controller has been applied to real-world A1 robot [53, 54] and shows better tracking of desired velocities for our test robots, as compared the Raibert controller from [22]. However, MPC is prohibitively slow and cannot be used for training RI. policies. Thus, we use Raibert for training dynamic policies, but evaluate using MPC. This difference in train and evaluation dynamics controllers has multiple purposes: (1) the evaluation using MPC improves performance of most policies, including dynamic policies, due to its better ability to track high-level commands; (2) the difference between the two dynamic controllers in simulation is also a proxy for the difference between our low-level controllers and closed- controllers from Spot. If dynamic policy cannot transfer from Raibert to MPC, it has low chance of transfer to Spot which has black box BD controllers, or even other robots in the real-world.
[22], which consists
Both dynamic controllers model the low-level physics interactions between the robot and the vironment. This makes them considerably slower than the kinematic controller, making training RL policies challenging. Moreover, for Spot, the low-level controller implementation is not openly available, making it hard to reproduce the low-level controller in simulation. For commercial legged robots that come with black-box controllers, kinematic simulations are the ideal fidelity for learning navigation policies. Our experiments in Section 5 show that the added fidelity of dynamic controllers does not benefit policy learning, or sim2real transfer.
Impact of Low-level Controllers Policy Learning. The low-level controller used in dynamic simulations can have significant impact on the learned policy. Low-level controllers both in sim and real are biased, and the status quo to make them biased in the same way, as the policy learns to compensate for the bias error. For example, [55] add learned actuation noise to their simulation, while [37] measure hardware characteristics and add them the simulation. However, given the high- nigh-dimensional nature of low-level physics, is very difficult to ensure that the biases incorporated in simulation actually hold for a larger range of motions that the data was collected on. Thus, there are several iterations of data-collection. bias updates, training and deployment needed for good
"Evaluation using Raibert [22] can be found in the appendix.
5

---

performance. Instead, kinematic controllers are unbiased by design, and can easily incorporate hardware bias through cow-dimesiional CoM motion noise models created from a small amount of real-world data, as shown in our experiments in Section 5.
5 Results and Analysis
In this section, we first study generalization of visual navigation policies across simulators (trained in one sim, tested in another) and across controllers (trained with one controller, tested with another). This shows the importance of fast simulation for learning high-level policies by comparing perfor- mance of kinematic and dynamic policies trained for the same wall-clock time. Next, we examine the performance of the different policies at zero-shot sim2real transfer on the Spot robot.
How large is the sim2sim gap? High for dynamic, and low for kinematic policies. We exhaustively study the combinatorial space of experiments - policies trained under 2 training conditions (with kinematic and dynamic simulation) 2 evaluation conditions (kinematic and dynamic simulation) x 2 simulators (Habitat and iGibson) x robots (A1, Aliengo, Spot). For each condition, train and report results with random seeds. Each policy is trained using 8 GPUs for 3 days, resulting in a cumulative training budget of 6,912 GPU-hours (288 GPU-days). The average success rates presented in Figure 5. Rows represent the evaluation conditions as tuples (simulator, fidelity), while columns represent the training conditions. We evaluate all policies across 1,100 episodes from 110 unique scenes in the HM3D Gibson validation split.
A1
Allengo
Spat
(Hob.Kin)
92.4
78.4
61.8
55.1
(Hab.Kin)
93.3
77.8
59.8
54.6
(Hai.Kin)
84.2
73.3
24.5
37.4
Eralusten Smn. Controler
Distueten Smm. Contraler
Braiueten Stm. Controler
80
(6tbKin)
83.2
82.7
50.5
(Gb.Kin)
90.8
68.8
62.7
73.4
92.4
34.8
40.0
60
40
(Hab.Dyn)
63.6
47.8
49.8
22.9
(Hab.Dyn)
55.4
51.5
51.8
41.2
@Hob.Dyn)
36.2
51.3
36 1
21.5
20
(Glb.Dyn)
58.2
22.3
(SIo,Dym)
75.2
45.8
43.9
22.9
(Mat,Kin) (6b,Kin Dyb. Taaning Sim, Controlir
(Mab.Kin) (G)b.Kn) (Hab.
JSIb byn Training Simm, Controller
(Hal,Kin) (GibKin) "Tainin Sim, Controlier
Figure 5: Average success rates for sim2sim and kinematic2dynamic transfer for A1, Aliengo and Spot. We see that the kinematic trained policies perform the best overall (red quadrants), and also often outperform the dynamic trained policies, even when evaluated using dynamic control (green quadrants orange quadrants).
We make two key observations here:
1. Kinematic-trained policies perform best overall, for all robots. In all cases, kinematic policies outperform the dynamic policies, even when evaluated using dynamic control, e.g. 62. 12.1% SR for A1 in (iGibson, Kinematic) vs. 24.2% SR in (iGibson, Dynamic). Fig. 5, left. This a surprising result because the kinematic policies are being evaluated in an out-of-distributin seen or accounted for during training. On the other hand, the dynamic policies are being evaluated in the domain that they were trained in, hence do not require control-related generalization.
setting, which was never
Dynamic policies are not robust different dynamic simulations. The dynamic policies from the two simulations observe significant performance drops when evaluated in the other dynamic sim- ulation. This points the dynamic policies overfitting the simulator dynamics during training, failing to generalize to a new setting, see e.g. column 3, rows 3 and 4; 49.8% SR for Al in (Habitat, Dynamic) vs. 22.3% SR in (iGibson, Dynamic). (iGibson, Dynamic) shows poor performance in both iGibson and Habitat, with slightly poorer performance in Habitat. This sensitivity to simulation makes training dynamics policies difficult, especially when the controller for the real-world robot is unknown. Even if the real-world controller is known, simulation physics and real-world are differ- ent, and sim2real transfer of the learned policy can suffer (as evidenced by low sim2sim transfer). On the other hand, kinematic policies, that have been trained with no physics, can generalize to the different dynamic controllers. Both of these results go to show that not only kinematic trained policies are able learn the task well, they have learned to reason without overfitting to simulation physics, making their chances of successful sim2real transfer high.
Why do kinematic-trained policies outperform dynamic ones? Scale. We plot the evaluation performance of both policies in Habitat kinematic and dynamic simulation in Figure 6. We train

---

both policies to convergence- 3 days for the kinematic policies, and 7 days for the dynamic poli- cies. While the kinematic policies are trained for 2.3x less wall-clock time, they still outperform the longer trained dynamic policies, even when evaluated out-of-distribution using dynamic con- trol (+12% SR). Kinematic training is much faster than training dynamically (right, Fig. 6); with kinematic training. the robot is able to learn from approximately 10x more steps of experience (500M steps vs. 50M steps). This increased experience allows the kinematic policies to learn in- telligent high-level reasoning. We contend that for any computational budget, there will always be more complex tasks that are bottlenecked by that budget. Wall-clock time is the true limitation for learning-base sim2real approaches (not experience, as different simulators have different speeds).
100
Evaluation Performance of A1 in Habitat
100
Evaluation Performance of A1 in Habitat
10x faster
Success Rate (%)
80
Success Rate (%)
80
+12% SR]
60
40
40
20
Kin Train, Kin Eval
20
Kn Train, Kin Eval
Kin Train. Dyn Evel
Knn Train. Cyn Eval
Dyn Train, Dyn
Eval
0
0
5 Wall Clock Time (Days)
3
6
0 M
100 M
200 M
300 M
400 M
500 M
Steps of Experience
Figure 6: Success rate of PointNav policies with Al trained and evaluated Habitat with kinematic or dynamic control. Left: Kinematic policies outperform the dynamic trained policies (+12% SR), even when evaluated using dynamic control. Right: Using kinematic control, we can train our robot for 10x more steps of experience than with dynamic control under identical compute budgets, despite training for 2.3x less wall-clock time.
How large is the sim2real gap for kinematic and dynamic trained policies? We evaluate the kinematic and dynamic policies on a Spot robot in the novel LAB environment described in Section Note that scans of LAB were not part of training. We evaluate 3 seeds of each policy over episodes in the real-world and report the average success rate (SR) and Success weighted by Path Length (SPL) [27] in Table (reported as a percentage for readability). Each control type is tested in 15 real-world episodes; one run of the Spot robot navigating LAB is shown in Figure 7. Success in the real-world is measured by computing final distance from the goal position using egomotion estimates provided by the Boston Dynamics SDK.
Train
Simulation
Reality
Sim2Real Gap
Simulator
Control
Noise
SR
SPL
SR
SPL
# Act.
# Coll.
SR
SPL
Habitat iGibson
Dynamic
60.0
38.7
40.0
28.2
107.9
41.2
20.0
10.5
Dynamic
20.0
14.0
67.7
46.6
76.8
12.9
-47.7
32.6
Habitat
Kinematic
93.3
76.9
100.0
82.7
26.4
3.1
-6.7
-5.8
iGibson
Kinematic
100.0
90.6
100.0
83.2
33.1
4.5
0.0
7.4
Habitat
Kinematic
Decoupled
80.0
72.0
100.0
87.8
27.1
2.5
-20.0
-15.8
Habitat
Kinematic
Coupled
80.0
74.1
100.0
88.8
22.7
2.8
-20.0
-1477
Table 1: Zero-shot sim2real transfer performance for the visual navigation policies. Success rate (SR) and path efficiency (SPL) are high for kinematic policies, while dynamic policies have lower performance due to the dynamics gap between the low-level control in training and the controller on the robot in the real-world
As reported in Table 1, all kinematic policies achieve a high success rate of 100% and SPL of 82- 83% (rows 3 and 4). On the other hand, the success rate drops to 40-67% for the dynamic policies (rows and 2). We notice that the dynamic policies typically commanded lower velocities, and often get stuck around obstacles (Figure 8). This is shown in the higher number of actions commanded and higher collision count for both dynamic policies; on average, dynamic policy trained in Habitat took 107.9 actions, and collided 41.2 times (row 1, columns 8 and 9), whereas a kinematic policy also trained in Habitat took 26.4 actions, and collided 3.1 times (row 3, columns 8 and 9). We attribute this to the impoverished experience of the dynamic policies; the policies did not learn robust navigation policies that could avoid obstacles during navigation. Additionally, they overfit to the levelevel behavior, which can be unstable at high velocities in sim, but not on hardware. Figure 8 (left) shows that the kinematic policy commands higher forward velocities, while the dynamic
7

---

Figure 7: One run the Spot robot navigating real-world L.A environment using kinematically trained policy from AI Habitat. The robot successfully navigates a hallway, moves around furniture and turns into the next hallway before stopping. In contrast, the native BD controllers without map can only reach visible goals.
policy commands slower velocities (right), which are often not achieved by the robot likely due to an obstacle." Successfully executed commands appear on the diagonal.
To improve kinematic simulation fidelity, we model actuation noise (difference between commanded and true velocity) on Spot and use it during kinematic training, similar to [16]. We collect 6,000 samples of decoupled (linear and angular velocities are actu- ated separately) and coupled (lincar and angular velocities tion noise. The parameters for noise each dimension, and details about data collec- tion and modeling can be found in the appendix. During training,
actuated together) actua-
sample from the Gaussian distribution for each dimension, and add it to the policy's predicted velocity.
We see that policies trained with noise also achieve 100% success in the real-world (rows and 6), and are able increase path efficiency (SPL) (4.6% using decoupled (row 4 vs. 5),
Resultant Vel. (m/s)
Kinematic Policy
Resultant Vel. (m/s)
Dynamic Policy
0.4
0.4
0.2
0.2
and 5.6% using coupled actuation noise (row
0.0
0.0
-0.2
-0.2
4 vs. 6)). The number of collisions and com- manded actions are also lower for these poli- cies, compared to kinematic policies trained with no noise (22.7 actions and 2.8 collisions vs. 26.4 actions and 3.1 collisions). These im- provements are due the added robustness that training with noise provides uncertainty during resulting fewer collisions in the real-world.
-0.4
-0.4
-0.5
0.5
Commanded Vel. (m/s)
Commanded Vel. (m/s)
Figure 8: Commanded vs. resultant velocities during
real-world trajectory rollouts for dynamically and kine- matically trained policies.
training forces the policy to take less risky actions
Conclusion, Limitations, and Future Work
In this work, we study the role of simulation fidelity for sim2real of visual navigation policies on three simulated and one real legged robot. Contrary expectations, find that higher simulation fidelity does not enable learning better high-level visual navigation policies. Dynamic policies tend overfit low-level simulation details, resulting in poor transfer to the real-world. On the other hand, kinematic policies are able to generalize well. These results raise important questions about the need for simulation fidelity for sim2real, especially in abstracted action spaces.
One limitation of this work is that we assume access to robust 'black box' controller on hardware. While most robots come shipped with canufacturer-proviiie differ between robots, and more robust noise modeling may be needed to better characterize the actuation noise. In the future, we plan to improve the modeling of real-world actuation noise by using a neural network conditioned on previous states and actions of the robot. Another limitation of this work is that specifically address only tasks that be abstracted with high-level actions, like navigation or object rearrangement (pick, place, open, close). We agree that there are many tasks that cannot be learned in a kinematic simulation e.g. tasks that require reasoning about low-level interactions with the environment, like dexterous manipulation, or low-level walking controllers. It essential create high-fidelity simulators for sim2real on such tasks.
controllers, the level of accuracy may
Actual velocity is measured using the Boston Dynamics SDK.

---

Acknowledgments
The Georgia Tech effort was supported in part by NSF, ONR YIP, ARO PECASE. JT was supported by an Apple Scholars AI/ML PhD Fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
License for dataset used Gibson Database of Spaces. License at http: /svll. stanf ord. edw. gibson2/asaets////// wag.ee/eettSss.eeeeeeeeee..eee pdf
References
[1] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: Learning near-perfect pointgoal navigators from 2.5 billion frames. In International Conference on Learning Representations (ICLR), 2020.
[2] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun,
J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research, In Interna- tional Conference Computer Vision (ICCV), 2019.
[3] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[4] B. Shen, F. Xia, C. Li, R. Martin-Martin, L. Fan, G. Wang. C. Pérez- Pérez-D'Arpino, S. Buch, S. Srivastava, L. P. Tchapmi. M. E. Tchapmi, K. Vainio. J. Wong, L. Fei-Fei, and S. Savarese. igibson 1.0: a simulation environment for interactive tasks in large realistic scenes. 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), page accepted. IEEE, 2021.
[5] Nvidia. Isaac Sim. https://developee wwtp::ealloe..i.idaaaaaaa nvidia. woom/ssaac--.. sim, 2020.
[6] M. Deitke, Han, Herrasti, A. Kembhavi, Kolve, R. Mottaghi, Salvador, Schwenk, E. VanderBilt, M. Wallingford, et al. Robothor: An open simulation-to-reet embodied Al plat- form. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 3164-3174, 2020.
[7] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, Kubilius, A. Bhandwaldar, N. Haber, M. Sano, al. ThreeDWorld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.0499 2020.
[8] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019 9-26,, 2020.
[9] Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environ- ment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[10] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[11] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax differen- tiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2 13281, 2021.
[12] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.

---

[13] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Un- dersander, W. Galuba, A. Westbury, A. X. Chang, et al. Habitat-mateeroot 3d dataset (hm3d); 1000 large-scale 3d environments for embodied ai. arXiv preprint arXiv: 109.08239. 2021.
[14] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, Song. A. Zeng, and Y. Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv: 1709.000000000 00,8, 2017.
[15] A. X. Chang, T. Funkhouser. L. Guibas, P. Hanrahan, Q. Huang, Z. Li. S. Savarese, M. Savva, Song, H. Su, et al. ShapeNer: An information-rich 3D model repository. arXiv preprint arXiv: 15 15120301 2015.
[16] Truong, S. Chernova, and D. Batra. Bi-directional domain adaptation for sim2real transfer of embodied navigation agents. IEEE Robotics and Automation Letters (RA-L), 6(2):2634 2021.
[17] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 20/9 International Conference on Robotics and Automation (ICRA), pages
IEEE, 2019.
[18] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), 2018
[19] J. Tobin, R. Fong, A. Ray. J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.
[20] A. Kadian, J. Truong, A. Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, S. Chernova, and D. Batra. Sim2real predictivity: Does evaluation in simulation predict real-world performance? IEEE Robotics and Automation Letters (RA-L), 2020.
[21] N. Yokoyama, S. Ha, and D. Batra. Success weighted by completion time: A dynamics-awan evaluation criteria for embodied navigation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021.
[22] Truong, D. Yarats, T. Li, Meier, S. Chernova, D. Batra, and A. Rai. Learning navigation skills for legged robots with learned robot embeddings. 2020.
[23] Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47)eebb59966, 2020. doi: 10. ht.1126/sciroboticssaee... 59666. URL https://www. science. sccis:/ org/ httg//di/abb///// 10. 1126/ scirobotics
scirobotics.abc5966. abc5986.
[24] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, x62))eab22822, 2022. 10 htt.01/10///ccroootsssaaaaaa ab22222.. URL sttps://wws.science wttrg//d//abs////
1125/scirobotics. abk2822.
[25] A. Kumar, Z. Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. Robotics: Science and Systems (RSS), 2021.
[26] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, P. P. Kaelbling, and Lozano-Pérez. Integrated task and motion planning. arXiv preprint arXiv:2010.01088. 2020.
[27] Anderson, A. Chang, D. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, et al. On Evaluation of Embodied Navigation Agents. arXiv preprint arXiv: 1807,06757, 2018.
[28) R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object- search strategies from human demonstrations at scale. In Proceedings of the IEEF/CVE Con- ference Computer Vision and Pattern Recognition, pages 3173-5183 2022.
10

---

[29] Т. Chen, S. Gupta, and A. Gupta. Learning exploration policies for navigation. arXiv preprint arXiv:1903.01959, 2019.
[30] D. S. Chaplot, D. Gandhi, S. Gupta, Gupta, and R. Salakhutdinov. Learning to explore using active neural slam. arXiv preprint arXiv:2004.05155 2020.
[31] S. Bansal, V. Tolani, S. Gupta, J. Malik, and C. Tomlin. Combining optimal control and learning for visual navigation in novel environments. In Conference on Robot Learning, pages 420-429, PMLR, 2020.
[32] W. Hess, D. Kohler, H. Rapp, and D. Andor. Real-time loop closure in 2d lidar slam. In ICRA, 2016.
[33] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar. Multi-agent manipulation via locomotion using hierarchical sim2real. arXiv preprint arXiv:1908.052: 2019.
[34] W. Yu, J. Tan, Y. Bai, E. Coumans, and S. Ha. Learning fast adaptation with meta strategy optimization. arXiv preprint arXiv: 12995, 2019.
[35] T. Li, N. Lambert, R. Calandra, F. Meier, and A. Rai. Learning generalizable locomotion skills with hierarchical reinforcement learning. arXiv preprint urXiv:/9099 /2324, 2019.
[36] T. Li, R. Calandra, D. Pathak, Y. Tian, F. Meier, and A. Rai. Planning in leamned latent action spaces for generalizable legged locomotion. IEEE Robotics and Automation Letters, 6(2): 2682-200 1882-2689,12 2021.
[37] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv: 10332, 2018.
[38] X. B. Peng, H. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. Robotics: Science and Systems (RSS), 2020.
[39] A. Rai, R. Antonova, S. Song. W. Martin, H. Geyer, and C. Atkeson. Bayesian optimiza- tion using domain knowledge on the atrias biped. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1771-1778. IEEE, 2018.
[40] Z. Fu, A. Kumar, A. Agarwal, Qi. J. Malik, and D. Pathak. Coupling vision and proprio- ception for navigation of legged robots. CVPR, 2022.
[41] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[42] D. Hoeller, L.. Wellhausen, F. Farshidian, and M. Hutter. Learning a state representation and navigation in cluttered and dynamic environments. IEEE Robotics and Automation Letters, (3):5081-5088, 2021.
[43] C. R. Garrett, R. Chitnis, R. Holladay, B. Kim, T. Silver, L. P. Kaelbling, and T. Lozano-Pérez Integrated task and motion planning- Annual review of control, robotics, and autonomous systems, 4:265-293, 2021.
[44] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pages 1470-1477 IEEE, 2011.
[45] Y. Lin, A. S. Wang, E. Undersander, and A. Rai. Efficient and interpretable robot manipulation with graph neural networks. IEEE Robotics and Automation Letters, 2022.
[46] Kuindersma, R. Deits, M. Fallon, Valenzuela, H. Dai, Permenter, T. Koolen, Marion, and R. Tedrake. Optinization-based locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 4055)449-445, 2016.
[47] Li, H. Geyer, C. G. Atkeson, and A. Rai. Using deep reinforcement learning to learn high- level policies on the atrias biped. In ICRA, pages 263-269. IEFF, 2019.
11

---

[48] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference Robot Learning (CoRI.), 2020.
[49] W. Yuan, C. Paxton, K. Desingh, and D. Fox. SORNet: Spatial object-centric representations for sequential manipulation. 5th Annual Conference on Robot Learning, 2021. URL https: / openreview
[50] Unitree robotics. https://ww.uuitrre..omm
[51] Boston dynamics. attpp://ssie http://wwwbbbttooddma.cc.c.c wtmpa//iiiiccoooottooeoeeoo// com/spot.
[52] Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese. Gibson env: Real-world percep- tion for embodied agents. In CVPR, 2018.
[53] Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots. Fast and efficient locomotion via learned gait transitions. Conference on Robot Learning, pages 773-783, PMLR, 2022.
[54] T. Li, J. Won, S. Ha, and A. Rai. Model-based motion imitation for agile, diverse and general- izable quadupedal locomotion. arXiv preprint 133322209 2021.
[55] Hwangho, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. arXiv preprint arXiv:1901.08632 2019.
12


Reviews:
[{"review_id": "anknv3fb3GC", "reviewer": "Reviewer_voBL", "summary": "The paper focuses on a practical evaluation of how higher-fidelity simulators impact sim2real transfer. It postulates that, contrary to current convention, simulators with high-fidelity are _actually detrimental_ in bridging the sim2real gap. To that end, the authors carried out a substantial amount of simulated and hardware experiments using a hierarchical control policy, where a high-level visual navigation policy is combined with a low-level dynamic controller to perform a Point Goal navigation task.\n\nTo corroborate their hypothesis, the authors perform an ablation study using three types of robots and two simulators and two types of simulations: kinematic, where the visual navigation policy running at 1 Hz sets the target linear and angular speeds and the simulator teleports the robot to the new location; and dynamic, where a low-level controller running at 240 Hz receives the target velocities and translates the signal to proper joint configurations and torque control.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and organised, with clear sections following a logical progression. The ablation study required a combinatorial number of computational experiments to be performed, exhausting the various combinations of robots, simulators and simulation types. Furthermore, the paper also presents real robotic experiments, for 15 different runs.\n\nThese results are analyzed and used as evidence to support the claim that lower-fidelity simulators are actually offer better sim2real transfer. Although the main idea of the paper is bold and refreshing, I think several points need to be addressed in order for such claim to be validated.\n\nFirstly, the authors use a common wall-clock period for training the visual navigation policy, but recognize that when using dynamical simulation the number of actual experience steps executed by the policy is much lower, due to the increased computational demand. They categorize this as an advantage for policies trained in simpler simulation and as one of the reasons a higher-fidelity simulator leads to worse sim2real transfer. Although this is a valid point from a practical standpoint, it is not a valid argument against the degree of fidelity of the physics simulation. That is to say, would there be a difference in sim2real performance if one could run a higher-fidelity simulation at the same rate as a lower-fidelity one? As it stands, this goes to show that a longer training experience improves the sim2real transfer. Since increasing the training time for dynamic policies is likely unfeasible, the training time for kinematic policies could be reduced to account for the experience difference.\n\nThe second point relates to the low-level controllers used. In the kinematic simulation, there is no need for a simulated low-level controller as the robot is simply teleported to the next state. In practice, the visual policy is providing set points for the built-in low-level controllers. On the other hand, for the dynamic simulation, it is necessary to include a simulated low-level controller. In the context of the current work, it was further necessary to use a different kind of simulated low-level in training and evaluation. The authors make the case that this serves as a proxy for the gap between the kinematic simulation and the built-in controllers response. However, this would not be the case the simulated low-level controller has a _biased_ error, since in that case the visual navigation policy would learn to compensate such bias. Since the kinematic simulator only has simulated noise in the euller integration, one can guarantee that error is unbiased. Furthermore, this makes it harder to untangle how much of the degraded performance during evaluation stems from the visual navigation policy or the low-level controllers.\n\nAdditionally, with regards to the claim in section 5 that the results presented indicate that the dynamic policies learned are overfitting to the simulator dynamics. I find that due to the two aforementioned points, it is hard to untangle the sources of error and it is not clear that overfitting is the leading cause of degraded sim2real transfer, especially when dynamic policies do not have enough experience to reach a plateau performance.\n\nFinally, although the current work is extensive, it covers a single task which can be reasonably well abstracted. One would imagine that in certain tasks, e.g. pouring or manipulating liquids, a higher-fidelity physics simulation could be more helpful. With that in mind, it would be important to study other tasks before making a general more general claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and brings a refreshing and impactful new perspective on sim2real transfer. However, there are important points that should be addressed before the paper is ready for publication, especially since it challenges the status-quo of high fidelity simulation.  \n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please refer to points made in the Strengths and Weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Nce-S339V2N", "reviewer": "Reviewer_MhUg", "summary": "This paper uses two different simulator abstractions to train a high-level navigation policy steering a legged robot around obstacles. The authors compare a kinematic simulation, where the robot is abstracted to a floating base, which is teleported based on the velocity commands, to a dynamical simulation, where a low-level controller is used to actuate the legs of the robot.\nAfter extensive experimentation, the authors conclude that policies trained in the simplified/kinematic simulation are superior to the ones trained in the dynamic setting. They achieve better performance in simulation and transfer better to a real Spot robot.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors conduct extensive experiments with a clearly presented and technically sound approach. However, I think the results are not as novel and surprising as presented. The real conclusion of the paper is that kinematic simulation is enough to train a navigation policy. This is not surprising since a legged robot with a good locomotion controller is an omnidirectional platform that can be steered with velocity commands. This abstraction has been shown to transfer easily to the real world before [1] (this work should have been cited)\nTwo simple reasons can explain the inferiority of the dynamically trained policies.\n1) As the authors explain, the dynamical simulation is much slower, so the policies used in the comparison had access to much less training data. In that case, why would we expect them to perform similarly? \n2) The authors use simplistic low-level controllers which “are unstable at higher velocities”. The left plot of figure 6 shows that kinematic policies also fail when deployed in the dynamical simulation, while they work much better in the real world. This suggests that the low-level controllers used in simulation are inferior to the real one provided by BD. If the robots keep falling in simulation, the high-level policy will struggle to learn.\n\n[1]: Hoeller, D., Wellhausen, L., Farshidian, F., & Hutter, M. (2021). Learning a State Representation and Navigation in Cluttered and Dynamic Environments. IEEE Robotics and Automation Letters, 6(3), 5081–5088. https://doi.org/10.1109/LRA.2021.3068639", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the experiments are relevant to the robotics community. I believe that the conclusions drawn by the authors are not well founded and should not be published in the current state. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "See above.\n\nMinor issues:\n\n“...can save simulation … and developer time spent building and designing these controllers.” Wouldn’t these controllers be needed on the real robot anyway?\n\n“In this work, we show that such end-to-end learning is also possible for complex, legged robots.” -> what is the difference between developing a navigation module for a wheeled or legged platform if the low-level controller is provided?\n\n[40] is not a blind policy.\n\nIn section 5, the authors add noise to the translation of the kinematic policy, thus showing that they need to close the sim2real to improve performance. This contradicts the message of the paper.\n"}}, {"review_id": "ADNIbe8P1PD", "reviewer": "Reviewer_Soyb", "summary": "The authors hypothesize that a decrease in robot simulation fidelity may result in a higher efficacy sim2real transfer of deep reinforcement learning policies for visual navigation tasks. A dichotomy is drawn between a low-frequency kinematic simulation environment (low-fidelity simulation) and a high-frequency dynamic simulation environment (high-fidelity).\n\nThis hypothesis is tested via training on 3 simulated quadrupedal robots across 2 simulation environments, and testing on one vehicle in the real-world setting.\n\nRobot behavior is governed by a hierarchical control architecture, with a high-level policy mapping depth imagery and GPS+compass measurements to velocity commands, and one of two low-level controllers (kinematic and dynamic) mapping velocity to a center-of-mass goal position or direct joint angles.\n\nThe authors find that use of kinematic-only simulation environments result in stronger sim2real transfer given equal training time.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n\nThe work applies a thorough experimental methodology, examining performance of learned policies in simulation and on a real-world Spot quadrupedal robot.  Performance comparison across multiple simulators (Habitat and iGibson) provides an additional degree of thoroughness, providing insight into current state-of-the-art simulation tools, as well as preventing limitations in a single simulator implementation from biasing the paper's results.  \n  \n- Weaknesses\n  - Insufficient discussion of negative effects of kinematic\n    simulation:\n\nThe need for agent knowledge of vehicle dynamics is repeatedly cited in the sim2real robotics literature, especially for complex motion platforms such as quadrupedal robots.  This is due to the lack of conformity of idealized vehicle kinematics to a real-world dynamic multi-legged system.  The paper in its current form fails to thoroughly address why the proposed system is immune to this constraint, and neglects to discuss task conditions under which a policy learned in kinematic simulation would be inapplicable.\n\n  - Artificial penalization of higher-complexity simulation:\n\nA key claim in this work is the stark contrast in efficacy and generalization of policies trained in physics-absent kinematic environments versus those in high-fidelity dynamic environments. However, this contrast is primarily due (as noted in Section 5) to the \"poverty\" of experience provided to agents trained in a dynamic environment.  The quantity of training provided to agents is controlled by wall clock time of the training task, resulting in 1/20th of the experience steps available to dynamic agents compared to their kinematic equivalent.  Benchmarking performance as a function of training wall-clock weakens the claim that lower fidelity simulation yields stronger sim2real transfer, instead reinforcing the need for sufficient experience for robust learned policy development.  Should the comparison been drawn between high and low fidelity training environments under the condition of equal steps of experience, stronger claims could be made regarding  the benefit of low-fidelity training environments.\n\nHowever, the contrast in requisite training time between low and high fidelity simulation is still important to note.  One could forsee a compelling claim being made for scrutinizing the costs and benefits of higher fidelity simulation, drawing a conclusion that under some task conditions, high-fidelity learning environments are needlessly expensive computationally.  Nonetheless, the wide claim of the higher sim2real transfer for lower fidelity simulation is not fully supported in the work as it stands.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "On the whole, this work provides a reasonable case for the use of lower-fidelity simulation for use in sim2real transfer. However, there are some crucial points that are analyzed in such a way as to limit the strength of the claims made. Despite relatively strong technical quality and thoroughness experimentally, there remain a select few outstanding questions on the part of the reviewer which provoke hesitation in recommending acceptance.  \n\nPost-Rebuttal: The authors' revisions have strengthened the paper and addressed several of the original concerns expressed above, and the rating for this paper has been amended accordingly. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Given the nature of this conference's 8-page paper limit, much of the below would be well-suited to inclusion in the paper's abstract, should space constraints prove problematic.  \n\n1. Additional discussion of kinematic-dynamic environment tradeoff: The work would be improved by addressing conditions under which kinematic simulation is unsuitable for sim2real transfer.  For the PointGoal Navigation task focused upon in this work, it is clear that simulation at kinematic fidelity is sufficient to produce an efficacious policy for real-world operation. However, there are many other task conditions in which robot kinematics alone are insufficient for commanding a vehicle in real-world environments. These should be noted and discussed.\n\n2. Description of high-level policy parameters: The work is very limited in its description of the exact architecture and parameterization of the high-level policy and reward function. Additional mathematical description would be beneficial.\n\n3. Clarify additions to work in Ref. [22] (Truong et al.): Much of the architecture and experimental setup appears to bear strong similarities to work put forward in reference 22 of the paper. A more explicit disambiguation of changes made in this work over Ref. 22 would serve to make clear why some of the concerns regarding dynamic modeling in that work do not apply here. Furthermore, some implementation details may bear repeating in this paper, as elements present in that work (such as mathematical rigor) are cited, but not expanded upon in a thorough way."}}, {"review_id": "3SchM4uGqS6", "reviewer": "Reviewer_BmX5", "summary": "The paper challenges the common assumption that sim2real transfer benefits from increasing physics fidelity and presents a contrary hypothesis that sim2real transfer can be improved with lower physics fidelity (abstracted physics). The authors conduct a systematic evaluation with three legged robots on two simulation platforms in the domain of visual PointGoal navigation. The authors claim that higher fidelity hurts performance because 1) it’s slow and therefore prevents large-scale training, 2) leads to overfitting to inaccuracies in simulation physics. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper conducted a large-scale study of sim2real transfer of legged robot locomotion: across 3 different robots, on 2 different simulators. All combination of (train, test) x (sim, fidelity) has been thoroughly evaluated. The authors also demonstrated the performance of their policy on real hardware in an unseen environment. More importantly, the paper proposes a very counterintuitive conclusion that challenges the almost self-evident assumption that sim2real gap involves creating simulators of increasing fidelity. This topic is very important and relevant for roboticists and practitioners interested in sim2real transfer for locomotion tasks and beyond. The lessons learned here have the potential to be widely applicable to the wider robotics domain when action abstraction can be applied (e.g. grasping, manipulation, etc). It’s appreciated that the authors take the effort to conduct this thorough study in sim and in real to answer this research question: do we need a higher-fidelity simulator?\n\nHowever, the paper also has a major weakness that needs to be addressed. The authors claim that high fidelity simulation hurts performance because 1) it’s slow and therefore prevents large-scale training, 2) leads to overfitting to inaccuracies in simulation physics. Unfortunately, these two aspects are entangled together, and therefore the conclusion is not entirely convincing. To be more specific, in Figure 6 (right plot), it’s very clear that the policies trained with dynamic simulation (dotted lines) have not converged at all. The authors gave a training budget of 3 days and showcased that the policies trained with kinematic simulation (converged) outperform those trained with dynamic simulation (not converged). Their argument is somewhat one-sided, because what if the practitioners have a training budget of 14 days, rather than 3 days? In this case, maybe training with dynamic simulation is better?\n\nFurthermore, the authors argue that policies trained with dynamic simulation “overfit” to the simulator dynamics, failing to generalize to a new setting (e.g. a black-box controller of a real robot and the real-world dynamics of the environment). The validity of this argument is questionable, because, again, this is based on the results of evaluating policies that have not converged yet. It’s very unclear if the policies “overfit” to the specific simulator dynamics, or if the policies actually underfit: they haven’t even had enough training examples to fit the training simulator, not to mention the other test simulator. In general, we should be very careful about any conclusion drawn from half-trained policies because the evaluation results from non-converged neural network-based policies can be arbitrary and meaningless.\n\nIt would be great if the authors can disentangle their two arguments and present a more impartial view for the practitioners and researchers to help them decide whether to increase or decrease the fidelity of their simulator. Feel free to check out the “Issues” section below for some concrete suggestions. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As mentioned above, although the problem domain is extremely relevant and interesting, and the authors did an amazing job of running thorough experiments across robot platforms and simulators and presenting their findings in the paper, the main conclusion of the paper is a bit one-sided and begs for further investigation. Since this paper is very focused and has essentially one strong message to deliver, this message has to be bullet-proof. If the authors can address the issues mentioned below, I am more than happy to change my rating/recommendation. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "To help present a more balanced, impartial view of whether to increase/decrease simulator fidelity for researchers and practitioners, here are some concrete suggestions:\n\nTraining:\nModel A. (Done already) Kinematic simulation: assume a perfect controller, fast to simulate\nModel B. (Done already) Kinematic simulation with actuation noise modeled from real-world data: fast to simulate, seems like a very promising way to close the sim2real gap. \nModel C. (New) Dynamic simulation with Raibert-style controller: slow to simulate, train until convergence*\n\nEvaluation:\nEval A. (New) Dynamic simulation with the actual controller of the robot (if possible, it seems that the controller for A1 is open-sourced?)\nEval B. (Done already) Real-world with the actual controller of the robot\n\nThe research questions that hopefully these additional experiments can help answer are:\nModel C: how long does it take for the policies trained in dynamic simulation to fully converge? If the researchers have more training time budget, is this a better option?\nModel C: once converged, how is its performance compared to kinematic policies? Does the conclusion of the paper still hold? Does “overfitting to inaccurate physics” actually happen?\nEval A: this is important because it isolates the actuation sim2real gap (there is no visual sim2real gap). It will solidify the authors’ claim that higher physics fidelity leads to “overfitting” of the inaccuracy of physics simulation IF model A outperforms model C in Eval A. Also, if model B is the best in Eval A, the authors might claim that doing kinematic simulation while modeling actuation noise with real-world data is the best way to close the sim2real gap and be robust against different testing dynamics (environment or black-box controller), which would be a great lesson to share with the community. Right now, this argument is not bulletproof yet.\nEval B is obviously the final test, with some confounding factor of visual sim2real gap. But still, it would be great to show model B outperform model A and C here.\nIn the very end, it would be great to present a fair assessment of when to increase/decrease physics fidelity for better sim2real performance. I am not fully convinced that the answer is as black-and-white as presented in the current paper.  \n\nTwo minor comments / questions\n- What causes the difference in dynamic simulation between Habitat and iGibson? Aren’t they both using bullet as the underlying physics engine? Maybe it’s two different versions of bullet?\n- It would be great if the authors can be consistent with the name of the simulators used. iGibson is mentioned in some places and Gibson is mentioned in others. Given the citation, it should probably always be iGibson?\n"}}]

Author rebuttal:

Decision: Accept (poster)