Legged Locomotion in Challenging Terrains using Egocentric Vision
Authors: Ananye Agarwal
Ashish Kumar
Jitendra Malik
Deepak Pathak
Venue: CoRL 2022 Oral (2022)
Keywords: legged robots
reinforcement learning
vision
locomotion
walking

Abstract: Animals are capable of precise and agile locomotion using vision. Replicating this ability has been a long-standing goal in robotics. The traditional approach has been to decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result on a medium-sized quadruped robot using a single front-facing depth camera. The small size of the robot necessitates discovering specialized gait patterns not seen elsewhere. The egocentric camera requires the policy to remember past information to estimate the terrain under its hind feet. We train our policy in simulation. Training has two phases - first, we train a policy using reinforcement learning with a cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning. The resulting policy transfers to the real world and is able to run in real-time on the limited compute of the robot. It can traverse a large variety of terrain while being robust to perturbations like pushes, slippery surfaces, and rocky terrain. Videos are at https://vision-locomotion.github.io

Legged Locomotion in Challenging Terrains using Egocentric Vision
Ananye Agarwal' 1 Ashish Kumar" 2, Jitendra Malik?? Deepak Pathak f1
Carnegie Mellon University, UC Berkeley
Figure I:
Our robot can traverse a variety of challenging terrain in indoor and outdoor environments, urban and natural settings during day and night using single front- facing depth camera. The robot can traverse curbs, stairs and moderately rocky terrain. Despite being much smaller than other commonly used legged robots, it is able to climb stairs and curbs of similar height. Videos httpa://visione. httr/vviiiioooooooooooo www.oo..gtiiit github.
Abstract: Animals are capable of precise and agile locomotion using vision.
Replicating this ability has been long-standing goal in robotics. The traditional approach has been decompose this problem into elevation mapping and foothold planning phases. The elevation mapping, however, susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implau- sible. In this paper, we present the first end-to-end locomotion system capable of traversing stairs, curbs, stepping stones, and gaps. We show this result medium-sized quadruped robot using a single front-facing depth camera. The small size of the robot necessitates discovering specialized gait patterns not elsewhere. The egocentric camera requires the policy to remember past informa- tion estimate the terrain under its hind feet. We train our policy in simulation. Training has two phases first, we train a policy using reinforcement learning with cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning. The resulting policy transfers to the real world and is able to run in real-time on the limited compute of the robot. It can traverse a large variety of terrain while being ro- bust perturbations like pushes, slippery surfaces, and rocky terrain. Videos are gi github.ii io.
at https: http:/vviiiiiooooooooooooo
Equal Contribution. 'qqual Advising.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Introduction
Of what is vision during locomotion? Clearly, there is a role of vision in navigation using maps or landmarks to find a trajectory in the 2D plane to a distant goal while avoiding obstacles. But given local direction in which to move, it turns out that both humans [1] and robots [2, 3] can do remarkably well at blind walking, Where vision becomes necessary is for locomotion in challenging terrains. In an urban environment, staircases are the most obvious example. In the outdoors, we can deal with rugged terrain such as scrambling over rocks, or stepping from stone to stone to cross a stream of water. There is fair amount of scientific work studying this human capability and showing tight coupling of motor control with vision [4, 5, 6]. In this paper, we will develop this capability for a quadrupedal walking robot equipped with egocentric depth vision. We use a reinforcement learning approach trained in simulation, which we are directly able to transfer to the real world. Figure I and the accompanying videos shows some examples of our robot walking guided by vision.
Humans receive an egocentric stream of vision which is used to control feet placement, typically without conscious planning. As children we acquire it through trial and error [7] but for adults it an automatized skill. Its unconscious execution should not take away from its remarkable sophistication. The footsteps being placed now are based on information collected some time ago. Typically, we don't 't look at the ground underneath our feet, rather at the upcoming piece of ground in front of us a few steps away[1, 4, 6]. short term memory is being created which persists long enough to guide foot placement when we are actually over that piece of ground. Finally, note that we learn to walk through bouts of steps, not by executing pre-programmed gaits [7].
We take these observations about human walking as design principles for the visually-based walking controller for an A1 robot. The walking policy is trained by reinforcement learning with recurrent neural network being used as short term memory of recent egocentric views, proprioceptive states, and action history. Such a policy can maintain memory of recent visual information to retrieve characteristics of the terrain under the robot or below the rear feet, which might no longer be directly visible in the egocentric view.
In contrast, prior locomotion techniques rely on the metric elevation map of the terrain around and under the robot [8, 9, 10] to plan foot steps and joint angles. The elevation map is constructed by fusing information from multiple depth images (collected over time). This fusion of depth images into single elevation map requires the relative pose between cameras at different times. Hence, tracking is required in the real world to obtain this relative pose using visual or inertial odometry. This is challenging because of noise introduced in sensing and odometry, and hence, previous methods add different kinds of structured noise at training time to account for the noise due to pose estimation drift [11, 12, 13]. The large amount of noise hinders the ability of such systems to perform reliably on gaps and stepping stones. We use vision as a first class citizen and show all the uneven terrain capabilities along with a high success rate on crossing gaps and stepping stones.
Obstructed
Topple
26cm
ANYmaiC
Spot
A1
(a) Robot size comparison
(b) Challenges due to size
(c) Emergent hip abduction
Figure 2: A smaller robot (a) faces challenges in elimbing stairs and curbs due to stair obstructing its feet while going up and tendency to topple over when coming down (b). Our robot deals with this by climbing using a large hip abduction that automatically emerges during training (c).
The design principle of not having pre- pre-programmed gait priors turns out to be quite advantageous for our relatively small robot (fig. 2). Predefined gait priors or reference motions fail to generalize to obstacles of even reasonable height because of the relatively small size of the quadruped. The emergent behaviors for traversing complex terrains without any priors enable our robot with a hip joint height of 28cm to traverse the stairs of height upto 25cm, 89% relative to its height, which is significantly higher than any existing methods which typically rely on gait priors.
Since our robot is small and inexpensive, it has limited onboard compute and sensing. It uses single front-facing D435 camera for exteroception. In contrast, AnymalC has four such cameras in addition to two dome lidars. Similarly, Spot has 5 depth cameras around its body. Our policy computes actions
Al standing height is 40cm measured by us. Spot, ANYmalC both are 70cm tall reported here and here.

---

with single feedforward pass and requires no tracking. This frees us from running optimization for MPC or localization which requires expensive hardware to run in real-time.
Overall, this use of learning "all the way" and the tight coupling of egocentric vision with motor control are the distinguishing aspects of our approach.
2 Method: Legged Locomotion from Egocentric Vision
Our goal is to learn a walking policy that maps proprioception and depth input to target joint angles at 50Hz. Since depth rendering slows down the simulation by an order of magnitude, directly training this system using reinforcement learning (RL) would require billions of samples to converge making this intractable with current simulations, We therefore employ two-phase training scheme. In phase I, we use low resolution scandots located under the robot as proxy for depth images. Scandots refer to set of (ır, y) coordinates in the robot's frame of reference which the height of the terrain is queried and passed as observation at each time step (fig. 3). These capture terrain geometry and are cheap to compute. In phase 2, we use depth and proprioception as input to an RNN to implicitly track the terrain under the robot and directly predict the target joint angles at 50Hz. This is supervised with actions from the phase policy. Since supervised learning orders of magnitude more sample efficient than RL, our proposed pipeline enables training the whole system on a single GPU in few days. Once trained, our deployment policy does not construct metric elevation maps, which typically rely on metric localization, and instead directly predicts joint angles from depth and proprioception.
One potential failure mode of this two-phase training is that the scandots might contain more information than what depth can infer. To get around this, we choose scandots and camera field-of- view such that phase 2 loss low. We formally show that this guarantees that the phase 2 policy will have close to optimal performance in Thm 2.1 below.
Theorem 2.1. M = (S.S.A.P,R.)) be an MDP with state space S, action space A, transition function P: S x AA+S, S, reward function R A x S + R and discount factor factor Let (s) be the value function of the phase I policy that is trained to be close to optimal value function V*(s), ie.. S, and n² (s) be the greedy phase I policy obtained from (=()). Suppose the phase 2 policy operates in different state space S' given by a mapping f : 5 S' the phase 2 policy is close to phase I (z) - 7 (f(ss) < y s and R, P are Lipschitz continuous, then the return of phase 2 policy is close to optimal everywhere, i.e., Vss V*((s) v** (f())) 2ertne
(**))) (s) < <v
where e x Eves V*)s) is a large but bounded constant. (proof in sec. A)
We instantiate our training scheme using two different architectures. The monolithic architecture is an RNN that maps from raw proprioception and vision data directly to joint angles. The RMA architecture follows [3, 14), and contains an MLP base policy that takes Ye (which encodes the local terrain geometry) along with the extrinsics vector z, (which encodes environment parameters [3]), and proprioception x, to predict the target joint angles. An estimate of 7t is generated by an RNN that takes proprioception and vision inputs. While the monolithic architecture is conceptually simpler, it implicitly tracks Yt and z; in its weights and is hard to disentagle. In contrast, the RMA architecture allows direct access to each input (7r 2,) through latent vectors. This allows the possibility of swapping sensors (like replacing depth by RGB) or using one stream to supervise the other while keeping the base motor policy fixed.
2.1 Phase 1: Reinforcement Learning from Scandots
Given the scandots m,, proprioception X,, commanded linear and angular velocity ugnd (ogmd, (osm", we learn policy using PPO without gait priors and with reward functions that minimize energetics to walk on variety of terrains. Proprioception consists of joint angles, joint velocities, angular velocity, roll and pitch measured by onboard sensors in addition to the last policy actions a,-1. Let o (m, xx, m denote the observations. The RMA policy also takes privileged information e, as input which includes center-of-mass of robot, ground friction, and motor strength.
3

---

Simulation
Phase 1 (in Simulation)
Phase Simulation)
Robot
Reinforcement Learning with privileged information
Supervised Learning to learn from onboard sensing
RMA Architecture
Monolithic Architecture
Scandots
Phase
Phase
Egocentric Depth
m
GRU
a
11 GRU
MLP
m
MLP
M.P P
Privieged
Sensing
Env Params (mass, friction....
Scandots
m
e,
Phase 2
Hegreas
Copy and Freeze
Phase 2
Copy to
intiaizze
Regnas
Onboard Sansing
Propriaception Ego Depth d
x
GRU
1 MUP
å
n GRU
Corv
Target Action
a,
GRU
Figure 3: We train locomotion policy in two phases to avoid rendering depth for too many samples. In phase 1, use RL train a policy nT that has access scandots that cheap compute. In phase 2, we use n' to provide ground truth actions which another policy x² trained to imitate. This student depth map from the front camera. We consider two architectures (1) monolithic one which is a GRU trained to output joint angles with raw observations as input (2) decoupled architectrre trained using RMA [3] that is trained to estimate vision and proprioception latents that condition base feedforward walking policy.
access to
Monolithic
The scandots m, are first compressed Y and then passed with the rest of the observations to GRU that predicts the joint angles.
7 MLP (m.)
(1)
a, GRU, (xz, 7t, u. (mdd
(2)
the subscript on the GRU indicates that stateful.
RMA Instead of using monolithic memory based architecture for the controller, use MLP as the controller, pushing the burden of maintaining memory and state on the various inputs to the MLP. Concretely, we process the environment parameters (e;) with an MLP and the scandots (m;) with GRU to get z, and 7/ respectively which are given as input to the base feedforward policy.
7/ GRU, (m²)
(3)
2: = MLP (c:)
(4)
= MLP (x,, Tt, Zı, u, m
(5)
Both the phase architectures are trained using PPO [15] with backpropagation through time [16] truncated at 24 timesteps.
Rewards We extend the reward functions proposed in [3, 17] to simply penalizing the energy consumption along with additional penalties to prevent damage hardware complex terrain (sec. B). Importantly, we do not impose any gait priors or predefined foot trajectories and let optimal gaits that are stable and natural to emerge for the task.
Absolute work penalty -IT ql where T are the joint torques. We use the absolute value so that the policy does not learn to get positive reward by exploiting inaccuracies in contact simulation.
wal where v, is velocity of robot in forward direction and W is yaw angular velocity (xc, are coordinate axes fixed to the robot).
Command tracking u ل Uz |w
Foot jerk penalty Eiez ||f fiull where f is the force at time on the ,th rigid body and F the set of feet indices. This prevents large motor backlash.

---

Feet drag penalty Eiex IN (1e² + +l/g)) where I is the indicator function, and e ui is velocity of 7t rigid body. This penalizes velocity of feet in the horizontal plane if in contact with the ground preventing feet dragging on the ground which can damage them. Collision penalty LiccuT [ 2 0.1N where C, T are the set of calf and thigh indices. This penalizes contacts at the thighs and calves of the robot which would otherwise graze against edges of stairs and discrete obstacles.
Survival bonus constant value at each time step to prioritize survival over following commands in challenging situations.
Similar to [18] we generate different sets of terrain (fig. 5) of varying difficulty level. Following [3], we generate fractal variations over each of the terrains to get robust walking behaviour, At training time, the environments are arranged in 6 10 matrix with each row having terrain of the same type and difficulty increasing from left to right. We train with a curriculum over terrain [18] where robots are first initialized on easy terrain and promoted to harder terrain they traverse more than half its length. They are demoted to easier terrain if they fail to travel at least half the commanded distance g where T is maximum episode length. We randomize parameters of the simulation (tab. 3) and add small i.i.d. gaussian noise to observations for robustness (tab. 2).
Training environment
2.2 Phase 2: Supervised Learning
In phase 2, we use supervised learning distil the phase policy into architecture that only has access to sensing available onboard: proprioception (xx) and depth d,.
Monolithic We create a copy of the recurrent base policy 2. We preprocess the depth map through a convnet before passing it to the base policy.
de ConvNet (d:)
(6)
a, GRU,(X,, åz, a
(7)
We train with DAgger [19] with truncated backpropagation through time (BPTT) minimize mean squared error between predicted and ground truth actions | ayl² In particular, we unroll the student inside the simulator for N = 24 timesteps and then label each of the states encountered with the ground truth action a, from phase 1.
Instead of retraining the whole controller, only train estimators of Yr and Zz, and use the same base policy trained in phase I (eqn. 5). The latent 5, which encodes terrain geometry, is estimated from history of depth and proprioception using GRU. Since the camera looks in front of the robot, proprioception combined with depth enables the GRU to implicitly track and estimate the terrain under the robot. Similar to [3]. history of proprioception is used to estimate extrinsics 2.
RMA
d ConvNet (d.)
(8)
f/2 GRU, (xe, u, and dz)
(9)
2. GRU, (Xe, un uamd)
(10)
a, = MLP (x,, uim .r.
(11)
As before, this is trained using DAgger with BPTT, The vision GRU 9 and convnet 8 are jointly trained minimize 1| 7ell while the proprioception GRU 10 minimizes || 2,
Deployment The student can be deployed as-is on the hardware using only the available onboard compute. It is able to handle camera failures and the asynchronous nature of depth due to the randomizations we apply during phase 1. It is robust to pushes, slippery surfaces and large rocky surfaces and can climb stairs, curbs, and cross gaps and stepping stones.
3 Experimental Setup
We use the Unitree AI robot pictured in Fig. 2. The robot has 12 actuated joints. The robot has front- facing Intel RealSense depth camera in its head. The onboard compute consists of the UPboard and Jetson NX. The policy operates at 50Hz and sends joint position commands which are converted to
5

---

Average z- Diisplacement (1)
Mean Time to Fall (s)
Terrain
RMA
MLith
Noisy
Blind
RMA
MLith
Noisy
Blind
Slopes
43,98
44.09
36.14
34.72
88.99
85.68
70.25
67.07
Stepping Stones
18.83
20.72
1.09
1.02
34.3
41.32
2.51
2.49
Stairs
31.24
42,4
6.74
16.64
69.99
90.48
15.77
39.17
Discrete Obstacles
40.13
28.64
29.08
32.41
85.17
57.53
59.3
66.33
Total
134.18
135.85
73.05
84.79
278.45
275.01
147.83
175.06
Table 1: We measure the average displacement along the forward axis and mean time to fall for all methods on different terrains in simulation. For each method, train single policy for all terrains and use that for evaluation. We see that the monolithic (MLith) and RMA architcctures of method outperform the noisy and blind baselines by 60-90% terms of total mean time to fall and average displacement. Vision is not strictly necessary for traversing slopes and the baselines make significant progress on this terrain, however, MLith and RMA travel upto 25% farther. The difference is more stark on stepping stones where blind and noisy baselines barely make any progress due to not being able locate positions of the stones, while MLith and RMA travel for around 20m. Noisy and blind make some progress on stairs and discrete obstacles, but our methods travel upto 6,3 times farther.
torques by low-level PD controller running 400Hz. Depth map is obtained from Intel RealSense camera inside the head of the robot. The camera captures images every 100ms + 20ms at a resolution of 480 848. We preprocess the image by cropping 200 white pixels from the left, applying nearest neighbor hole-filling and downsampling to 58 x 87. This is passed through a backbone to obtain the compressed de 8, 6 which is sent over a UDP socket to the base policy. This has latency of 10 10ms which we account for during phase 2.
We use the IsaacGym (IG) simulator with the legged_gym library [18] to train our walking policies. We construct large terrain map with 100 sub sub-terrains arranged in 20 10 grid, Each row has the same type of terrain arranged in increasing difficulty while different rows have different terrain.
Baselines We compare against two baselines, each of which uses the same number of learning samples for both RL phase and supervised learning phase.
Blind policy trained with the scandots observations m, masked with zeros. This baseline must rely on proprioception traverse terrain and helps quantify the benefit of vision for walking. Noisy Methods which rely on elevation maps need to fuse multiple depth images captured over time to obtain a complete picture of terrain under and around the robot. This requires camera pose relative to the first depth input, which is typically estimated using vision or inertial odometry [11, 13, 12]. However, these pose estimates are typically noisy resulting in noisy elevation maps [8]. To handle this, downstream controllers trained on this typically add large noise in the elevation maps during training. Similar to [8], we train a teacher with ground truth, noiseless elevation maps in phase and distill it to student with large noise, with noise model from [8], added to the elevation map. We simulate a latency of 40ms in both the phases of training to match the hardware. This baseline helps in understanding the effect on performance when relying on pose estimates which introduce additional noise in the pipeline.
4 Results and Analysis
Simulation Results We report mean time to fall and mean distance travelled before crashing for different terrain and baselines in Table 1. For each method, we train a single policy for all terrains and use that for evaluation. Although the blind policy makes non trivial progress on stairs, discrete obstacles and slopes, is significantly less efficient at traversing these terrains. On slopes our methods travel upto 27% farther implying that the blind baseline crashes early. Similarly, on stairs and discrete obstacles the distance travelled by our methods is much greater (upto 90%). On slopes and stepping stones the noisy and blind baselines get similar average distances and mean time to fall and both are worse than our policy. This trend even more significant on the stepping stones terrain where all baselines barely make any progress while our methods travel upto 20m. The blind policy has no way of estimating the position of the stone and crashes as soon as steps into the gap. For the noisy policy, the large amount of added noise makes it impossible for the student to reliably ascertain

---

Upstairs 17cm high, 30cm deep
Downstairs 17cm high, 30cm deep
Stepping Stones
Gaps
30cm wide, 15cm apart
26cm apart
Success
#Stairs
Success
#Stairs
Success
#Stones
Success
Ours
100%
13
Ours
100%
13
Ours
94%
9.4
Ours
100%
Blind
0%
2.2
Blind
100%
13
Blind
0%
0
Blind 0%
Figure 4: We show success rates and time-to-failure (TTF) for method and the blind baseline curbs, stairs, stepping stones gaps. We use a separate policy for stairs which is distilled to front camera, and use a separate policy trained on stepping stones distilled to the top camera which we use for gaps and stepping stones. We observe that our method solves all the tasks perfectly except for the stepping stone task in which the robot achieves 94% success. The blind haseline fails completely some progress, but fails complete the entire staircase even once, which is expected given the small size of the robot. The blind policy completes the downstairs task 100% success, although learns a very high impact falling gait to solve the task. In our experiments, the robot dislocates its real right leg during the blind downstairs trials.
gaps and stepping stones. For upstairs, it makes
the location of the stones since it cannot rely on proprioception any more. We note that the blind baseline is better than the noisy one on stairs. This is because the blind baseline has learnt to use proprioception to figure out location of stairs. On the other hand, the noisy policy cannot learn to use proprioception since it is trained via supervised learning. However, the blind baseline bumps into stairs often is not very practical to run on the real robot. The noisy baseline works well in [8] possibly because of predefined foot motions which make the phase 2 learning easier. However, as noted in sec. 1, predefined motions will not work for our small robot.
Real World Comparisons We compare the performance of our methods to the blind baseline in the real world. particular we have 4 testing setups as shows fig. 4: Upstairs, Downstairs, Gaps and Stepping stones. While train single phase 1 policy for all terrain, for running baselines, obtain different phase policies for stairs vs. stepping stones and gaps. Different phase policies obtained by changing the location of the camera. We use the in-built camera inside the robot for stairs and a mounted external camera for stepping stones and gaps. The in-built camera is less prone to damage but the stepping stones gaps are not clearly visible since it is horizontal. This is done for convenience, but we also have policy that traverses all terrain using the same mounted camera.
We see that the blind baseline is incapable of walking upstairs beyond few steps and fails to complete the staircase even once. Although existing methods have shown stairs for blind robots, we note that our robot is relatively smaller making it a more challenging task for blind robot. On downstairs, we observe that the blind baseline achieves 100% success, although learns to fall on every step and stabilize leading to very high impact gait which led to the detaching of the rear right hip of the robot during our experiments. We additionally show results in stepping stones and gaps, where the blind robot fails completely establishing the hardness of these setups and the necessity of vision to solve them. We show a 100% success on all tasks except for stepping stone on which we achieve 94% success, which is very high given the challenging setup.
Urban Environments We experiment on stairs, ramps and curbs (fig. 1). The robot was success- fully able to go upstairs as well as downstairs for stairs of height upto 24cm in height and 28cm as the lowest width. Since the robot has to remember terrain under its body from visual history, sometimes misses step, but shows impressive recovery behaviour and continues climbing or descending. The robot able to climb curbs and obstacles as high as 26em which is almost as high as the robot This requires emergent hip abduction movement because the small size of the robot doesn't leave any space between the body and stair for the leg to step up. This behavior emerges because of our tabula rasa approach to learning gaits without reliance priors datasets of natural motion.
7

---

Gaps and Stepping Stones We construct an obstacle course consisting of gaps and stepping stones out of tables and stools (fig. 4). For this set of experiments we use policy trained on stepping stones on gaps, and distilled onto the top camera instead of the front camera. The robot achieves 100% success rate on gaps of upto 26cm from egocentric depth and 94% on difficult stepping stones. The stepping stones experiment shows that our visual policy can learn safe foothold placement behavior even without an explicit elevation map or foothold optimization objectives. The blind baseline achieves zero success rate on both tasks and falls soon as any gap is encountered.
Natural Environments We also deploy our policy on outdoor hikes and rocky terrains next river beds (fig. 1). We see that the robot is able to successfully traverse rugged stairs covered with dirt, small pebbles and some large rocks. It also avoids stumbling over large tree roots on the hiking trail. On the beach, we see that the robot is able to successfully navigate the terrain despite several slips and unstable footholds given the nature of the terrain. We see that the robot sometimes gets stuck in the crevices and in some cases shows impressive recovery behavior as well.
5 Related Work
Legged locomotion an important problem which has been studied for decades. Several classical works use model based techniques, or define heuristic reactive controllers to achieve the task of walking [20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32J.This method has led to several promising results in the real world, although they still lack the generality needed to deploy them in the real world. This has motivated work in using RL for learning to walk in simulation [15, 33, 34, 35], and then successfully deploy them in a diverse set of real world scenarios [36, 37, 38, 39, 40, 41, 36, 42]. Alternatively, policy learned in simulation can be adapted at test-time to work well in real environments [43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53]. However, most of these methods are blind, and only use proprioceptive signal to walk.
Legged locomotion
Locomotion from Elevation Maps To achieve visual control of walking, classical methods de- couple the perception and control aspects, assuming a perfect output from perception, such as an elevation map, and then using it for planning and control [54, 55, 56, 57, 58]. The control part can be further decoupled into searching for feasible footholds on the elevation map and then execute it with a low-level policy Chestnutt [59]. The foothold feasibility scores can either be estimated heuristically [60, 61, 10, 62, 63, 9, 64) or learned [65, 66, 67, 68, 69]. Other methods forgo explicit foothold optimization and learn traversibility maps instead [70, 71, 72, 73]. Recent methods skip foothold planning and directly train a deep RL policy that takes the elevation map as input and outputs either low-level motor primitives [8. 74] or raw joint angles [18, 75, 76, 77J. Elevation maps can be noisy or incorrect and dealing with imperfect maps is a major challenge to building robust locomotion systems. Solutions this include incorporating uncertainty in the elevation map [54, 78, 11] and simulating errors at training time to make the walking policy robust to them [8].
Locomotion from Egocentric Depth Closest to ours is the line of work that doesn't I construct explicit elevation maps and predicts actions directly from depth. [53] learn a policy for obstacle avoidance from depth on flat (errain.[79] train a hierarchical policy which uses depth to traverse curved cliffs and mazes in simulation. [80] use lidar scans to show zero-shot generalization difficult terrains. Yu et al. [81] train a policy to step over gaps by predicting high-level actions using depth from the head and below the torso. Relatedly, Margolis et al. [82] train a high-level policy to jump over gaps from egocentric depth using a whole body impulse controller. In contrast, we directly predict target joint angles from egocentric depth without constructing metric elevation maps.
6 Discussion and Limitations
In this work, we show an end-to-end approach to walking with egocentric depth that can traverse a large variety of terrains including stairs, gaps and stepping stones. However, there can be certain instances where the robot fails because of a visual or terrain mismatch between the simulation and the real world. The only solution to this problem under the current paradigm to engineer the situation back into simulation and retrain. This poses fundamental limitation to this approach and in future, we would like to leverage the data collected in the real world to continue improving both the visual and the motor performance. Currently, our robot is only able to move through the environment but not interact with meaningfully. A future direction could be to combine vision- based policies with an articulated arm [83].
8

---

Acknowledgments
We would like to thank Kenny Shaw and Xuxin Cheng for help with hardware. Shivam Duggal, Kenny Shaw, Xuxin Cheng. Shikhar Bahl, Zipeng Fu, Ellis Brown helped with recording videos. We also thank Alex Li for proofreading. The project was supported in part by the DARPA Machine Commonsense Program and ONR N00014-22-1-20%..
References
[1] J. M. Loomis, A. Da Silva, N. Fujita, and S. S. Fukusima. Visual space perception and visually directed action. Journal of experimental psychology: Human Perception and Performance, 18 (4)9966 1992.
[2] Lee, Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 2020.
[3] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. In RSS, 2021.
[4] J. S. Matthis and B. R. Fajen. Visual control foot placement when walking over complex terrain. Journal of experimental psychology: human perception and performance, 40(1):106, 2014.
[5] A. E. Patla. Understanding the roles of vision in the control of human locomotion. Gait & posture, 5(1):54-69, 1997.
[6] A. A. Mohagheghi, R. Moraes, and A. E. Patla. The effects of distant and on-line visual information on the control of approach phase and step over an obstacle during locomotion. 2004.
Experimental brain research, 155(4):459-44)
[7] K. E. Adolph, W. G. Cole, M. Komati, S. Garciaguiree, D. Badaly, J. M. Lingeman, G. L. Chan, and R. B. Sotsky, How do you learn to walk? thousands of steps and dozens of falls per day. Psychological science, 2012.
[8] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eab 2022.
[9] D. Kim, D. Carballo, J. Di Carlo, B. Katz, G. Bledt, B. Lim, and S. Kim. Vision aided dynamic exploration of unstructured terrain with a small-scale quadruped robot. In ICRA, 2020.
[10] Jenelten, T. Miki, A. E. Vijayan, M. Bjelonic, and M. Hutter. Perceptive locomotion in rough terrain-online foothold optimization. RA-L, 2020.
[11] P. Fankhauser, M. Bloesch, C. Gehring, M. Hutter, and R. Siegwart. Robot-centric elevation mapping with uncertainty estimates. In Mobile Service Robotics, pages 433-440. World Scientific, 2014.
[12] T. Miki, L. Wellhausen, R. Grandia, Jenelten, T. Homberger, and M. Hutter. Elevation mapping for locomotion and navigation using gpu. arXiv preprint urXiv:2204. /2876, 2022.
[13] Y. Pan, X. Xu, X. Ding, S. Huang, Y. Wang, and R. Xiong, Gem: Online globally consistent dense elevation mapping for unstructured terrain. IEEE Transactions on Instrumentation and Measurement, 7-31-13, 2021. doi: 109.1I.../....00000002
[14] H. Qi, A. Kumar, R. Calandra, Y. Ma, and J. Malik. In-hand object rotation via rapid motor adaptation. arXiv preprint wwXii:22200048883 2022.
[15] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv: 0634666377700 2017.
[16] J. Werbos. Backpropagation through time: what does and how to it. Proceedings of the IEEE, 15500111500-156 1990.

---

[17] Fu, A. Kumar, A. Agarwal, H. Qi. J. Malik, and D. Pathak. Coupling vision and proprioception for navigation of legged robots. arXiv preprint arXiv:2112.0000 2021.
[18] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR, 2022.
[19] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011.
[20] H. Miura and I. Shimoyama. Dynamic walk of biped. IJRR, 1984.
[21] M. H. Raibert, Hopping in legged mysdems-modeling and simulation for the two-dimensional one-legged case. IEEE Transactions on Systems, Man, and Cyhernetics, 1984.
[22] H. Geyer, A. Seyfarth, and R. Blickhan. Positive force feedback in bouncing gaits? Proceedings of the Royal Society of London. Series B: Biological Sciences, 2003.
[23] K. Yin, K. Loken, and M. Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics, 2007.
[24] K. Sreemath, H.-W. Park, I. Poulakakis, and W. Grizzle. compliant hybrid zero dynamics controller for stable, efficient and fast bipedal walking mabel. IJRR, 2011.
[25] A. M. Johnson, T. Libby, E. Chang-Siu, M. Tomizuka, R. Full, and D. E. Koditschek. Tail assisted dynamic self righting. In Adaptive Mobile Robotics. World Scientific, 2012.
[26] M. Khoramshahi, H. J. Bidgoly, S. Shafiee, A. Asaei, A. J. Ijspeert, and M. N. Ahmadabadi. Piecewise linear spine for speed-energy efficiency trade-off quadruped robots. Robotics and Autonomous Systems, 2013.
[27] A. D. Ames, K. Galloway, Sreenath, and W. Grizzle. Rapidly exponentially stabilizing control Yyapunov functions and hybrid zero dynamics. IEEE Transactions on Automatic Control, 2014.
[28] D. J. Hyun, J. Lee, S. Park, and S. Kim. Implementation of trot-to-gallop transition and subsequent gallop on the mit cheetah IJRR, 2016.
[29] M. Barragan, N. Flowers, and M. Johnson. MiniRHex small, open-source, fully pro- grammable walking hexapod. In RSS Workshop, 2018
[30] G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim. Mit cheetah 3: Design and control of a robust, dynamic quadruped robot. IROS, 2018.
[31] M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V. Tsounis, J. Hwangbo, K. Bodie, Fankhauser, M. Bloesch, al. Anymal-a highly mobile and dynamic quadrupedal robot. In IROS, 2016.
[32] C. S. Imai, M. Zhang, Y. Zhang, M. Kierebinski, R. Yang, Y. Qin, and X. Wang. Vision-guided guadrupedal locomotion the wild with multi-modal delay randomization. arXiv:2109. 14549, 2021.
[33] T. P. Lillierap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In ICLR, 2016.
[34] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. ICML, 2016.
[35] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation emror in actor-critic methods. In ICML, 2018.
[36] Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In RSS, 2018.
10

---

[37] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation the real world. IROS, 2017.
[38] X. B. Peng, M. Andrychowicz. W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In ICRA, 2018.
[39] Z. Xie, X. Da, M. van de Panne, B. Babich, and A. Garg. Dynamics randomization revisited: A case study for quadrupedal locomotion. In ICRA, 2021.
[40] O. Nachum, M. Ahn, H. Ponte, S. Gu, and V. Kumar. Multi-agent manipulation via locomotion using hicrarchical sim2real. In CoRL. 2020.
[41] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Roboties, 2019.
[42] Hanna and Stone. Grounded action transformation for robot learning in simulation. In AAAI, 2017.
[43] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning a universal policy with online system identification. In RSS, 2017.
[44] W. Yu, C. K. Liu, and G. Turk. Policy transfer with strategy optimization. In ICLR, 2018.
[45] X. B. Peng, E. Coumans, T. Zhang, T.-W. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In RSS, 2020.
[46] W. Zhou, L. Pinto, and A. Gupta. Environment probing interaction policies. ICLR, 2019.
[47] W. Yu, V. C. V. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion, In IROS, 2019.
[48] W. Yu, J. Tan, Y. Bai, E. Coumans, and S. Ha. Learning fast adaptation with meta strategy optimization. RA-L, 2020.
[49] X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and J. Tan. Rapidly adaptable legged robots via evolutionary meta-learning. In IROS, 2020.
[50] I. Clavera, A. Nagabandi, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reiiforcemmnt learning. In ICLR, 2019.
[51] Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emergence gaits in legged robots. CoRL, 2021.
[52] L. Smith, J. C. Kew, X. B. Peng, S. Ha, Tan, and S. Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In ICRA, 2022.
[53] R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. In ICLR, 2022.
[54] Fankhauser, M. Bloesch, and M. Hutter. Probabilistic terrain mapping for mobile robots with uncertain localization. IEEE Robotics and Automation Letters, 3(4):3019-3026 2018.
[55] Y. Pan, X. Xu, Y. Wang, X. Ding, and R. Xiong. Gpu accelerated real-time traversability mapping. 2019 IEEE International Conference on Robotics and Biominetics (ROBIO), pages 734-740, 2019. doi: (00.100/R0B1044542200000 8961119.99.1.......
[56] I.-S. Kweon, M. Hebert, E. Krotkov, and T. Kanade. Terrain mapping for roving planetary explorer. In IEEE International Conference on Robotics and Automation, pages 997-1002. IEEE, 1989.
[57] I.S. Kweon and T. Kanade. High-resolution terrain map from multiple sensor data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14(2):278-22 292, 1992.
[58] A. Kleiner and Dornhege. Real-time localization and elevation mapping within urban search and rescue scenarios. Journal of Field Robotics, 2408-9):723-745,25

---

[59] J. Chestnutt. Navigation planning for legged robots. Carnegie Mellon University, 2007.
[60] M. Wermelinger, P. Fankhauser, R. Diethelm, P. Krüsi, R. Siegwart, and M. Hutter. Navigation planning for legged robots in challenging terrain. In IROS, 2016.
[61] A. Chilian and H. Hirschmüller. Stereo camera based navigation of mobile robots on rough terrain. In IROS, 2009.
[62] C. Mastalli, I. Havoutis, A. W. Winkler, D. G. Caldwell, and C. Semini. On-line and on-board planning and perception for quadrupedal locomotion. In 2015 IEEE International Conference on Technologies for Practical Robot Applications, 2015.
[63] P. Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter. Robust rough-terrain locomotion with a quadrupedal robot. In ICRA, 2018.
[64] A. Agrawal, S. Chen, A. Rai, and K. Sreenath. Vision-aided dynamic quadrupedal locomotion on discrete terrain using motion libraries. arXiv preprint arXiv:2110008891, 2021.
[65] J. Z. Kolter, M. P. Rodgers, and A. Y. Ng. A control architecture for quadruped locomotion over rough terrain. In ICRA, 2008.
[66] M. Kalakrishnan, J. Buchli, P. Pastor, and S. Schaal. Learning locomotion over rough terrain using terrain templates. In IROS, 2009.
[67] L. Wellhausen and M. Hutter. Rough terrain navigation for legged robots using reachability planning and template learning. In IROS, 2021.
[68] C. Mastalli, M. Focchi, I. Havoutis, A. Radulescu, S. Calinon, J. Buchli, D. G. Caldwell, and C. Semini. Trajectory and foothold optimization using low-dimensional models for rough terrain locomotion. In ICRA, 2017.
[69] O. A. V. Magana, V. Barasuol, M. Camurri, I.. Franceschi, M. Focchi, M. Pontil, D. G. Caldwell, and C. Semini. Fast and continuous foothold adaptation for dynamic locomotion through enns. RA-L., 2019.
[70] B. Yang, L. Wellhausen, T. Miki, M. Liu, and M. Hutter. Real-time optimal navigation planning using learned motion costs. In ICRA, 2021.
[71] R. O. Chavez-Garcia, J. Guzzi, L. M. Gambardella, and A. Giusti. Learning ground traversability from simulations. RA-L, 2018.
[72] J. Guzzi, R. O. Chavez-Gareia, M. Nava, L. M. Gambardella, and A. Giusti. Path planning with local motion estimations. RA-L, 2020.
[73] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning. In International Conference on Robotics and Automation (ICRA), 2021.
[74] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robotics and Automation Letters, (2):3699-3706, 2020.
[75] X. B. Peng, G. Berseth, and M. Van de Panne. Terrain-adaptive locomotion skills using deep reinforcement learning. ACM Transactions on Graphics (TOG). 35(4):1-12, 2016.
[76] X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG), 36(4): 1-13, 2017.
[77] Z. Xie, H. Y. Ling, N. H. Kim, and M. van de Panne. Allsteps: Curriculum- learning of stepping stone skills. In Computer Graphics Forum. Wiley Online Library, 2020.
[78] D. Belter, P. tabcki, and P. Skrzypezyński. Estimating terrain elevation maps from sparse and uncertain multi-sensor data. In 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 715-722, 2012. 1022001110//00110022244494999999
12

---

[79] D. Jain, A. Iscen, and K. Caluwaerts. From pixels to legs: Hierarchical learning of quadruped locomotion. arXiv preprint arXi::201..1172. 2020.
[80] A. Escontrela, G. Yu, Xu, A. Iscen, and Tan. Zero-shot terrain generalization for visual locomotion policies. arXiv preprint 1rxxi:2011.055/11 2020.
[81] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang. Visual- visual-locomotion: Learning to walk on complex terrains with vision. In 5th Annual Conference on Robot Learning. 2021.
[82] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. Kim, and P. Agrawal. Learning to jump from pixels. arXiv preprint arXiv:2110.15344 2021.
[83] Z. Fu, X. Cheng, and D. Pathak. Learning unified policy for whole-body control of manipula- tion and locomotion. Conference on Robot Learning (CoRI.), 2022.
[84] Singh and R. C. Yee. An upper bound on the loss from approximate optimal- value functions. Machine Learning, 16(3):227-233, 1994.
[85] D. P. Bertsekas. Dynamic programming: deterministic and stochastic models. Prentice-Hall, Inc., 1987.
13


Reviews:
[{"review_id": "kQz3CE1RJYI", "reviewer": "Reviewer_bvG9", "summary": "This paper presents a learning system for training vision-condition locomotion policies for quadruped robots that is able to traverse highly uneven terrains such as step stones, stairs, slopes, etc. The key idea is to first train a teacher policy in simulation that is conditioned on a heightmap near the robot, which is computationally cheaper, and then distill the policy into a student policy, which is conditioned on depth images, using DAGGER framework. During the training of the teacher policy, the range of the heightfield is designed to be roughly matching the fov of the depth camera, and sensor latencies are also added to facilitate student policy performance. The result method is demonstrated on a variety of indoor and outdoor uneven environments and is compared to several baseline methods. The experiments show the effectiveness of the proposed method in acquiring resilient vision-based locomotion skills for legged robots.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe proposed learning method for acquiring vision-based locomotion policy is solid and effective.\nThe model learned from the proposed method achieved impressive results in real-world environments.\nThe comparison to the baseline methods show the usefulness of incorporating vision input and also learning the teacher policy in a realistic setting.\n\nWeaknesses\nAlthough energy and jerk-related rewards are used, the motion quality is still not at the level of methods that use reference motion. It may be interesting to incorporate ideas in a motion imitation regime to achieve more natural motion.\nThe movement speed seems relatively slow (0.35 m/s) and it’s not clear if this is limited by the algorithm capability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The system proposed in this work enables quadruped robots to demonstrate impressive locomotion capabilities in challenging indoor and outdoor environments. The method is clearly presented and the experiments are solid. Thus I believe the work would be a good contribution to the robotics and ML community.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- Prior methods that leverages teacher-student framework (e.g. [3]) learns an adaptation module for the student to predict a latent representation of the privilege information while in this work the action is used as the supervision signal. I'm wondering if this design choice would impact the algorithm performance?\n- As mentioned above, the policy is trained to walk at 0.35 m/s on uneven terrains, which is well within the capability of the robot hardware. There is likely a trade-off between speed and stability in general and it can provide good insight to know what this trade-off looks like for the proposed method, e.g. at what speed does the method start to see drop in success rate.\n- The teacher policy is trained with realistic heightfield range and latency. Given these two affects policy in different ways, it would be nice to have a separate ablation for them to see the contribution from each component."}}, {"review_id": "eOB4FtwYOmU", "reviewer": "Reviewer_9Sia", "summary": "The paper proposes an end-to-end training for legged locomotion in significantly uneven terrains. The end policy is a recurrent network that uses depth camera images and trained in simulation to overcome different types of terrains that requires vision. It is trained in two phases, first in simulation  using RL then using supervised training using real depth images. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper shows the robot walking over very difficult terrains for a small robot. The results are very impressive. The proposed method is combination of a previously known methods but it is the first time I have seen it applied to the legged locomotion, so I would consider it novel.\n\nThe method is very well explained and it is compared to some baselines such as blind policy or privileged teacher. One concern I have is about the privileged teacher performing very poorly. The result was very suprising and I suspect it is due to the lack of latency in training. I think it would be a more fair comparison if the authors kept the same dynamics but just used privileged teacher for vision. The results might be more competitive to the proposed method.\n\nI think the impact of the paper would be higher if the authors could show the robot walking slightly faster or analyze / explain at what point and why it fails.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is novel within the context of legged locomotion. The results are impressive and there is significant amount of hardware results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "No issues, but some questions I mentioned in 'strengths and weaknesses'", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "b42nzZ-lMAU", "reviewer": "Reviewer_YYJj", "summary": "In this paper, the authors propose a two-stages training pipeline that enables quadruped robot to learn how to traverse uneven terrains using an onboard depth camera as inputs and without any SLAM algorithms. In the first stage of reinforcement learning, the teacher can leverage terrain elevation maps in front of the robot. Once trained, the teacher policy’s behavior is then cloned by a student policy using front mounted depth cameras in a second stage training. The authors train the policies in a diverse set of uneven terrains and deploy in the real world directly; The A1 robot demonstrates robust performance in various urban environments and on stepping stones. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Over all, it is an impressive paper with impressive results, especially on the hardware. It is the first work that I am aware of, which uses egocentric vision + pure learning to solve stepping stones. It shows the possibility of pure reinforcement learning to solve such challenging locomotion problems. \n\n\nThat said, here are my comments on the details of the paper:\n\n1, Line 5, “The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. ” I don’t think we necessarily need to be “biologically plausible” for robots. Anyway, animals don’t use motors and have a flexible spine to support more complex locomotion behaviors. \n\n3, Footnote on page 2, “A1 standing height is 40cm”. Should be 26 cm according to Figure 2.\n\n4, Theorem 2.1 is the main argument on why the proposed method can work better compared with baselines (i.e. information asymmetry between teacher and student policies). However, it does not cover the case where the state spaces between two training policies (from the two stages) are different. So I don’t think it is a concrete argument.\n\n5, It is unclear if the final policy learns the relative position of the environment, or if it learns a certain behavior that can conquer the task.\n\n6, The teacher policy uses an elevation map in front of the robot, but not below. Has the authors tried to have a teacher policy that uses the full elevation map (i.e. even below the robot) to see if there is a behavior difference? \n\n7, Line 91, “ This information bottleneck etc”. Reducing the number of neurons in a layer does not necessarily mean it is an information bottleneck, since these are floating point numbers. To correctly enforce an information bottleneck one needs to add the mutual information, as described in “Learning Agile Robotic Locomotion Skills by Imitating Animals”. \n\n8, Line 105, “the ith rigid body”. Do you mean the feet?\n\n\n9, Line 153. From the authors description it sounds like there is a single policy which handles or terrains. From the video the stepping stone policy seems to use an overhead camera that is looking ~45 degrees downwards, as can be told from the depth camera. However this same camera configuration does not appear in other environments such as stair climbing. This disparity in hardware configuration indicates that different policies are used. Please clarify this. \n\n\n10, For all the baselines, please also report the teacher policy’s performance since this is a two stage training. Also please report the training curve and number of samples needed to obtain a policy in each stage. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend accepting the paper conditioned that the authors revised the paper according to my suggestions. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See my comments in the \"Strengths And Weaknesses\" section. ", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "GztItmyhhHE", "reviewer": "Reviewer_evpZ", "summary": "This paper proposes a new learning based controller for legged robots which uses vision to navigate difficult terrain. Particularly they propose using depth images rather than elevation maps as perceptive input. The show extensive experiments on a real robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Technically this appears to be a very strong paper. As far as I am aware, this is the first end-to-end learning controller capable to navigating stepping stones.\n- The paper is very clear in the explanation of the method.\n\nWeaknesses:\n- I feel there is some discussion lacking in the paper. The authors don't really propose explanations for why their method can plan footsteps other than claim elevation maps have more noise. But I am not convinced on this point since depth images can also be very noisy.\n- I don' believe the experiment properly prove depth images are superior to elevation maps because they are not a proper A to B experiment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this paper shows significant promise and is simply lacking in discussion of the results. As such I recommend accept and have detailed my issues with the paper below. I hope the authors can expand on their paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Line 62 \"that has access to a depth map instead...\" I think should be **depth image** instead\n\n- It's not clear to me the reason for using depth images in phase 2. Converting depth images to elevation maps is fairly easy to compute and I would expect providing an elevation map would help learning. The authors claim elevation maps are noisier, but elevation maps are just another representation of the same noisy depth data in the images so I am not convinced. Perhaps a motivating example would be a good addition. \n\n- Following from my previous point, in my opinion the experiments do not completely justify the use of depth images. This is because the elevation maps used for training in Privileged and Noisy did not include lag which can be a huge failure point for sim-2-real problems. The authors should repeat those experiments but add lag and latency to the training of Noisy. \n\n- There should be a more robust discussion of why depth images enable foot step placement for stepping stones or for climbing high stairs. It seem non-intuitive to me and the authors don't provide any explanation of what the network is learning or how this might be working."}}]

Author rebuttal:

Decision: Accept (oral)