Visual-Locomotion: Learning to Walk on Complex Terrains with Vision
Authors: Wenhao Yu
Deepali Jain
Alejandro Escontrela
Atil Iscen
Peng Xu
Erwin Coumans
Sehoon Ha
Jie Tan
Tingnan Zhang
Venue: CoRL2021 Poster (2021)
Keywords: Legged Robot
Reinforcement Learning
Visual Locomotion

Abstract: Vision is one of the most important perception modalities for legged robots to safely and efficiently navigate uneven terrains, such as stairs and stepping stones. However,  training robots to effectively understand high-dimensional visual input for locomotion is a challenging problem. In this work, we propose a framework to train a vision-based locomotion controller which enables a quadrupedal robot to traverse uneven environments. The key idea is to introduce a hierarchical structure with a high-level vision policy and a low-level motion controller. The high-level vision policy takes as inputs the perceived vision signals as well as robot states and outputs the desired footholds and base movement of the robot. These are then realized by the low level motion controller composed of a position controller for swing legs and a MPC-based torque controller for stance legs. We train the vision policy using Deep Reinforcement Learning and demonstrate our approach on a variety of uneven environments such as randomly placed stepping stones, quincuncial piles, stairs, and moving platforms. We also validate our method on a real robot to walk over a series of gaps and climbing up a platform.

Visual-Locomotion: Learning to Walk on Complex Terrains with Vision
Wenhao Yu', Deepali Jain', Alejandro Escontrela 1.2, Atil Iscen',
Peng Xu', Erwin Coumans' Sehoon Ha 1.3 Jie Tan', and Tingnan Zhang'
Robotics at Google, United States
University of California, Berkeley, United States
3 Georgia Institute of Technology, United States
Email: (magiemelon, jaiddeeall aescontrela, atil, pengxu, erwincoumans, sehoonha, jietan, tingnan google.com
Abstract: Vision is one of the essential perception modalities for legged robots to safely and efficiently navigate uneven terrains, such as stairs and stepping stones. However, training robots to effectively understand high-dimensional visual input for locomotion is challenging problem. In this work, we propose a framework train a vision-based locomotion controller which enables a quadrupedal robot to traverse uneven environments. The key idea is to introduce hierarchical structure with high-level vision policy and a low-level motion controller. The high-level vision policy takes as inputs the perceived vision signals as well as robot states and outputs the desired footholds and base movement of the robot. These are then realized by the low level motion controller composed of position controller for swing legs and a MPC-based torque controller for stance legs. We train the vision policy using Deep Reinforcement Learning and demonstrate our approach variety of uneven environments such as randomly placed stepping stones, quincuncial piles, stairs, and moving platforms. We also validate our method on a real robot to walk over series of gaps and climbing up a platform.
Keywords: Legged Robot, Reinforcement Learning, Visual Locomotion
Introduction
Our approach trains a visual-locomotion policy for the Laikago robot in simulation (left) and transfers to the real world (right). We use two depth sensors on the robot, as marked in red.
Figure 1:
Reproducing natural vision-based locomotion skills seen in animals on artificial creatures such as legged robots has long been at the forefront of robotics research. Progress towards developing robust visual locomotion controllers not only deepens our understanding how humans and animals perceive the environment and control our limbs, but also enables us to build autonomous machines that can reliably traverse real-world environments.
To tackle this problem, most existing methods adopt a three-stage pipeline [1, 2, 3, 4, 5, 6, 7]: per- ception, motion planning, and control. In the perception stage, raw sensor data such as RGBD image and/or LiDAR point clouds are carefully fused with proprioceptive streams such as IMU, motor an- gles, and wheel odometry. For mobile robots, SLAM (simultaneous localization and mapping)
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

often used to produce elevation terrain map centered around the robot [8]. The generated elevation map is then fed the downstream motion planning modules to select paths, motion style, and foot placements (on legged rohots). Finally, the planned robot pose or joint angles are tracked by a low- level motion controller: model predictive control (MPC) [9] or whole body control methods (WBC) [4] are popular choices for unstable, highly dynamical platforms such as legged robots.
While there have been great results from previous works that adopt the three-stage control pipeline, it often leads to a system that is overly complex and requires significant manual effort to develop. For instance, SLAM algorithms require careful parameter tuning to achieve a balance between la- tency and accuracy for mobile robots [10]. On the other hand, recent developments in reinforcement learning (RL) based methods open an alternative path towards creating rision-based locomotion controllers without relying on terrain reconstruction or extensive prior knowledge for foothold selec- algorithms that teach robot arms to retrieve objects from cluttered environments [11] and teach drones to avoid obstacles [12] using vision input. They demonstrate the great potential in an end-to-end learning approach to achieve a low-latency control pipeline and reduce the prior knowledge required to design a working controller.
tion. Researchers have proposed learning-bass
Inspired by this recent progress, we propose a novel control architecture that enables legged robots to successfully solve various visual-locomotion tasks. Specifically, we adopt the philosophy of end-to-end learning and merge the perception and motion planning modules using neural network (NN), Our contribution is learnable hierarchical system that contains two individual layers: a high- level vision policy and low-level motion controller. The high-level vision policy takes two depth images and outputs the desired pose of the robot's base and foothold placements for all swing legs, thereby eliminating the need for a complex SLAM algorithm. The low-level locomotion controller takes the high-level vision policy output, and computes the target motor positions and torques to achieve the desired states. This hierarchical approach allows us to achieve dynamic locomotion in challenging simulated environments including randomly placed stepping stones, staircases, and moving platforms. We also demonstrate zero-shot sim-to-real transfer of visual locomotion policies on the real hardware for walking over stepping stones and climbing up platform.
2 Related Work
Recently, visual-locomotion researchers have developed various promising approaches that tackle the problem using on-board sensors [1, 2, 3, 4, 5, 6, 7, 13, 14, 15, 16). A main idea in many of these works is to perform explicit terrain shape reconstruction (i.e. local SLAM). For example, Fankhauser and Hutter [13] developed "Grid Map" stack to construct local elevation maps around the robot and Kim et al. [14] utilized an Intel RealSense D435 depth sensor and T265 tracking camera to obtain an elevation map of the robot's surroundings. In contrast, we do not employ any explicit mapping mechanism: our learning based visual policy consumes raw depth images from onboard cameras for decision making. Because we eliminate the highly complex mapping process which requires special expertise to tune, our system has a simplified and low latency data pipeline with less chance for manual error accumulation.
Given a representation of the environment provided by perception module, the controller needs to plan sequence of leg movements that can guide the robot through the environment. Unlike our approach where perception and motion planning are subsumed into a single neural network, previous works utilize heuristic-based approaches or local optimization to plan adequate motions (5, 3, 16, 2, 7, 17, 18, 19J. For instance, Jenelten et al. developed a metric for scoring each point on an elevation map based on its roughness, closeness to an edge, as well as slope degrees. They then employed an online batch optimization to choose the best landing positions for the feet. In contrast, our learning system directly learns successful foot targets from reward, which greatly simplifies the system architecture. Similar to our approach, Tsounis et al. also proposed to learn a high-level planner module to plan the gait sequence and low-level gait controller module to control the robot. They demonstrated impressive simulated results of traversing a variety of terrains using a height map input in simulation. In this work, we focus demonstrating a real-robot, which led us to different design choices such as depth images input, an MPC-based low-level controller, and the development of a sim-to-real transfer pipeline.
skills on
Foothold optimization is traditionally performed within a single swing step, however, recent works have proposed multi-step contact sequence optimization [3, 16, 21] to account for upcoming changes

---

in the terrain. For instance, Nguyen et al. proposed 2-step gait optimization to plan the footstep as well as transition between walking gaits. They demonstrated the approach on a biped robot walking over uneven terrains. In our work, we train the perception and planning module together using deep reinforcement learning, which endows the policy with implicit long-horizon planning capabilities. Although neural networks were also employed in prior works [2. 7] to decide optimal footholds, our system differs in that it also decides the desired body pose and speed. By simultaneous adjusting body pose and footholds, our system can learn challenging visual locomotion tasks such random stepping stones, which deemed as difficult to solve with constant body velocities [5],
In addition to methods that leverages the vision input for controlling the robot, recent works have also shown remarkable results for training robots traverse some uneven terrain without relying on visual perception [22, 23]. By carefully designing the reward function and training procedure, these methods produces policies that can robustly handle moderate uneven terrains by adjusting leg movements. Despite the impressive results demonstrated in these works, vision inputs are still beneficial for efficiently and safely traversing general unstructured terrains. For some tasks, such as walking over gaps, visual guidance is still necessary.
3 Methodoloy
3.1 Overview
Low level Control
CoM state, feet positions, feet contact state
CoM state, feet
positions, joint state
State Estimator
desired contact
Guit Scheduler
Vision Policy
angles
desired CoM state, swing feet targets
Swing Control
Stance Control
motor torques
depth images
Figure 2: Overview of the visual-locomotion control architecture.
In this work, we adopt a hierarchical architecture for rision-locomotion control, as illustrated in Figure 2. The controller consists of two components: high-level vision policy and low-level motion controller. The high-level vision policy uses raw depth images from the onboard cameras and robot state decide the desired center of mass (CoM) pose (and speed) and landing positions of the swing feet (Section 3.2). Then the low-level motion controller tracks the desired CoM state by combining a position-based swing leg controller and an model predictive control (MPC) based stance leg controller (Section 3.3). The vision policy runs at 20 Hz, while the motion controller runs at 250 Hz. We train our policy entirely in a physics simulation environment and transfer the trained policy to real robot platform using techniques described in Section 3.4. Unless otherwise specified, all quantities are defined in the yaw aligned inertial frame (Appendix A.1).
3.2 Learning-Bast Based Vision Policy
We formulate the visual-locomotion task Partially-observable Markov Decision Process (POMDP), (0, S, A, R, P. Po. 7), where O the observation space, is the state space, A is the action space, R S x A RR is the reward function, P S x A + S is the dynamics equation, Po the initial state distribution and 7 is the discount factor. Our goal is to find a policy π 0
that maximizes the expected accumulated reward: (TT) = (s0 EnD ('r
Observation and action spaces. We design the observation space in our experiments as 0 (I,,2 qs p. &. Ô ri...:; CI...; 21...4. Aprew), ie the two depth images IT22 from depth sensors shown in Figure (to cover terrains near both front and rear legs), subset of the CoM pose

---

q. - (p.. ②) (CoM height, roll, and pitch), estimated CoM velocity p (Appendix C), gyro- scope readings Φ ④ the robot's feet positions ri..., the feet contact states c1..,4 (one if in contact and zero otherwise), the phase & of each leg in its respective gait cycle (Appendix A.2), and the pre- vious action. The action space is (q 6 vff...4.. i.e. the desired CoM pose, velocity, and ith foot's s target landing position (Tzis readings: we compute the point cloud from the depth image take the average height of four nearest points. This enables us to achieve better exploration by elim- inating invalid foothold targets (e.g. inside obstacle or high in the air). We find this to be critical for the policy to traverse highly uneven terrains such as stairs.
The Z component of the landing location is inferred from the depth
the predicted (I, y) coordinate and
Reward function.
We design reward function:
R(s. a) = clip(pa: -p, - P -P=aa) w/ (lpyl + 1by1) wyy |P
where Pa. is the CoM velocity in the forward direction, Py the CoM displacement in the lateral direction, and V the base yaw angle. The first term rewards the robot to move forward with maximum speed controlled by paree, the second term penalizes the robot from moving sideways, and the last term encourages the robot walk straightly; W1, W2 modulates the importance of different terms. In our experiments, we chose pr = 0.375 m/s, W1 = 1.25, and wy = 0.125.
Early termination. A training episode is terminated if: the robot loses balance (CoM height Pz below 0.15 m, pitch [0] > 1 rad, or roll {b) > 0.3 rad in our experiments), 2) the robot steps within 0.02 m of the boundary of the stepping stones stairs, 3) the robot reaches invalid joint configuration, c.g. knee bending backwards.
3.3 Motion Controller
The motion controller computes appropriate motor position and torque commands to achieve the desired landing positions the swing legs and the target base poses from high level policy. Our motion controller separately controls swing and stance legs. The swing leg controller computes the feet positions by interpolating a time based curve a(t) between the swing start and target landing po- sitions. Instantaneous feet positions are converted to desired joint angles through inverse kinematics (IK) and tracked using proportional-derivatieee (PD) control. The stance leg controller, on the other hand, achieves the desired position and velocity of the robot base by computing sequences of con- tact forces between the feet and the ground. By approximating the robot dynamics using Centroidal Dynamies Model (CDM), we formulate this problem as a convex model predictive control (MPC) problem similar to [24]. The optimized contact forces are mapped to stance leg joint torques using Jacobian Transpose. More details regarding the motion controller can be found in Appendix A.
3.4 Sim-to-real Transfer
300
200
309
(A)
(B)
(D)
Figure 3: post-processing technique to reduce the sim-to-real gap perception. (A) An original
depth map obtained from the D435 camera with an invalid band on the left of the image and noises near edges. (B) A simulated depth map. (C) The filtered and down-sampled D435 depth image. (D)
The filtered and down- down-sampled simulation depth image.
Sim-to-real transfer is persisting problem in robotics, and is especially hard for visual locomotion problems because of distribution shift in image space and discrepancies robot dynamics.
Images from simulated and real-world sensors can be significantly different (Figure 3, A and B). To bridge the gap, we adopt a post-processigg procedure that maps both simulated and real depth images to similar domain. As shown in Figure 3, C and D, we first add random Gaussian noise

---

and randomly paint pixels black in the simulated depth image. To mimic the noise around the object edges stereo depth camera, pixels along an edges in the depth image identified by Canny edge detector have a higher probability of being dropped. We then apply an in-painting operation for both simulated and real images to fill the missing pixels[25], followed by down-sampling operation. This transforms the depth images in simulation and real world to similar distribution.
Discrepancies dynamics such as mismatch friction make precise foot-placements difficult. For example, we found the average difference of feet landing positions in the world frame can be 2 ~ em between sim and real when the same sequence of actions are executed. This enough to trigger a failure in the step-stone task. To compensate this issue, we adopt the domain randomization tech- nique [26] to train robust policy with randomized simulation parameters. More details regarding the randomization process can be found in Appendix D. Adding randomization not only allows the policy to be more robust, also enables the vision policy to observe more diverse states.
4 Experiment and Results
4.1 Experiment Setup
D
Figure 4: Examples of environments used for visual locomotion tasks: randomly placed stepping stones, quincuncial piles, staircases, and uneven terrains.
We evaluate our method on the Unitree Laikago [27] quadruped robot, which weighs 24kg. The robot is equipped with 12 actuated joints. To collect visual data, we install two depth cameras on robot: one Intel D435 the front for wider field of view and one Intel L515 on the belly with better depth quality in close range (Figure 1). At inference time. process all depth images as described in Section 3.4, which results in two 32 x 24 depth images as inputs to the policy.
We train our hicrarchical policies using the PyBullet physics simulator [28] with a distributed imple- mentation of augmented random search (ARS) [29]. We use N 256 perturbations per ARS itera- tion and run the algorithm until convergence with maximum of 2000 training iterations, amounting 1, 024, 000 simulation episodes per trial. We use a Multi-layer Perceptron (MLP)-basee policy architecture with 54, 656 parameters. For all experiments, we run ARS with grid search for four hyper-parameters, resulting 12 trials. We report the performance of top-3 policies by testing them on 150 randomized environments for each task. More training details can be found in Appendix E.
For experiments in both simulation and real world, we train policies with perception noise. We then fine-tune them with the dynamics randomization scheme as described in Section 3.4 before deploying the real robot. We empirically find this helpful for faster training convergence. The fine-tuned policy is deployed on the robot without the need of additional hardware data."
4.2 Simulation Results
We first evaluate the capability of our learning system variety of challenging visual locomotion tasks, including walking over randomly placed stepping stones, quincuncial piles, uneven terrains, and moving platforms. A subset of the simulation environments are shown in Figure 4. We train separate policies for each environment and show the statistical results in Figure 5. To measure
pf the performance of each policy, we design a metric, performance ratio, pairr, where pz is the distance travelled by the policy in the environment, and pm is the maximum distance the policy could reach, e.g. where the terrain ends.
Videos of our trained policies in simulation and world can be found at: https://youtu.be/// Bew.tttuu.e/e/HHHeeeeee-- M--NH-EEynnC
5

---

Perfrmance Ratto
0.8
0.4
0.2
Random Stepstone
Random Stepstone
Oulncuncial Files
Ouincuncial Files
Sair
Useven
Moving Platfomm
Figure 5: Performance ratio our policy for different simulated environments.
Randomly placed stepping stones. The first task is walk over a series of randomly placed stepping stones, Figure 4A. The widths, lengths and gap sizes of stepping stones are sampled from [0.55, 0.7], [0.5, 0.8], and [0.07, 0.2] meters respectively. The agent fails the task the robot steps outside of the stone or into the gaps. Since the stones and their positions are randomly sampled, this task cannot be accomplished without vision. To successfully achieve this task, the robot needs to identify the position and size of the stones and also plan for future footholds. We demonstrate that our method can obtain policies that solves this task using both trotting and pacing gaits.
Quincuncial piles. The random stepping stone task evaluates the robot's s ability to identify and handle terrain changes in the forward direction. A natural extension is to also include the lateral direction. In the second task we create a grid of 2 dimensional stepping stones (Figure 4B). Each stone has an area 0.15 x 0.15 m² with standard deviation 0.015 m in height, and separated by 10,13, 0.17] from each other in both and y directions. At the beginning of each episode, we also randomly rotate entire stone grid in [-0.1, 1, 0.1] rad. Despite being significantly more difficult, using framework
obtain policies that can traverse the field using trotting walking gaits.
Uneven terrains. far the tasks focus on evaluating the robot's ability to avoid undesired re- gions relatively even terrains. To test the policy's 's performance in handling different heights, we designed a staircase climbing task (Figure 4C). The depth of each stair uniformly sampled in [0.25, 0.33] and the height in [0.16, 0.19] to mimic the dimensions of real world stairs. This task quite challenging for Laikago robot because each stair as tall as the robot's 's knee joint. Fig- 4D shows another environment uneven terrains. In this environment, the height offsets of neighboring stones are uniformly sampled in [-0.13, 0.2] m, and gap of [0.05, 0.1] i is added between the stones.
designed for evaluating the ability of our approach to handle
Moving platforms. One benefit of using vision input to the policy is that it can potentially handle moving objects better than methods that relies explicit terrain reconstruction. To demonstrate the capability of our learning system, we take the random stepping stone environment (Figure 4A) and allow each piece to move. Each platform follows periodic movement whose magnitude and frequency are randomly sampled in [0.10, 0.15] m and [0.4, 1.0) Hz, respectively. Also, we randomly pick half of the platforms to move horizontally and the rest vertically. This task requires the robot to infer both the position and velocity of the platforms. To facilitate learning, we stack history of three recent images as input the policy for this task. As shown in the accompanying video, our policy learns to identify the moving objects in the scene and will slow-down and wait for the platforms to reach an ideal location before striding.
4.3 Comparison to baseline methods
We compare our proposed method to two baselines: end-to-end training and heuristics-based foot placement. For the sake of simplicity yet without loss of generality, all comparisons are done in the uneven terrain environment (Figure Top), which captures the difficulty of both the stepping stones and staircases. The results can be found in in Figure
4.3.1 End-to-end training
In the first baseline, we train end-to-end neural network policy that takes the images and the robot states as input (same as our hierarchical policy) and outputs desired motor angles. We choose the architecture of Policy Modulating Trajectory Generator (PMTG) [30], which has demonstrated

---

Figure 6: Laikago robot walking over uneven terrain and randomized pillar terrains. The black curve refers to the CoM trajectory, and colored curves represent the feet trajectories.
0.8
Performance
0.6
0.4
Qurs
Enditto-ent end
Heuristic
No Vision
No Depth based 2.component
tootplacement
Figure 7: Comparison of performance between our method and the baseline methods.
high-quality and transferrable locomotion policies [22]. Since PMTG generates the trajectory in the joint space, it was not able to achieve precise foot placement that needed for many our testing environments, shown in Figure 7.
4.3.2 Heuristics-based foot placement
In the second baseline, we compare our method to prior approaches that adjust the foot placement by computing a score map of surrounding terrains [2, 4, 18). In our baseline implementation, we define point on the nearby terrain to be valid if 1) it is within reach of the swing leg, 2) the z-component of the surface normal is larger than 0.9 (i.e. pointing upward), and 3) the standard deviation of the nearby heights (i.e. roughness) is less than 0.05m. For better comparison, we use the same motion controller as in our approach and train a high-level policy to output the desired base pose and velocity using the same observation space and learning procedure. As shown in Figure 7, using heuristics alone determine footplacement got lower score than our proposed approach. One important reason is that this heuristics does not take the base movement of the robot into consideration. As result, it may propose landing positions that are incompatible with the robot's 's CoM speed.
4.4 Ablation Study
A key hypothesis make in this work is that, despite that the images have low resolution, they contain critical information that enables our policy to traverse a large variety of environments. To validate this hypothesis, the observation space and retraining a vision-less policy. As seen in Figure 7, without vision input, the policy is not able to accomplish the task, indicating the importance of having vision as input.
first perform ablation by removing the vision component from
Another key component our method to use the depth input for inferring the z-component of the foothold location. This creates more meaningful action space for the policy to explore during learning. As shown Figure 7, the performance drops significantly we do not use depth infer the z-component of the foot placement.

---

4.5
Validation on Real Robot
Stepping
Climbing
Figure Laikago robot solving two challenging visual-locomotion tasks.
We deploy the trained policy to a Laikago robot for two tasks, walking over gaps and climbing onto stepping stones (see supplementary video). Our real-world setup (Figure 8 Top) consists of four stepping stones that are separated by three gaps with widths between [0.12, 0.18] m.
In the first task, the performance is measured by the number of gaps that are crossed successfully, summed over legs. For instance, all four legs cross one gap and the robot completely on the next stepping stone, the score would be 4. Therefore, the upper bound of the score is 12: three gaps times four legs. An experiment episode is terminated if the robot falls, any leg steps into the gaps or the robot steps outside the stones. As a baseline, the score is 0.9 + 1.4 for can rarely clear single gap. Over eight real-world episodes, our policy is able to consistently achieve score of 10.1 2.2. There are two episodes in which the robot completes the full course with a score of 12; For seven out of eight episodes, the robot reaches the last stepping stone. The real-world failure cases are mainly caused by the robot stepping too close to the edge of the terrain and slipping into the gap. This is likely due to the discrepancies in the dynamics and vision model. Despite applying domain randomization techniques precisely controlling the landing position the feet for a moving robot still challenging.
locomloocoooionn policy:
mitigate the model discrepancy. we found
In the second task, the robot needs to climb onto a stepping stone, which is 0.18 m above from the ground (Figure 8 Bottom). Similarly, we also define the performance score as the number of legs reached the top of the stone (4 is the max). The learned policy reaches a score of 3.7 + 0.7 over seven experiment episodes. The robot successfully steps onto the stone for six out the seven times. In the only failure case, the robot loses balance because its rear foot hits the edge of the stone.
5 Conclusion and Future Work
In this work, we present hierarchical learning system to tackle challenging visual-locomotion tasks. By using high-level learned vision policy that consumes raw camera images, we eliminate the need to explicitly construct 3D terrain maps, and thus reduce the control latency and architecture complexities, and more importantly, can handle dynamically moving terrains. The low-level, model predictive control based motion controller greatly reduces the motion tracking error on the hardware and thus narrows the sim-to-real gap. We demonstrate that policies trained using our system can reliably walk over challenging terrains such as stepping stones that require fine visual-motor control.
One limitation of this work is that the gaits are manually specified and fixed for each task. In contrast, animals can dynamically modulate their walking pattern to adapt to changes in the terrain. In the future, we plan to extend our framework to enable the visual policy to adjust the gait pattern online. Recent works have shown great potential in representation learning for vision-based robotic control problems [31, 32]. For example, Hoeller et al. trained VAE to encode the environment for a legged robot navigation task. Incorporating these techniques can further improve the learning performance of our framework. Additionally, animals can traverse complex terrains while paying attention to a small area, and can reason about their hind limbs that they cannot see. This makes adding memory [33] and attention [34] to the policy architecture another promising future direction.

---

Acknowledgement
We would like to thank Gus Kouretas, Thinh Nguyen, Noah Brown, Satoshi Kataoka, and the Oper- ations team at Robotics at Google for the help in setting up the testing environment, debugging robot hardware and camera issues. We would also like thank Ken Caluwaerts, Krzysztof Choromanski, Kuang-Huee Lee, Daniel Ho, Yuxiang Yang, and the anonymous reviewers for valuable discussion and suggestions.

---

References
[1] C. Mastalli, M. Focchi, I. Havoutis, A. Radulescu, S. Calinon, J. Buchli, D. G. Caldwell, and C. Semini. Trajectory and foothold optimization using low-dimensional models for rough ter- rain locomotion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1096- 1103. IEEE, 2017.
[2] O. A. V. Magana, V. Barasuol, M. Camurri, L. Franceschi, M. Focchi, M. Pontil, D. G. Cald- well, and C. Semini. Fast and continuous foothold adaptation for dynamic locomotion through CNNs. IEEE Robotics and Automation Letters, 4(2):2140-22 0044000-27,,. 2019.
[3] O. Villarreal, V. Barasuol, P. M. Wensing, D. G. Caldwell, and C. Semini. MPC-based con- troller with terrain insight for dynamic legged locomotion. In 2020 IEEE International Con- ference on Robotics and Automation (ICRA), pages 1436-2442 IEEE, 2020.
[4] Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter. Robust rough-terrain locomotion with quadrupedal robot. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 5761-5768. IEEE, 2018.
[5] Jenelten, T. Miki, A. Vijayan, M. Bjelonic, and M. Hutter. Perceptive locomotion in rough terrain online foothold optimization. IEEE Robotics and Automation Letters, 5(4)::337-5377, 2020.
[6] R. Grandia, A. J. Taylor, A. D. Ames, and M. Hutter. Multi-layered safety for legged robots via control barrier functions and model predictive control. arXiv preprint arXiv:2011.00032 2020.
[7] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. RLOC: Terrain- aware legged locomotion using reinforcement learning and optimal control. arXiv preprint arXiv:2012.03094, 2020.
[8] Fankhauser, M. Bloesch, and M. Hutter. Probabilistic terrain mapping for mobile robots with uncertain localization. IEEE Robotics and Automation Letters (RA-L), 3(49:301,-3 2018. 10.: (09/LRA.2018.28499)
[9] Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and Kim. Dynamic locomotion in the mit cheetah 3 through convex model-predictive control. In 2018 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 1-9, 2018. Noi:10.1109/IROS 2018. 8594448.
[10] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the peouut-perception age. IEEE Transactions on robotics, 1309-133311332 2016.
[11] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: /806. 10293, 2018.
[12] L. He, N. Aouf, F. Whidborne, and B. Song. Integrated moment-based Igmd and deep reinforcement learning for uav obstacle avoidance. In 2020 IEEE International Conference on Roboties and Automation (ICRA), pages 7491 7497. IEEE, 2020.
[13] P. Fankhauser and M. Hutter. A Universal Grid Map Library: Implementation and Use Case for Rough Terrain Navigation. In A. Koubaa, editor, Robot Operating System (ROS) The Complete Reference (Volume 1), chapter 5. Springer, 2016. ISBN 978-3-319-2600 26052-5. doi: 10. 10077977-3-319-22222 3-319-26054-9 URL http:/ /www.wwwooooo/ springer. com/ de htl/de//bboo book/ 978331 0113392205555
[14] D. Kim, D. Carballo, J. Di Carlo, B. Katz, G. Bledt, B. Lim, and S. Kim. Vision aided dy- namic exploration of unstructured terrain with small-scale quadruped robot. 2020 IEÉE International Conference on Robotics and Automation (ICRA), pages 2464-2470. IEEE, 2020.
10

---

[15] C. Mastalli, I. Havoutis, A. W. Winkler, D. G. Caldwell, and C. Semini. On-line and on-board planning and perception for quadrupedal locomotion. 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA), pages 1-7. IEFF, 2015.
[16] H.-W. Park, P. M. Wensing, S. Kim, et al. Online planning for autonomous running jumps over obstacles in high-speed quadrupeds. 2015.
[17] X. Da, Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Zhu, B. Babich, and A. Garg- Learning a contact-adaptive controller for robust, efficient legged locomotion. arXiv preprint arXiv:2009. 10019, 2020.
[18] Z. Xie, X. Da, B. Babich, Garg, and M. van Panne. Glide: Generalizable quadrupedal locomotion in diverse environments with a centroidal model. arXiv preprint 2rii: 2021.
[19] M. Xie, A. Escontrela, and Dellaert. A factor-graph approach for optimization problems with dynamics constraints. arXiv preprint arXiv:201 1.06/94, 2020.
[20] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning. IEEE Robolics and Automation Letters, 5(2):3699-3766, 2020.
[21] Q. Nguyen, A. Agrawal, X. Da, W. C. Martin, H. Geyer, J. W. Grizzle, and K. Sreenath. Dynamic walking on randomly-varying discrete terrain with one-step preview. Robotics: Science and Systems, volume 2, 2017.
[22] J. Lee, J. Hwangbo, I.. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science roboties, 5(47), 2020.
[23] Siekmann, K. Green, Warila, A. Fern, and Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. arXiv preprint arXiv:2/05.08826 2021.
[24] G. Bledt, M. J. Powell, B. Katz, J. Di Carlo, P. M. Wensing, and S. Kim. MIT Cheetah 3: Design and control of a robust, dynamic quadruped robot. 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2245-2252 IEEE, 2018.
[25] M. Bertalmio, A. L. Bertozzi, and G. Sapiro. Navier-stokes, fluid dynamics, and image and video impainting. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume 1, pages 1-1. IEEE, 2001.
[26] J. Tan, T. Zhang, F. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10333 2018.
[27] Unitree Robotics URL http: / www.
cc/.
[28] E. Coumans and Y. Bai. Pybullet, a python module for physics simulation in robotics, games and machine learning, 2017.
[29] H. Mania, A. Guy, and B. Recht. Simple random search provides competitive approach to reinforcement learning. arXiv preprint arXiv: 10X.:v:8833.70555 2018.
[30] A. Iscen, K. Caluwaerts, J. Tan, T. Zhang, E. Coumans, V. Sindhwani, and V. Vanhoucke. Policies modulating trajectory generators. In Conference on Robot Learning, pages 916-926. PMLR, 2018
[31] D. Hoeller, L. Wellhausen, F. Farshidian, and M. Hutter. Learning a state representation and navigation in cluttered and dynamic environments. IEEE Robotics and Automation Letters, (3):0081-5088, 2021.
[32] K.-H. Lee, I. Fischer, A. Liu, Y. Guo, H. Lee, J. Canny, and S. Guadarrama. Predictive infor- mation accelerates learning in rl. arXiv preprint arXiv:2007. 200.20007.240,,, 2020.
[33] Hochreiter and Schmidhuber. Long short-term memory. Neural computation, 973) 1780, 1997.
11

---

[34] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlos, P. Hawkins, J. Davis, A. Mohiuddin, Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[35] M. H. Raibert. Legged robots that balance. MIT press, 1986.
12


Reviews:
[{"review_id": "wj7Y5LAlyrl", "reviewer": "Reviewer_2GnU", "summary": "This paper proposes a hierarchical vision-based controller for quadruped locomotion on a variety of random terrains. The high-level component is trained using DRL and specifies the walking pattern for the legs and the robot's base. The low-level component separately controls the swing legs and the stance legs to realize the movement asked by the high-level controller. The system is successfully tested on a robot using zero-shot sim2real transfer.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The system design is straightforward.\n- The observation and action spaces are defined in a compact way.\n- The approach is tested in a wide range of randomized environments with good results. Comparision to the baselines and the ablation study look convincing as well.\n- The paper is well-written. I couldn't find any typos.\n\nWeaknesses:\n- The training details are only explained in the appendix. Even there the paper does not report any learning curves or the number of simulation budget for the training.\n- It seems that the approach only supports the gait that is encoded in the gait scheduler.\n- It would have been nice to see how the model works using other SOTA RL algorithms such as SAC & TD3.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I liked the paper since it is well-written, the approach is clear, and the results are looking good. I am a bit suspecious that the learning contribution is minor, but that should be OK cosidering the challenges of deploying such a system on an actual robot. I am happy with the paper as it is, but below you can find my suggestions for improving it.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Explain the training process details in the main paper body.\n- The policy archicture is only briefly explained in the appendix without any accompanying text. For example, I'd like to know if the vision inputs are processed using CNN networks. \n- If possible, test the approach with other SOTA RL approaches such as SAC & TD3."}}, {"review_id": "rMTntF3MWOW", "reviewer": "Reviewer_MZ8r", "summary": "The paper provides a hierarchical control that combines a high-level vision-based planner and a low-level MPC controller. The planner is an RL policy that takes the processed raw image data to decide the target CoM pose trajectory and the target feet locations. The main contributions include:\n1. Use raw image data as the policy input instead of a processed height map.\n2. Learn not only the foot location but also the target CoM trajectory.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is written clearly with enough discussion, explanation. I do appreciate the detailed appendix that provides useful information to elaborate the content in the main paper. I think the control method described in the paper is reasonable and the simulation results are good. I support the argument in the paper that learn both CoM pose and foot location is better than learn foot location only. \n\nFor the weak points, the paper may miss one case in the discussion. If we two actions: CoM pose and foot location and two controllers: learned and heuristic, we can come up with four combinations: 1. learned both CoM pose and foot location; 2. learned CoM pose, heuristic foot location; 3. heuristic CoM pose, learned foot location; 4. heuristic both CoM pose and foot location. The paper's method is 1. and the baselines are 4 and most of the literature is 3. However, citation [18] is case 2 where learn only and CoM pose and can pass all the tasks in the current papers. I would suggest discussing [18] in 4.3.2 Heuristic-based foot placement. It would support the last sentence that landing positions should compatible with the robot's CoM speed. \n\nOther weakness points are:\n1. Prefer more hardware experiments, like stair climbing. The current experiments are easy. The simple QP stand control + Raibert's heuristic foot placement can complete the tasks. The gap avoidance just needs to find the nearest feasible ground to adjust the target landing positions. I have tested this on A1. \n2. Prefer elaborating more about the action space. The MPC requires a trajectory of CoM and foot locations, not just an instant one. The action space in line 115 looks like a single state of CoM and foot locations. So just to be clear, does the policy output q_d(t=1) or {q_d(t=1), q_d(t=2), ...q_d(t=H))? So does the foot location. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper provides a clean hierarchical control strategy to include raw images in the legged locomotion control. Learning both CoM and foot location is useful. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "1. Please discuss citation [18]. Even if it uses heuristic foot placement, but performance-wise, it looks no different than the current method. \n2. Please elaborate more about the action space. The way it is present in line 115 looks like a vector with dimension 20, but MPC requires a sequence of target states as input.\n3. Sec 2 second paragraph, should you cite DeepGait?\n4. Line 112, there is no Appendix A.5. Should be A.4\n5. Appendix line 444, why it is SQP, not a QP? My understanding is SQP is a nonconvex problem, but the formulation looks the same as MIT chhetah3 convex-MPC formulation. "}}, {"review_id": "jtiXGyK_P9T", "reviewer": "Reviewer_tqbD", "summary": "Summary: This paper proposes a hierarchical approach to enable visual locomotion from onboard cameras. The high-level planner predicts the desired base pose, velocity and the target for the swing leg at a low frequency which is executed by an MPC running at a high frequency and uses a simplified robot dynamics model. The results are demonstrated in a wide variety of simulated terrains and also on real hardware for the stepping stone tasks.\n\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Comments:\n1. The idea of decoupling foot-step planner and leg controller is quite natural and elegant, and seems quite promising from the results presented in the paper.\n2. The heuristic controller used in Section 4.3.2 is not very clear and the failure mode analysis of not very convincing. Instead, the authors should use the heuristic method of foot placement as proposed in Section IV-B of [14], which takes the com speed into account before deciding the foot placement.\n3. One limitation of the proposed method is that there is a separate policy trained for each of these seemingly similar tasks. It would be good to know the benefits of the proposed method compared to [14], which uses the same heuristic across their experiments and only uses onboard cameras as well.\n4. The paper should at least discuss (and ideally compare) to [A] below which uses the same decoupling of a high-level planner and a low-level controller.\n5. It is quite interesting to see a simple visual processing pipeline that is quite effective in bridging the sim to real gap. It would be interesting to understand the limitations of this.\n6. The paper should talk about the major failure cases and the limitations of their overall method since the reported accuracies even in the simulation are not 100%.\n7. The paper is very well written and easy to follow and understand. The figures are great.\n\n[A] Tsounis, Vassilios, et al. \"Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning.\" IEEE Robotics and Automation Letters 5.2 (2020): 3699-3706.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper executes the idea high-level learned visual planner coupled with a low-level MPC-based controller quite well and my current inclination is to accept the paper if the heuristic baseline controller is modified as suggested above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "As mentioned earlier, I would like to see a discussion on the following points: \n1. One limitation of the proposed method is that there is a separate policy trained for each of these seemingly similar tasks. It would be good to know the benefits of the proposed method compared to [14], which uses the same heuristic across their experiments and only uses onboard cameras as well. \n2. The paper should at least discuss (and ideally compare) to [A] below which uses the same decoupling of a high-level planner and a low-level controller. \n3. It is quite interesting to see a simple visual processing pipeline that is quite effective in bridging the sim to real gap. It would be interesting to understand the limitations of this. \n4. The paper should talk about the major failure cases and the limitations of their overall method since the reported accuracies even in the simulation are not 100%.\n\n[A] Tsounis, Vassilios, et al. \"Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning.\" IEEE Robotics and Automation Letters 5.2 (2020): 3699-3706.\n"}}, {"review_id": "JHPCV1kXU7u", "reviewer": "Reviewer_jPiH", "summary": "This paper presents a reinforcement learning based **visual policy** for quadrupedal robots to walk on **structured rough terrain** with discrete footholds such as stepping stones and stairs with varying step lengths and step heights. The learning based visual policy takes in two depth images (one each attached near the front and rear of the robot) and provides high-level commands such as desired center-of-mass positions and velocities and swing foot positions to a low-level MPC controller. The policy is validated in a wide range of scenarios, both, in simulation and hardware on the Unitree Laikago quadruped and compared against a heuristic policy and an end-to-end trained policy. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The primary strength of this paper is that it does a good job of experimentally validating their policies on hardware, and also comparing against relevant baselines. \n\nAs also suggested in the paper, the main idea behind using a learning based approach is the avoidance of building an explicit elevation map and planning footsteps/base motions using that map. Instead, the visual policy outputs the plan for the footholds and base motions directly from the depth camera images and robot states. However, this is not very new and several works have previously looked at combining high-level visual inputs to low-level model based controllers like MPC. \n\nI see the following limitations:\n\n1. Related Works: There are key references missing from the related works and which might help the authors to further improve on their research as well. See end of this section for the list of additional references and below for a summary:\n\n  (a) Learning based approaches:\n\n  [R1] DeepGait - This paper introduces a learning based approach for legged robots to navigate on discrete terrain using the local height map data as the visual input and outputs desired base positions, velocities, foot positions and gait timing variable, which are then fed to a low-level controller. I believe this is very relevant and similar to the proposed method (except for the visual input representation). \n\n  [R2] This paper presents a learning based approach for visual navigation in cluttered environment that uses a VAE to build a predictive model of the depth images. This is then used to train a model-based RL policy that outputs high-level commands for a low-level model-based controller.\n\n[R3] is also very relevant to this current manuscript, where a highlevel CNN policy is used along with a low-level MPC controller.\n\n  (b) Control theoretic approaches [R4]-[R5]: There's quite a few works on using control theoretic (and with non-visual inputs) for legged robots to walk on discrete terrain like stepping stones. I believe the authors can use some ideas from these works to improve on the foot tracking performance without significant domain randomization of the dynamics parameters.\n\n2. Dynamic capabilities of the locomotion controller:\n\nThe learned policies only consider motions along a straight line. Are there any limitations on the turning capabilities of the policy (especially on the terrain with the stepping stones)? The authors should consider stating the limitations of the locomotion controller. \n\n3. MPC as a limitation to learn more complex behaviors: As stated in the manuscript, the current policy only considers gaits of a particular contact schedule (trotting/static walk/etc). In this regard, it seems like the MPC places additional constraints and limitations on the achievable robot behaviors. It would be interesting to see if this limitation can be lifted if desired joint trajectories can be learned from visual inputs. The authors do consider an end-to-end approach, however, it is unclear why this has poor performance.\n\n4. Details on the policy implementation:\nIt would be good to have some more info on whether the current policy can be run on-board or requires compute on an off-board workstation. \n\n5. CoM x-y position estimation: \nIt would be good to have some details on how the x-y positions are estimated on the robot since this is one of the inputs to the vision policy. \n\n6. Requirement of 2 cameras: Although this is a good engineering solution, I believe that this simplifies the learning problem quite a bit. \n\n\n[R1] Tsounis, Vassilios, et al. \"Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning.\" IEEE Robotics and Automation Letters 5.2 (2020): 3699-3706.\n\n[R2] Hoeller, David, et al. \"Learning a state representation and navigation in cluttered and dynamic environments.\" IEEE Robotics and Automation Letters 6.3 (2021): 5081-5088.\n\n[R3] Villarreal, Octavio, et al. \"MPC-based controller with terrain insight for dynamic legged locomotion.\" 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2020.\n\n[R4] Grandia, Ruben, et al. \"Multi-layered safety for legged robots via control barrier functions and model predictive control.\" arXiv preprint arXiv:2011.00032 (2020).\n\n[R5] Nguyen, Quan, et al. \"Dynamic Walking on Randomly-Varying Discrete Terrain with One-step Preview.\" Robotics: Science and Systems. Vol. 2. No. 3. 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the paper presents some good experimental results, there are several works that look into visual locomotion for legged robots in challenging terrain with discrete footholds. The use of two cameras simplifies the learning problem and avoids having to create an explicit map or use RNNs/LSTMs. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "It would be good to address the comments under the listed 'Weakness' of the paper.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author propose a deep vision based policy to enable a quadrupedal robot to walk on various terrains.  In particular, a RL policy is proposed that takes visual information as input and outputs commands to a low-level MPC controller. Simulation results as well as experimental validation is presented.  The reviewers agree that the paper is well written and presentation is good.  A comparision with other methods would be a good addition to further strengthen the contributions of the paper and establish why the proposed hierarchical policy is better.  Reviewers also point out several related papers that should be part of the literature review and potentially used for comparison.  Please also discuss failure cases.

I am recommending an accept as a poster.
Decision: Accept (Poster)