Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes an end-to-end training for legged locomotion in significantly uneven terrains. The end policy is a recurrent network that uses depth camera images and trained in simulation to overcome different types of terrains that requires vision. It is trained in two phases, first in simulation  using RL then using supervised training using real depth images. 
strengths_and_weaknesses: The paper shows the robot walking over very difficult terrains for a small robot. The results are very impressive. The proposed method is combination of a previously known methods but it is the first time I have seen it applied to the legged locomotion, so I would consider it novel.

The method is very well explained and it is compared to some baselines such as blind policy or privileged teacher. One concern I have is about the privileged teacher performing very poorly. The result was very suprising and I suspect it is due to the lack of latency in training. I think it would be a more fair comparison if the authors kept the same dynamics but just used privileged teacher for vision. The results might be more competitive to the proposed method.

I think the impact of the paper would be higher if the authors could show the robot walking slightly faster or analyze / explain at what point and why it fails.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is novel within the context of legged locomotion. The results are impressive and there is significant amount of hardware results.
issues: No issues, but some questions I mentioned in 'strengths and weaknesses'
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a new learning based controller for legged robots which uses vision to navigate difficult terrain. Particularly they propose using depth images rather than elevation maps as perceptive input. The show extensive experiments on a real robot.
strengths_and_weaknesses: Strengths:
- Technically this appears to be a very strong paper. As far as I am aware, this is the first end-to-end learning controller capable to navigating stepping stones.
- The paper is very clear in the explanation of the method.

Weaknesses:
- I feel there is some discussion lacking in the paper. The authors don't really propose explanations for why their method can plan footsteps other than claim elevation maps have more noise. But I am not convinced on this point since depth images can also be very noisy.
- I don' believe the experiment properly prove depth images are superior to elevation maps because they are not a proper A to B experiment.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think this paper shows significant promise and is simply lacking in discussion of the results. As such I recommend accept and have detailed my issues with the paper below. I hope the authors can expand on their paper.
issues: - Line 62 "that has access to a depth map instead..." I think should be **depth image** instead

- It's not clear to me the reason for using depth images in phase 2. Converting depth images to elevation maps is fairly easy to compute and I would expect providing an elevation map would help learning. The authors claim elevation maps are noisier, but elevation maps are just another representation of the same noisy depth data in the images so I am not convinced. Perhaps a motivating example would be a good addition. 

- Following from my previous point, in my opinion the experiments do not completely justify the use of depth images. This is because the elevation maps used for training in Privileged and Noisy did not include lag which can be a huge failure point for sim-2-real problems. The authors should repeat those experiments but add lag and latency to the training of Noisy. 

- There should be a more robust discussion of why depth images enable foot step placement for stepping stones or for climbing high stairs. It seem non-intuitive to me and the authors don't provide any explanation of what the network is learning or how this might be working.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a learning system for training vision-condition locomotion policies for quadruped robots that is able to traverse highly uneven terrains such as step stones, stairs, slopes, etc. The key idea is to first train a teacher policy in simulation that is conditioned on a heightmap near the robot, which is computationally cheaper, and then distill the policy into a student policy, which is conditioned on depth images, using DAGGER framework. During the training of the teacher policy, the range of the heightfield is designed to be roughly matching the fov of the depth camera, and sensor latencies are also added to facilitate student policy performance. The result method is demonstrated on a variety of indoor and outdoor uneven environments and is compared to several baseline methods. The experiments show the effectiveness of the proposed method in acquiring resilient vision-based locomotion skills for legged robots.
strengths_and_weaknesses: Strengths
The proposed learning method for acquiring vision-based locomotion policy is solid and effective.
The model learned from the proposed method achieved impressive results in real-world environments.
The comparison to the baseline methods show the usefulness of incorporating vision input and also learning the teacher policy in a realistic setting.

Weaknesses
Although energy and jerk-related rewards are used, the motion quality is still not at the level of methods that use reference motion. It may be interesting to incorporate ideas in a motion imitation regime to achieve more natural motion.
The movement speed seems relatively slow (0.35 m/s) and it’s not clear if this is limited by the algorithm capability.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The system proposed in this work enables quadruped robots to demonstrate impressive locomotion capabilities in challenging indoor and outdoor environments. The method is clearly presented and the experiments are solid. Thus I believe the work would be a good contribution to the robotics and ML community.
issues: - Prior methods that leverages teacher-student framework (e.g. [3]) learns an adaptation module for the student to predict a latent representation of the privilege information while in this work the action is used as the supervision signal. I'm wondering if this design choice would impact the algorithm performance?
- As mentioned above, the policy is trained to walk at 0.35 m/s on uneven terrains, which is well within the capability of the robot hardware. There is likely a trade-off between speed and stability in general and it can provide good insight to know what this trade-off looks like for the proposed method, e.g. at what speed does the method start to see drop in success rate.
- The teacher policy is trained with realistic heightfield range and latency. Given these two affects policy in different ways, it would be nice to have a separate ablation for them to see the contribution from each component.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In this paper, the authors propose a two-stages training pipeline that enables quadruped robot to learn how to traverse uneven terrains using an onboard depth camera as inputs and without any SLAM algorithms. In the first stage of reinforcement learning, the teacher can leverage terrain elevation maps in front of the robot. Once trained, the teacher policy’s behavior is then cloned by a student policy using front mounted depth cameras in a second stage training. The authors train the policies in a diverse set of uneven terrains and deploy in the real world directly; The A1 robot demonstrates robust performance in various urban environments and on stepping stones. 
strengths_and_weaknesses: Over all, it is an impressive paper with impressive results, especially on the hardware. It is the first work that I am aware of, which uses egocentric vision + pure learning to solve stepping stones. It shows the possibility of pure reinforcement learning to solve such challenging locomotion problems. 


That said, here are my comments on the details of the paper:

1, Line 5, “The elevation mapping, however, is susceptible to failure and large noise artifacts, requires specialized hardware, and is biologically implausible. ” I don’t think we necessarily need to be “biologically plausible” for robots. Anyway, animals don’t use motors and have a flexible spine to support more complex locomotion behaviors. 

3, Footnote on page 2, “A1 standing height is 40cm”. Should be 26 cm according to Figure 2.

4, Theorem 2.1 is the main argument on why the proposed method can work better compared with baselines (i.e. information asymmetry between teacher and student policies). However, it does not cover the case where the state spaces between two training policies (from the two stages) are different. So I don’t think it is a concrete argument.

5, It is unclear if the final policy learns the relative position of the environment, or if it learns a certain behavior that can conquer the task.

6, The teacher policy uses an elevation map in front of the robot, but not below. Has the authors tried to have a teacher policy that uses the full elevation map (i.e. even below the robot) to see if there is a behavior difference? 

7, Line 91, “ This information bottleneck etc”. Reducing the number of neurons in a layer does not necessarily mean it is an information bottleneck, since these are floating point numbers. To correctly enforce an information bottleneck one needs to add the mutual information, as described in “Learning Agile Robotic Locomotion Skills by Imitating Animals”. 

8, Line 105, “the ith rigid body”. Do you mean the feet?


9, Line 153. From the authors description it sounds like there is a single policy which handles or terrains. From the video the stepping stone policy seems to use an overhead camera that is looking ~45 degrees downwards, as can be told from the depth camera. However this same camera configuration does not appear in other environments such as stair climbing. This disparity in hardware configuration indicates that different policies are used. Please clarify this. 


10, For all the baselines, please also report the teacher policy’s performance since this is a two stage training. Also please report the training curve and number of samples needed to obtain a policy in each stage. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I recommend accepting the paper conditioned that the authors revised the paper according to my suggestions. 
issues: See my comments in the "Strengths And Weaknesses" section.
review_title: null

Meta-review: metareview: The paper proposes a learning-based approach for achieving legged locomotion in challenging terrains using onboard sensors.

**Strengths**
- The proposed learning method seems to be novel and effective.
- The paper shows an impressive demo of stepping stones on hardware.
- The paper is well written and easy to follow.

**Weaknesses**
- Reviewers requested a few technical details of the work for better reproducibility. Please respond to individual comments.

**Post-rebuttal updates**
N/A (in a positive way)
confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)