Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an approach to train control policies to execute agile gap jumping. A combination of reinforcement learning and model-based whole body control is used to generate robust policies. Policies is first trained with height map then distilled into a policy that takes raw camera images as input. Results demonstrate agile jumping motions across gaps, both in simulation and on the real robot.
strengths_and_weaknesses: Pro:

1. A system for training agile locomotion skills across gap that combines reinforcement learning and model-based control, with demonstration on the real robot.

Con:

1. One important/difficult part of gap crossing is the requirement of precise foot placement to avoid gap. It is not clear in the paper how this is being handled.  

2. While ablations show the importance of learning contact scheduling, it is not clear why learning body velocity is necessary.
summary_of_recommendation: This paper presents an approach to generate control policies for a quadruped to jump across terrains with gaps. The techniques look sounds, given that it work well both in simulation and on the real robot. But some clarifications are needed (see issues section). 

However, given that the title is "learning to jump from pixels" and this is a robotics conference, one would imagine a system that can enable jumping on real robot with real camera input, which is not the case in this paper.

That being said, given the preliminary real robot experiment and a system that works well in simulation, this paper still provides a good initial step in training agile robot motion with perception.
issues: 1. As mentioned above, it is not clear how foot placement strategy plays a role here, as one challenge of the tasks is precise foot placement. Please provide additional information about how this is implemented. My understanding is that a Raibert heuristic is used, but if the Raibert heuristic generates a target into the gap, how is this being handled? Especially in the case of using pure pixel, where gap avoidance becomes impossible?

2. While ablations show that learning contact scheduling is important, it is not clear why learning target body velocity is also important. Is it possible to just use the some reasonable fixed desired body pose/velocity? e.g, desired body height being the nominal height and desired body x, y velocities being some reasonable values.

3. In the Raibert heuristic (line 427 in appendix), how is the duration of the next placement calculated given that the contact schedule is not fixed?

4.  In the video, all scenarios seem to have gap width less than the body length (38cm). It will be interesting to see the larger gap that the gait-free policy is trained with, even if it is mostly failure.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper considers the problem of controlling a legged robot to jump over gaps. In its particular problem setting, a robot has no access to an elevation map. The paper proposes a hybrid controller to perform jumping maneuvers with an RNN of image input. The proposed algorithm is evaluated on a physical walking robot with the rate of successful jumps given different kinds of input. 
strengths_and_weaknesses: *Strengths*
* The paper seems to address a challenging problem with significance to the robotics community.
* Experiments use data from a physical robotic system.

*Weaknesses*
* The paper is more of a technical report than a report of scientific research. 
* The writing makes it difficult to identify the research questions.
* The writing makes it difficult to find and follow the paper's main narrative.
* The writing makes several vague and overly-general claims---these should be sharpened. For instance, the paper claims RL is data-inefficient (line 52). This statement is too vague to be meaningful. Furthermore, this statement seems generally false because some RL algorithms are quite efficient relative to their alternatives. Overall, the writing needs to reduce these instances of loose claims and make a more specific points.
* Line 61: Remove exclamation point.
* Algorithm 1 is not very informative. All the novel details one may need to implement and understand this work is in TRACK-TRAJECTORY(). 
* Lines 41--43: This is a subjective claim; it should be removed or edited.
* Behavioral cloning and the Sim-to-real transfer bits seemed strange to include in this study. Can the authors expand on why they think these parts are important for their study?  

summary_of_recommendation: Currently I am leaning to reject this paper because of concerns about its significance and correctness. The paper seems like it was put together in a rush. This made the writing difficult to parse, and it made it challenging to identify what research was being done---what the questions were and how the experiments shed light onto these issues. 

I can tell that the paper is trying to tackle a difficult control problem, and I appreciate how its experiments went the extra distance to deploy its proposed algorithm onto a physical robot. Unfortunately, because of issues with clarity and execution, those results appear somewhat inconclusive. How do the performance differences between baselines target the concerns mentioned in the introduction? For example, learning to jump without an elevation map. Is the conclusion that learning to jump can be sufficiently done with an RNN? Besides the results, the experiments do not contain an uncertainty analysis that would give readers confidence in their reproducibility.

I am open to adjusting my score if perhaps, during the rebuttal, the authors can clarify their position and expand on their results.
issues: 1. Figure 1 violates formatting guidelines, because it is placed before the abstract.
2. Writing needs to be improved. See list of weaknesses.
3. What should be taken away from Figure 5? All the feet are on the ground at the same time? How can that be?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this work, the authors proposed a novel architecture to train locomotion policies that can solve challenging visual locomotion tasks such as gap crossing. The controller contains a high level, learned policy that can modulate the desired body pose and velocity of the robot based on visual inputs (height map or depth map), and a low level MPC-WBC based motion controller to track the desired center of mass pose overtime. The training contains two stages. In the first stage, a teacher policy is trained using reinforcement learning and using ground truth height maps. In the second stage, a student policy, which takes raw depth image as inputs, is trained through behavior cloning. Using the learned policy, Mini Cheetah can trot or pronk over wide gaps in simulation. The authors also deployed the teacher policy to the real robot using ground truth height maps, and demonstrated great sim-to-real transfer results. 
strengths_and_weaknesses: The main strengths of the paper are:

(1) A novel architecture to combine learning with traditional control methods. Using a learned "planning" policy to generate reference center of mass motions and track the motion using a traditional, model predictive control based controller. This will minimize the sim to real gap especially for dynamic locomotion gaits such as pronking.

(2) A two-stage training with teacher and student policies. The teacher can solve the easier task with more info, i.e. the ground truth height map, and the student can mimic the teacher's behavior with limited inputs (front view depth images) and a RNN based memory module. This will increase the learning efficiency.

(3) Good sim-to-real transfer results of the teacher policy. The authors demonstrated dynamical pronking over a 30 cm wide gap which is a pretty significant result. 


The main limitations of the paper are:

(1) Not enough baselines to compare with. In many non-learning based papers,  there is this heuristic baseline where the foothold choice is based on the projected landing position of the feet:  A local optimization is used to search valid, nearest foothold location if the projected landing position is inside the gap. Consider adding this as a baseline to compare with at least the teacher policy proposed in this paper.

(2) Not enough real robot results for the student policy (i.e. depth RNN). The authors demonstrated successful sim-to-real transfer to the teacher policy which uses ground truth height maps. However, the results from the depth RNN policy is not reported. Even if there are difficulties in sim-to-real transfer of the student policy, it is still worth reporting and discussing the results. If the transfer fails, is it because of the state estimation, or because of the imperfect ego-motion model embedded in the RNN, or discrepancies in the visual inputs (i.e. distribution shift). 
summary_of_recommendation: How to elegantly use visual info in locomotion to solve challenging tasks is a hot and challenging topic. Based on my assessments of the strengths and limitations of the paper, I recommend accepting the paper conditioned on addressing my comments. 
issues: Please see my comments on the main limitations of the paper. 

On top of that, I think Table 1 can use more baseline results. Also, I think it is worthwhile to study the asymptotic performance of RNN only policy (RL trained). The RNN policy is expected to be much harder to train given the limited information, so naturally it can take more samples to reach the same performance. Since there is no training curve provided I am not even sure if the RNN policy is converged. Need more details on this. 

Another question I have is the quality of depth image. From the video the depth images look surprisingly good, which contradicts from my experiences with D435 cameras: There is no invalid band, no noise near the edges, etc. More discussions are needed on this.
review_title: null

Meta-review: This paper proposes to combine model-free learning and model-based control to enable a quadruped robot to jump over gaps using visual inputs. All reviewers agree that the paper is tackling an important and challenging control problem that could have potentially large impact. The combination of reinforcement learning and MPC is neat and could inspire follow-up works. The revision has significantly improved the quality of this paper. Original concerns about the clarity of writing, lack of baselines, and missing on-robot experiments using the student policies are sufficiently addressed in the revision. New experiments on the robots are impressive. 

The PMTG baseline for gap-crossing in simulation looks a bit weird though: many times the robot steps into the gap but seems to still get the contact force to keep it balanced (e.g. 00:06s in Trotting: Relaxed TG, rear left leg; also in the first plot Trajectory Plots). This does not change any conclusion of the paper because the PMTG baseline would perform even worse. But explaining/fixing this artifact will further improve the paper quality. 
Decision: Accept (Poster)