Learning to Jump from Pixels
Authors: Gabriel B Margolis
Tao Chen
Kartik Paigwar
Xiang Fu
Donghyun Kim
Sangbae Kim
Pulkit Agrawal
Venue: CoRL2021 Poster (2021)
Keywords: Locomotion
Vision
Hierarchical Control

Abstract: Today's robotic quadruped systems can robustly walk over a diverse range of rough but continuous terrains, where the terrain elevation varies gradually. Locomotion on discontinuous terrains, such as those with gaps or obstacles, presents a complementary set of challenges. In discontinuous settings, it becomes necessary to plan ahead using visual inputs and to execute agile behaviors beyond robust walking, such as jumps. Such dynamic motion results in significant motion of onboard sensors, which introduces a new set of challenges for real-time visual processing. The requirements of agility and terrain awareness in this setting reinforce the need for robust control. We present Depth-based Impulse Control (DIC), a method for synthesizing highly agile visually-guided locomotion behaviors. DIC affords the flexibility of model-free learning but regularizes behavior through explicit model-based optimization of ground reaction forces. We evaluate performance both in simulation and in the real world.


Reviews:
[{"review_id": "txTvY0g_If5", "reviewer": "Reviewer_8yhZ", "summary": "This paper presents an approach to train control policies to execute agile gap jumping. A combination of reinforcement learning and model-based whole body control is used to generate robust policies. Policies is first trained with height map then distilled into a policy that takes raw camera images as input. Results demonstrate agile jumping motions across gaps, both in simulation and on the real robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n1. A system for training agile locomotion skills across gap that combines reinforcement learning and model-based control, with demonstration on the real robot.\n\nCon:\n\n1. One important/difficult part of gap crossing is the requirement of precise foot placement to avoid gap. It is not clear in the paper how this is being handled.  \n\n2. While ablations show the importance of learning contact scheduling, it is not clear why learning body velocity is necessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents an approach to generate control policies for a quadruped to jump across terrains with gaps. The techniques look sounds, given that it work well both in simulation and on the real robot. But some clarifications are needed (see issues section). \n\nHowever, given that the title is \"learning to jump from pixels\" and this is a robotics conference, one would imagine a system that can enable jumping on real robot with real camera input, which is not the case in this paper.\n\nThat being said, given the preliminary real robot experiment and a system that works well in simulation, this paper still provides a good initial step in training agile robot motion with perception.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. As mentioned above, it is not clear how foot placement strategy plays a role here, as one challenge of the tasks is precise foot placement. Please provide additional information about how this is implemented. My understanding is that a Raibert heuristic is used, but if the Raibert heuristic generates a target into the gap, how is this being handled? Especially in the case of using pure pixel, where gap avoidance becomes impossible?\n\n2. While ablations show that learning contact scheduling is important, it is not clear why learning target body velocity is also important. Is it possible to just use the some reasonable fixed desired body pose/velocity? e.g, desired body height being the nominal height and desired body x, y velocities being some reasonable values.\n\n3. In the Raibert heuristic (line 427 in appendix), how is the duration of the next placement calculated given that the contact schedule is not fixed?\n\n4.  In the video, all scenarios seem to have gap width less than the body length (38cm). It will be interesting to see the larger gap that the gait-free policy is trained with, even if it is mostly failure."}}, {"review_id": "kPUBE_-Efmf", "reviewer": "Reviewer_NUCw", "summary": "This paper considers the problem of controlling a legged robot to jump over gaps. In its particular problem setting, a robot has no access to an elevation map. The paper proposes a hybrid controller to perform jumping maneuvers with an RNN of image input. The proposed algorithm is evaluated on a physical walking robot with the rate of successful jumps given different kinds of input. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n* The paper seems to address a challenging problem with significance to the robotics community.\n* Experiments use data from a physical robotic system.\n\n*Weaknesses*\n* The paper is more of a technical report than a report of scientific research. \n* The writing makes it difficult to identify the research questions.\n* The writing makes it difficult to find and follow the paper's main narrative.\n* The writing makes several vague and overly-general claims---these should be sharpened. For instance, the paper claims RL is data-inefficient (line 52). This statement is too vague to be meaningful. Furthermore, this statement seems generally false because some RL algorithms are quite efficient relative to their alternatives. Overall, the writing needs to reduce these instances of loose claims and make a more specific points.\n* Line 61: Remove exclamation point.\n* Algorithm 1 is not very informative. All the novel details one may need to implement and understand this work is in TRACK-TRAJECTORY(). \n* Lines 41--43: This is a subjective claim; it should be removed or edited.\n* Behavioral cloning and the Sim-to-real transfer bits seemed strange to include in this study. Can the authors expand on why they think these parts are important for their study?  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Currently I am leaning to reject this paper because of concerns about its significance and correctness. The paper seems like it was put together in a rush. This made the writing difficult to parse, and it made it challenging to identify what research was being done---what the questions were and how the experiments shed light onto these issues. \n\nI can tell that the paper is trying to tackle a difficult control problem, and I appreciate how its experiments went the extra distance to deploy its proposed algorithm onto a physical robot. Unfortunately, because of issues with clarity and execution, those results appear somewhat inconclusive. How do the performance differences between baselines target the concerns mentioned in the introduction? For example, learning to jump without an elevation map. Is the conclusion that learning to jump can be sufficiently done with an RNN? Besides the results, the experiments do not contain an uncertainty analysis that would give readers confidence in their reproducibility.\n\nI am open to adjusting my score if perhaps, during the rebuttal, the authors can clarify their position and expand on their results.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. Figure 1 violates formatting guidelines, because it is placed before the abstract.\n2. Writing needs to be improved. See list of weaknesses.\n3. What should be taken away from Figure 5? All the feet are on the ground at the same time? How can that be?"}}, {"review_id": "6Ji5uO9RM0", "reviewer": "Reviewer_dwZr", "summary": "In this work, the authors proposed a novel architecture to train locomotion policies that can solve challenging visual locomotion tasks such as gap crossing. The controller contains a high level, learned policy that can modulate the desired body pose and velocity of the robot based on visual inputs (height map or depth map), and a low level MPC-WBC based motion controller to track the desired center of mass pose overtime. The training contains two stages. In the first stage, a teacher policy is trained using reinforcement learning and using ground truth height maps. In the second stage, a student policy, which takes raw depth image as inputs, is trained through behavior cloning. Using the learned policy, Mini Cheetah can trot or pronk over wide gaps in simulation. The authors also deployed the teacher policy to the real robot using ground truth height maps, and demonstrated great sim-to-real transfer results. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper are:\n\n(1) A novel architecture to combine learning with traditional control methods. Using a learned \"planning\" policy to generate reference center of mass motions and track the motion using a traditional, model predictive control based controller. This will minimize the sim to real gap especially for dynamic locomotion gaits such as pronking.\n\n(2) A two-stage training with teacher and student policies. The teacher can solve the easier task with more info, i.e. the ground truth height map, and the student can mimic the teacher's behavior with limited inputs (front view depth images) and a RNN based memory module. This will increase the learning efficiency.\n\n(3) Good sim-to-real transfer results of the teacher policy. The authors demonstrated dynamical pronking over a 30 cm wide gap which is a pretty significant result. \n\n\nThe main limitations of the paper are:\n\n(1) Not enough baselines to compare with. In many non-learning based papers,  there is this heuristic baseline where the foothold choice is based on the projected landing position of the feet:  A local optimization is used to search valid, nearest foothold location if the projected landing position is inside the gap. Consider adding this as a baseline to compare with at least the teacher policy proposed in this paper.\n\n(2) Not enough real robot results for the student policy (i.e. depth RNN). The authors demonstrated successful sim-to-real transfer to the teacher policy which uses ground truth height maps. However, the results from the depth RNN policy is not reported. Even if there are difficulties in sim-to-real transfer of the student policy, it is still worth reporting and discussing the results. If the transfer fails, is it because of the state estimation, or because of the imperfect ego-motion model embedded in the RNN, or discrepancies in the visual inputs (i.e. distribution shift). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "How to elegantly use visual info in locomotion to solve challenging tasks is a hot and challenging topic. Based on my assessments of the strengths and limitations of the paper, I recommend accepting the paper conditioned on addressing my comments. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "Please see my comments on the main limitations of the paper. \n\nOn top of that, I think Table 1 can use more baseline results. Also, I think it is worthwhile to study the asymptotic performance of RNN only policy (RL trained). The RNN policy is expected to be much harder to train given the limited information, so naturally it can take more samples to reach the same performance. Since there is no training curve provided I am not even sure if the RNN policy is converged. Need more details on this. \n\nAnother question I have is the quality of depth image. From the video the depth images look surprisingly good, which contradicts from my experiences with D435 cameras: There is no invalid band, no noise near the edges, etc. More discussions are needed on this. "}}]

Author rebuttal:

Meta-review: This paper proposes to combine model-free learning and model-based control to enable a quadruped robot to jump over gaps using visual inputs. All reviewers agree that the paper is tackling an important and challenging control problem that could have potentially large impact. The combination of reinforcement learning and MPC is neat and could inspire follow-up works. The revision has significantly improved the quality of this paper. Original concerns about the clarity of writing, lack of baselines, and missing on-robot experiments using the student policies are sufficiently addressed in the revision. New experiments on the robots are impressive. 

The PMTG baseline for gap-crossing in simulation looks a bit weird though: many times the robot steps into the gap but seems to still get the contact force to keep it balanced (e.g. 00:06s in Trotting: Relaxed TG, rear left leg; also in the first plot Trajectory Plots). This does not change any conclusion of the paper because the PMTG baseline would perform even worse. But explaining/fixing this artifact will further improve the paper quality. 
Decision: Accept (Poster)