Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an architecture for adapting the locomotion of a legged robot in response to semantic terrain information. A human teleoperates the robot with a joystick to provide scalar velocity labels associated with different terrains. A mapping from RGB images to velocities is learned using the features of a pretrained vision module to promote generalization. Then, velocity commands are heuristically mapped to gait parameters, and the resulting locomotion skill is actuated using an MPC controller from prior work. The complete system enables the robot to automatically run faster on easier terrains and walk carefully on harder terrains. The system can complete novel trails in this manner without failure.
strengths_and_weaknesses: Strengths: The perception of semantic terrain information is a promising research area for locomotion, and this paper is the earliest known functional proposal. The proposed architecture is novel and demonstrates nice integration of several control components.

Weaknesses: A main conclusion of the paper is drawn from a single trial of a high-variance metric.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The perception of semantic terrain information is a promising research area for locomotion, and this paper is the earliest known functional proposal. The proposed architecture is novel and demonstrates nice integration of several control components. However, a main conclusion of the paper is drawn from a single trial of a high-variance metric. The authors should either (a) increase the number of trials and provide details about each failure, or (b) choose a less brittle evaluation metric. Also, the architectural choices in the paper may have some limitations, which the authors should express clearly to invite follow-up work in this area.
issues: - A main result of the paper, the failure rate in Table 1, is only reported for one trial per policy. This metric seems to have high variance and is also exposed to influence by the human operator. For example, the "unexpected bumps" you refer to in Section 7.5, which I read as small but dangerous features like potholes, would cause high variance in the failure rate depending on whether the robot happens to step on them or not in that trial. Do you have some video showing those failures? Can you report more trials or propose a less brittle evaluation metric?
- Also in Table 1, the Fixed-Medium controller has a much higher average speed than the Fully-Adaptive controller. It seems fair to add an evaluation of a fixed controller with equal avg speed (looks like 0.78m/s?)
- Some potential limitations of this framework should be discussed in the paper. The low-level policy is limited in how it can adapt to terrain properties. For example, there is a fixed friction value in the model used for MPC controller. Consider that if this were tuned for each terrain, the locomotion on slippery terrains could be stable at higher speeds. But it's impractical for the demonstrator to provide a demo that captures this, or in general to demonstrate adaptation in anything more than one or two highly interpretable dimensions of gait.
- During a discrete transition between terrains, something interesting happens. If transiting from challenging to easy terrain, the robot needs to speed up *after* it leaves the challenging terrain. If transiting the opposite way, from easy to challenging terrain, the robot must slow down *before* leaving the easy terrain. It seems like this behavior cannot be captured by your approach which is based on averaging in a fixed region. Please elaborate on this.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a system which learns to adapt walking speed based on semantics of terrain. The proposed approach learns, from human demonstration, to regulate forward speed as a function of RGB input. The model is pretrained on a semantic segmentation task and optimized via behavior cloning for the speed control problem. A gait selector is manually designed as a function of desired walking speed. The result is a system which is capable of traversing rough terrain more efficiently than the built-in Unitree controllers. 
strengths_and_weaknesses: Strengths - 

The paper studies an important problem - clearly some amount of semantic information is important for locomotion in offroad environments. Existing works typically fail to adequately address this in that they only model the geometry of the surface rather than material properties like friction, or by being entirely reactive. Thus the paper suggests an interesting innovation. 

Weakness - 

The learning problem proposed is perhaps too simple - controlling speed alone is likely insufficient for more challenging terrain, as illustrated in recent papers which additionally learn to regulate gait (e.g. the Hutter Science paper cited). It seems as though some combination of these ideas is required for robust locomotion, controlling both the desired forward speed and the parameters of the gait together as a function of anticipated terrain properties and proprioceptive data. That the method never fails on the test domains suggests that perhaps the tests are not sufficiently challenging. Moreover, comparison to a method which is purely geometry-based would be instructive in evaluating whether the semantic module indeed constitutes a substantial improvement in performance in the domain studied (e.g. even a simple variant without the semantic segmentation pretraining). 

Additionally, the reliance on human demonstrations may present a challenge in scaling the method. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I'm recommending weak rejection on the basis of the simplicity of the learning problem posed and the relatively limited experimental evaluation. 
issues: Abstract: consider changing "terrain type and property, reveals" to "...properties, reveal"
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a framework to learn a set of perceptive locomotion skills. The key idea of this work is to leverage existing offroad driving dataset for quadrupedal offroad navigation tasks. 
strengths_and_weaknesses: Strengths:
- A very clean and neat framework to enable perceptive locomotion skills
- The use of existing datasets from other domains for the target downstream tasks
- Both quantitative and qualitative evaluation of the approach, especially the robustness of the models.


Weakness:
- Line 96: Does this mean that authors assume there are no obstacles in terrain that would stop robots from traversing at all? The limitation section mentions future work of navigation, but that would require the robot to reorient its heading directions. This should also be mentioned in the limitations.
- Line 122: Authors should provide more intuition on why using FC on each pixel instead of using convolutional layers over pixels and directly output the forward speed. What is the intuition behind having this intermediate speed map?
- For real robot videos, it would be even better if there is a list of average speed over different domains. When watching the multi-terrain type video, it’s hard to tell if robots are executing at the same speed as individual evaluation. 
- Also, since the locomotion skills are not high-speed behaviors, it’s still difficult to tell the speed differences by playing each video individually. Maybe putting all video clips on one single canvas would give a better idea of the gaits’ differences.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I strongly recommend to accept this publication with the following two reasons: 1) it shows how to leverage existing datasets for robotics research. It is notoriously known that it’s hard to collect large-scale datasets for robot control as contrast to existing vision or nlp datasets. This work shows that we can leverage large datasets for robot control. And specifically, it’s a novel for the perceptive locomotion community. 2) It shows a working locomotion system that is very impressive. The no-cup video clip of full trail locomotion strongly shows the robustness of the approach. I think it’s quite rare to see in the robotics community nowadays, and it sets up a good example of what is a solid robot learning approach is. 
issues: Aside from the weakness mentioned, several comments on the details of writing:
- Line 25: Here, although it seems intuitive, lacks citations from either perception literature or previous works to support this claim. For example, maybe JJ Gibson's Affordance chapter could support this claim.
- Section 6: Since the MPC controller is from previous works, I suggest putting it into a single paragraph instead of a single section. 
- Line 172: "design" or "implement"? Maybe authors say "design" because they are different from how previous works implement impedance control for locomotion. I am confused if this is a novel design or an adaptation from a previous approach.  
- Line 182: I think unseen terrains refer to unseen "instances" of terrain types. It's ambiguous about "unseen terrains."
- Table 1: I think it's important to say what's the length of a single trail, as the authors only report a total number of lengths (6km) while the traverse time says around 10 min, which was quite surprising at first. Stating the length of trails themselves would help readers understand the result much clearer.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper presents a quadruped robot locomotion method, where MPC-based locomotion is enhanced with visual-based terrain type identification that drives correctly speed and gait. The whole method learns from human demonstrations on the real hardware.
strengths_and_weaknesses: Strengths:
- This is a clever work. It is a basic concept, i.e., when moving on different terrains, the speed and gait might need to adapt accordingly.
- The technical part is easy to follow and sound -- having said that the development part is not extremely challenging.
- The experiments are overall nice, with real-world locomotion on various terrains.

Weaknesses:
- Abstract: "close-to-optimal speed" is a bit of an exaggeration. It has not been shown anywhere in the results what is the optimal, and thus this sentence is not supported. Moreover, I have the feeling that it is also not true.
- Intro/RW: the intro and later the related work, lacks some work on terrain semantic classification for locomotion, e.g., "Terrain Classification and Locomotion Parameters Adaptation for Humanoid Robots Using Force/Torque Sensing", by K. Walas et al. (IJHR'16) for proprioception and "Terrain Segmentation and Roughness Estimation using RGB Data: Path Planning Application on the CENTAURO Robot", by V. Suryamurthy et al. (IJHR'19) for exteroceptive segmentation.
- Intro: "human expert demonstration" there is a big discussion now how those humans became experts and what does this mean?
- Intro: "Fast and Failure-free: this is again very vague and generic. In best case, terrain types need to be mentioned at least. Otherwise, this does not have any particular meaning.
- RW: In the end of the RW section, the selection to learn from real demonstration is highly proposed. Although, we know that learning in simulation (e.g., in isaac) achieves very good results without the need to adapt anything on the physical robot (e.g., the work on ANYmal in the latest Science publication). It needs to be justified better why in this case simulations could not do the job.
- Sec 4: "instead, we extract a semantic embedding" this is very vague and needs to be explained further. Why the semantic segmentation is not good, but the embedding is? Why it is feeded via a fully connected layer? More justification is needed.
- Sec. 4: "1m long, 0.3m wide in front of the robot." what if the camera changes orientation or position. How much of the method can work as is?  How much retraining is needed?
- Sec. 4: "using imitation learning": it is still confusing how the humans decided the "optimal" speed for each type of terrain.
- Sec. 5: the speed-gait selection seems to be very tedious and not extendable easily to other robots. Any alternative? Any less heuristic approach?
- Sec. 6: slope handling is claimed but none of the experiments include slopes. Not sure if it is good to be mentioned if it was not tried at all.
- Sec. 7: you compare with Unitree's controller, but in reality [21] should be your baseline. How much better than [21] do you perform?
- Sec. 7: " it can only walk up to 0.5m/s on rock", how was this evaluated? What if different gaits were applied? 
- Sec. 7.3: "traverses through all of them without failure", this is very generic and probably not true. Unless the authors claim we have found the optimal MPC-based control to deal with all terrains. In other words, a specification of gait and terrain type is needed to justify this.
- General: What happens when you traverse from one terrain type to another? How the robot adapts to the new gait and speed? What happens when terrain segmentation is noisy, and what happens when the robot walks on a misclassified terrain?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe the work requires some improvement on the weaknesses mentioned above, but in general it is nice. The main problem is overselling the approach. But if we ignore this part, the paper appears to be solid, sound, easy to follow, and nice.
issues: All weaknesses above need to be handled. Most are on the writing part and I think one is on the experimental part (slopes).
review_title: null

Meta-review: metareview: I believe many reviewers are convinced by the author's response. The motions and behaviors of the robot in the video are not impressive but I respect the reviewers' opinion. 

Quality: The reviewers agree that the paper's quality is high.

Clarity: All reviewers rated the clarity as good or above. However, a few comments about it in the reviews should be addressed.

Originality: The presented method is quite simple but the application to legged robotics seems original.

Significance: The reviewers provide a detailed list of issues. There are a few critical ones that the authors should answer in detail, especially the ones by the reviewer Rr19. First, changing the forward speed alone is insufficient to deal with many wild environments. The paper narrowed the scope of learning too much and the performance of the controller is highly limited. The accompanying video is not as impressive as those of the recent related works. The robot probably could run on most of the terrains if it used a better lower-level controller. Second, there are many heuristically designed components because the proposed controller is used as a combination of an MPC controller and a learned planner.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)