Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a gait controller for quadruped robots where the controller is parameterized by target velocities, feet patterns, stepping frequency, and the trunk height. So the controller can satisfy user-provided target velocity commands while generating various gait styles, which enables users to easily adapt their commands to achieve the desired tasks. More specifically, the controller is learned by deep RL where it is asked for achieving a random command generated by a sampling scheme that is adaptive to the current controller performance. Additionally, domain randomization is included for sim-to-real transfer. 

strengths_and_weaknesses: Strength

> The learned controller is robust and shows impressive sim-to-real performance, so it can be deployed in many challenging environments. The gait styles that the controller generates are diverse, which also include some agile motions. By considering these two factors, it has potential to be used as a base controller in many quadruped robot platforms.

> The paper is well-written and the exposition is mostly clear.


Weakness

> Not much of technical novelties were presented, where most components came from the existing methods.

> No failure case is explicitly mentioned in neither the paper nor the supplemental material.


Suggestion

> Why are the contact sensor inputs not given to the controller? Is there any particular reason for this? Because the command and the reward function requires specific foot patterns, I guess such information would be really helpful for better performance. 

> Figure3: it would be better if the horizontal axis could be” # of simulation step” instead of “training iteration” so that readers know how many samples would be necessary for the training.

> The motions generated do not seem energy efficient, can we improve this by adding more weights on it?

> Table6: the figure on the right is not symmetric w.r.t. w_z^{cmd}. Although I don’t think it should be perfectly symmetric due to randomness in nature, it should be approximately symmetric because there exists the adaptive sampling algorithm and the robot is perfectly symmetric at least in the simulation. For example, there is a significant performance difference between (-2.5, -2.5) and (-2.5, 2.5). 

> “One might hypothesize that the most energy-efficient gaits are also the easiest to learn, but our results go against this hypothesis (pronking emerges earlier than pacing; see Section 4.1).” -> This might also be due to the balancing problem where pronking or bounding has less balancing issue than the others. 

> I don’t find the definition for {\hat q} in Table1. Is it the previous joint positions?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Although there’s not many novel technical components, finding the right combination that would work well in the real world is not a trivial problem. I think that the combination presented in this paper is simple but very powerful enough to be used as a base controller in many existing quadruped platforms.
issues: Please see “Suggestion” above
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors propose a new system to train diverse locomotion skills in a single policy. Compared with previous works where only the moving speeds and angular speeds are tracked, the proposed policy architecture also takes into account the preferred gait parameters such as frequency and phase offsets. With a curriculum training procedure, the authors enable the quadruped robot to perform trotting, pacing, pronking, and bounding, all using a single learned policy. The authors zero-shot deploy the policy to a real go-1 robot and show that the gait conditioned policy’s performance is better than gait-free baselines. 

strengths_and_weaknesses: The strengths of this paper:

The paper is clearly written and easy to follow. The results are comprehensive and the hardware evaluations are rich. The authors measure various aspects including stability and power consumption.

The weaknesses of this paper:

1, Limited novelty. The delta between this and previous works seem to be mainly the inclusion of gait parameters during the RL training. All other aspects, such as policy architecture, reward shaping, and training curriculums, seem to be a reimplementation of previous works. 

2, Limited baselines. There is only a gait-free baseline that the author is compared with. I think it will be great to include other baselines such as these trained with motion imitation methods.



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper seems to have limited novelty and little comparisons with enough baselines. 
issues: 
Other issues of the paper, in no particular order:

1, In Table 2, the notation v_x^cmd appears twice.

2, In Table 4, prefer to use the cost of transport over power consumption, since CoT is more standard in locomotion literature.

3, There are also works related to gait transition and learning such as: Fast and Efficient Locomotion via Learned Gait Transition, which is not discussed in the related works. 

4, What is the control frequency of the policy?

5, The gait-free baseline. “The pacing and trotting gaits yield the best survival time during zero-shot deployment on this particular terrain, outperforming the gait-free baseline.” This is a bit surprising. Since the gait free baseline with auxiliary rewards pretty much will convert to trotting, and thus I would expect similar (good) stability. 

6, In Figure 4, please mark the gait transitions on the gait pattern subfigure to help visualization.  Also, mark the timing of each robot figure on top.

7, The “agile jumping” is hardly a successful jump as I can tell from the video. I think during the training there is no reward to incentivize the jumping height. I would only call this  as a bounding gait with a small aero phase.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work presents a new RL-based system that learns a single gait-conditioned policy to produce dynamics locomotion movements of a quadruped robot.
strengths_and_weaknesses: Strengths:
- The paper is easy to follow and as far as the reviewer is aware of, sound.
- Learning various gaits, with a single policy appears great as an idea.
- The system is implemented on real robots, demonstrating the sim2real challenge.

Weaknesses:
- The robot does not perform well in several cases, i.e., there are instances that it just drags the back feet on the ground, which is not optimal control.
- Is the paper, something more than combining [22], [16], and [25] (and mainly [22])?
- It is very confusing how energy consumption was measured (Table 4)? Some details are needed.

Overall, not much else the reviewer can comment on this work, as far as a clear statement about the differences with [22] are mentioned and of course Table 4 generation. Other than that, this is a really nice paper.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: If the novelty compared to [22] (and also [16], [25]) is explained and there is indeed some significant work that has been done (other than 2-to-4 legs), it is a nice paper in the field with real-world demonstrations.
issues: - A clear statement about the differences with [22] are mentioned.
- Table 4 energy measurement explanation.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a framework for learning policies for a quadrupedal robot that are conditioned on different gait parameters. This allows a single policy to generate a diverse set of gaits, including the common trotting, pacing, pronking and bounding. Analysis of the resulting policy shows that different gaits demonstrate different desired characteristics and are used to solve different downstream tasks.
strengths_and_weaknesses: Pro:
1. Demonstrate diverse gaits on a quadrupedal robot using a single policy, including successful sim-to-real transfer.

Con:
1. Motion quality degrades significantly when the desired velocity goes up, e.g., the agile jump sequence, compared to what is demonstrated here: https://sites.google.com/view/jumpingfrompixels, https://www.youtube.com/watch?v=9QV0Nnm80tU. Notably, the feet spread wide apart to gain more stability.
2. Some more analysis is needed, see issues below.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper extends the framework of learning a gait-conditioned policy from Siekmann et al to quadrupedal locomotion. While demonstration on the hardware is appreciated, the sim-to-real success is not surprising given similar demos in the past few years. The system is built upon several prior works with limited algorithmic innovation. Some further analysis of different gaits or more impressive demos that is not achievable by prior work can strengthen the paper. 
issues: 1. "No previously proposed reinforcement learning agent has been suitable to accomplish any of these tasks", (a) long jump: https://openreview.net/pdf?id=R4E8wTUtxdl, (b) through confined space: no work that I am aware of demonstrates this, but I don't think it is difficult to achieve with existing RL method, e.g., train a pacing gait to go through the narrow space (as is done in this paper), or train a crawling gait with low desired body height. One selling point is that one policy can achieve all these tasks, but the result is not surprising given that all necessary components are already available in the existing literature. Some suggestions, can we extend this framework so that it does less common gaits? e.g., two-legged hopping, skateboarding. 

2. In Table 6, similar heat maps for other gaits will be helpful information. What is the commanded stepping frequency for this table? I will imagine high-frequency stepping will be helpful to get better high-speed velocity tracking performance.

3.  In Figure 8, what is the unit of the temperature bar? What is the commanded stepping frequency?

4. Some analysis of how the stepping frequency affects performance can be useful, e.g., a high stepping frequency should be able to achieve faster locomotion as well as more robust to perturbation.

5. I think there are typos in Table 2, e.g., there are two $v_{x}^{cmd}$ in the column.
review_title: null

Meta-review: metareview: ### Initial Meta-review:

#### Strength:

The paper is clearly written and easy to follow. Successful real-world demonstration of the learned single policy controller on a physical quadruped robot is presented over different gait styles and on challenging terrains. Detailed evaluations on various aspects including robustness and energy consumption are provided.

#### Weakness:

One of the weaknesses of this paper is lack of novelty. The proposed method appears to be a combination of existing methods. In addition, the weights of the reward terms may need to be carefully chosen for successful implementation. It would be useful to include comparison to other baselines.

### Final Meta-review:

The authors have sufficiently addressed the comments by the reviewers with additional results, and clarified the novelty and significance. Majority of the reviewers agree that this paper be accepted.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)