Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a energy-based end-to-end reinforcement learning approach to realize natural locomotion gaits at different speeds in real quadruped robots. The authors show empirically that energy minimization criterion is enough to realize structured locomotion gaits at different speeds. The gaits show different behaviors for different speeds and match the behavior and energy profile of animals in nature. Finally, the authors validate their approach in both simulation and hardware showing effective speed tracking and robustness against challenging terrains.

strengths_and_weaknesses: Strengths
-Sometimes less is better. A simple approach such as only using energy consumption is shown to be enough to render natural locomotion gaits without using hand-coded sequences or reference trajectories. 
-The method presents an end-to-end model-free reinforcement learning approach that is simple but effective and does not require additional dynamic model information of the robots, which makes the approach general for different quadruped robots. 

Weaknesses
-It is hard to differentiate where the robustness of the method comes from since ARM is used on top of the proposed method to realize effective sim-to-real transfer. 
-The claim that energy minimization leads to the emergence of structured locomotion gait patterns in flat terrains and unstructured gaits in complex terrains may be imprecise and hard to demonstrate.  In particular, because additional rewards are added for uneven terrain with unstructured gaits and more aggressive environment randomization limits. 
-Different training sessions are needed for tracking different speed and therefore to emerge different behaviors.


summary_of_recommendation: Although the paper presents  good results in terms of realizing energy efficient gaits in both simulation and hardware, it is hard to identify the actual reason for the success of the method, specially in terms of the robustness and sim-to-real transfer. Also, there are some important observations in the contributions, as some of them may not really correspond to the implementation of the proposed method but an addition of a previous method on top of the proposed one. 
I would recommend to include the desired velocity as part of the state so that only one trained policy can realize different behaviors for walking at different speeds.


issues: - Bullet 1 and 4 of the main contributions are basically the same.
- Bullet 3 in main contributions is not really a contribution of this paper, but the result of ARM [17].
- Why extra penalties are added when doing uneven terrain with unstructured gaits?

Minor comments:
- According to section 1 in the appendix, the simulation time step is 0.025 seconds while the controller frequency is 100 Hz, which means the controller time step is 0,01 seconds. This results in the control actions being faster than the simulation itself, which adds unnecessary computational cost to the controller. 
- There is an error in the dimension of the state space in the appendix. It says 32 when it is 30.
- Move Table 2 to page 6.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper explores a hypothesis of gait emergence on a quadruped robot that energy minimization leads to different natural gait patterns such as walking, trot and bouncing gaits depending on the given target walking speed. First, the locomotion policy, which is a mapping from the current state and the previous action represented using a three-layer perceptron, is optimized via model-free reinforcement learning in numerical simulations using PPO. In order to obtain a robust gait policy on a flat surface, the policy is trained on uneven terrains instead of adding simulation noise. A simple reward function is used consisting of the terms associated with the forward velocity and the posture of the robot, penalizing energy consumption, and ground impact. Then, the learned policy in numerical simulations is transferred to a real quadruped robot using RMA. It is demonstrated that a robust walking policy can be obtained without any pre-programming or pre-defined movements for leg motions using the proposed framework on the physical robot. Furthermore, empirically, it is shown that different natural locomotion patterns can emerge depending on the speed of the gait as a consequence of energy minimization. Comparison of the efficiency of gaits obtained in the literature illustrates suggests the advantage of the proposed model-free approach.
strengths_and_weaknesses: Strength:
This paper is clearly written and easy to follow. As the supplementary video shows, the obtained policy works remarkably well in the real quadruped robot demonstrating the robustness against perturbations and adaptability to uneven terrain in real environments. In addition, the hardware experiments well-demonstrate the hypothesis raised in this work from an empirical point of view through the proposed reinforcement learning-based optimization approach. The resultant performance is impressive given that the locomotion policy can be obtained via model-free reinforcement learning without any pre-programming or pre-defined leg movements out of a simple energy minimizing reward, or any real world fine-tuning in transferring trained policies from simulations to real environments. 

Weakness:
Indeed, the successful application of reinforcement learning to train locomotion policy in generating natural different gaits on a real quadruped robot via energy minimization is highly impressive. First, the idea of minimizing energy for gait discovery has been previously explored in the case of a simplified biped model in the paper [a] listed below. The finding in this study [a] is that different gait patterns such as inverted-pendulum walk and bouncing run depending on the speed of walking can be found via energy minimization by numerically solving an optimal control problem using sequential quadratic programming (SQP). Although there is a difference in the complexity of the problem settings and optimization approaches (either model-based or model-free), the underlying idea seems to be conceptually similar. 

[a] Manoj Srinivasan and Andy Ruina, Computer optimization of a minimal biped model discovers walking and running, Vol. 439, 2006, doi:10.1038/nature04113

Second, it is not very clear from the description of the paper how the authors came to choose the learning architecture and the reward function. There is an impression that the proposed framework is somewhat a combination of existing methods. It would be quite surprising to see a relatively simple RL framework with a choice of an energy minimization reward could result in such a complex behavior of the robot in a high-dimensional setting without providing any pre-programming or pre-determined leg movements.

summary_of_recommendation: The paper is clearly written and well organized. The results with a real robot are very impressive while some of the technical details could be elaborated for the interest of the potential readers and the robot learning community. The paper could be further improved by discussing the motivation and the reason how the authors came to choose the proposed learning architecture, and the actual amount of manual effort in order to make the learning framework successful in such a complex system.
issues: The paper could be more informative if additional technical details that make successful implementation of the learning framework could be provided in the main text of the paper by addressing the following points:

1. How was the reward designed? How did the authors choose each term in the reward function and how were the weights tuned? 
2. How was the neural network structure (3-layer perceptron with 128 units) chosen for policy representation? The reviewer wonders whether authors explored any other choices for function approximators. 
3. How was the training initialized? 
4. In order to demonstrate the learning performance, it is suggested including a learning curve for multiple runs. Was learning always converge and robust?
5. It would be more informative if the reason why the authors did not choose domain randomization in this study.
6. Providing discussions on the advantage of the proposed model-free method over an MPC approach in terms of resultant efficiency would be more informative.
7. Providing detailed discussions on the technical differences in the approaches in comparison to the related work in Section 5 would be more informative.
8. It would be more interesting to see qualitative comparisons of the results to biomechanical studies of animal locomotion from a scientific aspect.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper explores the emergence of different quadruped gaits using reinforcement learning, by maximizing a forward reward and minimizing a lateral movement/rotation and an energy cost.
The agent is trained in simulation with domain randomization, and successfully deployed in the real world using Rapid Motor Adaptation (RMA).
strengths_and_weaknesses: Pros:
The paper shows that several quadruped gaits can be successfully learned in simulation and then deployed in the real world  by maximizing a forward reward and minimizing a lateral movement/rotation and an energy cost.
The obtained gaits have better energy efficiency than their MPC counterpart for different target speed.
The analysis shows that other quantities (for instance continuity cost of the torque) are implicitly minimized thanks to the energy term.

Cons:
The main issue with this paper is the unclear contribution with respect to Rapid Motor Adaptation (RMA), which is only briefly mentioned in section 2.5.
In fact, the current paper can be summarized as RMA minus the additional auxiliary rewards (action magnitude, smoothness, foot slip, ...).

The rest of the experimental setting is the same, and the robustness and sim2real transfer can be mainly attributed to RMA methodology:
- domain randomization
- domain adaption (RMA)
- fractal environment

summary_of_recommendation: Despite good real world results, as the contribution with respect to RMA is unclear, I would recommend this paper for rejection, but I'm willing to improve my rating upon clarification by the authors.
issues: The authors must clarify the contribution of the paper with respect to RMA.

The claim that minimizing energy consumption is sufficient for the emergence of natural locomotion gaits is also an overstatement given the experimental details from the paper.
It is true that is plays an important regularizer role, but it is not sufficient, as the task reward is encoded in the forward reward + lateral movement/rotation cost.

The result of the Froude number analysis is unclear: it is true that the learned gaits are closest to the sheep, but what can we do with such information?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an analysis that with simple energy minimization and training on fractal terrain can lead to different gaits at different target speeds. 
strengths_and_weaknesses: Pro:

1. Demonstrate that a simple reward (compared to other work that use similar control structure) is able to generate natural gaits that can work well on the real robots.

2. Froude number analysis with number compared with real animals.

Con:

No major flaw. See more comments below.
summary_of_recommendation: The idea in the paper is straightforward and the paper is well written. The analysis of gait, especially compared to those of animals' using Froude number is interesting.

However, similar result (gait emergence with simple energy minimization) is also demonstrated in Learning Symmetric and Low-Energy Locomotion, by Yu, et al. Although they did not give the throughout analysis of different gait patterns as provided by this paper.

Another impressive part of the paper is the robustness of the controller, although that mainly benefits from the approach in prior work, i.e.,  Learning quadrupedal locomotion over challenging terrain by Lee et al,  and Rma: Rapid motor adaptation for legged robots, by Kumar et al, and is not a contribution of this paper.

Overall, this paper presents a simple approach to train policies for different locomotion gaits for the A1 robot and can be valuable to the community, although the contribution is a bit incremental.






issues: Questions/Concerns:

1. The claim "Show that minimizing energy consumption is sufficient for the emergence of natural locomotion patterns" is not true, as mentioned in the paper, fractal terrain during training is also necessary.

2. How repeatable are the experiments? e.g., if I train with 10 different random seeds, will all seeds give us the same gait for the same speed?

3. Can we see what the gaits trained on flat ground looks like in the video? I can't get any information from Figure 8 alone.

4. It will be interesting to see the transition between different speeds.

5. Will the authors open source the code? Open source implementation will greatly benefit the community, especially given the affordable price of the robots.
review_title: null

Meta-review: This paper proposes to train locomotion controllers using reinforcement learning with a simple reward function for energy efficiency. Different locomotion gaits emerge automatically at different locomotion speeds, which is consistent with the findings in animal motor control. All reviewers appreciate the impressive robot experiments. The rebuttal cleared the original concern about its delta from the RMA paper. The revision of the paper and new experiments have sufficiently addressed reviewers' questions about details of the learning setup and the repeatability of experiments. In addition, the new results on gait transitions further improved the quality of the paper.
Decision: Accept (Poster)