Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper describes a mix of Model Predictive Control (MPC) and Direct Policy search for quadruped locomotion. The main objective is to allow robots to switch gaits (e.g., walk / trot / gallop) so that the gait is energy-efficient for the target speed. The main idea is that a “whole-body” state-of-the-art MPC (not learned, from previous work) computes (at 500Hz) the joint torques given target contacts and centroidal dynamics, whereas a neural network (optimized in simulation with CMA-ES) computes (at 20Hz) the contacts given the target speed and the current speed. The results show that the resulting “hierarchical controller” works well and does not need to be adapted to transfer to a real robot. The authors claim that end-to-end learning with CMA-ES does not give competitive results. The authors also attempted to learn the high-level policy using PPO, SAC and ARS and they concluded optimizing with CMA-ES gives the best result.

strengths_and_weaknesses: ## Strengths:
- the learning/optimization part compensates the weaknesses of MPC, which makes a very good combination of “classic control theory” and “learning-based robotics”
- convincing results on the videos
- experiments on a real quadruped robot
- comparisons with end-to-end learning and with different learning algorithms

## Weaknesses:
- no statistical analysis of the simulation results (Fig. 3): CMA-ES is a stochastic algorithm: the authors need to run the experiments several times and report median/confidence intervals (otherwise, how do we know that the authors were not lucky once or cherry-picked a particular “seed”?)
- no statistical analysis of the baselines (Table 2, section 5.4): all these algorithms are stochastic too. Also, it would have been good to show “learning curves” in supplementary material, at least to guess when the algorithms have converged (maybe PPO or SAC were stopped prematurely, for instance).
- the authors learned on a simple schedule of target speeds, then say (I am paraphrasing) that it works well for any change of speed and even novel terrains; this needs to be statistically analyses using a “test set” (like in supervised learning) and statistics.

summary_of_recommendation: I wanted to accept this paper because I agree with the authors that it is very interesting to use learning to improve the state-of-the-art (here MPC) instead of replacing it. This paper takes a good and effective approach and improve it, and I like it. 

However, the lack of any statistical analysis as well as the lack of details about the experiments cannot be ignored. 

issues: ## Main issue: 
- lack of statistical analysis

## Other issues
- Please change CMAES to CMA-ES (the official name: https://en.wikipedia.org/wiki/CMA-ES) 
- The MPC is called a “whole-body” controller but it handles legs separately depending on when they are in contact, or not. To me “whole-body” means that the “solution” is computed for all the joint in a single optimization/computation. I would suggest to call this “leg controller” or something similar.
- There are not enough details about the baselines. For instance, for end-to-end, we could suspect that the neural network would be too small, or too big to be optimized by CMA-ES (vs PPO/TRPO/SAC/etc.). But we have no information about the size. Similarly, how was PPO configured? what are the hyper-parameters?
- Releasing the source code would be very helpful, especially for the MPC part

### Minor remark(s):
- The general approach reminded me: https://arxiv.org/abs/2011.04741(Learning Task Space Actions for Bipedal Locomotion) in which the authors combine learning a policy with an inverse dynamics controller (not MPC).
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents an approach to devising a hierarchical framework for quadruped robots in order to develop an energy efficient controller. The proposed framework combines RL with optimal control that can learn fast and efficient locomotion controllers. The proposed approach consists of a high-level gait generator to be trained using evolutionary strategies formulated as an MDP and a low-level whole body controller using model predictive control with simple dynamics. Numerical simulations demonstrate that the learned controller achieves gait transition depending on the speed of locomotion from walking, trotting to fly-trotting (slow to fast locomotion). In addition, it is demonstrated that the learned policy can be directly transferred to a real quadruped robot without additional data collection or fine-tuning. On the real robot, gait transition and generalization to a variety of new terrains are also demonstrated. These results illustrate the effectiveness of the proposed approach. Furthermore, comparisons to non-hierarchical policies and different learning are presented to evaluate the efficacy of the proposed framework.
strengths_and_weaknesses: Strength:
This paper is concisely written and easy to read. The proposed framework is able to learn robust and efficient locomotion policy over different speed. The efficiency of the learned controller is evaluated considering the cost of transport. Depending on the speed of locomotion, gait transition can emerge, in which the robot switches from walking, trotting to fly-trotting gaits from slow speed to fast speed. The learned policy in numerical simulations can be directly transferred to a real robot without additional data collection or fine-tuning. Similar gait switching can be achieved on the real robot as well. In addition, comparison to non-hierarchical policies and different learning algorithms suggest the advantages of the proposed approach.

Weakness:
The resultant performance shown in the accompanied video looks quite impressive in terms of robustness of the gait on the robot hardware. To my understanding, the learned controller is confined to straight-line walking, it is not clear how this approach can learn to achieve complex maneuvers such as turning and steering, and possibly stair climbing behaviors which requires some sort of foot placement planning. It would be nice if this paper discusses how the proposed approach can be extended to design controllers for versatile behaviors. It is not clear how much time would be required to learn the behaviors described in this paper. Also, it convergence of the learning algorithm is not very clear. A learning curve during training should be presented to make the paper more convincing and informative to the potential readers.

summary_of_recommendation: This paper presents a hierarchical approach for optimizing quadruped locomotion by combining high-level RL and low-level whole body controller. The proposed framework is well demonstrated both on numerical simulations and hardware experiments. The paper could be improved by addressing the reviewer's comments.
issues: 1. There is an impression that the high-level clock-based controller is quite structured. It would be nice if the authors discuss the motivation of this choice. And, also citation of this kind of clock (phased)-based gait generation approach should be included. 
2. If I understand correctly, there could be interference between the high-level gait generator and the performance of the low level whole body controller. It would be more informative if this point could be further discussed.
3. The optimized contact forces are converted to joint torques via tau = J_transpose*F as in line 145 on page 4. This relationship generally holds in a quasi-static case. In the dynamic case, inverse dynamics may need to be used. As the locomotion is highly dynamic, I wonder if this quasi-static approximation would be sufficient for successful implementation.
4. In order to demonstrate the effectiveness of the proposed framework, the learning curve should be presented. Also, the number of requires iterations should be provided until convergence. 
5. How was the structure of the neural networks determined, i.e., the number of hidden layer and the number of units? Also, how were the weights in the reward function determined? It would be more informative if the information of such a manual tuning process be presented.
6. It is stated that the learned policy in simulations were successfully transferred to the real robot without additional data collection or fine tuning. It would be nice if the authors discuss the reasons why the proposed approach did not require such an additional process in comparison to many of typical RL sim-to-real cases.
7. It is interesting to compare the performance and efficiency against the built-in locomotion controller on the Unitree A1 robot if possible.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper develops energy-efficient controllers for quadrupedal robots.  Energy efficiency is achieved by switching the gait pattern depending on the robot's speed.  To this end, a hierarchical learning framework is introduced.  The gait policy is trained via reinforcement learning, while a body controller optimizes the robot's motion such that the desired velocity can be achieved.  The proposed method is demonstrated in real robot experiments.  The experiments demonstrate the generalization capability of the resultant controller to (i) sharp accelerations and decelerations, (ii) physical perturbations, and (iii) different terrains.  The experiments demonstrate that the resultant controller achieves lower energy across most of the spectrum of examined speeds (0-2.5m/s), when compared to the gait policies "walk", "fly-trot", "slow trot", "rapid trot".
strengths_and_weaknesses: A.  Strengths:

- In my opinion, the paper is concise and well-organized, being a pleasure to read.  The statement of contributions in the introduction is clear and easy to locate visually.  The related work appears extensive (differentiation of the paper's contributions with related work is also clear).  The paper's figures make it easy to understand the proposed method, and how it compares with existing state-of-the-art methods.

- Proposed method is mostly described in sufficient detail, even for the non-expert reader. 

- Real robot experiments appear extensive and are convincing on the generalization capabilities of the proposed method.  Comparison with other state-of-art methods appears extensive.  The accompanying video and supplementary material is similarly supportive.  

B.  Where I would make changes:

- I would provide intuition on how the evolutionary strategies for reinforcement learning work.

- I would specify the early termination criteria for infeasible gaits.

- I would quantify statements on the comparative performance of the proposed method (e.g., the statement on line 223).
summary_of_recommendation: In my opinion, the paper is well-written, with a clear set of technical and experimental contributions, which are supported by (i) a variety of real-world experiments that support the effectiveness of the proposed method (generalization, energy efficiency, and tracking error), and (ii) extensive comparisons with state-of-the-art methods, that further demonstrate the effectiveness of the proposed method.
issues: Please see my comments on where I would make changes, listed in the box "Strengths and Weaknesses".
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a hierarchical framework for building locomotion controllers that adaptively change the walking gait at different speeds. I liked that the system design is straightforward, the experiments are interesting, and the results look good. The design of the high-level gait generation task is very similar to other related work such as (1) phase-functioned neural networks and (2) simbicon (which are missing from the references). Plus, the low-level controller is based on some previous work (Di Carlo et al. [3]). These made me realize that, although this is a nice work with good-looking results, it's incremental and requires more novelty to be published in CoRL. Plus, framing CMA-ES as a RL algorithm seems misleading to me. So for now, I prefer to vote for a weak reject.
strengths_and_weaknesses: Strengths:
- The design is straightforward. I like the formulation of the high-level gait generation task. The low-level controller also seems robust.
- The paper is well-written and it's easy to read and follow.
- Video results are looking good. However, it's difficult to see the difference between trotting and fly-trotting.
Weaknesses:
- I believe the paper is misleading the reader by claiming that CMAES is a RL algorithm. If we accept this claim, doesn't this make any other classical search or optimization algorithm a RL algorithm, assuming that the search/optimization is performed in the policy parameter space?
- More references are needed.
- The parameters of the experiments (number of simulation timesteps, population size, learning rate of PPO/SAC, etc) are not reported.
summary_of_recommendation: I am willing to change my recommendation to accept if the authors do the followings:
- Either update their claims about using RL for building a locomotion controller, or employ state-of-the-art RL algorithms (e.g., PPO or SAC) to build a robust controller.
- Report the parameters for all training algorithms.
- Add more references, as suggested in the issues.
- Add a detailed discussion about the limitations of the system, showcasing failure cases in the supplemental video.
- Or finally, convince me if I'm wrong about the issues of the paper :)
issues: - In Eq. (3), it's not clear to me how the reward always stays positive. For example, the speed penalty can easily take a very large value if the robot starts from the idle state, where the actual speed (v_base) is almost zero. Later in Section 4.3 it's explained how the desired velocity is increased linearly after the episode starts, but (in my opinion) during the training the agent isn't able to follow the same speed pattern. So what happens if the actual speed is 0 (or 0.01) and the desired speed is 2.5?
- In Section 4.3, it's mentioned that the policy is trained using CMA-ES and the authors compare it to "other" RL algorithms. But CMA-ES is not a RL algorithm.
- In Section 4.3, the authors do not report the CMA-ES training configuration. It is a very important thing to miss, considering the fact that this is the only learning component of the system.
- For compuing the speed and energy penalties in Eq. (3), I think it would be better if the authors use exponential reward functions as, for example, explained in Section 5.3 of the DeepMimic paper:
Peng, X.B., Abbeel, P., Levine, S. and van de Panne, M., 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4), pp.1-14.
- It might not be a fair criticism since it's later explained by the authors, but in Section 5.2, I'd prefer to analyze Figure 3 differently. Animals optimize their energy consumption by changing their gait in different velocities. So, the correct plot in Figure 3 should have only one curve for the hand-tuned gaits, such that it uses the minimum CoT among the four hand-tuned gaits (or a linear combination of them). Then it would be observed that CoT of the learned gait and the hand-tuned one is almost identical. I believe this is what the paper was looking for in the first place (to show that their system can optimize the energy consumption in different speeds).
- It seems that in Figure 6.a, the labels for the desired and the actual velocities are misplaced. It does not make sense to me that the curve for actual speed is piecewise constant.
- In Section 5.5, the authors have done a poor job reporting the experiments using other algorithms. It is very important to report the number of timesteps, the size of the policy network, learning rate, etc. and also show the learning curves of the algorithms. 
- Watching the video, it's difficult to see the difference between trotting and fly-trotting. It would have been better to also know about the failure cases (both in the video and in the paper).
- I expected to see the following papers in the references, since they use similar designs for modelling locomotion controllers:
Holden, D., Komura, T. and Saito, J., 2017. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG), 36(4), pp.1-13.
Yin, K., Loken, K. and Van de Panne, M., 2007. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG), 26(3), pp.105-es.

Typos: 
- Line 15: traverses -> traverse
- Line 21: gates -> gaits
- Line 220: foot frequency -> stepping frequency?
review_title: null

Meta-review: There was a wide range of reviewer opinions on this paper. Most reviewers were in agreement that the restricted policy space used by the authors was chosen well and led to compelling results in the experimental section and the supplementary video. The main criticisms of the paper were a lack of detail about the methods employed in experiments as pointed out by several reviewers. There is no data on how the CMA-ES algorithm was actually run, and no learning curves or statistics of repeated testing were reported for the baseline methods. If the paper were to be accepted, the authors must do a substantial job revising the experimental description to provide these details. Moreover, as pointed out by reviewer 22ve the claim that their technique is "RL" should be rewritten since CMA-ES is a traditional metaheuristic optimization technique. The authors can claim they are "learning" using simulated experience, but "RL" is certainly a misnomer.

After receiving the author feedback, the reviewers are satisfied with the changes made to the paper, and are now in agreement that the paper should be accepted.
Decision: Accept (Oral)