Fast and Efficient Locomotion via Learned Gait Transitions
Authors: Yuxiang Yang
Tingnan Zhang
Erwin Coumans
Jie Tan
Byron Boots
Venue: CoRL2021 Oral (2021)
Keywords: Legged Locomotion
Hierarchical Control
Reinforcement Learning

Abstract: We focus on the problem of developing energy efficient controllers for quadrupedal robots. Animals can actively switch gaits at different speeds to lower their energy consumption. In this paper, we devise a hierarchical learning framework, in which distinctive locomotion gaits and natural gait transitions emerge automatically with a simple reward of energy minimization.  We use evolutionary strategies (ES) to train a high-level gait policy that specifies gait patterns of each foot, while the low-level convex MPC controller optimizes the motor commands so that the robot can walk at a desired velocity using that gait pattern. We test our learning framework on a quadruped robot and demonstrate automatic gait transitions, from walking to trotting and to fly-trotting, as the robot increases its speed. We show that the learned hierarchical controller consumes much less energy across a wide range of locomotion speed than baseline controllers.

Fast and Efficient Locomotion via Learned Gait Transitions
Yuxiang Yang', Tingnan Zhang", Erwin Coumans", Jie Tan', and Byron Boots' Uniivrsiity of Washington, "Robotics at Google
{yuxiangy, bboots @cs. washington. edu {tingnan, erwincoumans, bietan]0googe com
Abstract: We focus on the problem of developing energy efficient controllers for quadrupedal robots. Animals can actively switch gaits at different speeds to lower their energy consumption.
this paper, we devise a hierarchical learning frame- work, in which distinctive locomotion gaits and natural gait transitions emerge automatically with a simple reward of energy minimization. We use evolution- ary strategies (ES) train high-level gait policy that specifies gait patterns of each foot, while the low-level convex MPC controller optimizes the motor com- mands so that the robot can walk at desired velocity using that gait pattern. We test our learning framework on a quadruped robot and demonstrate automatic gait transitions. from walking to trotting and to fly-trotting, as the robot increases its speed. We show that the learned hierarchical controller consumes much less en- ergy across wide range of locomotion speed than baseline controllers.
Keywords: Legged Locomotion, Hierarchical Control, Reinforcement Learning
1
Introduction
Fast and energy efficient locomotion crucial for legged robots to accomplish tasks that traverse long distances. In the natural world, quadrupedal animals demonstrate wide variety of distinctive locomotion patterns known as gaits [1], such as walking, trotting, bounding and galloping. Each gait is characterized by unique foot contact schedule in a locomotion cycle. To lower their energy consumption, most quadrupedal animals switch to a preferred gait at each different speed range [1, 2]. While many locomotion gaits have been implemented on quadrupedal robots [3, 4], gait timing is often hand-engincered, and the switch between gaits is based on ad-hoc user commands. Can quadruped robots learn energy efficient gaits and natural gait transitions automatically?
In this work, we devise learning framework in which energy-efficient locomotion controllers emerge automatically. The learned controllers naturally switch between different gaits at differ- ent speeds to maximize energy efficiency. Learning speed-adaptive locomotion gait controllers challenging. Although reinforcement learning (RI.) has been used to train policies end-to-end for wide variety of continuous control tasks [5, 6, 7], these policies are often difficult deploy safely on real robots without additional sim-to-real effort such as reward shaping [8], domain randomization [9, 10], or meta learning [11, 12]. Alternatively, optimal control based controllers have demon- strated robust performance a number of quadruped robots [3, 13J. However, since gait patterns involve discrete contact events, it is difficult to optimize them together with other continuous forces. Therefore, most optimal control based controllers assume fixed, pre-selected gait timings.
To address the above challenges. we devise a hierarchical framework that combines the advantages of both RL and optimal control. This hierarchical framework consists of high-level guil generator and a low-level convex MPC controller, which decouples the locomotion task into gait generation and motor control. Instead of directly outputting motor commands, the gail policy functions as part of the gait generator and specifies key gait parameters, which determines the contact schedule for each leg. Based on this contact schedule, the conver MPC controller then determines which legs are in contact, and computes the optimal motor command for each leg. We formulate the high-level gait policy learning a Markov Decision Process (MDP), design simple reward function based on velocity tracking and energy efficiency, and train the gait policy using evolutionary strategies (ES).
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

We formulate the low-level convex MPC controller using model preec-prediceive control with simplified dynamics [3].
With this hierarchical framework and the simple reward function, the gait policy automatically learns distinctive gait patterns at different locomotion speeds, including slow walking, mid-speed trotting and fast fly-trotting. Moreover, the policy automatically transitions from one gait to another to gen- erate the most efficient gait all speeds. Thanks to the robustness of our hierarchical framework, the learned gait policy can be deployed successfully on a Unitree Al robot [14] in various environments (e.g. carpet, grass, short obstacle) without additional data collection and fine-tuning.
The main contributions of this paper include:
A hicrarchical learning framework effectively combines RL with optimal control, which can automatically learn fast and efficient locomotion controllers;
The learned controllers switch gaits across a wide range of locomotion speeds, similar to those demonstrated in the animal kingdom;
The learned controller can be deployed directly to the real world, and performs robustly in various environments.
2
Related Work
Quadrupedal animals demonstrate a wide variety of gaits [15]. Hoyt and Taylor [1] showed empir- ically that horses minimize their energy consumption when using the preferred gait each speed. Alexander and Jayes [2] generalized this result by providing a unifying theory of gait transitions for quadrupedal animals. wide variety of these gaits have been implemented in quadrupedal robots, including walking [16], trotting [3], pacing [4], bounding [17] and galloping [18]. In these works, model-based controllers [3, 19] optimize for motor commands a high frequency. These controllers usually assume a pre-defined contact sequence to keep the optimization problem tractable, which does not allow gait transitions. Alternatively, contact implicit optimization [20, 21, 22] optimizes contact forces and sequences together, but is not feasible for real-time use due to high computation cost. Using manually designed heuristics, Boussema et al. [23] achieved online gait transition by computing the Feasible Impulse Set for each leg, at speed up to 0.6m/s, or 1 body length/s. Owaki and Ishiguro [24] also demonstrated online gait transition on a 2kg quadruped robot using foot-force heuristics. Compared to these approaches, our learning-baset approach requires less manual tuning, achieves more agile motions (up to 2.5m/s, or 5 body length/s) on larger robot (15kg).
Recently, reinforcement learning became popular approach to learn locomotion policies for legged robots [9, 25, 26]. Since policies are often learned in simulation, extra effort is usually required to transfer the learned policies to the real robot, including building more accurate simulation [9, 25], dynamic randomization [9, 10], motion imitation [27, 28] and meta learning [11, 12]. Inspired by the periodicity of locomotion behaviors, several methods have been proposed to make the learned policy more predictable and safer for real robot deployment, such as cyclic trajectory generators [29], thase-functiinee neural networks [30] and state machines [31]. In contrast to these previous works, our learned policy achieves zero-shot sim-to-real transfer, and the controller is robust in multiple real-world environments.
Compared to directly learning end-to-end controller, hierarchical learning [32] can improve data efficiency, and achieve complex tasks. The low-level controller can be a learned policy [33, 34, 10], or a hand-tuned controller [35, 36, 37]. Li et al. [36] uses a learned policy to modulate objectives of the low-level MPC controller, with fixed contact sequence. Recently, Da et al. [35] looked into learning hierarchical controllers for gait control in quadrupeds, where the low-level controller is model- based and high-level policy selects from fixed set of gait primitives. We use similar hierarchical setup as Da et al. [35]'s, but extend the high-level policy to search continuous range of gaits with arbitrary gait changes, and achieve significantly faster walking.
3
A Hierarchical Framework for Gait Optimization
3.1 Overview
To learn fast and efficient locomotion, we build a hierarchical framework with high-level gait generator and low-level convex MPC controller (Fig. 1). The high-level gait generator includes

---

Base Vel, Desired Vel, IMU, Mictor Angles
Swing Controller
IK, PD Control
Yes
Foot Position
Base Vel Desired Vel
Gait Policy
Gait Parameters
Phase Integrator
11...
No
Stance Controller
J
Base Vel,
Desired Vel, IMU, Motor Anglos
Desired Foot Force
State Estimator
Motor Torque Command
Gait Policy (20Hz)
Phase Integrator (500
Convex MPC Controller (500Hz)
Figure 1: Our system consists of high-level gait generator and low-level convex MPC controller.
learnalle gait policy and phase integrator. To generate gait, the gait policy outputs gait param- eters, such as frequency, to the phase integrator. Based on these parameters and the robot's clock, the phase integrator increments the phase for each leg and determines its contact state. In each lo- comotion cycle, the phase progresses from 0 to 2n as the foot goes from liftoff to touchdown to the next liftoff. The low-level convex MPC controller consists of separate controllers for swing and stance legs, and controls each leg differently based on its contact state. Additionally, we implement Kalman Filter-based state estimator for the torso velocity, which cannot be measured directly us- ing onboard sensors and is used by both the gait generator and the convex MPC controller. We run the high-level gait generator at 20Hz avoid abrupt changes gait commands, and the low level controllers at 500Hz for fast replanning and stable torque control.
3.2 High-Level Gait Generation
locomotion gait determined by contact schedule, the time and duration that each leg in contact with the ground. To generate foot contact schedules, the phase integrator maintains set of phase variables &I...... one for each leg. The phase Φ; E [0, 2T) denotes the leg's progress in its current gait cycle (Fig. 2). Each leg starts with swing at the beginning of gait cycle (ø: 0)- As Φ; increases monotonically, switches to stance after a threshold Φ; @swingg After Φi reaches 2x, it wraps back to zero, and starts new gait cycle in the swing phase. The propagation of phase variables, as well as the transition from swing to stance, are controlled by three key parameters, including stepping frequency f. swing ratio Pswing and phase offsets 02, 03, 04, which are specified by gait policy. We choose such parameterization because is expressive enough to represent rich set of locomotion gaits. We now describe these parameters in detail:
Stepping Frequency As a notable feature in locomotion, the stepping frequency is usually ad- justed as a trade-off between speed and efficiency. While high stepping frequency allows the robot
Φ
Φ
Stance
Swing
44 d
sswing
Figure 2: We use phases to represent the state of each leg. Each leg is assigned with an independent phase variable à E [0, 2m). olwwng is the phase threshold that leg switches from swing to stance. In the figure above, front-right (01) and rear-left (64) legs in swing, while front-left (dx) and rear-right (da) legs are in stanec.

---

to run faster, stepping unnecessarily fast can cause additional energy consumption due to excessive leg swing. In our setup. the gait policy outputs the desired leg frequency f (OHz, 4Hz], which is used to increment the phase variable in the phase integrator. Specifically, at each control step, the phase advanced by:
[(n) < øfn 1] 2m f
(1)
where At is the time step of low-level controller (0.002s).
Swing Ratio Another important characteristic of gaits is the proportion of swing time in each gait cycle. For example, while a walking gait is usually characterized by spending less than 50% of a gait cycle in swing phase, running gait typically requires much longer swing time. To model this, we define another variable, Pswing E (0, 1). which controls the switching point in phase @ewing 2Tpswing between swing (ó < (swwngg) and stance (ø 2 (swing) in each gait eycle. Assuming that a leg moves at constant frequency, a larger Powing means that it will spend more time in air in a gait cycle, which usually results in more dynamic gait.
Phase Offsets Apart from careful design of individual gait cycles, careful coordination among legs is another critical component for efficient locomotion. While we use the same and Pswing across all legs, we allow each individual leg to have a different phase offset. Let 0g E [0. 2n denote the phase offset of leg i compared to first leg (the front-right leg); then the phase of leg is O; = Φ1 + 0, Note that the order of legs is [front-right, front-left, rear-right, rear-left), or [FR, FL, RR, RL] for short. For example, setting 0₂ 0 would make the rear-left leg in sync with the front-right leg, which is frequently seen in trotting gaits.
3.3 Low-Level Convex MPC Control
The low-level convex MPC controller computes and applies torques for each actuated degree of free- dom, given the leg phases from the high-level gait generator. Our low-level convex MPC controller is based Di Carlo al. [3]. We briefly describe the controller here for the completeness of the paper. Please refer to the Appendix A for more details.
Stance Leg Control In the stance leg controller, model the robot dynamics based on the Cen- troidal Dynamics Model [3], where the full robot is simplified as a rigid-body base with massless legs. Each stance leg can generate ground reaction force at the contact point, subject to torque limit and friction cone constraints. These ground reaction forces are solved as a short-horizon MPC prob- lem whose objective is for the robot base to closely track a given reference trajectory. We generate the reference trajectory based on user-specified velocity commands. The optimized contact forces J f. In our
are then converted motor torques using the Jacobian transpose method: MPC setup, we re-optimize for ground reaction forces every time step (2ms), and only apply the first command in the optimized sequence.
Swing Leg Control The swing leg controller calculates the swing foot trajectories and uses Proportional-Derivatyy (PD) controllers track these trajectories. The swing trajectory is com- puted by fitting a quadratic polynomial over the lift-off, mid-air and landing position of each foot, where the lift-off position is the foot location the beginning of the swing phase, the landing po- sition is calculated using the Raibert Heuristics [38]. and the mid-air location is set to ensure the minimum ground clearance. Please refer to Appendix A.2 for more details. Given the position in the swing trajectory, we convert it to the desired motor position using inverse kinematics, and apply motor torques using PD controllers.
Learning Gait Policies for Fast and Efficient Locomotion
Since locomotion gait involves discrete contact events, it is difficult to model and optimize them together with other continuous forces. Instead, we formulate a Markov Decision Process and apply Evolutionary Strategies (ES) to discover the most energy efficient gaits at different speeds.
4.1 Preliminaries
The reinforcement learning problem is represented as a Markov Decision Process (MDP), which includes the state space S, action space A, transition probability p(s (t, at), reward function

---

r: S x A + R. and initial state distribution (s() We aim to learn policy S + A that maximizes the expected cumulative reward over an episode of length T, which defined as:
T
J ( Se
as)
4.2 MDP Formulation
In our MDP formulation, we only learn the gait policy in the high-level gait generator, and consol- idate the other components, including phase integrator, the convex MPC controller, and the robot dynamics, into the environment. each step, the gait policy outputs gait parameters and receives reward, which is based on energy consumption and speed-tracking performance.
State and Action Space Since we aim to optimize the gaits based the current and desired speed, we only include the desired and actual linear velocity of the base the state space s = [Thases Phuse] We do not include prioperceptive information, such as joint angles and IMU readings, because the detailed control of balance and locomotion, which consumes these information, is delegated to the low-level controller. The action space is a 5-dimensional vector in Section 3.2.
[f, Pswing 02 03, O1]. as defined
Reward Design We design the reward function as linear combination of survival bonus, speed- tracking penalty, and energy penalty:
Tbuse
Ubase
E
max(Tww; + aT;
W.
We
Ubase
mgg
Speed Penalty
Energy Penalty (Cost TTransport
The survival bonus prevents the learning algorithm from falling into the local minima of early termination. The speed penalty is the L2 norm the relative error between the desired (ōase) and actual speed (Uhase) of the base. The energy penalty estimates the Cost of Transport (CoT), a standard metric for measuring the efficiency of locomotion [24, 39, 40). The numerator estimates total power consumption in all 12 motors based on the angular velocity (ws; and torque (Tx) of each motor, and the motor parameter (a = 0.3). (See Appendix for details.) The denominator consists of the mass of the robot (m 15kg). and gravity constant (g 9.8m/s* In each episode, the desired velocity Tasse starts at 0 m/s, increases linearly to 2.5m/s 5m/s at 1m/s² and stays at 2.5m/s for the rest of the episode. We use the same weights c 3, w. 1, w. 0.37 for all of our experiments.
Early Termination on Infeasible Gaits Despite the robustness of low-level whole body con- troller, the robot can still lose balance if the gait is infeasible, such standing with one leg for an extended amount of time. To avoid unnecessary exploration in suboptimal gaits, we terminate an episode early if the robot falls (i.e. orientation deviates significantly from the upright pose, or the robot's height becomes too low).
4.3 Policy Representation and Training
We represent our policy as neural network with one hidden layer of 256 units and tanh nonlin- earities. We chose this network architecture because is sufficiently expressive learn different gaits, and structurally compact to be efficiently optimized by our learning algorithm. We train our policies using Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [41], simple, paral- lelizable evolutionary algorithm that has been successfully applied to locomotion tasks [42, 43, 44]. Compared other RL algorithms, CMA-ES performs exploration in the policy parameter space and does not require accurate value function estimation at every step [45, 46], which is well-suited for our complex hierarchical system. See Appendix C.1 for details.
5 Results and Analysis
We design experiments to validate that our framework can learn fast and efficient locomotion con- trollers. Particularly, aim to answer the following questions:
5

---

0.5m/s
1m/s
1.5m/s
2m/s
2.5m/5
3.46
Shepping Frequency (HC)
0.55
FL
0.50
Sallm Rallo
RR
244
Waiking
Trotting
rotting
0335
Powin
2.20
Time/s
10
Figure 3: Robot gait during acceleration. Dark blue indicates foot contact. The robot switches from low-speed walking, mid mid-speed trotting, high-speed fly-trotting and increases the stepping frequeney and swing ratio Powing.
Learned (ours)
Freq (Hz)
Swing Ratio
Phase
Cost of Transport
Fly Trat
Slow
Walk
0.3
[0, k. 1.5m, 0.5m]
Rapid Trot
Slow Trot
2
0.5
[0, n. n,
Rapid Trot
4
0.5
0,
1.0
Fly Trot
0.6
[0, m. n. 0
0.8
1.25
2.25
2.50
Velocity (mis)
Table Parameters of hand-tuned gaits shown in Figure 4: Cost of Transport (CoT) of hand-tuned gaits and our learned gail policy at different speeds. The CoT of learned gait is averaged over 5 random seeds. The standard deviation is small to be visible in the figure.
Fig. 4. The order of leg phases [FR, FL, RR, RLJ.
Does our framework enable the robot to learn
locomotion controllers?
2. Does the gait switching behavior, which is widely observed in animals, emerge naturally in the learning process?
3. Can the gait policy learned in simulation be deployed on real robots?
4. What are the advantages of our hierarchical framework and what are important design decisions?
5.1 Experiment Setup
We use the Unitree A1 robot [14], a small-scale, 15kg quadruped robot with 12 degrees of free- dom. We use PyBullet [47] to simulate the robot dynamics. We implement the state estimation and high-level policy inference in Python, and the low-level Centroidal Dynamics-basee Convex MPC Controller in C++. We use Mac-Mini with M1 processor as our on-board computer, which runs the high-level gait generator at 20Hz and low-level convex MPC controller at 500Hz.
5.2 Emergence of Energy-Efficient Gaits
To demonstrate that our framework can learn efficient locomotion controllers, we evaluate the learned gait policy under different speed commands, and compare the Cost of Transport (CoT) with four desiinlly animal-inspired gaits: Walk, Slow Trot, Rapid Trot and Fly Trot. To find the parameters of these gaits, we first choose the phase offsets and swing-ratio based on the charac- teristics of each gait, and then perform grid search on stepping frequency to maximize the reward (Eq. 3) at different speed ranges. The parameters of these manually-designe gaits are summarized in Table. 1. We plot the CoT of each manually-designee gait and our learned controller (averaged over 5 random seeds) in Fig. 4. Clearly, the learned policy consistently achieves the lowest energy consumption at most the speeds.
A closer look at Fig. 4 reveals that walking gait is the most efficient when the speed is below 0.8m/s, the slow and rapid trotting gait are the most efficient between 0.8 and 2m/s, and the fly-trotting gait is the most efficient when the speed is above 2m/s. We execute the learned gait policy on an accelerating speed profile, and are glad to find that the policy switches gaits at similar boundaries (Fig. 3). At low speeds (less than 0.9m/s), the policy exhibits a four-beat walking gait by moving one leg at time in the order of [RR, FL., RI, FR]. intermediate speeds (between 0.9 and m/s), the policy synchronizes the diagonal legs and exhibits trotting gait. At the highest speeds

---

Walking
Trotting
Fly Trotting
Figure 5: Real world deployment of the learned policy. The robot moves at different speeds with different gaits.
Speed Tracking
Speedimiss
2-
Desired
FR
(a) Gait switching at abrupt speed changes.
(b) Walking over an 8em step. (Success rate: 4/5)
(c) Walking on grass.
(Success rate: 5/5)
(Success rate: 4/5)
Figure 6: Deployment of gait policy novel scenarios not encountered during training.
(1.8m/s or above), the policy exhibits a fly-trotting gait, with noticeable "airborne-phases" when all legs lift from the ground (characterized by Pswing > 0.5).
Compared to gaits observed quadrupedal animals, our gait policy discovered similar behaviors low and mid-range speeds (walking and trotting). However, at high speeds, many animals would gallop. To understand why our method does automatically learn galloping at higher speeds, we reproduced galloping gait in our framework by limiting the range of phase offsets. However, we found the learned galloping gait to be actually 30% less efficient than the flying trot on our robot, in contrast to animals [1]. This difference between animals and robots has been noted in [39]. and was hypothesized that such difference could result from differences morphology, kinematics limits and joint actuation.
5.3 Validation the Real Robot
We deploy hierarchical controller, including the learned gait policy and the low-level model- based controllers, to the real robot (Fig. 5). Please watch the accompanying video. In contrast to many RL works that focus on sim-to-real transfer [9, 11. 12), our gait policy, learned entirely in simulation with PyBullet, can be directly deployed to the real world without additional data collection fine-tuning. Similar to the simulation results, the robot accelerates, it dynamically switches between walking, trotting and fly-trotting gaits, and eventually reached the speed of 2.5m/s, body lengths per second.
We test the generalization of our learned controller in a number novel scenarios that were not seen during training. Although the policy is only trained using slowly accelerating desired speed profile, the robot remains stable with abrupt changes the desired speed, such as sharp accelcration and braking (Fig. 6a). The learned controller also generalizes over an 8-cm step (Fig. 6b) and on grass (Fig. 6c). This excellent generalization is attribute to our hierarchical setup with robust low-level convex MPC controller, which has demonstrated proven robustness on wide variety of robot systems [3, 48, 49].
terrains, including walking
5.4 Comparisons with Non-Hierarchical Policies
To demonstrate the importance of the hicrarchical setup, we compare the performance of our hier- archical framework with two non-hierarchical baselines: E2E and PMTG. E2E is fully-connected end-to-end policy that directly maps from state to motor actions. PMTG implements the Policies Modulating Trajectory Generator [29], which simplifies learning by incorporating cyclic motion pri- ors, and is widely used in the locomotion learning literature [33, 50, 51]. Following the prior work [52], we expand the state space to include IMU readings and motor angles, and modify the action space to be desired motor positions. Additionally, we carefully tuned the reward function in each baseline for optimal performance. See Appendix C.2 for details. We train policies using the same CMA-ES algorithm.

---

E2E
PMTG
FR
FR
FL
FL
RR
RR
RL
RL
0,50
0.00
0.50
0.75
Time/s
Timels
0.0
2.5
10.0
15.0 17.5
20.0
Timals
(a) Gaits Learned by Non-Hicrarchical Policies.
(h) Speed Tracking Performance.
Figure 7: Gaits and speed-tracking performance of learned non-hierarchical policies that directly output motor commands. Left: The E2E policy falls into the local minima and stands in-place without moving. The PMTG policy moves forward using three legs. Right: Both policies do not track the desired speed as well as our hierarchical framework.
We find that E2E only learns to stand, while PMTG learns an unnatural gait of using 3 legs, and cannot track the desired speed well (Fig. 7). This is likely due to the optimization landscape of the flat policy parameterization being more jaggy and the optimization converges to one of the bad local optima. While additional reward shaping could yield better results, our hierarchical system provides a simple alternative effectively learn locomotion policies.
5.5 Comparisons with Different Learning Algorithms
Using Evolutionary Strategies (ES) to train our hicrarchical controller is a critical design decision. To understand its importance, compare CMA-ES with other state-of-the-art methods, including PPO [53], SAC [7] and ARS [5] in training the hierarchical controller. Since both PPO and SAC can benefit from more comprehensive state information, also these algorithms with extended observation space, which includes IMU readings, motor angles and gait phases, in addition to current and desired velocities. Please refer Appendix C.1 for setup details and additional results.
As shown Table. 2, algorithms based on Evolutionary Strategies (ES), CMA-ES and ARS, sig- nificantly outperforms other algorithms. When using the original observation space, both PPO and SAC fail to complete the task, which is likely due to the lack of sufficient information to accurately estimate the value function. With the extended observation space, SAC learns to walk forward, but fails to track the speed closely, and consumes more energy, compared to ES-based algorithms. We hypothesize that the poor performance of these two popular reinforcement learning algorithms is because the low-level convex MPC controller a black box to the RL agent, which is not fully- observable and make the environment less Markovian.
Algorithm PPO
Success?
CoT
Avg Speed Tracking Error
No
2.29 + 0.96
0.080 + 0.092
PPO (extended obs)
No
3.60 + 1.05
0.15 1 0.17
SAC
No
2.22 + 0.47
0.069 + 0.025
SAC (extended obs)
Yes
1.19 + 0.04
0.030 = 0.025
ARS
Yes
0.87 + 0.0073
0.0082 + 0.00034
CMA-ES (ours)
Yes
0.84 + 0.017
0.0083+: 0.00075
Table 2: Cost of Transport (CoT) and speed tracking error (Eq. 3) for gait policies trained by different algo-
rithms. Error bar shows standard deviation.
Conclusion
We present hierarchical learning framework that automatically learn fast and efficient gaits for quadrupedal robots. Our framework combines a high-level gait generator with a low-level convex MPC controller, where gait policy is trained using evolutionary strategies with a simple reward function. Through learning, distinctive gaits and natural transitions between gaits emerge automat- ically. More importantly, the policy learned in simulation can be successfully deployed the real world, thanks to the hierarchical setup. Observations in our robotic experiments agree well with prior nio-mechanical studies, which showed that quadrupedal animals switch gaits different speeds to lower their energy expenditure. Our hierarchical control framework is general, and can be extended to modulate not only the gait patterns, but also other parts the low-level controller, such as foot placement positions and desired base pose, enable more agile and versatile locomotion skills. We plan to further develop this hierarchical framework through the lens of bi-level optimization, and apply it to other robotic platforms.
8

---

References
[1] D. F. Hoyt and C. R. Taylor. Gait and the energetics of locomotion in horses. Nature, 292 (5820):239- 240, 1981.
[2] R. M. Alexander and A. Jayes. A dynamic similarity hypothesis for the gaits of quadrupedal mammals. Journal of zoology, 201(1)113--15,, 1983.
[3] J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim. Dynamic locomotion in the mit cheetah 3 through convex model-predictive control. In 2018 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 1-9, 2018. doi:10. 1099000. 2018.. 8594448.
[4] M. H. Raibert. Trotting, pacing and bounding by quadruped robot. Journal of biomechanics, 23:79-98, 1990.
[5] H. Mania, A. Guy, and B. Recht. Simple random search of static linear policies competitive for reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 1805-1814, 2018.
[6] N. Heess, D. TB, S. Sriram, Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Wang. S. Eslami, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: 1e1.117077..2220 2017.
[7] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference Machine Learning, pages 1861-1870. PMLR, 2018.
[8] W. Yu, G. Turk, and C. K. Liu. Learning symmetric and low-energy locomotion. ACM Trans- actions on Graphics (TOG), 37 2018
[9] Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, June 2018. doi: 10. N0..5667/RSS.2188.....
[10] O. Nachum, M. Ahn, H. Ponte, S. S. Gu, and V. Kumar. Multi-agent manipulation via loco- motion using hierarchical sim2real. In L.. P. Kaelbling, D. Kragic, and K. Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 110-121. PMLR, 30 Oct-01 Nov 2020.
[11] W. Yu, Tan, Y. Bai, E. Coumans, and Ha. Learning fast adaptation with meta strategy optimization. IEEE Robotics and Automation Letters, 5(2):2950-2957, 2020.
[12] X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and J. Tan. Rapidly adapt- able legged robots via evolutionary meta-learning, In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3769-3776, 2020. 109.1109///001544.. 2020.9341571.
[13] T. Horvat, K. Melo, and A. J. Ijspeert. Model predictive control based framework for com control of a quadruped robot. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3372-3378. IEEE, 2017.
[14] AI Website. URL https: / /www uni tree. com wtt./oooo//u//e/ comm/products/al/
[15] R. M. Alexander. The gaits of bipedal and quadrupedal animals. The International Journal of Robotics Research, 3(2):49-59, 1984.
[16] R. Grandia, F. Farshidian, R. Ranftl, and M. Hutter. Feedback mpe for torque corque-controlled legged robots. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4730-4737, 2019. doi: 1000000090/0 1.
[17] P. Eckert, A. Spröwitz, H. Witte, and A. J. ljspeert. Comparing the effect of different spine and leg designs for a small bounding quadruped robot. 20/5 IEFE International Conference on Robotics and Automation (ICRA), pages 3128-3133, IEEE, 2015.

---

[18] D. W. Marhefka, D. E. Orin, J. P. Schmiedeler, and K. J. Waldron. Intelligent control of quadruped gallops. JEEE/ASME Transactions On Mechatronics, 103:446-556, 2003.
[19] Farshidian, E. Jelavic, A. Satapathy, M. Giftthaler, and J. Buchli. Real-time motion planning of legged robots: A model predictive control approach. In 2017 IEEE-RAS 17th International Conference on Humanoid Robotics (Humanoids), pages 577- 577-584. IEEE, 2017.
[20] I. Mordatch, Z. Popović, and E. Todorov. Contact-invariant optimization for hand manipula- tion. In Proceedings of the ACM SIGGRAPH/EEaggaphiics symposium on computer anima- tion, pages 137-144, 2012.
[21] M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 33(1);69-81, 2014.
[22] Z. Manchester and S. Kuindersma. Variational contact-implicit trajectory optimization. In Robotics Research, pages 985-1000. Springer, 2020.
[23] C. Boussema, M. J. Powell, G. Bledt, A. J. Ijspeert. P. M. Wensing, and S. Kim. Online gait transitions and disturbance recovery for legged robots via the feasible impulse set. IEEE Robotics and automation letters, 412):1611-1618, 2019.
[24] D. Owaki and A. Ishiguro. A quadruped robot exhibiting spontaneous gait transitions from walking to trotting to galloping. Scientifiic reports, 7(1):1-10, 2017.
[25] Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26). 2019.
[26] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. arXiv preprint arXiv:2105.08325 2021.
[27] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Example-guided deep re- inforcement learning of physics-bacee based character skills. ACM Transactions on Graphics (TOG), 17,4):1-14, 2018.
[28] X. B. Peng, E. Coumans, T. Zhang, T.W. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 07 2020. 205507//SS..2200XVV.
[29] A. Iscen, K. Caluwaerts, J. Tan, T. Zhang. E. Coumans, V. Sindhwani, and V. Vanhoucke. Policies modulating trajectory generators. In Conference on Robot Learning, pages 916-926. PMLR, 2018.
[30] D. Holden, T. Komura, and J. Saito. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG), 36(4):1-13, 2017.
[31] K. Yin, K. Loken, and M. Van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG), 2653):05-es, 2007.
[32] D. Precup. Temporal abstraction reinforcement learning. 2001.
[33] D. Jain, A. Iscen, and K. Caluwaerts. From pixels to legs: Hicrarchical learning of quadruped locomotion. arXiv preprint arXiv:2011.11722 2020.
[34] T. L. N. Lambert, R. Calandra, F. Meier, and A. Rai. Learning generalizable locomotion skills with hierarchical reinforcement learning. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 413-419. IEEE, 2020.
[35] X. Da, Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Y. Zhu, B. Babich, and A. Garg- Learning a contact-adaptive controller for robust, efficient legged locomotion. arXiv preprint arXiv:2009.100 10019, 2020.
[36] T. Li, H. Geyer, C. G. Atkeson, and A. Rai. Using deep reinforcement learning to learn high- level policies on the atrias biped. 20/9 International Conference on Robotics and Automa- tion (ICRA), pages 263-269, IEEE, 2019.
10

---

[37] H. Duan. J. Dao. K. Green, T. Apgar, A. Femn, and J. Hurst. Learning task space actions for bipedal locomotion. arXiv preprint 1xXi::2011.047:2 2020.
[38] M. H. Raibert. Legged robots that balance. MIT press, 1986.
[39] D. J. Hyun, J. Lee, S. Park, and S. Kim. Implementation of trot-to-gallop transition and sub- sequent gallop the mit cheetah i. The International Journal of Robotics Research, 35(13): 1627-1650, 2016.
[40] V. Radhakrishnan. Locomotion: dealing with friction. Proceedings of the National Academy of Sciences, 95(10):5448-5455 1998.
[41] N. Hansen. The ema evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772 2016.
[42] Tan, Y. Gu, G. Turk, and C. K. Liu. Articulated swimming creatures. ACM Transactions on Graphics (TOG), 30(4):1-12, 2011.
[43] J. Tan, Z. Xie, B. Boots, and C. K. Liu. Simulation-basee design of dynamic controllers for humanoid balancing. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2729 2736. IEEE, 2016.
[44] T. Geijtenbeek, M. Van De Panne, and A. F. Van Der Stappen. Flexible muscle-based locomo- tion for bipedal creatures, ACM Transactions on Graphics (TOG), 32(6):1-11, 2013.
[45] T. Salimans, J. Ho, X. Chen, S. Sidor, and I. Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv: 038644 2017.
[46] A. Vemula, W. Sun, and Bagnell. Contrasting exploration parameter and action space:
A zeroth-order optimization perspective. In The 22nd International Conference Artificial Intelligence and Statistics, pages 2926-2935. PMLR, 2019.
[47] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybullet.. org,
[48] C. D. Bellicoso, F. Jenelten, P. Fankhauser, C. Gehring, Hwangbo, and M. Hutter. Dynamic locomotion and whole-body control for quadrupedal robots. In 2017 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3359-3365. IEEE, 2017.
[49] S. Fahmi, C. Mastalli, M. Focchi, and C. Semini. Passive whole-body control for quadruped robots: Experimental validation over challenging terrain. IEEE Robotics and Automation Let- ters, ((3):2553-256)) 2019.
[50] J. Lee, J. Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47). 2020.
[51] Y. Yang, K. Caluwaerts, A. Iscen, T. Zhang, Tan, and V. Sindhwani. Data efficient rein- forcement learning for legged robots. In Conference on Robot Learning, pages 1-10. PMLR, 2020.
[52] X. B. Peng and M. van de Panne. Learning locomotion skills using deeprl: Does the choice of action space matter? In Proceedings of the ACM SICGRAPH/Einograpiis Symposium on Computer Animation, pages 1-13, 2017.
[53] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint 10X.v:177700644111 2017.
[54] N. Hansen, Y. Akimoto, and P. Baudis. 901:100.281/eemndo2556666 Feb. 2019. URL https://dbi.org:: wttg/1..22///ennddd. 2559634.
CMA-ES/pycma
Github. Zenodo,
[55] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines3. attps://es //gi thub. BttM//DLRRRM///babb htt.a--RRW//taallbebeeeeeeeee baselines3, 2019.


Reviews:
[{"review_id": "haUsYnH6oyo", "reviewer": "Reviewer_J9Ji", "summary": "The paper develops energy-efficient controllers for quadrupedal robots.  Energy efficiency is achieved by switching the gait pattern depending on the robot's speed.  To this end, a hierarchical learning framework is introduced.  The gait policy is trained via reinforcement learning, while a body controller optimizes the robot's motion such that the desired velocity can be achieved.  The proposed method is demonstrated in real robot experiments.  The experiments demonstrate the generalization capability of the resultant controller to (i) sharp accelerations and decelerations, (ii) physical perturbations, and (iii) different terrains.  The experiments demonstrate that the resultant controller achieves lower energy across most of the spectrum of examined speeds (0-2.5m/s), when compared to the gait policies \"walk\", \"fly-trot\", \"slow trot\", \"rapid trot\".", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "A.  Strengths:\n\n- In my opinion, the paper is concise and well-organized, being a pleasure to read.  The statement of contributions in the introduction is clear and easy to locate visually.  The related work appears extensive (differentiation of the paper's contributions with related work is also clear).  The paper's figures make it easy to understand the proposed method, and how it compares with existing state-of-the-art methods.\n\n- Proposed method is mostly described in sufficient detail, even for the non-expert reader. \n\n- Real robot experiments appear extensive and are convincing on the generalization capabilities of the proposed method.  Comparison with other state-of-art methods appears extensive.  The accompanying video and supplementary material is similarly supportive.  \n\nB.  Where I would make changes:\n\n- I would provide intuition on how the evolutionary strategies for reinforcement learning work.\n\n- I would specify the early termination criteria for infeasible gaits.\n\n- I would quantify statements on the comparative performance of the proposed method (e.g., the statement on line 223).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In my opinion, the paper is well-written, with a clear set of technical and experimental contributions, which are supported by (i) a variety of real-world experiments that support the effectiveness of the proposed method (generalization, energy efficiency, and tracking error), and (ii) extensive comparisons with state-of-the-art methods, that further demonstrate the effectiveness of the proposed method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see my comments on where I would make changes, listed in the box \"Strengths and Weaknesses\".", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {}}, {"review_id": "bSdsv0wGpW", "reviewer": "Reviewer_ED3s", "summary": "This paper describes a mix of Model Predictive Control (MPC) and Direct Policy search for quadruped locomotion. The main objective is to allow robots to switch gaits (e.g., walk / trot / gallop) so that the gait is energy-efficient for the target speed. The main idea is that a “whole-body” state-of-the-art MPC (not learned, from previous work) computes (at 500Hz) the joint torques given target contacts and centroidal dynamics, whereas a neural network (optimized in simulation with CMA-ES) computes (at 20Hz) the contacts given the target speed and the current speed. The results show that the resulting “hierarchical controller” works well and does not need to be adapted to transfer to a real robot. The authors claim that end-to-end learning with CMA-ES does not give competitive results. The authors also attempted to learn the high-level policy using PPO, SAC and ARS and they concluded optimizing with CMA-ES gives the best result.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- the learning/optimization part compensates the weaknesses of MPC, which makes a very good combination of “classic control theory” and “learning-based robotics”\n- convincing results on the videos\n- experiments on a real quadruped robot\n- comparisons with end-to-end learning and with different learning algorithms\n\n## Weaknesses:\n- no statistical analysis of the simulation results (Fig. 3): CMA-ES is a stochastic algorithm: the authors need to run the experiments several times and report median/confidence intervals (otherwise, how do we know that the authors were not lucky once or cherry-picked a particular “seed”?)\n- no statistical analysis of the baselines (Table 2, section 5.4): all these algorithms are stochastic too. Also, it would have been good to show “learning curves” in supplementary material, at least to guess when the algorithms have converged (maybe PPO or SAC were stopped prematurely, for instance).\n- the authors learned on a simple schedule of target speeds, then say (I am paraphrasing) that it works well for any change of speed and even novel terrains; this needs to be statistically analyses using a “test set” (like in supervised learning) and statistics.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I wanted to accept this paper because I agree with the authors that it is very interesting to use learning to improve the state-of-the-art (here MPC) instead of replacing it. This paper takes a good and effective approach and improve it, and I like it. \n\nHowever, the lack of any statistical analysis as well as the lack of details about the experiments cannot be ignored. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "## Main issue: \n- lack of statistical analysis\n\n## Other issues\n- Please change CMAES to CMA-ES (the official name: https://en.wikipedia.org/wiki/CMA-ES) \n- The MPC is called a “whole-body” controller but it handles legs separately depending on when they are in contact, or not. To me “whole-body” means that the “solution” is computed for all the joint in a single optimization/computation. I would suggest to call this “leg controller” or something similar.\n- There are not enough details about the baselines. For instance, for end-to-end, we could suspect that the neural network would be too small, or too big to be optimized by CMA-ES (vs PPO/TRPO/SAC/etc.). But we have no information about the size. Similarly, how was PPO configured? what are the hyper-parameters?\n- Releasing the source code would be very helpful, especially for the MPC part\n\n### Minor remark(s):\n- The general approach reminded me: https://arxiv.org/abs/2011.04741(Learning Task Space Actions for Bipedal Locomotion) in which the authors combine learning a policy with an inverse dynamics controller (not MPC)."}}, {"review_id": "WYb6cu8plr8", "reviewer": "Reviewer_22ve", "summary": "This paper presents a hierarchical framework for building locomotion controllers that adaptively change the walking gait at different speeds. I liked that the system design is straightforward, the experiments are interesting, and the results look good. The design of the high-level gait generation task is very similar to other related work such as (1) phase-functioned neural networks and (2) simbicon (which are missing from the references). Plus, the low-level controller is based on some previous work (Di Carlo et al. [3]). These made me realize that, although this is a nice work with good-looking results, it's incremental and requires more novelty to be published in CoRL. Plus, framing CMA-ES as a RL algorithm seems misleading to me. So for now, I prefer to vote for a weak reject.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The design is straightforward. I like the formulation of the high-level gait generation task. The low-level controller also seems robust.\n- The paper is well-written and it's easy to read and follow.\n- Video results are looking good. However, it's difficult to see the difference between trotting and fly-trotting.\nWeaknesses:\n- I believe the paper is misleading the reader by claiming that CMAES is a RL algorithm. If we accept this claim, doesn't this make any other classical search or optimization algorithm a RL algorithm, assuming that the search/optimization is performed in the policy parameter space?\n- More references are needed.\n- The parameters of the experiments (number of simulation timesteps, population size, learning rate of PPO/SAC, etc) are not reported.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am willing to change my recommendation to accept if the authors do the followings:\n- Either update their claims about using RL for building a locomotion controller, or employ state-of-the-art RL algorithms (e.g., PPO or SAC) to build a robust controller.\n- Report the parameters for all training algorithms.\n- Add more references, as suggested in the issues.\n- Add a detailed discussion about the limitations of the system, showcasing failure cases in the supplemental video.\n- Or finally, convince me if I'm wrong about the issues of the paper :)", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- In Eq. (3), it's not clear to me how the reward always stays positive. For example, the speed penalty can easily take a very large value if the robot starts from the idle state, where the actual speed (v_base) is almost zero. Later in Section 4.3 it's explained how the desired velocity is increased linearly after the episode starts, but (in my opinion) during the training the agent isn't able to follow the same speed pattern. So what happens if the actual speed is 0 (or 0.01) and the desired speed is 2.5?\n- In Section 4.3, it's mentioned that the policy is trained using CMA-ES and the authors compare it to \"other\" RL algorithms. But CMA-ES is not a RL algorithm.\n- In Section 4.3, the authors do not report the CMA-ES training configuration. It is a very important thing to miss, considering the fact that this is the only learning component of the system.\n- For compuing the speed and energy penalties in Eq. (3), I think it would be better if the authors use exponential reward functions as, for example, explained in Section 5.3 of the DeepMimic paper:\nPeng, X.B., Abbeel, P., Levine, S. and van de Panne, M., 2018. Deepmimic: Example-guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4), pp.1-14.\n- It might not be a fair criticism since it's later explained by the authors, but in Section 5.2, I'd prefer to analyze Figure 3 differently. Animals optimize their energy consumption by changing their gait in different velocities. So, the correct plot in Figure 3 should have only one curve for the hand-tuned gaits, such that it uses the minimum CoT among the four hand-tuned gaits (or a linear combination of them). Then it would be observed that CoT of the learned gait and the hand-tuned one is almost identical. I believe this is what the paper was looking for in the first place (to show that their system can optimize the energy consumption in different speeds).\n- It seems that in Figure 6.a, the labels for the desired and the actual velocities are misplaced. It does not make sense to me that the curve for actual speed is piecewise constant.\n- In Section 5.5, the authors have done a poor job reporting the experiments using other algorithms. It is very important to report the number of timesteps, the size of the policy network, learning rate, etc. and also show the learning curves of the algorithms. \n- Watching the video, it's difficult to see the difference between trotting and fly-trotting. It would have been better to also know about the failure cases (both in the video and in the paper).\n- I expected to see the following papers in the references, since they use similar designs for modelling locomotion controllers:\nHolden, D., Komura, T. and Saito, J., 2017. Phase-functioned neural networks for character control. ACM Transactions on Graphics (TOG), 36(4), pp.1-13.\nYin, K., Loken, K. and Van de Panne, M., 2007. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics (TOG), 26(3), pp.105-es.\n\nTypos: \n- Line 15: traverses -> traverse\n- Line 21: gates -> gaits\n- Line 220: foot frequency -> stepping frequency?"}}, {"review_id": "Ln1iQAeVJnF", "reviewer": "Reviewer_Zoo3", "summary": "This paper presents an approach to devising a hierarchical framework for quadruped robots in order to develop an energy efficient controller. The proposed framework combines RL with optimal control that can learn fast and efficient locomotion controllers. The proposed approach consists of a high-level gait generator to be trained using evolutionary strategies formulated as an MDP and a low-level whole body controller using model predictive control with simple dynamics. Numerical simulations demonstrate that the learned controller achieves gait transition depending on the speed of locomotion from walking, trotting to fly-trotting (slow to fast locomotion). In addition, it is demonstrated that the learned policy can be directly transferred to a real quadruped robot without additional data collection or fine-tuning. On the real robot, gait transition and generalization to a variety of new terrains are also demonstrated. These results illustrate the effectiveness of the proposed approach. Furthermore, comparisons to non-hierarchical policies and different learning are presented to evaluate the efficacy of the proposed framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThis paper is concisely written and easy to read. The proposed framework is able to learn robust and efficient locomotion policy over different speed. The efficiency of the learned controller is evaluated considering the cost of transport. Depending on the speed of locomotion, gait transition can emerge, in which the robot switches from walking, trotting to fly-trotting gaits from slow speed to fast speed. The learned policy in numerical simulations can be directly transferred to a real robot without additional data collection or fine-tuning. Similar gait switching can be achieved on the real robot as well. In addition, comparison to non-hierarchical policies and different learning algorithms suggest the advantages of the proposed approach.\n\nWeakness:\nThe resultant performance shown in the accompanied video looks quite impressive in terms of robustness of the gait on the robot hardware. To my understanding, the learned controller is confined to straight-line walking, it is not clear how this approach can learn to achieve complex maneuvers such as turning and steering, and possibly stair climbing behaviors which requires some sort of foot placement planning. It would be nice if this paper discusses how the proposed approach can be extended to design controllers for versatile behaviors. It is not clear how much time would be required to learn the behaviors described in this paper. Also, it convergence of the learning algorithm is not very clear. A learning curve during training should be presented to make the paper more convincing and informative to the potential readers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a hierarchical approach for optimizing quadruped locomotion by combining high-level RL and low-level whole body controller. The proposed framework is well demonstrated both on numerical simulations and hardware experiments. The paper could be improved by addressing the reviewer's comments.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. There is an impression that the high-level clock-based controller is quite structured. It would be nice if the authors discuss the motivation of this choice. And, also citation of this kind of clock (phased)-based gait generation approach should be included. \n2. If I understand correctly, there could be interference between the high-level gait generator and the performance of the low level whole body controller. It would be more informative if this point could be further discussed.\n3. The optimized contact forces are converted to joint torques via tau = J_transpose*F as in line 145 on page 4. This relationship generally holds in a quasi-static case. In the dynamic case, inverse dynamics may need to be used. As the locomotion is highly dynamic, I wonder if this quasi-static approximation would be sufficient for successful implementation.\n4. In order to demonstrate the effectiveness of the proposed framework, the learning curve should be presented. Also, the number of requires iterations should be provided until convergence. \n5. How was the structure of the neural networks determined, i.e., the number of hidden layer and the number of units? Also, how were the weights in the reward function determined? It would be more informative if the information of such a manual tuning process be presented.\n6. It is stated that the learned policy in simulations were successfully transferred to the real robot without additional data collection or fine tuning. It would be nice if the authors discuss the reasons why the proposed approach did not require such an additional process in comparison to many of typical RL sim-to-real cases.\n7. It is interesting to compare the performance and efficiency against the built-in locomotion controller on the Unitree A1 robot if possible.\n"}}]

Author rebuttal:

Meta-review: There was a wide range of reviewer opinions on this paper. Most reviewers were in agreement that the restricted policy space used by the authors was chosen well and led to compelling results in the experimental section and the supplementary video. The main criticisms of the paper were a lack of detail about the methods employed in experiments as pointed out by several reviewers. There is no data on how the CMA-ES algorithm was actually run, and no learning curves or statistics of repeated testing were reported for the baseline methods. If the paper were to be accepted, the authors must do a substantial job revising the experimental description to provide these details. Moreover, as pointed out by reviewer 22ve the claim that their technique is "RL" should be rewritten since CMA-ES is a traditional metaheuristic optimization technique. The authors can claim they are "learning" using simulated experience, but "RL" is certainly a misnomer.

After receiving the author feedback, the reviewers are satisfied with the changes made to the paper, and are now in agreement that the paper should be accepted.
Decision: Accept (Oral)