Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
Authors: Zipeng Fu
Ashish Kumar
Jitendra Malik
Deepak Pathak
Venue: CoRL2021 Poster (2021)
Keywords: Locomotion
Gaits
Biomechanics
Energetics
Reinforcement Learning

Abstract: Legged locomotion is commonly studied and expressed as a discrete set of gait patterns, like walk, trot, gallop, which are usually treated as given and pre-programmed in legged robots for efficient locomotion at different speeds. However, fixing a set of pre-programmed gaits limits the generality of locomotion. Recent animal motor studies show that these conventional gaits are only prevalent in ideal flat terrain conditions while real-world locomotion is unstructured and more like bouts of intermittent steps. What principles could lead to both structured and unstructured patterns across mammals and how to synthesize them in robots? In this work, we take an analysis-by-synthesis approach and learn to move by minimizing mechanical energy. We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots. The emergent gaits are structured in ideal terrains and look similar to that of horses and sheep. The same approach leads to unstructured gaits in rough terrains which is consistent with the findings in animal motor control. We validate our hypothesis in both simulation and real hardware across natural terrains. Videos at https://energy-locomotion.github.io

Minimizing Energy Consumption Leads to the Emergence of Gaits in Legged Robots
Zipeng Fu', Ashish Kumar?, Jitendra Malik?, Deepak Pathak CCannegie Mellon University, 2UC Berkeley
Legged locomotion is commonly studied and expressed as a discrete set of gait patterns, like walk, trot, gallop, which are usually treated as given and pre-pogrrameed in legged robots for efficient locomotion at different speeds. However, fixing a set of pre-programmed gaits limits the generality of locomotion. Recent animal motor studies show that these conventional gaits are only prevalent in ideal flat terrain conditions while real-world locomotion is unstructured and more like bouts of intermittent steps. What principles could lead to both structured and unstructured patterns across mammals and how to synthesize them in robots? In this work, we take an analysis-by bisiis-----ytnhssii approach and learn to move by minimizing mechanical energy. We demonstrate that learning to minimize energy consumption plays a key role in the emergence of natural locomotion gaits at different speeds in real quadruped robots. The emergent gaits structured in ideal terrains and look similar to that of horses and sheep. The same approach leads to unstructured gaits in rough terrains which is consistent with the findings in animal motor control. We validate our hypothesis in both simulation and real hardware across natural terrains. Videos at ht httpp:://eeeet /tttpp://eeeeee eeneegy-locooottt
Abstract:
gi thub. io.
Keywords: Locomotion, Biomechanies, Energetics, Reinforcement Learning
Introduction
common approach understanding mammalian locomotion is study in terms of a set of discrete gaits. Seminal works include Muybridge [1]'s Is's motion video of a galloping horse and Hildebrand [2]'s study of different gaits observed at different speeds. This has inspired research on replicating these gaits in robots by pre-programming them to achieve effective locomotion. This approach of pre-programming fixed gait patterns has lead to immense progress in robotic locomotion [3, 4, 5, 6].
However, pre-programmed gaits limit the ability of legged systems to perform general locomotion in diverse terrains and different speeds. Studies of motor control in both animals and human infants [7] in the last few decades have shown that general locomotion in natural settings is irregular and does fit predefined fixed gaits. It better defined in terms of bouts of steps than periodic cycles [8, 9]. That being said, we do see consistent regular patterns in ideal conditions when an animal is moving in straight line over flat terrain at different speeds, e.g., walk, trot, canter, gallop, etc. [2, 10, 11, 12] and these patterns are shared across species of diverse animals [13]. How do we reconcile these seemingly contradictory arguments about unstructured locomotion patterns on complex terrains with that of regular patterns seen in flat terrains under the same rubric?
One way make sense of these differences is see them from the lens of minimizing energy consumption. Locomotion consumes a significant fraction of an animal's metabolic energy budget which suggests that there would be evolutionary selection pressure for locomotion strategies that minimize energy consumption. Indeed, this has been a focus of several studies in biomechanics and energetics. Hoyt and Taylor [12] used energy minimization to explain why animals switch from one gait to another as their speed increases. Similar studies explain gaits across different animals including humans [14, 15]. However, most of the biomechanics studies are limited the straight- line motion of animals in ideal flat terrains. Furthermore, these studies take gaits as given and offer simplified analytical models to explain them from an energy perspective without any statement about their synthesis on relatively complex real robot systems.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Bouncing
1.5 m/s
Slow
Fast
Trot Gait
0.9 m/s
Minimize
Energy
Consumption
Walking Gait
0.375 m/s
Unstructured
Minimize
Gait
Energy
Consumption
0.35 m/s
Figure 1: We demonstrate via mealysis-by-synhheee approach that learning to move forward minimizing energy consumption plays a key role the emergence natural locomotion patterns in quadruped robots. do not pre-pre pre-program any primitives for leg motions and the policy directly output desired joint angles. Top: Bio-energetics driven learning on flat terrain leads to different gaits namely walk, trot and bounce (similar to gallop) the speed inereases. Our high speed bounce gait displays emergent flight phase despite low energy usage. This corresponds the nearest animals (sheep/horse) with similar Froude numbers. Bottom: The same pipeline on diverse uneven terrains leads to unstructured gait as true general animal locomotion. Please see videos on the website.
In this work, we take an the emergence structured locomotion gait patterns in flat terrains as well as unstructured gaits in complex terrains. We use energetics to design an end-to-end learning framework and display the resulting gait patterns
by-synthesis approach to show how energy minimization leads to
real quadruped robot. We employ model-free reinforcement learning as the optimizer to learn controllers to make the quadruped move forward while simultaneously minimizing the bio-energetics constraints of minimizing work [16]. We train our policies on simple fractal terrains with varying frequency of terrain heights instead of perfectly flat terrain. This mimics the real world more closely and achieves efficient and robust gaits without the need for artificial rewards for foot clearance periodic external push during training. Low frequency fractal terrains resemble the simple flat terrain settings and high frequency simulates complex terrains. In addition, present learning pipeline that enables smooth gait transitions given changing target speeds by combining RL rewards and distillation losses from trained expert gait policies. We then transfer the policies onto the real robot using online adaptation at test time [17].
We further analyze our emergent walking behaviors by comparing them to biological quadrupeds. Our quadruped robot is closest to sheep and horse in terms of physicality determined by the Froude number [18] which is a scalar metric characterizing gaits of quadrupeds (discussed more in Section 4.2). Therefore, energetics-based training leads to similar three gaits in our robot as found in horses and sheep, namely, walk, trot and gallop as the speed gradually increased in flat terrains Figure 1. Interestingly, we show that the gait selection is inherently linked with the speed of the robot. For instance, trotting is only energy efficient around the medium speed of 0.9m/s. If we increase or decrease the speed of the robot, it takes more energy for trot than to gallop (similar to bouncing gait) or walk as shown in Figure 2. Finally, when the same pipeline is run in uneven complex terrains, leads to unstructured gaits consistent with animal locomotion in usual diverse conditions [9, 8).
Prior works have used energy penalty to obtain energy-efficient gaits [19, 20] but have not shown the correspondence to different gaits at different speeds. Furthermore, out of these works, the ones which deploy these gaits on a real robot either use a predefined hand-coded gait library, use reference motions [19]. only learn high-level contact sequences for predefined swing leg motion [21].

---

These approaches have some fundamental limitations in contrast to what we propose. Firstly, the use of predefined foot motions never optimize for efficient swing leg motions, and using reference trajectories gives gaits that look realistic, but are not necessarily energy-efficient for the specific robot we have. Second, learning controllers end to end without the use of any priors (reference motions or predefined motions) allows for the possibility of learning behaviors that don't 't follow any predefined gaits, We show the robustness of this complex terrain policy by successfully deploying it in complex terrains in the real world.
The main contributions of this paper include:
Show that minimizing energy consumption plays key role in the emergence of natural loco- motion patterns both flat as well as complex terrains at different speeds without relying on demonstrations predefined motion heuristics.
Show that the emergent gaits at different target speeds correspond to conventional animals in the similar Froude number range (sheep/horse) without any sort of rre-programming.
Present a distillation-based learning pipeline to obtain velocity-conditione policy that displays smooth gait transition as the target speed is changed.
Demonstrate the emergent behaviors, robustness analysis, and gait patterns in simulation as well as real-world budget quadruped robot.
2 Method
2.1 Claim: Energy Minimization leads to Emergence of Natural Gaits
We would like to validate that indeed certain gaits will emerge at certain speeds if we min- imize the energy consumption. To do so, on the right, we plot the energy consumption speed of the Al robot in simulation for walk, trot and bounce (modification of gallop/canter) gaits. We use [6] to achieve the different gaits by hard-coding the contact sequences for walk, trot and bounce. This plot is qualitatively sim- ilar to the plot in [12], which shows the energy consumption of different horse gaits across dif- ferent speeds. We additionally plot the energy
MPC
140
MPC trot
Energy Consumed to Move One Meter
MPC bounce,
100
80
bounce
trot
60
walk
walk our trot bounce
40
consumption of the three policies we get from
our approach at different speeds (0.375 m/s, 0.9
m/s, 1.5 m/s). This plot shows two things. First,
20
it is observed that certain gaits are only efficient at certain speeds and using them for other speeds suboptimal. For instance, walk is optimal at low speeds with transition to trot and bounce. Secondly, we observe that our learning frame- work converges to the optimal gait at each of the speeds, and beats the efficiency of MPC gaits from [6], which does not optimize the swing foot trajectory for energy. Details of MPC method are in the supplementary.
0-
0.25
0.50
0.75
1.25 1.50 1.75 2.00 Speed (m/s)
1.00
Figure 2: Energy consumed in moving Im distance using our method and MPC shows why certain gaits are stable at certain speeds. Videos on the wehsite.
2.2 Learning Locomotion Policy
We learn the locomotion policy æ which takes as input the current state T, E 30 previous action a-- 12 to predict the next action a, (Equation 1). The predicted action a, is the desired joint position for the 12 robot joints which is converted to torque using a PD controller.
0: = T ot-1)
We implement π as MLPs (details in Section 3) and train the base policy π end to end using model-free reinforcement learning. At time step f, æ takes the current state I₁, previous action a:- to predict an action a, RL maximizes the following expected return of the policy m:
T-1
J J(T) = E. ET-p(T/F)
Σ 1=
rr
(2)
3

---

Figure Complex rcal-world behaviors the low speed settings. Top: key frames rocky terrain. Bottom: Foot contact plots and key frames on unstable moving planks. Videos on the website.
where T = {(Fo, do, ro); (x1, on, ri ).... is the trajectory of the agent when executing policy n, and p(r nT) represents the likelihood of the trajectory under m.
2.3 Stable Gait through Natural Constraints
We train our agent with the following natural constraints. First, the reward function is motivated from bio-energetic constraints of minimizing work [16]. We found these reward functions to be critical for learning realistic gaits in simulation. Second, we train our policies on uneven terrain (Figure as substitute for additional rewards used by [22] for foot clearance and robustness external push. We see that these constraints are enough to achieve all gaits and results we demonstrate this paper.
2.4 Energy Consumption-Basse Reward
Let's denote the linear velocity as and the angular velocity as w, both in the robot's base frame. We additionally define joint torques and joint angles and joint velocities q. We define reward as sum of the following three terms:
r = Fforward + 01 Tenergy + Talive
where,
Fforward = -02 * |t' 2 target] - |0,) 12 WWyaw 2
(4)
Tenergy -T² - 4.
(5)
lforward rewards the agent for walking straight at the specified speed, Vonergy penalizes energy consumption and Talive is the survival bonus.
We simple rules to set the hyper- parper--aramteers... At given target linear speed starget the survival bonus e is set to 20 ytarget We set a1 = 0.04, and a2 = 20 across all settings.
We estimate the unit energy consumption per time step by summing the instantancous power of the 12 motors by multiplying the torque and the joint velocity each motor. Also notice that this reward is much more concise than those used in prior works. We discuss the role of minimizing energy consumption in Section 4.4.
2.5 Sim to Real Transfer
For simulation to real transfer, we use RMA [17] which consists of the an adaptation module on top of the base policy. During deployment, the adaptation module uses the state history to estimate the vector of extrinsics (which contains environment information) online. Once the base policy is learned using RI, this adaptation module be trained in simulation itself using supervised learning with the true extrinsics parameters as supervision. The environment perturbations we train on is listed in Table
3 Experimental Setup and Training Details
Hardware and Simulation: We use Unitree's 's Al robot our hardware platform [23] and use its AI URDF [23] to simulate the Al robot in RaiSim simulator [24]. The full details of terrain and environment setup are provided in supplementary due to space constraints.

---

BR
RR
(a) Walking gait for a 10-sccond period
(h) Walking gait for a 2-sccond period
1R
6.0
1 (d) Trotting gait for a 2-second period
406
1.5
Trotting gait for 10-second period
1A
Line Sa
gimetsl
(e) Bouncing gait for 10-second period
(1) Bouncing gait for 2-second period
Figure 4: Foot contact plots for walking, trotting and bouncing gait of Al robot in the real world. Bold color means the corresponding foot is in contact with the ground and the light color means the foot is in the air. (RF: Right-Front font, LF: Left-Front foot, RR: Right-Rear foot, LR:: Reft foot)
State-Action Space: The state is 30 dimensional containing the joint positions (12 values). joint velocities (12 values), roll and pitch of the torso and binary foot contact indicators (4 values). The action space is 12 dimensional corresponding to the target joint position for the 12 robot joints. The 12 is converted torques T using a PD controller with target
predicted target joint angles joint velocities set to O.
Environmental Variations: All environmental varia- tions with their ranges are listed in Table 1. Policies with simple terrain and mild environment variations achieve walking, trotting and bouncing gaits. To deploy our robot in the complex terrain, we use aggressive environment randomization with sharper fractal terrains which gives us the unstructured walking terrain.
Normal Perturbation
Aggressive
Parameters
Perturbation
Friction Coeff.
[0.6, 1.2]
[0.05, 4.5]
Kg KJ
[50, 60]
[50, 60)
[0.4, 0.8]
[0.4, 0.8]
Payload (kg) Canter of Mass (m)
[0.0, 0.5]
[0.0, 6.0]
[-0.15, 0.15]
[-0.15, 0.15]
Motor Sirength Re-sample Pmob.
[0.95, 1.05]
[0.90. 1.10]
0.02
0.02
The policy is multi-layer perceptron with 3 layers which takes the current state x2 E RR30 previous action d;-1 12 and outputs 12-dim target joint angles. The hidden layers have 128 units. We train the policy and the environment encoder network using PPO [25]. The training rums for 15, 000 iterations with batch size of 100, 000 split into 4 mini- batches and learning rate 5e-4. We train for total of 1.5 billion samples which takes roughly 24 hours on a desktop machine with GPU.
Policy Learning:
Table 1: Ranges of the environment parame- ters in simulation. Normal perturbation used regular gait emergence. Aggres- sive perturbation is used for unstructured gait emergence.
4 Results and Analysis
4.1 Emergence of Locomotion Patterns in Complex Terrain
We analyze the performance of the walking policy on complex terrain with aggressive perturha- tions in the real world (videos on website). We show two un- even terrain deployments of our method in Figure 3. During the deployment of the robot in the rocky terrain, we see that the robot roughly follows the walk- ing gait while stepping over rocks of different heights which prematurely terminate the swing motion of the leg. Consequently, the gait we observe is an unstructured walking gait. We further analyse this in another deployment unstable planks which move as the robot tries cross it (see video).
Method
Walk mls)
Trot
Bounce (1.5 m/s)
Speed (m/s)
Energy
Speed (m/s)
Energy
Speed (mus)
Energy
Ours
0,353
30.7
0.970
41.2
1.49
90.7
MPC [6]
0.360
87.0
0.953
77.2
1.482
93.9
Table 2:
We show the actual speeds and energy consumed to move
one meter of the our method and convex MPC, which does not directly optimize for energy efficiency, for walking gait target speed 0.375 m/s, trotting gait target speed 0.9 m/s, and bouncing gait at target speed 1.5 m/s in simulation. Our method shows the advantages in energy consumption in all gaits.
5

---

Figure 5: Trotting with kg payload (two bottles of 500 ml water are strapped onto the robot). The 1 kg
payload is out of the normal perturbation of environment parameters used in simulation training shown in Table 1. We show qualitative and gait patterns for other two gaits in the appendix. We find that the gait patters robust and remain the same even in the presence of disturbances.
The foot contact plot for this setup is not periodic and changes depending on the complexity of the terrain. In the video, this can been seen when the front right foot of the robot steps on an unstable plank which moves as a result, but to maintain stability, the robot violates the periodic gait notions automatically.
4.2 Emergence of Walking, Trotting and Bouncing
Simulation Gaits: We train separate policies for target speeds 0.375 m/s (low), 0.9 m/s (median) and 1.5 m/s (high) in simulation. We use the same hyper-parameter across all target speeds. We observe that the walking gait emerges at target speed 0.375 m/s, trotting emerges 0.9 m/s and hopping emerges at 1.5 m/s in simulation. Note that we use RMA [17] (Section 2.5) to transfer our policies to the real world without any fine-tuning. We observe the same foot contact sequence the real world as in simulation. In Table 2, we compare the performance of our policies with MPC [6] and observe that our walking gait can achieve accurate speed tracking while being 50% more energy efficient than the MPC baseline.
Saal-Wolld Behaviors: the low speed (0.375 m/s), the rohot demonstrates a quarter-off walking gait with the following foot contact sequence: Right-Front (RF), Left-Rear (LR), Left-Front (LF) and Right-Rear (RR). At the median speed (0.9 m/s), two beat trot gait emerges where diagonal legs (RF & LR or LF & RR) are synchronized and hit the ground the same time. At the high speed (1.5 m/s), the policy converges to bouncing with approximately half the gait cycle as flight phase. Key frames of the 3 gaits are shown in Supplementary and the foot contact plots in Figure 4. We use the foot contact sensor reading from the Al robot to generate these plots.
We also measured the actual speed of the robot in the real world by taking the average of 3 trials per target speed. In each trial, we log the total distance divided by total time taken to traverse. The average real-world speeds
0.396 m/s at target speed 0.375 m/s, 0.914 m/s target speed 0.9 m/s, and 1.714 4 m/s at target speed 1.5 m/s. The average real-world speeds closely match the target speeds in all settings. The standard deviation of speeds across 3 trials is negligible.
We test the robustness of the emergent gaits by adding a Kg payload on the robot by strapping two bottles with 500 ml water. In Figure 5, show 4 key frames of trotting gait. Note that the kg payload is outside the range of training perturbations (Table 1), but our policies with the Sim-to-Real Adaptation in Section 2.5 learn to compensate the increase in weight by implicitly predicting changes in dynamics via latent extrinsics. We didn't 't test bouncing with payload, because the extra 1 kg prevents the human operator from swiftly preempting the robot high speed by lifting.
Froude Number Analysis: In Biomechanics, Froude number F is a scalar metric characterizing gaits of quadrupeds and bipeds [18]. Concretely, F = v2 /2/gh, where is the linear speed, g is the gravitational acceleration, and is the height of the hip joint from the ground. Existing research shows that animals with similar morphology but with different sizes tend to the same gait when walking with equal Froude numbers [26, 10]. We calculate the Froude numbers of our robot in the real world for the low, median and high speeds, and compare them with the Froude numbers of quadrupeds in nature dogs, horses and sheep [12, 10]. We compare the Froude number of animals at the gait transition boundaries and observe that our robot is closest to the sheep in Table 3.
Gaits (Froude #)
A1
Sheep [10]
Horses [12]
Dogs [10]
Walk
0.059
0.05
0.07
0.
Trot
0.316
0,30
0.61
0.50
Bounce
1.110
1.10
1.70
1.73
Table 3: We compare the Froude numbers of our emergent gaits with other quadruped animals at their gait transition boundaries. Our policies are closest sheep in all gaits and horse in walking gait. Bounce includes canter and gallop.

---

Energy Consumpton
Torque
Data Torque
Foot Siip
Joint Speod
Action
1.
spest recaa
2.
24
90
Fanning spocTs
Figure 7: Energy consumption plot during training. Even though our reward function does not include the hand-designed locomotion penalties that are commonly used in prior works (torque, delta torque, foot slip, joint speed, and action), we show that all these penalties are minimized in our training framework hyproduct of minimizing energy consumption.
4.3
Gait Transition via a Velocity-Conditioeed Policy
We have shown walking, trotting and bouncing emerged at three different target speeds. To demon- strate foot patterns at a continuous range of velocities, we propose a learning scheme (Figure 6) for a velocity-condttioned policy that enables smooth gait transition when the command velocity changes.
Naive Multi-Task Training Fails: We first tried the naive approach to train the velocity- conditioned policy in a multi-task fashion by randomly sampling desired velocities and using the corresponding palocit-conditiones emerged gaits collapse to only two modes: walking and trotting, but trotting at high speeds is not energy-efficienn (Figure 2). We believe the reason for failure is difficulty in optimization as the robot is now tasked not only to learn to move forward but also do it by learning different gaits which causes it to collapse.
reward (Section 2.4). However, this did not work and the resulting
Learning via Distillation and RL: We sidestep this issue via stage-wise distillation approach. We
first train our fixed-velocity policies as described in the paper which lead to walking, trotting and bouncing (galloping) gaits. We
then treat these policies as ex- perts and collect demonstration
data from them to self- sulf-supee and bootstrap the initial training phase of the velocity-coneen conditioned policy. These three policies serve as experts at three velocities at low (0.375 m/s), median (0.9 m/s) and high (1.5 m/s), repre- sented three-dimensional one- hot vectors, which are fed into the velocity-conditioned policy as input. Expert supervisions along with the RL rewards guide policy
Foldy
Wilking Pollay Kuskey
Physics Sam.delin
Masor DoMt Friction, ln Motor Strenght
Brstader (
Acaptttion Modiin
ConttyCenntiinnn Polcy
(1I)
Command Velocity
Figure 6: Learning scheme for smooth gait transition. Trainable modules are red. The blue path indicates forward and backward passes for the L2 loss on action, whereas the green path is for training by RL.
the to follow the energy-efficient efficient gaits the three expert velocities. To learn motor skills and smooth gait transition at intermediate velocities in the continuous range of 0.375 m/s to 1.5 m/s, the learning relies RI. rewards only without expert supervision. To represent randomly
on the sampled target velocity to feed into the policy, we use an interpolated vector based on the closest two experts' one-hot velocity representations (e.g. 1.2 m/s is represented as [0, 0.5, 0.5) and 0.5 m/s [0.238, 0.762, OJ). We linearly anneal the L2 supervision loss from the expert supervision (for the three velocity modes) as the training progresses. Towards the end, the learning relies only on the velocity-condittoned RL loss for the entire velocity range. Details are in the supplementary.
Real-World Smooth Gait Transition: We test our relocity-conditione policy in the wild and want to highlight the smooth gait transition happening at different speeds. Video is in the supplementary.
4.4 Ablation Studies
The two key components we use during policy training to get reliable walking gaits is an energy based reward function, and a fractal terrain. We posit that these ingredients are minimal, and we demonstrate this by ablating each of them below.
Minimizing Energy Consumption Also Minimizes Artificial Penalties: To demonstrate the suf- ficiency of minimizing energy consumption, we plot the decreasing trend of the artificial penalties during training Figure 7, even though these artificial penalties are not included in our reward function. These include Torque penalty, delta torque penalty, foot slip penalty, joint speed penalty, and
7

---

(a) Low target speed
(b) Median target speed
(c) High target speed
Figure 8: Resulting gait plots of policies without minimizing energy consumption. All policies exhibit a high-frequency tapping behavior that is not energy-efficient and not transferable to robots in the real world. In (c), the robot in simulation falls off the terrain after 4 seconds.
the action regularization [21, 27, 22, 28, 29]. A high positive correlation between energy consumption and artificial penalties can be observed. All these artificial penalties are minimized in our training framework byproduct of minimizing energy consumption. During training, the robot first learns to stand still with a minimal energy consumption (around epoch 500), and then starts moving forward with slightly higher energy consumption.
Minimizing Energy Consumption: In Figure 8, we show the the resulting gaits of policies trained without the unit energy consumption term in the reward function. All policies show seh-frreuency tapping behaviors that are not natural and inefficient. The high-free high-freuency joint movements can burn the joint motors in the robot and are not transferable the real hardware.
Fractal Terrain during Training: We train the policies with energy minimization (Section 2.4) but on flat terrain without fractal perturbations (Section 2.3). All training trials converge unnatural and unstable gaits. We found that adding fractal terrain also facilitates a larger foot clearance and robustness which improves real-world hardware deployment. We show the key frames of 2 example unnatural gaits in the supplementary.
5 Related Works
Model-Based Optimization for Gaits: Model-based controllers to generate gaits [6] optimize for the stance legs for a given foot contact sequence. They use predefined swing leg motion and show stable gait generation real robot system. One class of method is contact-implicit optimization [30, 31, 32], optimizing contact forces along with sequences. However, these gaits do not emerge as an mergyy-inimmzzing solution to moving at different speeds. [33] analyzes biped gaits by optimizing energy consumption, but uses a minimal mechanical model assuming point-mass mass body and massless legs, In contrast, swing leg motion, stance leg force and contact sequence, all emerge from minimizing energy consumption of moving of a real robot. We take a functional approach to gaits where we achieve them not predefined movements, but consequence of energy minimization.
Learning for Legged Locomotion and Gaits: Data-driven learning for legged locomotion has shown robust controllers for quadrupedal robots [22, 34, 35, 36. 37, 38]. Of these, [34, 22] focus on a controller on complex terrains but do not synthesize and analyze leg patterns of the robot across diverse gaits at different target speeds. [35] demonstrates learning method to select gait controllers, but the low-level gait controllers are predefined primitives. Concurrent work [21] shows the learned gaits and gait transition, but their method relies on many human priors (e.g. predefined cyelic motion priors), shows trotting at high speed with short periods of flight phase, and requires powerful on- on-board computation power (Apple MI chip) to solve online MPC optimization. In graphics, [39] shows low energy also plays role in generating natural locomotion animations.
6 Conclusion
This work demonstrates that energy consumption plays key role in the emergence of natural locomotion patterss in animals by following an maulysis-by-synthest approach of showing the result in real quadruped robots. The allows generation of the straight line lab gaits, well unstructured complex terrain gaits in animals. We perform thorough analysis of gait patterns, show their robustness under disturbances, and propose learning pipeline for smooth gait transition. Interestingly, the gaits obtained by our robot are most similar to those in horses and sheep the animals which are close to in terms of the Froude number metric space.
Acknowledgments This work was supported by the DARPA Machine Common Sense program. DP is partly supported by Good AI research grant. We would like to thank Chris Atkeson for high-quality feedback, and Qingqing Zhao and Shivam Duggal for many video recordings.
8

---

References
[1] E. Muybridge. The horse in motion. Library of Congress Prints and Photographs Division, 23778/.
1878. URL htttp://wwwte http:://www...cc.oooo/////////ep
[2] M. Hildebrand. Symmetrical gaits of horses. Science, 1965.
[3] M. H. Raibert. Trotting, pacing and bounding by quadruped robot. Journal of biomechanics, 23:79-98, 1990.
[4] D. W. Marhefka, D. E. Orin, J. P. Schmiedeler, and K. J. Waldron. Intelligent control of quadruped gallops. JEEE/ASME Transactions On Mechatronics, 8(4):446 2003.
[5] P. Eckert, A. Spröwitz, H. Witte, and A. J. Ijspeert. Comparing the effect of different spine and leg designs for a small bounding quadruped robot. In 2015 IEFE International Conference on Robotics and Automation (ICRA), pages 3128 3128-3133. IEEE, 2015.
[6] J. Di Carlo, P. M. Wensing, B. Katz, G. Bledt, and S. Kim. Dynamic locomotion in the mit cheetah through convex model-predictive control. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEFE, 2018.
[7] K. E. Adolph, W. G. Cole, M. Komati, J. S. Garciaguiree, D. Badaly, J. M. Lingeman, G. L. Chan, and R. B. Sotsky, How do you learn to walk? thousands of steps and dozens of falls per day. Psychological science, 2012.
[8] W. G. Cole, S. R. Robinson, and K. Adolph. Bouts of steps: The organization of infant exploration. Developmental psychobiology, 2016.
(9] D. L.. Kramer and R. I.. McILauglin. The hehavioral ecology of intermittent locomotion. American Zoologist, 2001.
[10] A. Jayes and R. M. Alexander. Mechanics of locomotion of dogs (canis familiaris) and sheep (ovis aries). Journal of Zoology, 1978.
[11] Z. Afelt, J. Blaszczyk, and C. Dobrzecka. Speed control in animal locomotion: transitions between symmetrical and nonsymmetrical gaits in the dog. Acta Neurobiol Exp, 1983.
[12] D. Hoyt and C. R. Taylor. Gait and the energetics of locomotion in horses. Nature, 1981.
[13] R. M. Alexander and A. Jayes. A dynamic similarity hypothesis for the gaits of quadrupedal mammals. Journal of zoology. 201(1):135-152, 1983.
[14] J. E. Bertram. Constrained optimization in human walking: cost minimization and gait plasticity. Journal of experimental biology. 208(6))979 9911 2005.
[15] D. T. Polet, T. Schroeder, and E. Bertram. Reducing gravity takes the bounce of running. Journal of Experimental Biology, 221(3), 2018.
[16] D. T. Polet and J. E. Bertram. An inelastic quadrupedal model discovers four- walking, two-beat running, and pseudo-elastic actuation as energetically optimal. PLoS computational biology, 2019.
[17] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. In Robotics: Science and Systems, 2021.
[18] C. L. Vaughan and M. O'Malley. Froude and the contribution of naval architecture to our understanding of bipedal locomotion. Gait &amp; posture, 2005.
[19] J. Tan, T. Zhang, F. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems, 2018.
[20] X. Song, Y. Yang, K. Choromanski, K. Caluwaerts, W. Gao, C. Finn, and Tan. Rapidly adapt- able legged robots via evolutionary meta-learning. In International Conference on Intelligent Robots and Systems (IROS), 2020.

---

[21] Y. Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots. Fast and efficient locomotion via learned gait transitions. In 5th Annual Conference on Robot Learning (CoRL). 2021.
[22] Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
[23] X. Wang. Unitree Robotics. https://wwws uni wwitree.ce. www.ree.com..
[24] J. Hwangbo, J. Lee, and M. Hutter. Per-contact iteration method for solving contact dynamics. IEEE Robotics and Automation Letters, 2018. URL www www.raisim.mm com.
[25] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (00.006377. 2017.
[26] R. M. Alexander. The gaits of bipedal and quadrupedal animals. The International Journal of Robotics Research, 1984.
[27] W. Yu. C. V. Kumar, G. Turk, and C. K. Liu. Sim-to-real transfer for biped locomotion. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems. IEEE, 2019.
[28] S. Gangapurwala, M. Geisert, R. Orsolino, M. Fallon, and I. Havoutis. Real-time trajectory adaptation for quadrupedal locomotion using deep reinforcement learning. In International Conference on Robotics and Automation (ICRA), 2021.
[29] W. Yu, J. Tan, C. K. Liu, and G. Turk. Preparing for the unknown: Learning universal policy with online system identification. In Robotics: Science and Systems, 2017.
[30] Z. Manchester and Kuindersma. Variational contact-implicit trajectory optimization. In Robotics Research. Springer, 2020.
[31] M. Posa, C. Cantu, and R. Tedrake. A direct method for trajectory optimization of rigid bodies through contact. The International Journal of Robotics Research, 2014.
[32] I. Mordatch, Z. Popović, and E. Todorov. Contact-invariant optimization for hand manipulation. In Proceedings of the ACM SIGGRAPH/Eunngypphis symposium on computer animation, 2012.
[33] M. Srinivasan and A. Ruina. Computer optimization of minimal biped model discovers walking and running. Nature, 2006.
[34] Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 2020.
[35] X. Da, Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Y. Zhu, B. Babich, and A. Garg. Learning a contact-adaptic controller for robust, efficient legged locomotion. arXiv preprint arXiv:2009. 10019, 2020.
[36] T. Haarnoja, S. Ha. A. Zhou, Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning, arXiv preprint 1rXiv:/812.11 2018.
[37] X. B. Peng. E. Coumans, T. Zhang, T.-W. E. Lee, J. Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 2020.
[38] G. B. Margolis, Chen, K. Paigwar, X. Fu, D. Kim, S. bae Kim, and Agrawal. Learning to jump from pixels. In 5th Annual Conference on Robot Learning (CoRL), 2021.
[39] W. Yu, G. Turk, and C. K. Liu. Learning symmetric and low-energy locomotion. ACM Transactions on Graphics (TOG), 2018.
10


Reviews:
[{"review_id": "lYXZKMgaDq5", "reviewer": "Reviewer_hrLb", "summary": "The paper explores the emergence of different quadruped gaits using reinforcement learning, by maximizing a forward reward and minimizing a lateral movement/rotation and an energy cost.\nThe agent is trained in simulation with domain randomization, and successfully deployed in the real world using Rapid Motor Adaptation (RMA).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\nThe paper shows that several quadruped gaits can be successfully learned in simulation and then deployed in the real world  by maximizing a forward reward and minimizing a lateral movement/rotation and an energy cost.\nThe obtained gaits have better energy efficiency than their MPC counterpart for different target speed.\nThe analysis shows that other quantities (for instance continuity cost of the torque) are implicitly minimized thanks to the energy term.\n\nCons:\nThe main issue with this paper is the unclear contribution with respect to Rapid Motor Adaptation (RMA), which is only briefly mentioned in section 2.5.\nIn fact, the current paper can be summarized as RMA minus the additional auxiliary rewards (action magnitude, smoothness, foot slip, ...).\n\nThe rest of the experimental setting is the same, and the robustness and sim2real transfer can be mainly attributed to RMA methodology:\n- domain randomization\n- domain adaption (RMA)\n- fractal environment\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Despite good real world results, as the contribution with respect to RMA is unclear, I would recommend this paper for rejection, but I'm willing to improve my rating upon clarification by the authors.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The authors must clarify the contribution of the paper with respect to RMA.\n\nThe claim that minimizing energy consumption is sufficient for the emergence of natural locomotion gaits is also an overstatement given the experimental details from the paper.\nIt is true that is plays an important regularizer role, but it is not sufficient, as the task reward is encoded in the forward reward + lateral movement/rotation cost.\n\nThe result of the Froude number analysis is unclear: it is true that the learned gaits are closest to the sheep, but what can we do with such information?"}}, {"review_id": "b2K9M1geX4r", "reviewer": "Reviewer_CQpm", "summary": "This paper presents an analysis that with simple energy minimization and training on fractal terrain can lead to different gaits at different target speeds. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n1. Demonstrate that a simple reward (compared to other work that use similar control structure) is able to generate natural gaits that can work well on the real robots.\n\n2. Froude number analysis with number compared with real animals.\n\nCon:\n\nNo major flaw. See more comments below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The idea in the paper is straightforward and the paper is well written. The analysis of gait, especially compared to those of animals' using Froude number is interesting.\n\nHowever, similar result (gait emergence with simple energy minimization) is also demonstrated in Learning Symmetric and Low-Energy Locomotion, by Yu, et al. Although they did not give the throughout analysis of different gait patterns as provided by this paper.\n\nAnother impressive part of the paper is the robustness of the controller, although that mainly benefits from the approach in prior work, i.e.,  Learning quadrupedal locomotion over challenging terrain by Lee et al,  and Rma: Rapid motor adaptation for legged robots, by Kumar et al, and is not a contribution of this paper.\n\nOverall, this paper presents a simple approach to train policies for different locomotion gaits for the A1 robot and can be valuable to the community, although the contribution is a bit incremental.\n\n\n\n\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Questions/Concerns:\n\n1. The claim \"Show that minimizing energy consumption is sufficient for the emergence of natural locomotion patterns\" is not true, as mentioned in the paper, fractal terrain during training is also necessary.\n\n2. How repeatable are the experiments? e.g., if I train with 10 different random seeds, will all seeds give us the same gait for the same speed?\n\n3. Can we see what the gaits trained on flat ground looks like in the video? I can't get any information from Figure 8 alone.\n\n4. It will be interesting to see the transition between different speeds.\n\n5. Will the authors open source the code? Open source implementation will greatly benefit the community, especially given the affordable price of the robots."}}, {"review_id": "S7lOs7TmpRW", "reviewer": "Reviewer_gEhV", "summary": "This paper explores a hypothesis of gait emergence on a quadruped robot that energy minimization leads to different natural gait patterns such as walking, trot and bouncing gaits depending on the given target walking speed. First, the locomotion policy, which is a mapping from the current state and the previous action represented using a three-layer perceptron, is optimized via model-free reinforcement learning in numerical simulations using PPO. In order to obtain a robust gait policy on a flat surface, the policy is trained on uneven terrains instead of adding simulation noise. A simple reward function is used consisting of the terms associated with the forward velocity and the posture of the robot, penalizing energy consumption, and ground impact. Then, the learned policy in numerical simulations is transferred to a real quadruped robot using RMA. It is demonstrated that a robust walking policy can be obtained without any pre-programming or pre-defined movements for leg motions using the proposed framework on the physical robot. Furthermore, empirically, it is shown that different natural locomotion patterns can emerge depending on the speed of the gait as a consequence of energy minimization. Comparison of the efficiency of gaits obtained in the literature illustrates suggests the advantage of the proposed model-free approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThis paper is clearly written and easy to follow. As the supplementary video shows, the obtained policy works remarkably well in the real quadruped robot demonstrating the robustness against perturbations and adaptability to uneven terrain in real environments. In addition, the hardware experiments well-demonstrate the hypothesis raised in this work from an empirical point of view through the proposed reinforcement learning-based optimization approach. The resultant performance is impressive given that the locomotion policy can be obtained via model-free reinforcement learning without any pre-programming or pre-defined leg movements out of a simple energy minimizing reward, or any real world fine-tuning in transferring trained policies from simulations to real environments. \n\nWeakness:\nIndeed, the successful application of reinforcement learning to train locomotion policy in generating natural different gaits on a real quadruped robot via energy minimization is highly impressive. First, the idea of minimizing energy for gait discovery has been previously explored in the case of a simplified biped model in the paper [a] listed below. The finding in this study [a] is that different gait patterns such as inverted-pendulum walk and bouncing run depending on the speed of walking can be found via energy minimization by numerically solving an optimal control problem using sequential quadratic programming (SQP). Although there is a difference in the complexity of the problem settings and optimization approaches (either model-based or model-free), the underlying idea seems to be conceptually similar. \n\n[a] Manoj Srinivasan and Andy Ruina, Computer optimization of a minimal biped model discovers walking and running, Vol. 439, 2006, doi:10.1038/nature04113\n\nSecond, it is not very clear from the description of the paper how the authors came to choose the learning architecture and the reward function. There is an impression that the proposed framework is somewhat a combination of existing methods. It would be quite surprising to see a relatively simple RL framework with a choice of an energy minimization reward could result in such a complex behavior of the robot in a high-dimensional setting without providing any pre-programming or pre-determined leg movements.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is clearly written and well organized. The results with a real robot are very impressive while some of the technical details could be elaborated for the interest of the potential readers and the robot learning community. The paper could be further improved by discussing the motivation and the reason how the authors came to choose the proposed learning architecture, and the actual amount of manual effort in order to make the learning framework successful in such a complex system.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "The paper could be more informative if additional technical details that make successful implementation of the learning framework could be provided in the main text of the paper by addressing the following points:\n\n1. How was the reward designed? How did the authors choose each term in the reward function and how were the weights tuned? \n2. How was the neural network structure (3-layer perceptron with 128 units) chosen for policy representation? The reviewer wonders whether authors explored any other choices for function approximators. \n3. How was the training initialized? \n4. In order to demonstrate the learning performance, it is suggested including a learning curve for multiple runs. Was learning always converge and robust?\n5. It would be more informative if the reason why the authors did not choose domain randomization in this study.\n6. Providing discussions on the advantage of the proposed model-free method over an MPC approach in terms of resultant efficiency would be more informative.\n7. Providing detailed discussions on the technical differences in the approaches in comparison to the related work in Section 5 would be more informative.\n8. It would be more interesting to see qualitative comparisons of the results to biomechanical studies of animal locomotion from a scientific aspect.\n"}}, {"review_id": "B27dC8QJS7n", "reviewer": "Reviewer_h727", "summary": "This paper presents a energy-based end-to-end reinforcement learning approach to realize natural locomotion gaits at different speeds in real quadruped robots. The authors show empirically that energy minimization criterion is enough to realize structured locomotion gaits at different speeds. The gaits show different behaviors for different speeds and match the behavior and energy profile of animals in nature. Finally, the authors validate their approach in both simulation and hardware showing effective speed tracking and robustness against challenging terrains.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-Sometimes less is better. A simple approach such as only using energy consumption is shown to be enough to render natural locomotion gaits without using hand-coded sequences or reference trajectories. \n-The method presents an end-to-end model-free reinforcement learning approach that is simple but effective and does not require additional dynamic model information of the robots, which makes the approach general for different quadruped robots. \n\nWeaknesses\n-It is hard to differentiate where the robustness of the method comes from since ARM is used on top of the proposed method to realize effective sim-to-real transfer. \n-The claim that energy minimization leads to the emergence of structured locomotion gait patterns in flat terrains and unstructured gaits in complex terrains may be imprecise and hard to demonstrate.  In particular, because additional rewards are added for uneven terrain with unstructured gaits and more aggressive environment randomization limits. \n-Different training sessions are needed for tracking different speed and therefore to emerge different behaviors.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper presents  good results in terms of realizing energy efficient gaits in both simulation and hardware, it is hard to identify the actual reason for the success of the method, specially in terms of the robustness and sim-to-real transfer. Also, there are some important observations in the contributions, as some of them may not really correspond to the implementation of the proposed method but an addition of a previous method on top of the proposed one. \nI would recommend to include the desired velocity as part of the state so that only one trained policy can realize different behaviors for walking at different speeds.\n\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Bullet 1 and 4 of the main contributions are basically the same.\n- Bullet 3 in main contributions is not really a contribution of this paper, but the result of ARM [17].\n- Why extra penalties are added when doing uneven terrain with unstructured gaits?\n\nMinor comments:\n- According to section 1 in the appendix, the simulation time step is 0.025 seconds while the controller frequency is 100 Hz, which means the controller time step is 0,01 seconds. This results in the control actions being faster than the simulation itself, which adds unnecessary computational cost to the controller. \n- There is an error in the dimension of the state space in the appendix. It says 32 when it is 30.\n- Move Table 2 to page 6.\n"}}]

Author rebuttal:

Meta-review: This paper proposes to train locomotion controllers using reinforcement learning with a simple reward function for energy efficiency. Different locomotion gaits emerge automatically at different locomotion speeds, which is consistent with the findings in animal motor control. All reviewers appreciate the impressive robot experiments. The rebuttal cleared the original concern about its delta from the RMA paper. The revision of the paper and new experiments have sufficiently addressed reviewers' questions about details of the learning setup and the repeatability of experiments. In addition, the new results on gait transitions further improved the quality of the paper.
Decision: Accept (Poster)