Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior
Authors: Gabriel B. Margolis
Pulkit Agrawal
Venue: CoRL 2022 Oral (2022)
Keywords: Locomotion
Reinforcement Learning
Task Specification

Abstract: Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack a mechanism for fast tuning when they fail in an out-of-distribution test environment. This necessitates a slow and iterative cycle of reward and environment redesign to achieve good performance on a new task. As an alternative, we propose learning a single policy that encodes a structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB). Different strategies generalize differently and can be chosen in real-time for new tasks or environments, bypassing the need for time-consuming retraining. We release a fast, robust open-source MoB locomotion controller, Walk These Ways, that can execute diverse gaits with variable footswing, posture, and speed, unlocking diverse downstream tasks: crouching, hopping, high-speed running, stair traversal, bracing against shoves, rhythmic dance, and more. Video and code release: https://gmargo11.github.io/walk-these-ways

Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior
Gabriel B. Margolis Pulkit Agrawal
Improbable AI Lab, Massachusetts Institute of Technology (gmargo, malkitag/emit.eee
2
G
G
Galt
Sprintin
Stalr Traversal
Bracing Against Shives
Climbingg CuerCurba
Payloed Manipulation
Crouching Obsticies
Rhythmic Dance
Figure I: Multiplicity Behavior (MoB) enables human to tune single quadruped policy trained on flat ground to diverse unseen environments. Top row: A low-frequency gait fails sprint on slippery terrain (Gait 2; inset) but tuning it to high frequency results in success (Gait 1). However, low frequency and high footswing height are necessary for stair traversal (Gait 2; middle image). A low footswing and wide stance (Gait 3) makes the robot robust to leg shoves, but Gait I, which succeeded at sprinting, fails. Tuning gait thus aids in generalizing different tasks. Bottom row: Examples of other behaviors enabled by our controller.
Abstract: Learned locomotion policies can rapidly adapt to diverse environments similar to those experienced during training but lack mechanism for fast tun- ing when they fail an out-of-distribution test environment. This necessitates slow and iterative cycle of reward and environment redesign to achieve good performance on a new task. As an alternative, we propose learning a single pol- icy that encodes structured family of locomotion strategies that solve training tasks in different ways, resulting in Multiplicity of Behavior (MoB). Different strategies generalize differently and can be chosen in real-time for new tasks environments, bypassing the need for time-consuming retraining. We release a fast, robust open-source MoB locomotion controller, Walk These Ways, that execute diverse gaits with variable footswing, posture, and speed, unlocking di- verse downstream tasks: crouching, hopping, high-speed running, stair traver- sal, bracing against shoves, rhythmic dance, and more. Video and code release: https::/ /ttps:///gmargol11 github. io/ walk-these-ways.
Keywords: Locomotion, Reinforcement Learning, Task Specification
Introduction
Recent works have established that quadruped locomotion controllers trained with reinforcement learning in simulation can successfully be transferred to traverse challenging natural terrains [1, 2,
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

3]. Adaptation to diverse terrains is accomplished by estimating terrain properties from sensory observations that are then used by the controller (i.e., online system identification). The success of this paradigm relies on two assumptions: priori modeling of environment parameters that can vary during deployment and the ability to estimate these parameters from sensory observations. To bypass the first assumption, one possibility is to widely vary a large set of environment parameters during training. However, this creates a hard learning problem due to creation of challenging or infeasible locomotion scenarios. To simplify learning, typically the designer chooses a subset of parameters that are randomized in a carefully restricted range. Even in this setup, additional measures such as a learning curriculum and reward shaping are necessary for successful learning in simulation.
As a result of these practical restrictions on the expressiveness of the simulation, quite often the robot encounters scenarios during deployment that were not modeled during training. For instance, if the robot only presented with flat ground and terrain geometry not varied during training, it may fail to traverse non-flat terrains such stairs. In such a case, it is common to tweak the training environments or the reward functions and re-train the policy. This iterative loop of re- training and real-world testing is tedious. To make things worse, in some scenarios such iteration is insufficient because it is not possible to accurately model or sense important environment properties. For example, thick bushes are both hard to simulate due to compliance and hard to sense because depth sensors do not distinguish them from walls. Thus, the robot may attempt to climb over thick bushes like a rock move through them with an overly conservative gait that leaves the robot stuck.
The examples above illustrate that even for the most advanced sim-to-real systems, the real world offers new challenges. We broadly refer to scenarios that can be simulated but are not anticipated during training and the situations which cannot be simulated or identified from sensory observations as out-of- dit-o---distiibtiion cases. We present a framework for policy learning that enables improved per- formance in out-of-distribution scenarios under some assumptions detailed below. Our key insight is that given task, there are multiple equally good solutions (i.e., under-specification [4]) that have equivalent training performance but can generalize different ways. For instance, the task of walk- ing on flat ground only imposes a constraint on the velocity of robot's body, but not on how the legs should move, or high should the torso be above the ground, etc. Consider two different walking behaviors: crouch where the robot keeps its torso close to the ground and stomp where the torso is high and also the legs have a high foot swing. While both crouch and stomp succeed at walking on flat ground, their generalization to out-of-distributtin scenarios is different: with crouch the robot can traverse under obstacles but not stairs, whereas with stomp it can climb over curbs/stairs but not move under obstacles (Figure 1).
Out of the many possible locomotion behaviors that succeed in the training environment, typical reinforcement learning formulations result in policy that only embodies one solution and therefore expresses a single generalizing bias. To facilitate generalization to diverse scenarios, we propose a technique, Multiplicity of Behavior (MoB), that given the same observation history and small set of behavior parameters outputs different walking behaviors. When faced with an unseen scenario, one can test different behaviors by varying these parameters, which affords much quicker iteration than re-training a new policy and facilitates collection of online demonstrations by a human pilot.
The utility of MoB depends on the assumption that some subset of behaviors successful in the restricted training environment will also succeed in the out-of-distribution target environment. To demonstrate that this is true in meaningful sense, we chose an extreme case: train single policy for quadrupedal walking only on flat ground and evaluate on non-flat terrains and new tasks. We show that human operator can tune behaviors in real-time to enable successful locomotion in the presence of diverse unseen terrains and dynamics including uneven ground, stairs, external shoves, and constrained spaces (Figure 1). The same tuning mechanisms can be used to compose behaviors to perform new tasks such as payload manipulation and rhythmic dance (Figure 1).
This work contributes a robust low-level quadruped controller that can execute diverse structured behaviors, which we we hope will be a useful building block for future locomotion applications. can serve as a platform for collecting quadruped demonstrations for diverse tasks which is enabled by an interpretable high-level control interface. Furthermore, our controller showcases MoB as a practical tool for out-of -it-of--distibution generalization. While our implementation of MoB leverages expert knowledge in the domain of locomotion, in general, the technique of learning multiple meth- ods of achieving goals to facilitate generalization is a promising approach with potential for broad application in sim-to-real reinforcement learning for robotics.
2

---

Figure 2: A learned controller transitions between classical structured gaits: trotting, pronking, pac- ing, and bounding in place at alternating frequencies 2Hz and 4Hz. Images show the robot achieving contact phases of each gait, with stance feet highlighted in red. Black shading in the bottom plot re- flects the timing reference variables t, for each foot; colored bars report the contact states measured by foot sensors. As we later expose, diverse behaviors can facilitate novel downstream capabilities.
2 Background
Auxiliary Rewards. Locomotion gaits learned using only the task reward (e.g. velocity tracking) have not been shown to successfully transfer the real world. is necessary train using auxiliary rewards that bias the robot to maintain a particular contact schedule, action smoothness, energy consumption, or foot clearance [1, 2, 3,5,6 6, 7]. to compensate for the sim-to-real gap. Such auxiliary rewards can be interpreted as biases for generalization. For example, foot clearance enables the robot to be robust when the terrain in the real world more uneven, or the robot's body sinks more in the real world than in simulation [7]. If an agent fails on a real-world task, it is common practice to manually tune auxiliary rewards to encourage the emergence of successful real-world behavior. However, such tuning is tedious because it requires repeated iterations of training and deployment. Furthermore, such tuning is task-specific and must be repeated for new tasks commanded to the agent. The difficulty of designing single set of auxiliary rewards that promote generalization in diverse set of downstream tasks is illustrated in the top row insets of Figure I: each shows an instance where a fixed auxiliary reward would lead to failure for one task, despite working well for another.
Learning Locomotion with Parameterized Style. Closely related recent work on bipeds has ex- plicitly included gait parameters in the task specification through reward terms, including tracking rewards for contact patterns [8] and target foot placements [9]. [8] used gait- depeedennn reward terms in bipedal locomotion to control the timing offset between the two feet and the duration the swing phase. The reward structure of [8] inspired ours, but due to its small space of command parameters, did not propose and would not support application to com- positional tasks or -uttr--distribution environments. To similar effect, other works have imposed parameterized style through libraries of reference gaits. [10, 11] generated a library of reference trajectories and trained a goal- goal-conditioned policy to imitate them. [12] demonstrated that a small discrete set of motion styles can be learned simultancously from reference trajectory library.
auxiliary
Learning with Diversity Objectives. Several prior methods aim to automatically learn a collection of
and diverse behaviors. Quality diversity (QD) methods [13, 14, 15] learn a li- brary of diverse policies by enforcing a novelty objective defined among trajectories. They typically perform this optimization using evolutionary strategies. QD has demonstrated benefits including improved optimization performance and reuse skills for online adaptation [13, 14). Another line of work uses unsupervised objectives for skill discovery in RI, towards improving optimization [16] or out-of- oi-diitribbttin generalization [17, 18]. Unsupervised diversity approaches hold promise, but they have not yet scaled to real robotic platforms and do not produce an grounded interface of pa- rameters for guiding behavior.
Hierarchical Control Using Gait Parameters. Several works have learned high-level policy to accomplish downstream tasks by modulating the gait parameters low-level masel-based controller. This approach has been previously applied to energy minimization [19] and vision- guided foot placement [20, 21]. These works relied on model-predictive low-level controller to execute the different gaits in absence of gait-conditioned learned controller. The tradeoffs between model-based control and reinforcement learning are well discussed in prior literature [3, 22]. Our work enables revisiting hierarchical approaches with learning at both the high and low levels.

---

Term
Equation
Weight
If lag
velocily tracking
expl
0.02
Task
"Gunne" yaw velocity tracking
expl
0.01
t ( Londd swing phase tracking (force)
Lmel
Onae (B
expp
0.08
Tenad stance phase tracking (velocity)
Soal [Ciae (0 43] exp
-0.08!
"werss body height tracking
Aa )a
-0.21
Augmented Auxiliary
"geme: body pitch tracking
(
-0.1
"agua" raibert heuristic footswing tracking
you (s 1)²
0.2
Va Soft footswing height tracking
(h
h Crad
0.6,
Fixed Auxiliary
iz velocity
16-14
coll-pitch velocity
Wxy
2e-51
slip
Yay
thigh/ealf collision
Lonton
-000
joant limit violation
1 2mmma Ceoai
-0.21
joint tomues
20-51
foint velocities
2
joint accelerations
5e-9
action smoothing
2e-31
action smoothing. 2nd order
a--2
2e-3
Table 1: Reward structure: task rewards, augmented auxiliary rewards, and fixed auxiliary rewards.
3
Method
To obtain MoB, we train conditional policy (-|cc, b,) that achieves tasks specified by the com- mand (ct) in multiple ways that result from different choices of behavior parameters, b,. The ques- tion arises of how to define b,. We could learn behaviors using an unsupervised diversity metric, but these behaviors might not be useful [4] and are not human tunable. To overcome these issues, we leverage human intuition about useful behavior parameters (b,) corresponding to gait properties like foot swing motion, body posture, and contact schedule [6, 7, 8, 23]. During training, the agent receives a combination of task rewards (for velocity tracking), fixed auxiliary rewards (to promote sim-to-real transfer and stable motion), and finally augmented auxiliary rewards (that encourage lo- comotion in the desired style). During deployment in a novel environment, a human operator can tune behavior of the policy by changing its input b;.
3.1 Task Structure for MoB
Task Specification. We consider the task of omnidirectional velocity tracking. This task is specified by 3-dimensional command vector [v vend, ( where vend veni are the desired linear velocities in the body-frame and y- axes, and w is the desired angular velocity in the yaw axis.
Behavior Specification. We parameterize the style of task completion by an s-dimensional vector of behavior parameters, b,:
by T @
gemd
cmd
emd
sy h fcmd]
amd
a m0 Oomd 0g are the timing offsets between pairs of feet. These express gaits includ- ing pronking (gamd (0.0, O, 0)), trotting (@amd (0.5, O, 0)), bounding, (6emd (0, 0.5, 0)), pacing (0omd 0.,5, well as their continuous interpolations such as galloping (@amd (0.25, 0, 0)). Taken together, the parameters gemd can express all two-beat quadrupedal contact pat- terns; Figure provides a visual illustration. femal the stepping frequency expressed in Hz. As an example, commanding gomd = 3 Hz will result in each foot making contact three times per second. h. is the body height command; & gond is the body pitch command. ggmdl is the foot stance width command: fcnd cmd is the footswing height command.
Reward function. All reward terms are listed in Table 1. Task rewards for body velocity track- ing are defined as functions of the command vector e,, Auxiliary rewards are used constrain the

---

quadruped's 's motion for various reasons. "Fixed" auxiliary rewards are independent of behavior pa- rameters (b,) and encourage stability and smoothness across all gaits for better sim-to-real transfer. During training, one concern is that the robot might abandon its task or choose an early termination when the task reward is overwheimed by penalties from the auxiliary objectives. To resolve this, as in [7], we force the total reward to be a positive linear function of the task reward by computing it as Trask exp (CauxTauxx where Ttask is the sum of (positive) task reward terms and raux is the sum of (negative) auxiliary reward terms (we use Caux = 0.02). This way, the agent is always rewarded for progress towards the task, more when auxiliary objectives are satisfied and less when they are not.
For MoB, we define augmented auxiliary rewards as functions of the behavior vector b, We de- signed these rewards to increase when the realized behavior matches b, and to not conflict with the task reward. This required some careful design of the reward structure. For example, when implementing stance width as behavior parameter, naive approach would be to simply reward a constant desired distance between left and right feet. However, this penalizes the robot during fast turning tasks requiring relative lateral motion of the feet. To avoid this, we implement the Raibert Heuristic, which suggests the necessary kinematic motion of the feet to achieve a particular body velocity and contact schedule [23, 24]. The Raibert Heuristic computes the desired foot position in the ground plane. Play quul( (s5mm)) ), as an adjustment to the baseline stance width to make it consistent with the desired contact schedule and body velocity. To define the desired contact schedule, function Cooo (@emd, t) computes the desired contact state of each foot from the phase and timing variable, as described in [8], with details given in the appendix.
3.2 Learning Diversified Locomotion
Task and Behavior Sampling In order to learn graceful online transitions between behaviors, we resample the desired task and behavior within each training episode. To enable the robot to both run and spin fast, we sample task c2 (v yend ( using the grid adaptive curriculum strategy
from [3]. Then, we need to sample target behavior Dy. First, we sample (0TTh 0 : 00gma) as one of the symmetric quadrupedal contact patterns (pronking, trotting, bounding, or pacing) which known more stable and which we found sufficient basis for diverse useful gaits. Then, the remaining command parameters (vgmd good h.nd o s h and uniformly. Their ranges are given in Table 6.
/emd .sg sqmdd) are sampled independently
Policy Input. The input to the policy is 30-step history of observations Oz-H...t: commands C. behaviors b.- previous actions a.- and timing reference variables H.. The observation space consists of joint positions and velocities q, 4, (measured by joint en- coders) and the gravity vector in the body frame g, (measured by accelerometer). The timing refer- ence variables t |sin(2
sin R2 (Rn gond 0gmdd gqmd gemd .t + gomd I + 0gmd), where f is a counter variable that advances from 0 to 1 during each gait cycle and FR FL RR RL are
are computed from the offset
timings of each foot: [1 FFL ,RR jRL]-
the four feet. This form is adapted from [8] to express quadrupedal gaits.
Policy Architecture. Our policy body is an MLP with hidden layer sizes [512, 256 128] and ELU activations. Besides the above, the policy input also includes estimated domain parameters: the velocity of the robot body and the ground friction, which are predicted from the observation history using supervised learning in the manner of [7]. The estimator module is an MLP with hidden layer sizes [256, 128) and ELU activations. We did not analyze the impact of this estimation on performance but found it useful for visualizing deployments.
Action Space. The action a, consists of position targets for each of the twelve joints. A zero action corresponds to the nominal joint position, q. The position targets are tracked using a proportional- derivative controller with kip = 20, ka = 0.5.
3.3 Design Choices for Sim-to-Real Transfer
Domain Randomization. For better sim-to-real transfer, we train policy that is robust to range of robot's body mass, motor strength, joint position calibration, the ground friction and restitution, and the orientation and magnitude of gravity. As we are interested in studying out-of -ii-rii-iitiittionn generalization, we only train on flat ground without any randomization of terrain geometry. This choice also simplified training. The randomization ranges of all parameters are in Appendix A.
5

---

Power Consumption
Gait
m/
1.0m/s
2.0m//s
m/0
Trotting
Trotting
9+1
2441
532s
98
120
Pronking
Pronking
3211
4312
68
1124s
Facing
Pacing
1343
2542
55ga
9946
Consumption (18)
100
Bounding
Bounding
22+2
39+4
78+6
127+a5
Gait-free Baseline
1743
35
64
142
80
Trotring (fams Hxx)
11+2
25+1
55
104+s
60
Trotring (fans 3 Hx)
9+1
24+1
63-+6
98++9
Trotinn (fum 4 Hx)
941
2640
60
114
40
Trotinn (hema 20 cm)
941
26
5623
10248
Trotring ind
cm)
94:
24
53
98.1
Trotting (hem
40 cm)
10+1
23+1
62:
95.49
0.0
0.5
1.0
1.5
2.0
2.5
3.0
1gth
Table 3: Behavior tuning enables interventional studies the relationship between gait proper- ties and performance criteria within single policy. Here, we illustrate how power consumption varies across speeds for common quadrupedal gaits and for baseline policy without gait constraint. Several structured gaits surpass the efficiency of unconstrained gait across all speeds.
Latency and Actuator Modeling. Directly identifying invariant properties avoids overly conserva- tive behavior from unnecessary domain randomization. We perform system identification to reduce the sim-to-real gap in the robot dynamics. Following [22], we train actuator network to capture the non-ideal relationship between PD error and realized torque. Separately, we identify latency of around 20 ms in system and model this constant action delay during simulation.
3.4 Materials
Simulator and Learning Algorithm. We define training environment in the Isaac Gym simu- lator [6, 25J. We train policies using Proximal Policy Optimization [26]; details in Appendix A.
Hardware. We deploy our controller in the real world on the Unitree Gol Edu robot [27]. An onboard Jetson TX2 NX computer runs our trained policy. We implement an interface based on Lightweight Communications and Marshalling (LCM) [28] to pass sensor data, motor commands, and joystick state between our code and the low-level control SDK provided by Unitree. For both training and deployment, the control frequency is 50Hz.
Gait-free Baseline. To understand the impact of MoB on performance, we compare our controller to a baseline velocity-trackipg controller (the "gait-free baseline")). The gait-free baseline is trained by the method above, but excludes all augmented auxiliary rewards (Table 1). Therefore, only learns one solution to the training environment and its actions are independent of behavior parameters b,.
4 Experimental Results
4.1 Sim-to-Real Transfer and Gait Switching
We deploy the controller learned in simulation in the real world and first evaluate its performance on flat ground similar to the training environment. To start, we demonstrate generating and switching between structured gaits that are well-known in the locomotion community. Figure shows torques and contact states during transition between trotting, pronking, bounding, and pacing while alter- nating famul between 2 Hz and 4 Hz. We find that all gait parameters are consistently tracked after sim-to-real transfer. Videos (i)-(iv) on the project website visualize the different gaits obtained by modulating each parameter in b, individually.
4.2 Leveraging MoB for Generalization
Tuning for New Tasks. After training using generic locomotion objective, one might wish to tune a controller's behavior optimize new metric in the original environment. MoB facilitates this some subset of learned behaviors outperform the gait-free policy by the new task metric.

---

Gait
Vornd
Tware
Vegnt
Survival
Trotting
0.98
0.88
Pronking
0.
0.96
0.97(0.96)
0.82
Pacing
0.76
0.76
0.94
(0.96)
0.98(0.90)
0.87
Bounding
0.80
0.7310.01
0.04100.00
0.98
10.00
0.82
Gait-free
0.81
(0.96)
0.7440..06
0.83
20.03
Table 4: Zero-shot generalization to platform terrain (visualized right). Pacing and trotting yield the best survival time in out-of-distribution deployment, outperforming the gait-free baseline. Pronk- ing attains the best velocity tracking performance, with similar survival time to the baseline. We report the fraction of maximum episodic reward. Superscript reports performance in flat training environment with platforms. Subscript reports standard deviation across three random sceds.
Energy efficiency (simulated): We consider the task of minimizing the mechanical power consump- tion (J/s) measured by summing the product of joint velocity and torque at each of the 12 motors: Σ; max(T 0). As shown in Figure 3, several choices of the contact schedule gemd height h and frequency found outperform the gait-free policy in this unseen metric, consuming less energy facilitate tuning performance
across all speeds. Therefore, tuning the behavior parameters new objective (energy efficiency) in the original training environment.
Payload manipulation: We experiment with another task where the robot is required to transport a ball from one place to another, then bend its body so the ball is deposited on the ground. gait- free policy couldn't 't do this; while many possible body posture profiles are valid to solve the training environment, the gait-free policy will simply converge to one at random. In a real- world experiment, we demonstrate how MoB-enables body posture control can be repurposed for teleoperated payload manipulation. The operator pilots the robot to the delivery location with the body level, then pitches the body backward, modulating &f amd to dump the payload (Figure 1, bottom row, second from left).
Tuning for New Environments. In the previous section, we showed that MoB can support repur- posement to novel tasks in the training environment. In the real world, there is also always a long tail of environments that are not modeled in training. We demonstrate the behavior of our controller trained only on flat ground in the presence of challenging non-flat terrains and disturbances such as curbs, bushes, hanging obstacles, and shoves. Successes suggest that the space of policies learned by MoB contains some behaviors that transfer better than the baseline to unseen environments.
Platform terrain (simulated): We evaluate the performance of our robot in traversing randomly positioned platforms with height up 16 (Table 4). We report two metries: mean reward and mean survival time as a fraction of the maximum episode length (10s). For each metric, we find that by modulating the contact schedule grma or footswing height h f we can outperform the gait-free policy. (Figure 4, Appendix Figure 9). Therefore, it is possible improve performance in an out-of distiiibbbttiinn terrain by modulating the parameters of the MoB policy.
Climbing over curbs: Previous works demonstrating blind obstacle traversal on quadruped [1. 2] learned a foot-trapping reflex where the robot first trips, then raises its foot over the obstacle. In contrast, with the help of human pilot, our gait-conditioned policy with high footswing command enables fast and smooth obstacle traversal without tripping, despite training on a simpler flat-ground terrain. In the real world, we demonstrate that modulating footswing height h cmd enables our controller to climb smoothly across stairs and curbs (Figure 1, bottom left).
Hacking through thick bushes: Extremely thick bushes pose a methodological challenge for state-of- the-art perceptive locomotion controllers. They are hard to simulate, because they comply against the whole body of the robot, and they are hard to sense as they are indistinguishable from solid obstacles to depth sensors. Therefore, prior works would either attempt to climb over bushes as obstacles or fall back on a robust proprioceptive controller that is unaware of the semantic context. In contrast, by modulating the gait parameters for gait frequency fond and footswing height h.omd in situ, human operator can guide our system quickly through challenging brush
Navigating confined spaces: Consider the scenario where the robot needs to go under a bar. The gait-free baseline cannot accomplish this; in the absence of such constraints during training, it will
7

---

We demonstrate that behavior transitions can be performed even in quick sequence at high speed for synthesis of agile maneuvers. Emulating gap crossing on flat ground, we show that this can facilitate crossing a gap wider than the robot's body length in a single leap.
Figure 3:
converge to fixed body height profile. Our system with MoB can navigate confined spaces through modulation of the body height and In real-world example, the robot was able crawl under 22 em bar; the robot body thickness is 13 cm, leaving 9 cm of clearance beneath the robot.
Anticipative bracing against shoves: Widening the stance can make quadruped more robust to hard shoves, but absent perception module to anticipate human kick, the robot would always need to walk in widened stance be prepared for shove. This interferes with performance in other tasks like running efficiently, so learned locomotion controllers without MoB often provide incentive to keep the feet nominally below the hips [2, 7]. With MoB, widening the stance width snnd in the pilot's anticipation of shove enables the robot to remain upright.
Exploring the Speed and Range of Behavior Adaptation. If we want to troller for high-level tasks, one thing we might desire is to switch between gaits even at high speeds, which might be useful for applications such as parkour. Another is to transition between diverse gaits with precise timing for a synchronous task like dancing.
low-level con-
Agile forward leap: As a demonstration of gait transitions at high speed, we modulate contact sched- ule, velocity, and gait frequency to encode an agile forward leap (Figure 3). The robot first accel- erates to target speed of 3 m/s at a trot while increasing its step frequency from 2 Hz to 4 Hz, then switches to pronking
Hz for one second, then decelerates the leap phase, the distance from the location of the robot's front feet at takeoff to the location of the hind feet upon landing 60 cm.
standstill while trotting. During
Choreographed dance: To demonstrate precisely timed transition between diverse gaits, we program sequence of gait parameters generate dance routine synchronized a jazz song with tempo of 90 bpm. At this tempo, combinations of phases 0, 0.25, and 0.5 with frequencies of 1.5 Hz and Hz yield eighth, quarter, half, and full beat gaps between consecutive footsteps. We also modulate body height and velocity in time with the music. An assistant script procedurally generates gait parameters, fed into the controller in open-looy fashion.
Discussion and Limitations
Our experiments show that the benefits of adding MoB can come at a cost to in-distribution task performance, specifically limiting the robot's flat-ground sprinting performance (Table 5, appendix). Heat maps reveal that our behavior parameterization is restrictive for combinations of high lincar and angular velocity. To quantify and control the tradeoff between task performance and reward shaping an interesting future direction, for which some prior methods have been proposed [29].
MoB confers a single learned policy structured and controllable space of diverse locomotion be- haviors for each state and task in the training distribution. This yields a set of "knobs' to tune the performance of motor skills in unseen test environments. The system currently requires a human pilot to manually tune its behavior. In the future, the autonomy of our system could be extended by automating behavior selection using imitation from real-world human demonstrations, or by using a hierarchical learning approach to automatically self-tune the controller during deployment.

---

Acknowledgments
We thank the members of the Improbable AI lab for the helpful discussions and feedback on the paper. We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources. This research Program, the MIT-IBM Watson AI Lab, and the National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Inter- actions, http://iaifi.orr /rgg/ This research was also sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and accomplished under Cooperative Agreement Number 1--8770011-2-10000 in this document are those of the authors and should not be interpreted as representing the offi- cial policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
supported by the DARPA Machine Common Sense
The views and conclusions contained
Author Contributions
Gabriel B. Margolis concieved, designed, and implemented the controller, ran all experiments, and played the primary role in paper writing.
Pulkit Agrawal advised the project and contributed to its conceptual development, experimental design, positioning, and writing.
References
[1] J. Lee, J. Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain, Sci. Robot., 5(47):eabc5986, Oct. 2020. doi:10. /26/sccrobooss. abc5986.
[2] A. Kumar, Z. Fu, D. Pathak, and Malik. RMA: Rapid motor adaptation for legged robots. In Proc. Robot.: Sci. and Syst. (RSS), Virtual, July 2021. doi: db::10:88550arx::21000004333
[3] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforce- ment learning. Proc. Robot:: Sci. and Syst. (RSS), June 2022. 000110448550/aaxX1222550000 02824..
[4] P. Agrawal. The task specification problem. In Conference Robot Learning, pages 1745- 1751. PMLR, 2022.
[5] Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emer- gence of gaits in legged robots. In Proc. Conf. Robot Learn. (CoRL), pages 928 937, London, UK, Nov. 2021. 11.:0044850//xxx22111..0011
[6] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Pruc. Conf. Robot Learn. (CoRL), pages 91-100, London, UK, Nov. 2021. doi: 10.485500arxivv210999
[7] G. Ji, Mun, H. Kim, and J. Hwangbo. Concurrent training of control policy and state estimator for dynamic and robust legged locomotion. IEEE Robot. Automat. Lett. (RA-L), 7 (2):4630-a - 4637, Apr. 2022. 1o 101.0000....202 2.291..RA.002211513339
[8] J. Siekmann, Y. Godse, A. Fern, and J. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In Proc. IEEE Int. Conf. Robot. Automat. (ICRA), pages 7309-7315, Xi' an, China, June 2021. doi: 000.:011100//RA445500.200000 810.4444556.221.56111811
[9] H. Duan, A. Malik, J. Dao, A. Saxena, K. Green, J. Siekmann, A. Fern, and J. Hurst. Sim-to- real learning of Automat. (ICRA), Philadelphia, USA, May 2022. doi: 10.1 10990/CA446999.0222222
bipedal dynamic walking. In Proc. IEEE Int. Conf. Robot.
[10] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 2811-2817. IEEE, 2021.

---

[11] Y. Shao, Y. Jin, X. Liu, W. He, H. Wang, and W. Yang. Learning free gait transition for quadruped robots via phase-guided guuded controller. IEEE Robot. Automat. Lett. (RA-L), 7(2):1230- 1237, 2021.
[12] E. Vollenweider, M. Bjelonic, V. Klemm, N. Rudin, J. Lee, and M. Hutter. Advanced skills through multiple adversarial motion priors in reinforcement learning. arXiv preprint arXiv:2203. 14912, 2022.
[13] J.-B. Mouret and J. Clune. Illuminating search spaces by mapping elites. arXiv preprint 1XXv::554.04909, 2015.
[14] A. Cully, J. Clune. D. Tarapore, and J.-B. Mouret. Robots that can adapt like animals. Nature, 521(7553):503-507 2015.
[15] B. Lim, L. Grillotti, L. Bernasconi, and A. Cully. Dynamics-aware diversiyy for effi cient learning of skill repertoires. In 2022 International Conference on Robotics and Automa- fion (ICRA), pages 5360-5366, IEEE, 2022.
[16] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function, arXiv preprint arXiv: htt:::800200011 2018
[17] S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198-8210. 2020.
[18] J.-B. Gaya, L. Soulier, and L. Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. arXiv preprint arXiv:2110.05169, 2021.
[19] Y. Yang, Zhang, E. Coumans, J. Tan, and Boots. Fast and efficient locomotion via learned gait transitions. In Proc. Conf. Robot Learn. ( (CoRL), pages 773-783, Nov. 2022. doi: 10.485500/axxiv22100 04.00644.
[20] G. B. Margolis, T. Chen, K. Paigwar, X. Fu, D. Kim, S. Kim, and P. Agrawal. Learning to jump from pixels. In Proc. Conf. Robot Learn. (CoRL), pages 1025-1034, London, UK, Nov. 2021. doi:10.450 10.48550/arxiv.2111 10.110151444,
[21] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang. Visual-locomotion: Learning to walk on complex terrains with vision. Proc. Conf. Robot Learn. (CoRL), Oct. 2021.
[22] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Sci. Robot., 4(26):aau5772, Jan. 2019. doi: 10.1 1 /2t///iroootics.auu3333
[23] D. Kim, J. Di Carlo, B. Katz, G. Bledt, and S. Kim. Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control. arXiv preprint, 2019. doi: 10. 10.48550/arxiv..b 065886
[24] M. Raibert. Legged Robots That Balance. MIT press, 1986.
[25] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin,
A. Allshire, A. Handa, et al. Isaac Gym: High performance GPU-based physics simulation for robot learning. arXiv preprint, 2021. doi: 10..48550/axii.210000 104400
[26] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint, 2017. 1ot.:/00/8555///rx111000.0
[27] Unitree Robotics, Gol, 2022, http:p/swwwww.iiiaee...cooo///ep wtt.://ww.ee..coo/pppprttttttoop [Online; accessed Jun. 2022].
[28] A. S. Huang, E. Olson, and D. C. Moore. LCM: Lightweight communications and marshalling. In 2010 JEEE/RS/ International Conference Intelligent Robots and Systems, pages 4057- 4062. IEEE, 2010.
[29] E. Chen, H. Zhang-Wee J. Pajarinen, and P. Agrawal. Redeeming intrinsic rewards via con- strained optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
10


Reviews:
[{"review_id": "_E9IPn6-M7O", "reviewer": "Reviewer_E9ME", "summary": "In this paper, the authors propose a new system to train diverse locomotion skills in a single policy. Compared with previous works where only the moving speeds and angular speeds are tracked, the proposed policy architecture also takes into account the preferred gait parameters such as frequency and phase offsets. With a curriculum training procedure, the authors enable the quadruped robot to perform trotting, pacing, pronking, and bounding, all using a single learned policy. The authors zero-shot deploy the policy to a real go-1 robot and show that the gait conditioned policy’s performance is better than gait-free baselines. \n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of this paper:\n\nThe paper is clearly written and easy to follow. The results are comprehensive and the hardware evaluations are rich. The authors measure various aspects including stability and power consumption.\n\nThe weaknesses of this paper:\n\n1, Limited novelty. The delta between this and previous works seem to be mainly the inclusion of gait parameters during the RL training. All other aspects, such as policy architecture, reward shaping, and training curriculums, seem to be a reimplementation of previous works. \n\n2, Limited baselines. There is only a gait-free baseline that the author is compared with. I think it will be great to include other baselines such as these trained with motion imitation methods.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper seems to have limited novelty and little comparisons with enough baselines. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "\nOther issues of the paper, in no particular order:\n\n1, In Table 2, the notation v_x^cmd appears twice.\n\n2, In Table 4, prefer to use the cost of transport over power consumption, since CoT is more standard in locomotion literature.\n\n3, There are also works related to gait transition and learning such as: Fast and Efficient Locomotion via Learned Gait Transition, which is not discussed in the related works. \n\n4, What is the control frequency of the policy?\n\n5, The gait-free baseline. “The pacing and trotting gaits yield the best survival time during zero-shot deployment on this particular terrain, outperforming the gait-free baseline.” This is a bit surprising. Since the gait free baseline with auxiliary rewards pretty much will convert to trotting, and thus I would expect similar (good) stability. \n\n6, In Figure 4, please mark the gait transitions on the gait pattern subfigure to help visualization.  Also, mark the timing of each robot figure on top.\n\n7, The “agile jumping” is hardly a successful jump as I can tell from the video. I think during the training there is no reward to incentivize the jumping height. I would only call this  as a bounding gait with a small aero phase. \n"}}, {"review_id": "CQjfS9yozF", "reviewer": "Reviewer_fCqU", "summary": "This paper presents a framework for learning policies for a quadrupedal robot that are conditioned on different gait parameters. This allows a single policy to generate a diverse set of gaits, including the common trotting, pacing, pronking and bounding. Analysis of the resulting policy shows that different gaits demonstrate different desired characteristics and are used to solve different downstream tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n1. Demonstrate diverse gaits on a quadrupedal robot using a single policy, including successful sim-to-real transfer.\n\nCon:\n1. Motion quality degrades significantly when the desired velocity goes up, e.g., the agile jump sequence, compared to what is demonstrated here: https://sites.google.com/view/jumpingfrompixels, https://www.youtube.com/watch?v=9QV0Nnm80tU. Notably, the feet spread wide apart to gain more stability.\n2. Some more analysis is needed, see issues below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper extends the framework of learning a gait-conditioned policy from Siekmann et al to quadrupedal locomotion. While demonstration on the hardware is appreciated, the sim-to-real success is not surprising given similar demos in the past few years. The system is built upon several prior works with limited algorithmic innovation. Some further analysis of different gaits or more impressive demos that is not achievable by prior work can strengthen the paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. \"No previously proposed reinforcement learning agent has been suitable to accomplish any of these tasks\", (a) long jump: https://openreview.net/pdf?id=R4E8wTUtxdl, (b) through confined space: no work that I am aware of demonstrates this, but I don't think it is difficult to achieve with existing RL method, e.g., train a pacing gait to go through the narrow space (as is done in this paper), or train a crawling gait with low desired body height. One selling point is that one policy can achieve all these tasks, but the result is not surprising given that all necessary components are already available in the existing literature. Some suggestions, can we extend this framework so that it does less common gaits? e.g., two-legged hopping, skateboarding. \n\n2. In Table 6, similar heat maps for other gaits will be helpful information. What is the commanded stepping frequency for this table? I will imagine high-frequency stepping will be helpful to get better high-speed velocity tracking performance.\n\n3.  In Figure 8, what is the unit of the temperature bar? What is the commanded stepping frequency?\n\n4. Some analysis of how the stepping frequency affects performance can be useful, e.g., a high stepping frequency should be able to achieve faster locomotion as well as more robust to perturbation.\n\n5. I think there are typos in Table 2, e.g., there are two $v_{x}^{cmd}$ in the column."}}, {"review_id": "1v1_jmWM42", "reviewer": "Reviewer_kWnF", "summary": "This paper presents a gait controller for quadruped robots where the controller is parameterized by target velocities, feet patterns, stepping frequency, and the trunk height. So the controller can satisfy user-provided target velocity commands while generating various gait styles, which enables users to easily adapt their commands to achieve the desired tasks. More specifically, the controller is learned by deep RL where it is asked for achieving a random command generated by a sampling scheme that is adaptive to the current controller performance. Additionally, domain randomization is included for sim-to-real transfer. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n> The learned controller is robust and shows impressive sim-to-real performance, so it can be deployed in many challenging environments. The gait styles that the controller generates are diverse, which also include some agile motions. By considering these two factors, it has potential to be used as a base controller in many quadruped robot platforms.\n\n> The paper is well-written and the exposition is mostly clear.\n\n\nWeakness\n\n> Not much of technical novelties were presented, where most components came from the existing methods.\n\n> No failure case is explicitly mentioned in neither the paper nor the supplemental material.\n\n\nSuggestion\n\n> Why are the contact sensor inputs not given to the controller? Is there any particular reason for this? Because the command and the reward function requires specific foot patterns, I guess such information would be really helpful for better performance. \n\n> Figure3: it would be better if the horizontal axis could be” # of simulation step” instead of “training iteration” so that readers know how many samples would be necessary for the training.\n\n> The motions generated do not seem energy efficient, can we improve this by adding more weights on it?\n\n> Table6: the figure on the right is not symmetric w.r.t. w_z^{cmd}. Although I don’t think it should be perfectly symmetric due to randomness in nature, it should be approximately symmetric because there exists the adaptive sampling algorithm and the robot is perfectly symmetric at least in the simulation. For example, there is a significant performance difference between (-2.5, -2.5) and (-2.5, 2.5). \n\n> “One might hypothesize that the most energy-efficient gaits are also the easiest to learn, but our results go against this hypothesis (pronking emerges earlier than pacing; see Section 4.1).” -> This might also be due to the balancing problem where pronking or bounding has less balancing issue than the others. \n\n> I don’t find the definition for {\\hat q} in Table1. Is it the previous joint positions?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although there’s not many novel technical components, finding the right combination that would work well in the real world is not a trivial problem. I think that the combination presented in this paper is simple but very powerful enough to be used as a base controller in many existing quadruped platforms.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see “Suggestion” above", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "-N2FxhaMYH2", "reviewer": "Reviewer_hEjf", "summary": "This work presents a new RL-based system that learns a single gait-conditioned policy to produce dynamics locomotion movements of a quadruped robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is easy to follow and as far as the reviewer is aware of, sound.\n- Learning various gaits, with a single policy appears great as an idea.\n- The system is implemented on real robots, demonstrating the sim2real challenge.\n\nWeaknesses:\n- The robot does not perform well in several cases, i.e., there are instances that it just drags the back feet on the ground, which is not optimal control.\n- Is the paper, something more than combining [22], [16], and [25] (and mainly [22])?\n- It is very confusing how energy consumption was measured (Table 4)? Some details are needed.\n\nOverall, not much else the reviewer can comment on this work, as far as a clear statement about the differences with [22] are mentioned and of course Table 4 generation. Other than that, this is a really nice paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "If the novelty compared to [22] (and also [16], [25]) is explained and there is indeed some significant work that has been done (other than 2-to-4 legs), it is a nice paper in the field with real-world demonstrations.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- A clear statement about the differences with [22] are mentioned.\n- Table 4 energy measurement explanation.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)