Learning H-Infinity Locomotion Control
Authors: Junfeng Long
Wenye Yu
Quanyi Li
ZiRui Wang
Dahua Lin
Jiangmiao Pang
Venue: CoRL 2024 (2024)
Keywords: Robot Learning
Quadrupedal Robot
Robust Locomotion
TL;DR: The actor acts, the disturber disturbs, and the H-Infinity constraint helps them enhance the robot's disturbance resistance capability.

Abstract: Stable locomotion in precipitous environments is an essential task for quadruped robots, requiring the ability to resist various external disturbances. Recent neural policies enhance robustness against disturbances by learning to resist external forces sampled from a fixed distribution in the simulated environment. However, the force generation process doesn’t consider the robot’s current state, making it difficult to identify the most effective direction and magnitude that can push the robot to the most unstable but recoverable state. Thus, challenging cases in the buffer are insufficient to optimize robustness. In this paper, we propose to model the robust locomotion learning process as an adversarial interaction between the locomotion policy and a learnable disturbance that is conditioned on the robot state to generate appropriate external forces. To make the joint optimization stable, our novel $H_{\infty}$ constraint mandates the bound of the ratio between the cost and the intensity of the external forces. We verify the robustness of our approach in both simulated environments and real-world deployment, on quadrupedal locomotion tasks and a more challenging task where the quadruped performs locomotion merely on hind legs. Training and deployment code will be made public.

Learning H-Infinity Locomotion Control
Junfeng Long", Wenye Yu 12,, Quanyi Li" Zirui Wang Dahua Lin Jiangmiao Pang
'Shanghai AI Laboratory FShanghai Jiao Tong University 'Zhejiang University "The Chinese University of Hong Kong
Plane Slope
UpwDown Stairs
High platform
Slippery
Bipedal Walking
Figure I:
We deploy the policy trained by our method to real robots. Whether in quadrupedal or bipedal states, the robots successfully resist disturbances under various conditions.
Abstract:
Stable locomotion in precipitous environments is an essential task for quadruped robots, requiring the ability to resist various external disturbances. Recent neu- ral policies enhance robustness against disturbances by learning to resist external forces sampled from a fixed distribution in the simulated environment. However, the force generation process doesn't consider the robot's current state, making it difficult identify the most effective direction and magnitude that push the robot to the most unstable but recoverable state. Thus, challenging cases in the buffer are insufficient to optimize robustness. In this paper, propose model the robust locomotion learning process as an adversarial interaction between the locomotion policy and a learnable disturbance that is conditioned on the robot state to generate appropriate external forces. To make the joint optimization sta- ble, our novel Ho constraint mandates the bound of the ratio between the cost measurement for performance drop) and the intensity of the external forces. We verify the robustness of approach in both simulated environments and real- world deployment, on quadrupedal locomotion tasks and more challenging task where the quadruped performs locomotion merely on hind legs. Please refer to our project page for videos of real-world deployment.
Introduction
Recent end-to-end learning-bassed quadruped controllers exhibit various capabilities during deploy-
ment in real-world settings [1, 2, 3, 5, 6, 7, 8. 9, 10, 11, 12, 13, 14, 15, 16). Moreover, the learning-based approach enables skills beyond locomotion including target tracking in a bipedal manner [17, 18), manipulation using front legs [19], jumping over obstacles [17] and parkour [20].
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Successful real-world deployment requires the control policy to be able to resist various distur- bances like strong wind and falling debris. Previous learning-basee controllers acquire this ability with domain randomization [21, 22] where environment parameters like external forces [23, 24] are randomly sampled and exerted on the robot trunk during training. However, this method is not efficient enough to generate high-quality disturbance-resisting training samples and hinders the policy from acquiring adequate robustness. To be specific, excessively severe disturbances in early training procedures could undermine the training, whereas insufficiently challenging disturbances in late training stages may hinder the robot from developing a more resilient policy. The preliminary experiments in Appendix A provide evidence of this hypothesis.
For generating more effective training samples, an ideal external force sampler is supposed to affect the policy the extent that the agent experiences an obvious performance drop but is still able to recover from the disturbance, which guarantees not only the training feasibility but the weakness of the policy is attacked precisely. To this end, we introduce a disturber network conditioned on the current states of the robot to generate adaptive external forces. Compared to the actor that aims to maximize the cumulative discounted overall reward, the disturber is modeled as a separate learnable module to maximize the cumulative discounted error between the task reward and its upper bound. To ensure stable optimization between the actor and the disturber, we implement an additional learning objective derived from the constraint inspired by the classical He theory [25, 26, 27], which mandates the bound of the ratio between the cost and the intensity of external forces generated by the disturber. Following this constraint, we naturally derive an upper bound for the cost function with respect to a certain intensity of external forces, which is equivalent to performance lower bound for the actor with a theoretical guarantee.
We train our method in Isaac Gym simulator [28] and utilize dual gradient descent method [29] for joint optimization. We evaluate our locomotion policy by comparing it against baseline approaches in terms of their command-trackee train policies with baseline methods and our method in the non-stationary bipedal walking setting and measure their abilities to resist collision. In all evaluations, our method outperforms the baseline method, suggesting the effectiveness and superiority of our method. We deploy the learned policy on Unitree Aliengo robot and Unitree Al robot in real-world settings. As shown Fig. 1, the robot manages to traverse planes, slopes, stairs, high platforms, and greasy surfaces whether the external foree is applied the trunk or legs. The robot can even walk with its hind legs while withstanding the impact from heavy objects.
ability under various types of disturbances and terrains. We also
2 Related Work
Quadruped robots are expected to stabilize themselves face of noisy observations and external forces. While large quantities of research have been carried out to resolve the former issue either by modeling observation noises explicitly during training procedure [9, 24] introducing visual inputs by depth images robots [3, 20]. few works shed light confronting potential physical interruptions. While some works claim to achieve robust performance during real-world deployment [7], they fail to model external forces as learnable modules and introduce extreme disruptions to either training or real-world deployment, resulting in vulnerability to harsher conditions.
However, simply modeling external forces as a learnable module causes the problem fall into the setting of adversarial reinforcement learning, which is particular case of multi-agent reinforcement learning. One critical challenge in this field is training instability. During training, each agent's 's pol- icy changes over time, which results in the environment becoming non-stationary from the view of any individual agent. Directly applying single-agem algorithm will lead to the non-stationary prob- lem. For example, Lowe et al. [30] found that the variance of the policy gradient grows exponentially when the number of agents increases. Although following works utilized centralized critic [30, 31] which can stabilize training, the learned policy may be sensitive its training parameters and con- verge poor local optimal. This problem is more severe for competitive environments because if the opponents change their policies, the learned policy may perform even worse [32].

---

There exist some previous works on adversarial reinforcement learning aiming to enhance robustness of control policy. For instance, Gleave et al. [33] constructed an adversarial agent that produced ef- fective attack on the actor, but did not proceed to refine the actor for more robust performance. Pinto al. [34] and Pan et al. [35] drew inspiration from H-infinity theory and adopted an adversarial RL framework. However, the former modeled the problem not mandate this setting. The latter averted risk by lowering the variance of multiple value functions while our approach introduces explicit estimation of performance drop through cost value networks. Besides, they both failed to regulate the ratio between output error and disturbance norm stated in H-infinity theory, granting no performance lower bound for the actor in face of disturbances. Sim- ilar to [34], Rigter et al. [36] modeled zero-sum game for model-base offline RL. Besides, they constructed worst-case scenario for the actor by tuning the model to predict transitions with the most unpromising next-state value, but an explicit disturbance was absent.
zero-sum game while our method does
In light of that, we introduce a novel training framework for quadruped locomotion by modeling an external disturber explicitly, which is the first attempt to do so as far as we are concerned. Based on the classic H. method from control theory [25, 26, 27], we devise a brand-new training pipeline where the external disturber and the actor of the robot can be jointly optimized in an adversarial manner. With more experience of physical disturbance in training, quadruped robots acquire more robustness against external forces in real-world deployment.
3 Preliminaries
Classic He control [37] deals with system where disturbance is involved. We denote G as the plant, K the controller, as the control input, as the measurement available to the controller, w as an unknown disturbance, and z as the error output which is expected to be minimized. In general, we wish the controller to stabilize the closed-loop system based on a model of the plant G. As shown in Fig. 2, the goal of Hz control is design controller K that minimizes the error while minimizing the H norm of the closed-loop transfer function Tew from the disturbance to the error 2.
min
||T 2oulll
sup 1ill m=00
Tolls
W
However, minimizing [o usually challenging. In prac- tice, we instead wish to find an acceptable 7) > and a con- troller K satisfying [|Twll n, which is called suboptimal Hie control. We denote this type of controller as n-optimal in this paper. According to Morimoto and Doya [25], if < 1J. it is guaranteed that the system will remain stabilized for any disturbance mapping d - with |d] <
Figure 2: Illustration of the classic Ho control theory.
Finding a n-optiall Hx controller modeled as a min-max problem. We consider a plant with dynamics given by
* f(x, w.
where X R is the state, U R is the control input, and e w R' is the disturbance input. Then, the His control problem can be viewed finding controller that satisfies:
Iew = sup
(1)
where z is the error output. Given that the Euclidean norms |lz||a and ||w are defined as:
| = zZ (t)z(()) dt, [w]22= = Je wT (t)w((()))) dt
Our goal, hereby, is to find control input satisfying:
= (z (f)((t) - -1 (W <0,
(2)
3

---

Policy Data Collection
Optimization
PPO Certtic (Tt
c(5
F'r (s,)
Action
V(sp)
PPO (T)
Policy
n(: (s²)
AL Wi Min| (Tt)
d( (s₂) peast (sp)
Force
Novel Loss Ldisturber
St
Id( 1s₂)
St+1
peost (Sc+1)
Disturber
Disturber Data Collection
Disturbance
Movement
Overview of Hx locomotion control method. At every time step during the training process, we perform simulation step based on the robot's action and the external force generated by the disturber. The agent thus moves towards the rewarded direction and resists the disturbance. During the optimization process, values are calculated for batched training samples and carry out Ho policy gradient by optimizing the PPO loss of the actor while taking into consideration the novel constraint H.H.n,, Value estimators (Critic) are also updated to approximate the state value.
Figure 3:
where w is any possible disturbance and the state vector x satisfies zero initialization (x(0) = 0)- By solving the following min-max game, we can find the best control input u while the worst disturbance is chosen maximize V:
minmax S (zT(1)(()) -/w"(()w(()d <0
(3)
Learning Hic Locomotion Control
In this section, we first give the definition of the robust locomotion problem. After that, we describe our method in detail and give practical implementation.
4.1 Problem Definition
As described in the former sections, we wish the disturber to learn more effective disturbances. We model it a one-step decision problem. Given Markov Decision Process (MDP) M = (S, A, T, R. 7), we define the disturhance policy to be a function d : - c R which maps observations to forces. Let C S x D R cost function that measures the gap between current performance and the best performance in theory measured by reward.
Additionally, (s) -r(ss))((((ssdd())) C(s, a, d) denotes the gap between expected performance and actual performance given policy and disturber d. Similar to Sec. 3, for given > O, we wish to find an admissible policy such that
lim Σ (C 7/ dell)) < 0,
(4)
We define policy satisfying the above condition opttmmll More intuitively, if policy is oopptimal, then an external force f can get performance decay up f/l Additionally, we wish the disturbances to be effective, which means that it can maximize the cost of policy with limited intensity. Therefore, for policy π,, and a discount factor & 72 < 1, the target of d is to maximize:
8 Eal
7 (s) n/(d/112)
(5)
1-0

---

4.2 Method
In reinforcement learning-based locomotion control, the reward functions are usually complicated [5, 23, 1, 7]. Some of them guide the policy to complete the task, and some them act as regu- larization to the policy. In our work, we divide the reward functions into two categories, the task rewards and the auxiliary rewards. The former part leads the policy achieve command tracking, maintain good orientation and stay at desired base height, while the latter part leads the policy to satisfy the physical constraints of robot and give smoother control. We present the details of our reward functions in Table 1 and 2, which can be found in Appendix C.I.
Now we denote the rewards from each part as task rewards Rlask and auxiliary rewards Rawe re- spectively, and the overall reward as R. Firstly, we assume that the task reward has an upper bound Rinak and the cost can be formulated as C Raask Rlask With R and C, we can get value functions for overall reward and cost, denoted as V and Veast, We adopt PPO [38] as basic policy optimization method. Then the goal of the actor at each iteration is to solve:
maximize subject to
Er
x(aile))
A(ss)
Kolt (ar (+c)
E,
KL Told (- 8t)
9t <
E,
1/
de |2 (s))
0,
As PPO [38] is used to optimize the disturber as well, the goal of the disturber each iteration is to solve:
where the advantage function [39].
E, d(delee) A (s)) dar (de 0e) subject to E, [L [doa (.1 8t), (- [t]]] < 6,
maximize
However, requiring a high-freyuency controller to be strictly robust in every time step is unpractical, so we replace the constraint E, [n) d(s)) 12 - (st) > 0 with a more flexible substitute:
B, [rl|dd]]2 (x1) + (xost(s)) Vrest (s(+1) > 0,
(8)
where Feost is the value function of the disturber. Intuitively, if the policy guides the robot to better state, the constraint will be slackened, otherwise the constraint will be tightened. We will show that using this constraint, the actor is also guaranteed to be n-optimal.
We follow PPO to deal with the KL divergence part and use dual gradient decent method [29] to deal with the extra constraint, denoted as T, f (TT) d E, [r]]]dd:l (st) + (sost(ss) Vcost (8++1)]
0, then the update process of policy can be described as:
argmaxI, PPO
x H Hins (x)
= argmax (d)
X = >
(inn(())
where I.P (T) is the PPO objective function for the actor, Laisturber is the objective function for disturber with similar form PPO objective function, À is the Lagrangian multiplier of the proposed constraint, and a is the step-size of updating X. An overview of our method is in Fig. 3.
4.3 7-optimality
We assume that & C(s, Cmar where Cmaz OD a constant, and there is value function yeost such that 0 & Veost(s (s) & Vcost for any s S, where Veost co. Besides, we denote (s) P(st s|su. x). where su is sampled from initial states, assuming that the limit of distri- bution under policy x is (8) = lime- (s(() and it exists. Then we have the following theorem:
Theorem 1. If Cs) - n/| (s))l22 < Ey-PL (s) - (sC))(('))) ( for s S with 3z (s) > 0.
the policy स is opptimall.
Detailed derivation of Theorem can be found in Appendix B.
5

---

4.4 Practical Implementations
We use Isaac Gym [28, 40J with 4096 parallel environments and rollout length of 100 time steps. Our training platform is RTX 3090. During training, we randomize ground friction, restitution coefficients, motor strength, joint-level PD gains, system delay and initial joint positions in each episode. The randomization ranges for each parameter are detailed in Table 3 in Appendix C.2. The algorithm is summarized in Algorithm in Appendix C.3.
5 Experimental Results
In this section, we conduct experiments to show the effectiveness of our method. We use the lat- est non-visual locomotion method [9] as our baseline which trained with continuous stochastic disturbances drawn from uniform distribution. By changing the disturbance sampling strategy to ours and its ablated versions, we can show to what extent our method exceeds the baseline and the effectiveness of specific modules of our methods. Our experiments aim to answer these questions:
1. Can our method and its variants handle continuous disturbances as well as the bascline?
2. Can all methods handle the challenges of sudden extreme disturbances?
3. Can all methods resist deliberate disturbances that intentionally attack the policy?
4. Is our method applicable to other tasks that require stronger robustness?
5. Can our method be deployed to real robots?
Specifically, we design four different training settings for comparison studies. First, we train a policy in complete settings where both H-infinity loss and a disturber network are exploited, which refer to ours. We clip the external forces have intensity of no more than 100N for sake of robot capability. Next, we remove the H-infinity loss from the training pipeline and obtain another policy, which we refer to ours without hinf loss. Then, we keep the H-infinity loss but remove the disturber network from ours and replace it with a 'disturaance curriculum whose largest intensity grows linearly from ON to 100N with the training process and whose direction is sampled uniformly. We call this policy ours without learnable disturber. Finally, we train a vanilla policy without both H-infinitt loss and disturber network, which also experiences random external forces with curriculum disturbance described above. We refer this policy as baseline. All four policies are trained the same set of terrains (Stairs, Slopes, and Diserete heightfield) as is shown in Appendix C.2. The training process for all policies lasts 5000 epochs.
After obtaining the well-trained 4 policies, we evaluate them on terrains with 3 types of distur- bances (continuous disturbance, sudden force, and deliberate attack) and measure their command- tracking performance. For each evaluation, we repeat the rollout 32 times with different seeds and report the average performance with 95% confidence interval.
5.1 Can our method and its variants handle continuous disturbances well as the baseline?
To answer question 1, we test all policies with random continuous disturbances which are drawn from uniform distribution ranging from 0-100N with the same frequency as controllers. same type of disturbance experienced by the baseline at the final training stage. We command the robot to move forward with velocity of 1.0 m/s. The tracking curves in Fig. 4 show that our method has the same capability of dealing with continuous disturbances on rough slopes bascline methods and it even performs better on discrete height fields, and stairs. In an overall sense, our method can achieve comparable or even better performance against the baseline method in the continuous disturbance setting, even if the baseline methods have been trained with the same type of disturbances. Also, the policy trained without H-infinity loss fails immediately regardless of the terrain, demonstrating that vanilla adversarial training doesn't 't work well, highlighting the effectiveness of the novel H-infinity loss.
the
Without the curriculum scheme, the training will collapse large force may be sampled in the early training phase, which is also confirmed by our preliminary experiments in A.

---

Ours
Ours W/O Learnable Disturber
Ours W/O Hinf Loss
Baseline
Command
Slopes
Heightfields
1.5
Base Velocity Along A [mris]
1.25
1.0
0.75
0.5
0.25
0.0
2
10
12
14
15
18
20
12
14
16
18
20
10
1.2
14
16
18
Time [s]
Time [5]
Time [s)
Figure 4: Tracking curve of our method and baselines under continuous random forces.
Ours
Ours W/O Learnable Disturber
Ours W/O Hinf Loss
Baseline
Command
Slopes
Stairs
Discrete Heightfields
Dade Velacity Along X.Axia [mis]]
2
10
12
14
10
10
20
30
12
14
16
10
70
10
12
34
10
18
20
Time [s]
Time [s]
Time
[s]
Figure 5: Tracking curve of our method and baselines under sudden large forces.
5.2 Can all methods handle the challenges of sudden extreme disturbances?
To answer question we evaluate all policies by applying sudden large external forces on the trunk of robots. We apply identical forces to all robots with intensity of 150N and random direction sampled uniformly. The external forces are applied every 4 seconds and last 0.5 seconds. In Fig. 5, spike or pit appears at the moment the force is applied, indicating the robot is trying to offsst the external force. Robot controlled by our policy shows better precision in tracking the command and the ability to recover from sudden force, especially on stairs and heightfields.
Ours W/O Learnalee Disturber
Hinf
Baseline
Command
Slopes
Stairs
Discrete Heightfieeds
1.5
Base Velocity Along Axiis [m]s]]
1.25
1.0
0.75
0.5
0.5
0.0
.025
Figure 6: Tracking curve for all methods tested with disturbers trained to intentionally attack them.

---

5.3 Can all methods resist deliberate disturbances that intentionally attack the policy?
To answer question 3, we freeze the parameters of four well-trained policies and train a disturber from scratch for each policy using our method. By doing this, each disturber is optimized to discover the weakness of the corresponding policy and try to undermine its performance as much as possible. We perform the disturber training for 500 epochs and examine the tracking performance of the four policies on different terrains with the specifically trained adversarial disturber. The disturbance applied continuously as well. The results shown in Fig. 6 suggest disturbers can identify the weakness for other policies immediately, and these policies fail upon encountering the attach for the first time, whereas our method can withstand the deliberate disturbance many times across three challenging terrains, especially on slopes and discrete heightfields.
5.4 Is our method applicable to other tasks that require stronger robustness?
To answer question 5, we train the robot walk with two hind legs and test the policy by exerting intermit- tent large external forces. We train
Standing with on Standing External y-axis
10--
Out
Namber of Falls
Number Falle
40-
the policy for 10000 epochs for the sake of stronger demands of this task. Identical to the quadrupedal locomo- tion task, the baseline bipedal pol- icy is trained with normal random disturber while our method is trained with the proposed adaptive disturber. Both disturbers have the same sam- ple space ranging from ON to 50N. To evaluate the performance of both methods, we count the total times of falls in one episode when external forces are exerted. Each evaluation episode lasts 20 seconds. Every 5 seconds, the robot receives large external force with an intensity of 10ON to 150N that lasts 0.2 seconds. We carry out two different experiments where the directions of the forces are set to æ, y axes respectively. For each method, the evaluation runs 32 times repeatedly and we report the average number of falls. As shown in Fig. 7, our method outperforms the baseline policy by a large margin no matter the force comes from x or y axes.
60
30-
10000
112.0
120.0
196.0
1900
100
110.3
120.0
100.0
1600
Intonsity (N)
Foroo Intorsity (N)
Figure 7: Comparison between the baseline and our method in terms of the number of falls.
5.5
Can our method be deployed real robots?
To answer question 4, we train our policy with the learnable disturber with force limit of 100 N and deploy trained policies Unitree Aliengo quadrupedal robots in the wild. As shown in Fig. 1, it can traverse various terrains such staircases, high platforms, slopes, and slippery surfaces, withstand pulling on the trunk, legs, and even arbitrary kicking, and accomplish different tasks such as sprinting. In addition, We deploy the bipedal walking policy to the Unitree Al robot. As shown in Fig. 1, the standing policy is able to withstand collisions with heavy objects and random pushes on its body while retaining a standing posture.
Conclusion
In this work, we propose Ho learning framework for quadruped locomotion control. Unlike pre- vious works where the external force is drawn from a fixed distribution, we propose to train an adversarial disturber generate external force dynamically. To stabilize the learning process, we introduce a novel H. constraint to policy optimization, providing guarantee for the actor's perfor- mance lower bound in face of external forces with certain intensity. We demonstrate our method achieves notable improvement in robustness in both locomotion and standing tasks and can be de- ployed in world settings. To inspire further research, all code and checkpoints are made public.

---

Acknowledgments
We thank Shanghai AI Laboratory for hardware support and provision of experimental site.
References
[1] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 2019.
[2] C. Yang, K. Yuan, Q. Zhu, W. Yu, and Z. Li. Multi-expert learning of adaptive legged locomo- tion. Science Robotics, 2020.
[3] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang. Visual-locomotion: Learning to walk on complex terrains with vision. In Conference on Robot Learning (CoRL), 2021.
[4] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal, Rapid locomotion via rein- forcement learning. In Robotics: Science and Systems, 2022.
[5] G. B. Margolis and P. Agrawal. Walk these ways: Tuning robot control for generalization with multiplicity of behavior. In Conference on Robot Learning (CoRL), 2023.
[6] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In Conference on Robot Learning (CoRL), pages 403-415. PMLR, 2023.
[7] M. Nahrendra, B. Yu, and H. Myung. Dreamwaq: Learning robust quadrupedal loco- motion with implicit terrain imagination via deep reinforcement learning. In International Conference Robotics and Automation (ICRA), pages 5078-5084. IEEE, 2023.
[8] Y. Kim, H. Oh, J. Lee, J. Choi, G. Ji, M. Jung, D. Youm, and J. Hwangbo. Not only rewards also constraints: Applications on legged robot locomotion, 2024.
[9] J. Long, Z. Wang, Q. Li, J. Gao, L. Cao, and J. Pang. Hybrid internal model: Learning agile legged locomotion with simulated robot response. In International Conference on Learning Representations, 2024.
[10] Ha, Xu, Z. Tan, S. Levine, and J. Tan. Learning to walk in the real world with minimal human effort, 2020.
[11] T. Haarnoja, S. Ha. A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning, 2019.
[12] R. Yang, M. Zhang, N. Hansen, Xu, and Wang. Learning vision-guided quadrupedal comotion end-to-end with cross-modal transformers. In International Conference on Learning Representations, 2022.
[13] R. Yang, G. Yang, and X. Wang. Neural volumetric memory for visual locomotion control, In Conference on Computer Vision and Pattern Recognition, 2023.
[14] Lee, Hwangbo, L. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 2020.
[15] F. Jenelten, J. Hwangbo. F. Tresoldi, C. D. Bellicoso, and M. Hutter. Dynamic locomotion on slippery ground. IEEE Robotics and Automation Letters, 4(4):4170 2019. doi: 10.1 R..10099R
[16] J. Reher, W.-L. Ma, and A. D. Ames. Dynamic walking with compliance cassie bipedal robot, 2019,

---

[17] L. Smith, J. C. Kew, T. Li, L. Luu, X. B. Peng, S. Ha, J. Tan, and S. Levine. Learning and adapting agile locomotion skills by transferring experience, 2023.
[18] Y. J. Li, W. Fu, and Y. Wu. Learning agile bipedal motions on quadrupedal robot, 2023.
[19] X. Cheng, Kumar, and D. Pathak. Legs as manipulator: Pushing quadrupedal agility beyond locomotion, 2023.
[20] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots, 2023.
[21] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic con- trol with dynamics randomization. In International Conference on Robotics and Automation (ICRA), pages 3803-3810. IEEE, 2018.
[22] J. Tobin, R. Fong, A. Ray. J. Schneider, W. Zaremba, and P. Abbeel. Domain randomiza- tion for transferring deep neural networks from simulation to the real world. In International Conference on Intelligent Robots and Systems (IROS), pages 23-30, IEEE, 2017.
[23] C. Zhang, N. Rudin, D. Hoeller, and M. Hutter. Learning agile locomotion on risky terrains, 2023.
[24] K. Paigwar, L. Krishna, S. Tirumala, N. Khetan, A. Sagi, A. Joglekar, S. Bhatnagar, A. Ghosal,
B. Amrutur, and S. Kolathaya. Robust quadrupedal locomotion on sloped terrains: A linear policy approach, 2020.
[25] J. Morimoto and K. Doya. Robust reinforcement learning. Neural Computation, 17(2)3335- 359, 2005.
[26] A. Aalipour and A. Khani. Data-driven h-infinity control with real-time and efficient inforcement learning algorithm: An application to autonomous mobility-on-demand systems. 2023. doi: 000445500/AXXI122000.088888 URL htt../ttss//aaaai..ro//abb////// http://b//23099.. 08880.
[27] D. Larby and Forni. A passivity preserving h-infinity synthesis technique for robot control. In 2022 IEEE 61st Conference Decision and Control (CDC), pages 1416-1422 IEEE, 2022.
[28] V. Makoviychuk, Wawrzyniak, Guo, M. Lu, Storey, M. Macklin, Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physics simula- tion for robot learning, 2021.
[29] Paternain, M. Calvo-Fullana, F. O. Chamon, and Ribeiro. Safe policies for reinforce- ment learning via primal-dual methods. 2022.
[30] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and Mordatch. Multi-agent actor-critic for mixed cooperative-compettivee environments. Advances in neural information processing systems, 30, 2017.
[31] J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[32] A. Lazaridou, A. Peysakhovich, and M. Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint 11Xiv:1612.07188 2016.
[33] A. Gleave, M. Dennis, C. Wild, N. Kant, S. Levine, and S. Russell. Adversarial policies: Attacking deep reinforcement learning, 2021.
10615. 10615.
[34] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcementi learning, 2017. URL https:://seeet wwwpzxi..rggabs///eeee orgg/bs/////// 1703. .27022.
[35] X. Pan, D. Seita, Y. Gao, and J. Canny. Risk averse robust adversarial reinforcement learning,
2019. URL https: / /rxivv www./ogg////eeooooooo//../...ett ab9/9444//99 00511.
10

---

[36] M. Rigter. B. Lacerda, and N. Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning, 2022. URL /tatps://arxiv.... 2015/4.//2200..... 12581.
[37] K. Zhou and J. C. Doyle. Essentials of robust control, volume 104. Prentice hall Upper Saddle River, NJ, 1998.
[38] Schulman, Wolski, Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, 2017.
[39] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1 2015.
[40] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning (CoRL), pages 91- 100. PMLR, 2022.

---

Appendix
A Preliminary Experiments
To verify the necessity of having an adaptive disturber, we conduct preliminary experiment. The most com- mon disturber for legged locomotion randomly samples external forces from [0,30] N [28]. We increase the up- per bound of the uniform distribution to 100 N and get the Baseline method. As shown in Fig. 8, the training of Baseline collapses under external forces with the tremely large upper bound (100N). Although another method, Baseline-C. C, overcomes this by curriculum learn- ing where the upper bound of the forces linearly increases throughout the training, the trained policy fails to achieve comparable final performance against our method, as the training samples generated may be not challenging enough in the late training stage in terms of only the magnitude but the direction of the force. However, our method keeps optimizing for better adversarial perfor- mance and producing valid training samples throughout the training.
Baseline
Baseline-C
Tracking Reward
0.1
0
1000
2000
3000
4000
5000
Figure 8: Preliminary experiment compar- ing policies trained with fixed disturber and our method.
B Proof of Theorem 1
Proof.
lim
E (st) dell))
7-000
T
lim
Es, (
(Veot((s)) Veost (e
(8++1))) (Hypothesis of the theorem)
1-000
T
t=0
1
lim
(st)
T-w
T
(=0
P(s 16,, (Vonst - Vonat (==1)) (Based on the definition of 3 (s:))
JS
lim
T-
T
8 (s))
((544286,,T)
(8+11)s944dd) (B04))))))
JS
1
-im
Eag (c0st(s))
T-o T
(st P( (ss
convergence theorem)
J8
lim
Eg (ea))(st) - Eesg. ceast (s(+1))
T-3
T
lim
(Eag Vroos( (s0) Esrra peost (eost (ST+1))
T-2
T
lim
(Vcost 0) (Given peost 00)
T-x00
12

---

Table 1: Reward functions for Unitree A1 standing task
Term (* indicates Rlask)
Calculation
Scale
linear velocity tracking" exp( - |uay
Eay I|? //trakk) Fors 1.0
angular velocity tracking* exp(- -11s, - w / /track)) Tork
0.5
joint velocities
| 2
-2e-1
joint accelerations
|| 2
2.5e
action rate
at+1 aq 1
-0.01
joint position limits
[[ $ (gmin, Imax)
10000
joint velocity limits
L $ (gminy Imaz)
10000
torque limits
1 T $ (Tmin: Tmaxx) )
10.0
collision
Ejep jeontact jeontact//1 /|P|
-10
extra collision front feet contact
D JeeR jeontact //ppl
1.0
EΣ (E(FL.FFR poontact 0J
1.0
orientation Yori root height
(0.5 cos (ty -0*) 0.5)²
1.0
min( (e 0.55)
1.0
Therefore we obtain lim
+ Σ Es, (s) ///r/2) 0, and thus, the following inequality is
T-
derived:
lim >
B., (C - <
(10)
T-
1-0
C
Training details
C.1 Reward function scales for Unitree Aliengo locomotion task and Unitree Al standing task
Detailed reward functions are shown in Table and Table 2. To clarify the meaning of some symbols used in the reward functions, P denotes the set of all joints whose collisions with the ground are penalized, and Ep denotes the set of joints with stronger penalization. frontact stands for whether foot f has contact with the ground. Moreover, denotes the projection of gravity onto the local frame of the robot, and h denotes the base height of the robot. the standing task particularly, we define an ideal orientation n for the robot base, which we assign the value y = (0.2, 0.0, 1.0), and accordingly define the unit ideal orientation 6* .*
| We expect the local r-axis of the metric for
robot, which we denote Vy, be aligned to 6* and thus adopt cosine similarity the orientation reward. Besides, scale the tracking rewards by the orientation reward Fors in the standing task because expect the robot to stabilize itself in standing pose before going on to follow tracking commands.
C.2 Terrains and domain randomization details
We exploit three different types of terrains, slopes, stairs, and discrete height fields during the train- ing procedure, as is presented in Fig. We also introduce terrain curriculum strategy, where the level of terrain difficulty is dynamically adjusted according to the distance that the robot can travel during fixed duration. Besides, we exploit domain randomization for some simulation parameters, as is shown in Table 3.
13

---

Table Reward functions for Unitree Aliengo locomotion task
Term (* indicates Rlask)
Calculation
Scale
linear velocity tracking*
exp( exp(
Uuy
1/ 1 /Otrack)
1.0
angular velocity tracking*
11 We
cand 1/2/ //otack))
0.5
z-axis linear velocity
2
-2.
roll-pitch angular velocity
1 1
-0.05
joint power
Σ || + [g]
-e 5
joint power distribution
Var O [d]
-
joint accelerations
|
2.5e
action rate
1a a1-1 1
0001
smoothness
|00 2ap-1 + a4-2 1
0.01
joint position limits
1[g $ (gmin, Iman
5.0
joint velocity limits
1 # (ámin: Imax)
5.0
torque limits orientation
T $ (Tminy Tmaz)
5.0
1 12
002
base height
h h farget 12
-1.
(a) Slopes
(b) Stairs
(c) Discrete height fields
Figure 9: Demonstration different terrains used in simulated training environments
Table 3: Domain Randomizations and their Respective Range
Parameters
Range[Min, Max]
Unit
Ground Friction
[0.2, 2.75]
Ground Restitution
[0.0, 1.0)
Joint Kp Joint K Initial Joint Positions
00.8, 1.2
20
0088, 1.2
0.5
[0.5, 1.5]
x nominal value rad
14

---

C.3
Pseudo code for Ho locomotion control
Algorithm 1: Learning Hx Locomotion Control
Input:
Initial actor To, disturber dg. overall value function Vo, task value function Viestt initial guess no, initial multiplier So. upper bound of task reward Reat
Output: policy n, disturber cl
Told Ng. dotd du. Vold Vo. V Vonst
for iteration = 1,2, ... max iteration do
Run policy Vold in environment for I' time steps
Compute values of each states with Vold
Compute cost values of each states with old
Compute costs C man R
Compute advantage estimation A.
Optimize ı with Li + X * L Hinf
Optimize d with Laisturber
À À
Hind
Cr Ct
7
0.9 17 + 0.1 27 Idoula
Told = π
end
D Limitations
Although our method able to enhance the robustness of control policy across various robots and different tasks, it still suffers from a few limitations. First, task reward functions need to be defined for different embodiments and tasks. Besides, in order to generalize to other robotic systems such as robot arms, method requires new definitions of "disturbances", other than external force. Future works can be carried out to resolve the aforementioned issues.
15


Reviews:
[{"review_id": "VbDgrEWxpK", "reviewer": "Reviewer_p1Hn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a reinforcement learning framework for quadrupedal robot locomotion that a) learns a disturbance network conditioned on the current state of the robot to maximize the cumulative discounted error in the task performance, and b) learn a policy that can withstand the disturbances applied by the disturbance network. Using ideas from classical robust control theory (H-infinity control), the authors propose a reward function to jointly train the policy and disturbance networks in a stable manner. The proposed approach is validated on two quadrupedal robot platforms under quarupedal and bipedal modes.", "summary_of_recommendation": "The method needs to be motivated better, limitations of the approach need to be discussed", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please address the points in the Review section. \n\nParticularly, motivate the use of H-infinity control to devise the disturbance network and discuss limitations.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eEPzjrRKUF", "reviewer": "Reviewer_ATZL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A robust control inspired loss for learning robust control policies in adversarial settings.", "summary_of_recommendation": "Robust control regularizes learning, while learning addresses the performance trade-off in robust control—an elegant and powerful approach to control synthesis.", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "The reviewer has no questions or clarifying remarks on the paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rofIRAxS95", "reviewer": "Reviewer_YAqL", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a creative technique for robust RL, which embeds the objective in the traditional zero-sum formulation of an H_inf robust control problem as a constraint during policy learning. Results indicate that this can dramatically improve robustness to a variety of disturbances.", "summary_of_recommendation": "The paper contains the kernel of an interesting idea which I believe could have good impact on the community. However, the paper also contains many technical and writing-related issues which should be addressed before publication.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Please see my detailed comments above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: This paper proposes an adversarial interaction model for robust quadruped locomotion learning, where a learnable, state-conditioned disturbance generates optimal external forces to push the robot to unstable but recoverable states, verified in both simulated and real-world environments, including challenging hind-leg locomotion tasks.

Strength:
* Demonstrates competitive performance on real quadrupedal robots in both quadrupedal and bipedal modes, showing real-world applicability.
* Evaluated on multiple platforms and compared against state-of-the-art methods, showing significant robustness improvements.
* Addresses practical challenges by relaxing constraints to alleviate conservatism typically observed in classical robust control.

Weakness:
* Some sections, such as the connection between H-infinity control and the proposed objective, are not clearly explained.
* Important aspects, such as the detailed explanation of the progressive training and the role of specific parameters, are not well-covered.
* Relies on assumptions like static environments and accurate odometry, which may limit applicability in dynamic real-world scenarios.
* Lacks comprehensive comparison with domain adaptation methods and explicit adversarial training in RL, and does not outperform all baselines.
* The approach appears to rely on heuristics without rigorous proof of correctness, and the uncertainty estimation lacks discussion on false positive and negative rates.

---- 

Post rebuttal: The paper received two accept and one reject. The AC carefully went throughs the rebuttal and revised paper and discussion and agreed with the majority of the reviewers to accept the paper. But the AC will encourage to improve the clarity of the paper.
Decision: Accept