In-Flight Attitude Control of a Quadruped using Deep Reinforcement Learning
Authors: Tarek El-Agroudi
Finn Gross Maurer
Jørgen Anker Olsen
Kostas Alexis
Venue: CoRL 2024 (2024)
Keywords: Deep Reinforcement Learning
Legged Robotics
TL;DR: We use DRL to train and demonstrate an in-flight attitude control law for a small low-cost quadruped with a five-bar-linkage leg design using only its legs as reaction masses.

Abstract: We present the development and real world demonstration of an in-flight attitude control law for a small low-cost quadruped with a five-bar-linkage leg design using only its legs as reaction masses. The control law is trained using deep reinforcement learning (DRL) and specifically through Proximal Policy Optimization (PPO) in the NVIDIA Omniverse Isaac Sim simulator with a GPU-accelerated DRL pipeline. To demonstrate the policy, a small quadruped is designed, constructed, and evaluated both on a rotating pole test setup and in free fall. During a free fall of 0.7 seconds, the quadruped follows commanded attitude steps of 45 degrees in all principal axes, and achieves an average base angular velocity of 110 degrees per second during large attitude reference steps.

In-Flight Attitude Control of a Quadruped using Deep Reinforcement Learning
Tarek El-Agroudi" Finn Gross Maurer" Jorgen Anker Olsen Kostas Alexis
Norwegian University of Science and Technology
O. S. Bragstads Plass 2D, 7034 Trondheim
(tareke, finngm, jorgen. a. olsen, konstanting allexi @ntnu.... no Hardware design Code
Paper video GitHub IO page
We present the development and real world demonstration an in- flight attitude control law for small low-cost quadruped with a five-bar- -inkkae leg design using only its legs reaction masses. The control law is trained us- ing deep reinforcement learning (DRL) and specifically through Proximal Policy Optimization (PPO) in the NVIDIA Omniverse Isaac Sim simulator with GPU- accelerated DRL pipeline. To demonstrate the policy, a small quadruped is de- signed, constructed, and evaluated both on rotating pole test setup and in free fall. During a free fall of 0.7 seconds, the quadruped follows commanded atti- tude steps of degrees all principal axes and achieves an average base angular velocity of 110 degrees per second during large attitude reference steps.
Abstract:
Keywords: Deep Reinforcement Learning, Legged Robotics
-45'
-180"
-45'
120'
45' Roll
30' 20//30'330
Yaw
Pitch
Combo
Figure Camera frames for selected free fall experiments, with commanded attitude setpoints.
1 Introduction
The exploration of extraterrestrial environments with robots is daunting endeavor. On one hand, the impossibility of fast human intervention requires unprecedented robustness and autonomy while, on the other, the complexity and uncertainty extraterrestrial surfaces requires dexterous locomotion.
One particularly complex class of environments are martian and lunar lava tubes, the exploration of which might provide valuable geological insights and the potential for early human bases [1]. Wheeled rovers, of which several have been deployed on Mars and on the Moon [2], are inadequate
Indicates equal contribution.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

to explore these terrains due to the need for significant vertical motion for lava tube exploration. Helicopters, such as Ingenuity [3], have also been demonstrated on Mars but have limited payload- carrying capacity. With the recent advances in quadrupedal robots (quadrupeds) such as the Boston Dynamies Spot [4], the ANYbotics ANYmal [5] and the Unitree Go 1 [6], a proposed solution for the robotic exploration of lava tubes has been to use specially designed quadrupeds that can exploit the lower gravity experienced on Mars or on the Moon by jumping and experiencing long phases of "flight" to seamlessly traverse complex terrain [7].
In order to cancel out any accumulated angular velocity during a prolonged flight-phase and to en- able landing on inclined surfaces, an in-flight attitude' control law a prerequisite for such a robotic platform. A proposed solution for the actuation of such capabilities is for the quadruped to use its own legs as reaction masses in maneuvers that resemble the self-righting motions of a falling cat. This has certain ad advantages compared to other orientation approaches, such as the reaction wheels exploited in satellites [8]. Primarily, the considered approach does not require extra hardware and thus keeps the overall system weight smaller, which has significant mission payload implications.
With the recent advances in simulation technologies and parallelization hardware, many complex control tasks have been solved with DRL-based approaches. DRL allows for the exploitation of the equivalent of many hours of real-time experience in just a few minutes [9), while also being able to generate lightweight control policies implementable on resource-constraing motivates the use of DRL, for synthesizing an in-flight attitude control law for a jumping quadruped and the implementation of such policy on quadruped in real life.
hardware. This
Driven by the above, the main contribution of this work is the development of a DRL- attitude- control policy for a quadruped with five-bar-linkage leg design and lateral leg movement capabili- ties, and the demonstration of this policy on a bustom-designed prototype all degrees of freedom (DOFs). The proposed test setups include maneuvering on rotating pole and free fall experi- ments (a subset of the results is depicted in Figure 1) with the system following commanded attitude setpoints. The considered testing setups are inexpensive and easy to replicate. Furthermore, the developed quadruped design features low-cost hardware with of-the-shelf actuators and electron- ics, demonstrating the potential for robust DRL. without exact knowledge of factors like actuator dynamics. Key insights that transfer to related robotic hardware are the treatment of low-cost RC servo motors through a reference model and motor model, and the explicit simulation of the closed kinematic chain of the robot's legs. We further show how the same reward function can be used to learn policy for both fast and slow dynamics. To the authors' best knowledge, this work is the first real-world demonstration of an in-ftight attitude-control feedback policy for quadruped in 3D.
This paper is structured as follows: Related work is outlined in Section 2, while Section formally defines the control problem to be solved using DRL. An overview of the robotic research platform and experimental setups is provided in Section 4, and the proposed approach for training and de- ploying the DRL-based policy is outlined in Section 5, Section 6 evaluates the experimental results, and Section 7 outlines central conclusions, limitations and future work.
2 Related Work
This work builds upon range of previous contributions. The work in Rudin al. [10] presented SpaceBok. quadruped specifically designed for the exploration of lunar surfaces, and demonstrated a DRL-based policy in 2D on a test bed designed for micro-gravity experiments [11]. Kurtz et al. [12] demonstrates self-righting on the quadruped Mini Cheetah by restricting attitude errors to pitch and utilizing a combination of trajectory optimization and supervised learning techniques. The work on the tripedal robot SpaceHopper [13] has trained and demonstrated a DRL-based policy for attitude control in 3D on a parabolic flight and a specially designed gimball setup. Furthermore, Olsen and Alexis [14] presented OLY MPUS, quadruped with five-bar-linkage leg design tailored for extraterrestrial exploration. The modeling and control of closed kinematic chains (CKC), such as the N- bar-linkage legs of OLYMPUS and SpaceBok, has been extensively researched [15, 16),
We use attitude to denote the 3D orientation of the quadruped.

---

along with free-floatin rre--flating dynamical systems with manipulator structures [17]. Kamidi et al. [15] explicitly simulates the CKC and SpaceBok establishes an open-loop equivalent. For the synthesis of the policy using DRL, this work builds heavily upon [9], which demonstrated the learning of high- performance locomotion policies for quadrupeds mere minutes by exploiting GPU parallelization. The main technique used for sim2real transfer of the learned policy for SpaceBok and SpaceHopper domain randomization, which has been evaluated against range of sim2real methods in [18].
3 Problem Formulation
With reference to Figure 2, the considered control problem can be formally defined as follows: Given the orientation quaternion' g the quadruped's body frame in an inertial world frame W and a target orientation 7 expressed in W through q²" control the quadruped towards g. q." The latter is equivalent to controlling the angle of the axis-angle representation of the quaternion error q = (q @ L zero, denoted Zgf + O, which motivates the reward function presented in Section 5. The inputs made available to the control law are the the quaternion error q. its angular velocity w and the positions and velocities of the 12 actuators, O. and i.... forming the observation vector [ w/j 0m. I,,'"' R Actuation restricted to the actuators of the quadruped, and the output of the control law thus vector of motor position references 0, E R sent to the motors, which in turn apply torque Tm R 12 the actuated joints. We note that the target application of canceling angular momentum from jump is velocity tracking problem rather than regulation problem. However, if position tracking is demonstrated, velocity tracking can be achieved by the introduction of a moving position target.
B
w
Lateral
Transversal
motors
1V
341
m
g
m
150
808 paw
W
Figure 2: World, body and target frames for the attitude control problem. The black arrow indi- cates the angular velocity vector.
Figure 3: Eurepus dimensions and labeled leg. Or,, 8% and of are the inner transversal, outer transversal and lateral motor angles.
System Overview
The proposed research platform, called Eurepus, consists of four legs with five-bar-ling lie-baarlinkkge design, each consisting of lateral motor and two transversal motors. The base is constructed with interior rails to allow the quadruped to be mounted on forked rotating pole for single DOF attitude-control tests. The quadruped has a weight of 2.5 kg and the dimensions presented in Figure 3. Key design considerations include aluminum joints and carbon-foam links to ensure low weight and robustness. 80 gram aluminum paw masses equip the quadruped with the necessary inertia for using the legs as reaction masses, and constitute about an eighth of the system weight. The IB53BHP servo from AGF-RC is used for all three motors torque (2Nm) and low-cost servo with potentiometer position feedback. The motors are driven by a PWM driver and powered by a 2 cell LiPo battery. Position feedback is converted from analogue potentiometer readings via 12 ADS1115 ADCs. The Khadas Vim 3 (A311D) SBC is the central
leg. It is fast (180 RPM), light- (70 g). high-
We use the Hamilton convention of the quaternion that defines a transformation from frame B to frame W according xx qi¹ x (q""" where and denote the quaternion product and conjugate respectively.

---

control unit of the quadruped. To test the system at lower joint velocities and over extended periods of time, a stiff forked aluminum pole is mounted to an open ball bearing, such that the quadruped can be slid onto it in all principal axes, illustrated in Figure 4. The pole fixes the center of rotation and restricts attitude emrors to 2D. We therefore conduct 3D free fall experiments, shown in Figure 5, by raising the quadruped into mount designed lock it into a known initial configuration. A magnetic switch is installed to remotely release the quadruped into foam pit. The resulting configuration allows for freee fall duration of 0.7 seconds. Attitude feedback is provided through motion capture (MoCap) system for the free fall tests and through encoder for the pole setup.
Encoder
Reference
Roll
Pich
Yau
Figure 4: Rotating pole experimental setup.
Figure 5: Quadruped free fall pitch test.
Proposed Approach
We create simulated environment of the Eurepus quadruped in NVIDIA Omniverse Isaac Sim and simulate 4096 environments in parallel to collect aation-state-reward trajectories. The modified PPO implementation from [9] is leveraged to learn control policy from the collected trajectories during simulation. Two policies are learned, for slow and fast motor velocities respectively. Implementation wise, the work builds on the OnmilsaacGymEnns repository [19] for GPU-accelerated DRL.
5.1 The Reward Function
The reward function is defined weighted sum of eight reward terms, Reoal = Σ. C. R4, where the terms R, and scaling factors C; are given in Table 1. The terms split into three mostly positive reward terms and five negative penalty terms. Rí rewards closeness to the target orientation. To encourage small orientation errors, it is constructed as an exponential kernel with temperature Torien around the absolute value the error angle Zgz, where the latter was defined in Section 3. R2 provides an extra discrete reward whenever the quadruped is within threshold crien radians of the target, where corient > tuned according to the required accuracy. Finally R3 acts as a sparse reward provided only when the quadruped terminates due to success criteria of having been within Corient degrees of the target for more than half second consecutively. Its design renders it negative when the orientation error is larger than 90 degrees and positive when it is less.
i Reward/Penalty
R,
C2
Orientation Reward
exp
(Lgg)) ( /oricat)
[
[<(
Eorient
Inside Threshold Reward
otherwise.
3 Terminal Reward
|<(g))
100
2
4 Self-Collision Clamping Penalty
1
intterppl
1,
1 x 10
5 Torque Saturation Penalty
θm.) Km6m Tmax ||
1 x 10
Velocity Penalty
1
2 x 10-5
7 Torque Regularization Penalty
Tm (t) - Tm (t 1)112
3.5 X 10
-1 (m)() 0m 1) Cnoise
8 Change Direction Penalty
2 x 10-3
otherwise.
Table 1: Reward function terms and their scaling factors. denotes the angle in radians.

---

With reference to Section 5.3, R& punishes the squared norm of the difference between interpolated motor targets, 0 interpl. and the motor targets after being clamped to prevent self-collisions, 0,. Simi- larly, Rs punishes the excess torque that is clipped away during saturation at maximum torques Tima. where Kp and Ka are the parameters of the motor's interior PD controller. Both terms punish the setting of targets that are clipped or clamped in subsequent steps, and are inspired by [10]. The final reward terms Re, Ry and Rg are regularization penalties and serve the purpose of enforcing smooth policies that minimize actuator effort, to reduce motor wear and tear. Rg punishes the squared norm of the motor velocity vector and Rt punishes the squared norm of the difference between consecu- tive applied torques, while both terms serve the purpose of minimizing wear and tear. Finally, Rs is a discrete penalty given anytime the motors change direction, to enforce smooth maneuvers, where enoise > 0 a threshold included exclude numerical noise.
A few important practical considerations were necessary for successful training, Since the loss function PPO considers only the total reward Rwall the reward weights C; are chosen such that the reward terms are comparable order of magnitude. Furthermore, implementing the orientation rewards as positive rewards instead of penalties was essential avoid the local optimum of self- termination. The most important reward the orientation reward R1, and it alone suffices to achieve working policy with PPO, but increased performance is experienced with the addition of the other terms. While self-collisions of leg are prevented by explicit clamping, as will be outlined in Section 5.3, no explicit collision filter is implemented to avoid collisions between separate legs. Instead, the quadruped is reset upon the occurrence of collisions such that the accumulated reward after the trajectory smaller (total reward in general positive), thus implicitly incentivising collision- free maneuvers. Another approach that was explored was to explicitly punish collisions through negative penalties, but this led to conservative policies. The developed policy can be combined with an explicit collision filter during deployment on hardware. Finally, when training policy for lower motor velocities, we use the exact same reward function as for the unconstrained velocity problem. We find that reducing the inference frequency and extending the time horizon parameter of PPO with the same ratio as the ratio of maximum motor velocities results in successful learning. This likely stems from the fact that a trajectory will contain similar degrees of exploration.
5.2 Simulation and sim2real
A series of steps were necessary to achieve a stable and realistic simulation in Isaac Sim. Isaac Sim simulates systems of rigid bodies through articulations, a reduced-come coordinaae representation which is fully determined by root body and joint angles between all bodies. This mechanism does not support loops, which required separately modeling the ankle joint constraint. In Isaac Sim, this is achieved by excluding the joint from articulation. This constraint and the significant mass differences between the links of the quadruped render the dynamics stiff. To ensure stability of the resulting simulation model, the most important consideration was the need for low simulation time step At = 1/480s, which eight times smaller than the simulation step in [9].
A prerequisite for the successful sim2real transfer of the policy was an actuator model. The actuators are modeled by treating the internal motor dynamics as a static torque-speed mapping, Fi(()) where is the motor joint angle and Fss is a linear curve provided by the manufacturer. This simplification based on the intuition that electrical dynamics are much faster than the mechanical dynamics ô. The parameters Kp and Ka of the actuator's internal PD controller were unknown and hence identified from step response data. The resulting simplified motor model is given by:
T = L0,(KK,0,, 0) 0) K20, Tmax), where
(0).
Timxx
Table presents the sim2real techniques employed during training in preparation for deployment on the real system, where 24 and V indicate sampling from uniform and normal distributions. The significant randomization of the controller parameters stems from the simplifications used in (1) and uncertainty in the system identification procedure. The other randomization parameters originate from the inaccurate potentiometer feedback from the low-cost actuators. Factors that were less uncertain, such as the system mass and inertia, are not randomized.
5

---

Entity
Type of sim2real
Amount
Applied on
Kp Ka
Domain randomization
+(20 - 50)%
Reset
Motor positions measurement
Measurement bias
4(-2,29)
Reset
Motor positions measurement
Additive noise
(V)
Measurement time step
Motor velocities measurement
Additive noise
WO 20%)
Measurement time step
Table 2: Sim2Real variables
5.3
Policy Architecture and Deployment
Collision Clamping
Normalisation
Sampling
Scaling
Vine
02
Nu. o)
cli(% 11
(1 affaedal all
dm]
Observation
MIP
Clipping
Interpolation
Motor Position References
Figure 6: Architecture of the attitude-control policy, from observation to motor references 0,.
Figure illustrates the policy architecture. The same network is used for both the actor and critic networks of PPO. The observation o, is normalized by a running mean À and variance y which are updated during training, before being fed through three layer feed-forward neural network with layer width 128, 64 and 64, and exponential linear unit (ELU) activation. The network output sists of the value function estimate ((+.) to form the critic, and the mean and standard deviation a of a Gaussian distribution to form the actor. The distribution serves the purpose of ensuring ex- ploration during training. During deployment, the standard deviation is ignored, and the network output for the mean, denoted 9wm. is used directly in the subsequent steps of the policy. Before being applied as joint targets the motors, On is clipped to the range [-1, and mapped joint- specific motor limits [O mmxx] to form scaled targets Oscaled- These then sent through linear interpolation (2) with exponential kernel interpolation factor with temperature interpoo, where 0m are the measured joint positions, in order to reduce control efforts around the target orientation.
interrpol = (1 (0-xx)0waaa Oscated + a0m; where a = exp (-<(G))) ((
)
innterpol
Finally, the interpolated joint targets Interpol are clamped to avoid self collisions of a leg. This is implemented as a condition based on the sum Pwm = + 0g of the inner and outer transversal motors angles 0; and 0,, and acceptable range for this sum [Omin sum 0max sum). according to the following conditions, where - and + are the decrement and increment operators:
Condition
Clamping Operation
Condition
Clamping Operation
(mmax Osum)
0 + (Omin
0smm
0.um threshold
2
0sum < threshold
2
0. (0 max sum 0smm)
e. (Amin sum
6 0amm)
2
If a policy is trained deployed at reduced motor velocities, smooth commanded target sequence 0, is achieved by sending the clamped and interpolated motor targets 0,, through the saturated second order reference model given by (3). Ve denotes the commanded velocity, and the absolute damping factor Ç and natural frequency wh are filter tuning parameters. second order reference model advantageous as it generates smooth velocities in addition to smooth position signals.
è_ = sat(
(sgn(x) ( if |x
x max
where
sat(x)
(3)
s. + 2(wn/ + w.O0 = 220,
x
else

---

Evaluation Studies
To evaluate the performance of the learend policy Eurepus, tests were conducted both per DOF on rotating pole and during free fall 3D maneuvering. A comparison with other control approaches and specifically with Model Predictive Control is presented in the supplementary materials.
6.1 Rotating Pole
We evaluate a 3D policy trained with joint velocities below 300% through (3) against a time varying reference on the rotating pole setup. Step responses for roll, pitch and yaw are given in Figure 7. Figure presents an ablation study of the paw masses. The policy successfully follows the reference for all DOFs, validating the test setup. The spike in yaw indicates that the policy has learned to sometimes take the longest path to reference, and the degradation in the pitch DOF after 60 seconds originates from leg collision and the corresponding accumulated angular velocity.
A key finding is that the 3D policy trained in pole-less environment and with variations of paw masses is robust to the effects of the pole well as variations of the paw mass. This indicates that the learned feedback policy has not overfitted to the system dynamics and justifies the simple feed-forward network structure as opposed to an adaptive approach through for instance recurrence in the policy. A potential drawback of the policy excessive control effort and oscillations at the reference. Possible reasons for this are noise in joint measurements, which directly transmits to the actuators through the linear interpolation (2), inadequate motor model (I) leading to excessive actuation slower dynamics the real system, and potential collisions of the legs with the pole.
100
30
Azpe (0og)
Axgle (dag)
1000
10
20
30
50
63
70
80
10
20 30
40
50
60
70
130
Time
Time (6)
Figure 7: 3D policy response to references in yaw, pitch and roll on the real system.
Figure 8: 3D policy roll response for differ- ent paw masses. Policy trained 120 g.
6.2 Free Fall
We demonstrate attitude control of the quadruped through range of free-fall experiments from height of about 3.5 meters and landing into foam pit. Figure 9 displays the simulated and real response of the error quaternion decomposed into Euler angles (ZYX convention) for 45 degrees setpoint in roll, pitch and yaw. Each experiment is repeated thrice and the plot indicates the mean and variance of the response to demonstrate repeatability. The target orientation is reached in less than 0.4 seconds for yaw and pitch, and 0.6 seconds for roll. Next, Figure 10a demonstrates non- principal axis attitude control by applying a simultaneous degrees reference in roll, pitch and yaw. Finally, we show that the policy is capable of controlling 80 degrees of orientation during the free fall time of 0.7 seconds by applying large reference steps in pitch and yaw, 10b and 10c. Extrapolating the results yields an estimated acheivable average attitude velocity of 110 degrees per second. Frames from recording of selected experiments are provided in Figure 1.
in Figures
Compared with the simulated trajectories, the real responses demonstrate robust sim2real transfer, but are simultaneously characterized by slower transient and oscillations at the reference orienta- tion. The main contributing factors to this are a) a time lag before a measurement received from the MoCap system as well as b) a higher time constant in the real motor control loop than in the sim- ulation, experimentally identified to 50% higher for the real actuators. The former is estimated be 0.03 0.05 seconds, and such time lags can cause oscillations in feedback control systems dealing with fast dynamics in Eurepus. The higher time constant in the actuators likely stems from the static approximation of the internal motor dynamics employed in the simplified motor model (1).
7

---

60
Yaw
Y Yaw
Pitch
40
Roll
40
Angle (deg)
Angle (deg)
Angle (deg)
20
20
-20
.40
Pitch
Roll
-60
(s)
Time (s)
Time (s)
(a) Yaw (simulated)
(b) Pitch (simulated)
(c) Roll (simulated)
60
60
20
Yow
Pitch
Pinch
40
Rall
40
Roll
Angle (deg)
Angle (deg)
Angle (deg)
-40
Yaw
Pitch
-20
-20
-60
0.2
0.4
0.6
0.2
0.4
0.6
0,2
0.4
0.6
Time (s)
Time (s)
Time (s)
(d) Yaw (real)
Pitch (real)
(f) Roll (real)
Figure 9: Euler angle errors for simulated and real robot for 45 degrees attitude steps.
10
120
180
100
160
Angle (deg)
Angle (deg)
Angle (deg)
80
140
-20
Y
60
120
Pitch
Roll
-40
40
100
0.2
0.4
0.6
0
0.2
0.4
0.6
0.2
0.4
0.6
Time (s)
Time
Time
(a) 30 degrees roll, pitch and yaw (b) 120 degrees pitch reference (c) 180 degrees yaw reference
Figure 10: Selected results for free fall tests on the real robot. Plot (a) displays the decomposed error quaternion, and plots (b) and (c) display the absolute angle error.
7 Conclusions, Limitations and Future Work
We demonstrate fast attitude control on custom-built low-cost quadruped with five-bar-linkage leg design rotating pole test setup lower speeds and in free fall. Tuning the horizon and inference frequency of PPO to replicate another problem with faster dynamics proved to be an effective way to relearn a problem for slower dynamics. Limiting assumptions of the work are disturbance-frre environment and the set- pontt--t-rrgnl problem formulation. Future work should consider more realistic extraterrestrial conditions and also address the velocity-trackinn problem. Key limitations of the proposed approach are the simplified motor model and the possibility of leg collisions. This should be addressed through better actuator models and an explicit safety collision filter. Before operational system deployment, considerations should be made to whether the leamned attitude-control policy efficiently prepares the quadruped for landing.

---

Acknowledgments
This work was partially supported by the Research Council of Norway under Grant Agreement 338694. The authors extend their gratitude to Michalis Papadakis for providing an MPC baseline the supplementary material. We would further like to thank the reviewers for their time and comments, and for improving the quality of the paper.
References
[1] A. Daga, C. Allen, M. M. Battler, J. D. Burke, I. A. Crawford, R. J. Léveillé, S. B. Simon, and L. T. Tan. Lunar and martian lava tube exploration as part of an overall scientific survey. White
Paper submitted to the Planetary Sciences Decadal Survey 2013-2022, 2022.
[2] M. Zarei and R. Chhabra. Advancements in autonomous mobility of planetary wheeled mobile robots: A review. Frontiers in Space Technologies, 3, 2022. ISSN 2673-5075. doi:10.3389/
rist.2022211802211
[3] J. Balaram, M. Aung, and M. P. Golombek. The ingenuity helicopter on the perseverance rover. Space Science Reviews, 217(4):56, 2021.
[4] B. Dynamics. Spot the agile mobile robot, 2024. URL http::://///ttt http:::://www.bbstoddlnaaaaccce bow.onnnnddddi..s... com com/products/spot.
[5] M. Hutter et al. Anymal highly mobile and dynamic quadrupedal robot. In 2016 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 38 48-44, Oct 2016.
[6] Y. J. Ma, W. Liang, H.-J Wang, S. Wang, Y. Zhu, L. Fan, O. Bastani, and D. Jayaraman. DrEureka: Language Model Guided Sim-To- Real Transfer. github. io/ to/dr-eureka, 2024.
[7] M. Trentini, P. Arm, G. Valsecchi, H. Kolvenbach, and M. Hutter. Concept study of small- scale dynamic legged robot for lunar exploration. In IAC 2023 Conference Proceedings, page 78250. International Astronautical Federation, 2023.
[8] Y. Johnson, M. S. Parvathy Thampi, A. Das, and M. Binz. Attitude control of satellites with reaction wheels and its formation keeping. In T. Senjyu. C. So-In, and A. Joshi, editors, Smart Trends in Computing and Communications, pages 279-292, Singapore, 2024. Springer Nature Singapore. ISBN 978-981 197-9999----
[9] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning, 2022.
[10] N. Rudin, H. Kolvenbach, V. Tsounis, and M. Hutter. Cat-like jumping and landing of legged robots in low gravity using deep reinforcement learning. IEEE Transactions Robotics, 38 (1):317-328, 2021.
[11] H. Kolvenbach and K. Wormnes. Recent developments orbit, a 3-dof free floating contact dynamics testbed. 01 2016.
[12] V. Kurtz, H. Li, M. Wensing, and H. Lin. Mini cheetah, the falling cat: A case study in machine learning and trajectory optimization for robot acrobatics, 2022.
[13] A. Spiridonov, Buehler, M. Berclaz, V. Schelbert, Geurts, Krasnova, E. Steinke, Toma, J. Wuethrich, R. Polat, W. Zimmermann, P. Arm, N. Rudin, H. Kolvenbach, and M. Hutter. Spacehopper: A small-scale legged robot for exploring low-gravity celestial bod- ies, 2024.
[14] A. Olsen and K. Alexis. Martian lava tube exploration using jumping legged robots: A concept study. In 74th International Astronautical Congress (IAC), Baku, Azerbaijan, October 2-6 2023. IAF.

---

[15] V. R. Kamidi, A. Williams, and P. Ben-Tzvi. A framework for modeling closed kinematic chains with a focus on legged robots. 2018 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), Madrid, Spain, October 1-5 2018.
[16] L. C. Tokuz Dulger and S. Uyan. Modelling, simulation and control of four-bar mechanism with brushless servo motor. Mechatronics, 7(4):369-383, 1997. ISSN 0957-4158. Received 9 January 1996; revised 10 January 1997; accepted 29 January 1997.
[17] S. Dubowsky and E. Papadopoulos. The kinematics, dynamics, and control of free-flying and free-floating space robotic systems. IEEE Transactions on Robotics and Automation, 9(5): 531-543, 1993.
[18] S. Hofer, K. Bekris, A. Handa, C. Gamboa, M. Mozifian, al. Sim2real in robotics and automation: Applications and challenges. IEEE Transactions on Automation Science and En- gineering, 18(2), 2021.
[19] NVIDIA. Omniverse isaac gym reinforcement learning environments for isaac sim. https: wwtp://apooooo/////////. wttpyyapppsRLROmnnnIaaaaaaa hts//OmmiisaaacyeEEEvv.s.
10

---

Appendix Supplementary Material
1 Baseline Comparison with Model Predictive Control (MPC)
To motivate the use of DRL for solving the attitude-control problem, and to evaluate the learned policy's performance against classical control methods, we present the results for model predictive control (MPC) approach to the attitude-control problem on Eurepus.
The implemented controller features cascade structure, with an outer torso position controller to calculate the needed net torque, an inner torque-tracking controller to plan the leg motions of single leg, and finally, symmetry-basee allocation strategy to map the leg motions to the remaining three legs in collision-fiee manner. Furthermore, the controllers (FSM) which changes between a torque-applying stage and predefined resetting stage when the legs have reached the endpoints of their range of motion. An important implication of the enforced symmetry in the leg manoeuvres is that only principal-axio rotations (yaw, pitch, roll) are possible. Hence, the control approach achieves non-principal-aric rotations by regulating one DOF time, in order of decreasing regulation error. This switching also handled by the FSM. The described approach represents one possible avenue design MPC solution the problem.
encased in finite state machine
The controller is implemented with the simulation model of Eurepus and tuned for regulation per- formance. The performance of the controller against the DRL-based policy is presented in Figure when given a 45-degree reference in yaw, pitch and roll. One central reason for the inferior perfor- mance as compared to that of the DRL-based policy that the simplifications made in the controller restrict the space of available manoeuvres at any time. Such measures include in particular the sym- metry restriction and the enforcing of an explicit retraction and resetting stage through the FSM.
While the disadvantage pointed out above is implementation-speciiee it generalizes to classical op- timal control approaches. While DRL directly optimizes the control task's objective, the optimal control approach often needs to split this objective into the sub-problems of planning and control (primarily for feasibility and computational tractability reasons), and the restrictions imposed by this can limit the range of behaviour significantly !
Several other factors put the MPC-based method at a disadvantage to the DRL-based policy. A well- known disadvantage of such receding horizon control methods is their computational cost which is typically high. Specifically, the computation time of the implemented MPC is about 13 ms, which to be compared with the inference time of 0.06 ms for the DRL-based policy network. This additional processing time in the control loop has a detrimental impact both on the computational resources used and the achievable control frequency during deployment. In particular, the control frequency of 80 Hz used for the DRL-based policy is not achievable with the MPC. This is because the latency in the other parts of the real-time control loop already comprises about 8 ms. Adding the computational time of the MPC results in maximum achievable control frequency of 50 Hz. reduction in control frequency is a disadvantage because of the particularly short gait during the free fall experiments and the requirement for extremely fast actuation in the target application.
Y. Song, A. Romero, M. Müller, V. Koltun, and Scaramuzza. Reaching the limit in ing: Optimal control versus reinforcement learning. Science Robotics, 8(82), Sept. 2023. ISSN 2470-9476. 10.:10.1 126//ccroboticc.adgee. 1462. URL http://dx..oo.org////// 10 126/scirohottessaag14...

---

Finally, tuning of the MPC is no trivial task, due to the cascade structure and number of parameters. While tuning is also non-trivial for the proposed DRL-basedd method, we find the required tuning efforts comparable, such that ease of tuning not an advantage of the MPC-based approach.
60
Pitch
40
Roll
Angle (dog)
DRL
MPC
-20
2
4
Time (s)
(a) Yaw
60
Yaw
Pitch
Roll
Angle (deg)
DRL
MPC
-20
0.5
1.5
2.5
Time (s)
(b) Pitch
20
Aggle (deg)
-40
Yaw
Pitch
-60
Roll
-80
DRL
MPC
-100
Time (s)
(c) Roll
Figure 1: 45-degree setpoint regulation performance for DRL-based policy (dashed line) and MPC-
based controller (solid line) in simulation. The plot shows angle errors.

---

2
Motor Model Ablation
Tow
Part
Pic1
Pirtt
Firt
Rell
3el
Axgle (blg)
(bbg))
Augle (tteg)
Azgle (deg))
20
-20
-20
-20
0.2
0.6
0.2
0.6
0.1
64
0.6
69)
(a) Kp
(b) 2 Kp
(c) 5 x Kp
(d) 10 X Kp
Figure 2: Response to a 45 degrees reference in pitch for different motor dynamics, where Kp has been modified.
To assess the impact of the motor model on the performance of the policy, we repeat a 45-degree pitch experiment with four different configurations of the Kp gain of the motor controller sim- ulation (Figure 2). K is the nominal proportional controller gain and is the one employed during training. The ablation demonstrates degraded performance from changes to the gain indicating the importance of an accurate motor model. We further note the similarity between the degraded per- formance with the real experimental results of Figure 9d-9f in the paper, which may highlight the merit of further effort into motor modelling, through for instance an actuator network.
3
Angular Velocity Tracking
100
100
100
80
Pilch
Pitch Roll
Fiizh
Roll
Angle (deg)
60
Rel
Angle (deg)
60
Aggle (deg)
40
40
20
20
0
-20
-20
-20
2
0
2
Time (s)
Time (s)
Time (s)
(a) Yaw
(b) Pitch
(c) Roll
Figure 3: 30 degrees per second velocity tracking performance of the attitude control policy in yaw, pitch and roll. The reference angle is indicated by a dashed line.
To assess the suitability of the developed policy to velocity tracking problems, we evaluate the ve- locity tracking performance to ramp references with a rate of 30 degrees per second in all principal axes (Figure 3). The velocity evaluated is motivated by the estimates of the angular velocity fol- lowing a jump derived related work2. The oscillation observed around the target is result of the interpolation step of the policy. However, tracking is demonstrated in yaw and pitch. Inferior track- ing is observed in roll, although velocity of 30 degrees per second in roll is more than what can reasonably be expected from normal jump according to ? This further clarifies the applicability of the learned policy to offset any initial angular velocity from jump.
AA. Olsen and K. Alexis. Martian lava tube exploration using jumping legged robots: A concept study. In 74th International Astronautical Congress (IAC), Baku, Azerbaijan, October 2-6 2023, IAF.

---

Training Times
We present in Table I the training times for the two developed DRL-based policies. The GPU used is the NVIDIA GeForce RTX 3090, and the GPU memory usage was 4GB out of the available 24 GB RAM. The training time for the vow-mitor-velocity policy for the pole experiments is slower due to the inherently slower dynamies. Furthermore, training times are generally slower than similar related work using GPU-parallelisatonn.. This is mainly attributed to the need for extremely small simulation time steps due to stability issues of the Closed Kinematic Chain (CKC) simulation.
Table 1: Training time and epochs until convergence for trained policies
Policy
Training time (hrs) Number of Epochs
3D pole (slow)
7.0
800
3D free fall (fast)
2.6
1500
5
Reduced Reward Function
20
60
20
Pitch
40
Roll
Angle (dog)
Angle (deg)
Angle (deg)
-20
20
-20
-40
Y aa
-40
am
Pitch
Pitch
Roll
Roll
-60
-20
-60
0.5 Time (s)
0.5
0
0.5
Time (s)
Time (s)
(a) Yaw
(b) Pitch
(c) Roll
Figure 4: 45-degree step responses where the policy is trained with only the three first rewards in
Table 1 in the manuscript.
We train and evaluate the performance of a policy trained with a reduced reward function comprising only the first three reward terms, Ruval C2 R1 + C2R2 C3 The five terms omitted correspond to the negative penalty terms of the original reward function. The scaling factors are not altered.
The results, in simulation, of 45-degree setpoint following in roll, pitch and yaw are provided in Figure 4. The results show that successful attitude control policies can be trained with the reduced reward function, which demonstrates that the main task objective is sufficiently represented by only three terms. An important observation is the oscillatory response when compared to the responses presented in the work. This is a natural consequence of the removal of the penalty terms which reduce actuator wear and tear (reward terms Rs, Rg and R7) and directly punish oscillatory motor action (reward term Rg).
6 Hardware Design
The proposed robotic platform is designed to be modular, and all parts can be 3D printed. CAD and STL files will be open-sourced, such that the robot can be replicated easily.
As is illustrated in Figure 6, the base of Eurepus consists of three main plates. The four legs slide onto the corners of a central base plate through the four lateral motor mounts. The electronics are
N Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning, 2022.
iv

---

SBC Khadas VIM
Power divider
PWM driver
P
Battery
12C mux
ADC
Top view
Bottom view
Figure Picture of the electronics on Eurepus.
screwed to the upper and lower electronics plates and fit top of the lateral motor mounts. The space between the base plate and the electronics plates is occupied by configuration of rails. These, along with the parallel holes in the centre of each plate, are designed to allow a forked pole to be slid through the quadruped all three principal axes. To protect the electronics on the upper plate and the ADCs mounted on top of the lateral motor mounts, a base cage and ADC cage are fitted over these. The entire configuration is fastened through three screws in each corner of the plates.
Each leg consists of lateral motor and two transversal motors. The transversal motors fit snugly into a transversal motor mount that is mounted directly to the rotational axis of the lateral motor. The two thighs are mounted to the transversal motors with spacer plates to provide additional room when the legs traverse over the base. The leg itself is held together by parts designed to clamp to thigh or shank on one end and connect in a ball-bearing joint on the other end. The paw clamp at the bottom of the leg contains additional fastening mechanisms for the paw mass.
Finally, present annotated picture of the electronics on Eurepus in Figure 5 to highlight how they fit in the mechanical design.

---

Carbon fiber foam sandwich, 10mm
Carbon fiber, 2mm
POM, machined
Resin/PLA,, 3D printed
Aluminium, machined
Electronics
3
10
14
16
15
12
17
18
20
22
Base cage
7
Base plate
13
Lateral motor
Lower electronics
2 PWM driver
8
plate
4
Transversal motors
19
Shank clamp
3
12C mux
9
Battery with mount
15
Spacer plates
20
Shank
Transversal motor
4
Khadas VIM 3
10
ADC cage
16
mount
21
Paw clamp
Upper electronics plate
5
3 x ADC
17
Thigh
22 Paw weight
Lateral motor mount
6
Base rails
12
18
Thigh clamp
Figure 6: Exploded view of the Eurepus quadruped design.
vi


Reviews:
[{"review_id": "wdWF9XpUPf", "reviewer": "Reviewer_dUvT", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper describes the design of a fast attitude controller for a customized quadrupled robot platform. The attitude controller is learned using PPO via simulated experiences using Isaac Sim. The policy is transferred to real hardware in two tests: free falling and  pole rotating using domain randomization. The learned policy seems to work well on hardware.", "summary_of_recommendation": "The paper is a nice technical write-up but lacks novelty to become a science paper as limited new insights are provided.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- If possible, please add more comparisons to baselines, especially with classical controllers, to validate the need for using reinforcement learning in this setting\n- In Fig. 9 and Fig. 10, I assume the errors are plotted, but it isn’t clear from the description text and the figures’ caption. Also, in Fig.9 and Fig. 10, why is only one trial plotted? Also, the same for Fig.7 and Fig. 8.\n- Can the authors clarify why the actuator model is important for successful sim2real transfers? Also, where is the actuator model in Fig. 6?\n- Can authors elaborate on the difference between the regulation and velocity tracking problems and the paper was formulated as a regulation problem?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "emHVHrS2Fx", "reviewer": "Reviewer_82XZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to use reinforcement learning (RL) for in-flight attitude control of a quadrupedal robot. The setup is discussed in detail, and both simulation and real-world experiment results are presented.", "summary_of_recommendation": "Despite the lack of ground-breaking novelty, I think the paper provides useful insights and could be accepted after revisions.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "I would like the authors to answer the following questions in the rebuttal:\n* In section 5.1 (Line 112), the authors mention that \"two policies are learned\" for slow and faster motor velocities. What is the motivation behind the two-policy design? Which policy is used for the evaluation results in Section 6? How do you determine which policy to use, and how do you handle the switch between policies?\n\n* Is the result presented in section 6.1 in simulation or in the real world?\n\n* How would the proposed method compared with the results in [10]?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FvNfun6eRD", "reviewer": "Reviewer_LWaG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A modified PPO for quadruped robot in in-flight attitude control", "summary_of_recommendation": "More detials of real-world experiment and some comparsion with RL baselines could further improve this paper. It is also suggested to explain the proposed method as one unique approach for the target task rather than just a modified PPO.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "More information related to the hard ware and RL setting should be added. More figures related to real-world task are welcome.\nWhy the original PPO in [9] can not be directly used in this task? The impact of the Modified PPO to the target task should be carefully analyzed.\nIt is suggested to discuss the potential of the proposed method to pure real-world task without relying on simulation and sim2real. Also the generalization ability of the proposed method when the real hardware and simulation are mismatch is worth discussing.\nIt is also suggested to add comparison with other RL baselines to further demonstrated the advantage of the proposed method.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: # Strengths:
Multiple reviewers acknowledge the writeup clarity, effort in the experiments, and the detailed insights presented in the paper

# Weakness:
Reviewers are failing to recognize the contribution of the proposed work beyond well-known techniques (domain randomization, sim2real, etc) in legged locomotion. Lack of baseline comparisons has also been raised.

# Recommendations:
It is recommended to improve the positioning of the paper. Authors are encouraged to clarify the contributions made by the paper and improve baselines and ablations. Not necessarily, all papers need a novel method. Often well-known techniques applied in interesting ways move the needle in a tough problem. Leverage improvements in baselines and ablations to highlight such contributions and strengthen the submissions.

# Rebuttals
The authors added new experiments and clarified concerns raised by the reviewer. Two authors increased their scores. Reviewers are split on this paper - two in favor and one not in support. Based on the points raised by all reviewers, clarity of presentation, experimental details, and the contributions the paper will make to the field AC recommends acceptance of the paper. Authors are strongly advised to make a best care effort to address all comments from all reviewers, especially dUvT.
Decision: Accept