Tuning Legged Locomotion Controllers via Safe Bayesian Optimization
Authors: Daniel Widmer
Dongho Kang
Bhavya Sukhija
Jonas Hübotter
Andreas Krause
Stelian Coros
Venue: CoRL 2023 Poster (2023)
Keywords: Legged Robotics
Bayesian Optimization
Controller Tuning
Locomotion
Machine Learning
Safe Learning

Abstract: This paper presents a data-driven strategy to streamline the deployment of model-based controllers in legged robotic hardware platforms. Our approach leverages a model-free safe learning algorithm to automate the tuning of control gains, addressing the mismatch between the simplified model used in the control formulation and the real system. This method substantially mitigates the risk of hazardous interactions with the robot by sample-efficiently optimizing parameters within a probably safe region. Additionally, we extend the applicability of our approach to incorporate the different gait parameters as contexts, leading to a safe, sample-efficient exploration algorithm capable of tuning a motion controller for diverse gait patterns. We validate our method through simulation and hardware experiments, where we demonstrate that the algorithm obtains superior performance on tuning a model-based motion controller for multiple gaits safely.

Tuning Legged Locomotion Controllers via Safe Bayesian Optimization
Daniel Widmer'
Dongho Kang*
Bhavya Sukhija
Jonas Hübotter
Andreas Krause
Stelian Coros
ETH Zürich
fuidmdani, kangd, sukhijab, jhuebotter, krausea, acoros)@ethz.co ch
Abstract: This paper presents data-driven strategy to streamline the deployment of model-based controllers in legged robotic hardware platforms. Our approach leverages model freee-free safe learning algorithm to automate the tuning of control gains, addressing the mismatch between the simplified model used in the control formulation and the real system. This method substantially mitigates the risk hazardous interactions with the robot by sample-efficienly optimizing parameters within a probably safe region. Additionally, we extend the applicability of our approach to incorporate the different gait parameters contexts, leading to safe, sample-efficient exploration algorithm capable of tuning motion controller for diverse gait patterns. We validate our method through simulation and hardware experiments, where we demonstrate that the algorithm obtains superior performance on tuning a model-based motion controller for multiple gaits safely.
Keywords: Legged robot, Bayesian optimization, Safe learning, Controller tuning
Introduction
A model-based control strategy facilitates quick adaptation to various robots and eliminates the need for offline training, thereby streamlining the design and test phases. However, it requires an accurate dynamics model of the system, which is often unavailable due to our limited understanding of real-world physics and inevitable simplifications to reduce the computational burden. As result, these controllers typically underperform on actual hardware without considerable parameter fine-tuning. This tuning process is not only time-consuming but can also harm the hardware platform. Additionally, it often requires reiteration for diverse environments movement patterns.
This work explores the challenge of determining optimal control gain parameters for model-based legged locomotion controller. doing so, we aim bridge the disparity between simplified models and actual hardware behavior, consequently improving the controller's robustness and tracking accuracy. To this end, we employ safe learning algorithm,, namely GOSAFEOPT [1] automate the parameter tuning process, enabling the online identification of optimal control gain parameters extend GOSAFEOPT by incorporating various gait parameters as contexts [2]. This facilitates more sample-efficient learning of control gains tailored for distinct gait patterns and allows for fluid online adjustments of the control gains during operation.
within a safe region. Furthermore,
We demonstrate our method on the quadruped robot Unitree Gol [3] in both simulation and hardware experiments. In our simulation experiments, show that contextual GOSAFEOPT outperforms other model-free safe exploration baselines while ensuring zero unsafe interactions. Moreover, when trained across varied gait patterns, the experimental results clearly indicate that our contextual GoS AFEOPT delivers considerable performance boost. Moving to our hardware experiments, contextual GOSAFEOPT finds optimal feedback controller gains for both frot and crawl gaits in only
* TTheee authors contributed equally.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

50 learning steps, all while avoiding any unsafe interaction with the real robot. The resulting controller gains, together with our model-based controller, ensure robust legged locomotion against perturbations and environmental uncertainties. In addition, our tests reveal that GOSAFEOPT can effectively suggest reasonably good controller gains for previously unseen gait patterns such as flying trot and pronk.
In summary, (i) we formulate the problem of safe control parameter tuning for model-based legged lo- comotion controller as constrained optimization, (ii) we extend GOSAFEOPT to account for contextual scenarios while providing theoretical safety and optimality guarantees, (iii) we demonstrate the superi- ority of contextual GoS AFFOPT over other state-of-the-art safe exploration algorithms in supporting diverse gait patterns, and (iv) we show that our method successfully and safely tunes control gains on the hardware and enhances the robustness and tracking performance of the controller significantly.
2 Related Work
Bridging the reality gap in legged locomotion tasks Several previous studies have emphasized the importance of considering an actuator behavior and identifying the system latency to successfully bridge the reality gap in legged robot systems [4, 5, 6]. These studies develop a simulation model of a legged robot system incorporating either modeled or learned actuator dynamics and train a control policy that can be effectively deployed to the robot hardware.
Incorporating this strategy into a model- based control framework is an area of active investigation. Rather, in the context of model-based control, it is typically more straightforward to introduce adjustable control gain parameters and fine-tune them to align with the real-world behaviors of the robot. For instance, Kim et al. [7] use joint position- and velocity-level feedback to joint torque command in order to address any discrepancy between the actual torque output and the intended torque command for robots with proprioceptive actuators [8]. However, the fine-tuning of these parameters continues present significant challenge. Schperberg et al. [9] utilize the unscented Kalman filter algorithm to recursively tune control parameters of model-based motion controller online, and they successfully demonstrate it on the simulated quadrupedal robot in the presence of sensor noise and joint-level friction. However, their proposed tuning method is inherently unsafe and can therefore lead to arbitrary harmful interactions with the system. In contrast, our method aims to optimize control gains while avoiding any unsafe interactions with the robot hardware.
Safe exploration for controller parameter tuning Training controller directly on hardware a challenging task, as it requires sample efficient and safe exploration to avoid possible damage to the robot. In such settings, Bayesian optimization (BO [10]) emerges as a suitable framework due to its sample efficiency. A notable example in the field of legged robotics comes from Calandra et al. [11], who successfully employed BO to learn optimal gait parameters for a bipedal robot platform.
BO methods can be easily adapted to constrained settings for safe learning. Gelbart et al. [12), Hernández-Lobato al. [13], Marco et al. [14] utilize constrained BO for finding safe optimal controller parameters. However, these works do not provide safety assurance during exploration. In contrast, methods such as SAFEOPT [15, 16] and its extensions [1, 17, 18, 19] guarantee safety throughout the entire learning and exploration phases. SAFEOPT leverages regularity properties of the underlying optimization to expand the set of safe controllers. This expansion is inherently local, and accordingly SAFEOPT can miss the global optimum. For dynamical systems, Baumann et al. [19] introduced GOSAFE, a global safe exploration algorithm which, unlike SAFEOPT, is capable of identifying the global optimum. However, the BO routine proposed in GoS is expensive and sample inefficient which limits the scalability of the method. To this end, Sukhija et al. [1] introduce GoSAFEOPT AOSAFEOPT leverages the underlying Markovian structure of the dynamical system to overcome the GOSAFE's restrictions. As result, it can perform global safe exploration for realistic and high-dimensional dynamical systems.
In this work, we extend GoS to a contextual setting and apply it to systematically tune the model-based controller of quadruped robot for various gait patterns. Our proposed method not only guarantees safety and global optimality but also scales effectively to systems with relatively nigh-dimensional search space that involves forny--fourdiimnniinnall state space, six-dimensional parameter space, and five-dimensional context space.

---

3
max Problem Setting
Safe learning formulation The dynamics of robotic systems can generally be described as an ordinary differential equation (ODE) of the form = f(s, u) where u E U C. R is the control signal and & E S C IRd. is the state of the robot. Due to the reality gap, disparities can arise between the real-world dynamics and the dynamics model f. This often results in a significant divergence between the behaviors of models and actual real-world systems, thereby making the control of intricate and highly dynamic systems like quadrupeds particularly challenging.
common solution to this problem is using a feedback policy to rectify the model inaccuracies. Given a desired input signal u² desired state s", and true system state 8, we formulate a parameterized feedback control policy in the form u To (u" ss" s)) that steers to closely align with s*. The parameters 0 are picked minimize the tracking error. common example of such a feedback policy is PD control, where Typically, choosing the parameters 0 involves a heuristic process, requiring experimental iterations with the physical hardware. However, such interactions can be unpredictably risky and could possibly cause damage to the hardware.
u* ((s s). where 0 E I2 xd, corresponds to the controller gains.
In this work, we formalize the tuning process as constrained optimization problem:
g(0) such that g: (0) 2 0, Vi E I,, GE
where g is an objective function (or reward function), qi constraints with Ig {1,.... c}. and O is compact set of parameters over which we optimize. Since the true dynamics are unknown, we cannot solve Equation (1) directly. Instead, we interact with the robot to learn g(0) and g,))), and solve the optimization problem in a black-box fashion. As we interact directly with the robot hardware, it is important that the learning process is sample-efficient and safe, i.c., constraints q, are not violated during learning.
Extension contextual setting Our goal is to find optimal control gains specific to individual gait patterns and facilitate seamless online transitions across various gaits. Each gait pattern demonstrates unique dynamic properties. Therefore, the optimal feedback parameters O vary depending on the gait pattern in question. We consider gaits as contexts from a (not necessarily finite) set of contexts Z [2]. Contexts are essentially external variables specified by the user. We broaden our initial problem formulation from Equation (1) to accommodate these contexts;
g) such that ge (0, z) 2 0, V, E Iyy
o0e
where z E Z is the context, which in our scenario, is the parameters of the gait of interest.
Assumptions We reiterate and discuss the assumptions for GOSAFEOPT [1] in Appendix A. To summarize, we assume the following: (i) an initial safe set of parameters is known, (ii) the objective and constraints lie in a reproducing kernel Hilbert space with bounded norm, (iii) measurement noises are i.i.d. sub-Gaussian, (iv) the control frequency sufficiently high to capture the state evolution, and (v) constraints (0, 2) can be defined as the minimum of state-dependent function (0, 8, along the trajectory starting in so with policy ne-
4
Control Gain Optimization for Model-based Legged Locomotion Control
4.1 Control Pipeline
Model-based locomotion controller Our locomotion controller utilizes a combination of the model predictive control (MPC) and the whole-body control (WBC) method following the previous work by Kim et al. [7]. Kang et al. [20], and Kang et al. [21]. The MPC generates dynamically consistent base and foot trajectories by finding an optimal solution of a finite-horizon optimal control problem, using a simplified model. To convert these trajectories into joint-level control signals, we implement a WBC method that incorporates a more sophisticated dynamics model and takes into account the physical con- straints of the robot. More specifically, use WBC formulation similar to the one presented by Kim et al. [7]. This method calculates the desired generalized coordinates x", speed ă" and acceleration
3

---

Inining Line
Safi: Reyosion optimization
Robot stase olocrvation
Conrrl grim
Control gain tuner
Ru,Ay
Locoenctons controller
Gak Gait planner
úiszlisz
Base
MPC
WBC
targnee
controller
stopats
Robot
Beay velecity commrnd
Food
Figure 1: Overview of the system. The control gain tuner determines the optimal gains kp. ka for the locomotion controller given gait parameters 2 as a context variable. In order to learn the map between the optimal gains and context variable, use safe Bayesian optimization algorithm, which finds optimal gains by minimizing the mismatch between desired joint states s* and actual joint states & while ensuring no safety breach during the learning process.
" on kinematic level while respecting task priority via the null-space projection [22]. Subsequently, it finds the desired joint torques T" by solving quadratic program that aligns with the desired general- ized acceleration, adhering to the motion equations of the floating base and other physical constraints. For more detailed explanation of the WBC formulation, the reader is referred to Appendix D.
We emphasize that the feed-forward torque commands T² by themselves fail to produce the desired motion on the robot hardware due to model discrepancies. Particularly, we observed the actuator dy- namies and joint friction, which are impractical include in the system model, contribute significantly to this model mismatch. As practical solution, compute the final joint torque commands --dd T" + hpp - x)- + la á:) with the feedback gains kp Rd, xdx and ka E Rdd xdz and send them to the robot. Here, 2, ē represent the joint angles and speeds (we use to represent the concate- nated vector of æ and Ø), while x", #" denote their desired values. We treat the feedback gains k and ka as the parameters that we want to optimize using data samples collected from hardware directly.
Gait parameterization We parameterize quadrupedal gait pattern with = [d9, , tg of, og, og], where d is the duty cycle for gait g, tg is the gait duration, and of are the phase offsets of legs two to four respectively, starting counterclockwise with the rear left leg. The duty cycle is defined the contact duration divided by the stride duration. In general, the optimal feedback parameters (k) kz) change with the gait. We show this empirically in Section 5.
4.2 Contextual GOSAFEOPT
We model the unknown objective and constraint functions h(,, () (i for the objective, E Iy for constraints) through Gaussian Process regression [23]. To this end, given a dataset (u,, Vilisn with 2 (0). zz) and the kernel k, we calculate mean and uncertainty estimations of h(, i):
pn (v. i) k, (K, 12 Bnnt.
(3)
i) k(v, -x,
+ +021)-"k,,00
where Un. = B are the observations of h(;,2,, (v) = [k(u, w,)] and K = [k(u,, et),;,s is the kernel matrix. We leverage these estimates to provide high- geobbbobybyllty frequen- tist confidence intervals.
Lemma (Confidence intervals, Theorem of [24] and Lemma 4.1 of [16]). Let h be defined as h(0, 2, (9)02)) 2)
i
(0,z) E
For any 5 E 1) and under Assumptions and 3 from Appendix A, with probability at least 5 holds jointly for all n, i, z, 0 that
|0,0,, pn (0, 2, 1)1 S (8) ox (0, 2,
(5)
with ( 5(B 40 (T//z 11 log(1/0)) where
7n
max
I(yA: hA).
(6)
Ace x III Al

---

Here, I(yA; ha) denotes the mutual information between IΛ = [h(+)]vEA. if modeled with GP, and the noisy observations y at ha- It quantifies the reduction in uncertainty about h upon observing YA at points A. The quantity Yhu is a Bayesian construct, however, in the frequentist setting quantifies the complexity of learning the function h. is instance-dependent and can be bounded I and kernel function (see Appendix B).
depending on the domain θ
Given the confidence interval from Equation (5), we define confidence set for each context parameter & and index 0 v i S as
00 oo| 1000, 00, oc]
if E So(z) and i 2 1,
Co(O, i)
otherwise,
(,, - (0, 2,
(0, 2, + (o) (0,
(8)
We refer to (0, min C. (0, 2, as the lower bound, un (0, z, max C, (0, upper bound, and wi, (0, 2, Un (0, 2, L, (0, 2, the width of our confidence set.
z, the
4.2.1 Algorithm
Given (user-specified) context Zn E 2, episode of contextual GOSAFFOPT is performed in one of two alternating stages: local safe exploration (LSE) and global exploration (GE),
Local safe exploration During the LSE stage, we explore the subset the parameter space θ which known to be safe, and learn backup policies for all the states on the trajectories visited during LSE. In this stage, the parameters are selected according to the acquisition function
0, = OEG (M.
argmax max Wn-1(0,4 Zn: i) RET
(9)
where, Gn(2n) (n,) is (z() Sn(zn) is set of maximizers (c.f., Equation (18) in Appendix B). Intuitively, (z,) U.M(zz,) represents those parameters that can potentially lead to an expansion of the safe set (2x)) or potentially be a solution to the optimization problem of Equation (2) with context zn.
of expanders (c.f., Equation (16) in Appendix B) and
Global exploration Once LSE converges (see Equation (21) in Appendix B), we run the GE stage where we evaluate possibly unsafe policies and trigger backup policy whenever necessary. If no backup policy is triggered, we conclude that the evaluated policy is safe and add it to our safe set. After new parameter is added the safe set during GE, we continue with LSE.
The parameters are selected according to the acquisition function
argmax O, (S.
max Wn-11(.)) Zn
(10)
where & denotes parameters which have been shown be unsafe (see line of Algorithm 4 in Appendix B). If all parameters have been determined either safe or unsafe, i.c., θ (z,) U E(2,)) 0, then GE has converged.
Summary A detailed deseription of the contextual GOSAFROPT algorithm is provided in Ap- pendix B.2. GOSAFEOPT alternates between local safe exploration and global exploration. Therefore, it can seek for the optimum globally. In Figure 5 of Appendix C, we analyze the algorithm using simple example for better understanding.
The only difference between the contextual and non-contextual variants is that contextual GOSAFEOPT maintains separate Sn, Cr, Bn, Dny and X for each context Z. For any given context E Z, the running best guess of contextual GOSAFEOPT for the optimum is 0).
(z) (4) I, (0,
4.2.2 Theoretical Results
In the following, we state main theorem, which extends the safety and optimality guarantees from Sukhija et al. [1] to the contextual case.
5

---

We say that the solution to Equation (2), 0' is discoverable if there exists finite á such that 0" (z) E R (Sa Here, R2 (S) C θ represents the largest safe set which can be reached safely from S - Q up to c-precision (c.f., Equation (20) in Appendix B).
Theorem 1. Consider any e > 0 and ó E (0, 1). Further, let Assumptions I to 5 from Appendix A
hold and (,(8) be defined as in Lemma 1. For any context & E Z, let r(z) be the smallest integer such that
a(z)
n(z)
where n(z) = 1(2 =
Σ
(11)
(6) (z)
and C = 32/ log(1 1 + o 3). Here, (z()) = max AC LyA. hA,x) S In denotes the mutual information between A = [h(0, 2, ()) and corresponding observations.
Then, when running contextual AOSAFE and if (=) discoverable, the following inequalities jointly hold with probability at least - 28:
1. Vn 2 0, 0,1 Iyp (0., s st), 2) 2 0,
(safety)
2. Vz E Z, 2 iz): (() =) 2 ())
(optimality)
It is natural to start for each E I with kernels hi and Ki on the space of contexts and the space of parameters, respectively, and to construct composite kernels k; = k @ ke or k; = k2 e k as the product or sum of the pairs of kernels (see section 5.1 of [2]). In this case, the information gain On sublinear in n for common choices of kernels k, and A implying that n n²(z) (z) finite.
The theorem is proven in Appendix B.3, Comparing to contextual SAFEOPT [16] which is only guaranteed to converge to safe optima in (z)))), the global exploration steps of contextual GOSAFEOPT can also discover a safe optimum which was not reachable from the initial safe seed. We remark that Theorem is a worst-case result and, in particular, disregards a possible statistical dependence between different contexts. practice, if kernel chosen which does not treat all contexts independent, then the convergence can much faster as knowledge about particular context can be transferred to other contexts.
Experimental results
We evaluate the performance of contextual GOSAFEOPT using the Unitree Gol robot both physical simulation and hardware experiments. In the experiments, we use the following objective and constraint functions:
g(0, 2) = Σ (t)
s(t. x, 0)1,, q(,, 2) min (t) s(t. z. 0)), +20
(12)
where ø is joint-level states of the system (i.e. joint angles and speeds), sx denotes its desired values, and both Q2 and Q are positive semi-definite matrices. Additionally, we define an error threshold, U, that the norm the state error shouldn't It surpass throughout the entire duration. Further details of the experimental setup are provided Appendix We also made the implementation available online! and uploaded video showcasing the experiments".
Simulation experiments In simulation experiments, contrasted the learning curves of contextual GOSAFEOP to SAFEOPT, and AFEAPT without contexts. Additionally, we evaluate GP-UCB [25], an unconstrained BO algorithm. To simulate the model mismatches and uncertainties, we introduced disturbances the form of joint impedances at every joint (see Appendix for more details). These disturbances destabilize the system, leading to constraint violations. To adhere to the prerequisites of the safe BO algorithms, we initiate all experiments with roughly hand-tuned control gains that are safe, yet suboptimal.
htttp:://yottu..bb/bouuug/g

---

Contextual GOSAFEOPT
GoS
Contextual SAFE
SAFEOPT
UCB
1.00
Reward
0.50
Troi
Crawl
0.00
0
100
200
300
400
500
0
100
150
200
250
Iteration steps
Figure 2: Simulation experiments. On the left, we display the learning curves of the BO algorithms trained with trol gait. After completing this training, we started new training for crawl gait and included the contextual variants of GOSAFEOPT and SAFFOPT in the assessment to investigate the impact of contextual settings on learning performance as illustrated on the right.
Reward
Reward for crawl
Tracking error [deg]
1.0
5.0
Saled
0.5
2.5
Trot
Craw!
0.0
0.0
50
100
Trained for trot Truined for crow!
1000
2000
Iteration steps
Iteration steps
Figure 3: Hardware experiments. On the left, we present the learning curve of Contextual GOSAFEOPT. It shows that the algorithm successfully tunes the controller gains for trot, and then subsequently for crawl. In the center, we compare the performance the optimized control gains of trof when applied to crawl, against the gains specifically optimized for crawl. On the right, present the tracking error of the hip joint for the front-left leg with toot and crawl gait at initialization (trot: yellow, crawl: violet) and after optimization (trot: green, crawl: blue).
We optimized joint-level feedback gains for two different gaits; trot and crawl sequentially. All simu- lation experiments were conducted using ten different seeds, and we report the mean with one standard error the Figure 2. Throughout our experiments, all of the safe algorithms met safety constraints. In contrast, the standard GP-UCB method violates the constraints in 4.7% and 8% of all evaluations for the trot and crawl gaits respectively. Figure 2's left plot, we illustrate the normalized performance of GOSA AFEOPT and SAFEOPT w.r.t. objective for the trot gait. The learning curves clearly indicate that GOSAFEOPT's global exploration facilitates faster identification of better control gains. Notably, the GOSAFEOPT algorithm performs nearly as well as GP-UCB but without any constraint violations.
In the second test, we contrasted the contextual variants of GoS AFEOPT and SAFEOPT with their non-contextual counterparts. The results, as depicted in the right part of Figure 2, suggest that the contextual variants yield superior optima, with the contextual GOSAFEOPT algorithm emerging as the standout performer. The contextual variants leverage the information collected from the previous training with the trot gait, enabling them to identify better optima for the newly introduced crawl gait more efficiently. Additionally, they evade unsafe or unstable evaluations, unlike GP-UCB. The gait parameters we used for the trof and crawl gaits are provided Appendix E.
Hardware experiments Similarly to the simulation experiments, we first tune the controller for the trot gait and subsequently for the crawl gait. In Figure 3's left plot, we report the mean performance with standard error, based experiments conducted using three different seeds. In all our experiments, note that the contextual GOSAFEOPT algorithm results in zero constraint violations.
In our hardware experiments, we confirmed that different gait patterns require distinct sets of control gains. As shown in the center plot of Figure 3, the best- best-performing parameters for the trot gait do not perform well on the crawl gait. However, as we train with context for the new gait pattern crawl, there is notable improvement in the reward. Here, highlight that the contextual GOSAFBOP can harness previously gathered data when encountering new gait patterns, accelerating the discovery of optimal gains for the new gait.
7

---

Inirial gains
Optimized gains
Push
Slippey surBoee
Figure 4: Robustness test. Compared to the roughly hand-tuned initial gains (left), the optimal gains derived from our method (right) significantly improve the motion controller's robustness against external pushes (top) and slippery contacts caused by socks on the robot's s feet (bottom).
Additionally, in the right plot of Figure 3, we evaluate the tracking performance of our tuned controller, focusing on the hip joint's joint-angle tracking. When comparing the initial and the tuned controller across both trol and crawl gaits, it's evident that the tuned controller has a significantly reduced tracking error. For a comprehensive view, the error plots for other joints are provided in Appendix
To assess the robustness of the locomotion controller with the optimal control gains, introduced uncertainties in the form of external forces and simulated slippery conditions by placing socks on the robot's s feet, depicted in Figure 4. Our experiments demonstrate that the robustness of our controller is significantly enhanced after the tuning process, and it is able to recover from pushes and retain stability on a slippery surface. On the other hand, the controller with the initial control gains is more vulnerable to these uncertainties and tends to easily crash.
Finally, also highlight the zero-shot generalization capabilities of our method for unseen gait patterns through learned model. While flying trot and pronk gaits were not presented during the training, the learned model effectively suggests reasonably good control gains for these gaits. We encourage readers to view the accompanying video? for more in-depth understanding.
6 Conclusion
In this work, we extend GOSAFEOPT to the contextual setting and showcase its efficacy in adjusting the control parameters for a model-based legged locomotion controller through both simulation and hardware experiments. In our experiments, contextual GOSAFEOPT demonstrated superior conver- gence for newly introduced gait patterns by drawing upon information from previous training sessions. Additionally, our results confirmed that contextual GOSAFEOPT can effectively identify better optima without violating safety constraints. Across all of our experiments, contextual GOSAFEOPT outper- forms prior approaches by large margin and successfully finds optimal control gains for different quadrupedal gait patterns. After the fine-tune the-tunnng process, we found that our model-based controller exhibits a considerable improvement in robustness to various types of uncertainties, significantly enhancing the system's reliability. We highlight that the applicability of our proposed algorithm extends beyond our current scenario. For instance, we are interested applying this method to more diverse set of quadrupedal gait patterns and extend its scope to encompass non-periodic and unstructured gait patterns. Furthermore, stress that the algorithm is controller- robot-agnostic, making it pivotal tool for addressing the reality gap across various contexts.
Limitations While the algorithm provides theoretical safety guarantees, it is often uncertain in real-world applications whether all theoretical prerequisites are fulfilled. For instance, even though the surrogate model might be Lipschitz-cootinuouus the Lipschitz constant is generally not known a priori, i.e., Assumption 2 from Appendix A may not be satisfied. This often results in a too conservative choice of parameters. Furthermore, a wrong parameter choice for the backup prior can result unsafe global exploration or no global exploration at all. general, while safe exploration methods such SAFEOPT and GOSAFEOPT have been successfully applied several practical domains [1, 17, 18, 26, 27, 28), bridging the disparity between theoretical foundations and application remains topic of active investigation [29, 30, 31].

---

Acknowledgements
We thank Lenart Treven and Flavio De Vincenti for their feedback on this work.
This project has received funding from the Swiss National Science Foundation under NCCR Automa- tion, grant agreement 5INF40 180545, the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme, grant agreement No. 866480, and the Microsoft Swiss Joint Research Center.
References
[1] B. Sukhija, M. Turchetta, D. Lindner, A. Krause, S. Trimpe, and D. Baumann. Gosafeopt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence, 2023.
[2] A. Krause and C. Ong. Contextual gaussian process bandit optimization. In Advances in Neural Information Processing Systems, 2011.
[3] Unitree Robotics. attps://wwe wttp://www.w.ee.ee comeee.ccoeeeeeeeeeeeeeeeeeeeeee homm/en/// en/go..
[4] Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke, Sim- to-real: Learning agile locomotion for quadruped robots. In Proceedings of Robotics: Science and Systems, 2018.
[5] J. Hwangbo. J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 2019.
[6] D. Kang, J. Cheng, M. Zamora, Zargarbashi, and S. Coros. RI + model-based control: Using on-demand optimal control learn versatile legged locomotion. IEEE Robotics and Automation Letters, 2023.
[7] D. Kim, D. Carlo, B. Katz, G. Bledt, and S. Kim. Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control. arXiv preprint arXiv:1909.06580 2019.
[8] M. Wensing, A. Wang. S. Seok, D. Otten, J. Lang, and S. Kim. Proprioceptive actuator design in the mit cheetah: Impact mitigation and high-bbanwidth physical interaction for dynamic legged robots. IEEE Transactions on Robotics, 2017.
[9] A. Schperberg, S. D. Cairano, and M. Menner. Auto-tuning of controller and online trajectory planner for legged robots. IEEE Robotics and Automation Letters, 2022.
[10] J. Mockus, V. Tiesis, and A. Zilinskas. The application of Bayesian methods for seeking the extremum. Towards Global Optimization, 1978.
[11] R. Calandra, A. Seyfarth, J. Peters, and M. P. Deisenroth. Bayesian optimization for learning gaits under uncertainty: An experimental comparison on dynamic bipedal walker. Annals of Mathematics and Artificial Intelligence, 2016.
[12] M. Gelbart, J. Snoek, and R. Adams. Bayesian optimization with unknown constraints. Confer- ence on Uncertainty in Artificial Intelligence, 2014.
[13] J. M. Hernández-Lohato, M. A. Gelhart, R. P. Adams, M. W. Hoffman, and Z. Ghahramani, A general framework for constrained Bayesian optimization using information -based search. The Journal of Machine Learning Research, 2016.
[14] A. Marco, D. Baumann, M. Khadiv, P. Hennig, L. Righetti, and S. Trimpe. Robot learning with crash constraints. IEEE Robotics and Automation Letters, 2021.

---

[15] Y. Sui, A. Gotovos, J. Burdick, and A. Krause. Safe exploration for optimization with Gaussian processes. In International Conference on Machine Learning, 2015.
[16] F. Berkenkamp, A. Krause, and A. Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, 2021.
[17] Y. Sui, V. Zhuang, Burdick, and Y. Yue. Stagewise safe Bayesian optimization with Gaussian processes. In International Conference on Machine Learning, 2018.
[18] C. König, M. Turchetta, J. Lygeros, A. Rupenyan, and A. Krause. Safe and efficient model-free adaptive control via bayesian optimization. In IEEE International Conference on Robotics and Automation, 2021.
[19] D. Baumann, A. Marco, M. Turchetta, and S. Trimpe. GoSafe: Globally optimal safe robot learning. In JEEE International Conference on Robotics and Automation, 2021. Proofs in extended online version: arXiv 1105.1321..
[20] D. Kang, S. Zimmermann, and S. Coros. Animal gaits quadrupedal robots using motion matching and model-based control. In IEEE/RSJ International Conference Intelligent Robots and Systems. IEEE, 2021.
[21] D. Kang, F. De Vincenti, N. C. Adami, and Coros. Animal motions on legged robots using nonlinear model predictive control. In IEEE/RSJ International Conference Intelligent Robots and Systems. IEEE, 2022.
[22] B. Siciliano, L.. Sciavicco, L.. Villani, and G. Oriolo. Robotics: Modelling, Planning and Control. Springer Publishing Company, Incorporated, 1st edition, 2008
[23] C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005.
[24] S. R. Chowdhury and A. Gopalan. On kernelized multi-armed bandits. In International Conference Muchine Leurning, 2017.
[25] N. Srinivas, A. Krause, S. Kakade, and M. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference Machine Learning. Omnipress, 2010.
[26] A. Wischnewski, J. Betz, and B. Lohmann. A model-free algorithm to safely approach the handling limit of an autonomous racecar. In IEEE International Conference on Connected Vehicles and Expo, 2019.
[27] M. Fiducioso, S. Curi, B. Schumacher, M. Gwerder, and A. Krause. Safe contextual Bayesian optimization for sustainable room temperature PID control tuning. In International Joint Conference on Artificial Intelligence, 2019.
(28) S. F. Cooper and T. I. Netoff. Multidimensional bayesian estimation for deep brain stimulation using the safcopt algorithm. medRxiv, 2022.
[29] C. Fiedler, C. W. Scherer, and Trimpe. Practical and rigorous uncertainty bounds for Gaussian process regression. AAAI Conference Artificial Intelligence, 35(8), 2021.
[30] Berkenkamp. A. P. Schoellig, and Krause. No-regret bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 2019.
[31] J. Rothfuss, C. Koenig, A. Rupenyan, and A. Krause. Meta-learning priors for safe bayesian optimization. In 6th Annual Conference on Robot Learning, 2022.
[32] S. Vakili, K. Khezeli, and V. Picheny. On information gain and regret bounds in gaussian process bandits. In International Conference Artificial Intelligence and Statistics, 1021.
10

---

[33] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint 1606./6606.01: 2016.
[34] D. Kang, F. De Vincenti, and S. Coros. Nonlinear model predictive control for quadrupedal locomotion using second-order sensitivity analysis. arXiv preprint arXiv: 2207. 2007.10465, 2022.
[35] R. Smith et al. Open dynamics engine, 2007.
[36] K. Serkan, I. Turker, and G. Moncef. Multidimensional particle swarm optimization for machine learning and pattern recognition. Springer- Verlag Berlin Heidelberg, 2014.
[37] Constrained Bayesian optimization with particle swarms for safe adaptive controller tuning. IFAC-PapersOnLLnn 2017.
[38] G. Pleiss, Gardner, K. Weinberger, and A. G. Wilson. Constant-time predictive distributions for Gaussian processes. In Proceedings of the 35th International Conference on Machine Learning. PMLR, 2018.

---

Contents of Appendix
Assumptions
13
Proofs
13
B.1 Definitions
13
B.2 Algorithm
14
B.3 Proof of Theorem
16
Comparison SAFEOPT and GOSAFEOPT
16
Control Formulation
17
E Experimental Details
18
E.1 Gait parameters
18
E.2 Bayesian optimization
19
E.3 Simulation
19
E.4
Hardware
19
Practical modifications
19
F.1
Boundary conditions
19
F2 Optimization
F3
Fix iterations and discard unpromising new detected safe regions
F.4
Posterior estimation
21
12

---

A Assumptions
In this section, we reiterate the assumptions by Sukhija et al. [1] for GOSAFEOPT.
Assumption 1 (Initial safe seed). For any episode 2 1 with (user-specified) context 2n 2, non-empty initial safe set of parameters (zn) C θ is known. That is, for all 0 E (nn112n) and all i E Iup (0, z,)) 2 O.
Here, (z) 2 Su(z) denotes the safe set after episode for the given context z as defined in Equation (15) in Appendix B. Given the prior knowledge of the dynamics, a conservative safe set of parameters represents some initial stable feedback controller. Accordingly, this assumption is typically satisfied in practice. The assumption necessary as, in principle, during each iteration, an adversarial context could be chosen for which the initial safe does not include any safe parameters.
Assumption (Continuity of objective and constraints). Let be defined as
(9) h(0, 2, ( (g,)0
ifi=0. O,
(13)
We assume that h lies in a reproducing kernel Hilbert space (RKHS) associated with a kernel and has a bounded norm in that RKHS, that is, ||/]| S B. Furthermore, we assume that and qs (Vi E Ig) are Lipschitz-continuuur with known Lipschitz constants.
This is common assumption in the model-free safe exploration literature [16, 19, 1]. Sukhija et al. [1] discuss the practical implications of this assumption in more detail.
Assumption 3. We obtain noisy measurements of with measurement noise i.i.d. Ssub-Gauusiia. Specifically, for measurement ys of h(0, z, i), we have ys h(0,z.i) + c, with 6, Sub-Gaussian for all i I where write I (0, co..
Assumption 4. We observe the state s(t) every At seconds. Furthermore, for any s(t) and p E [0, 1], the distance to st + pLt) induced by any action is bounded by a known constant E, that is, ||s(1 + pAi) s()]l E.
Assumption is crucial to guarantee safety in continuous time even though the state is measured at discrete time instances. For highly dynamical systems, such quadrupeds, the observation frequency is typically very high, e.g., 500 Hz kHz, and accordingly E is small.
Assumption 5. We assume that, for all i E (1...., c), qs defined the minimum of state- dependent function qj along the trajectory starting 8o with controller no. Formally,
q:(0, 2)
min 8'E5(ag.0.e)
qo (=6,,2,00
(14)
with E(.0.0.2) {so (s(T), (o(()),))))())...))))))))))>))) policy parameter O and context z starting from at time 0.
2)) dr it 0} representing the trajectory of s(t) under
Assumption is an assumption on our choice the constraint. Many common constraints, such as the minimum distance to an obstacle along a trajectory, satisfy this assumption.
B Proofs
B.1 Definitions
We begin by re-stating definitions of sets used by GOSAFEOPT [1] with an additional context variable. Fix arbitrary context Z. The safe is defined recursively
(x)z)
n
U
10 ( θ (0', 2.1) Lz(z) 10 0'll > 0}
(15)
iEI, (2E)
13

---

where Le is the joint Lipschitz constant of and the constraints under context z. The expanders are defined as
(.( (0 E (z) e. (0, 2) 0} with
(16)
en (0,z) 1(0' e (x)=)) Di E Eg: un (0, 2, Le(2) 00'| 0}
(17)
and the maximizers are defined
Mn(z) = {0 E S,())) un (0,00)) max (,00,)))
(18)
O'ES,(=)
The analysis requires the e-slacked safe region (S) given initial safe seed S θ which is defined recursively as
(S) = S u (0 E 30' E S such that Vi E Iy: (,'0,, - e Le(>) 0'll 2 0}, (19) (20)
R (S) = lim (S) 11
where (R5)* denotes the nihh composition of R with itself.
B.2 Algorithm
B.2.1
Local Safe Exploration
During LSE, we keep track of a set of backup policies ((x) C θ x X' and observations of h for each context E 2 which we denote by ((z) S θ x R An LSE step is described formally in Algorithm 1.
Algorithm Local Safe Exploration (LSE)
Input: Current context 2n, safe sets S, sets of backups B, datasets D, Lipschitz constants Le 1: Recommend parameter 6,, with Equation (9)
2: Collect R = U [(On, skk)) and h(On, Zn, i) En
keN
3: B(z,) = B(z,)) U U. D(z,) = D(z,) ( h(O,, Zn, i) + E,))
4: Update sets S(z), ((z), and M(z) for all 2 E Z
Equations (15), (16) and (18)
Return: 5, B3, D
The LSE stage terminates for some given context æ E Z when the connected safe set is fully explored and the optimum within the safe set discovered. This happens when the uncertainty among the expanders and maximizers less than and the safe set is not expanding
max
max 10-1 2,,
and
= =p-1(2)=S,(z))
(21)
OEG. (3) (2) IEI
B.2.2 Global Exploration
GE step conducts an experiment about a candidate parameter 6, E which may not be safe. If the safety boundary approached, GE conservatively triggers safe backup policy. If, on the other hand, the experiment successful, new (potentially disconnected) safe region was discovered which can then be explored by LSE in the following steps. A GE step is described formally in Algorithm 2.
B.2.3 Boundary Condition
The boundary condition checks when the system is in the state whether there is a backup (0., s.) ((x) such that is sufficiently close to s to guarantee that 0, can steer the system back to safety for any state which may be reached in the next time step. no such backups exist for the next states, a backup triggered at the current state. In this case, the backup parameter 0 with the largest safety margin is triggered:
0 88..8,) (.,,),,,ll (*., ) FE,
min (0.. Zn. 1) L I|s szll-
(22)
14

---

Algorithm 2 Global Exploration (GE)
Input: Zn, safe sets S, confidence intervals C, sets of backups B, datasets D, fail sets & and Faa
1: Recommend global parameter 0, with Equation (10)
2: 0 = on. Ftuil 0, Boundary False
3: while Experiment not finished do
Rollout policy
4: if Not Boundary then
5:
Boundary, 0 = BOUNDARYCONDITIONE: ())), 6: if Boundary then
Trigger backup policy
7:
0 0r SFail = s(k) = U ). Apail XPail {SPall}
8:
Update fail sets
9:
Execute until skk
10: Collect R = U [(O,, (k))) and h((O,, Zn, + En
KEN
11:
if Not Boundary then
>> Successful global search
B(zn) = zn
and (zn,) = D(zn) {(On h(On: Zn, i) E..)
13:
S(zn) = S(z,) U (
C(O Zn, = C(O,, nn 1) [0, for all In
Return: S, C, B, D, D, Faail
Algorithm 3 BOUNDARYCONDITION
Input: context 2,, state backups B
1: if v(O,,,, 8.) E ((z,), di E Inp L,, kn, i) - Lz 1ss 8.1l +3 < 0 then
2: Boundary True, Calculate 0 (Equation (22))
3: else
4:
Boundary False, 0 Null
return: Boundary, 0:
B.2.4
Contextual GOSAFEOPT
The algorithm stops for particular context E Z when
Equation (21) is satisfied
and
(Z. E((z))
0
(23)
LSE converged
GE cunverged
The full algorithm is described in Algorithm 4.
Algorithm 4 Contextual GOSAFEOPT
Input: Domain θ, Contexts Z, Sequence of contexts {2, E 2 2)211,1 k(. Sp. Co. Do.
1: Initialize GP h(0, z, i). E(z) 0. (z) = Bo(z) ) = {(0, 2o) E So} 2: while 7F Z such that GOSAFEOPT has not terminated for z (Equation (23)) do
3: if GOSAFEOPT has terminated for zn (Equation (23)) then
> Skip finished contexts
4: continue
5: for E Xiaa (z.) do
b Update fail sets
6: Not BOUNDARY CONDITION(2,,.. B,,) then
7:
E(z,)) {z,,) {0}. Xtai (,,,) = Niaal (2,) {8} Update (0, z, i) V0 O, I
8:
Update confidence intervals, Equation (8)
9:
if LSE not converged for context 2n (Equation (21)) then
10:
Sa+1, B
LSE(Z,, Sn, B,, D.,)
11: else
Sn+1, Ca41, Br+ D Xami GB(z,, Sn, Cny Br, Dny E, XFai)
return:
(0)(z)
Z}
15

---

B.3 Proof of Theorem 1
Proof. We first derive the sample complexity bound of non-contextual GOSAFEOPT. Then, we extend this sample complexity bound to contextual GOSAFEOPT. We assume without loss of generality that Bn is monotonically increasing with n.
Sample complexity Assume first that the context is fixed, that is, Vn 1: 2. In this the safety guarantee (with probability at least 1 6) follows directly from Theorem 4.1 of Sukhija et al. [1]. Thus, it remains to show that the optimality guarantee with the given sample complexity holds also with probability at least 1 - 6, as then their union holds jointly with probability at least
1 26 using a union bound.
It is straightforward to see (by employing Theorem 4.1 of Berkenkamp al. [16]) that Theorem 4.2 of Sukhija et al. [1] holds for n² being the smallest integer such that
n" C|O Bn. (x))11)
(24)
2e²
where we use that [S)] < [0] for any 5 S θ and | + < 2/0). Thus, whenever a new disconnected safe region is discovered by GE, LSE is run for at most n steps.
It follows from the stopping criterion of GE, θ (S, UE) 0, that GE is run for most [
consecutive steps (i.c., without LSE-step in between). Clearly, new disconnected safe region can be discovered by GE at most [ times, and hence, GOSAFEOPT terminates after at most [+] iterations of at most n LSE steps and at most | GE steps. Altogether, we have that the optimality for ñ being the smallest integer such that guarantee holds with probability at least
O 2 (6)
| (n* + 10)),
(25)
2
completing the proof of Theorem 1 for non-contextual GOSAFEOPT.
Multiple contexts Visiting other contexts 3 {=} in between results in additional measurements and increases the constant 8, ensuring that the confidence intervals are well-calibrated. The only difference in the proofs is the appearance of (n-(=)) rather than Br(()) in Equation (24). In the contextual setting, n²* (z) the smallest integer such that
C|O) (0))))))2))
n(z) N
(26)
2ea
where
(2)
n(z)
Σ
=,]=
counts the number of episodes with context z until episode n (z). The bound on ñ(z) then follows analogously to Equation (25).
C Comparison of SAFEOPT and GOSAFEOPT
To visually analyze the difffrrnt exploration properties of SAFEOPT and GOSAFEOPT we use the Pendulum Environment from OpenAl [33]
example. The ideal trajectory is given by some undisturbed controller. In our toy problem, we use simple PD control which is sufficient for the pendulum swing-up problem and various oscillating trajectories. To simulate the sim to hardware gap, we artificially add a disturbance to the applied torque in the form of joint impedances T = T' (5 z) + f where 6% and 0 are unknown disturbance parameters and "", 2 are the desired and observed motor angles. We use AOSAF to tune an additional PD controller which should follow the ideal trajectory and compensate for the artificial disturbance. Figure 5 shows an example run of SAFEOPT and GOSAFEOPT. Whereas SAFEOPT is restricted to expanding the initial safe region, GOSAFEOPT can discover safe regions, and thus find a better optimum.
16

---

Table 1: Here, we summarize different magnitudes of 7/n for composite kernels from Theorems and 3 of Krause and Ong [2] and for individual kernels from Theorem 5 of Srinivas et al. [25] and Remark of Vakili et al. [32]. The magnitudes hold under the assumption that the domain of the kernel is compact. 7 and 72 denote the information gain for the kernels % and h², respectively. By the modified Bessel function.
Kernel
k(v,v')
Product
(w, v²) h2t v0) v') if has rank at most d 1. (0) + h (0,
dh + dlog(n) + n + 2 log(n)
Sum
Linear
v'v'
o
(dlog(n))
RBF
212
0 log (og=+((n)) (n))
Matérn
Viv v-v'
V20
O
n 20+ log 2o (n)
SAFEOPT
2
GOSAFEOPT
kp
Figure 5: Example run of SAFEOPT and GOSAFEOPT. The red circle denotes the initial safe point. The black dots denote observed points. The green circle denotes the true safe optimum and the blue circle denotes the optimal point determined by SAFEOPT and GOSAFEOPT after 150 iterations respectively. The discovered safe sets are shown in black. GoS AFEOPT gets closer to the true optimum by discovering new safe regions which are not connected to the initial safe region.
D Control Formulation
Our model-based motion controller integrates the MPC and WBC methods to enhance both robustness and maneuverability, The MPC is responsible for generating base and foot trajectories, while the WBC converts these trajectories into joint-level commands. For the MPC component, employ the model predictive control formulation proposed by Kang al. [21, 34]. This formulation represents
17

---

finite-horizon optimal control problem as a nonlinear program utilizing the variable-heige -hight inverted pendulum model. The optimal solution of the nonlinear program is determined by using a second- order gradient- gradient-based method. For a more in-depth understanding the MPC formulation, we direct readers to the prior work by Kang et al. [21, 34].
We employ slight modification of the WBC formulation introduced by Kim et al. [7], adapting it to align with our MPC method. Following the method proposed by Kim et al. [7], we compute the desired generalized coordinates x", speed i and acceleration ø' for quadruped system at the kinematic level. This process involves translating desired task space (Cartesian space) positions, velocities, and accelerations into configuration space counterparts. Throughout this process, we enforee task priority through iterative sull-space projection [22]. The top priority is assigned the contact foot constraint task, followed by the base orientation tracking task. The base position tracking task given the third priority, and the swing foot tracking task is assigned the final priority.
Subsequently, we solve the following quadratic program:
min
|10$10
(27a)
f..fe
s.t.
Sy + b+b = SjJJ fa
(27b)
a" = x" + +
(27c)
Wf 220 0,
(27d)
where oş denotes a relaxation variables for the floating base acceleration and fa denotes contact forces with the contact Jacobian Je. Equation (27a) is the objective function that penalizes the weighted norm of 53 with the weight matrix Q. Equation (27b) corresponds to the equation of motion of the floating base, representing the first six rows of the whole-body equation of motion, with S being the corresponding selection matrix. Lastly, Equation (27d) sets forth the Coulomb friction constraints. This procedure refines the desired generalized acceleration i', which is calculated the kinematic level, by incorporating the dynamic impacts of the robot's 's movements.
Upon determining i determined, we compute the joint torque commands as follows:
Mæ * + +b+g J.
(28)
The final torque commands are calculated using -emal T" + ksp (x - x) + kald" ) and dispatched to the robot with the feedback gains kp E Rd. xds and ka RRd, xds, Here, T, 2 are the joint angles and speeds, while x., â" denote their desired values. As previously noted, this step is crucial in dealing with model mismatches, specifically, the differences in joint-level behavior stemming from actuator dynamics and joint friction.
E
Experimental Details
E.1
Gait parameters
We used the following gait parameters for the simulation and the hardware experiments.
Table 2: Gait parameters.
Trot
Crawl
Flying trot
Pronk
Duration [s]
0.5
1.2
0.6
0.6
Duty cycle
0.5
0.75
0.4
0.9
0.5
0.25
0.5
0.0
Phase
0.5
0.5
0.5
0.0
Offsets
0.75
0.0
18

---

E.2 Bayesian optimization
For all our experiments, we use a Matérn kernel with v = 1.5 for the underlying Gaussian Process. The lengthscales are fixed during the whole optimization process and set to
Table 3: Kernel lengthscales.
lengthscales
Simulation
[0.1, 0.05, 0.1, 0.05 0.0. 01100000110000000000011011111111
Hardware
[0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.5, 0.5 0.5]
where the first n parameters correspond to the (kpp k2) pairs and the last parameters to the context. For all experiments, we use 3 = 16 for the LCB on the constraints.
E.3 Simulation
We developed an emulator to simulate the control gain tuning process, utilizing the open-source rigid- body simulation engine, the Open Dynamics Engine (ODE) [35]. To account for model mismatches and uncertainties, we introduced disturbances based on the model detailed in subsequent sections.
Disturbance model We introduced joint-level disturbances by altering the torque exerted by each motor. This method emulates the torque tracking discrepancies in motors, the damping effects stemming from joint friction, and other model mismatches attributed inaccuracies in the model. More specifically, for ith motor of leg I, the applied motor torque is given by Ti.
applied ouT
0, [ 3t,i, = In this equation, aad is the torque command computed as described in Appendix D, a is disturbance factor for leg with [0.73, 0.9, 0.73, 0.9]T, and 2, ă are the joint angles and speeds, while ~" =² denote their desired values.
The joint states variable E R221 all 12 joints is described as a concatenated vector of joint angles and joint speeds. We set the matrices from Equation (12) to Q2 = 244224 and Q 1i= Ai <= 12).
Reward function
E.4 Hardware
We slightly modify the reward function for the hardware experiment and include penalty term the joint velocities.
3 2)) =((,)))((,=)) ( ||s(t)]1,,
where the velocity state errors in Q, and Q2 in Equation 12 are set to zero, since the joint speed measurements are noisy finite difference approximations of the joint angles. Furthermore, we define Qp to only include the noisy joint speeds observations. More specifically, we define Q = Q = A <= 12} and Q = li
A > 12} and Qg. Q Qp E 244x24 Experimental results have shown, that adding a penalty term on the joint velocities acts as a regulator to prefer solutions where motor vibrations are low. This has shown to improve overall convergence and to visibly avoid solutions where motor vibrations are high.
Figure 6 shows that the optimal feedback control parameters drastically reduce motor vibrations and increase the tracking performance.
F Practical modifications
F.1 Boundary conditions
We use the idea from Sukhija et al. reduce computational complexity by defining interior and marginal set. Intuitively, the interior set contains all observed states for which the safcty margin is
19

---

Tracking performance
Tracking error [deg]
Joint angle [deg]
Trot untrained
Hip
-2
Tror trained
-5
Crawl untrained
Crawl trained
0
Thigh
10
Joint configurations
-10
Hip
0
Thigh
Calf
Calf
3
-5
-10
Iteration steps
Iteration steps
Figure 6: Joint angle tracking performance comparison. Joint angle tracking errors in degrees (left) and joint angle measurements in degrees (right.) The ycllow and violet regions represent the initial control gains for trot and crawl respectively. Conversely, the green and blue regions indicate the optimized gains for trot and crawl. It is evident from the plots that the refined gains yield substantially reduced tracking error with diminished jitter.
high and the marginal set includes all states where the safety margin greater than a certain threshold. More formally, Sukhija al. [1] defines the interior and marginal set as
Q1,n {a, X (0, x.) B : Vi (0,)) 2 nn QM {x, X (0, æ.) B, vi Iq. m S Ln ( i) < nu
(29)
(30)
The boundary condition is defined separately for the interior and marginal set. Firstly, the Euclidean distance d, between the observed state and all the backup states is calculated. If dmin = min, d: = 0, a backup policy for the observed state is known to be safe. Intuitively, the uncertainty if a backup policy can safely recover from the observed state increases as dain grows. If the observed state moves too far away from the set of backup states, the closest backup policy is triggered. More formally, a backup policy triggered, if the Ad, s.t p((2| 2 d:) > т. The distribution over æ is defined as NO0 d²) and Tm 2 r for the interior and marginal set, respectively. With x² and 7, there are two adjustable parameters to influence how conservative the backup policy acts.
Table 4: Boundary condition parameters
Parameter
Value
Description
Simulation
0
2
Standard deviation of backup distribution Interior lower bound probability Marginal lower bound probability
T
0.2
Tm
0.6
Hardware
0
2
Standard deviation of backup distribution Interior lower bound probability Marginal lower bound probability
0.05
Tin
0.1
20

---

F.2 Optimization
The solution the acquisition optimization problem formulated 10 is approximated with the standard particle swarm [36] algorithm, similar to [37].
At the beginning of each acquisition optimization, ny particle positions initialized. Rather than initializing the positions over the whole domain, the positions are sampled from a list of known safe positions in the current safe set.
For all experiments, the parameters in Table are used.
Table 5: Swarmopt parameters
Parameter
Value
Description
O
Social coefficient
θ
Cognitive coefficient
w
0.9
Inertial weight
100
Number of iterations
T4r
100
Number of restarts if no safe is found
E,3 Fix iterations and discard unpromising new detected safe regions
In practice, it is not practical to fully explore safe set before the global exploration phase. For our experiments, the number of iterations for the local and global exploration phase are fixed to = 10 and ny = 5, respectively. To avoid exploring for all ny steps in unpromising regions, we define nd = 5 < ni and switch to local exploration set is much less than the best global reward estimate. That is, we switch the best set if fz < cx and ni Tht.
best set if the best reward estimation of the current
F,4
Posterior estimation
Each BO step requires the optimization of the GOSAFEÔPT acquisition function to predict the next parameters to evaluate. This paper uses the standard particle swarm [36] algorithm, which requires the
computation of the posterior distribution at each optimization step for all particles. To speed up the computation of the posterior distribution, the paper uses Lanczos Variance Estimates Pleiss et al. [38].
21


Reviews:
[{"review_id": "tEC18jZP5t", "reviewer": "Reviewer_bsr3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a method to safely and automatically tune parameters (feedback gains) for legged locomotion controllers. The underlying algorithm is an extension of GoSafeOpt[1] that includes contextual information (gait parameters). The authors provide both theoretical and experimental validation for the proposed method, and shows that parameters tuned using GoSafeOpt can increase the overall stability and smoothness of the resulting controller.", "summary_of_recommendation": "Overall I find this to be a clear, well-written paper with well-demonstrated theoretical and experimental results. Therefore, I recommend acceptance for this paper. I'm holding back from a strong accept because I find the theoretical contribution to be relatively incremental and the experimental validation to be more of a proof-of-concept and does not fully capture the complexity of real-world.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please refer to the weakness section in my review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strengths:**\n\n1. Convincing demonstration on real hardware\n\nIn the supplemental video, the authors clearly demonstrated the effectiveness of the proposed method. Tuning the feedback gains (as well as other parameters) in optimal-control-based controllers is a long and tedious process, and I truly appreciate the authors' effort in developing an automated and safe approach to this problem.\n\n2. Clarity of presentation\n\nOverall, I find the paper to be clearly written and easy to follow. I especially appreciate the fact that the authors added explanations and intuitions next to the assumptions and theorems, which makes the paper very easy to read and understand. The experimental results is also concise and convincing.\n\n**Weaknesses:**\n\n1. Incremental contribution on algorithms\n\nIf I understand correctly, the proposed method is an extension of GoSafeOpt with contextual information (in this case gaits). While the authors performed comprehensive evaluation of the proposed method, I find the overall algorithm contribution to be relatively incremental and straightforward, because the inclusion of context in Bayesian Optimization (e.g. Contextual Bandits or [CGP-UCB](https://papers.nips.cc/paper_files/paper/2011/hash/f3f1b7fc5a8779a9e618e1f23a7b7860-Abstract.html)) has been proposed in previous studies.\n\n2. More diverse context information would be desirable\n\nIn this paper, the \"context\" seems to be mostly restrained to the gait of the robot, while in practice many other factors might affect the controllers' performance, such as speed command and terrain type. Would it be beneficial to include these information in the context vector? How to encode these information in the context vector? Would the proposed algorithm still work well with the expanded context?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "cQW9ZdSDES", "reviewer": "Reviewer_2qzt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an algorithm to learn optimal feedback control gains for model-based quadruped controllers. Theoretical results are presented to demonstrate the proposed method's optimality and safety, and hardware experiments are used to support the claims.", "summary_of_recommendation": "This paper presents an algorithm to safely tune control parameters on real robots, with sufficient theoretical and hardware results. I believe this can be valuable to the robot learning community and recommend acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. It will be interesting to see how the gains evolve during the optimization process to understand how much work the optimization need to do.\n\n2. In the demo video, there is a clip (starting at around 22 seconds) where a controller fails to directly transfer to the hardware, however, this is not used as a starting point for fine-tuning. Can the authors elaborate on if it is possible to fine-tune from that controller and if possible present relevant results?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is presented with both theoretical results and hardware results and addresses the important question of how to fine-tune controllers safely on real robots, so I believe this is a good contribution to the robot learning community.\n\nMy only concern is that the controllers presented are already working pretty well before tuning, and the algorithm seems to only need to do minimal tuning to further improve them. It will be interesting to see more interesting situations where the original controller can barely work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "VCUOrivtfAc", "reviewer": "Reviewer_2pRp", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a data-driven method to safely and automatically tune controller feedback gains.\nThe authors validate their work by deploying an MPC style policy on an Go1 robot and tuning the torque control loop's gains.\nOn the theoretical side, the work builds on a prior method called GoSafeOpt and extends it by introducing a notion of context, making it possible to discover gait-specific gains.\n", "summary_of_recommendation": "The method in itself is interesting, however it is highly complex to achieve a relatively basic outcome: tuning PD gains.\nThe main hardware result is that by tuning the PD gains of the torque control loop in a gait-specific way, the joint position tracking error is lower and the robot is less noisy as a result of less over/undershoot during deployment. \nHowever, the evaluation is done on very basic gaits under perfect conditions. Go1 is capable of highly agile skills, yet this work focuses on basic trotting and crawling behaviors, which make it hard to assess its value with respect to the current state of the art.\nThe theoretical contribution appears incremental (adding a context variable to GoSafeOpt for each gait). ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Hardware results: It's not clear how the data was collected to optimize the gains? Was this in sim and then deployed to real or did you iteratively improve the performance on hardware?\n- Consider demonstrating the usefulness of the method with state of the art skills (jumping, agile skills) and in non-ideal scenarios (disturbances, uneven terrain etc.).\n- How long are the rollouts? The paper mentions iteration steps throughout: Are these timesteps or complete rollouts?\n- When does the method fail? \n- The hardware parameter space was rather small (6D). How does the method scale to a higher number of dimensions?\n- Please compare to existing methods (RMA, Actuator Net, manually tuning gains etc.) to make it possible to understand the relative performance of the proposed method.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "While the proposed method is interesting, it is a highly complex approach to implement a rather basic feature: tuning PD gains for a well-known torque control loop. A lot of assumptions are made (e.g. noise in Section 3.1) to ensure safe exploration. However, the experimental section does not validate these assumptions.\n\nThe method is evaluated on a very simple test case (basic trotting and crawling on flat terrain) rather than with state-of-the-art policies.\nThe manuscript mentions that \"Even though we have a good model of the robot dynamics, the motors are typically difficult to model sufficiently accurately.\" This is true. However, there are multiple recent methods available to address this issue (e.g. actuator net), which could be used as a comparison to ground the results.\n\nFinally, the method appears incremental (adding a context variable to GoSafeOpt) and beyond the basic experiments in the paper, the value of adding the context variable is not clear.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-BRdahiiVH", "reviewer": "Reviewer_9Nt1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a safe an efficient method to tune model of a robot based on the hardware data. This allows running MPC on a more accurate model. The authors extend a previous algorithm to have context and apply it to quadruped walking in different gaits. The authors describe the algorithm and its safety guarantees in details. The proposed method is tested on quadrupeds both in sim and in real.", "summary_of_recommendation": "I recommend weak reject because of the evaluation. The claims and theory is well explained, the contribution is incremental, but the evaluation lacks the domain and experiments to prove the claims further. Given further experiments or domains the paper can be very significant, but it is not well supported in its current form.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Can the authors clarify the above question on torque vs pd controls? Does the controller purely a position control?\n\nCan the method handle unkown latency in the system? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper takes a previous method, extends it with contexts, shows the safety guarantees and applies it to quadruped problem. The intro, motivation and the overall idea is very interesting. As the authors state, the sim-to-real gap cause by the model discrepancies makes MPC hard to use for some problems.\n\nThe authors do a great job at introducing the problem and the method. But the author's contribution is very incremental and mainly composed of having context. The idea of applying to quadruped locomotion is interesting, because MPC is known to work quite well for these quadrupeds as the model is well studied. \n\nThe formulation of the problem is very similar to the PD controller and setting the gains. So I'm a little confused by the usage of MPC here which outputs desired torques. If I understand correctly, the authors convert it to position, then they track the position while tuning the gains? \n\nThe idea of safe exploration is very interesting and the results show significant improvement. On the other hand, the problems the author study is limited to 2 different gaits for walking. I think the method had great potential to overcome different behaviors such as hopping, jumping or very fast galloping behaviors.\n\nI think that the paper has a high potential, but it can benefit from a problem where the model is unknown or very hard to model (i.e. attach a bucket filled with water?) and different behaviors where safety is critical while MPC needs to push the boundaries of the robot. In its current state, the results show behaviors that can be already accomplished.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for safe and automatic parameter tuning of feedback gains in a legged locomotion controller. The underlying algorithm is an extension of GoSafeOpt that includes contextual information. The authors provide both theoretical and experimental validation for the proposed method. Please prepare your camera-ready paper by including the results of additional experiments you made during the rebuttal phase and the clarifications discussed with reviewers.
Decision: Accept (Poster)