Learning Safe Control via On-the-Fly Bandit Exploration
Authors: Alexandre Capone
Ryan Kazuo Cosner
Aaron Ames
Sandra Hirche
Venue: ICML 2025 poster (2025)
Keywords: Safety
Gaussian processes
control barrier functions
Bayesian models
Primary area: applications->robotics
TL;DR: We guarantee safe control by learning on the fly, without requiring a prior model or backup controller

Abstract: Control tasks with safety requirements under high levels of model uncertainty are increasingly common. Machine learning techniques are frequently used to address such tasks, typically by leveraging model error bounds to specify robust constraint-based safety filters. However, if the learned model uncertainty is very high, the corresponding filters are potentially invalid, meaning no control input satisfies the constraints imposed by the safety filter. While most works address this issue by assuming some form of safe backup controller, ours tackles it by collecting additional data on the fly using a Gaussian process bandit-type algorithm. We combine a control barrier function with a learned model to specify a robust certificate that ensures safety if feasible. Whenever infeasibility occurs, we leverage the control barrier function to guide exploration, ensuring the collected data contributes toward the closed-loop system safety. By combining a safety filter with exploration in this manner, our method provably achieves safety in a general setting that does not require any prior model or backup controller, provided that the true system lies in a reproducing kernel Hilbert space. To the best of our knowledge, it is the first safe learning-based control method that achieves this.

Learning Safe Control via On-the-Fly Bandit Exploration
Alexandre Capone' Ryan Cosner
Aaron Ames? 2 Sandra Hirche
Abstract
mous with guaranteeing that the system always satisfies pre-specified set of constraints. While there exist vari- ous tools to achieve safety whenever the system is known perfectly (Agrawal & Sreenath, 2017; Ames et al., 2017; Bansal et al., 2017; Wabersich & Zeilinger, 2018; Zeng et al., 2021), doing so if the dynamics have to be learned poses significantly more difficult challenge.
Control tasks with safety requirements under high levels of model uncertainty are increasingly com- mon. Machine learning techniques are frequently used to address such tasks, typically by leveraging model error bounds to specify robust constraint- based safety filters. However, if the learned model uncertainty is very high, the corresponding filters are potentially invalid, meaning no control input satisfies the constraints imposed by the safety filter. While most works address this issue by assuming some form of safe backup controller, ours tackles it by collecting additional data on the fly using a Gaussian process bandit-type algo- rithm. We combine a control barrier function with a learned model to specify a robust certificate that ensures safety if feasible. Whenever infeasibility occurs, we leverage the control barrier function guide exploration, ensuring the collected data con- tributes toward the closed-loop system safety. By combining safety filter with exploration in this manner, our method provably achieves safety in a setting that allows for a zero-mean prior dynam- ies model, without requiring a backup controller. To the best of our knowledge, is the first safe learning-based control method that achieves this.
To address safety in control setting, existing approaches typically attempt to bound the model error and assess how its worst-case estimate affects safety. To estimate the model error, frequently used frameworks include ensembles of neural networks (Chua et al., 2018; Curi et al., 2020) or Gaussian processes (GPs) (Capone et al., 2022; Rodriguez et al., 2021). The impact of the model error on safety is then frequently estimated by predicting the worst-case trajectory multiple steps into the future (Akametalu al., 2014; Hew- ing et al., 2020; Koller et al., 2018). Alternatively, safety filters attempt to condense the long-term impact of con- trol input on safety (Ao et al., 2025; Berkenkamp et al., 2017; Castañeda et al., 2021; Cheng et al., 2019; Jagtap et al., 2020). A crucial drawback of most of these methods is that they assume the model uncertainty always small enough for the conservative safety constraints to be feasible. In other words, if the uncertainty becomes too large, the underlying methods are no longer applicable.
Motivated by the aforementioned challenges, we present a learning- -bsee control algorithm that ensures safety by exploring on the fly. We use Gaussian process model es- timate model uncertainty, which combine with a control barrier function to compute robust safety filter. Whenever our safety filter becomes infeasible, we employ the control barrier function to guide exploration in a manner that is in- formative for safety. Formally, this is done by leveraging the lower confidence bound of the certificate constraint function for certifiably safe control inputs, and the upper confidence bound for exploration. Our approach draws heaviliy from ideas from Bayesian optimization, where exploration is done in similar manner. Overall, our main contributions are as follows:
1. Introduction
With the growing proliferation of robotics in safety-critical fields, e.g., autonomous vehicles, medical robotics, and aerospace systems, the need for methods that ensure the safety of systems and their users has become paramount. In control tasks, guaranteeing safety has become synony-
"quual contribution 'Robotics Institute, Cameeicc Mellon Uni- versity, Pittsburgh. PA, USA Department of Mechanical and Civil Engineering California Institute of Technology. Pasadena, CA, USA 'TUM School of Computation, Information and Technology, Technical University of Munich, Munich, Germany. Correspon- dence to: Alexandre Capone <acapone2 @ indrew.emuedd
We present the first safe control algorithm that ad- dresses infeasible safety filters by exploring online using a Gaussian process bandit algorithm.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Learning Safe Control via On-the-Fly Bandit Exploration
Starting state
Feasible (safe) region
Infeasible (safe) region
Unsafe region
System trajectory
X
(a) System state exits the region where the safety filter is feasible.
(b) Our algorithm computes exploratory control inputs using bandit-type tech- nique.
(c) Theorem 3.2 guarantees that the safety filter always becomes feasible before the boundary of the safe set is reached.
Figure 1. Illustration of our method (Algorithm 1) and theoretical result (Theorem 3.2).
2. We rigorously analyze approach and discuss our main theorem, which shows the interdependence be- tween model uncertainty, sampling time, and safety.
R - R is used, define the closed-loop sys- tem:
4 = -(x) +g()))(())
(2)
3. We showcase our approach using two numerical simu- lations where we achieve safety using control barrier function, dynamics model with zero-mean prior, and no prior measurements. To the best of our knowl- edge, this setting cannot be addressed by current state- of-the-art methods.
We also assume to know upper bound Lg R4 for the time-derivative of the dynamics, specified by the following assumption.
Assumption 2.1. There exists a known positive constant Lg. such that || S L3 holds for all x E and u E U.
The remainder of this paper is structured follows. In Section 2 we describe the problem setting considered in this paper, together with some background on control barrier functions and Gaussian processes. Our main contribution, which includes an exploration-drive safe control algorithm and safety guarantees, is presented Section In Section 5, we apply our method using numerical simulations of cruise control and quadcopter. We finalize the paper with some conclusions in Section 7.
We define safety by considering a safe set C, as specified in the following.
Definition 2.2 (Safety). The closed-loop system (2) said to be safe (with respect C) if (0) E C implies xx)) for all t > 0.
We consider the case where the safe set C corresponds to the superlevel set of some known continuously differentiable function R - IR with a regular value !
2. Background and Problem Setting
C (x M(x) 2 0}.
We consider a system that affine in the control inputs
This specification of C allows to achieve safety through the control barrier function framework, which we discuss in the following section. This paper considers a compact C and a Lipschitz continuous functions as stated in the following assumption.
t == f(x) + g(z)u
(L)
where x E X' C Rd and u E 21 S IR"" and f R + R and g R" + Raxm are (partially) unknown locally Lip- schitz continuous functions that represent the drift dynam- ies and the input matrix, respectively. Eq. (1) represents broad class of physical systems. Though our method can be straightforwardly extended to the more general case i = fxx u), this structure significantly facilitates the com- putation of safe control inputs, as described in Section 2.1. We further assume that X' is an open and connected set and that 24 is compact. When locally Lipschitz controller
Assumption 2.3. The safe set C is compact and the function h admits Lipschitz constant Ln, i.e., (h(z) h(x')l < La 2|2 holds for all x' E X.
Given an arbitrary nominal controller Tnom X + U, we aim to design a control law л that renders the closed loop
A function & has 0 regular value if h((z) = =
du (a) o
2

---

Learning Safe Control via On-the-Fly Bandit Exploration
system (2) safe while following nom as closely as possible. We make the following assumption regarding the nominal controller:
Given the existence of a CBF, we note that this assumption is not very conservative. Intuitively, it states that we can violate the non-robust CBF by margin of at most
Assumption 2.4. The nominal controller Tnom is locally Lipschitz in X.
2.2. Gaussian Processes and Reproducing Kernel Hilbert Spaces
Assumption 2.4 is not restrictive, as it includes a broad class of controllers, including most types of neural networks.
We now introduce the tools used to model the the unknown functions f and g.
2.1. Safety through Control Barrier Functions
Consider a prior model of the system f and ÿ. which can be set to zero if no prior knowledge is available. For the re- mainder of this paper, we assume f 9 for simplicity. We then model the residuals f-f and g - 9 jointly by using GP model. To obtain theoretical guarantees of the model error as new data is added, we assume that the residual model belongs to reproducing kernel Hilbert space (RKHS). This is codified later in this section. In the following, we review both GPs and RKHSs, and provide some preliminary theoretical results.
Our method attempts to ensure safety using control barrier functions, which we introduce in the following.
Definition 2.5 (Control Barrier Function (CBF) (Ames et al., 2017)). C C X' be the O-superlevel set of continuously differentiable function h: _= + IR with 0 regular value. Furthermore, let be a radially unbounded, strictly mono- tonically increasing function with a(0)= = O. The function is a control barrier function (CBF) for (1) on C if, for all a X.
the growth
A GP is collection of random variables, of which any finite number is jointly Gaussian distributed. GPs are fully specified by prior mean, which we set to zero without loss of generality, and kernel k: X' x X - RR We model each dimension of the unknown dynamics f(x) + g())u using a separate GP. This is achieved by employing composite kernels
0h
sup M(,, u) A sup well
weM dr (z) (f(z)) + g(x))) -a(h())).
(3)
Intuitively, (3) states that there exists control input that decreases the time derivative of the CBF as the boundary of the safe set C is approached, effectively slowing down the system as the boundary gets closer. If f and g were known perfectly, we could leverage this property to compute certifiably safe control inputs by including a CBF increase condition constraint when synthesizing control inputs:
m
(z,zz') 2 kR. 2') +
Σ
Euykon ('), (6)
where - (x=,uTT) The kernels ky, and kg., capture the behavior of the individual entries of the residuals f and g. respectively. In this paper, we do not require specific form of kernel k,, and kg.,, though the choice of kernel significantly affects performance. We employ a composite kernel (6) because this allows us to leverage measurements of å to improve our learned model.
(x) argmin wEll
|/ Taom mm) (z)
(4)
s.t. i(x, u) ((h(2))
Given the affine form of this constraint, this safety filter is quadratic program (QP) that has closed form solution (Ames et al., 2017) and can be solved fast enough for online safe controller synthesis (Gurriet et al., 2018).
Consider N noisy measurements D A (2/9), Vi (4) Jy=1....N of the i-th entry of the time-derivative of the state, where
Since we are dealing with unknown systems (3) is not di- rectly applicable, and we need to formulate a robust CBF constraint that accounts for potential modeling errors. To this end, assume the CBF constraint is feasible up positive margin 0:
m
y =f((x()) +
Σ Mi (2(00)) + =((
and the measurement noise g(0) satisfies the following as- sumption.
Assumption 2.6 (Robust CBF Feasibility). There exists radially unbounded, strictly monotonically increasing func- tion a with a(0)- = and a positive scalar > 0, such that
Assumption 2.7. The measurement noise g(1) is iid ai sub Gaussian for every i E (I....,7).
Oh sup
(1) ( -a(h(x)) te uell Or
(5)
holds for all x E X
The posterior of the model at an arbitrary state z" is then
3

---

Learning Safe Control via On-the-Fly Bandit Exploration
given by
computed
(2,)(2") - k. (K + t ,n Yi:
Oh I.C u) (x) - La,N V ir /r (2r( u))
dr
(x") - kz(zz, 2²)) - k,, (Ki,N + o IN -1 ki.e;
0h (x) + LA.N (En (z, u)),
where k - (Ra(z", z2(1), )..,,, lz'",, ())))) and the en- tries of the covariance matrix are given by [Kg = k (z(((, ) In our approach, we employ M4,N as model of our system, and IN as a measure of uncertainty, which employ inform the data-collection trigger.
UCBN(Z, dr
(10)
where Ln > is the global Lipschitz constant of h, specified in Assumption 2.3 and
In the following, we refer to the mean and covariance matrix of the full multivariate Gaussian process model as
NN - max Bi + oz.m 2 + + log (nd (nnd-1))
(11)
(x(+) (1.,,())...,,,,,,)) 2 ()) e di on,
Mn, T
(8)
Here, Vi denotes the maximal information gain after N rounds of data collection, and is computed as
Furthermore, we employ DN = {z(0), ()()) y I....... where y(s) A (u( ...., yo )T, to refer to the full data set corresponding to system measurements.
H.N
max 21 (N))
log IN + o K
To be able ensure safety, we need to estimate how data affects the worst-case model error. Though this is impossi- ble in a general setting, an estimate is possible if we assume that the entries of f and g belong to an RKHS, a very rich function space specified by the composite kernel (6).
Intuitively, the LCB (9) and UCB (10) respectively corre- spond to the lowest and highest permissible value for h(x,, u) given the model error bound.
Given the nominal controller Tnom. we leverage the LCB (9) to compute robustly safe control inputs by solving the second-order cone program
safe (x) argmin
| mnom (x)ll2
Assumption 2.8. For every i = 1,.., n, the i-th entry of (z))) belongs to the RKHS with reproducing kernel ks. and the corresponding RKHS norm is bounded by a known positive scalar B, E (0)..).
uell
ICBB(E, -a((())) +
(12)
Note that, due to the nature of the composite kernels (6), M u) is linear function of and ΣN (1, u) is di- agonal matrix whose entries are positive definite quadratic functions of u. Hence, (12) can be solved straightforwardly with conventional second-order cone program optimizers.
The satisfaction of Assumption 2.8 hinges on choosing the kernels correctly. In practice, if little is known about the system, then so-called "universal kernels" (e.g. Matérn or squared-exponential kernels) are often employed, which can approximate continuous functions arbitrarily accurately (Micchelli et al., 2006). Furthermore, there exist several techniques to mitigate kernel specification that can easily be applied to the methods described in this paper (Berkenkamp et al., 2019; Capone et al., 2022; Fiedler et al., 2021).
We include the term & in the constraint in (12) because this keeps the system from getting too close to the boundary. This ensures that when (12) becomes infeasible, there is still some distance go until it reaches the boundary of the safe set, leaving us with enough time explore the system.
As long as the robust safety filter (12) is strictly feasible, the resulting control policies are locally Lipschitz continuous (Castañeda et al., 2022), yielding unique and well-behaved closed-loop system trajectories.
3. Safe Control Through Bandit Exploration
If (12) is feasible for all C, we can leverage standard results from CBF theory (Ames et al., 2017) to show that the resulting trajectory is safe. However, since (12) includes model uncertainty, it becomes potentially infeasible in re- gions of the state space where uncertainty is high. We address this by employing a bandit-type exploration algo-
In this section, we present our main algorithm and theoreti- cal guarantees.
Our algorithms employs two key quantities: the lower confi- dence bound (LCB) and upper confidence bound (UCB) of the time derivative of h given the data. These quantities are
4

---

Learning Safe Control via On-the-Fly Bandit Exploration
rithm to acquire new data whenever infeasibility occurs. Our algorithm is designed collect sufficiently informative data to render (12) feasible before exiting the safe set. We now describe the corresponding sampling scheme in detail.
Algorithm 1 Safe Control via On-the-Fly Bandit Explo-
ration
Input:
Sampling time At, GP prior, CBF h, class-K func-
tion
1: Set EXPLORE = FALSE
Let N denote the number of data collected up to the current time. If (12) is strictly feasible, is used to compute control inputs. As soon as strict feasibility is impossible, we switch our bandit exploration strategy, This corresponds to the first instance in time when
2: for 1. E [0, o0) do
3:
if EFAL then
4: if max LCBB (z, u) > -a(((x)) + then
5:
Solve (12) and apply (a))
else
max LCBr(1,,U, -o((())) NEWl
(13)
Set EXPLORE-TRUE
8:
Set N = 11
9:
Set IN = L.
holds and can be easily verified using quadratic program.
10:
Set x(() =
We denote this point time by IN+1- At this point, we switch to an exploration-driver controller N which can be chosen freely, provided that it is locally Lipschitz tinuous and satisfies 7 (xx((x+)) = (N++)), where (N++) maximizes the upper confidence bound
11:
Compute (N) by solving (14).
12:
end if
13:
end if
14:
if EXPLORE==TRUE then
15:
ift < tN At then
(++1) arg max UCBW well
16:
Apply locally Lipschitz controller क with {(x(N) = (N)).
(14)
17:
Collect noisy measurement y(N) f(tx)- (x(tt) 9 (I(tN)) (N) g(N)
In our experiments in Section 5, choose (2x) = u (N) Choosing control input this way is very similar to bandit exploration algorithms, which aim to find the maximum of an unknown function. In our case, we wish to maximize the unknown time derivative of the CBF h(z u) given the state a, which we know to satisfy Assumption 2.6. We apply N++1 for a time length of At, where Al corresponds to the data sampling frequency and is specified priori. We then collect a noisy measurement of the time-derivative of the system
18:
else if iN + At then
19:
Set EXPLORE==FLSE
20:
Set DN DN-1 (2(N), y(N)) and update
GP.
21:
end if
22: end if
23: end for
= f(x((x++)) (alls)))))(((((())) + ((((( + g(N++)
detrimental control.
We then show that, if we choose At high enough, the closed- loop system under Algorithm is safe with high probability.
and update the GP model with the measurement data pair (z (N11)
y(N+1)), ()N++.))) where
Theorem 3.2. Let Assumptions 2. 2.3, 2.4 and 2.6 2.8 hold. Choose 8 E [0,, 1] and BN as in (11). Moreover, choose the sampling time as
z (N+1)= ((a(tx==)) (u(Vtt)) (u(tt))))(((((tt ")
After time of At has passed, we check the optimization problem (12) is strictly feasible. To this end, we check
At
(16)
La LOLA LALAAN
where AN max satisfies
max LCBN (z. u) -a(h()) NEW/
(15)
3232
L2
AN max 22 2oo (1 1+ G.
A
N...
(17)
holds. If so, we apply sa.e Otherwise, we repeat the sampling procedure. These steps are summarized in Algo- rithm 1.
2
Let the initial state (0) be strictly within the safe set, such that u(h(((())) 2 e. Then, with probability at least 8, the closed-loop system under the policy specified by Algo- rithm is safe. Furthermore, the closed-loop stops collect- ing data after most ANmax collected observations.
Remark 3.1. The condition (13) specifies that exploration starts when strict infcasibility is not given, as opposed to waiting until infeasibility occurs. In general, it is not ad- visable to wait for infcasibility to start exploring, as this means that the control law not well-defined, which can be
The proof of Theorem 3.2 can be found in Appendix B.
5

---

Learning Safe Control via On-the-Fly Bandit Exploration
66
60
60
CBF level set
CBF constr.
35
20
50
-20
WW
40
0
5 10
15
20
25
30
40
20
Time
Time
(b) Minimum of LCB of CBF time derivative for cruise control example. Spikes are due to training data set updates, leading to decreased model uncertainty. Positive values indicate infea- sibility of the robust safety filter (12).
(a) Value of control barrier function h(z) for cruise control example. Since prior model is available for control, safety is obtained by efficientl learning model online.
Figure 2. CBF and LCB for single cruise control simulation using sampling time of At = Ss
Intuitively, Theorem 3.2 states that if we choose the data collection frequency high enough, we guarantee safety after collecting a finite number of data. The quantity Pm Σi= Nam is the only part in the right-hand side of (17) that depends on N It is closely related to the regret metric frequently found in bandit literature, which measures the deviation from optimal decisions. This quan- tity grows sublinearly for most commonly employed kernels, meaning there exists Aa that satisfies (17). In practice, we can approximate by sampling the state and input space. Alternatively, we can obtain a (potentially crude) bound for this term based on the kernel hyperparameters. This is discussed in Appendix C.
(14) corresponds to choosing the optimistically safest input under uncertainty. Hence, it is reasonable to expect the corresponding input to be acceptable to the system, i.e., not damaging.
82
The primary focus of this work is the safety the closed- loop function, and we do not address closed-loop control performance. However, the tools presented here can be straightforwardly extended to address performance, e.g., by designing data collection scheme that aims to reduce model uncertainty whenever control performance is not satisfactory. Furthermore, our method can be potentially improved by preemptively exploring before infeasibility occurs, increasing the space of permissible control actions.
4. Discussion and Limitations
Although Assumption 2.6 seem strict, it still offers flex- ibility, and there are ways relax it in practice. Typically, only a portion of the safe set is visited during control. Hence, it is enough if the CBF is valid only for the corresponding subset of the state space. Furthermore, we can achieve con- servatism by initially restricting the safe set to a region that is easy control and then iteratively expanding data is collected. This potentially allows us to gradually improve the CBF as more system knowledge allows to expand the safe set. Alternatively, we can include conser- vatism in the safety requirement by computing the CBF condition from collection of (potentially valid) CBFs and taking the worst case.
Although we assume that the system structure (1) is affine in the control input, this is not strictly requirement for theoretical guarantees and can be relaxed, provided that the optimization problems (12), (13) and (14) can be solved adequately.
more
Theorem 3.2 points interaction between various quan- tities, including the sampling frequency, model uncertainty, measurement noise, CBF violation tolerance, and Lipschitz constants of the system and CBF. In simple terms, Theo- rem 3.2 indicates that higher sampling frequency is likely more beneficial for safety. However, other practical con- siderations can also improve safety. For instance, we can reduce the Lipschitz constant Li of the system by cleverly restricting the permissible control input space U. thereby reducing the bound (17).
5. Numerical Experiments
We now showcase how approach performs using two nu- merical simulations: a cruise control system and a quadrotor with ground dynamics. We assume to have either no prior model (cruise control example) or to know only the model component corresponding to the time derivatives (quadro- tor). Note that this is insufficient to implement any state- of-the-art approach, the prior model does not include
A practical concern that may arise with control inputs u geared towards exploration is that the corresponding inputs may strain the system and lead to undesirable behavior, e.g.. if the input exhibits high frequency and amplitude. However, computing the input by maximizing the UCB
6

---

Learning Safe Control via On-the-Fly Bandit Exploration
3.-
(a) The quadrotor starts at state within the safe region where (12) infeasible.
(b) Our algorithm computes exploratory control inputs without leaving the safe set. The observations are used to update the model. This phase corresponds roughly to the first two seconds of the simulation.
(c) After finite amount of data is collected without exiting the safe set, the uncertainty is small enough to achieve the feasibility of the safety filter (12). This is expected from Theorem 3.2.
Figure 3. Visualization of a single trajectory of the quadrotor experiment using a sampling time of At = 10 10-
30
CBF for height
Dif. Hatness-- based CBF
1000
25
Est. CBF decrease cond.
800
c00000
hright flatmest- besed
CBF level set
20
600
600
400
15
400
200
10
200
0.5
-200
0
10
20
30
40
50
10
20
30
40
50
Time
Time
(a) Value of control barrier functions for quadrotor example.
(b) Minimum of the LCB of the CBF time derivative for quadrotor example.
Figure 4. CBF and LCB for single quadrotor run using sampling time of A 1s
the influence of the control input. We additionally perform experiments that aim answer the following questions:
corresponds to the desired velocity. We employ squared- exponential kernels model f and g and assume to have N 10 data points at the start of the simulation, which we employ exclusively to learn the kernel hyperparameters. We do so by minimizing the posterior likelihood (Rasmussen & Williams, 2006).
How does our method compare with random explo- ration during infeasibility?
How does the choice of sampling time At affect safety?
We simulate the system for 100 seconds. The CBF value for single simulation run using a sampling time of At 10 10-5s s is shown in Figure 2a. The minimum of the LCB is depicted in Figure 2b. The CBF value h(()) is always above zero, meaning the system is safe. This is to be expected from Theorem 3.2. As can be seen in Figure 2b, the min- imum of the LCB is frequently positive in the beginning of the simulation. This means that the safety filter (12) infeasible during these time instances, particularly in the beginning. This leads high rate of exploratory inputs, which suddenly decreases model uncertainty, causing the LCB to spike. After approximately 6 seconds, enough in- formation was collected to recover feasibility, after which a safe input can be obtained without further exploration.
5.1. Cruise Control
We employ our approach to learn the road vehicle model pre- sented Castañeda et al. (2021) while applying an adaptive cruise control system. The states are given by x = [v 2]", where the distance between the ego vehicle and the target vehicle in front, and v denotes the ego vehicle speed. Our setup is fully described in Appendix A.1.
As control barrier function, we employ h((x)= = Twv, where Th = 1.8, which aims to maintain a safe distance between the ego vehicle and the vehicle in front. The nomi- nal controller Taom (x) used for the robust CBF-SOCP (12) P-controller Tnom 10 va), where va = 24
7

---

Learning Safe Control via On-the-Fly Bandit Exploration
Our agpronch
Random imputs
Our approach
Random inputs
Rate of fallure
Rate of failure
0.5
0.5
10 Hz
100 Hx
1000 Ha
2000 H&
Hx
100 Hz
1000 Hz
2000 H&
(a) Cruise control.
(b) Quadrotor.
Figure 5. Rate of failure using our approach and random inputs for the cruise control (left) and quadrotor (right) settings varying sampling frequencies (At)-1.
5.2. Quadrotor
5.3. Comparison with Random Exploratory Control
We additionally showease approach using numeri- cal simulation of quadrotor with ground dynamics. The quadrotor dynamics are described in detail in Appendix A.2.
Theorem 3.2 states that it is sufficient to collect data by applying (14) to the system in order to guarantee safety. However, other types of control inputs may also satisfy this requirement. In the following, we investigate how our ap- proach performs compared to random exploratory control inputs. More specifically, we apply our method to the sys- tem with the following difference: instead of computing exploratory inputs by solving (14), sample the control inputs from uniform distribution on U. We again consider the adaptive cruise control and quadrotor settings and inves- tigate how our approach and the random input-based one perform using various sampling frequencies At. This is especially relevant, as many practical settings do not allow for arbitrarily high sampling frequencies.
The quadrotor states are given by x = [p. v, q]. where where IR3 the global velocity,
p E R33 is the global position, and q g the system orientation in quaternion for- [T w| consist of the thrust
mat. The control inputs acceleration T and the body- body-frame angle rates SO(3).
In this example, we employ two CBFs. The first is h(x)= 10(p- - v2), where Tg = 0.1, and is geared toward keeping altitude higher than zero. The second CBF utilizes the differential flatness of the quadrotor to restrict velocities, orientation we add rotation, and the orientation of the thrust vector. It is fully described in Appendix A.2. When computing the exploratory control inputs (14), we alternate between CBFs. The nominal controller Taom (x) used for the robust safety filter (12) corresponds to differentially flat controller, computed as in Faessler et al. (2018), and we consider bounded thrust, with [T'] < 15000. Similarly to the kernels and assume to have N 10 data points at the start of the simulation to learn the kernel hyperparameters.
We perform 100 simulations with different initial conditions, uniformly sampled from a region within the safe set. We report how often each method fails, i.e., leads to positive value for the CBF during the simulation. The average num- ber failures for both settings is shown in Figure 5. non-zero failure rate expected at low sampling frequen- cies since too little data is collected to learn a model quickly. However, approach nonetheless performs better than the random control input-based approach. This is because the in- puts applied to the system are geared towards recovering the feasibility of the robust safety filter (12), whereas random inputs are not. We also observe that data efficiency higher at higher sampling, leading to less collected data. This because higher sampling rate means that the elapsed time between data collection and model update is smaller, i.e., the model captures the true system more faithfully imme- diately after an update. This suggests that, if the maximal permissible sampling frequency is low, then particular effort should be put into updating the model as fast as possible.
cruise control setting, we use gguared-exponentg
We simulate the system for 50 seconds. The CBF value for a single run using sampling time of At = 10-5s can be seen in Figure 4a, the minimum the LCB is depicted in Figure Figure depicts the corresponding quadrotor tra- jectory. Similarly to the cruise control setting, infeasibility is frequently encountered initially, causing our algorithm to explore. After approximately 2 seconds, feasibility is recovered, causing the quadrotor to hover in place for the remainder of the simulation.
8

---

Learning Safe Control via On-the-Fly Bandit Exploration
6. Related Work
Acknowledgements
6.1. Control Barrier Functions
This work is supported by the DAAD programme Kon- rad Zuse Schools of Excellence in Artificial Intelligence (relAI), sponsored by the German Federal Ministry of Education and Research and by the European Research Council (ERC) Consolidator Grant "Safe data-driven control for human- centric systems (CO-MAN)" under grant agreement number 864686.
CBFs have been studied extensively in settings with per- fectly known dynamies (Agrawal & Sreenath, 2017: Ames al., 2017; Bansal et al., 2017: Wabersich & Zeilinger, 2018; Zeng et al., 2021). However, the setting where known CBF is used to guide data collection when dynam- ies are unknown is an ongoing research question that has recently garnered increased attention. In (Choi et al., 2020; Taylor et al., 2020), the authors employ a neural network to reinforcement learning set-
Impact Statement
learn the residual dynamics ting. The safety of the learned policy enforced by includ- ing CBF constraint during training. An event-triggered approach for improving the learned dynamics model is pro- posed in (Lederer et al., 2024), where CBF and the system structure are leveraged to guarantee safety.
This paper presents work toward advancing the safety of Machine Learning-based methods. Safety plays crucial role in developing Machine Learning techniques, as guaran- teeing safety greatly enhances the applicability of existing methods. We feel that developing and further improving safe learning-basee based techniques such as the one presented in this paper will have significant societal impact, as the spectrum of applications is substantial, including health- care, autonomous driving. human- interaction, and various other robotics fields.
6.2. Safe Exploration with Gaussian Processes
Gaussian processes have been used extensively in the con- text of safe control and exploration. The work of (Sukhija et al., 2023) leverages contextual bandits to safely tune a parametric controller. By formulating an upper confi- dence bound for the safety function, the method switches to safe parameters whenever the boundary of the safe set is reached. The works of Prajapat et al. (2022); Sui et al. (2015); Turchetta et al. (2016); Wachi & Sui (2020) con- sider exploration with finite state spaces, known dynamics, and unknown functions specifying safe state-action pairs. A lower confidence bound of the safety constraints ensures safe actions, whereas an upper confidence bound for the reward drives exploration. A significant difference between our method and those described above is that they explore safe states to expand the safe set and maximize return, whereas exploration in our approach is driven by the need to ensure safety. The methods of (Hewing et al., 2019; Koller et al., 2018) use a GP to formulate safety constraints over a finite horizon for a model predictive control algorithm.
References
Agrawal, A. and Sreenath, K. Discrete control barrier func- tions for safety-criticnn control of discrete systems with application to bipedal robot navigation. In Robotics: Sci- ence and Systems, volume 13, pp. 1-10. Cambridge, MA, USA, 2017.
Akametalu, A. K., Fisac, J. F., Gillula, J. H., Kaynama, S., Zeilinger, M. N., and Tomlin, C. J. Reachabillty- safe learning with Gaussian processes. In 53rd IEEE Conference on Decision and Control, pp. 1431, 2014. doi: 10.1 119/CCDC.2014..03366))
Ames, A. D., Xu, X., Grizzle, J. W., and Tabuada, P. Control Barrier Function Based Quadratic Programs for Safety Critical Systems. IEEE Transactions on Automatic Con- trol, 62(8):3861-3877 August 2017. ISSN 0018-9286 1558-2523.
7. Conclusion
doi: N0.10099AAC.2201.6638886
URL
www.i/i.wrg/b///o..... 06408. arXiv:
We have presented a learning- based control approach for exploring on the fly to guarantee safety. Using Gaussian process model, we have formulated a robust safety filter that guarantees safe inputs whenever feasible. Whenever infea- sibility occurs we revert to a bandit exploration algorithm that is geared toward ensuring that feasibility is recovered before exiting the safe set. We show that if the sampling rate is chosen high enough, we can provably ensure safety with high probability. In future, we aim to apply the proposed approach to real-life and other complex systems.
1609.06408.
Ao, Y., Esfandiari, H., Carrillo, F., Laux, J. As, Y. Li, R., Van Assche, K., Davoodi, A., Cavalcanti, N. A., Farshad, M., Grewe, B. F., Vander Poorten, E., Krause, A., and Fürnstahl, P. Saferplan: Safe deep reinforcement learning for intraoperative planning of pedicle screw placement. Medical Image Analysis, 99:103345, 2025. ISSN 1361- 8415.
Bansal, S., Chen, M., Herbert, S., and Tomlin, C. J. Hamilton-jacobs reachability: A brief overview and re- cent advances. In 2017 IEEE 56th Annual Conference

---

Learning Safe Control via On-the-Fly Bandit Exploration
on Decision and Control (CDC), pp. 2242-2253. IEEE, 2017.
Danjun, L., Yan, Z., Zongying, S., and Geng. L. Au- tonomous landing of quadrotor based ground effect modelling. In 2015 34th Chinese Control Conference (CCC), pp. 5647-5652, 2015. doi: 10.1109/ChiCc.0115. 7260521.
Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A. Safe model-based reinforcement learning with sta- bility guarantees. In Advances in Neural Information Processing Systems, pp. 908-919, 2017.
Faessler, M., Franchi, A., and Scaramuzza, D. Differen- tial flatness of quadrotor dynamics subject to rotor drag for accurate tracking of high-speed trajectories. 1EEE Robotics and Automation Letters, 3(2):620-626, 2018. doi: 10. 11110//RA.2201.2776653.
Berkenkamp, F., Schoellig, A. P., and Krause, A. No-regret Bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20(50):1-24, 2019.
Fiedler, C., Scherer, C. W., and Trimpe, S. Practical and rigorous uncertainty bounds for Gaussian process regres- sion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7439-7447, 2021.
Capone, A. and Hirche, S. Backstepping for partially un- known nonlinear systems using Gaussian processes. IEEF Control Systems Letters, 3:416-421, 2019.
Gurriet, T., Singletary. A., Reher, J., Ciarletta, L., Feron, E., and Ames, A. Towards a framework for realizable safety critical control through active set invariance. In 2018 ACM/IEEE 9th International Conference on Cyber- Physical Systems (ICCPS), pp. 98-106. IEEE, 2018
Capone, A., Lederer, A.,
and Hirche, S. Gaussian process uniform error bounds with unknown hyperparameters for cafety-critical applications. In International Conference on Machine Learning, pp. 2609-26624, PMLR, 2022.
Castañeda, F., Choi, J. J., Zhang, B., Tomlin, C. J., and Sreenath, K. Pointwise feasibility of Gaussian process- based safety-critical control under model uncertainty. In 2021 60th IEEE Conference on Decision and Control (CDC), pp. 6762-6769, 2021. doi: 10.1109/CDC45484. 2021.9683743.
Hewing, L., Kabzan, J., and Zeilinger, M. N. Cautious model predictive control using Gaussian process regres- sion. IEEE Transactions on Control Systems Technology, 2019.
Hewing, L., Wabersich, K. P., Menner, M., and Zeilinger, M. N. Learning- based model predictive control: To- ward safe learning in control. Anmual Review of Control, Robotics, and Autonomous Systems, 3:269-298 2020.
Castañedaa, F., Choi, J. J., Jung, W., Zhang, B., Tom- lin, C. J., and Sreenath, K. Probabilistic safe online learning with control barrier functions. arXiv preprint arXiv:2208. 10733, 2022.
Jagtap, P., Pappas, G. J., and Zamani, M. Control barrier
functions for unknown nonlinear systems using Gaussian processes. In 2020 59th IEEE Conference on Decision and Control (CDC), pp. 3699-3704. IEEE, 2020.
Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the AAAI conference on artificial intelli- gence, volume 33, pp. 3387-3395, 2019.
Khalil, H. K. Noninear systems. Prentice-Hall, New Jersey, 2(5):5-1, 1996.
Koller, T., Berkenkamp, F., Turchetta, M., and Krause, Learning-bais based model predictive control for safe explo- ration. In IEEE Conference Decision and Control, pp. 6059-6066, 2018.
Choi, J., Castañeda, F., Tomlin, C. J., and Sreenath, K. Reinforcement learning for safety-critical control under model uncertainty, using control Iyapunov functions and control barrier functions. Robotics: Science and Systems (RSS), 2020.
Krause, A. and Ong, C. Contextual Gaussian process bandit optimization. In Advances in Neural Information Process- ing Systems, volume 24. Curran Associates, Inc., 2011.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in handful of trials using proba- bilistic dynamics models. Advances in neural information processing systems, 31, 2018
Lederer, A., Begzadić, A., Hirche, S., Cortés, J., and Her- bert, S. Safe barrier-constrained control of uncertain systems via wrigg-trgggerd learning. arXiv preprint arXiv:2408.16 2024.
Curi, S., Berkenkamp, F., and Krause, A. Efficient model- based reinforcement learning through optimistic policy search and planning. In Advances in Neural Informa- tion Processing Systems, volume 33, pp. 14156-1411 14170. Curran Associates, Inc., 2020.
Micchelli, C. A., Xu, Y., and Zhang, H. Universal kernels. Journal of Machine Learning Research, 7(Dcc):2651- 2667, 2006.
10

---

Learning Safe Control via On-the-Fly Bandit Exploration
Nguyen, Q. and Sreenath, K. Exponential control barrier functions for enforcing high relative-degree safety-critical constraints. In 2016 American Control Conference (ACC), pp. 322-328. IEEE, 2016.
Zeng, J., Zhang, B., and Sreenath, K. Safety-critical model predictive control with discrete-time control barrier func- tion. In 2021 American Control Conference (ACC), pp. 3882-3889. IEEE, 2021.
Prajapat, M., Turchetta, M., Zeilinger, M., and Krause, A. Near-optimal multi-agent learning for safe coverage con- trol. Advances in Neural Information Processing Systems, 10.14998-15012. 2022.
Rasmussen, C. E. and Williams, C. K. Gaussian processes for machine learning. 2006. The MIT Press, Cambridge, MA, USA, 2006.
Rodriguez, I. D. J., Rosolia, U., Ames, A. D., and Yue, Y. Learning unstable dynamics with one minute of data: A differenniation--aasee Gaussian process approach. arXiv preprint, 2021.
Srinivas, N., Krause, A., Kakade, M., and Seeger, M. W. Information thhooretictteeeceeeeooooooooeett regret bounds for Gaussian process optimization in the bandit setting. IEEE Transactions Information Theory, 58(5):3250-322 3263, 2012.
Sui, Y., Gotovos, A., Burdick, J., and Krause, A. Safe exploration for optimization with Gaussian processes. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 997-1005, Lille, France, 07-09 Jul 2015. PMLR.
Sukhija, B., Turchetta, M., Lindner, D., Krause, A., Trimpe, S., and Baumann, D. Gosafeopt: Scalable safe explo- ration for global optimization of dynamical systems. Ar- tificial Intelligence, 320:103922, 2023. ISSN 0004-3702.
Taylor, A., Singletary, A., Yue, Y., and Ames, A. Learning for safety-critical control with control barrier functions. In Proceedings of the 2nd Conference on Learning for Dynamics and Control, volume 120 of Proceedings of Machine Learning Research, pp. 708-717. PMLR, 10-11 Jun 2020.
Turchetta, M., Berkenkamp, F., and Krause, A. Safe explo- ration in finite markov decision processes with Gaussian processes. In Advances in Neural Information Processing Systems, pp. -332, 2016.
Wabersich, K. P. and Zeilinger, M. N. Linear Model Predic- tive Safety Certification for Learning Based Control. In 2018 IEEE Conference on Decision and Control (CDC), pp. 7130-7135, December 2018. doi: 10.1 C0. 2018.8619822 ISSN: 2576-2370.
Wachi, A. and Sui, Y. Safe reinforcement learning in con- strained Markov decision processes. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pp.
PMLR, 13-18 Jul 2020.

---

Learning Safe Control via On-the-Fly Bandit Exploration
A. Full Description of Numerical Experiments
A.1. Cruise Control
In the following, we omit physical dimensions when describing the system model. The state space model the cruise control given by 1 f(x)) g(x) j,, with state-dependent functions
f(x) = -/ (So + <iv + 6avi) g
(18)
20
and state x = [v 2 where denotes the distance between the ego vehicle and the target vehicle in front, v denotes the ego =
vehicle speed, m = 1650 its mass, and So = 0.2, S1 = 10, <2 = 0.5 are parameters that specify the rolling resistance.
A.2. Quadrotor
The quadrotor dynamics are specified by the functions
U:
à = gg + Re.T,
(19a)
R = R[w]x
(19b)
where p E Rd is the global position, v E R the global velocity, and R E SO(3) the system orientation. The parameter Ogr 9.81 denotes gravity, Ix R3 - R3x3 is the skew-symmetric mapping, and e, [0 0 1] is the unit-z vector. The
function Ç R4 [0, 1] models ground effects and takes the quadrotor height pa as input variable. It is computed as Danjun et al. (2015)
2
((0z) = - pp
Tiul
(20)
Apz
where p and Tiot 0.09 is the rotor radius.
The first CBF is hxx) 10(p- T=,z), where T,, 0.1. It is geared toward keeping altitude higher than zero. The second CBF utilizes the differential flatness of the quadrotor to synthesize viable safe set. In particular, we concerned with restricting the system to safe positions defined as the O-superlevel set of hp(p)) = r² - llpl23. We extend hp to include velocities as he(p, v)) (p, v) (p) for some > 0 to produce relative degree-1 I CBF for double integrator system as in Nguyen & Sreenath (2016). To include orientation we add rotation term to produce CBF for the drone, 1 0h Re,), with some A E (2/)) which shrinks the safe set to ensure that the thrust vector is pointing inwards whenever (p. v) 0.
h(p, v, R) = (p. u) A(1 2r Bp
B. Proof of Theorem 3.2
We prove Theorem 3.2 in two steps. First, we show that the control inputs are well-behaved and the closed-loop system is safe the robust CBF-SOCP (12) is strictly feasible. Afterward, we show that strict feasibility is always recovered before exiting the safe set when using our exploration approach.
We begin by leveraging Assumptions 2.7 and 2.8 obtain a bound on the model error, which is crucial to establish safety.
Lemma B.1. Let Assumptions 2.7 and 2.8 hold and let
B; + di 2 (nHN + log (nō-
BN
max i5
Then, with probability at least 1-8.
llf(x") + g(x") yu' "n""-"(a",aa S 8N BNr (EN(x*, u*))
holds for all x E X, u E U and all N E N.
12

---

Learning Safe Control via On-the-Fly Bandit Exploration
Proof. By Capone Hirche (2019, Lemma 1), with probability at least 6,
f((x*) + ( - NI,NN2" x²)) B; + oi,ns y (76, + 11 log (nò (né-1))
Σ
oi, oN,N (x", (x², ,
==1
holds for all 1,..., n, all X, u E U, and all N E N. We then have
m
[If(x)) + g(x")='' )u- =1x"","" 2 11 -
fi(0
Σ
ge (z) -1.N(2")
1=1
No i (2*)
3 Roo
i=1
= Rxt (En(x*, u²
Lemma B.1 allows bound the GP model error any given point z" given any data with high probability. This plays crucial role in establishing robust system safety in the face of model uncertainty.
Safety under Striet Feasibility
We first demonstrate that when the robust safety filter (12) is strictly feasible, the control law WT ensures the system remains strictly within the safe set with high probability. Specifically, the control barrier function value of the closed-loop system increases over time whenever a<h(x)) < 2- To prove this statement formally, we first present some preliminary results.
Lemma B.2. Let the robust safety filter (12) be strictly feasible for some x E X. Then the policy N, specified by the safety filter (12) is Lipschitz continuous a neighborhood of x.
Proof. The proof is identical to that of Castañeda et al. (2022, Lemma 6).
Lemma B.2 is important to guarantee that the trajectory resulting from applying TT unique and well-behaved, allowing establish safety.
Lemma B.3. Let Assumptions 2.1, 2.3 and 2.6 2.8 hold. Then, with probability at least - 8, the following holds for all E X, all u E U, and all N E N:
LCBr(I, h(x, UCBx(z,
(21)
Proof. We show the result only for the lower bound LCBN since the argument is identical for the upper bound UCBy. By
13

---

Learning Safe Control via On-the-Fly Bandit Exploration
Lemma B.1, with probability at least 6,
0h
h(,, u) (x) (f(x) + g(x)u)
8x
0h
Oh
(x) (I u)) +
(a) (f(z) g()) ux(
dx
dz
0h
0h
(x) (1N(
[1[(()) + g(x)u un(a, u))l
Ox
dx
0h
Oh
(a) (IN(
(x)
AN BN tr (EN( + +
0x
dz
0h
Or (x) (H,( LN V (EN(r,
(I,
holds for all x E X, u E U, and all N N.
We additionally require the following result, which is direct consequence of classical result on CBFs (Ames et al., 2017, Theorem 1).
Lemma B.4. For some Tmax R U 00, let =(1), & [0, Tmax), be the unique and well-defined solution to the differential equation
$ = f()), BEX, #(0) $00
(22)
where f is Lipschitz continuous around a neighborhood of (t) for all t E Tmax) h+(()) > 0 and
ah (+(1))((((1) 2 - -a((((())) +
(23)
a5
holds for all t E [0, Tmaxx), then a(h(E(t))) > 0 for all t [0, Tmax)).
Proof. The proof is identical to that of Ames et al. (2017, Theorem 1). Note that, although Ames et al. (2017, Theorem 1) requires ff to be locally Lipschitz continuous in X' and (23) hold for any state in X, these properties are only used generalize the result all solutions of (22) with arbitrary ((0) E X, which we do not require in this case.
We now show that finite-time trajectories obtained with the controller safe are always strictly within the safe set, provided that the safety filter (12) strictly feasible.
Lemma B.5. Let Assumptions 2.1, 2.3 and 2.6 to 2.8 hold. For all N N, define
XN = E
max I., -ax (h(x)) +
wELl
and the dynamical system
IN = f(IN) + g(xN) ( Taate JuseeeNN (IN)), t [ At,tN+)))
(24)
specified within the time interval [tN + At, (N1). where the initial condition satisfies (y + At) E XN, tN41 is given by
c 2 25
t A inf t2tN tN + At
max LC = - -a(((=)) WEld
(25)
and (h((xx(tx + At))) > O, and saafe.N is given by (12). Then, if XN non-empty, (24) has a unique solution x (t). Furthermore, with probability at least the trajectories x (t) satisfy the following conditions for all N N:
14

---

Learning Safe Control via On-the-Fly Bandit Exploration
(i) zy(() E C. E [lN (I)(N-1)
(i)) I (I)) < for any i E [IN + At,tN), then a (h(ax(())) > a (M(xx(0)) for all t (i, (N).
(ii) I (h(@N (0)) s 5/2 for any i E 1NN + b, then a (h(#v(()) (h(zx for all t [i,
Proof. We begin by showing condition (i), then show (ii) and (ii).
(i) Due to the definition of A', the safety filter (12) is strictly feasible for all E Xy. Furthermore, the definition of +1 implies that a candidate solution IN (t) satisfies (() E XN for all 1. E [lN + At)( Moreover, N, is well-defined and strictly feasible in Xx. By Lemma B.2, N is Lipschitz continuous around a neighborhood of any x E Ax.. Since and g are locally Lipschitz continuous, this implies that f(z) + Hence, by the Picard- Picard-Lindelöf Theorem, (24) has a unique solution zx((). (t). Since Xy is bounded and zy E Xy for all t [ty At, IN+1), (t) is also well-defined for all t E [tN (+1)++). Moreover, due the definition of Nafeee Lemma B.3 implies that, with probability at least 1 0, 6,
is locally Lipschitz continuous in Ax.
Oh dr
(IN) (f(#x) + (( -a((((w))) +
holds for all N and all IN E XN. By Lemma B.4, (t) E C holds for all t E [ty + (x,))+1) and all N E N with probability at least 1 - 5.
(ii) The following holds with probability at least 1 - 6: For all xx(() and all N E N, if (((((N((())) < for any e [tN + At,t)+1), then, by Lemma B.3 and the definition of Tsale.N:
h(z, ae (EN (t))) -a (h(z(()))
6 =0 2 -
Due to continuity with respect to time, this means there exists an instance time to E (t', tv41) and an ē > 0, such that h(#N(t) (f), N (z (t))) > holds for all (t', By the Comparison Lemma (Khalil, 1996). h(xx > b(ZN (f") + E(t. - f' > (t)) for all t (f',e) By the same argument, h( (t)) is strictly increasing t whenever Mxx((t (t)) < (z)((t))
2. hence h(zx(t)) (t)) for all f (fc, Since is strictly increasing, this implies
the desired result.
(iii) The proof is identical to that of (ii,, except that the strict inequality is replaced by non-strict one.
A consequence of Lemma B.5 is that safety is guaranteed if infeasibility never occurs within the safe set:
Corollary B.6. Let Assumptions 2.3, 2.4 and 2.6 to 2.8 hold. With probability 1 8, if
max (CBN(2, u) > a(h(2)) + 2' VxEC well
(26)
holds for any N E N, Nnfe, is well-defined for all x E C. Furthermore, tN41 oc and the closed-loop system
IN [(xN) + g(xN)) J (T. LE [LN + xN,
is safe.
Proof. Due to the definition of Tsaee,,, (26) directly implies that its feasible region is non-empty, i.e., Taaee is well-defined, for all x E C. We now show that 4x+1 = oc by contradiction. Hence, suppose ( < co holds. By Item (ii) in Lemma B.5, the trajectory xN stays strictly in the interior of C for & E [IN + AN, fx+1). Due to continuity with respect to time, (1 = lime-tt-- (1) E C. Recall that (N41 denotes the earliest instance when the CBF-SOCP specifying N is no longer strictly feasible. Since (t) e C and s is strictly feasible in C, this is a contradiction, meaning tN+1 co.
Recovering Feasibility Through Bandit Exploration
We now analyze the effect of data collected under Algorithm on the feasibility of the safety filter. Specifically, we demonstrate that by gathering enough data before exiting the safe set C, strict feasibility ensured anywhere in C. To
15

---

Learning Safe Control via On-the-Fly Bandit Exploration
achieve this, will need the following preliminary results, which bound the growth of the cumulative worst-case model emor.
Lemma B.7. The posterior variance satisfies
N Σ 4=1
oi (2l0-1)) 5 log (1 + di
K.N
(27)
for all N, all and all 200,, ( &.
Proof. Employing the argument in the proof of Srinivas al. (2012, Lemma 5.4), get
2 (4)
log (1 + 0, o. (2(5))
log (1 os,ss)
The result then follows from Srinivas al. (2012, Lemma 5.3).
Lemma B.8. For all N, the posterior covariance matrices satisfy
N
2 Vi log (1 )
Σ -
tr
(28)
Proof. By Lemma B.7,
Σ (Ev(x(N), ,((0))
Σ Σ of (11)
2
N.N-
log (1 + dins)
i=1
Lemma B.8 bounds the growth the posterior covariance matrices.
We now show that strict feasibility in C is recovered after collecting most A ANmu data points.
Lemma B.9. Let Assumppionss 2.3, 2.4 and 2.6 to 2.8 hold. The following holds with probability at least 1 8: If Algorithum I collects A N measurements in C, where A N satisfies
3232
T2
AN max
AN
Σ
Y.AN...
2 log (1 J,nss
then the safety filter is strictly feasible everywhere in C.
Proof. We employ a proof by contradiction. Hence, assume that after AN
observations have been collected, there
exists an d C, such that
max LCB (2t -a(h(z)) e
(29)
well
In the following, set
(ANNa))) A x
(NNu)) A arg max LCB, New. (z u).
well
16

---

c z. N Learning Safe Control via On-the-Fly Bandit Exploration
Recall that Algorithm I only collects observations x:(N), (u(() if (12) is not strietly feasible, i.e.,
LCBN (2).)) ,,(()))
max LCBN (x((), ,u)
S
weul
<-a (a(b((())) + 2
for all u E U. Furthermore, due to Lemma B.3, with probability at least 1 -- Ō,
sup i (2(N), (alM)u) S WEL
max UCBN (D6M) well
-UCB (I(N), (2(M),(()))) u(m))
holds for all u E U and all N E N. By employing the identity
LCBN( u) UCBN (2, u) 2L,BN (EN u)),
then obtain, with probability least 1 - 5, for all N N:
-a(b(z(N2)) + 2CCB (2(N) u())
ZLCB (2(N), (1())) well
sup h(r(O),)))))))))))))) UCBN (2(N), vd(o))
sup sup h(o .u) -u))-Tnn v ir (IN(x()), a(N3)) uell
2 - a(h(x((0)) + t 2L urr (EN u())))
where the last inequality is due to Assumption 2.6. Hence, with probability at least 6,
2L,3N v tr (Ex (o00),(00))
By squaring and summing up from to A then employing Lemma B.8 and (29), we get
ANas
( 2 = Σ
max
2
A uas
2L BL ir (EN(x( (ooo))) 2((23))
N=1
A
2 Naaa
4
Σ (EN(a(()) u(()))
N=No
S.A. Nena
4L3
log (1 + + -2 o
V.AN Nan
This implies
32,3% Nun 2 log
AN
Σ NN...
(1 of 1m
which is a contradiction. Hence, the safety filter is strictly feasible in all of C.
Next, we analyze the amount of data collected under Algorithm before potentially exiting the safe set C. Specifically, we show that the closed-loop system under Algorithm collects at least AN Nma data points before exiting the safe set. To do so,
17

---

Learning Safe Control via On-the-Fly Bandit Exploration
will require the following result:
Lemma B.10. Let Assumptions 2.1, 2.3, 2.4 and 2.6 2.8 hold. Then, with probability least 1 8: consider the closed-loop system (2) and assume there exists time interval [t', t"] with t' < t", such that
a(h(x(t)) 2 a(h(x(t)) <a(M(z(t)); AL (t,
(30)
and let
t maxtN: S.t. tN St NEN
(31)
denote the last time instance when data is collected up until V. Then, the safety filter is not strictly feasible at times t' and t, and
(a(h(a((x))) - -a(((((())) LI
holds, where At is the sampling frequency used by Algorithm
Proof. We begin by showing that the safety filter is not strictly feasible at time t' by contradiction. Hence, assume that the safety filter is strictly feasible L'. Since ((h(x((!)))) S ş. by Item (iii) in Lemma B.5, (<((())))) 2 2 holds for some t e (t', (" This contradicts (30).
We now show that strict feasibility does not hold ty by contradiction. Hence, assume strict feasibility holds at txy. Due to the definition of t y (31), new data point is not collected until strictly after t'. This implies strict feasibility time t', which is contradiction.
Since the safety filter is not strictly feasible at time Ix,, and Lx is the last time instance when data is collected before L', from the sampling frequency At, we have
At t - tx > tN-
The result then follows from the Lipschitz continuity of a. x and h.
We now show that if the closed-loop system ever exits the safe set, it must collect at least ANmax data points before doing so.
Lemma B.11. Let Assumptions 2.1, 2.3, 2.4 and 2.6 2.8 hold and let z(0) be strictly within the safe set, such that ((((((0))) 2 Then the following holds with probability at least 1 - 8: if the closed-loop system under Algorithm ever exits the safe set C, then it collects at least N data points before doing so.
Proof. Assume that the closed-loop trajectory x(t) reaches the boundary of the safe set C at some time taxi before exiting it:
texit sup { 2 a(h(=(t)) 2 S [0,
(32)
Due to the continuity of x, and h, and a<h()x())) 2 e, there exists time tg texits such that
ts sup { 0 a(h(((()) = /. a(h(z(t)) < S. VtE (f, } Let N₂ and Nexit denote the number of collected observations at time to and fexir. respectively:
(33)
N₂ arg max INs NEN
s.t. 5/g
Next = arg max tN. NEN
s.t. to S texit-
18

---

Learning Safe Control via On-the-Fly Bandit Exploration
Then, with probability at least - 6: Due Lemma B.10, have
|a(h(a(tx, )) ((((2(())))))))))))))))))(()))() LOLA [a(h(I((NN.)))) a(h((((eex)))) S LALnLAA
(34)
(35)
We now analyze the growth of a(h(r(t))) between tN and IN+1- We separately analyze the case where the safety filter is strictly feasible and the where it is not. If the safety filter is not strictly feasible at time 1--1, due to Assumption 2.1 and the sampling frequency At, the CBF changes between -1-1 and & most by
la(h(a(t>))) a(h(c((x-1)))
ILgLa - tv-1l
(36)
LL
By Item (iii) in Lemma B.5, if the safety filter is strictly feasible at time (x-1 and (<h((((N--1)) s ş. then a(h(x((())) a(h(x((x-1)) Hence, for all Ix-1 E (4g fexi]. whether the safety filter feasible or not,
a(h(x(tN-a)) a(h(x(tN))) s LILaLh
(37)
holds. We then have
=a(b(x(t))) (((l((t)))((((((taaa))) a(h(x(teui)
2
=a(b(x((4)) - a(h(x((enii) a(h(x((N.m)) - -al00-a(((((aa a(h(o((NN))) a(h(x((NNa))) a(h(o((xg))) sa(M(a((N))) - a(h(o((NN.))) + TIgA
Nad
Σ
-n Ln
N=N++1
(Noxit N2 1)LOLALLAA LLLLa
-Noxit + 1 Ng)LolnLAAI
where the first inequality is due to (34), and the second inequality due to (37). This implies
Nesi N₂ +12 LLgLaL
AN
where the last equality follows from the sampling frequency specification (16). Since Next, Ng and A are integers, this implies that the amount collected data satisfies Nexi N₂ 2 ANmax.
We now put everything together and prove Theorem 3.2.
Proof of Theorem 3.2. By Lemma B. Algorithm collects at least A Nma. observations before ever exiting the safe set C. Lemma B.9 states that the safety filter is strictly feasible within all of after collecting at most A observations. Together with Corollary B.6 this implics that, with probability at least 6, the closed-loop system is safe and Algorithm collects at most A AN Nmax observations.
C. Kernel-Dependent Bounds for AN Nmax
Theorem 3.2 specifies that
3282
L3 A Ni..
A max
22 log
(1 + o.nss
N...
(38)
19

---

Learning Safe Control via On-the-Fly Bandit Exploration
sufficient condition to guarantee safety. We now analyze how the right-hand side term behaves for different kernels. We begin by analyzing how the maximal information gain "i depends the maximal information gain of the kernels k,, and Kggs used to model f and
Lemma C.1. Let & be given as in (6), and let "Y, and NNN denote the maximal information gain after N rounds of data collection under the kernels k and kig.. respectively. Then
7,N S TA,N +
Jon.. + 3m (og(N)
(39)
Proof. The proof follows directly the definition of the composite kernel k; Krause & Ong (2011, Theorem 2) and Krause Ong (2011, Theorem 3).
We now dissect the right-hand side of (40). Define 7AN, - maxign Nm.. and B A maxiga B,. From the definition of BaNma it follows that
BA N
Auaa S BBaNan 11A Nuxx
= B MYNN 2B / (7ANN 1+ log logg(n-11)) (nó-1)) + mANNa Rim (1ANmu + 1 log (nd (nd-
3 <Ci MANNa + Cz as 12 Am.. + C2(B, oi,ss, 1))2N.... Num
< (C((B, 12 + Cg (B, B i. n) (B, in mi) max Nuns 1}
((Ci(B, 11 + Cg B, ns n) CB (ims (72N + +),
where
(B nB2 + n2Bo,, log (nd (no-1)) + no + log (no-1))
C3 [B dt.ss n) = n22Bo 2
Cz (B, oi,,s, n) 2no? i.ns
For most commonly used kernels. the growth rate the maximal information gain is of the form O(N" log( where S w (Srinivas al., 2012). Hence, using Lemma C.I, we can bound 7N as YN (log(()(N))) for some positive Cy, w and f. and N large enough. Using estimate of this type, can compute (potentially crude) upper bound for the right-hand side of (40) as
3282 N.... L
max Σ MNNND I CCB n)) (C,,N (log(N))) +1 S Č(B s, A2 2w max
(log(N))22 (40)
22 log (1 + i, i=1
where C' and Ö are appropriate constants.
This yields the sufficient condition for choosing AN
AN (C(B, n)ogg (A A
(41)
which can casily be solved using numerical methods. For -quared-exponente kemels,, w 0 and 0 0= For linear kernels, = 0 and 0 1, whereas for Matérn kernels, 0 = O and depends on its smoothness parameter (Srinivas et al., 2012).
20


Reviews:
[{"review_id": "UNTfhXMV6E", "reviewer": "Reviewer_kKqd", "summary": "The work presents an approach to design certifiably safe feedback controllers for a priori unknown systems. The authors propose using gaussian processes to approximate the learned model (or its error with respect to the true model). Then, they leverage bandit theory to propose upper and lower bounds to the model estimates, using the lower bounds for ensuring safety through a control barrier function and deciding when infeasibility occurs. Under infeasible conditions (when the controller is not able to ensure safety given the current model bound), the method uses the upper confidence bound to guide exploration and collect data to improve the model estimates. They provide theoretical results showing that under assumptions, the method ensures safety up to a given probability bound. They demonstrate the effectiveness of the work in several control problems.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you kindly for your review. Below, you will find our answers to your comments.\n\n**Regarding the claim of solvability of the benchmarks:** This claim is tied to the controller having a prior mean model of zero, no measurement data, and only a CBF to guide it. Although existing methods can effectively control these systems, the control design in existing works is informed by a prior (non-zero) model of the dynamics. Alternatively, a backup controller is required. Our approach only requires the CBF and a well-specified GP to inform the control design. We will reformulate the sentence and include this discussion in the revised paper.\n\n**The choice of kernel is not obvious:** We agree with this statement. Choosing appropriate kernels for modeling dynamical systems is generally a nontrivial task, and there exists an extensive body of literature on how to do so while mitigating model misspecification; see, e.g., Berkenkamp et al. (2019), Fiedler et al., (2021), Capone et al., (2022). Crucially, although many kernels can satisfy the requirements of our approach, some kernels can generalize far better than others, significantly reducing the amount of data required to achieve safety. For example, if the system has a linear component, it is typically beneficial to include a linear component in the kernel, and sometimes, a linear kernel can even outperform a universal kernel in this sense. We will include this discussion in the paper.\n\n**Knowing a CBF without knowing the dynamics:** While it is generally impossible to formulate a valid CBF without any form of model knowledge, doing so is easier than obtaining an actual system model, since the existence of a CBF typically implies the existence of (potentially infinitely) many others. Furthermore, the assumption that the CBF is valid can be relaxed in many ways: the CBF only needs to be valid in parts of the state space that are visited by the closed-loop system. Moreover, our approach can gradually expand the safe set, enabling incremental improvements to the CBF. We also note that the assumption that a CBF is known despite not having an accurate model is common in the literature; see, e.g., Cheng et al. (2019), Choi et al., (2020), Taylor et al., (2020). \n\nBerkenkamp, F., Schoellig, A. P., & Krause, A. (2019). No-regret Bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20(50), 1-24.\n\nCapone, A., Lederer, A., & Hirche, S. (2022, June). Gaussian process uniform error bounds with unknown hyperparameters for safety-critical applications. In International Conference on Machine Learning (pp. 2609-2624). PMLR.\n\nCheng, Richard, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. \"End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks.\" In Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, pp. 3387-3395. 2019.\n\nChoi, J., Castañeda, F., Tomlin, C. J., & Sreenath, K. (2020, July). Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions. In Robotics: Science and Systems (RSS). 2020.\n\nFiedler, C., Scherer, C. W., & Trimpe, S. (2021, May). Practical and rigorous uncertainty bounds for Gaussian process regression. In Proceedings of the AAAI conference on artificial intelligence (Vol. 35, No. 8, pp. 7439-7447).\n\nTaylor, A., Singletary, A., Yue, Y., & Ames, A. (2020, July). Learning for safety-critical control with control barrier functions. In Learning for dynamics and control (pp. 708-717). PMLR.", "claims_and_evidence": "The claims proposed are well supported, and the theoretical results and experiments justify the contributions proposed.", "methods_and_evaluation_criteria": "The theoretical results are sensible, and the proposed experiments seem adequate for the method proposed.", "theoretical_claims": "I checked the presented proofs for the main theorem (and necessary lemmas presented in the appendix) to the best of my capacity (given the reviewing load). The results seem correct and the proofs were well structured. I did not find any issues.", "experimental_designs_or_analyses": "The experiment results are thorough and showcase the effectiveness of the work. I do have a question regarding the statement in the introduction on the given examples not being solvable by the current state of the art. The experiments are interesting, but I do not really understand how these are not solvable by current methods, given that they seem fairly standard control problems.", "relation_to_broader_scientific_literature": "The authors do a good job at relating the work to existing safe control literature. I do not have any comments on this.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strengths:\n- The paper has solid theoretical results, does not overstate the claims and showcases the method effectively. \n- The idea of leveraging bandit bounds for model uncertainty decision making is interesting.\n\nWeaknesses:\n- Some (critical) statements are made and quickly dismissed, see below.\n- The method still depends on critical knowledge or careful heuristics when picking the kernels, and requires having valid control barrier functions.", "other_comments_or_suggestions": "See above and below. Overall, I find this a solid paper.", "questions_for_authors": "- The main question I have is regarding the assumption that we do not know the system dynamics and we need to learn them, and yet we do know a control barrier function for the system. Is this realistic for most applications?\n- The choice of the kernel (although it is discussed) is not obvious. Is there any disadvantage to choosing \"universal kernels\"? And what are their limitations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WbEDPzZrwk", "reviewer": "Reviewer_kuHs", "summary": "The paper proposed an online safe control algorithm with Gaussian process models of the dynamics and bandit-type exploration to learn the dynamics. Then, the learned dynamics are combined with a control barrier function to ensure online safety. The control signal is solved with a safety filter with a lower confidential bound for robustness. The data collection requires a high frequency to prevent safety violations and infeasibility of the safety filter optimization.\n\n## Update after rebuttal\n\nMy major concern is the connection to previous thoeritical papers about safe exploration, which is resovled by the rebuttal.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you kindly for reviewing our paper. Please find our answers to your questions and comments below.\n\n**Comparison with other references and baselines:** Thank you for suggesting the additional references. Although the methods of Sui et al. (2015), Wachi et al. (2018) and Prajapat et al. (2022) employ similar tools (the most notable being a combination of GP-based LCBs and UCBs for safety and optimistic exploration), there exist some fundamental differences between their methods and ours. Arguably the biggest difference on a conceptual level is that exploration in their approaches is driven by return maximization. In contrast, exploration in our approach is driven by the need to ensure safety. Moreover, their approaches do not allow the (pessimistically) unsafe states to be visited, whereas ours does, using exploration to reduce uncertainty while in those states. Other differences are as follows:\n- Their methods require a tabular MDP setting, which scales poorly with the number of state space dimensions and actions. \n- They are designed for a discrete-time setting, whereas our approach considers a continuous-time setting.\n- Their methods require the transition dynamics to be known perfectly, whereas our approach does not.\n- A further difference tied to the previous point is how optimistic exploration is used. The methods of Sui et al. (2015) and Wachi et al. (2018) utilize exploration to expand the safe set and maximize the returns under optimism. However, the safe set is expanded only by visiting safe states, i.e., by incrementally moving toward the unsafe states. Note that this is typically only achievable with perfectly known dynamics. In contrast, our approach expands the safe set by directly exploring the states that are unsafe under pessimism.\n\nAfter carefully analyzing the method and code of Wachi et al. (2018), we have concluded that to implement their method on one of our baselines, we would require significant changes to their method: \n- First, we would need to discretize the dynamical systems. While this is possible for the cruise control example, it is practically infeasible for the quadrotor. This is because the state space is 10-dimensional, and the nonlinear dynamics are strongly state-dependent, meaning that a dense graph is required. However, this implies a graph with 10^20 nodes for just 20 support points per dimension.\n\n- Secondly, we would need to address the fact that the dynamics are unknown in our setting. To this end, we could either use the GP mean to estimate the state-transition dynamics or a pessimistic estimate of the transition dynamics under model uncertainty and the pessimistic safety constraint. Moreover, the state transition graph would have to be updated after every time step, which was not intended under the original algorithm.\n\nOverall, we feel that directly implementing their code on our setting would entail design decisions that would significantly alter the original method, making a fair comparison difficult. However, we will thoroughly discuss these papers in the revised manuscript. Moreover, we will include a comparison with the heuristic proposed by Choi et al. (2023), which corresponds to an equation that attempts to maximize the LCB whenever infeasibility occurs.\n\n**Regarding computation time:** Our method mainly requires computation time for GP predictions and to solve the optimization problems involving the LCB (exploration) and UCB (safe control) of the CBF. Please note that, due to the nature of the CBF optimization problem, we only require a single GP pass to formulate the required vectors and matrices, after which we only have to solve a convex optimization problem. Hence, the GP computation scales with the data, whereas the CBF optimization does not. However, sparse inference tools can be easily applied to improve scalability. Below, we report the average computation time for each operation as a function of the data using vanilla GPs (we use the GPyTorch toolbox without GPU acceleration):\n\n| | GP  | Optim. |\n| ---- | ------ | ---- |\n| n=0 | 0.0399 | 0.0346 | \n| n=10| 0.0407 |  0.233 | \n| n=50| 0.0410 |  0.0174|\n| n=200| 0.0448 | 0.0194 |\n| n=500| 0.0654 | 0.0182 | \n\nThe GP column corresponds to the average time of GP computations per iteration, which scales with the data, as expected. The Optim column corresponds to the average time taken by the optimization problem, which stays approximately constant with the amount of data.", "claims_and_evidence": "I examined all the claims in the paper, and all of them are supported by clean and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method makes sense. The assumptions might be too strong, but it is a general problem of control barrier function-based methods, not the problem of authors.\n\nThe evaluation criteria might miss some baselines; now, it only includes random exploration. I suggest the authors to include [1] and its following work like [2] to compare the performance.\n\n[1] Sui, Yanan, et al. \"Safe exploration for optimization with Gaussian processes.\" International conference on machine learning. PMLR, 2015.\n\n[2] Wachi, A., Sui, Y., Yue, Y., & Ono, M. (2018). Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).", "theoretical_claims": "I briefly checked the proof of Theorem 3.2, it looks good and corret to me. I didn't check the correctness line by line.", "experimental_designs_or_analyses": "Like I said in Methods And Evaluation Criteria, The evaluation criteria might miss some baselines; now, it only includes random exploration. I suggest the authors to include [1] and its following work like [2] to compare the performance.\n\n[1] Sui, Yanan, et al. \"Safe exploration for optimization with Gaussian processes.\" International conference on machine learning. PMLR, 2015.\n\n[2] Wachi, A., Sui, Y., Yue, Y., & Ono, M. (2018). Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).", "relation_to_broader_scientific_literature": "The results are novel in the sense of combining control barrier functions, but similar ideas have been explored in other safe exploration literature like \n\n[1] Sui, Yanan, et al. \"Safe exploration for optimization with Gaussian processes.\" International conference on machine learning. PMLR, 2015.\n\n[2] Wachi, A., Sui, Y., Yue, Y., & Ono, M. (2018). Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).\n\n[3] Prajapat, Manish, et al. \"Near-optimal multi-agent learning for safe coverage control.\" Advances in Neural Information Processing Systems 35 (2022): 14998-15012.", "essential_references_not_discussed": "The results are bandit-like exploration algorithm to ensure safe control with control barrier functions, but similar ideas have been explored in other safe exploration (without using control barrier functinos) literature like", "other_strengths_and_weaknesses": "# Strength \n1. The authors explain a lot about the solution existence, problem difficulties, and limitations. I appreciate them very much.\n2. The experimental results show the effectiveness of pthe roposed algorithm.\n\n# Weakness\n1. The required sampling time might not be realistic in the real world.\n2. As the data accumulates, the dimension of the safety filter optimization problem will increase.", "other_comments_or_suggestions": "N/A. Good paper.", "questions_for_authors": "1. Can you report the computation time in the experimental section?\n2. Can you explain the difference between this paper and previous safe exploration with GP and bandit style in the following references?\n\nIf the differences are properly justified, I will further improve my score. Overall, it is a good paper.\n\n[1] Sui, Yanan, et al. \"Safe exploration for optimization with Gaussian processes.\" International conference on machine learning. PMLR, 2015.\n\n[2] Wachi, A., Sui, Y., Yue, Y., & Ono, M. (2018). Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).\n\n[3] Prajapat, Manish, et al. \"Near-optimal multi-agent learning for safe coverage control.\" Advances in Neural Information Processing Systems 35 (2022): 14998-15012.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XlKl6ichtP", "reviewer": "Reviewer_23em", "summary": "## update after rebuttal\nMy evaluation of the paper has not changed after the rebuttal. The technical part of the paper is correct but does not bring much new insight on the topic. The assumption of known CBF for a system is stronger than existence of partial models and backup controllers. The experiments do not show particular benefits of using CBF, compared to reasonable designs of other exploration schemes. \n\nThe paper described a method for using control barrier functions to guide exploration under model uncertainty. Theoretical analysis on sample complexity is given in the bandit exploration setting. Experiments are shown on quadrotor and cruise control which have dynamics that are well-understood.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you kindly for your review. We have addressed your comments and questions below.\n\n\n\n**Knowledge of CBF and difference to nominal understanding of the system dynamics/backup controller:** Although the CBF assumption cannot be ensured in a general setting, it still offers flexibility, and there are ways to relax it in practice. Typically, only a portion of the safe set is visited during control; hence, the CBF function only has to satisfy the corresponding conditions for the corresponding subset of the state space. Furthermore, we can achieve conservatism by initially restricting the safe set to a region that is easy to control and then iteratively expanding it as more data is collected. This potentially allows us to gradually improve the CBF as more system knowledge allows us to expand the safe set. Alternatively, we can include conservatism in the safety requirement by computing the CBF condition from a collection of (potentially valid) CBFs and taking the worst case. \n\nAdditionally, we note that the setting where a CBF is known but the dynamics are unknown is an open ongoing research question that has garnered increased attention recently:\n- End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks (Cheng et al., 2019)\n- Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions (Choi et al., 2020)\n- Learning for safety-critical control with control barrier functions (Taylor et al., 2020)\n- Constraint-Guided Online Data Selection for Scalable Data-Driven Safety Filters in Uncertain Robotic Systems (Choi et al., 2023)\n- Safe Barrier-Constrained Control of Uncertain Systems via Event-triggered Learning (Lederer et al., 2024)\n- Learning-Based Prescribed-Time Safety for Control of Unknown Systems With Control Barrier Functions (Huang et al., 2024)\n\nNote that these works require additional assumptions beyond a known CBF to achieve safety.\n\nFurthermore, we believe our method still has value even in settings where the CBF is not strictly valid. One of our paper's main contributions is that we use exploration to recover safety if all other safety certificates fail. This strongly contrasts with other approaches, which either assume enough controllability to stay within the safe set at all times or require a safe backup controller. Hence, it provides a principled way to design an exploration-based controller that aims to achieve safety whenever other certificates become invalid. We will include this discussion in the revised paper.\n\n\n\n**Comparison with other baselines:** We are currently implementing a comparison with the heuristic proposed by (Choi et al., 2023), which computes control inputs by maximizing the LCB whenever infeasibility occurs instead of exploring the state space. We will also include a comparison with the baseline with a fully known model. We will report the results during the second round of reviews.\n\n**Designing a CBF with a zero model prior:** In general, it is impossible to design a CBF without any prior model knowledge. However, it is still less challenging than deriving an accurate system dynamics model, and this requirement can frequently be relaxed, as discussed above. Moreover, note that allowing a prior model of zero is not strictly an assumption of our method but rather something that it allows for, e.g., if we want to learn an unbiased model purely from data. Our paper highlights this because it contrasts with other methods, which require more assumptions/backup controllers even in settings where a prior model is available.\n\n**CBF design in experiments:** The CBF was taken from the cruise control setting from Ames et al. (2021). It restricts the velocity of the vehicle it comes closer to the vehicle in front. For the quadrotor, the first CBF (for avoiding ground collision) is similar to the cruise control setting, whereas the second CBF (for avoiding overrotation) corresponds to a quadratic equation that constrains the velocities and the orientation of the quadrotor.\n\n**What happens if the collected data show that the learned dynamics violates the CBF that is assumed given?** The CBF is not necessarily learned from the model. Hence, it can violate the learned dynamics. In fact, it can violate the dynamics under pessimism (this corresponds to the CBF LCB), which is when our method explores.\n\n\n\n\nChoi, J. J., Castaneda, F., Jung, W., Zhang, B., Tomlin, C. J., & Sreenath, K. (2023). Constraint-guided online data selection for scalable data-driven safety filters in uncertain robotic systems. arXiv preprint arXiv:2311.13824.\n\nAmes, A. D., Grizzle, J. W., & Tabuada, P. (2014, December). Control barrier function based quadratic programs with application to adaptive cruise control. In 53rd IEEE conference on decision and control (pp. 6271-6278). IEEE.", "claims_and_evidence": "The claims on safety-guided bandit exploration are supported, modulo the limitation of experiments only on system that have well-understood dynamics with high controllability and limited uncertainty.", "methods_and_evaluation_criteria": "The main problem is although the paper claims to provably achieves safety \"in a general setting that does not require any prior\nmodel or backup controller\" there is the requirement of a fully known control barrier function. This assumption is very much questionable, because the difficulty of coming up with CBFs are well-known even under fully specified dynamics models. The proposed framework can only be useful when the model uncertainty can be dominated by the known CBF and control authority, which is a setting that hardly requires the whole CBF mechanism in the first place. For instance, this assumption is equivalent to requiring a backup controller that's very easy to formulate. It is thus misleading to claim that the methods achieve safety in a general setting.", "theoretical_claims": "Under the assumptions the theoretical claims on sample complexity is reasonable.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "The assumptions about the systems make the need of CBF questionable. By only comparing with CBF-based methods, the paper is not well-positioned in the literature.", "essential_references_not_discussed": "It should compare with methods that do use prior knowledge about the dynamics and backup controllers, because the need for a predefined CBF is implicitly imposing even stronger assumptions.", "other_strengths_and_weaknesses": "none", "other_comments_or_suggestions": "none", "questions_for_authors": "The main part of the paper says it is ok to assume prior model of the system $\\hat f$ and $\\hat g$ to be zero. How do you come up with the CBF as input to the system in that case? What happens if the collected data show that the learned dynamics violates the CBF that is assumed given? In the setting of the experiments, how are the assumption different from having a nominal understanding of the system dynamics and backup controller?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qBxHxxL5g8", "reviewer": "Reviewer_b8R8", "summary": "This paper proposes a safe control method to address the problem of infeasible safety filter. The authors use Gaussian processes to learn the system dynamics. They use the lower confidence bound of the control barrier function (CBF) to identify the feasibility of the safety filter, and sample exploratory controls according to the upper confidence bound of CBF to reduce the model uncertainty when it is infeasible. They provide the guarantee of feasible safety filter under finite samples of online exploration, and evaluate the proposed safe control methods on numerical simulation of cruise control and quadrotor control.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you kindly for your review. We have corrected all typos and addressed all your major comments below. We want to stress at this point that our method does, in fact, guarantee safety, even throughout exploration.\n\n**Safety guarantee during exploration:** Our method does guarantee safety during exploration. This is, in fact, the whole point and one of the main contributions of our method. Our method explores in a way that recovers the feasibility of the CBF safety filter **before** the safe set is exited. In other words, the closed-loop system is always in the safe set during exploration. This is illustrated by Figure 1 and shown theoretically in Theorem 3.2. The key idea in the proof is to show that if we collect data with our exploration strategy quickly enough, we *must* recover feasibility before exiting the safe set. This is achieved by upper-bounding the information that can be gained inside the safe set, then showing that each collected data point contributes sufficiently toward reducing the information within the safe set (Lemma B.9). This is a crucial aspect of our method, and we will make this clearer in the revised manuscript by explicitly stating it in the abstract and introduction.\n\n**Specifications for experiments:** In the experiments, we set $\\beta=10$, $\\epsilon=0$ and $h$ equal to the identity function. The high choice of $\\beta$ aims to avoid underestimating the model error. Note that $\\beta=2$ is otherwise frequently chosen in the literature, e.g., in Berkenkamp et al. (2017) and Wachi et al. (2018). The choice of $\\epsilon=0$ corresponds to assuming that the CBF constraint can be enforced robustly, whereas $h$ equal to the identity function is for simplicity.\n\nBerkenkamp, F., Turchetta, M., Schoellig, A., & Krause, A. (2017). Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30. \n\nWachi, A., Sui, Y., Yue, Y., & Ono, M. (2018). Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).\n\n**Sampling time:** Although the sampling time required to achieve safety consistently in the examples is high, note that they correspond to a setting with no prior knowledge. Hence, it is reasonable to expect a high sampling time, as otherwise, sufficient information cannot be collected before exiting the safe set. \n\n**GoSafeOpt paper:** Thank you for pointing out this paper to us. The paper does indeed employ similar tools and has a similar motivation. Some important differences include:\n- On a high level, the goal of GoSafeOpt is to expand a set that is already safe. Although our method can also be used for the same end, its main goal is to render the system safe under high levels of uncertainty. This is particularly useful for settings where other safety certificates can potentially fail.\n- A further difference that is related to the one above is that GoSafeOpt requires a set of initially safe control parameters at the beginning of exploration to guarantee safety. In contrast, our approach can guarantee safety purely through exploration, without requiring an initially safe set of control parameters. \n- In GoSafeOpt, the goal is to tune a parametric controller, and the GP-based safety conditions depend on the controller's parametric structure. In our setting, the GP model is independent of any underlying control structure.\nWe will include this comparison in the revised paper.\n\n**Less safety violations at 10Hz:** We agree that this is somewhat unexpected since a higher sampling rate should yield a better model faster, which translates to safer control. However, it is important to remember that at lower sampling frequencies, the system can quickly reach the boundary of the safe set before more than a handful of data has been collected. In these settings, the exploratory input is applied for a considerable amount of time before potentially reverting to the safety filter-based control, making its role non-neglegible. Hence, we speculate that the exploratory control is particularly beneficial for safety in this particular instance.", "claims_and_evidence": "The authors claim that the proposed safe control method is safe with high probability, and shows evidences via\n\n1. theoretically proving the feasibility of safety filter within finite exploratory samples.\n\n2. experimentally demonstrating the numerical results over the safe control of two different dynamical systems.\n\nHowever, I think these evidences may not fully support the claim:\n\n1. The author did not provide safety guarantee during the exploratory sampling procedure.\n\n2. The default sampling frequency in the experiment of is 1e5 hz, which may not be practical in real-world experiments.", "methods_and_evaluation_criteria": "Overall I think the proposed method makes sense and benchmark used for evaluation are common-used embodiments in safe control.", "theoretical_claims": "As mentioned above, the absence of safety analysis during the exploration phase hinders the guarantee of high-probability safety. I think further analysis about the unsafe probability of using UCB to sample should be conducted.", "experimental_designs_or_analyses": "During the experiment, the authors track the value and LCB of the CBF function, and demonstrate the effectiveness of the proposed optimistic sampling method. They also show the control trajectories of the quadrotor.\nI have following concerns regarding to the experiment part:\n\n1. As mentioned above, the 1e5 sampling frequency may be too high for real-world deployment, Under low sample frequency the proposed method cannot guarantee high safe probability, as shown in Figure 5.\n\n2. Some implementation details are missing (e.g. the setting of $\\beta$，$\\alpha$, and $\\epsilon$).", "relation_to_broader_scientific_literature": "Existing safe controls requires that the safety filter is always feasible, or there is a backup policy to ensure safety during exploration. The authors propose to use UCB of the CBF to sample optimistically safe controls to learn the CBF to address the issue of infeasibility safety filter or unavailable backup policy. However, I think the provided theoretical and empirical results cannot fully support this claim.", "essential_references_not_discussed": "The main idea of this paper is to introduce an optimistic exploratory phase when pessimistic safety estimation is infeasible. I think an important work [1] is missing, which also introduces a UCB-based exploration strategy and online learns a backup policy to guarantee high-probability safety. I think the authors should include this paper in discussion.\n\n[1] Sukhija, Bhavya, et al. \"Gosafeopt: Scalable safe exploration for global optimization of dynamical systems.\" Artificial Intelligence 320 (2023): 103922.", "other_strengths_and_weaknesses": "**Strength**\n\n1. The addressed safe control problem without feasible safe filter is important.\n\n**Weakness**\n\n1. Theoretical and experimental concerns have been mentioned above.\n\n2. The proposed method uses GP to jointly predict over the concatenation of states and controls, which may be difficult to scale to higher-dimensional systems.", "other_comments_or_suggestions": "1. Some typos: line 412: duplicated number. line 414: should be Figure 3?", "questions_for_authors": "1. In Figure 5, seems the rate of failure under 10 hz is better than 100 hz and 1000 hz. Can you explain why this happens?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper is squarely borderline but the AC argues for (weak) acceptance after weighing the reviews for strengths and weaknesses. Firstly, safety of ML-based control --particularly during exploration and data collection -- is an active area of research and bringing together the CBF methodology along with GPs to characterize system uncertainty is a coherent approach to attempt to advance further. The primary criticisms against the paper are that  (1)"the assumption of known CBF for a system is stronger than existence of partial models and backup controllers" and (2) the experiments are limited, shown for Quadrotor/Cruise systems with well understood dynamics and unclear implications for high-dimensional systems. 

 That said, the proposed method explores in a way that recovers the feasibility of the CBF safety filter before the safe set is exited --  by upper-bounding the information that can be gained inside the safe set, then showing that each collected data point contributes sufficiently toward reducing the information within the safe set. This kind of argument is non-trivial -- requiring connecting dots between CBF synthesis, sample complexity and choice of function families to capture uncertain dynamics; hence the could be useful for future iterations of controllers that admit safe exploration.
Decision: Accept (poster)