On Safety in Safe Bayesian Optimization
Authors: Christian Fiedler
Johanna Menn
Lukas Kreisköther
Sebastian Trimpe
Venue: Accepted by TMLR ()

Abstract: Safe Bayesian Optimization (BO) is increasingly used to optimize an unknown function under safety constraints, a central task in robotics, biomedical engineering, and many other disciplines. Due to the safety-critical nature of these applications, it is crucial that theoretical safety guarantees for these algorithms translate into the real world. In this work, we investigate three safety-related issues in SafeOpt-type algorithms, a popular class of safe BO methods. First, these algorithms critically rely on frequentist uncertainty bounds for Gaussian Process (GP) regression, but concrete implementations typically utilize heuristics that invalidate all safety guarantees.  We provide a detailed analysis of this problem and introduce Real-$\beta$-SafeOpt, a variant of the SafeOpt algorithm that leverages recent GP bounds and thus retains all theoretical guarantees. Second, we identify a key technical assumption in SafeOpt-like algorithms, the availability of an upper bound on the reproducing kernel Hilbert space (RKHS) norm of the target function, as a central obstacle to real-world usage.
To address this issue, we propose to rely instead on a known Lipschitz and noise bound, and we introduce Lipschitz-only Safe Bayesian Optimization (LoSBO), a SafeOpt-type algorithm using the latter two assumptions. We show empirically that this algorithm is not only safe, but also outperforms the state-of-the-art on several function classes. Third, SafeOpt and derived algorithms rely on a %gridding of the search space, discrete search space, complicating their application to higher-dimensional problems. To broaden the applicability of these algorithms, we introduce Lipschitz-only Safe GP-UCB (LoS-GP-UCB), a LoSBO variant that is applicable to moderately high-dimensional problems, while retaining safety. By analyzing practical safety issues in an important class of safe BO algorithms, and providing ready-to-use algorithms that overcome these issues, this work contributes to bringing safe and reliable machine learning techniques closer to real world applications.

Published in Transactions on Machine Learning Research (09/2024)
On Safety in Safe Bayesian Optimization
Christian Fiedler'
Institute for Data Science in Mechanical Engineering (DSME) RWTH Aochen University
Johanna Menn
Institute for Data Science in Mechanical Engineering (DSME)
RWTH Aochen University
Lukas Kreisköther
Sebastian Trimpe
Institute for Data Science in Mechanical Engineering (DSME)
RWTH Aachen University
(* Equal contribution)
Reviewed OpenReview: http:p//ppppoorr.....aoo/ommrrap
Abstract
Safe Bayesian Optimization (BO) increasingly used to optimize unknown function un- der safety constraints, central task in robotics, biomedical enginoering, and many other disciplines. Due to the safety-critical nature of these applications, it is crucial that theo- retical safety guarantees for these algorithms translate into the real world. In this work, we investigate three safety-related issues in SafeOpt-type algorithms, popular class of safe BO methods. First, these algorithms critically rely on frequentist uncertainty bounds for Gaussian Process (GP) regression, but concrete implereentations typically utilize heuris- tics that invalidate all safety guarantees. We provide detailed analysis of this problem and introduce Real-3-SafeOpt,, variant of the SafcOpt algorithm that leverages recent GP bounds and thus retains all theoretical guaranlees. Second, we identily a key technical as- sumption in SafeOpt-like algorithms, the availability of an upper bound on the reproducing kernel Hilbert space (RKHS) norm of the target function, as a central obstacle to real-world usage. To address this issue, we propose to rely instead on a known Lipschitz and noise bound, and we introduce Tipschitz-only Safe Bayesian Optimization (LoSBO), a SafeOpt- type algorithm using the latter two assumptions. We show empirically that this algorithm is not only safe, but also outperforms the state-of-the-ars SafeOpt and derived algorithms rely discrete search space, complicating their applica- tion to higher-dimennionaa problems. To broaden the applicability of these algorithms, we introduce Lipschitz-only Safe GP-UCB (LoS-GP-UCB), LoSBO variant that is applicable to moderately high-dimensional problems, while retaining safety. By analyzing practical safety issues an important class of safe BO algorithms, and providing ready-to-use al- gorithms that overcome these issues, this work contributes bringing safe and reliable machine learning techniques closer real world applications.
several function classes. Third,
1 Introduction
In science, engincoring, and business often necessary to optimize an unknown target function. Typically, such functions are expensive evaluate and only noisy function values are available. If it is possible actively query the function, i.e., to select the inputs that are to be evaluated, this problem is commonly

---

Published in Transactions on Machine Learning Research (09/2024)
addressed using Bayesian Optimization (BO), see (Shahriari et al., 2015) or (Garnett, 2023) for an overview. However, in many real-world applications, BO algorithms should avoid the use of certain inputs, frequently for safety reasons. For example, the target function might be a reward function for a robotic task, with the imput a control policy for a physical robot. In this example, any imputs leading to physical damage or unsafe behavior should be avoided. An important special case of such a safety constraint is the restriction of query inputs to those with function values not lower than given threshold (Kim et al., 2020). Inputs that satisfy this constraint are termed safe inputs, and BO algorithm is considered safe if it queries only safe imputs throughout its run. This type of safety constraint has been introduced in (Sui et al., 2015) and arises for example in biomedical applications (where the target function is patient comfort and the imput corresponds to treatment settings) or controller tuning (where the target function is a measure of controller performance and the inputs are luning parameters). A popular BO algorithm for this problem setting is SafeOpt (Sui et al., 2015). Starting from a given set of safe inputs, this algorithm iteratively searches for a maximum while aiming to avoid unsafe inputs with high probability. It achieves this by utilizing Gaussian Processes (GPs) together with a frequentist uncertainty bound (Srinivas et al., 2010; Chowdhury & Gopalan, 2017) and known upper bound on the Lipschitz constant of the target function. Provided the algorithmic parameters are set correctly, then with (high) pre-defined probability, SafeOpt demonstrably converges to the safely reachable maximum while avoiding unsafe inputs. SpfeOpt and its variants have been used in various applications, e.g., safe controller optimization (Berkenkamp et al., 2016), automated deep brain stimulation (Sarikhani et al., 2021) and safe robot learning (Baumann et al., 2021). To ensure safety, SafeOpt and its variants require frequentist uncertainty sets that are valid (i.e., they hold with specified high probability) and explicit (i.e., they can be numerically evaluated). However, two issues arise here: First, the uncertainty bounds from (Srinívas et al., 2010; Chowdhury & Gopalan, 2017) used in most SafeOpt-type algorithms tend to be conservative, even if all necessary ingredients for their computation are available. This restriction can entirely prevent exploration of the target function. Second, such uncertainty bounds rely upon a particular property of the target function (a known finite upper bound on the Reproducing Kernel Hilbert Space (RKHS) norm), which in practice is very difficult to derive from reasonable prior knowledge. As a consequence of these issues, algorithmicet Indeed, to the best of our knowledge, all implementations of SafeOpt and its variants have instead used hcuristics (Berkenkamp et al., 2016; Kirschner et al., 2019a; Baumann et al., 2021; Koller al., 2019; Helwa et al., 2019).' This shortcoming means that in practice such implementations lose all their theoretical safety guarantees. In this work, we carefully investigate this issue and propose practical solutions, bringing safe BO closer to real-world usage.
usable uncertainly sets are not yet available for SafeOpt..
Outline In Scction 2, we recall some background matcrial. The problem sctting and our objectives are presented in Section 3, and Section 4 provides comprehensive discussion of related work. In Section 5, we investigate safety issues of SafeOpt-type algorithms arising from using heuristics instead of valid frequentist uncertainty bounds. In Section 6, we discuss the practical safety problems arising from the assumption of a known upper bound on the RKHS norm of the target function, and propose to rely instead on a known Lipschitz and noise bound to address this issue, with Tipschitz-only Safe Bayesian Optimization (LoSBO) as an appropriate algorithm for this. To enable this type of safe BO beyond low dimensional input spaces, Section introduces another algorithmic variant, called Lipschitz-only Safe Gaussian Process- Upper Confidence Bound (LOS-GP-UCB), which is suitable for moderately high dimensions. Finally, Section 8 closes the article with summary and outlook. Additional background, discussion and experimental details can be found in the appendix, and the source code for all experiments is available in GitHub repository."
2 Background
In this section, we provide brief review of Gaussian Process (GP) regression, reproducing kernel Hilbert spaces (RKHSs), and frequentist. uncertainty bounds for GP regression, as these three components form the foundations for SafeOpt-type algorithms.
The precise experimental scttings are not reported in (Sui al., 2015). However, based the descriptions provided, it can be inferred that some form of heuristic was used.
2 Phttps://github. wwwesittuu..oom//eee com//Daaa-ciinneee in- Nechanical wettaaanneaannngnnnnnnngggg/ee Engineering/Loso
2

---

Published in Transactions on Machine Learning Research (09/2024)
2.1 Gaussian processes and reproducing kernel Hilbert spaces
A GP is collection of R-valued random variables, here indexed by the set D, such that every finite collection of those random variables has a multivariate normal distribution. A GP is uniquely defined by its mean function m(x)= = [z] and covariance function k(x, x") = [[((()) m(z))(((((a)) - mm('))],, and we denote such a GP by g ~ GP (m,,m,)) In GP regression, we start with a prior GP. Assuming independent and identically distributed (i.i.d.) additive normal noise, 6t ~ NN o²), this prior GP can be updated with data (21,31),... v1),..., (3ee, Mt), leading to a posterior GP. Without loss of generality we assumne that the prior GP has zero mean. Then the posterior mean, posterior covariance and posterior variance are given by pe(z)= ke(x)² (K, - 1,) ye: x1,)='))= = - (KK) ( + ==),))))k(((")) and df(x) = (x, x)), respectively, where Vr = [yi...., yrl" is the vector of observed, noisy function values of f. the kernel matrix K; E Bext has entries [k<2,, x'),.'' the vector (x(r)= - [k(=1, =)...( a²}² contains the covariances between I and the observed data points, and I is the txt identity matrix. In practice, the prior mean, prior covariance function, and noise level, are (partially) chosen based on prior knowledge. Tf no specific prior knowledge regarding the mean is available, the zero function usually chosen. Furthermore, in practice these three components are only partially specified, usually up to some parameters, which are then called hyperparameters in the context. In BO, the latter are often determined during the optimization via hyperparameter optimization (Garnett, 2023).
If k is the covariance function of a GP, then it is symmetric and positive semidefinite. Conversely, if k is the reproducing kernel of a RKHS, then k is symmetric and positive semidefinite, and there exists a GP having as its covariance function, and the GP can be chosen with a zero mcan function (Berlinct & Thomas-Agnan, 2004). Finally, for every symmetric and positive semidefinite function k (and hence for every covariance function of GP) there exists a unique associated Hilbert space of functions, called a reproducing kernel Hilbert space (RKHS) and denoted by (H&, (s), with (i) k(,,z)) E Ha for all x E D, and (ii) f(x) = ((, x))x for all f E H2 and x E D. For the reader's convenience, Section A.1 contains a brief introduction and discussion of these function spaces.
For more details on GPs, GP regression, and related method, we refer to (Rasmussen & Williams, 2006) and (Garnett, 2023), and for more background on RKHSs, we refer to (Steinwart & Christmann, 2008, Chapter 4).
2.2 Frequentist uncertainty bounds
An important ingredient in SafeOpt-type algorithms are upper and lower bounds on the unknown target function, and these bounds have to hold uniformly in both time and input space. If we adopt stochastic setup, then this can be formalized by finding upper and lower bounds such that for given user-specified confidence 8 E (0, 1), we have TP [f((x)) S fxx S We (1), Var E D, 2 0) 2 1 1-& 5, where the probability is with respect to the data-generating process, while f is assumed fixed and nonrandom. In GP regression, the posterior mean H4 can be interpreted as a nominal estimate of f, and the posterior variance of as a measure of uncertainty of this estimate. However, using the posterior variance to build upper and lower bounds in the SafeOpt sctting is not straightforward. Firstly, the posterior variance is a pointwise mcasure of uncertainty about the ground truth, but the upper and lower bounds have to hold uniformly over the input set, and also uniformly in time. Secondly, GP regression is by its nature Bayesian method. However, the SafeOpt setting is typical frequentist sotup - we have a fixed, yet unknown ground truth, about which we receive noisy information. In particular, any stochasticity arises only through the data-generating process (e.g., via random noise on the function values), and not through epistemic uncertainty, as is the case in the Bayesian setup. This difficulty is well-known, see (Fiedler et al., 2021a), and is particularly relevant in the context of robust control and related arcas (Fiedler et al., 2021b). We thus need bounds (ve)121, such that for a user-specified 8 E (0, 1) we have
(1) The bounding function v, must only depend on data collected up to time t, as well as reasonable prior knowledge about f and the data-generating process egg., about the noise statistics. These bounds are
P (lf(z)) - S (z) E 2 1-5
3This means that all matrices (k(#) 3y (33)))))))) J for AN and N E Nso are symmetric positive semidefinite.
3

---

Published in Transactions on Machine Learning Research (09/2024)
(a)
(b)
2
posterior mean uncertainty set true function data points
2
0
-2
4
0
4
4
-
0
Figure Illustration of the required GP error bounds. Consider a fixed ground truth (solid black line), of which only finitely many samples are known (gray dots). Applying GP regression leads to posterior GP fully described by the posterior mean (solid blue line) and the posterior variance, from which high- probability uncertainty set can be derived (shaded blue). Left: The ground truth is completely contained in
the uncertainty set. Right: The ground truth violates the uncertainty bound around x = 1.
illustrated in Figure All common bounds are of the form
(x(x) (,0,(2),
(2)
where 3r E Ryo is some scaling factor. Let & be the covariance function used in GP regression, and Hş the unique RKHS with & as its reproducing kernel. Assume that the ground truth is contained in this RKHS, i.e., f Hy. Let F (Fi)eza be filtration defined on the underlying probability space, and assume that the sequence of inputs (24))20 chosen by the algorithm is adapted to F.1 The first bound of the form (2) was introduced in the seminal work (Srinivas et al., 2010), cf. their Theorem which holds in the case of bounded noise. This is also the bound that was used in the original SafcOpt paper (Sui et al., 2015). At the moment, the most commonly used uncertainly bound in the analysis of SafeOpt-type algorithms is (Chowdhury & Gopalan, 2017, Theorem 2). Assume that (4), a martingale difference sequence that is conditionally R-subgaussian w.r.t. for some R E R20: which holds for example for i.id. bounded or normal noise. Additionally, assume that À > 1, or > and the covariance function positive definite, where À is the nominal noise variance used in GP regression. Under these conditions, the uncertainty bound (1) holds with
3. = 11 222((111+1+1) + log(1/5))
in (2), where Y is the mazimum information gain after rounds, (Srinivas al., 2010) for thorough discussion of this quantity. In contrast to (Srinivas et al., 2010, Theorem 6), this bound allows subgaus- sian noise (including bounded noise and normal noise) and involves only fairly small numerical constants. However, it still requires the maximum information gain to work with in practice, and it introduces some conservatism. Motivated by these shortcomings, Fiedler et al. (2021a) proposed a data-dependent scaling factor in (2), based on (Chowdhury & Gopalan, 2017, Theorem 2). Assume the same sctting as this latter result, and also assume that the covariance function k is positive definite, then we can set
upper bound thereof, which can be difficult
R VI Indet(///AK, + (L)) - In(),,
8. = =1ff+++
(4)
where we defined À max(1,A), and À again the nominal noise variance used in GP regression, cor- responding to the regularization parameter in kernel ridge regression. This bound no longer involves the maximum information gain, and numerical experiments demonstrate that the resulting uncertainty bounds are rarely significantly larger than common heuristics, (Fiedler et al., 2021a). In fact, the bounds are
40f course, this requires that measurable space, which a problem practice.

---

Published in Transactions on Machine Learning Research (09/2024)
sufficiently small to use in algorithms, e.g. (Fiedler et al., 2021b). Finally, from the results in the doctoral thesis (Abbasi-Yadken 2013), which was published in 2012, an uncertainty bound can be deduced that is superior to (Chowdhury & Gopalan, 2017, Theorem 2), and therefore also improves over (4). Consider the same setting as introduced above. Combining Theorem 3.11 with Remark 3.13 in (Abbasi- Yadakri,, 2013),
we find that for all 8 E (0, 1) we can set
R VX
1 ( det I [ k,
1
Be llf
2
(5)
in (2). Like (4), this bound can be casily evaluated, but it appears to have been used only infrequently in the machine learning community though it has recently been rediscovered, for example, in (Whitehouse et al., 2024).
3 Problem setting and objectives
We now formalize our problem setting following the seminal paper (Sui et al., 2015), describe SafeOpt-type algorithms in detail, and specify our objectives for the remainder of this work. Consider nonempty set D, the impul set, and a fixed, but unknown function f D + R, the target function or ground truth. We are interested in an algorithm that finds the maximum of f by iteratively querying the function. At time step E No, such an algorithm chooses an imput zt E D and receives a noisy function evaluation yt = f(x)) + where & additive mcasurement noisc. As a safety constraint, all chosen inputs must have a function value above given safety threshold h E R, i.e., f(aa)) 2 h for all I. Furthermore, the algorithm should be sample-efficient, i.., use few function queries possible to find input with a high function value. To make progress on this problem, it is clear that some restriction on the function f must be posed. Central to our developments is the next assumption.
Assumption 1. D is equipped with a metric d: D x D + R2o- Additionally, I is L-Lipschilz continuous, where L E R20 is a known Lipschitz constant.
The second assumption means that for all x. x, E D, we have [(x)) S dd(a,,')). From now on, we work under Assumption 1. Furthermore, we assume that we have access to non-empty set of known safe inputs So S D. i.., for all E So we have f(z) h. The SafeOpt algorithm and its derivatives use iteratively updated model Mr that provides estimated upper and lower bounds u, and & on f, ie., with a certain confidence it holds that (x(x) S fxx I (x(x) for all æ E D and all t 2 1. These bounds are also used to provide a measure of uncertainty defined as wx(x) uz(x) - (x) (xx). In each step L 2 1, the previous model Mi-1 together with the Lipschitz assumption is used to determine a new set S, C D of safe inputs, starting from the initial safe set Sg. Subsequently, a set M, S S, of potential maximizers of the target. function, and a set G, S S, of potential expanders computed. The latter contains inputs that are likely lead to new safe inputs upon query. Finally, the target. function is queried at the input de argmax w,(x), noisy function value yt is received, and the model M,-1 is updated with the data point (It, Mr). Different variants of SafcOpt result from different choices of models and computations of S,, M, and G.. To the best of our knowledge, in all SafeOpt-type algorithrss, the unknown ground truth ʃ is modeled as a GP. To compute appropriate upper and lower bounds, it is assumed that an appropriate scaling factor 3e is available, cf. (2). For each time step 6, define Cx(x) = (e--nQQtle
where Qx(x) = [ + 3, ot-1 and, starting with Qu(z)= R for all x E D, Co(z) = [h, oo) for all E So and Co(x) = for x E D Sg. The corresponding estimated upper and lower bounds are given by ua(x) = max (xr and l((x)) min (x),, respectively. Tn
the original SafcOpt algorithm from (Sui et al., 2015), for cach step t 1, the new safe sets are calculated by
S, =
U La' DD)(())) SES
h}
(6)
the potential maximizers are given by
M = (x E Se (((x) 2 max ((s)),
(7)
#gES,
S Note however that for À A and positive definite, (5) reduces to (4).
5

---

Published in Transactions on Machine Learning Research (09/2024)
Iteration 0
Iteration 5
Iteration 30
true function
uncertainty bounds
posterior mean data points optimum
initial safe set safe set maximizer set expander set
I
Figure Illustration of SafeOpt. The safe set (gray bar), expanders (blue bar), and maximizers (green bar), derived from the current GP model (with solid blue line the posterior mean, shaded blue areas the uncertainty sets), used to find the safely reachable optimum (red box). In each iteration, the next imput is chosen from the union of the current expanders and maximizers (a subset of the safe set) by maximizing the acquisition function.
A Algorithm SafeOpt
Require: Lipschitz constant L, algorithm to compute 3t, initial safe set So, safety threshold h
1: Qu(z) + R for all x E
b Initialization uncertainty sets
Cn(x) -- oo) for all So
Co(x) t for all x E 00)55 So
for t= 1....... do
Ci(x) C--((z) ( C for all x E D & Compute upper and lower bounds for current iteration
6:
4(x) - m (x)), (x) + max C((x) for all x DED
7:
if then
b. Compute new safe set
8:
S, = S U Hx, E S l((xs) - Ld(x,, x) 2 h}
else
10:
Si Su
11:
end if
12:
G₁ f- fx E St Ex E D) uc(x) Ld(æ, x¹) 2 h}
Compute set of potential expanders b Compute set of potential maximizers Determine next input
13:
M, = fx S, u(zx) 2 max (x(x))}
14:
+ argmax,eG,UM, We(x)
15:
Query function with 2t, receive = f(xx) + 6,
16:
Update GP with new data point (I,, 9t), resulting in mean μr and or
17:
Compute updated Be
18:
(x(x) (m((x) 3e or ((() 3. (x)) for all D
19:
end for
and the potential expanders by
G (xs E St Er E D St uxxs) Ld(xs, 2 h).
(8)
The resulting algorithm is illustrated in Figure 2. A formal description of the algorithm using pseudocode is provided by Algorithm 1. In some popular variants of the SafeOpt algorithm, no Lipschitz bound is used in the computation of the safe sets (Berkenkamp et al., 2016). However, since knowledge of such Lipschitz bound additional knowledge that should be used by the algorithm, and we strongly rely on this assumption from Section onwards, we do not consider these algorithmic variants in the present work.
Our primary objective is investigate and improve practically relevant safety aspects SafeOpt-type algo- rithms. We will consider three specific objectives, which will be explored in Sections and 7, respectively.

---

Published in Transactions on Machine Learning Research (09/2024)
(01) While SafeOpt and related algorithms come with precise theoretical safety guarantees, all known im- plementations to date use heuristics (usually by setting 33 to some constant) instead of theoretically sound uncertainty bounds. These heuristics generally invalidate all theoretical safety guarantees, raising the question of whether their use leads to safety problems in practice.
(02) Since SafeOpt and its variants are adopted for scenarios with stringent safety requirements where no failure should occur with high probability and no tuning phase is allowed, it is important that all underpinning assumptions are reasonable and verifiable by users. The most delicate of these assumptions is knowledge of an upper bound of the RKHS norm of the target function, raising the question of whether the RKHS norm bound assumption is reasonable, and if not, how it can be replaced.
(O3) In many relevant applications of safe BO, the input space is a continuous domain in moderately high dimensions. Since SafeOpt-type algorithms rely on discrete input space, they cannot be used in these settings. It is therefore necessary to devise alternatives that work in moderately high dimensions, while retaining safety guarantees.
4
Related work
In the following, related work will be reviewed. We first provide an overview of safe BO methods, particularly those closely related to the present setting. As Lipschitz bounds play a central role from Section 6 onwards, we also review previous work on methods that rely on such an assumption.
4.1 Safety in Bayesian optimization
We first consider safety in the context of Bayesian optimization (BO) (Shahriari et al., 2015; Garnett, 2023). The type of safety constraint considered in this work was first introduced in the seminal paper (Sui et al., 2015), which also proposed and analyzed the original SafeOpt algorithm. Several variations to the original algorithm have been proposed, with a SafeOpt-variant that does not use a Lipschitz constant (Berkenkamp et al., 2016) proving particularly popular in the robotics community. While the original SafeOpt algorithm from Sui al. (2015) interleaves safe exploration and function optimization over certified safe sets, a two- stage variant was introduced and analyzed in (Sui et al., 2018). Since SafeOpt relies space, and hence requircs discretization in the case of continuous optimization problem, applying this algorithm to continuous problems in even moderate dimensions can be very challenging, ef. also Section 7. This motivated the introduction of a variant based on swarmm-ptiimiztionn (Duivenvoorden et al., 2017), albeit with hcuristic safety guarantecs, as well as a variant tailored to dynamical problems in high dimensions (Sukhija et al., 2023). method based on random projections was also proposed and analyzed in (Kirschner et al., 2019a), and applied to tuning of a particle accelerator (Kirschner et al., 2019b). In terms of problem formulations, instead of just one safety constraint for the input function, multiple safety constraints can also be enforced, each encoded by unknown constraint function (Sui et al., 2018; Berkenkamp et al., 2023). In many applications of SafeOpt-type algorithms, properties of a dynamical system need to be optimized under safety constraints, for example, in controller tuning with BO. The dynamic aspect can be included in the optimization algorithm and its safety mechanisms, for example, in order to use a backup controller (Baumann et al., 2021). Furthermore, while formally SafeOpt-type algorithms are BO algorithms having input constraints, this is different from constrained BO as considered for example in (Hernández-Looatl et al., 2016). In the latter type of BO, the focus is on finding good inputs (i.e., corresponding to a high objective function value) that fulfill the (usually unknown) constraints, and violation of the constraints during the optimization process is not considered problematic (though of course it can be advantageous to avoid this). By contrast, safe BO aims to avoid constraint violations during the optimization process since these are considered problematic or even harmful. For an in-depth discussion of this difference, and further connections between the two problem settings, we refer to the excellent survey (Kim et al., 2020). SafeOpt-type algorithms are motivated by problems where safcty violations are considered to be very costly, so any safety violations should be avoided with very high probability. This contrasts with a whole spectrum of related, but different safe BO (and reinforcement learning) settings. For example, in robot learning
discrete input
7

---

Published in Transactions on Machine Learning Research (09/2024)
small amount of crashes might deemed acceptable, in which case crash-aware BO algorithm (Marco et al., 2021) would be preferable over SafeOpt-type algorithm. Similarly, in the context of bandit algorithms (Slivkins et al., 2019; Lattimore & Szepesvári, 2020), instead of avoiding bad option at all costs, one might instead want to be conservative, carefully exploring alternatives, starting from a default strategy. This setting is lormalized in the context of conservative bandils (Wu el al., 2016), and again, is related to, but distinct from, the SafcOpt setting. Corresponding bandit formulations in the context of hard safety are also available (Amani et al., 2019), where the focus is on regret bounds under safety constraints. Finally, parallel to the situation of conservative bandits, there are cautious variants of BO (Fröhlich et al., 2021).
4.2 Lipschitz-based methods
In order to address safety-related issues uncovered and discussed in Sections 5.1 and 6.1, we will introduce algorithms based on a Lipschitz assumption. While regularity properties like Lipschitz (and the more general Hölder) continuity play an important role in the theory of statistical learning, especially in nonparametric statistical estimation (Tsybakov, 2009), learning algorithms based on Lipschitz assumptions have received relatively little attention in the machine learning community. Relevant exceptions to this are Lipschitz bandits (Kleinberg al., 2019) and the original SafeOpt algorithm from Sui et al. (2015). The situation is considerably different in the field of global optimization and the systems and control community, respectively. In the former, Lipschitz continuity with a specific Lipschitz constant is a standard assumption, used in a varicty of algorithms for (cortified) global optimization, cf. (Hansen et al., 1992; Pintér, 1995), though usually a noise-free sel ling is assumed in this literature. Global optimization of Lipschitz functions has recently also received attention from the machine learning community (Malherbe & Vayatis, 2017). Similarly, a specified Lipschitz constant is also used in the context of Lipschitz interpolation (Beliakov, 2006). Furthermore, closoly related to our approach taken in Section 6, a deterministic variant of SafeOpt has been considered in Sergeyev et al. (2020). However, the latter reference only works with functions on a compact interval, and does not use any BO techniques. In the systems and control community, Lipschitz assumptions have long been used, especially in the context of systems identification, where they were explicitly introduced and popularized by Milanese & Novara (2004), though similar methods had been used before, e.g., (Cooper, 1995). A known bound on the Lipschitz constant and on the size of additive noise is commonly used to derive uncertainty bounds in the context of regression. This approach has been further popularized and extended to the case of Hölder continuous functions by Calliess (2014), widely known as kinky inference in the systems and control community. A central assumption in this context is knowledge of a concrete, numerical upper bound on the Lipschitz constant of the target function. This assumption has clear geometric and practical interpretation, namely a bounded rate of change of the larget quantity. As such, it is related to the well-established field of sensitivity analysis (Da Veiga et al., 2021), for example. Approaches to estimate the Lipschitz constant of an unknown function have been proposed both in the context of global optimization (Strongin, 1973) and in Lipschitz-based regression methods, particularly in the context of systems identification (Milanese & Novara, 2004; Novara et al., 2013; Calliess et al., 2020), see (Huang et al., 2023) for an overview and very recent sample-complexity results. We would like to stress that these approaches are not suitable for the present setting of hard safety constraints, since the estimation of Lipschitz constant bound requires queries to the target function, which in turn already need to be safe, see also the discussion in Section 6.1. The developments in Sections 6 and 7 combine kernel-based methods (here GP regression) with Lipschitz assumption to overcome the requirement of a known bound on the RKHS norm of the target function, see Section 6.1 for details. The problematic nature of an RKHS norm bound in the context of learning- based control has been recognized for some time (Lederer et al., 2019; Fiedler et al., 2022). In (Lederer et al., 2019), using probabilistic Lipschitz bounds together with a space discretization was suggested to derive GP uncertainty bounds. However, this approach relies in the context of SafeOpt-type algorithms. The work (Fiedler et al., 2022) proposes the usage of geometric constraints as prior knowledge in the context of uncertainty sets for kernel-based regression, with Lipschitz constant bounds as a special case. The resulting kernel machines, which provide nominal predict and smoothed uncertainty bounds adhering to the geometric constraints, are not necessary in our setting, though using more general geometric constraints than Lipschitz constant bounds might be a promising avenue in the context of SafeOpt-type algorithms. Finally, combining kernel methods with Lipschitz assumptions is natural approach as there is close connection between regularity properties of a kernel and Lipschitz
probabilistic setting, and is therefore not suitable
8

---

Published in Transactions on Machine Learning Research (09/2024)
continuity of functions in the RKHS generated by the kernel. For thorough discussion of this point, we refer to (Fiedler, 2023).
5 Frequentist uncertainty bounds and practical safety issues in SafeOpt
We now investigate practical safety implications of commonly used heuristics in the frequentist uncertainly bounds in Safeept-type algorithms. addressing objective (01). In Section 5.1, we discuss why these heuristics are problematic and demonstrate safety issucs using numerical experiments. To overcome these issues, in Section 5.2, we propose using state-of-the-art frequentist uncertainty bounds in the actual algorithm.
5.1 Practical safety issues in SafeOpt
Safety in SafeOpt-type algorithms is ensured by restricting query inputs to safe sets, which are computed using frequentist uncertainty bounds, typically in the form (2) using (3). However, these bounds are often too conservative for algorithmic use, which has led to implementations adopting heuristic choices for Be, for example, P 2 in (Berkenkamp et al., 2016; Turchetta et al., 2016), 8g = 3 in (Helwa et al., 2019; Baumann et al., 2021), 3 4 in (Sukhija et al., 2023). Using such heuristics instead of evaluating B, invalidates all safety guarantees. In practice, choosing some 3r can be a useful heuristic, as demonstrated by the reported success of SafeOpt-type algorithms (Berkenkamp et al., 2016; Baumann et al., 2021; Sukhija et al., 2023). However, it should be stressed that in the sctting of SafcOpt as outlined in Section 3, the learning algorithm has to fulfill a hard safety constraint namely, that no unsafe inpuls are queried by the algorithm (potentially only with high probability). In particular, no burn-in or tuning phase for 3r is allowed. Such heuristics not only invalidate the theorctical safety guarantoes, but can also actually lead to safety violations.
First, we demonstrate empirically that a simple heuristic like setting 3, = 2 can lead to significant propor- tion of bound violations. To do so, we follow the general approach adopted in (Fiedler et al., 2021a). We randomly generate 100 RKHS functions on [0, 1] with RKHS norm 10 using the squared exponential (SE) kernel, and for each of the functions, we generate 10000 data sets, run GP regression on it, and finally check whether the uncertainty set (2) with &r 2 fully contains the respective function. 2727:3882 (average + SD) of these runs (all 10000 repetitions for all 100 functions) led to a bound violation, sizcable proportion. Second, we show that these bound violations can indeed lead to safety violations when running SafeOpt. We generate an RKHS function f and run SafcOpt with 3e = 2, but all other algorithmic parameters set cor- rectly (or even conservatively) for 10000 independent runs. This leads to 2862 (out of 10000) runs with safely violations (see Table 1), which is unacceptable for most application scenarios of SafeOpt-type algorithms.
These experiments illustrate that using houristics in SafcOpt can be highly problematic. Even in the relative benign setting used above, both uncertainty bound and safety violations occur. Moreover, in application scenarios for SafeOpt-type algorithms, tuning the heuristic scaling factor is not possible, as it is the primary mechanism for safety. Dispensing with the need for such heuristics, and retaining safety guarantees both in theory and practice, is therefore the primary motivation for this work.
5.2 Real--3-aafeOp
As a first step, we propose using modern uncertainty bounds in SafeOpt that can be computed numerically, avoiding the replacement with unreliable heuristics. For this purpose, we investigate the original SafeOpt algorithm, as described by Algorithm 1, with 8, from (5). To clearly distinguish this variant of SafeOpt from previous work, we call it Real-B-SafeOpt, emphasizing that we use a theoretically sound choice of 3.. The bound (5) requires the determinant to be computed, which can be computationally expensive, but typical applications of SafeOpt. and related algorit allow few evaluations, so this does not pose problem. Furthermore, the additive noise needs to be conditionally R-subgaussian (martingaleddiffernce)) sequence with a (known upper bound on) R. This assumption is standard, has a clear interpretation, and is in many cases harmless. Finally, for frequentist uncertainty bound, we also need the next assumption.
Assumption 2. Some B E R2o is known with | B.
9

---

Published in Transactions on Machine Learning Research (09/2024)
Combining these ingredients allows us to compute the bound (5), Real-3-SafeOpt can actually be imple- mented. We would like to stress that both the underlying algorithm (SafeOpt) and the used uncertainty bound have been known, so we do not claim any new technical developments here. However, to the best of our knowledge, this concrete version of SafeOpt has not been considered in the litcrature so far, and no empirical evaluation on it has been conducted so far. To ralustrate the advantages of Real-3-SafeOpt, we run it on the same function f as before (cf. Section 5.1), and set = 0.01, as well as the true RKHS norm in (5). Running this experiment results in 0 failures (ef. Table 1), evidently well within the â range required. An obvious question arising at this point is how Real-3-SafeOpt compares in terms of performance with previous SafeOpt-variants relying on heuristics. Unfortunately, a meaningful comparison is impossible since the two algorithmic variants address different problems. If for the latter the heuristic constant B is determined by trial-and-error, and SafeOpt is then run with this constant that leads to no or almost no safety violation, then we essentially end up with a different algorithm overall. If the heuristic constant 3 is chosen ar arbitrarily or basod on experience with previous usages of SafeOpt-type algorithms, then the original setting of hard safcty constraints is abandoned, and instead we end up with a form of cautious BO. By contrast, Real-3-SafeOp tries to stay within the original setting of SafeOpt that requires hard safety constraints. To investigate how Real-3-SafeOpt typically behaves, we will perform a careful empirical evaluation of this algorithm variant in Section 6.4.
6 Lipschitz-only safe Bayesian optimization (LoSBO)
In this section, we address objective (02), in particular, we investigate the central Assumption 2 of a known upper bound on the RKHS norm of the target function, finding that this is a problematic assumption in practice. To overcome this issue, we propose to rely instead on a known Lipschitz and noise bound, and introduce Lipschitz-only Safe Bayesian Optimization (LoSBO) as an appropriate algorithm, on which we perform exlensive numerical experiments.
6.1 Practical problems with the RKHS norm bound
A central ingredient in the Real-3-SafeOpt algorithm is an upper bound on the RKHS norm of the target function. In particular, the safety and exploration guarantees inherited from (Sui et al., 2015) hinge on the knowledge of concrete numerical upper bound on the RKHS norm. Unfortumately, while the RKHS norm is very well understood from thcoretical point of view, in practice it is currently not possíble to derive concrete nunerical upper bounds on the RKHS norm from realistic assumptions in non-trivial cases, to the best of our knowledge, cf. the additional discussion in Section A.2. This issue is known in e.g. the learning-basee control community (Fiedler et al., 2022), but has not yet been addressed in the context of safe BO. However, it has immediate consequences for SafcOpt-type algorithms. These algorithms are intended for hard safcty settings, i.e., scenarios where any safety violation very costly and must be avoided. In these scenarios it is not possible to have a tuning phase before the actual optimization run where algorithmic parameters are set, since the safcty requirements hold from the start. For SafcOpt-type algorithms this mcans that all algorithmic parameters have to be set beforehand, and these parameters need to ensure the safety and lead to satisfying exploration behavior. However, this entails an upper bound on the RKHS norm of the target function, which currently appears to be impossible to derive from reasonable prior knowledge in practically relovant scenarios.
Furthermore, using an invalid RKHS upper norm bound can indeed easily lead to safety violations. In order to illustrate this, we run Real-3-Safeepp on the same function f from Section 5.1, and set 8 8= 0.01, but this time, we use misspecified RKHS norm of 2.5 in (5). Running this experiment now results in 1338 failure runs out of 10000, cf. Table 1, which is much more than what would be expected from safety probability of 1 - 8 = 0.99. For a summary of this experiment, see again Table 1. Finally, simply using a very conservative upper bound on the RKHS norm is not a viable stralegy to overcome this problem. A severe overestimation of the RKHS norm leads to very large and overly conservative uncertainty bounds, which in turn leads to performance issucs. In particular, since the uncertainty bounds are used to determine the safety sets in SafeOpt-type algorithms, a supposed RKHS upper norm bound that is too conservative can result, in the algorithm "getting stuck", i.e., no more exploration is possible. It. should also be noted
10

---

Published in Transactions on Machine Learning Research (09/2024)
that it is not even clear what "very conservative" means in the present context. While an extensive body of theory and strong qualitative intuitions on these objects are available, cf. Section A.2, the lack of concrete, quantitative bounds amenable to numerical evaluation makes it very difficult for users of BO to judge what a conservative estimate of the RKHS norm in a given situation could be.
We argue that as a consequence, any SafeOpt-like algorithm should not depend on the knowledge of an upper bound on the RKNS norm of the target function for safety, at least in the setting of hard safety requirements where any failure should be avoided (with high probability). More generally, in order guarantee safety, should only use assumptions that are judged as reliable and reasonable by practitioners. In particular, all assumptions that are used for safety should have a clear interpretation for practitioners and a clear connection to established prior knowledge in the application area of the safe BO algorithm. In the end, it is up to the user of safe BO to decide which assumptions used in the safety mechanism can be considered as reliable.
6.2 Describing LoSBO and its safety guarantees
Motivated by the popularity of SafeOpt, which combines GPs with a Lipschitz assumption, and the extensive experience of the systems and control community with Lipschitz bounds and bounded noise (cf. Section 4), we propose to use an upper bound on the Lipschitz constant of the target function and known noise bound as the ingredients for safety in BO. The key idea is to ensure that the safety mechanism works reliably independent of the (statistical) exploration mechanism. In a generic SafeOpt algorithm, safety is guaranteed by ensuring that the safe sets S, contain only safe inputs (potentially only up to high probability), i.e., roquiring that f(x) 2 h for all x E St. Once this property is fulfilled, the rest of the algorithm can no longer violate the safely constraints anymore. Based on the earlier discussion, the construction of the safe set should only rely on the Lipschitz and noise bound. As is well-known, these two assumption allow the construction of lower bounds on the function (Milanese & Novara, 2004), and the corresponding safe sets should therefore be defined for all t 2 1 as
U
E Id(x;-1, h):
where L E R20 is a bound on the Lipschitz constant of the unknown target function, E E R2 a bound on the magnitude of the noise, and So the initial safe set. We propose using this variant of the safe set, and leaving the rest of the generic SafeOpt algorithm unchanged, which leads to an algorithm we call Lipschitz- only Safe Bayesian Optimization (LoSBO). Our proposed modification applies any algorithm instantiating the generic SpfeOpt strategy outlined in Section 3. For concreteness, we focus in the following the original SafeOpt algorithm from (Sui et al., 2015). The algorithm fulfills the following safety guarantee.
Proposition Let f : D + R be an L-Lipschitz function. Assume that |ea S E for all 2 1 and let 0 # So C D such that f(x) h for all x E So- For any choice of the scoling factors B, > 0, running the LoSBO algorithm leads to a sequence of only safe inpuls, i.e., we have [(x) 2 h for all t 2 1.
Proof. It is enough to show that Vt 0 and St, we have f(x) 2 h. Induction on t: For t 0, this follows by assumption. For i 2 1, let E S = S.-1 Uf E D Vt-1 - E Ld(x)-1,x) 2 hJ. If x E S then f(x) 2 h follows from the induction hypothesis. Otherwise we have
f(x) ((x) + - f(x) f(x.) 2 v E Ld(x,, h,
where we used the T-Liischitz continuity of f and the noise bound leel S E in the first inequality, and the definition of S, in the second inequality.
The argument in the proof above is well-known in e-g. the systems identification literature, and the resulting bounds even fulfill certain optimality properties (Milanese & Novara, 2004). We would like to stress that the safety guarantce of LoSBO, as formalized in Proposition 1, is deterministic, i.c., it always holds and not only with high probability. This type of safety is often preferred in the context of control and robotics (Hewing et al., 2020; Brunke et al., 2022).
11

---

Published in Transactions on Machine Learning Research (09/2024)
6.3 Discussion
While LoSBO arises from a rather minor modification of the generic SafeOpt algorithm class (by changing the computation of the safe sets St), a conceptual level significant differences arise. Inspecting the proof of Proposition 1 shows that the safety guarantee of LoSBO is independent of the underlying model sequence (Mz).. As an important consequence, the choice of the uncertainty sets used in the optimization of the acquisition function cannot jeopardize safety. One consequence is that the assumption that the target function f is contained in the RKHS of the covariance function used in GP regression is not necessary anymorc. In particular, in order to ensure safety, we need only Assumption together with a noise bound, and not Assumplion 2 anymore. Sinilarly, hyperparameter tuning is not an issue for safely, cf. also to our discussion in Section 2.2. Of course, appropriate function model is important for good exploration performance, but this issuc is now independent of the safety aspect. As another, even more important consequence, in the context of the concrete LoSBO variant described in Algorithm 3 (Appendix), the scaling parameter 3e is now a proper tuning parameter. Modifying them, even online, in order to improve exploration no longer interferes with the safety requirements. This differs from previous variants (and practical usage) of SafeOpt, where the scaling factors Be cannot be freely tuned since they are central for the safety mechanism of these algorithms. This aspect is illustrated in Figure 3. In the situation depicted there, the uncertainty bounds do not hold uniformly, ie., the target function is not completely covered by them, and deriving safety sets from these uncertainty bounds, regardless of whether to include the additional knowledge of the Lipschitz bound (Sui et al., 2015) or not (Berkenkamp et al., 2016), results in potential safety violations. However, since in LoSBO these bounds are ignored for the safe sets, this problem does not occur.
true function posterior mean uncertainty set Lipschitz cone data points unsafe area "wrong" safe set safe
safe set LoSBO
Figure Illustration of LosBO being safe, while safe set based on invalid uncertainty bounds leads potential safety violations. The safe set of LoSBO (gray set) is determined by the constant E (black arrow) and the Lipschitz cone (orange). The GP mean and the confidence bounds are illustrated in blue. The points in the safe set given by the lower confidence bound are green if they are safe and red if they are unsafe.
Using a Lipschitz bound instead of an RKHS norm bound comes with several advantages. First, a (quantitative) Lipschitz bound has a clear interpretation it an upper bound on the slope of the target function. Second, the Lipschitz assumption can be related to established prior knowledge: A known upper bound the Lipschitz constant corresponds to priori bound the rate of change of function, i.e., it is related to the sensitivity of the underlying problem. As discussed in-depth in Section 6.1, to the best of our knowledge this is not possible for the RKHS norm in non-trivial cases. Third, already mentioned above, the Lipschitz assumption is independent of the internal model used for exploration, and hence much kernels. Finally, the exact Lipschitz
less prone to model misspecifications like wrong hyperparameters constant rarely known in practice, and hence an upper bound has to be used. is clear that a very conservative upper bound will inhibit exploration, just as a conservative upper bound on the RKHS norm, but this issue appears to be unavoidable with any quantitative a priori assumption.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm
SafeOpt
Real-3-SafeOpt
Real-3-SafeOpt
Real-B-SafeOpt == 20 (overestimate)
LosBO
B = (heuristic)
B = 2.5 (underestimate)
B = 10 (correct)
3 = 2
Experimental setting
Not started % Safety violations % Safety violations
1.93
3.16
30.40
68.34
0.018
3.95
0.859
0
0
28.62
13.38
0
worst case %
Final performance %
88.75
88.76
82.45
76.69
90.90
Table 1: Safety-performance tradeoff in SafeOpt. We evaluated 100 functions sampled from an SE-kernel with B = 10. On each function we ran each algorithm 10000 times, starting from two initial safe points.
The original SafeOpt algorithm comes with conditional exploration guarantees. Since our modification leading to LoSBO essentially separates the safety and exploration mechanisms, the exploration guarantees from SafeOpt are rendered inapplicable in the present context. An inspection of the proof of (Sui et al., 2015, Theorem 1) shows that it cannot easily be modified to apply to LoSBO again, as the argument used there relics on the GP model interacting with the safety mechanism". Furthermore, we suspect that pathological situations exist where LoSBO fails to properly explore. However, we have not observed such situuatonn in our extensive experiments. While providing (again conditional) exploration guarantees for LoSBO is an interesting focus for future work, we argue that the present lack of such theoretical guarantees does not di- minish the relevance and usefulness of this algorithm. First, LoSBO shows excellent exploration performance, as demonstrated in the experiments described in the next section. Second, since the scaling parameters 3 (which have an important influence on the exploration performance) are proper tuning parameters in LoSBO, unsatisfying performance of the algorithm can be overcome by using this tuning knob. We would like stress again that in the previous variants of SafeOpt, this option is not available the scaling parameters need to lead to valid uncertainty sets.
6.4 Experimental evaluation
As discussed in Section 5.2, a meaningful comparison with SafeOpt implementations relying on heuristics is impossible, since the latter essentially address different problem setting. For this reason, we will compare LoSBO with Real-33-afeOOt, which precisely adheres to the original SafeOpt setting. For the empirical evaluations a frequentist approach will be used as this is the most natural setting for SafcOpt-like algorithms, see (Srinivas et al., 2010; Fiedler et al., 2021a;b) for further discussion on this. Unless noted otherwise, in each experiment, we generate 100 RKHS functions (using the pre RKHS and ONB approaches), each with RKHS norm 10, and we compute a (slightly) conservative Lipschitz bound as well as an appropriate initial safe set. Following the previous safe BO literature, we restrict ourselves to compact subsets of R4, and in this section we use d and D [0, 1] for simplicity. Furthermore, we use independent additive noise, uniformly sampled from [-B,, B.], with (5) to compute 3 in Real-8-SafeOpt, and E 2B, and Br = 2 in LoSBO. For each function we run the algorithms 10000 times to allow a frequentist evaluation of the behavior. The performance is measured by the (shifted and normalized) gap to the optimal function value ft. Additional details on the experimental setup are provided in the appendix, cf. Section C.
Well-specified setting We start by comparing LoSBO and Real-3-SafeOps in a well-specified setting, This mcans that all the algorithmic parameters are set correctly, in particular, the covariance function used in GP regression is the kernel generating the RKHS from which the target functions are sampled. In Figure 4, the results for this setting are presented. The thick solid lines are the means over all 100 functions and all of their repetitions, the fine lines are the means for each of the individual 100 target functions (over
By this the keruel. Inspecting the expression for the time I in this latter result, find that this result requires appropriate growth behavior of the maximum information gain tu in order lead non-vacuous exploration guarantces. More precisely, with LoSBO we cannot ensure that the inequality the last display in the proof of (Sui al., 2015, Lemma 7) holds.
that the exploration guarantee given in (Sui et al., 2015, Theorem is conditional the choice of
13

---

Published in Transactions on Machine Learning Research (09/2024)
Pre-RKHS Matern
Pre-RKHS Matern
ONB SE
Pre-RKHS Matern
lGp 41g
ler 0.2le
1.0
0.5
0.0
0
10
20
0
10
20
0
10
20
0
10
20
Iteration
Iteration
Tteration
Tteration
RealBetaSafeOpt
RealBetaSafeOpt
LoSBO
LoSBO
Figure Comparison of LosBO and Real-3-SaleOOl a well-specified setting. Thick solid lines are the means over all functions and repetitions, thin solid lines are the means over all repetitions for each individual function, shaded area corresponds to the 10 % and 90 % quantiles over all runs.
10000 repetitions for each function), and the shaded area shows the % and % quantiles, again over all functions. Figure 4, left, displays the results for functions from the Squared Exponential RKHS, sampled using the ONB approach. Interestingly, LoSBO exhibits superior performance compared to Real---SafeOpt, despite providing the latter algorithm with the correct ingredients (Lipschitz bound, kernel, RKHS norm bound, noise variance). Sampling functions using the pre RKHS approach shows only minor differences, cf. Section C.3 in the appendix. Figure 4, second plot, shows the results for RKHS functions corresponding to a Matern-3/2 kernel and sampled with the pre RKHS approach. Qualitatively, we see the same picture, though the performance of both LoSBO and Real-3 S-afeOt appear to be slightly weaker compared to the previous setting. Intuitively, this is to be expected Matern RKHS functions are generically less smooth than Squared Exponential RKHS functions, and both LoSBO and Setl-3-SafeOpt rely Lipschitz bound, which is in turn related to regularity of functions.
Misspecified sctting We turn to misspocified settings, where the algorithmic paramcters do not match the true setling of the larget function. This is particularly interesting in the present siluation, since the underlying GP model does not impact the safety of LoSBO, and therefore becomes amenable to tuning, We start with the longth scale used in the kernel, which is arguably the most important type of hyperparameter in practice. In Figure 4, third plot, we show the results for overestimating the length scale in GP regression, using Matern kernels. More precisely, a Matern kernel used both as the kernel for generating the target functions and as a covariance function in GP regression, but the length scale of the covariance function in GP regression is 4 times the ength scale used in the kernel to generate the target function. The qualitative picture remains the same, though it appears that the performance of Real-33-aafeOp suffers from the misspecification more than LoSBO. More importantly, in this setting safety violations occur in 12.57 % of all runs. Moreover, for the worst-behaving target function 943 out of 10000 runs lead to safety violations, which is unacceptable in a real-world use case. Figure 4, rightmost plot, shows the complementary situation of underestimating the length scale in GP regression, again using Matern kernels. The length scale of the covariance function used GP regression 0.2 times the length scale of the kernel that is used to generate the target function. Again, the qualitative picture remains the same, but the performance degradation is worse for both algorithms in this case. Finally, consider the case where different kernel is used to generate the target functions than the covariance function in the GP regression. We use Matern-3/2 kernel to generate the target functions, and a Squared Exponential kernel as covariance function in the GP regression, with the same length scale for both. The results are displayed in Figure 7, lower left. Interestingly, essentially no qualitative difference can be noticed compared to the well-specified Matern caso, Figure 7 the appendix. We suspoct that this is due the correct specification of the length scale, which in the present setting is more important than the kernel misspecification.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Y0
-E 12
X2
XO
X1
82
D
Figure 5: Illustration of safe sets for a 2-dimensional input set. Left: Safe set in LoSBO resulting from three function evaluations. Right: Illustration of the acquisition function optimization in LoS-GP-UCB over a safe set resulting from four function evaluations. Two initial guesses for the local optimization are used for each of the balls that span the safe set according to (12).
7 LoS-GP-UCB
In SafeOpt-type algorithms, including our variant LoSBO (Algorithm 3), the central computational step is the optimization of the acquisition function over the expander and maximizer sets, see Algorithm 1. Inspecting the definition of the latter two sets, see Section 3, makes clear that computing these sets For many typical application scenarios, at least parts of the input set will
requires a discrete input be continuous (e.g., the optimization variables include physical parameter that can vary continuously). This mcans that some form of discretization necessary before a SafeOpt-type algorithm Typically, equidistant gridding of the (continuous parts of the) input set is used discretization, see (Berkenkamp et al., 2016) for typical example. As a result, SafeOpt-type algorithms become impractical for even moderate dimensions, e.g., D C Rd for d > 3 (Kirschner et al., 2019a; Sukhija et al., 2023), and as a member of this class, LoSBO inherits this limitation. In this section, we present and investigate an approach to overcome this issue. Instead of adapting existing solution approaches like (Duivenvoorden al., 2017: Kirschner et al., 2019a; Sukhija et al., 2023), we suggest pragmatic and straightforward variant that is motivated by three obscrvations.
be applicable.
Firsi, safety in SafeOpt-type algorithms is ensured by restricting the optimization of the acquisition function to (subsets of) safe sets, i.e., sets S C D such that fls> h, where f is the unknown target function. In other words, as long as we ensure that the acquisition function optimization is restricted to such sets S, the resulting algorithm will be safe, no matter how this optimization is performed or whether an additional restriction added (as in SafeOpt, where the optimization is only over expander and maximizer sets). In the case of LoSBO, these safe sets are of a particularly simple form, as they are the union of closed spheres in metric space, see Figure left, for an illustration for the case D C RR?. In typical application scenarios of SafeOpt-type algorithms, the number of input queries is relatively low, and hence the aforementioned union is only over relatively few sets. Second, discrete input set for SafeOpt-type algorithms is necessary due to the involved definition of the expander and maximizer sets, which in turn are defined to guarantee proper exploration in the original SafeOpt setting (Sui et al., 2015). However, for concrete practical problems, such an underexploration might not pose severe challenge, and an existing BO algorithm can simply be made safe by restricting the optimization of the acquisition function to safe set S as described above. In fact, the original SafeOpt paper (Sui et al., 2015) already discussed safe variant of GP-UCB (Srinivas
15

---

Published in Transactions on Machine Learning Research (09/2024)
et al., 2010). This indicates that it might be possible to avoid the complicated sets involved in SafeOpt-type algorithms, and still attain practically useful exploration performance. Third, in the current practice of BO, moderately high dimensions of the input space are not problematic for modern BÔ algorithms* Typically, acquisition functions are optimized by running a local optimization method (usually gradient-based) from several initial guesses (which might be randor,, or based on heuristics). In particular, no gridding is necessary, and this strategy can even deal with moderately high dimensions since local search methods behave well despite increasing dimensionality. In fact, state-of-the-art BO libraries like BOTorch (Balandat al., 2020) implement exactly this approach as the default option.
Based on these three observations, we now propose straightforward safe BO algorithm that works even in moderate input dimensions, is compatible with modern BO libraries, and retains all the safety guarantees from LoSBO. In Section 7.1, we describe this algorithm in detail and provide some discussion. In Section 7.2, we evaluate the algorithm empirically and compare to LoSBO.
7.1 Algorithm
Consider the setting of LoSBO as described in Section 6.2. Motivated by the preceding discussion, we start with a standard GP-based BO algorithm that does not need expander and maximizer sets (or similar complicated sets requiring discretization). Due to its relation to SafeOpt, we choose GP-UCB (Srinivas et al., 2010) for this task, so at step & 2 1, the next input is
T++1 = argmax + (,Or(T),
(10)
for an appropriate scaling factor Be E 30 As usual, ties are broken arbitrarily. Using the (scaled) posterior variance as the acquisition function would be even closer to SafcOpt, but numorical experiments indicato that GP-UCB performs slightly better in this context. Next, we restrict the acquisition function optimization to the safe sets St defined for LoSBO in (9),
rt+1 = argmax -e5, + BLor(x).
(11)
Observe now that
N S. = U (2,) j-1
(12)
for some N, Nso, UN, E Rso and ,N, D,, and Bx(2) = fx E D dp(z,z) S r is the closed ball with radius r E Eso and center e D in the metric space D. For example, if the initial safe set has only one element Fo and no input is repeatedly sampled, then N₂ = 1 + 1, 21 To and 2j = Iz-1 for = ......tt + 1. Using the decomposition (12), we now have
T++1= arggma arg.maa BEB, (e()) + (x),
(13)
Each of the inner optimization problems ma (2z) (z( + (z), j = I....,N. No. is a maximization
problem over the convex sets B,,(2), (z5), and each of these inner problems are independent. In particular, these optimizations can be trivially parallelized. In practice, one usually has D C R2 (often with simple geometry) and a difforentiable covariance function k so it is possible to use a gradient-based local optimization mothod started from multiple initial guesses. This is illustrated in Figure 5, right. All of these multistarts are independent, and can therefore also be parallelized. We thus arrive at Algorithm 2, which we call Lipschitz- only Safe Gaussian Process Upper Confidence Bound (LoS-GP-UCB) algorithm in the following. LoS- GP-UCB can casily be implemented with state-offthh-art BO libraries. For the numerical experiments described in the next section, we have chosen Borch (Balandat et al., 2020), which allows an easy parallel implementation of the acquisition function optimization. Finally, LoS-GP-UCE retains the safcty guarantces from LoSBO. In particular, the scaling factors (8t), remain tuning factors, and the safely of LoS-GP-UCB is independent of their choice. Furthermore, safety of LoS-GP-UCB does not require Assumption 2.
Heree are referring handling the dimensionality within the algorithm, specifically optimization of the acquisition function, and not the exploration performance. Of course, the latter poses challenges, especially if not enough structural or qualitative prior knowledge is encoded in the BO function model.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 2 LOS-GP-UCB
Require: Lipschitz constant L, algorithm compute Sr, noise bound E, initial safe set So, safety threshold
h
1: Compute No, 21,..., NNo ,l...,, Noo 80
2: for 1= 1,2.... ..2....du do
3: = ar.. ar (2,) (2,) (()) (0)++ + 8 (0-)))(1(2).
b Determine next input
4: Query function with æt, receive J/ = f(x)) +
Update GP with new data point (It, yt), resulting in mcan μr and di
Compute updated Be
Compute N₂ and add new 2z, F3
8: end for
7.2 Experimental evaluation
Analogously to the case of Real-3-SafeOpt and LoSBO, a frequentist setup will be used, i.e., the algorithm will be rum on fixed target function for many independent noise realizations. In the experiments two aspects will be investigated. First, LoS-GP-UCB will be compared with Real-B-SafeOpp and LoSBO. Second, we apply LoS-GP-UCB several benchmark functions with moderate input dimensions.
We start with the comparison to Real-3-SafeOpt and LoSBO. Since these two algorithms rely discrete input space, this comparison necessarily has to be performed on functions with low dimensional input. We choose essentially the same experimental settings as in Section 7.2 and consider only the well-specified case. As for LoSBO, we use 3. in LOS-GP-UCB, and we consider one-dimensional RKHS functions. The algorithms are evaluated on 100 target functions sampled from the pre RKHS corresponding to a Matern-3/2 kernel, and for each function, we run the algorithms 1000 times". The results of this experiment are shown in Figure (a), where we again use the evaluation metric (16). Thick solid lines are the means over all functions and repetitions and shaded areas correspond to the 10 % and 90 % quantiles (again over all functions and realizations). To avoid clutter, the means for each individual function are plotted only for LoS-GP-UCB (thin lines). It is clear from Figure 6 that the performance of Los-GP-UCB is only slightly inferior to the original LosBO algorithm, yet still superior to Seal-B-SafeOpt.. This outcome indicates that LoS-GP-UCB is not severely affected by the under-exploration problem described for a safe variant of GP-UCB in (Sui al., 2015). We suspect that, similar to LoSBO, this is due the overoptimism resulting from setting Br = 2, corresponding to moderately aggressive exploration.
(a) Pre-RKHS Matern
(b)
(c)
(d) Camelback2D
Gaussian10D
Hartmann6D
1.0
4
0.5
0.0
0
10
20
0
50
100
50
100
50 Iterations
100
Iteration
Iterations
Iterations
RealBeta
LosBO
LoS-GP-UCB LoS-GP-UCB
Random
RealBeta
LosBO
Random
Figure 6: (a) Comparison of LOS-GP-UCB to Real-8-SafeOpt and LoSBO. (b)-(d) Comparison of Los-GP-
UCB to random sampling more high-dimensional test functions.
9The qualitative picture already clear for 1000 repetitions, hence we choose computational resources and 10000 repetitions
Section 7.2.
17

---

Published in Transactions on Machine Learning Research (09/2024)
Let us turn to the evaluation of dimensions. As test functions, use the standard benchmarks Camelback (2d) and Hartmann (6d). Similar to (Kirschner et al., 2019a), we also use a Gaussian function f(x) = exp (-4[|zr))) in ten dimensions (10d) as a benchmark. For the Camelback and Hartmann functions, we choose a random initial point in the safe set. For the Gaussian function, we choose a random safe set. from the level f(x)) 0.4. We assume uniformly bounded noise with a noise level of 0.01. In this synthetic setting the Lipschitz constant L is determined by evaluating the function on a fine grid. As a model we use a Squared Exponential kernel with output variance set to and length scale set to L/L. For the prior mean choose 0.5 as the function values are between O and 1. Finally, in all these settings, we compare LoS-GP-UCB to random search, and run both algorithms from the same initial safe set for 100 iterations, repeating this 100 times (for different random choices of the initial safe set). The results of this experiment is shown in Figure 6 (b), (c) and (d).
several benchmark functions with moderate to high imput
8 Conclusion
In this work, we are concerned with practically relevant safety aspects of the important class of SafeOpt-type algorithms. We identified the use of heuristics to derive uncertainty bounds as potential source of safety violations in the practical application of these algorithms. This prompted us to use recent, rigorous uncer- tainty bounds in SafeOpt, which allowed us to numerically investigate the safety behavior of this algorithm. We further identified the knowledge of an upper bound on the RKHS norm of the larget function a serious obstacle to reliable real-world applicability of SafeOpt-type algorithmss. To overcome this obstacle, we proposed LoSBO, a BO algorithm class relying only on a Lipschitz bound and noise bound to guarantee safety. Numerical experiments demonstrated that this algorithm is not only safe, but also exhibits supe- rior performance. However, analogously to related algorithms, LoSBO is only suitable for low dimensional problems. We therefore also proposed an additional variant (LOS-GP-UCB) suitable for moderately high dimensional problems.
The two key assumptions to ensure safety are known Lipschitz bound on the target function and a known bound on the additive measurement noise, which have clear interpretations, appear natural in many ap- plications, and are established in domains like control engineering. The safety guarantees of the proposed algorithms rely on these assumptions, and they have to be judged on a case-by-case base by practitioners.
Ongoing work is concerned with implementing the presented algorithms for safe learning in an automotive context. While this article focuses safety, providing exploration guarantees for LoSBO is interesting aspect of future work. We expect that the approach outlined in Section 6 applies to most SafeOpt variants. The derivation, implementation, and evaluation of the corresponding LoSBO-type algorithms for these vari- ants is thus another interesting direction for future work. Our findings in combination with evidence in the literature that SafeOpt and related algorithms have been successfully used in various applications indicate that this algorithm class does not ensure hard safety constraints (in practice), but instead yields "cautious" behavior. The precise connection to conservative bandits and existing cautious BO approaches is another interesting topic for further investigations.
Broader Impact Statement
This work is concerned with safety issues of popular BO algorithm that has already found numerous applications in real-world scenarios. Henceforth we contribute to the improved safety and reliability of machine learning methods for rcal-world applications. Furthermore, we expect no adverse socictal impact of our work.
Acknowledgments
We thank Paul Brunzema and Alexander von Rohr for very helpful discussions and Sami Azirar for support when generating plots. This work was performed in part within the Helmholtz School for Data Science in Life, Earth and Energy (HDS-LEE). Furthermore, the research was part funded by the German Foderal Ministry for Economic Affairs and Climate Action (BMWK) through the project EEMotion and by the Deutsche Forschugssemeinschaaft (DFG, German Research Foundation) under Germany's Excellence Strategy EXC-
18

---

Published in Transactions on Machine Learning Research (09/2024)
2023 Internet of Production 390621612. Computations were performed with computing resources granted by RWTH Aachen University under project rwth1459.
References
Yasin Jobasi-Yadkori Online learning for linearly parametrized control problems. 2013.
Sanae Amani, Mahnoosh Alizadeh, and Christos Thrampoulidis. Linear stochastic bandits under safety constraints. Advances in Neurol Information Processing Systems, 32, 2019.
Marc Alleia. Hilbertian kernels and spline fumclions. Elsevier, 1992.
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. BoTorch: A framework for efficient. Monte-Carlo Bayesian optimization. Advances in neural information processing systems, 33:21524 21538, 2020.
Dominik Baumann, Alonso Marco, Mattco Turchetta, and Sebastian Trimpe. Gosafe: Globally optimal safe robot learning. In 2021 IEEE International Conference on Robolács and Aulomation (ICRA), pp- 4452-4458. IEEE, 2021.
Gleb Beliakov. Interpolation of Lipschitz functions. Journal of computational and applied mathematics, 196 (1):20-44, 2006.
Felix Berkenkamp, Angela P Schoollig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. In 2016 IEEE international conference on robotics and automation (ICRA), pp. 491-496. IEEE, 2016.
Felix Berkenkamp, Angela Schoellig, and Andreas Krause. No-regret Bayesian optimization with unknown hyperparameters. Journal of Machine Learning Research, 20:1-24, 2019.
Felix Berkenkamp, Andreas Krause, and Angela P Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning, 112(10):37133 3747, 2023.
Alain Berlinet and Christine Thomas-Agnan. Reproducing kernel Hilbert spaces in probability and statistics. Springer Science Business Media, 2004.
Ilija Bogunovic and Andreas Krause. Misspecified Gaussian process bandit optimization. Advances Neural Informalion Processing Systems, 34:3004-31 -014-, 2021.
Lukas Brunke, Melissa Greeff, Adam W Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P Schoellig. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5:411 444, 2022.
Jan-Peter Calliess. Conservative decision-makiig and inference in uncertain dynamical systems. PhD thesis, Oxford University, UK, 2014.
Jan-Peter Calliess, Stephen Roberts, Carl Edward Rasmussen, and Jan Maciejowski. Lazily adapted constant kinky inference for nonparametric regression and model-reference adaptive control. Automatica. 122:109216, 2020.
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. 34th International Con- ference on Machine Learning, 1CML 2017,
2017.
Duane Cooper. Learning Lipschitz functions. International journal of computer mathematics, 59(1-2): 15 26, 1995.
Sébastien Da Veiga, Fabrice Gamboa, Bertrand Iooss, and Clémentine Pricur. Basics and trends in sensitivity analysis: Theory and practice in R. SIAM, 2021.
19

---

Published in Transactions on Machine Learning Research (09/2024)
Rikky RPR Duivenvoorden, Felix Berkenkamp, Nicolas Carion, Andreas Krause, and Angela P Schoel- lig. Constrained Bayesian optimization with particle swarms for safe adaptive controller tuning. TFAC- PapersOnLine, 50(1):11800 11807, 2017.
Christian Fiedler. Lipschitz and Hölder continuity in reproducing kernel Hilbert spaces. arXiv preprint arXiv:2310.18013 2023.
Christian Fiedler, Carsten W. Scherer, and Sebastian Trimpe. Practical and rigorous uncertainty bounds for Gaussian process regression. Proceedings of the AAAI conference on artificial intelligence, 35, 2021a.
Christian Fiedler, Carsten W. Scherer, and Sebastian Trimpe. Learning-en enhanced robust controller synthesis. 60th TEEE Conference on Decision and Control (CDC), 2021b.
Christian Fiedler, Carsten W. Scherer, and Sebastian Trimpe. Learning functions and uncertainty sets using gcometrically constrained kernel regression. In 2022 IEEE 61st Conference on Decision and Control (CDC), pp. 2141-2146, IEEE, 2022.
Christian Fiedler, Michael Herly, and Sebastian Trimpe. On kernel-based statistical learning in the mean field limit. In Advances in Neural Information Processing Systems, volume 36, 2024.
Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. The Journal of Machine Learning Research, 21 (1)4:8464,, 2020.
Lukas Fröhlich, Melanie Zeilinger, and Edgar Klenske. Cautious Bayesian optimization for efficient and scalable policy search. In Learning for Dynamics and Control, pp. 227-240. PMLR, 2021.
Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
Pierre Hansen, Brigitte Jaumard, and Shi-Hui Lu. Global optimization of univariate Lipschitz functions: I. survey and properties. Mathematical programming, 55(133):251-2 272, 1992.
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
Mohamed K. Helwa, Adam Heins, and Angela P. Schoellig, Provably robust learning- based approach for high-aceuracy tracking control of Lagrangian systems. IEEE Robotics and Automation Letters, 4(2):1587- 1594, 2019. ISSN 23773766. doi:
José Miguel Hernández-Lobato, Michael A Gelbart, Ryan P Adams, Matthew W Hoffman, Zoubin Ghahra- mani, et al. A general framework for constrained bayesian optimization using information-best based search. Journal of Machine Learning Research, 17(160):1-53, 2016.
Lukas Hewing, Kim P Wabersich, Marcel Menner, and Mclanic Zeilingor. Learning- model predictive control: Toward safe learning in control. Annual Review of Coutrol, Robolies, and Aulonomous Systems, 3:269 296, 2020.
Julien Walden Huang, Stephen J Roberts, and Jan-Peter Calliess. On the sample complexity of Lipschitz constant estimation. Transactions on Machine Learning Research, 2023.
Toni Karvonen and Chris Oates. Maximum likelihood estimation in Gaussian process regression ill ill-posed. Journal of Machine Learning Research, 24(120):1-47, 2023.
Toni Karvonen, George Wynne, Filip Tronarp, Chris Oates, and Simo Sarkka. Maximum likelihood es- timation and uncertainty quantification for Gaussian process approximation of deterministic functions. SIAM/ASA Journal on Uncertainty Quantification, 8(3):926 958, 2020.
Youngmin Kim, Richard Allmendinger, and Manuel Tópeze Safe learning and optimization techniques: Towards a survey of the state of the art. In International Workshop on the Foundations of Trustworthy AT Inlegrating Learning, Optimization and Reasoning, pp- 123 139. Springer, 2020.
20

---

Published in Transactions on Machine Learning Research (09/2024)
Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces. In International Conference on Machine Learning, pp. 3429 3438. PMLR, 2019a.
Johannes Kirschner, Manuel Nonnenmacher, Mojmir Mutný, Andreas Krause, Nicole Hiller, Rasmus Is- chebeck, and Andreas Adelmann. Bayesian optimisation for fast and safe parameter tuning of SwissFEL. In FEL2019, Proceedings of the 39th International Free-Electron Laser Conference, pp. 707-710. JACoW Publishing, 2019b.
Robert Kleinberg, Aleksandrs Slivkins, and Eli Upfal. Bandits and experts in metric spaces. Journal of the ACM (JACM), 66(4):1-71 17: 2019.
Torsten Koller, Felix Berkenkamp, Matteo Turchetta, and Andreas Krause. Learning-based model predictive control for safe exploration. Proceedings of the IEEE Conference Decision and Control, 2018-Decem: 6059-6066, 2019. ISSN 07431546. doi: 20////DC.22100000000
Hayri Korezlioglu. Reproducing kernels in separable Hilbert spaces. Pacific Journal of Mathematics, 25()):
305 314, 1968.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Armin Lederer, Jonas Umlauft, and Sandra Hirche. Uniform error bounds for Gaussian process regression with application to safe control. Advances in Neural Information Processing Systems, 32, 2019.
Milan Lukić and Jay Beder. Stochastic processes with sample paths in reproducing kernel Hilbert spaces. Transactions of the American Mathematical Society, 353(110)3945-3969 2001.
Cédric Malherbe and Nicolas Vayatis. Global optimization of Lipschitz functions. In International Conference on Machine Learning, pp. 2314-2323. PMLR, 2017.
Alonso Marco, Dominik Baumann, Majid Khadiv, Philipp Hennig, Ludovic Righetti, and Sebastian Trimpe. Robot learning with crash constraints. IEEE Robotics and Automation Letters, 6(2):1439 1446, 2021.
Mario Milanese and Carlo Novara. Set membership identification of nonlinear systems. Automatica, 40(6): 957-975, 2004.
Carlo Novara, Lorenzo Fagiano, and Mario Milanese. Direct feedback control design for nonlinear systems. Automatica, 49(4):8449888) 2013.
Baver Okutmuştur. Reproducing kernel Hilbert spaces. PhD thesis, Bilkent Universitesi (Turkey), 2005.
Vern I Paulsen and Mrinal Raghupathi. An introduction to the theory of reproducing kernel Hilbert spaces, volume 152. Cambridge university press, 2016.
János D Pintér. Global optimization in action: continuous and Lipschitz optimization. Algorithms, imple- menialions and appliculions, volume 6. Springer Science & Business Media, 1995.
C. E Rasmussen and C 1 Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, USA, 2006.
Parisa Sarikhani, Benjamin Ferleger, Jeffrey Herron, Babak Mahmoudi, and Svjotlana Miocinovie. Au- tomated deep brain stimulation programing with safety constraints for tremor suppression. Braân Stimulation, 14(6):1699-177) 1700, 2021. ISSN 1935861X. doi: 20211.... URL https: //doi.org//i 1tw./.11....... btt.://..br...000 www.b........................... 10.357.
Yaroslav 1) Sergeyev, Antonio Candelieri, Dmitri E Kvasov, and Riccardo Perego. Safe global optimization of expensive noisy black-box functions in the 5-Lipschitz framework. Soft Computing, 17713)3177155177755
2020.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando De Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEFF, 104(1):148-11) 2015.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trends in Machine Learning, 12(1-2):1-286, 2019.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. ICML 2010 Proceedings, 27th International Conference Machine Learning, pp. 1015-1022, 2010. doi: 10.1109/ (..1099/TIT.20 TTT.TTTT22211122005.5
Ingo Steinwart and Andreas Christmann. Support vector machines. Springer Science & Business Media, 2008.
Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approzimation, 35:363 11363-11,, 2012.
Ingo Steinwart, Don R Hush, Clint Scovel, et al. Optimal rates for regularized least squares regression. In COLT, pp. 79-93, 2009.
RG Strongin. On the convergence of algorithm for finding global extremum. Eng. Cybernetics, 11: 549 555, 1973.
Yanan Sui, Alkis Gotovos, Joel Burdick, and Andreas Krause. Safe exploration for optimization with Gaus- sian processes. In International conference on machine learning, pp. 997 1005. PMLR, 2015.
Yanan Sui, Vincent Zhuang, Joel Burdick, and Yisong Yue. Stagewise safe Bayesian optimization with Gaussian processes. In International conference on machine learning, pp. 4781-4789. PMLR, 2018.
Bhavya Sukhija, Matteo Turchelta, David Lindner, Andreas Krause, Sebastian Trimpe, and Dominik Bau- mann. Gosafeopt: Scalable safe exploration for global optimization of dynamical systems. Artificial Intelligence, 320:103922, 2023.
Aretha L Teckentrup.
Convergence of Gaussian process regression with estimated hyper parameters and applications in Bayesian inverse problems. STAM/ASA Journal on Uncertainty Quantification 834):1310- 1337, 2020.
Abdullah Tokmak, Christian Fiedler, Melanie N Zeilinger, Sebastian Trimpe, and Johannes Köhler. Au- tomatic nonlinear MPC approximation with closed-loop guarantees. arXio preprint arXiv:2312.10199,
2023.
Alexandre Tsybakov. Introduction Nonparametric Estimation. Springer New York, NY, 2009.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite Markov decision processes with Gaussian processes. Advances in Neural Information Processing Systems, (Nips):4312- 4320, 2016. ISSN 10495258.
Holger Wendland. Scattered data approzimation, volume 17. Cambridge university press, 2004.
Justin Whitchouse, Aaditya Ramdas, and Steven Z Wu. On the sublincar regret of GP-UCB. Advances in Newral Information Processing Systems, 36, 2024.
Yifan Wu, Roshan Shariff, Tor Lattimore, and Csaba Szepesvári. Conservative bandits. In International Conference on Machine Learning, pp. 1254 1262. PMLR, 2016.
George Wynne, François-Xavaa Xavier Briol, and Mark Girolami. Convergence guarantees for Gaussian process means with misspecified likelihoods and smoothness. The Journal of Machine Learning Research, 22(1): 1468-5507, 2021.
22

---

Published in Transactions on Machine Learning Research (09/2024)
A Background on RKHSs
A.1 A brief introduction
Consider a function k: D x D + R. We call k positive semidefinite if for all 21,..., EN E D, N E Nso, the matrix (.).....))) is posilive semidefinite. Equivalently, the function l is symmelric (k(z); kzz,, for all i.j = NN,. and for all aN E R we have Σnj==1 a,ayk(1, 2T) 2 0. In the literature such a function is often called positive definite or of positive type. Additionally, is called positive definite, if for all pairwise distinct 2I,...,IN E D, the matrix (k(T,; "=)),. is positive definite. This property is sometimes referred to as strict positive definiteness in the literature.
i.=
Let H be a Hilbert space functions on D. We call H a reproducing kernel Hilbert space (RKHS) if every evaluation functional is continuous, ie., for all x E the mapping H t [+ [(x) E is contimuous w.r.b. the norm induced by the scalar product in H. Furthermore, h called a reproducing kernel (for H) if 1) 2(xxx) E H for all D, and 2) f(x) (f, k)) hr for all f E H and x E D.
As is well-known, H is RKHS if and only if it has reproducing kernel, and in this case the latter is unique (Steinwart & Christmann, 2008, Lemma 4.19, Theorem 4.20). Furthermore, every reproducing kernel is positive semidefinite, and every positive semidefinite function is a reproducing kernel for a unique RKHS (Steinwart & Christmann, 2008, Theorem 4.16, 4.21). If k is positive semidefinite, then we denote its unique RKHS as (He, ()), and the induced norm by || llk- Furthermore, we define the pre RKHS by
N
m = span (k(-,z) a D) = Σ an, In) | N E NNo. on E In E D, n= D.n=L....NN
(14)
and this subspace is dense in H3 w.r.t. || llk-- Given f = En ank(( 9= =EM= Pmnkl, Yre) H we have
(f.9)= -E Σ (mmmkkymmszn)
(15)
see (Steinwart & Christmann, 2008, Theorem 4.21).
k is the covariance function of a GP, then it is positive semidefinite (since every covariance matrix is positive semidefinite), and hence the reproducing kernel of unique RKHS. Conversely, if the reproducing kernel of RKHS, then & is positive semidefinite, there exists a GP having & as its covariance function, and the GP can be chosen with zero mean function (Berlinet & Thomas-Agnan, 2004).
Furthermore, consider GP regression with prior GP k), then μe 1 E H where μ is the posterior mean corresponding to finite data set with t points. In particular, the posterior mean for a zero mean prior GP is always in the pre RKHS corresponding to the covariance function. As is customary in machine learning with GPs (Rasmussen & Williams, 2006), and also in many BO scenarios (Garnett, 2023), in the following we will use zero mean GP prior, m O, without loss of generality.
A.2 Additional discussion
Theory for the RKHS norm For an arbitrary kernel, one can use liseretization-based variational char- acterizations of the RKHS norm (and RKHS functions), for example, by maximization over a family of lower bounds on the RKHS norm (Fiedler et al., 2024, Section B), (Atteia, 1992, Chapter 1), by minimization over certain bounds on function values at finitely many inputs (Okutmuştur, 2005, Theorem A.2.6), by minimiza- tion over finite interpolation problems (Paulsen & Raghupathi, 2016, Theorem 3.11), by minimization over certain matrix incqualitics (Paulsen & Raghupathi, 2016, Thcorem 3.11). For separable RKHSs, the RKHS norm can be expressed using sampling expansion (Korezlioglu, 1968), or as the limit of norms of RKHSs over finite inputs (Lukić & Beder, 2001, Lemma 4.6). On the one hand, all of these variational problems have an explicit form and they work for any kernel (any kernel with separable RKHS, respectively). However, it is not at all clear how to relate these representations to common properties of functions that might be used reliable prior knowledge to derive upper bounds the RKHS norm. Furthermore, these variational
23

---

Published in Transactions on Machine Learning Research (09/2024)
problems generally cannot be used in numerical methods to estimate upper bounds the RKHS norm, but only lower bounds, though they may be used for estimating bounds in heuristies (Tokmak et al., 2023). Since these characterizations are based on discretizations of a given RKHS function, in particular, using the exact function values, they are not suitable in the present setting of unknown target functions only accessible through noisy evaluations.
If one considers more specific classes of kernels, other characterizations of the RKHS norm become available. For example, contimuuus kernels compact metric space equipped with a measure having full support (often called Mercer kernel in this context) allow description of the RKHS norm as weighted Ey-norm (Steinwart& & Christmann, 2008, Section 4.5), based on Mercer's theorem. This has a clear interpretation in the context of kernel methods, in particular, giving insight into the regularization behavior of the RKHS norm in optimization problems in kernel machines (Hastie et al.; 2009, Section 5.8), which in turn can be used to derive learning rates for various statistical learning problems (Steinwart et al., 2009). More general forms of Mercer's theorem are available (Stcinwart & Scovel, 2012), which in turn lead to improved learning theory results (Fischer & Steinwart, 2020). While the RKHS norm representation for Mercer kernels is an important tool for statistical learning theory and provides intuition about the regularization behavior, it is again unclear how it can be used to derive quantitative RKHS norm bounds. Expressing the RKHS norm for Mercer kernels as a weighted fg-norm provides valuable qualitative intuition about the corresponding RKHS norm, but we are not aware of any practically relevant example where this has been used to translate realistic prior knowledge into a concrete upper bound on the RKHS norm.
Similarly, for sufficiently regular iosslaaton-ivvaiiann kernels, the RKHS norm can be expressed as a weighted integral over the Fourier transform of RKHS functions (Wendland, 2004, Theorem 10.12). This formulation allows intuitive interpretation of the RKHS norm as generalized energy, penalizing high-frequency behavior of RKHS functions (as determined by the Fourier transforms of the kernel). Several important function spaces are related to RKHSs, for example certain Sobolev spaces (Wendland, 2004, Chapter 10) or Fock spaces (Stcinwart & Christmann, 2008, Scction 4.4), which again have their own reprosentations of the RKHS norm (potentially after some embedding). Again, all of these representation offer insights inlo the RKHS norm, and are important theoretical tools, but how this can be used to derive practically useful quantitative upper bounds on the RKHS norm remains unclear.
To summarize, while an extensive body of work on characterization and representation results for the RKHS norm is available, these results appear to be unsuitable to derive numerical upper bounds on this norm using practically meaningful prior knowledge.
RKHS membership The assumption that the target function f is contained the RKHS 1l is standard in the BO literature (Abbasi- Yadkori, 2013; Srinivas et al., 2010; Chowdhury & Gopalan, 2017; Whitehouse et al., 2024), where k is a kernel that is used as the covariance function in GP regression. In general, if functions from H₂ are to be used as models of the underlying target function, the RKHS Hę has to be sufficiently rich. In many cases, as a qualitative assumption, this is quite mild. For example, D is a compact metric space, then a large variety of universal kernels are available kernels with an RKHS that is dense (w.r.t. the supremum norm) in the set of all continuous functions D (Steinwart & Christmann, 2008, Section 4.6). However, the assumption f E H3 is considerably more stringent and delicate. Specifically, even containment of the target function within an RKHS generated by a kernel from the same class as the one used as a covariance (e.g,, a squared exponential or Matern kernel) may not be sufficient, because there can be a mismatch of hyperparameters. For the case of squared exponential kernels, a complete characterization of the inclusions of RKHSs w.r.t. different. hyperparameters is described in (Steinwart & Christmann, 2008, Section 4.4), revealing that this situation can indced occur. While there is work on covariance function misspecification in GP regression (Wynne al., 2021)10 and GP-based BO (Berkenkamp et al., 2019; Bogunovic & Krause, 2021), these works do not help in the present situation, because of the form of the resulting guarantees in the former case, and the potential for schemes sampling unsafe inputs in the latter case. The situation is further complicated by the fact that in practice the hyperparameters are adapted during BO (Garnelt, 2023), theoretical guarantees of which are only slowly emerging (Teckentrup, 2020; Karvonen et al., 2020; Karvonen & Oates, 2023). While these issues do not directly apply to LoSBO
1ve likelihood misspecification.
24

---

Published in Transactions on Machine Learning Research (09/2024)
and LoS-GP-UCB, cf. also the discussion in Section 6.3, they are relevant for Real-3-SafeOpt. In Sections 5 and 6, we work (unless noted otherwise) with the assumption adaption during the optimization process. This is justified from two perspectives. First, it is a common approach in the BO litcrature, inchuding safe BO (Sui et al., 2015; 2018). Second, and more importantly, the safety-related issues that we are concerned with this work are independent of this issue (though of course, is an additional problematic factor).
Hı, and do not consider hyperparameter
B Additional implementation details and practical considerations
Pseudocode for LoSBO For the reader's convenience, provide detailed description of LoSBO in Algorithm 3. As discussed in Section 6, the only change compared to SafcOpt (and Real-3-SafiOptt) is in the computation of the safe sets.
Algorithm 3 LoSBO
Require: Lipschitz constant L, algorithm to compute Sr, noise bound E, initial safe set Sy, safety threshold h
Qo(x) + R for all E D Cu(x)= + [h. for all E So 3: Ca(x) +- R for all x E D S.
> Initialization of uncertainty sets
4: for tort= t= =, do
5: C+(x) - (z-1 n (x) for x E & Compute upper and lower bounds for current iteration
((x) min (x)), (z) max (x(r) for all
if 1 then
Compute new safe set
8:
S. S.-1 fx E Vt-1 E
h
else
10:
S = So
11:
end if
12:
G. - {a E St E ur(x) - Ld(x, x²) 2 hi
Compule set of potential expanders b Compute set of potential maximizers Determine next input
13:
M, fI St u(x)= 2 max (x(=))}
14:
3t f argmax,eG..M, wr(x)
15:
Query function with 24, receive y f(x)) + E
16:
Update GP with new data point (Ir, resulting in mean la and dx
17:
Compute updated St
18:
Qx(x) = [x)(2) - 3e o,(x), + B, (x) for all x e D
19:
end for
Choice of noise bound in LoSBO Proposition states that if E. E Ry0 is a bound on the noise magnitude, then LoSBO is safe. Suppose we know that |e.l s B, for some constant B, E R20, then we could set E Be, and assume that the bound B, is sharp. For example, 6 is the Dirac distribution with atom Proposition 1, i.c., for all inputs E D that are queried by the algorithm, we have f(x.) 2 h. However, if we additionally assume that there are sizeable parts of the input space D where f Rs h, and since the border the safe sets will be such a region, it is likely that inputs from this area will be queried. This means that it likely that measurements //2 with Yt < will be received this happens f(x) a -B. While according to the formal model such an input x, is safe, to the user looks as if a safety violation occurred. Since in practice detected (not necessarily real) safety violation might lead to some costly action (e.g., emergency stop), such situation is undesirable. To avoid this, While this introduces some conservatism, it avoids the described apparent safety violations essentially, this option mitigates false alarms. Whether to choose in the present situation E = B. or E = 2B, (or even an option in between) is ultimately a practical question to be addressed by the practitioner using the algorithm.
might have & _00g 26-B, where If we choose E Be, then LoSBO is indeed safe according to
and
set E = 2B..
Assumptions of LoSBO Inspecting the proof of Proposition reveals that considerably more general (and weaker) assumptions can be used with the same argument.
25

---

Published in Transactions on Machine Learning Research (09/2024)
1. Instead of fixed noise bound E, one can mutatis mutandis use asymmetric, time-varying and heteroscedastic noise. Formally, one can assume that two functions Ee, F.u. : D x No + Rzo exist, such that for all t 2 1 and x E D, it holds that t)(x,t) S S Eu t), if x is the input used at time t.
2. Instead of Lipschitz continuity, one can assume that there exists a continuous and strictly increasing function Φ IRza -> Rzo with (0)) 0, such that for all x, x' D it holds that f(x') 2 f(x) (adn((x,')) This includes the case of Hölder continuity, which has previously been used in a similar context (Calliess, 2014).
To keep the presentation focused and casy to follow, we do not use this additional flexibility in the present work, but all developments in Sections 6 and 7 applies to these more general cases.
C Additional material on experiments
C.1 Additional details for Section 5.1
In Section 5.1, we work exclusively with the squared exponential (SE) kernel on the input set [0, with length scale 0.2/v2 In order to generate a synthetic RKHS function with known RKHS norm, we follow the approach from Fiedler et al. (2021a) and use the explicit orthonormal basis (ONB) of the SE kernel RKHS as described in (Steinwart & Christmann, 2008, Section 4.4). More preciscly, we select a random subset of these basis funetions, generate randomly a vector of coefficients and rescale it (to achieve the required RKHS norm), and then form the weighted sum of the selected basis functions.
For the first experiment, in order to generate the data sets, 100 inputs are uniformly sampled from [0, 1], the corresponding RKHS function is evaluated on these inputs, and finally i.i.d. normal noise with variance 0.01 is added to the function values. For each data set, we apply GP regression with a zero mcan prior, and the SE kernel as covariance function, using the same length scale as for the generation of the target functions, and we use the actual noise variance in the GP regression. Finally, we use the uncertainty set (2) with 8, 2, and check on fine grid on [0, 1] whether the target function from the RKHS is completely contained within this uncertainty set. As soon as the function value of one input is not covered by the uncertainty, we count this run as a failure. Note that this might be even slightly optimistic, since bound violation might happen in the between the grid points (which is very unlikely in this setup, however).
For the second experiment, we use the same approach as above to generate RKHS functions f. We define the safety threshold h by setting h M(f) - 0.25D(f), where ((f) and SD(F) are the empirical mean and standard deviation of the test function f evaluated on fine grid of the input space. This procedure is necessary to ensure that we get non-empty safe sets, but that the problem instance is sufficienlyy challenging. We further evaluate If'll on fine grid on the imput space, take the maximum of this, and multiply it by 1.1 to find a (slightly conservative) upper bound of the Lipschitz constant of f, which is used as the Lipschitz bound in the algorithm. We then run SafeOpt on 10000 times from & random safe initial state, again using i..d. additive normal noise with variance 0.01 (and using the correct noise variance in the GP regression). Finally, we select a target function that exhibits significant safety violations. Note that this approach is appropriate this context, since the safety property has hold for all potential target functions, and due to the large number of independent runs, no multiple testing problem arises.
C.2 Additional details for Section 6.4
A target function is therefore fixed, and the algorithms are run multiple times this same function with independent noise realizations. To enable the performance of the algorithms to be clearly evaluated, synthetic target functions will be used, and since we want to compare LoSBO with Real-3-SafeOpt- the latter requiring a larget function from an RKHS and with a known RKHS norm upper bound - we generale target functions from an RKHS. Unless noted otherwise, in each experiment we generate 100 RKHS functions, each with RKHS norm 10, and we compute a (slightly) consorvative Lipschitz bound as well as an appropriate initial safe set. Following the previous safe BO literature, we restrict ourselves to compact subsets of Rd, and in this section for simplicity further restrict. ourselves to 1.
26

---

Published in Transactions on Machine Learning Research (09/2024)
The frequentist setup is inherently worst-case, but for numerical experiments it is necessary to restrict ourselves to finitely many RKHS functions. Nevertheless, the RKHS functions used should be somewhat representative to give a meaningful indication of the algorithmic performance. Specifically, any bias due to the function generating method should be minimized. In the following experiments, we sample functions from the pre RKHS, i.e, given a kernel k, we randomly choose some M E Nso, :i......0M. E R and T1,.... .. E D and then use f Σm= (;) E H₃ as a target function, which works for any kernel. In the case of the squared exponential kernel, we also utilize the ONB described in Steinwart & Christmann (2008, Section 4.4), which we have already used for some of the experiments in Sections 5.1 and 6.1. Generating RKHS functions with more than one method ensures more variety of the considered RKHS functions. Moreover, with both approaches, the exact RKHS norm is available (and can be set by normalization), and the generated functions can be evaluated at arbitrary inputs. Unless noted of herwwwise, we generate RKHS functions with an RKHS norm of 10, ie., we consider target functions f E Hx with l| = 10. For a more thorough discussion of gencrating RKHS functions and subtle biases due to the chosen method, we refer to (Fiedler et al., 2021a).
LoSBO and Real-3-SafeOpl. work on arbitrary metric spaces, long as a kernel can be defined on that space. Following the previous safe BO literature, we restrict ourselves to compact subsets of Rd, and in this section for simplicity we further restrict ourselves to d = 1. To run LoSBO and Real-3-SafcOOt, we need a bound on the Lipschitz constant of the target function, as well as an imitial safe set. For the former, we restrict ourselves to kernels inducing contimuously differentiable RKHS functions, since the latter are Lipschitz continuous due to the compact domain. To determine a bound on the Lipschitz constant, we evaluate the target function on a fine discretization of the input domain, numerically compute an approximation of the Lipschitz constant, and multiply the result by 1.1 to counterbalance the discretization error. Since the target. functions are randomly generated, we compute an appropriate safety threshold for cach function that some portions of the input space are safe, and some are unsale. This avoids trivial situations for safe BO. More precisely, for given target function f, we compute its empirical mean p(f) and empirical standard deviation SD(f) on fine grid, and then set h = fuf)) 0.25D(f). Next, for cach target function f and safety threshold h, we need to generate initial safe set. Similar to the choice of the safety threshold, trivial situations should be avoided, particularly cases where no safe exploration is possible. To achieve this goal, we first determine some To E argmax,eD.fxt) then consider the set Dr where Izo is the largest interval such that xo E Imn and flr., 2 h + E. Finally, one input is randomly selected from this set, and the singleton set containing this latter input is then selected as the initial safe set. Using a singleton initial safe set common in the litcrature on SafeOpt-type algorithms, see (Berkenkamp et al., 2016).
The typical application scenario for SafeOpt-type algorithms is the optimization of some performance mea- sure by interacting with physical system. In particular, each function query is relatively expensive, hence in these sccnarios only a few function values are sampled. Motivated by this, in all of the following experiments, for each larget function, LoSBO and Real-3-SafeOpt are run for 20 iterations, starting from the same safe set. For each target function, this is repeated 10000 times to allow a frequentist evaluation of the behavior. Finally, each type of experiment is run with 100 different randomly generated target functions. To make runs with different target functions comparable, we evaluate the performance in a given run of a target function f by
fr I prngmaxpes,,t h
(16)
where (e(z) is the predictive mean (in LoSBO and Safeept the posterior mean) at time t 2 1, evaluated at input ær and f is the maximum of f. This metric will be averaged over all runs for a given f. and over all target functions, respectively, in the following plots.
For simplicity, independent additive noise, uniformly sampled from [-B,, is used in all of the following experiments. As is well-known, bounded random variables are subgaussian, and we can set R = B, in Real- B-SafeOpt. Additionally, we choose 5 = 0.01 and the true RKHS norm as the RKHS norm upper bound in Real-3-SafeOpt, unless noted otherwise. We further set the nominal noise variance equal to It in both LoSBO and Real-3-Safee S-tfeO... Following the discussion in Section B, we choose E = 2B, in LoSBO. Finally, we must specify a strategy to compute 8, in LoSBO. Recall from Section 6.2 that these scaling factors are now proper tuning parameters. In all of the following experiments, we use 8 2 in LoSBO, as this is a common choice in the literature on SafeOpt and GP-UCB. Choosing such simple rule also simplifies the
27

---

Published in Transactions on Machine Learning Research (09/2024)
Pre-RKHS SE
Misspecified Kernel lap = la
1.0
RealBetaSafeept RealBetaSafeOpt
4=
0.5
LoSBO
LoSBO
0.0
0
10
20
0
10
20
Iteration
Iteration
Figure 7: Additional experimental rosults for LoSBO. Left: Well-spocified setting using the SE covariance function and sampling larget functions using the pre RKHS approach. Left: Misspecified covariance function.
experimental evaluation, as additional tuning parameters or further algorit hmicc choices introduced. Unless noted otherwise, in all of the following experiments B, 0.01 is used.
C.3 Additional experimental results for LoSBO
We have repeated the first experiment from Section 6.4, however, now with the RKHS functions sampled using the pre RKHS approach. Figure 7, left, displays the results. While no real difference in performance is noticable for LoSBO, Real-3-SafeOpt appears to perform slightly better compared to target functions generated using the ONB approach. A potential explanation lies in the shapes of functions that typically arise in the two different sampling methods. As observed in (Fiedler et al., 2021a), functions sampled from the ONB look more 'bumpy" compared to pre RKHS functions, and appear to more challenging for the uncertainty bounds. Since Real-f3SafeOOt needs precisely adhere to these bounds, its exploration performance diminished. By contrast, LoSBO behaves overly optimistically underlying RKHS functions have RKHS norm 10, see also the evaluations in (Fiedler et al., 2021a). It appears that this over-optimism leads to better performance, and since for safety LoSBO does not rely scaling factors Bg that correspond to valid frequentist uncertainty sets, this over-optimism does not jeopardize safety.
Br = 2 usod, but the
28


Reviews:
[{"review_id": "euTZTsvzYw", "reviewer": "Reviewer_ZXHe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1) The theoretical results are solid, and the paper is well-written.\n\n2) The motivations are clearly articulated, with some supported by observations from experiments. Summaries in the main text provide valuable insights.\n\n3) The paper addresses a key question regarding safety in safe Bayesian optimization.\n\nSuggestions:\nOverall, I think this work is good. Here are some suggestions (not necessary) that might be helpful.\n\n1) I realize that some experiments in Section 5.1 may be sufficient to demonstrate that using heuristics in SafeOpt can be highly problematic. However, a theoretical analysis here would be more appreciated. Additionally, examining the underlying theory would provide more insight into why these heuristics are problematic and could lead to more interesting strategic approaches. I am trying to understand this part intuitively but find it difficult. Could you provide an interpretation of why such heuristics are problematic in an intuitive way?\n\n2) Regarding the experimental design, I am curious about the performance of each strategy for each question. For example, how does the proposed solution compare to the heuristic solution on benchmarks, similar to the comparison of RealBetaSafeOpt versus LoSBO in Section 6. What about the remaining two solutions for the two questions, O(2) and O(3), in Section 3?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "By deeply examining the practical implementation of a wide range of methods, specifically SafeOpt-type algorithms, the author questions three important aspects of these algorithms: 1) the use of heuristics instead of theoretically sound uncertainty bounds, 2) the assumption of an RKHS norm bound, and 3) the discrepancy between discrete input space in theory and continuous domain in moderately high dimensions in application. Overall, the motivations for addressing these three points are clear, and the proposed strategies for tackling these problems are novel.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7pMQ8SJ4Mz", "reviewer": "Reviewer_QF7s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this paper has some good critiques of previous safe BO methods and proposes a reasonable algorithm. That being said, after a lot of criticism of previous papers for unrealistic assumptions of bounding constants/etc, the authors provide a pretty flimsy justification for why we would know the Lipschitz constant of the function being optimized. Furthermore, for what it provides, the paper is too long in my opinion- it feels like a good 6-page paper trapped in the body of a 24 page paper. I think the authors can cut a lot out. More on these points below.\n\n## Strength 1: criticisms\n\nI like the analysis of the following criticisms:\n- Previous works not setting $\\beta_t$ to explicitly satisfy a uniform high-probability bound for containing the true function values, instead just using a heuristic\n- Assuming a known upper bound to the RKHS norm (this is very hard to calculate)\n\n## Weakness 1: known Lipschitz constant\n\nThe authors correctly mention that there are many established methods for estimating Lipschitz constants from partial observations of a function. The authors vaguely argue that it is easier to get a good estimate of a Lipschitz constant compared to an RKHS norm, which I agree with. However, after repeatedly emphasizing how it is critical to have a super-high probability safety _guarantee_ and avoid hacky methods, the authors estimate the Lipschitz constant using a grid of evaluations, then multiply by 1.1 \"just to be safe.\" This is very hypocritical: 1.1 is clearly another \"heuristic constant\" and this method could easily fail for suitably varying functions- think something like the [Weierstrass function](https://en.wikipedia.org/wiki/Weierstrass_function), which is \"smooth\" at a coarse level, but has low-level fluctuations.\n\nOf course, Weierstrass-like functions are clearly pathological, and would not be anticipated in practice. Yet, if the authors assume some sort of  general smoothness, this feels like a qualitatively similar assumption to $f$ belonging in a smooth RKHS, no?\n\n## Weakness 2: writing\n\nThis paper is, in my opinion, way too long for what it provides. It is clear that the authors have done a lot of reading and I appreciated thorough citations, but a lot of the discussion seemed tangential to the main points of the paper. Here are my thoughts section-by-section:\n\n1. Good intro, but in the future if the paper is made shorter, maybe remove the huge \"outline\" paragraph (since it would be less necessary)\n2. Intro to GPs makes sense. The introduction to RKHSs seems a bit too formal and long, given that RKHSs are not really used anywhere in the paper  (except for criticizing not knowing the RKHS norm). Section 2.3 seems very long and also not necessary: in the end all that is used is the bound in equation (7). Also, the sentence \"_where the probability is with respect to the data-generating process, rather than to the target function f_\" was confusing to me: are we not assuming that data is generated using $f$? Or are we assuming some fixed distribution over input values in addition to $f$?\n3. This seemed like it should also belong in the \"background\", probably at the beginning, since it is the definition of the problem. The list O1-O3 seem duplicated with the intro and sections 5-6. The reader is already expecting this content, why list it again?\n4. 4.1 has a lot of duplicate content with the intro. I think the reader gets the idea of why you would want to avoid \"bad\" function evaluations, no need to repeat this over and over again. 4.2 seems like a bunch of sentences that should just be elsewhere in the paper. The parts justifying the Lipschitz constant should probably be near Assumption 1. The parts mentioning sections 5-6 should probably just come after these sections.\n5. This section basically says \"you can't just set algorithm constants arbitrarily and expect safe behavior\". I think this could be said much more succinctly. I don't think the experimental demonstration provided much value (of course heuristic values will not always work).\n6. 6.1 seemed like it belongs in section 5.2, no? What does it have to do with Lipschitz constants? In 6.2, the algorithm could probably be stated first and then analyzed (currently Proposition 1 comes even before the statement of the algorithm). Also, I think it could be explained more intuitively (it is essentially just finding the regions which are guaranteed safe by the Lipschitz constant)\n7. This algorithm seems very similar to the one in section 6. Maybe they can be presented together? The experiments were also very long. Consider moving some of it to the experiments?\n8. Conclusion without much of a discussion of limitations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies \"safe\" Bayesian optimization (BO), which I understand is maximizing a function $f(x)$ while avoiding the evaluation of any inputs $x$ where $f(x)<h$ for some constant $h\\in\\mathbb{R}$.\nThe paper describes some theoretical and practical limitations of existing algorithms for safe BO,\nthen proposes a new algorithm called LoSBO (Lipschitz-only safe BO) which uses a Lipschitz continuity assumption to guarantee the \"safety\" behavior.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The main change I would request is to add discussion contrasting the pros/cons of estimating RKHS norm bounds vs Lipschitz constant bounds. Also, acknowledging the failure mode of underexploring if the Lipschitz constant is high is also important to guide future users of the proposed algorithms.\n\nI also highly highly suggest re-writing the paper to be much shorter (ideally < 10 pages). To do this, you can look at my notes above and think about re-structuring the paper by merging semi-duplicated sections and moving things to appendices. Here are some additional notes:\n- No need to repeatedly say how important safety is: you can just mention it once in the intro.\n- You can use appendices to add content which could be helpful to the paper, but not important for the main narrative. Here are some things which could go into an appendix?\n  - Experiments where existing BO algorithms are not safe if the parameters are not tuned correctly (it should not be surprising that this can happen; you can just mention your constructed example and refer readers to the appendix if they are interested in the details.)\n  - Extended definitions of RKHSs\n  - Extended literature review (where not super related to main point of paper)\n- Keep remarks short- e.g. remark 1 is not really a \"remark\" in my opinion... ([reading](https://math.stackexchange.com/questions/2097999/in-writing-mathematical-papers-what-qualifies-as-a-remark-and-not-just-some-d?rq=1))\n- Clarify what input dimension was used for your experiments in sections 6-7 (this detail was not clear)\n\nAlso, small typos:\n- You use \">>\" instead of \"\\gg\" to type $\\gg$\n- The notation \"SD\" with a hat looks awkward. You probably don't know the `\\widehat{}` command.\n- in your citations, you cite \"hernandez et al\" when it should be \"Hernández-Lobato et al\" (note the accents and a hyphenated last name)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NLTuncToXo", "reviewer": "Reviewer_Eqfg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1.\tThe paper is well motivated since it is clear to me that each problem the authors identify with current SafeOpt algorithms is important to solve. \n2.\tVery good use of diagrams to illustrate points/algorithms. \n3.\tAlgorithm blocks make it very clear which each algorithm is doing and imporves readability. \n4.\tThe paper is well written and easy to follow. \n\nWeaknesses: \n1.\tSee in requested changes below", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors identify and analyze three problems with current Safe Bayesian Optimization (SafeOpt) algorithms, propose three novel SafeOpt algorithms - one to address/solve each problem. First, the authors demonstrate that using typical Safe BO algorithms often leads to safety guarantees being invalid in practice. To solve this problem, the authors propose a new algorithm (the Real-β-SafeOpt algorithm) which only requires relatively small changes to the standard SafeOpt algorithms to recover the desired theoretical safety guarantees. Secondly, the authors point out that many SafeOpt algorithms are difficult to apply in real-world settings since they make the assumption that the target function has an upper bound on the reproducing kernel Hilbert space (RKHS) norm. To solve this problem, the authors introduce an additional new SafeBO algorithm (the Lipschitz-only Safe BO algorithm) which maintains safety guarantees without the need for the limiting RKHS norm assumption. In addition to removing this assumption, the authors demonstrate that this algorithm outperforms other state-of-the-art SafeBO algorithms on several test functions. Third, the authors point out that SafeOpt and other similar algorithms are difficult to apply to higher-dimensional problems since they assume a discrete search space. To fix this, the authors propose a third algorithm (Lipschitz-only Safe GP-UCB) which is makes Safe BO applicable to higher-dimensional functions while still having the same safety guarantees.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None of the changes list below are critical to securing my recommendation for acceptance of the paper, but I do think they could strengthen the work. \n1.\tThe authors claim that the significance of LoS-GP-UCB is that it allows SafeBO algorithm to be applied to higher-dimensional functions. However, in the experimental results section, the highest dimensional function that the authors apply LoS-GP-UCB to is 10-dimensional. It is still exciting and significant that LoS-GP-UCB is able to scale SafeBO up to 10 dimensions! But it would be very interesting to see what the limit is on the dimensionality where LoS-GP-UCB can be applied successfully? Could it be applied to say 20-dimensional function? How about 100-dimensional functions? Additional experiments on higher-dimensional functions could show what the upper limit is on the dimensionality where LoS-GP-UCB is no longer very useful. I think this would be informative for readers who may want to use this algorithm in practice, but again, this addition is not critical to securing my recommendation for acceptance. \n2.\t While the text is clear and easy to follow, some of the text could likely be made more concise (i.e. some things are said in several sentences that could be condensed down to one sentence).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the paper should be accepted in TMLR. See comments above regarding claims. 

I do agree with Reviewer QF7s that this paper is quite a bit longer than it needs to be. I would strongly encourage the authors to shorten the paper by making the writing more concise throughout. The usual length of a TMLR paper is around 12 pages.
Decision: Accept (poster)