Information-Theoretic Safe Exploration with Gaussian Processes
Authors: Alessandro Giacomo Bottero
Carlos E. Luis
Julia Vinogradska
Felix Berkenkamp
Jan Peters
Venue: NeurIPS 2022 Accept (2022)
Keywords: Safe exploration
Gaussian process

Abstract: We consider a sequential decision making task where we are not allowed to evaluate parameters that violate an a priori unknown (safety) constraint. A common approach is to place a Gaussian process prior on the unknown constraint and allow evaluations only in regions that are safe with high probability. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. Moreover, the way in which they exploit regularity assumptions about the constraint introduces an additional critical hyperparameter. In this paper, we propose an information-theoretic safe exploration criterion that directly exploits the GP posterior to identify the most informative safe parameters to evaluate. Our approach is naturally applicable to continuous domains and does not require additional hyperparameters. We theoretically analyze the method and show that we do not violate the safety constraint with high probability and that we explore by learning about the constraint up to arbitrary precision. Empirical evaluations demonstrate improved data-efficiency and scalability.

Information-Theovee Safe Exploration with Gaussian Processes
Alessandro G. Bottero 12 Carlos E. Luis! Julia Vinogradska' Felix Berkenkamp', Jann Peters?
Bosch Center for Artificial Intelligence, Germany 2 Technische Universität Darmstadt, Germany Al AlessandroGiacomo Btt.eaaee..oottttreeeeeeeee. Bottero@de. com
Abstract
We consider sequential decision making task where not allowed to eval- uate parameters that violate an priori unknown (safety) constraint. A common approach is place a Gaussian process prior on the unknown constraint and al- low evaluations only in regions that are safe with high probability. Most current methods rely on a discretization of the domain and cannot be directly extended to the continuous case. Moreover, the way in which they exploit regularity as- sumptions about the constraint introduces additional critical hyperparameter. In this paper, we propose an information-theoretii safe exploration criterion that directly exploits the GP posterior identify the most informative safe parameters to evaluate. Our approach is naturally applicable to continuous domains and does not require additional hyperparameters. We theoretically analyze the method and show that we do not violate the safety constraint with high probability and that we explore by learning about the constraint up to arbitrary precision. Empirical evaluations demonstrate improved data-efficiency and scalability.
Introduction
In sequential decision making problems, we iteratively select parameters in order optimize a given performance criterion. However, real-world applications such as robotics (Berkenkamp et al., 2021), mechanical systems (Schillinger et al., 2017) or medicine (Sui al., 2015) are often subject to additional safety constraints that we cannot violate during the exploration process (Dulac-Arool Arnold al., 2019). Since is a priori unknown which parameters lead to constraint violations, we need to actively and carefully learn about the constraints without violating them. That is, we need to learn about the safety of parameters by only evaluating parameters that are currently known to be safe.
Existing methods by Schreiter et al. (2015); Sui et al. (2015) tackle this problem by placing Gaussian process (GP) prior the constraint and only evaluate parameters that do not violate the constraint with high probability. To learn about the safety of parameters, they evaluate the parameter with the largest posterior variance. This process made more efficient by SAFEOPT, which restricts its safe set expansion exploration component to parameters that are close to the boundary of the current set of safe parameters (Sui et al., 2015) the cost of an additional tuning hyperparameter (Lipschitz constant). However, uncertainty about the constraint is only proxy objective that only indirectly learns about the safety of parameters. Consequently, data-efficiency could be improved with an exploration criterion that directly maximizes the information gained about the safety of parameters.
Our contribution In this paper, propose Eeformation-Thhoreeff Safe Exploration (ISE), safe exploration algorithm that directly exploits the information gain about the safety of parameters in order expand the region of the parameter space that we can classify as safe with high confidence. By directly optimizing for safe information gain, ISE is more data-cfficient than existing approaches without manually restricting evaluated parameters be on the boundary of the safe set, particularly
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in scenarios where the posterior variance alone is not enough to identify good evaluation candidates, as in the case of heteroskedastic observation noise. This exploration criterion also means that we do not require additional hyperparameters beyond the GP posterior and that ISE is directly applicable to continuous domains. We theoretically analyze our method and prove that it learns about the safety of reachable parameters to arbitrary precision.
Related work Information-based selection criteria with Gaussian processes models are success- fully used in the context of unconstrained Bayesian optimization (BO, Shahriari et al. (2016); Bubeck and Besa-Bianchi (2012)), where the goal find the parameters that maximize an a priori unknown function. Hennig and Schuler (2012); Jenrández-Lobard et al. (2014); Wang and Jegelka (2017) 7) select parameters that provide the most information about the optimal parameters, while Fröhlich et al. (2020) consider the information under noisy parameters. The success of these information based approaches also relies on the superior data efficiency that they demonstrated. We draw inspiration from these methods when defining an information-based criterion w.r.t. the safety of parameters to guide safe exploration.
In the presence of constraints that the final solution needs to satisfy, but which we can violate during exploration, Gelbart et al. (2014) propose combine typical BO acquisition functions with the probability of satisfying the constraint. Instead, Gotovos et al. (2013) propose an uncertainty-base criterion that learns about the feasible region of parameters. When we are not allowed to ever evaluate unsafe parameters, safe exploration is a necessary sub-routing of BO algorithms to learn about the safety of parameters. To safely explore. Schreiter et al. (2015) globally learn about the constraint by evaluating the most uncertain parameters. SAFFOPT by Sui et al. (2015) extends this to joint exploration and optimization and makes more efficient by explicitly restricting safe exploration to the boundary of the safe set. Sui et al. (2018) proposes STAGEOPT, which additionally separates the exploration and optimization phases. Both of these algorithms assume access to a Lipschitz constant to define parameters close to the boundary of the safe set, which is a difficult tuning parameter in practice. These methods have been extended to multiple constraints by Berkenkamp et al. (2021), 1), while Kirschner et al. (2019) scale them to higher dimensions with LINEBO, which explores in low-dimensional sub-spaces. To improve computational costs, Duivenvoorden et al. (2017) suggest continuous approximation SAFEOPT without providing exploration guarantees. All of these methods rely on function uncertainty to drive exploration, while we directly maximize the information gained about the safety of parameters.
Safe exploration also arises in the context of Markov decision processes (MDP), (Moldovan and Abbeel, 2012; Hans et al., 2008). In particular, Turchetta et al. (2016, 2019) traverse the MDP to learn about the safety of parameters using methods that, at their core, explore using the same ideas as OPFEOPT and STAGEOPT to select parameters to evaluate. Consequently, our proposed method for safe exploration is also directly applicable to their setting.
2 Problem Statement
In this section, we introduce the problem and notation that we use throughout the paper. We are given an unknown and expensive to evaluate safety constraint f X - R s.t. parameters that satisfy f() 2 0 are classified as safe, while others are unsafe. To start exploring safely, we also have access an initial safe parameter 2o that satisfies the safety constraint, f (x0) 2 We sequentially select safe parameters In E X' where to evaluate f in order to learn about the safety of parameters beyond 2p. At each iteration n, we obtain a noisy observation of yn == fxx,) + Vn that is corrupted by additive homoscedastic Gaussian noise in ~ (O, starting from 2o we aim to safely explore the domain so that we ultimately classify as safe all the safe parameters that are reachable from æo.
We illustrate the task in Figure 1a, where
As f is unknown and the evaluations yn are noisy, it is not feasible to select parameters that are safe with certainty and we provide high-probabile that the safety constraint/ f has bounded norm in the Reproducing Kernel Hilbert Space (RKHS) (Schölkopf and Smola, 2002) H4 associated to some kernel k X' x X + R with k(æ,, x²) - 1. This assumption allows us to to model ff as a Gaussian process (GP) (Srinivas et al., 2010).
safety guarantees instead. To this end, we assume
Gaussian Processes A GP is a stochastic process specified by mean function μ X + IR and a kernel h (Rasmussen and Williams, 2006). defines a probability distribution over real-valued functions on X, such that any finite collection of function values at parameters [æ1,.... x,] is

---

Safety threshold f((x) =0
Value of safe seed No
(a
Safety constraint f
7a (x:
Largest safe containing
(4/()) + (x)
(a) Problem components.
(h) ISE mutual information.
Figure I: In (a) we illustrate the safe exploration task. Based on the unknown safety constraint f. we are only allowed to evaluate safe parameters æ with values f()) 2 0 above the safety threshold (dashed line). Starting from a safe seed æn safe exploration strategy needs to discover the largest reachable safe region of the parameter space containing æg- (b) we show the mutual information 1xx a0,yy y) (z)) in green for different inside the safe set and for fixed outside (red dashed line). ISE maximizes this mutual information jointly over æ and z.
distributed multivariate normal distribution. The GP prior then be conditioned on (noisy) function evaluations D, = {(ar, y1) ) If the noise is Gaussian, then the resulting posterior is also GP and with posterior mean and variance
pa = ((x)) k(x)) T (K + I0 1t);
(xx) = k<x,, x)) - k(x)) (K + Io
where μ [u(=r,;.... is the mean vector at parameters 2, E D, and [y]. = y(æ;) the corresponding vector of observations. We have [k())] = k( ax), the kernel matrix has entries [K] == k(z,,,)) and the identity matrix. In the following, we assume without loss of generality that the prior mean identically zero: p(x) = O.
Safe set Using the previous assumptions, we can construct high-probabbility confidence intervals on the function values (ar). Concretely, for any > O it is possible find sequence of positive numbers { such that f()) E [un (z)) = (,)) with probability at least - 6, jointly for all
a X and n 2 1. For a proof and more details see (Chowdhury and Gopalan, 2017). We use these confidence intervals to define safe set
(2) which contains all parameters whose Bowlower confidence bound is above the safety threshold and the initial safe parameter Øp. Consequently, we know that all parameters in Sn, are safe, f(x) 2 0 for all æ E S., with probability at least 5 jointly over all iterations
S fxr E X' (x) - (x) 2 0} (zo).
Safe exploration Given the safe set Sn, the next question is which parameters in S, to evaluate in order to efficiently expand it. Most existing safe exploration methods rely uncertainty sampling over subsets of Sn. SAFEOPT-like approaches, for example, use the Lipschitz assumption on f to identify parameters in S, that could expand the safe and then select the parameter that has the biggest uncertainty among those. In the next sections, we present and analyze our safe exploration strategy, ISE, that instead uses information gain measure identify the parameters that allow us to efficiently learn about the safety of parameters outside of S..
InformatiionTThorree Safe Exploration
We present
Safe Exploration (ISE), which guides the safe exploration by using an coommation-thoreetie criterion. Our goal is to design an exploration strategy that directly exploits the properties of GPs to learn about the safety of parameters outside of S,, We draw inspiration from Hennig and Schuler (2012); Wang and Jegelka (2017) who exploit information-theonm design data-efficient BO acquisition functions for their respective optimization objectives.
insights to
Information gain measure In our case, we want to evaluate f at safe parameters that are maxi- mally informative about the safety of other parameters, in particular of those where we are uncertain
3

---

Algorithm 1 Information-Theereei Safe Exploration
1: Input: GP prior (Jo. k, σw), Safe seed æ00
2: for = 0, ...,, N do
3: 2n+1 + arg mas, max In
4: Un+1 + (1++1)++ +
5: Update GP posterior with 17441994+11
about whether they are safe or not. To this end, we need corresponding measure of information gain. We define such a measure using the binary variable (ar) = f())20. which is equal to one iff f(x) 2 O. Its entropy is given by
Hn [<))] P
(11- - (n(1-P%(zz)
(3)
where PTz (z) the probability of z being unsafe: + - erf (- 1 ( The random (
=
variable (z) has high-entropy when we are uncertain whether a parameter is safe not; that is, its entropy decreases monotonically as [pn [z)] increases and the GP posterior moves away from the safety threshold. It also decreases monotonically as (z) decreases and we become more certain about the constraint. This behavior also implies that the entropy goes to zero as the confidence about the safety of z increases, as desired.
Given our definition of V we consider the mutual information [fxx, y) (=)) between an observa- tion y at parameter z and the value of vp at another parameter 2. Since Vi is the indicator function of the safe regions of the parameter space, the quantity I, (xx, y): (=))) measures how much informa- tion about the safety of z we gain by evaluating the safety constraint f at at iteration n, averaged over all possible observed values y- This interpretation follows directly from the definition mutual
information: 7, ((ær, y); (z)) = H, [z)) Fy Hn [V)z (a. w)]] where H, (z)) the
entropy of (z) according to the GP posterior at iteration n, while Ha+1 [1x)) {x, y}} is its entropy
at iteration n + 1, conditioned on a measurement y at æ at iteration n. Intuitively, L, (1x,, w); (=)) negligible whenever the confidence about the safety of is high or, more generally, whenever an evaluation at æ does not have the potential to substantially change our belief about the safety of z. The mutual information large whenever an evaluation the safety of z to increase significantly. As an example, in Figure I we plot I, (fz, y (z)) as function of x: E Sn, for a specific choice of z and for an RBF kernel. As one would expect, we see that the closer it gets to the bigger the mutual information becomes, and that it vanishes in the neighborhood of previously evaluated parameters, where the posterior variance is negligible.
on average causes the confidence about
To compute I, ((x, v}: (z)). we need to average (3) conditioned on an evaluation over all possible values of y. However, the resulting integral is intractable given the expression of H, [x)] in (3). In order to get tractable result, we derive a close approximation of (3).
1
Pn (x(2) 21 Gn(z)
Ha [x))) H. [T)) = (n(2)
(4)
In(
The approximation in (4) is obtained by truncating the Taylor expansion of Hn, [ [2)) at the second order. and it recovers almost exactly its true behavior (see Appendix B for details). Since the posterior mean at z after an evaluation æ depends linearly on n,, (x). and since the probability density of y de-
pends exponentially on - (æ)), using (4) reduces the conditional entropy Ey, [i (x)
to Gaussian integral with the exact solution
E, Ha [z) (z, vi]]
of -1(=))11- 2), 1n(2) of + (z) (x, exp 0
(2(())
of + 22 oz(() of + (z) + (x,) f
1
(5)
where Pa (ar, 2) is the linear correlation coefficient between f(x) and f(z). and with e and C2 given by ci == 1/ ln(2) and C = 2c 1. This result allows us to analytically calculate the approximated

---

(a) Iteration
(b) Iteration
Iteration 8
Example run of ISE at different iterations. The GP's posterior mean (blue line) and confidence interval (4, + 3, (blue shaded) approximate the true safety constraint f (red line) based on selected data points (green crosses) and, together with the safety threshold at zero (orange line), identify the current safe set Sp (light blue bar, bottom). The vertical green line indicates the location of the next parameter En+1 selected by (6). Initially, ISE evaluates parameters the boundary of the safe set, but eventually also evaluates inside the safe set if that provides the most information. ISE quickly discover the largest reachable safe set (dark blue bar, bottom).
Figure 2:
which we use to define the ISE acquisition function, and which analyze theoretically in Section 4.
mutual information (12, v); V2)) [=)) -By BBy
(2)) [ w}|
ISE acquisition function Now that we have defined a way to measure and compute the information gain about the safety of parameters, next parameters to evaluate. The natural choice for such selection criterion is to select the parameter that maximizes the information gain; that is, we select 2n+1 according
use it to design an exploration strategy that selects the
En++1 arg max in (fzr v/: V (())). #ESn
#EX
where we jointly optimize over x in the safe set Sn and an unconstrained second parameter z. Evaluating In+1 according to (6) maximizes the information gained about the safety of some parameter x E X, so that allows us to efficiently learn about parameters that are not yet known to be safe. While can lie in the whole domain, the parameters where the safety constraint lie outside the safe set. By leaving z unconstrained, we show in our theoretical analysis in Section 4 that,
the most uncertain about
have learned about the safety of parameters outside the safe set, (6) resorts to learning about the constraint function also inside S,, An overview of ISE can be found in Algorithm and show example run one-dimensional illustration of the algorithm in Figure 2.
Theoretical Results
In this section, we study the expression for ī, (fxx.) (); (=()) obtained using (4) and (5) and analyze the properties of the ISE exploration criterion (6). By construction of S, in (2) and the assumptions on in Section 2, we know that any parameter selected according to (6) is safe with high probability, see Appendix for details. To show that we also learn about the safe set, we first need to define what it means to successfully explore starting from Co. The main challenge is that it is difficult to analyze how a GP generalizes based on noisy observations, so that it is difficult to define a notion of convergence that is not dependent the specific run. SAFEOPT avoids this issue by expanding the safe set not based on the GP, but only using the Lipschitz constant L. Contrary to their approach, we depend the GP to generalize from the safe set. In this case, the natural notion of convergence is provided by the the posterior variance. In particular, we say that at iteration n we have explored the safe up e-accuracy if of (x) < for all parameters ø in S,- In the following, ISE asymptotically leads either to e-accurate exploration the safe set or to indefinite expansion of the safe set. future work it will be interesting further investigate the notion of generalization and to derive a similar convergence result as those obtained by Sui et al. (2015).
show that
Theorem 1. Assume that an+t1 is chosen according to (6), and that there exists ñ such that for all
ñt Sa+1 C Sn, Moreover, assume that for all 2 ft (1,((r)) ) S M for some M >> for all
Sn, Then, for all > 0 there exists No such that 0 (x) S. for every S, if 2 ft + No.
5

---

The smallest such N₂ is given by
CYN N
N₂ min N N
SE
where b(n) == (n( { exp
M
0 2
maxpex; DI=N (D(;;; y (D)) is
IN
the maximum information capacity of the chosen kernel (Srinivas et al., 2010; Contal el al., 2014).
and C ln(2)/o3 In(1 + a.
Proof. See Appendix A.
Theorem 1 tells us that if some point the set of safe parameters S,, stops expanding, then the posterior variance over the safe set vanishes eventually. The intuition behind Theorem I is that there were a parameter æ in the safe set whose posterior mean remained finite and whose posterior variance remained bounded from below, then an evaluation of f at such æ would yield non negligible average information gain about the safety of æ. that, since æ is in the safe set, at some point ISE will be forced to choose to evaluate T, reducing its posterior variance. This result guarantees that, should the safe set stop expanding. ISE will asymptotically explore the safe set up to an arbitrary e-accuracy. In practice, we observe that ISE first focuses on reducing the uncertainty in areas of the safe set that are most informative about parameters whose classification is still uncertain (e.g. the boundary of the safe set), and only eventually turns to learning about the inside of the safe set. This behavior is what ultimately leads to the posterior variance to decay over the whole S,,. Therefore, even if in general it is not always possible to say whether or not the safe set will ever stop expanding, we can read Theorem as an exploration guarantee for ISE, as it rules out the possibility that the proposed acquisition function forever leaves the uncertainty high in areas of the safe set that, if better understood, could lead to an expansion of the safe set.
Theorem 1 requires bound on the GP posterior mean function, which is always satisfied with high probability based on our assumptions about f. Specifically, we have that |p, (x)] S 2,3, with probability of at least 1 - 5 for all n (see Appendix A for details). Therefore, it does not represent an additional restrictive assumption for f. Finally, we also note that the the constant N₂ defined by (7) always exists since the function b is monotonically increasing, long 7N grows sublinearly in N. Srinivas et al. (2010) prove that this the case for commonly-used kernel and, more generally, is prerequisite for data-efficient learning with GP models.
5 Discussion and Limitations
ISE drives exploration of the parameter space by selecting the parameters to evaluate according to (6). An alternative but conceptually similar approach to this criterion would be to consider the parameter that yields the biggest information gain on average over the domain, i.c., substituting the inner max in (6) with average over X. The resulting integral, however, is intractable and would require further approximations. Moreover, the parameter found by solving (6) will also yield a high average information gain over the domain, due to the regularity of all involved objects.
Being able to work in continuous domain, ISE can deal with higher dimensional domains better than algorithms requiring a discrete parameter space. However, as noted in Section 4, finding + as (6) means to solve a non-convex optimization problem with twice the dimension of the the parameter space, which can also become a computationally challenging problem as the dimension grows. In a high-dimensional setting, we follow LINEBO by Kirschner et al. (2019), which at each iteration selects random one-dimensional subspace to which it restricts the optimization of the acquisition function.
In Sections 2 and 3, we assumed the observation process to be homoskedastic. However, it needs not to be the case, and the results can be extended the case of heteroskedastic Gaussian noise. The observation noise at a parameter æ explicitly appears in the ISE acquisition function, since it crucially affects the amount information that we can gain by evaluating the constraint ʃ at æ. On the contrary, STAGEOPT-like methods do not consider the observation noise in their acquisition functions. As consequence, ISE can perform significantly better an heteroskedastic setting, as we also show in Section 6.

---

Lastly, we reiterate that the theoretical safety guarantees offered by ISE are derived under the assumption that f is bounded norm element of the RKHS space associated with the GP's kernel. In applications, therefore, the choice of the kernel function becomes even more crucial when safety is an issue. For details on how to construct and choose kernels see (Garnett, 2022). The safety guarantees also depend on the choice of 3,,- Typical expressions for S,, include the RKHS norm of the constraint f (Chowdhury and Gopalan, 2017; Fiedler et al., 2021), which is in general difficult to estimate in practice. Because of this, usually in practice a constant value of B,, is used instead.
6 Experiments
In this section we empirically evaluate ISE. Additional details about the experiments and setup can be found in Appendix C. As commonly done in the literature (see Section 5), we set 3, = 2 for all experiments. This choice guarantees safety per iteration, rather than jointly for all n and it allows for a less conservative bound than the one needed for the joint guarantees.
GP samples For the first part of the experiments, we evaluate ISE on constraint functions / that we obtain by sampling a GP prior at a finite number of points. This allows us to test ISE under the assumptions of the theory and we compare its performance to that of the exploration part of S STAGEOPT (Sui et al., 2018). STAGEOPT is a modified version of SAFEOPT, in which the exploration and optimization parts are performed separately: first the SAFEOPT exploration strategy is used to expand the safe set as much as possible, then the objective function is optimized within the discovered safe set. We further modify the version of STAGEOPT that we use in the experiment by defining the safe set in the same way ISE does, i.e., by means of the GP posterior, as done, for example, also by Berkenkamp et al. (2016). We select 100 samples from a wwo-dimensional GP with RBF kernel, defined in 2.5, 2.5] x 2.5, 2.5) and run ISE and STAGEOPT for 100 iterations for each sample. As STAGeOPT requires a discretization of the domain, we use this discretization to compare the sample efficiency of the two methods, by computing, at each iteration, what percentage of the discretized domain is classified as safe. Morcover, we also compare with the heuristic acquisition inspired by SAFEOPT proposed by Berkenkamp et al. (2016). This method works exactly as STAGEOPT, with the difference that the set of expanders is computed using directly the GP posterior, rather than the Lipschitz constant. More precisely, parameter is considered an expander if observing value of pm (x) + ,,,,(()) (z) at x would enlarge the safe set. For the STAGEOPT run, we use the kernel metric to compute the set of potential expanders, for different values of the Lipschitz constant L. From the results shown in Figure 3a, we see not only that ISE performs as well or better than all tested instances of STAGEOPT, but also how the choice of 7. affects the performance of the latter. This plot makes it also evident how crucial the choice of the Lipsehitz constant is for STAGROPT and SAFROPT-like algorithms in general. In Table 1, in Appendix C, we report the average percentage of safety violations per run achieved by ISE and STAGEOPT. As expected, we see that the percentage of safety violations is comparable among all algorithms.
To show that for STAGEOPT exploration not only overestimating the Lipschitz constant, but also underestimating it can negatively impact performance.
consider the simple one-dimensional constraint function fxx = e + 0.05 and run the safe exploration for multiple values of the Lipschitz constant. This function gets increasingly away from the safety threshold for x -o0, while it asymptotically approaches the threshold for x -> oc, so that a good exploration algorithm would, ideally, quickly classify as safe the domain region for boundary of the safe set for x > 0. The results plotted in Figure 3b show how both a too high and a too low Lipschitz constant can lead sub-optimal exploration. In the case of too small constant, this is because STAGEOPT considers expanders almost all parameters in the domain, leading to additional evaluations in the region for x that are unlikely cause expansion of the safe set. On the other hand, a too high value of the Lipschitz constant can lead to the set of expanders to be empty as soon as the posterior mean gets close to the safety threshold for x > 0.
O and then keep exploring the
OpenAl Gym Control After investigating the performance of ISE under the hypothesis of the theory, we apply it two classic control tasks from the OpenAl Gym framework (Brockman et al., 2016), with the goal of finding the set of parameters of a controller that satisfy some safety constraint. In particular we consider linear controllers for the inverted pendulum and cart pole tasks.
For the inverted pendulum task, the linear controller is given by u, 0x + az0., where de is the control signal at time t, while 0, and é are, respectively, the angular position and the angular velocity
7

---

*8 of demain classifedd ao safe
1.225
% of domaiin clusaified as safe
18E
80
1,00
Stagetat. L+10
60
180 [Dur)))))
Stagelpt.
40
L L-
0.50
Heuriik:
Stagedgs
0.25
Bhagcops L-30
Stagelpa 10.0
0.00
Heartvi
200 300
lteration n
lteration
Comparison with STAGEOPT for GP samples.
(b) Comparison in ID exponential example.
Figure 3: (a) Average expansion of safe set over 100 swo-dimensional GP samples. The average percentage of the domain classified as safe is plotted as a function of n with its standard error. The lines for L = 0 and L 1 overlap. We can see that ISE obtains an higher sample efficiency than the best instance of STAGEOPT and a comparable one with the heuristic acquisition function proposed by Berkenkamp et al. (2016). The plot also shows that STAGEOPT performance is heavily affected by the choice L. In (b) the average percentage of the domain classified as safe in the one dimensional + 0.05 is plotted function of th eiteration n with its standard error, and shows the detrimental effect of over- and underestimating L.
example for fxx)
of the pendulum. Starting from position close to the upright equilibrium, the controller's task is the stabilization of the pendulum, subject safety constraint on the maximum velocity reached within one episode. For some given initial controller configuration On == (af, aς). we want to explore the controller's parameter space avoiding configurations that lead the pendulum swing with too high velocity. We apply ISE to explore the a-space with æo = ao and the safety constraint being the maximum angular velocity reached by the pendulum in an episode of fixed length. this case, the safety threshold is not at zero, but rather some finite value 0s and the safe parameters are those for which the maximum velocity below 0M. The formalism developed in the previous sections can be easily applied to this scenario if we consider f(a) = (max, (a - 0.y ). Figure 4a we show the true safe set for this problem, while in Figures 4b 4b-4d one can see how ISE safely explores the true safe set. These plots show how the ISE acquisition function (6) selects parameters that are close to the current safe set boundary and, hence, most informative about the safety of parameters outside of the safe set. This behavior eventually leads to the full true safe set to classified as safe by the GP posterior, as Figure 4d shows.
The cart pole task similar the inverted pendulum one, but the parameter space has three dimensions. The controller we consider is given by u, a,0, + az0, + azst, where 0g and 0, are, respectively. the angular position and angular velocity of the pole at time f, while & is the cart's s velocity. We set the initial state to zero angular and linear velocity and with the pole close to the vertical position, with the controller's 's goal being to keep the pole stable in the upright position. A combination of the three parameters at, Q2 and 0g considered safe if the angle of the pole does not exceed given threshold within the episode. Again, we can easily cast this safety constraint in terms of the formalism developed the paper: f(a) = - (max; (a)) - 0as), where Oas is the maximum allowed angle. Figure 5a shows the expansion of the cart pole space promoted by ISE, compared with STAGEOPT for different values of the Lipschitz constant. Both methods achieve a comparable sample efficiency and both lead to the classification as safe of the full true safe set.
High dimensional domains Many interesting applications have high dimensional parameter space. While SAFEOPT-like methods are difficult to apply already with dimension > 3 due to the discretization of the domain, ISE can perform well also in four or five dimensions. To see this, we
apply ISE to the constraint function (xx = + 2 (m===;) + 5E -===,,² 0.2. Figure 5b shows the ISE performance in dimension 5. We that ISE is able to promote the expansion of the safe set, leading to an increasingly bigger portion of the true safe set to be classified as safe.
Heteroskedastic noise domains For even higher dimensions, we can follow similar approach to LINEBO, limiting the optimization the acquisition function to randomly selected one-dimensional subspace of the domain. Moreover, as discussed in Section 5, it is also interesting to test ISE in the case of heteroskedasticc observation noise, since the noise a critical quantity for the ISE acquisition function, while it does not affect the selection criterion of STAGEOPT-like methods. Therefore, in

---

Utsafe set
Safe set
-2
02
03
-6
5
-2
[4)
(a) True safe
(b) Iteration
-2
-2
5
2
(3)
(c) Iteration 30
(d) Iteration
Figure 4: Safe exploration of the linear controller's parameter space in the inverted pendulum experiment. In (a) we see the true safe set, while in (b-d) we see the safe set (blue region) as identified by ISE various iterations. The point marked by the green dot is 0x++1 selected by (6), while the black cross is the imitial safe seed Cu.
100
% of trae safe set explored
% of Irue safe vet explored
80
40
(Qurs)
40
StageOpt L 0.1
StageOpt L 1.0
20
Stagedpt 5.0
0
100
200
300
400
0-0
100
200
300
400
Iteration
Iteration
(a) Exploration of the safe set for the cart pole task.
(b) Expansion of the safe set in dimension five.
Figure 5: In (a) we plot the percentage of the true safe set the cart pole task classified as safe by ISE and STAGEOPT, while in (b) ISE, for the safety constraint f used in the high dimensional experiments.
the expansion of the five-dimensional safe set promoted by
this experiment we combine a high dimensional problem with heteroskedastic noise. particular, we apply a LINEBO version of ISE to the constraint function f(x) = _ == + -a
3e (= 0.2 dimension nine and ten, with the safe seed being the origin. This function has two symmetric global optima at fæ² and we set two different noise levels in the two symmetric domain halves containing the optima. To assess the exploration performance, regret, defined as the difference between the current safe optimum and the true safe optimum. As the results Figure show, ISE achieve greater sample efficiency than the other STAGEOPT- methods. Namely, for a given number of iterations, by explicitly exploiting knowledge about the observation noise, ISE able to classify as safe regions the domain further away from the origin, in which the constraint function assumes its largest values, resulting in smaller regret. On the other hand, AAFEOPT-like methods only focus on the posterior variance, so that the higher observation noise causes them remain stuck in smaller neighborhood of the origin, resulting in bigger regret.
use the simple

---

2.0
2.0
Simple regret
Simple regret
1.5
I8E (0un)
ESE (Oun)
StageOps.
Bragodps.6-11 2-10
1.0
StageOps, L-100
StagnOps, 1-00
L1000
ttceatic
Himnet
0.5
400
600
800
200
400 Iteration n
600
Iteration
(a) Dimension 9.
(b) Dimension
Figure 6: Example of high dimensional exploration, for d -9 and d =10. After every ten iterations, we perform safe optimization with UCB acquisition within the current safe set and plot the simple regret T, with respect to the safe optimum. (a) and (a) show, respectively, the average regret over 70 runs in dimension nine and ten, as a function of the number of iterations. We can see that this adapted version of ISE promotes expansion of the safe set, leading to classifying the latent function attains its largest value. The plots also show that ISE achieves a better sample efficiency than both STAGEOPT-like exploration and the STAGEOPT inspired heuristic acquisition.
safe regions where
Conclusion and Societal Impact
We have introduced Information Theoretic Theoreiii Safe Exploration (ISE), novel approach to safely explore space in sequential decision task where the safety constraint is a priori unknown. ISE efficiently and safely explores by evaluating only parameters that are safe with high probability and by choosing those parameters that yield the greatest information gain about the safety of other parameters. We theoretically analyzed ISE and showed that leads arbitrary reduction of the uncertainty in the largest reachable safe set containing the starting parameter. Our experiments support these theoretical results and demonstrate an increased sample efficiency scalability of ISE compared to SAFEOPT-based approaches.
In many safety sensitive applications the shape of the safety constraints is unknown, so that an important prerequisite for any kind of process is to identify what parameters are safe to evaluate. By providing principled way to do this, the contributions of this paper allow to deal with safety in broad range of applications, which can favor the usage of ML approaches also in safety sensitive settings. On the other hand, misuse of the proposed method cannot be prevented in general.
10

---

References
Felix Berkenkamp, Angela P. Schoellig, and Andreas Krause. Safe controller optimization for quadrotors with gaussian processes. In IEEE International Conference Robotics and Automation (ICRA), 2016.
Felix Berkenkamp, Andreas Krause, and Angela P Schoellig. Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics. Machine Learning Journal, Special Issue Robust Machine Learning, 2021.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
Sébastien Bubeck and Nicolò Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi- armed bandit problems. Foundations and Trends@ in Machine Learning, 5(1):1-112, 2012.
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning (ICML), 2017.
Emile Contal, Vianney Perchet, and Nicolas Vayatis. Gaussian process optimization with mutual information. In International Conference on International Conference on Machine Learning
(ICML), 2014.
Rikky R.P.R. Duivenvoorden, Felix Berkenkamp, Nicolas Carion, Andreas Krause, and Angela P. Schoellig. Constrained Bayesian optimization with particle swarms for safe adaptive controller tuning. In Proc. of the IFAC (International Federation of Automatic Control) World Congress, pages 12306 12313, 2017.
Gabriel Dulac-Amold, Da Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. In International Conference on International Conference on Machine Learning (ICML), 2019.
Christian Fiedler, Carsten W. Scherer, and Sebastian Trimpe. Practical and rigorous uncertainty bounds for gaussian process regression. In AAAI Conference on Artificial Intelligence, 2021.
Lukas Fröhlich, Edgar Klenske, Julia Vinogradska, Christian Daniel, and Melanie Zeilinger. Noisy- input entropy search for efficient robust bayesian optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020.
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2022. in preparation.
Michael A. Gelbart, Jasper Snoek, and Ryan P. Adams. Bayesian optimization with unknown constraints. In Conference on Uncertainty in Artificial Intelligence (UAI), 2014.
Alkis Gotovos, Nathalic Casati, Gregory Hitz, and Andreas Krause. Active learning for level set estimation. In International Joint Conference on Artificial Intelligence (IJCAI). 2013.
Alexander Hans, Daniel Schneegass, Anton Schäfer, and Steffen Udluft. Safe exploration for
reinforcement learning. In European Symposium on Artificial Neural Networks (ESANN), 2008.
Philipp Hennig and Christian J. Schuler. Entropy search for information-efficcent global optimization. Journal of Machine Learning Research, 13:1809-1837, 2012.
José Miguel Genrández-Lobatte Matthew W. Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In International Conference Neural Information Processing Systems (NIPS), 2014.
Johannes Kirschner, Mojmir Mutný, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe bayesian optimization in high dimensions via onc-dimensional subspaces. In International Conference for Machine Learning (ICML), 2019.
Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision processes. In International Coference on International Conference Machine Learning (ICML), 2012.
11

---

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. MIT Press, Cambridge, MA, USA, 2006.
Mark Schillinger, Benjamin Hartmann, Patric Skalecki, Mona Meister, Duy Nguyen-Tuong, and Oliver Nelles. Safe active learning and safe bayesian optimization for tuning a pi-controller. Papersspprs,,,,nee, 50(1):5967-597) 2017.
Bernhard Schölkopf and Alexander J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT Press, Cambridge, MA, USA, 2002.
Jens Schreiter, Duy Nguyen-Tuong, Mona Eberts, Bastian Bischoff, Heiner Markert, and Marc Toussaint. Safe exploration for active learning with gaussian processes. In European Conference on Machine Learning and Knowledge Discovery in Databases (ECMLPKDD), 2015.
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan Adams, and Nando de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104: 148-175 175, 2016.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimiza- tion in the bandit setting: No regret and experimental design. In International Conference Machine Learning (ICML), 2010.
Yanan Sui, Alkis Gotovos, Joel W. Burdick, and Andreas Krause. Safe exploration for optimization with Gaussian processes. In International Conference Machine Learning (ICML), 2015.
Yanan Sui, Vincent Zhuang, Joel Burdick, and Yisong Yue. Stagewise safe Bayesian optimization with Gaussian processes. In International Conference on Machine Learning (ICML), 2018.
Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration in finite markov decision processes with gaussian processes. Advances Neural Information Processing Systems (NeurIPS), 2016.
Mattco Turchetta, Felix Berkenkamp, and Andreas Krause. Safe exploration for interactive machine learning. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient bayesian optimization. In International Conference Machine Learning (ICMI.), 2017.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations are discussed in Section 5
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] The code is available https://githul: can/boschressaachh inf information-thee whwoonnnthrrriic--o safe- eafe-explooation Experiment details to reproduce results can be found in Appendix C.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C.
(c) Did report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] They are in the respective plots.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix C.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 6 and Appendix C
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either in the supplemental material or as a URL? [No] No new assets have been used nor introduced.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hyc6HASwLIe", "reviewer": "Reviewer_Se3u", "summary": " The paper focuses on an important stage of GP safe exploration algorithms: safely expanding the size of the set of parameters believed to be safe, in a data-efficient manner.\nTo do this, the authors propose an information-theoretic approach inspired by other non-safety-constrained Bayesian Optimization (BO) literature.\nCompared to standard SafeOpt (Sui et al. \"Safe exploration for optimization with Gaussian processes.\" PMLR15), the method is shown to be more efficient at expanding the safe set.\n", "questions": "why didn't the experimental section analyse the safety spec? \n\nIn figure 2 (c), the left-most green evaluation cross is clearly under the orange safety bound line, which suggests that the agent has broken the safety specification in this run. Is this a formatting error or representative of an actual failed run?\n\nPlease clarify the point about Theorem 1 above.", "limitations": "Limitations on scalability are mentioned in text but not analysed empirically. Negative societal impacts are addressed with a boilerplate sentence in he end of the paper, bit unsatisfactory; probably better just not saying anything.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe originality and contribution the paper makes is good:\n\n* It adapts mutual-information-based methods to the safe set expansion setting, which no other work (that I was able to find) does. Usually information theoretic measures for BO simplify to just sampling at the parameters with maximum variance (Schillinger et al., 2017) or highest UCB, but in this case it is more interesting and the authors are able to come up with a function to optimise to find the maximum information gain about expanding the safe set.\n\n* Their acquisition function simplification to make optimization tractable seems justified and well explained in the appendix.\nThis is not the only work to avoid the inclusion of a Lipschitz constant hyperparameter (Berkenkamp et al. \"Safe controller optimization for quadrotors with Gaussian processes.\", ICRA16) (they use hypothesised GP observations for safe set expansion) or to avoid SafeOpt-style discretisation (Schillinger et al., 2017) (this is another method that works in continuous parameter space). However, these two problems are solved here in an elegant way.\nIn particular, it avoids the basic heuristic/arbitrariness of the expander state definition in SafeOpt.\n\n* I believe the authors have covered all the relevant related work – although not sure the paper experimentally compares to enough related work (see below).\n\n* I believe the clarity of the paper is generally good until the experiments section.\n\n\nWeaknesses:\n\n* The acquisition function simplification should be more explained in the main body of the paper. Furthermore, it would be good to have further discussion on the approximation on the rest of the paper, e.g. Theorem 1.\n\n* The evaluation with GP samples, shown in Figure 3, is confusing. The only alternative method comparison is standard StageOpt with differing Lipschitz (L) parameter values. The justification for the choice of plotted L-values isn’t clear, and furthermore there is a clear trend of decreasing L leading to better performance. From this graph, it would suggest that reducing L further <1 might result in it outperforming ISE.\nFor those with background knowledge, it is clear why it’s not possible to keep reducing L. Eventually, reducing L enough will result in StageOpt classifying unsafe states as safe and then failing the safety specification. However this section contains no discussion of whether/how any algorithms fail the safety specification, and only shows the first few % of safe set exploration. Evaluation should include plots on how many time s the safety spec is broken, as that is a crucial part of the problem\n\n* I think evaluation might have benefited comparing to SafeOpt variants that do not require the Lipschitz constant, such as (Berkenkamp et al. \"Safe controller optimization for quadrotors with Gaussian processes.\", ICRA16). That would better differentiate which performance improvements are from more intelligent expansion behaviour, vs tuning of the Lipschitz constant.\nFigure 6 does not have enough baseline comparison. Only one StageOpt plot is shown in 6 (a) and it is not clear which Lipschitz value was used for this. There is no StageOpt plot in 6 (b) and it is unclear why. It's also unclear why the plot only shows the first 5% of exploration.\n\n* There should be some experiments on scalability, e.g. it would be nice to see a scalability plot on number of dimensions.\n\n\n\nOther  points:\n\nThe last paragraph of related work should also mention Turchetta, Matteo, Felix Berkenkamp, and Andreas Krause. \"Safe exploration in finite markov decision processes with gaussian processes.\" Advances in Neural Information Processing Systems 29 (2016).\n\n“This process is made more efficient by SafeOpt, which restricts exploration to parameters that are close to the boundary of the current set of safe parameters (Sui et al., 2015)”: I think it is more accurate to say that StageOpt does this restriction (in its expansion phase). SafeOpt can also choose to sample potential maximisers in M_t, not just from the expander set G_t.\n\nClarity-wise, I don’t fully understand how Theorem 1 justifies the sentence “in practice it will first focus on reducing the uncertainty in areas of the safe set that are most informative about parameters whose classification is still uncertain (e.g. areas close to the boundary of the safe set), and only eventually turns to learning about the inside of the safe set”. If it doesn’t justify that sentence, the theorem doesn’t seem to actually prove anything about safe set expansion.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dMkd5NWVHhZ", "reviewer": "Reviewer_HY4V", "summary": "This paper provides a newer approach to explore and uncover the safe set of parameters. The exploration is principled and is guaranteed to be safe with high probability. The developments in the theoretical section provide closed-form approximations for the information-gain relevant quantities. The experiments show that their approach can provide the maximally safe set faster than existing methods. The novelty with respect to prior work is that the information gain criteria is used for bayesian optimization with an emphasis on safety. As an illustration of the algorithm, the GP prior is known beforehand with at least one point in the safe set. The next point is chosen to be in the region that is known to be safe such that decreasing the variance around this region will provide maximal information about the safety of any point beyond the safe set. Rather than the Lipschitz constant, the use of a smoothness prior does seem interesting and novel.", "questions": "From a theory perspective, [A] proves a finite-time regret bound while this paper only proves an asymptotic guarantee. The first type of bounds are stronger and provide more information about the rate of convergence. In practice, the method actually has an advantage but this is lost in the theory. What is proven here is that once we have found the maximal safe set, the posterior variances of the points inside the safe set will eventually become small. The exploration for uncovering the safe set is optimal at least assuming the prior is correct. Is it relevant to prove the number of samples it would take to uncover the safe set or let us say the regret wrt an algorithm that knows the optimal prior?\n\nThe comparison to previous related work is at a very high-level. It would help to provide a deeper theoretical comparison to the 2-3 most related papers. For example, can you place the theorem being proven in the context of the literature in this topic?\n\n[A] Max-value Entropy Search for Efficient Bayesian Optimization - Wang and Jegelka", "limitations": "There is a great discussion on limitations of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I believe this is overall a good paper as it is a principled approach to Bayesian optimization and safety. It is improving the Lipschitz constant based methods for sampling-based safety verification.\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JMqU0JJ10r5", "reviewer": "Reviewer_ZWAM", "summary": "Optimization of an unknown function under unknown constraints is a problem which has been often addressed based on Gaussian process surrogate models in the past. Due to the unknown constraints, existing approaches typically start with a given set of safe parameters, which is subsequently expanded. In order to ensure the safety of the approach, this expansion must not lead to constraint violations. State-of-the-art methods determine evaluation points for the expansion of the safe set by finding the most uncertain point in discretized sets. However, this leads to additional hyperparameters and tends to scale poorly to high dimensional input domains. This paper aims to avoid the issues resulting from discretization by employing the information gain of the binary random variable describing the constraint violation at a given input point. Since this information gain cannot be computed exactly in practice, an efficient approximation is proposed. It is shown that determining evaluation points using the proposed criterion is guaranteed to achieve any upper bounds on the posterior GP variance in a finite number of evaluation points if the safe set is not expanded anymore. The practical strengths of the method are demonstrated in simulations with GP sample functions and control examples. In particular in high dimensional problems, a significantly better exploration of the true safe region can be observed.", "questions": "Apart from my comments mentioned above, I only have minor suggestions/questions:\n- Highlighting ISE using a bold line in Figs. 3 and 6 might be a good idea\n- How is beta chosen in the simulations?\n- Line 47: in $\\rightarrow$ on\n- Lines 231-232: exploration exploration\n- Lines 262-263: the true safe for this problem\n- The y axis label of Fig. 6 seems wrong. I think it should be the found optimum.\n- Line 280: I think it should be referred to Fig. 5b instead of 6b here.\n", "limitations": "I think the limitations of the method and potential societal impact are generally well discussed. While the choice of the kernel function is highlighted as a potential challenge, it is not sufficiently discussed that the RKHS bound also needs to be known for the theory to hold. This bound is generally difficult to obtain in practice, which crucially limits practical safety guarantees. In my opinion, this should be discussed in the limitations section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is on an interesting and relevant topic. The proposed ideas are novel to the best of my knowledge and nicely relate to entropy search and similar approaches in unconstrained Bayesian optimization. The paper combines theoretical analysis and demonstration of practical advantages well. In general, the paper is well-structured and the presentation of the method is excellent in my opinion. Merely in the supplemental material I have noticed that everything is proven for the approximation proposed by the authors, while it is stated that the analysis is performed for the mutual information itself. It took me some time to realize this discrepancy, such that I would recommend to clarify this. The derivation of the theoretical results looks quite straightforward, but the result is certainly interesting and fits well into the overall paper. However, it does not become clear if the scenario analyzed in Theorem 1 does actually ever occur. The reason for this is that it requires that the safe set needs to stop growing at some point in time, which could potentially never happen if the increase of the size of the safe set becomes smaller and smaller every iteration, but never becomes zero. This case is also not discussed in the paper, such that the significance of the theoretical results strongly diminishes in my opinion. I did not check the theoretical results in detail, but they seem to be correct to the best of my knowledge. The simulations are properly executed with sufficiently many random seeds including functions satisfying the assumptions of the approach and functions for which this is not clear. The effect of the dimensionality on the performance of the different approaches is illustrated nicely. I merely have a small technical comment regarding the statement in section 6 that the ISE approach is evaluated on samples from a GP because this would allow to test ISE under the assumptions of the theory. While this is true for the exploration, the usage of GP sample function poses the theoretical challenge that samples from a GP are well-known to almost surely have an unbounded RKHS norm. Therefore, the definition of the safe set based on (Chowdhary and Gopalan, 2017) does not work anymore because it requires a bound for the RKHS norm.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept