Title: The Behavior and Convergence of Local Bayesian Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: Bayesian optimization

convergence rates
Abstract: A recent development in Bayesian optimization is the use of local optimization strategies, which can deliver strong empirical performance on high-dimensional problems compared to traditional global strategies. The "folk wisdom" in the literature is that the focus on local optimization sidesteps the curse of dimensionality; however, little is known concretely about the expected behavior or convergence of Bayesian local optimization routines. We first study the behavior of the local approach, and find that the statistics of individual local solutions of Gaussian process sample paths are surprisingly good compared to what we would expect to recover from global methods. We then present the first rigorous analysis of such a Bayesian local optimization algorithm recently proposed by Müller et al. (2021), and derive convergence rates in both the noisy and noiseless settings.

The Behavior and Convergence of Local Bayesian Optimization

Abstract

A recent development in Bayesian optimization is the use of local optimization strategies, which can deliver strong empirical performance on high-dimensional problems compared to traditional global strategies. The 'folk wisdom' in the literature is that the focus on local optimization sidesteps the curse of dimensionality; however, little is known concretely about the expected behavior or convergence of Bayesian local optimization routines. We first study the behavior of the local approach, and find that the statistics of individual local solutions of Gaussian process sample paths are surprisingly good compared to what we would expect to recover from global methods. We then present the first rigorous analysis of such a Bayesian local optimization algorithm recently proposed by Müller et al. (2021), and derive convergence rates in both the noiseless and noisy settings.

1 Introduction

Bayesian optimization (BO) [11] is among the most well studied and successful algorithms for solving black-box optimization problems, with wide ranging applications from hyperparameter tuning [23], reinforcement learning [19, 27], to recent applications in small molecule [18, 26, 7, 13, 12] and sequence design [25]. Despite its success, the curse of dimensionality has been a long-standing challenge for the framework. Cumulative regret for BO scales exponentially with the dimensionality of the search space, unless strong assumptions like additive structure [15] are met. Empirically, BO has historically struggled on optimization problems with more than a handful of dimensions.

Recently, local approaches to Bayesian optimization have empirically shown great promise in addressing high-dimensional optimization problems [10, 19, 20, 27, 17]. The approach is intuitively appealing: a typical claim is that a local optimum can be found relatively quickly, and exponential sample complexity is only necessary if one wishes to enumerate all local optima. However, these intuitions have not been fully formalized. While global Bayesian optimization is well studied under controlled assumptions [24, 3], little is known about the behavior or convergence properties of local Bayesian optimization in these same settings - the related literature focuses mostly on strong empirical results on real-world applications.

In this paper, we begin to close this gap by investigating the behavior of and proving convergence rates for local BO under the same well-studied assumptions commonly used to analyze global BO. We divide our study of the properties of local BO in to two questions:

Q1 Using common assumptions ( e.g ., when the objective function is a sample path from a GP with known hyperparameters), how good are the local solutions that local BO converges to?

Kyurae Kim

Jacob R. Gardner

1

2

3

4

5

6

Input dimension (

d

)

-15

-10

-5

0

Min value found

GIBO (

σ

= 0

)

1

2

3

4

5

6

d

-15

-10

-5

0

GIBO (

σ

= 0

.

05

)

1

2

3

4

5

6

d

-15

-10

-5

0

GIBO (

σ

= 0

.

2

)

0

1

2

3

4

5

d

0

10

10

10

20

10

30

Local Opt. Effective Grid Size

1

2

3

4

5

6

d

-15

-10

-5

0

Min value found

GP-UCB (

σ

= 0

)

1

2

3

4

5

6

d

-15

-10

-5

0

Random Search (

σ

= 0

)

1

5

10

20

30

50

1

5

10

20

30

50

1

5

10

20

30

50

1

5

10

20

30

50

1

5

10

20

30

50

1

5

10

20

30

50

dimension

d

dimension

d

dimension

d

Q2 Under these same assumptions, is there a local Bayesian optimization procedure that is guaranteed to find a local solution, and how quickly can it do so?

For Q1 , we find empirically that the behavior of local BO on 'typical' high dimensional Gaussian process sample paths is surprising: even a single run of local BO, without random restarts, results in shockingly good objective values in high dimension. Here, we may characterize 'surprise' by the minimum equivalent grid search size necessary to achieve this value in expectation. See Figure 1 and more discussions in §3.

For Q2 , we present the first theoretical convergence analysis of a local BO algorithm, GIBO as presented in [19] with slight modifications. Our results bring in to focus a number of natural questions about local Bayesian optimization, such as the impact of noise and the increasing challenges of optimization as local BO approaches a stationary point.

We summarize our contributions as follows:

We provide upper bounds on the uncertainty of the gradient when actively learned using the GIBO procedure described in [19] in both the noiseless and noisy settings.

By relating these uncertainty bounds to the bias in the gradient estimate used at each step, we translate these results into convergence rates for both noiseless and noisy functions.

We investigate the looseness in our bound by studying the empirical reduction in uncertainty achieved by using the GIBO policy and comparing it to our upper bound.

We empirically study the quality of individual local solutions found by a local Bayesian optimization, a slightly modified version of GIBO , on GP sample paths.

2 Background and Related Work

Bayesian optimization. Bayesian optimization (BO) considers minimizing a black-box function f :

through potentially noisy function queries y = f ( x ) + ε . When relevant to our discussion, we will assume an i.i.d . additive Gaussian noise model: ε ∼ N (0 , σ 2 ) . To decide which inputs to

be evaluated, Bayesian optimization estimates f via a surrogate model, which in turn informs a policy (usually realized by maximizing an acquisition function α ( x ) over the input space) to identify evaluation candidates. Selected candidates are then evaluated, and the surrogate model is updated with this new data, allowing the process to repeat with the updated surrogate model.

Gaussian processes. Gaussian processes (GP) are the most commonly used surrogate model class in BO. A Gaussian process GP ( µ, k ) is fully specified by a mean function µ and a covariance function k . For a finite collection of inputs X , the GP induces a joint Gaussian belief about the function values, f ( X ) ∼ N ( µ ( X ) , k ( X , X )) . Standard rules about conditioning Gaussians can then be used to condition the GP on a dataset D , resulting in an updated posterior process reflecting the information contained in these observations:

Existing bounds for global BO. Srinivas et al. [24] proved the first sublinear cumulative regret bounds of a global BO algorithm, GP-UCB , in the noisy setting. Their bounds have an exponential dependency on the dimension, a result that is generally unimproved without assumptions on the function structure like an additive decomposition [15]. This exponential dependence can be regarded as a consequence of the curse of dimensionality. In one dimension, Scarlett [21] has characterized the optimal regret bound (up to a logarithmic factor), and GP-UCB is near-optimal in the case of the RBF kernel. In the noiseless setting, improved convergence rates have been developed under additional assumptions. For example, De Freitas et al. [6] proved exponential convergence by assuming f is locally quadratic in a neighborhood around the global optimum, and Kawaguchi et al. [16] proved an exponential simple regret bound under an additional assumption that the optimality gap f ( x ) -f ∗ is bounded by a semi-metric.

Gaussian process derivatives. A key fact about Gaussian process models that we will use is that they naturally give rise to gradient estimation. If f is Gaussian process distributed, f ∼ N ( µ, k ) , and k is differentiable, then the gradient process ∇ f also is also (jointly) distributed as a GP. Noisy observations y at arbitrary locations X and the gradient measured at an arbitrary location x are jointly distributed as:

This property allows probabilistic modeling of the gradient given noisy function observations. The gradient process conditioned on observations D is distributed as a Gaussian process:

3 How Good Are Local Solutions?

Several local Bayesian optimization algorithms have been proposed recently as an alternative to global Bayesian optimization due to their favourable empirical sample efficiency [10, 19, 20]. The common idea is to utilize the surrogate model to iteratively search for local improvements around some current input x t at iteration t . For example, Eriksson et al. [10] centers a hyper-rectangular trust region on x t (typically taken to be the best input evaluated so far), and searches locally within the trust region. Other methods like those of Müller et al. [19] or Nguyen et al. [20] improve the solution x t locally by attempting to identify descent directions d t so that x t can be updated as x t +1 = x t + η t d t .

Before choosing a particular local BO method to study the convergence of formally, we begin by motivating the high level philosophy of local BO by investigating the quality of individual local solutions in a controlled setting. Specifically, we study the local solutions of functions f drawn from Gaussian processes with known hyperparameters. In this setting, Gaussian process sample paths can be drawn as differentiable functions adapting the techniques described in Wilson et al. [29]. Note that this is substantially different from the experiment originally conducted in Müller et al. [19], where they condition a Gaussian process on 1000 examples and use the posterior mean as an objective.

To get a sense of scale for the optimum, we first analyze roughly how well one might expect a simple grid search baseline to do with varying sample budgets. A well known fact about the order statistics of Gaussian random variables is the following:

Algorithm 1: A Local Bayesian Optimization Algorithm

Input: A black-box function f over a convex compact set X . for t = 1 , 2 , · · · , T do X = argmin Z α trace ( x t , Z ) where Z ∈ R b t × d // b t is the batch size evaluate the black-box function f on X , obtaining (possibly noisy) measurements y D t = D t -1 ∪ ( X , y ) // add ( X , y ) to the training data x t +1 = x t -η t ∇ µ D t ( x t ) end

Remark 1 ( e.g ., [8, 14]) . Let X 1 , X 2 , · · · , X n be (possibly correlated) Gaussian random variables with marginal variance s 2 , and let Y = max { X i } . Then the expected maximum is bounded by

This result directly implies an upper bound on the expected maximum (or minimum by symmetry) of a grid search with n samples. The logarithmic bound results in an optimistic estimate of the expected maximum (and the minimum): as n →∞ the bound goes to infinity as well, whereas the maximum (and the minimum) of a GP is almost surely finite due to the correlation in the covariance.

With this, we now turn to evaluating the quality of local solutions. To do this, we optimize 50 sample paths starting from x = 0 from a centered Gaussian process with a RBF kernel (unit outputscale and unit lengthscale) using a variety of observation noise standard deviations σ . We then run iterations of local Bayesian optimization (as described later in Algorithm 1) to convergence or until an evaluation budget of 5000 is reached. In the noiseless setting ( σ = 0 ), we modify Algorithm 1 to pass our gradient estimates to BFGS rather than applying the standard gradient update rule for efficiency.

We repeat this procedure for d ∈ { 1 , 5 , 10 , 20 , 30 , 50 } dimensions and σ ∈ { 0 , 0 . 05 , 0 . 2 } and display results in Figure 1. For each dimension, we plot the distribution of single local solutions found from each of the 50 trials as a box plot. In the noiseless setting, we find that by d = 50 , a single run of local optimization ( i.e ., without random restarts) is able to find a median objective value of -12 . 9 , corresponding to a grid size of at least n = 10 36 to achieve this value in expectation!

For completeness, we provide results of running GP-UCB and random search also with budgets of 5000 and confirm the effectiveness of local optimization in high dimensions.

4 A Local Bayesian Optimization Algorithm

The study above suggests that, under assumptions commonly used to theoretically study the performance of Bayesian optimization, finding even a single local solution is surprisingly effective. The natural next question is whether and how fast we can find them. To understand the convergence of local BO, we describe a prototypical local Bayesian optimization algorithm that we will analyze in Algorithm 1, which is nearly identical to GIBO as described in [19] with one exception: Algorithm 1 allows varying batch sizes b t in different iterations.

Define k D∪ Z ( x t , x t ) = k D ( x t , x t ) -k D ( x t , Z )( k D ( Z , Z ) + σ 2 I ) -1 k D ( Z , x t ) . Namely, k D∪ Z ( · , · ) is the posterior covariance function conditioned on the data D as well as new inputs Z . Crucially, observe that the posterior covariance of a GP does not depends on the labels y , hence the compact notation k D∪ Z . The acquisition function α trace of GIBO is defined as

which is the trace of the posterior gradient covariance conditioned on the union of the training data D and candidate designs Z .

The posterior gradient covariance ∇ k D ( x t , x t ) ∇ ⊤ quantifies the uncertainty in the gradient ∇ f ( x t ) . By minimizing the acquisition function α trace ( x t , Z ) over Z , we actively search for designs Z that minimizes the one-step look-ahead uncertainty, where the uncertainty is measured by the trace of the posterior covariance matrix. After selecting a set of locations Z , the algorithm queries the (noisy) function values at Z , and updates the dataset. The algorithm then follows the (negative) expected gradient ∇ µ D t ( x t ) to improve the current iterate x t , and the process repeats.

5 Convergence Results

In light of the strong motivation we now have for local Bayesian optimization, we will now establish convergence guarantees for the simple local Bayesian optimization routine outlined in the previous section, in both the noiseless and noisy setting. We first pause to establish some notations and a few assumptions that will be used throughout our results.

Notation. Let X ⊂ R d be a convex compact domain ( e.g ., X = [0 , 1] d ). Let H be a reproducing kernel Hilbert space (RKHS) on X equipped with a reproducing kernel k ( · , · ) . We use ∥·∥ to denote the Euclidean norm and ∥·∥ H to denote the RKHS norm. The minimum of a continuous function f defined on a compact domain X is denoted as f ∗ = min x ∈X f ( x ) .

In our convergence results, we make the following assumption about the kernel k . Many commonly used kernels satisfy this assumption, e.g ., the RBF kernel and Matérn kernel with ν > 2 .

Assumption 1. The kernel k is stationary, four times continuously differentiable and positive definite.

The following is a technical assumption that simplifies proofs and convergence rates.

Assumption 2. The iterates { x t } ∞ t =1 stay in the interior of the domain X .

Intuitively, with the domain X large enough and an appropriate initialization, the iterates most likely always stay in the interior. In the case when the iterate x t +1 slides out of the domain X , we can add a projection operator to project it back to X , which will be discussed in §5.2.

We also define a particular (standard) smoothness measure that will be useful for proving convergence.

Definition 1 (Smoothness) . A function f is L -smooth if and only if for all x 1 , x 2 ∈ X , we have:

Next, we define the notion of an error function , which quantifies the maximum achievable reduction in uncertainty about the gradient at x = 0 using a set of b points Z and no other data, i.e ., the data D is an empty set in the acquisition function (2).

Definition 2 (Error function) . Given input dimensionality d , kernel k and noise standard deviation σ , we define the following error function :

5.1 Convergence in the Noiseless Setting

We first prove the convergence of Algorithm 1 with noiseless observations, i.e ., σ = 0 . Where necessary, the inverse kernel matrix k D ( Z , Z ) -1 should be interpreted as the pseudoinverse. In this section, we will assume the ground truth function f is a function in the RKHS with bounded norm. This assumption is standard in the literature [ e.g ., 3, 24], and the results presented here ultimately extend trivially to the other assumption commonly made (that f is a GP sample path).

Assumption 3. The ground truth function f is in H with bounded norm ∥ f ∥ H ≤ B .

Because k is four times continuously differentiable, f is twice continuously differentiable. Furthermore, on a compact domain X , f is guaranteed to be L -smooth for some constant L (see Lemma 9 for a complete explanation). To prove the convergence of Algorithm 1, we will show the posterior mean gradient ∇ µ D approximates the ground truth gradient ∇ f . By using techniques in meshless data approximation [ e.g ., 28, 5], we prove the following error bound, which precisely relates the gradient approximation error and the posterior covariance trace.

Lemma 1. For any f ∈ H , any x ∈ X and any D , we have the following inequality

Since f has bounded norm ∥ f ∥ H ≤ B , the right hand side of (4) is a multiple of the posterior covariance trace, which resembles the acquisition function (2). Indeed, the acquisition function (2) can be interpreted as minimizing the one-step look-ahead worst-case gradient estimation error in the RKHS H , which justifies GIBO from a different perspective.

Next, we provide a characterization of the error function E d,k, 0 under the noiseless assumption σ = 0 . This characterization will be useful later to express the convergence rate.

Lemma 2. For σ = 0 , the error function is bounded by E d,k, 0 ( b ) ≤ C max { 0 , 1 + d -b } , where C = max 1 ≤ i ≤ d ∂ 2 ∂x i ∂x ′ i k ( 0 , 0 ) is the maximum of the Hessian's diagonal entries at the origin.

Now we are ready to present the convergence rate of Algorithm 1 under noiseless observations.

Theorem 1. Let f ∈ H whose smoothness constant is L . Running Algorithm 1 with constant batch size b t = b and step size η t = 1 L for T iterations outputs a sequence satisfying

As T →∞ , the first term in the right hand side of (5) decays to zero, but the second term may not. Thus, with constant batch size b , Algorithm 1 converges to a region where the squared gradient norm is upper bounded by B 2 E d,k, 0 ( b ) with convergence speed O ( 1 T ) . An important special case occurs when the batch size b t = d +1 in every iteration. In this case, E d,k, 0 ( b ) = 0 by Lemma 2 and the algorithm converges to a stationary point with rate O ( 1 T ) .

Corollary 1. Under the same assumptions of Theorem 1, using batch size b t = d +1 , we have

Therefore, the total number of samples n = O ( dT ) and the squared gradient norm ∥∇ f ( x t ) ∥ 2 converges to zero at the rate O ( d/n ) .

We highlight that the rate is linear with respect to the input dimension d . Thus, as expected, local Bayesian optimization finds a stationary point in the noiseless setting with significantly better nonasymptotic rates in d than global Bayesian optimization finds the global optimum. Of course, the fast convergence rate of Algorithm 1 is at the notable cost of not finding the global minimum. However, as Figure 1 demonstrates and as discussed in §3, a single stationary point may already do very well under the assumptions used in both analyses.

We note that the results in this section can be extended to the GP sample path assumption as well. Due to space constraint, we defer the result to Theorem 4 in the appendix.

5.2 Convergence in the Noisy Setting

We now turn our attention to the more challenging and interesting setting where the observation noise σ > 0 . We analyze convergence under the following GP sample path assumption, which is commonly used in the literature [ e.g ., 24, 6]:

Assumption 4. The objective function f is a sample from a GP, f ∼ GP ( µ, k ) with known mean function, covariance function, and hyperparameters. Observations of the objective function y are made with added Gaussian noise of known variance, y = f ( x ) + ε , where ε ∼ N (0 , σ 2 ) .

Since the kernel k is four times continuously differentiable, f is almost surely smooth.

Lemma 3. For 0 < δ < 1 , there exists a constant L > 0 such that f is L -smooth w.p. at least 1 -δ .

Challenges. The observation noise introduces two important challenges. First, by the GP sample path assumption, we have:

Thus, the posterior mean gradient ∇ µ D t ( x t ) is an approximation of the ground truth gradient ∇ f ( x t ) , where the approximation error is quantified by the posterior covariance ∇ k D t ( x t , x t ) ∇ ⊤ . For any f , we emphasize that the posterior mean gradient ∇ µ D t ( x t ) is therefore biased whenever the posterior covariance is nonzero. Unfortunately, because of noise σ > 0 , this is always true for any finite data D t . Thus, the standard analysis of stochastic gradient descent does not apply, as it typically assumes the stochastic gradient is unbiased .

The second challenge is that the noise directly makes numerical gradient estimation difficult. To build intuition, consider a finite differencing rule that approximates the partial derivative

where e i is the i -th standard unit vector. In order to reduce the O ( h 2 ) gradient estimation error, we need h → 0 . However, the same estimator under the noisy setting

where ϵ 1 , ϵ 2 are i.i.d . Gaussians, diverges to infinity when h → 0 . Note that the above estimator is biased even with repeated function evaluations because h cannot go to zero.

We first present a general convergence rate for Algorithm 1 that bounds the gradient norm.

Theorem 2. For 0 < δ < 1 , suppose f is a GP sample whose smoothness constant is L w.p. at least 1 -δ . Algorithm 1 with batch size b t and step size η t = 1 L produces a sequence satisfying

with probability at least 1 -2 δ , where C t = 2log ( ( π 2 / 6)( t 2 /δ ) ) .

The second term 1 T ∑ T t =1 C t E d,k,σ ( b t ) in the right hand side of (6) is the average cumulative bias of the gradient. To finish the convergence analysis, we must further bound the error function E d,k,σ ( b t ) . For the RBF kernel, we obtain the following bound:

Lemma 4 (RBF Kernel) . Let k ( x 1 , x 2 ) = exp ( -1 2 ∥ x 1 -x 2 ∥ 2 ) be the RBF kernel. We have

where m ∈ N and W denotes the principal branch of the Lambert W function.

The error function (3) is an infimum over all possible designs, which is intractable for analysis. Instead, we analyze the infimum over a subset of designs of particular patterns (based on finite differencing), which can be solved analytically, resulting the first inequality. The second equality is proved by further bounding the Lambert function by its Taylor expansion at -1 /e .

In addition, we obtain a similar bound for the Matérn kernel with ν = 5 2 by a slightly different proof.

Lemma 5 (Matern Kernel) . Let k ( · , · ) be the ν = 2 . 5 Matérn kernel. Then, we have

Interestingly, Lemma 4 and Lemma 5 end up with the same asymptotic rate. Writing the bound in terms of the batch size b , we can see that E d,k,σ ( b ) = O ( σd 3 2 b -1 2 ) for both the RBF kernel and the ν = 2 . 5 Matérn kernel. 1 Coupled with Theorem 2, the above lemmas translate into the following convergence rates, depending on the batch size b t in each iteration:

Corollary 2. Let k ( · , · ) be either the RBF kernel or the ν = 2 . 5 Matérn kernel. Under the same conditions as Theorem 2, if

with probability at least 1 -2 δ . Here, T is the total number of iterations and n is the total number of samples queried.

With nearly constant batch size b t = d log 2 t , Algorithm 1 converges to a region where the squared gradient norm is O ( σd ) . With linearly increasing batch size, the algorithm converges to a stationary point with rate O ( σd 1 . 25 n -0 . 25 log n ) , significantly slower than the O ( d/n ) rate in the noiseless setting. The quadratic batch size is nearly optimal up to a logarithm factor - increasing the batch size further slows down the rate (see Appendix D for details).

To achieve convergence to a stationary point using Corollary 2, the batch size b t must increase as optimization progresses. We note this may not be an artifact of our theory or our specific realization of local BO, but rather a general fact that is likely to be implicitly true for any local BO routine. For example, a practical implementation of Algorithm 1 might use a constant batch size and a line search subroutine where the iterate x t is updated only when the (noisy) objective value decreases otherwise, the iterate x t does not change and the algorithm queries more candidates to reduce the bias in the gradient estimate. With this implementation, the batch size is increased repeatedly on any iterate while the line search condition fails. As the algorithm converges towards a stationary point, the norm of the ground-truth gradient ∥∇ f ( x t ) ∥ decreases and thus requires more accurate gradient estimates. Therefore, the line search condition may fail more frequently with the constant batch size, and the effective batch size increases implicitly.

We provide two additional remarks on convergence in the noisy setting. First, the convergence rate is significantly slower than in the noiseless setting, highlighting the difficulty presented by noise. Second, when the noise is small, the convergence rate is faster. If σ → 0 , the rate is dominated by a lower-order term in the big O notation, recovering the O ( 1 T ) rate in the noiseless setting (see Appendix D for details). This is in sharp contrast with the existing analysis of BO algorithms. Existing convergence proofs in the noisy setting rely on analyzing the information gain, which is vacuous when σ → 0 , requiring new tools. It is interesting that no new tools are required here.

Finally, we revisit Assumption 2. In the case when it does not hold, we use a modified update:

where the projection operator proj X ( · ) is defined as proj X ( x ) = argmin z ∈X ∥ z -x ∥ , i.e ., the closest feasible point to x . When the iterates stay in the interior of the domain X , the projection operator is an identity map and the update rule simply reduces to x t +1 = x t -η t ∇ µ D t ( x t ) .

Define the gradient mapping

which is a generalization of the gradient in the constrained setting: when x t -η t ∇ f ( x t ) lies in the interior of X , the gradient mapping reduces to the (usual) gradient ∇ f ( x t ) . The following gives convergence rates of ∥ G ( x t ) ∥ .

Theorem 3. Under the same conditions as Corollary 2, without Assumption 2, using the projected update rule (7) , Algorithm 1 obtains the following rates:

with probability at least 1 -2 δ . Here, n is the total number of samples queried.

These rates are slower than Corollary 2. We defer more details to Appendix C.

6 Additional Experiments

In this section, we investigate numerically the bounds in our proofs and study situations where the assumptions are violated. We focus on analytical experiments, because the excellent empirical performance of local BO methods on high dimensional real world problems has been well established in prior work [ e.g ., 10, 19, 20, 27]. Detailed settings and additional experiments are available in Appendix E. The code is available at https://github.com/kayween/local-bo-convergence .

6.1 How loose are our convergence rates?

This section investigates the tightness of the bounds on the error function E d,k,σ ( b ) -akey quantity in our convergence rates. We plot in Figure 2 the error function E d,k,σ for the ν = 2 . 5 Matérn

10

2

10

3

10

4

batch size

b

10

-

1

10

0

10

1

error function

E

d,k,σ

(

b

)

bound

σ

=0

.

05

err.

func.

σ

=0

.

05

bound

σ

=0

.

1

err.

func.

σ

=0

.

1

bound

σ

=0

.

1

err.

func.

σ

=0

.

1

10

0

10

1

10

2

dimension

d

10

-

2

10

-

1

10

0

10

1

10

2

error function

E

d,k,σ

(

b

)

bound

σ

=0

.

05

err.

func.

σ

=0

.

05

bound

σ

=0

.

1

err.

func.

σ

=0

.

1

bound

σ

=0

.

1

err.

func.

σ

=0

.

1

(a) fixed dimension, varying batch sizes

(b) fixed batch size, varying dimensions

-

20

-

19

-

18

-

17

local solution value

probability density

0

20

40

60

80

100

-

20

-

19

-

18

repeats

best value found

kernel and our bound O ( σd 3 2 b -1 2 ) implied by Lemma 5. The error function is empirically estimated by minimizing the trace of posterior gradient covariance (3) using L-BFGS. Figure 2a uses a fixed dimension d = 10 and varying batch sizes, while Figure 2b uses a fixed batch size b = 500 and varying dimensions. Both plots are in log-log scale. Therefore, the slope corresponds to the exponents on b and d in the bound O ( σd 3 2 b -1 2 ) , and the vertical intercept corresponds to the constant factor in the big O notation.

Figure 2a shows that the actual decay rate of the error function E d,k,σ ( b ) may be slightly faster than O ( b -1 2 ) , as the slopes of the (empirical) error function have magnitude slightly larger than 1 2 . Interestingly, Figure 2b demonstrates that the dependency d 3 2 on the dimension is quite tight - all lines in this plot share a similar slope magnitude.

6.2 What is the effect of multiple restarts?

In §3, we analyze local solutions found by a single run on different GP sample paths. Here, we investigate the impact of performing multiple restarts on the same sample path. In Figure 3, we plot (left) a kernel density estimate of the local solution found for a series of 10 000 random restarts, and (right) the best value found after several restarts with a 90% confidence interval. We make two observations. First, the improvement of running 10-20 restarts over a single restart is still significant: the Gaussian tails involved here render a difference of ± 1 in objective value relatively large. Second, the improvement of multiple restarts saturates relatively quickly. This matches empirical observations

-1.0

-0.5

0.0

0.5

1.0

-0.5

0.0

0.5

1.0

1.5

data  

x

0.0

0.3

0.6

0.9

1.2

1.5

1.8

2.1

-1.0

-0.5

0.0

0.5

1.0

-0.5

0.0

0.5

1.0

1.5

subgradient

data  

x

0.0

0.4

0.8

1.2

1.6

2.0

2.4

2.8

-1.0

-0.5

0.0

0.5

1.0

-0.5

0.0

0.5

1.0

1.5

subgradient

data  

x

0.0

0.4

0.8

1.2

1.6

2.0

2.4

2.8

(a) ∇ µ D ( x )=(0 . 039 , 1 . 018) . The ground truth is (0 , 1) .

(b) ∇ µ D ( x )=(0 . 005 , 0 . 318) . Subgradient is [ -1 , 1] ×{ 1 } .

∇

µ

D

(

x

) = (

-

Subgradient is

0

[

.

031

-

1

,

,

1]

1

.

053)

×{

1

}

.

made when using local BO methods on real world problems, where optimization is often terminated after (at most) a handful of restarts.

6.3 What if the objective function is non-differentiable?

In this section, we investigate what happens when f is not differentiable - a setting ruled out in our theory by Assumption 1. In this case, what does the posterior mean gradient ∇ µ D learn from the oracle queries? To answer this question, we consider the ℓ 1 norm function x ↦→ ∥ x ∥ 1 in two dimensions, which is non-differentiable when either x 1 or x 2 is zero. The ℓ 1 norm is convex and thus its subdifferential is well-defined:

In Figure 4, we use the posterior mean gradient ∇ µ D to estimate the (sub)gradient at (0 , 1) . As a comparison, we also show the result for the differentiable quadratic function x ↦→ 1 2 ∥ x ∥ 2 . Comparing Figure 4a and 4b, we observe that the ℓ 1 norm has higher estimation error than the Euclidean norm despite using exactly the same set of queries. This might suggest that non-differentiability increases the sample complexity of learning the 'slope' of the function. Increasing the sample size to n = 10 eventually results in a ∇ µ D ( x ) much closer to the subgradient, as shown in Figure 4c.

7 Discussion and Open Questions

For prototypical functions such as GP sample paths with RBF kernel, the local solutions discovered by local Bayesian optimization routines in high dimension are of surprisingly high quality. This motivates the theoretical study of these routines, which to date has been somewhat neglected. Here we have established the convergence of a recently proposed local BO algorithm, GIBO , in both the noiseless and noisy settings. The convergence rates are polynomial in both the number of samples and the input dimension, supporting the obvious intuitive observation that finding local optima is easier than finding the global optimum.

Our developments in this work leave a number of open questions. It is not clear whether solutions from local optimization perform so well because the landscape is inherently easy ( e.g ., most stationary points are good approximation of the global minimum) or because the local BO has an unknown algorithmic bias that helps it avoid bad stationary points. This question calls for analysis of the landscape of GP sample paths (or the RKHS). Additionally, we conjecture that our convergence rates are not yet tight; §6.1 suggests that there is likely room for improvement. Further, it would be interesting to establish convergence for trust-region based local BO [ e.g . 10, 9].

In practice, while existing lower bounds imply that any algorithm seeking to use local BO as a subroutine to discover the global optimum will ultimately face the same exponential-ind sample complexities as other methods, our results strongly suggest that, indeed as explored empirically in the literature, local solutions can not only be found, but can be surprisingly competitive.

(c)

.

Acknowledgments and Disclosure of Funding

KWwould like to thank Thomas T.C.K. Zhang and Xinran Zhu for discussions on RKHS. RG is supported by NSF awards OAC-2118201, OAC-1940224 and IIS-1845434. JRG is supported by NSF award IIS-2145644.

References

Ahmad Ajalloeian and Sebastian U Stich. On the convergence of SGD with biased gradients. arXiv preprint arXiv:2008.00051, 2020.

Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231-357, 2015.

Adam D. Bull. Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research , 12(88):2879-2904, 2011.

Ioannis Chatzigeorgiou. Bounds on the Lambert function and their application to the outage analysis of user cooperation. IEEE Communications Letters , 17(8):1505-1508, 2013.

Oleg Davydov and Robert Schaback. Error bounds for kernel-based numerical differentiation. Numerische Mathematik , 132:243-269, 2016.

Nando De Freitas, Alex Smola, and Masrour Zoghi. Exponential regret bounds for Gaussian process bandits with deterministic observations. In Proceedings of the 29th International Conference on Machine Learning , 2012.

Aryan Deshwal and Jana Doppa. Combining latent space and structured kernels for Bayesian optimization over combinatorial spaces. In Advances in Neural Information Processing Systems , volume 34, pages 8185-8200, 2021.

Luc Devroye and Gábor Lugosi. Combinatorial Methods in Density Estimation . Springer Science & Business Media, 2001.

David Eriksson and Matthias Poloczek. Scalable constrained Bayesian optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130, pages 730-738. PMLR, 2021.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In Advances in Neural Information Processing Systems , volume 32, 2019.

Roman Garnett. Bayesian Optimization . Cambridge University Press, 2023.

Antoine Grosnit, Rasul Tutunov, Alexandre Max Maraval, Ryan-Rhys Griffiths, Alexander I Cowen-Rivers, Lin Yang, Lin Zhu, Wenlong Lyu, Zhitang Chen, Jun Wang, et al. Highdimensional Bayesian optimisation with variational autoencoders and deep metric learning. arXiv preprint arXiv:2106.03609, 2021.

Rafael Gómez-Bombarelli, Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science , 4(2):268-276, 2018.

Gautam Kamath. Bounds on the expectation of the maximum of samples from a Gaussian. 2015. URL http://www.gautamkamath.com/writings/gaussian_max.pdf .

Kirthevasan Kandasamy, Jeff Schneider, and Barnabas Poczos. High dimensional Bayesian optimisation and bandits via additive models. In Proceedings of the 32nd International Conference on Machine Learning , volume 37, pages 295-304, 2015.

Kenji Kawaguchi, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Bayesian optimization with exponential convergence. In Advances in Neural Information Processing Systems , volume 28, 2015.

Natalie Maus, Haydn Jones, Juston Moore, Matt J Kusner, John Bradshaw, and Jacob Gardner. Local latent space Bayesian optimization over structured inputs. In Advances in Neural Information Processing Systems , volume 35, pages 34505-34518, 2022.

Natalie Maus, Kaiwen Wu, David Eriksson, and Jacob Gardner. Discovering many diverse solutions with Bayesian optimization. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 1779-1798, 2023.

Sarah Müller, Alexander von Rohr, and Sebastian Trimpe. Local policy search with Bayesian optimization. In Advances in Neural Information Processing Systems , volume 34, pages 20708-20720, 2021.

Quan Nguyen, Kaiwen Wu, Jacob Gardner, and Roman Garnett. Local Bayesian optimization via maximizing probability of descent. In Advances in Neural Information Processing Systems , volume 35, pages 13190-13202, 2022.

Jonathan Scarlett. Tight regret bounds for Bayesian optimization in one dimension. In Proceedings of the 35th International Conference on Machine Learning , volume 80, pages 4500-4508, 2018.

Yao Shu, Zhongxiang Dai, Weicong Sng, Arun Verma, Patrick Jaillet, and Bryan Kian Hsiang Low. Zeroth-order optimization with trajectory-informed derivative estimation. In The Eleventh International Conference on Learning Representations , 2023.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian optimization of machine learning algorithms. In Advances in Neural Information Processing Systems , volume 25, 2012.

Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning , pages 1015-1022, 2010.

Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating Bayesian optimization for biological sequence design with denoising autoencoders. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 20459-20478, 2022.

Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. In Advances in Neural Information Processing Systems , volume 33, pages 11259-11272, 2020.

Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimization using Monte Carlo tree search. In Advances in Neural Information Processing Systems , volume 33, pages 19511-19522, 2020.

Holger Wendland. Scattered Data Approximation , volume 17. Cambridge University Press, 2004.

James T Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Peter Deisenroth. Pathwise conditioning of Gaussian processes. The Journal of Machine Learning Research , 22(1):4741-4787, 2021.

A Technical Lemmas

Lemma 6. Let p n ( x ) = ∑ n i =0 1 i ! x i be the degreen Taylor polynomial of e x around x = 0 . Then for any k ≥ 1 and any x ∈ R , we have

Proof. The proof is an induction on k . The base case k = 1 is trivial: e x ≥ 1 + x .

Consider the general case p 2 k +1 . Define g ( x ) = e x -p 2 k +1 ( x ) . It is easy to see that g ′ ( x ) = e x -p 2 k ( x ) and g ′′ ( x ) = e x -p 2 k -1 ( x ) . By the induction hypothesis, g ′′ ≥ 0 and therefore g is convex. Thus, the minimum of g is given by its stationary points. It is easy to observe that x = 0 is indeed a stationary point. Thus, min x ∈ R g ( x ) = g (0) = 0 , which finishes the proof.

Lemma 7. Let σ > 0 and W be the principal branch of the Lambert W function. For any m ∈ N , we have

Proof. The problem is reduced to proving

for all -1 e ≤ x ≤ 0 . In fact, the right hand side is exactly the first-order Taylor expansion of the left hand side at x = -1 e [ e.g ., 4].

Square both sides. It is equivalent to prove (1 + W ( x )) 2 ≤ 2(1 + ex ) for all x ∈ [ -1 e , 0] . Define

The derivative of g in ( -1 e , 0) is

By the definition of the Lambert W function, we have

since -1 < W ( x ) < 0 for x ∈ ( -1 e , 0) . Thus, g ′ ( x ) < 0 for x ∈ ( -1 e , 0) and g is a decreasing function in [ -1 e , 0] . Observe that g ( -1 e ) = 0 . Therefore g ( x ) ≤ 0 for all x ∈ [ -1 e , 0] , which completes the proof.

Next, we present a lemma which states that the error function bounds the posterior covariance trace in each iteration of Algorithm 1.

Lemma 8. In the t -th iteration of Algorithm 1, we have

Proof. Without loss of generality, we assume x t = 0 . Otherwise, shift the data D t and x t by -x t , which does not change the value of the left hand side becauce of stationarity of the kernel k . Let Z ∈ R b t × d be arbitrary candidates. Then, we have

Because the LHS conditions on both D t -1 and Z but the RHS only conditions on Z . Now, we minimize Z on both sides.

On the one hand, the LHS becomes tr ( ∇ k D t ( 0 , 0 ) ∇ ⊤ ) . This is because D t is the union of D t -1 and the minimizer of the acquisition function α trace ( 0 , Z ) = tr ( ∇ k D t -1 ∪ Z ( 0 , 0 ) ∇ ⊤ ) .

On the other hand, the RHS becomes E d,k,σ ( b t ) by definition of the error function, which completes the proof.

Note that the edge case, where the minimizer of the acquisition function argmin Z α trace ( 0 , Z ) does not exist ( e.g ., when σ = 0 ), can be handled by a careful limiting argument using the same idea.

A.1 Lemmas for the RKHS Assumption

By Assumption 1, the kernel function k is four times continuously differentiable. The following lemma asserts the smoothness of f ∈ H .

Lemma 9. Suppose f ∈ H maps from a compact domain X to R . Then f is L -smooth for some L .

Proof. Since the kernel k is four times continuously differentiable, f is twice continuously differentiable. On a compact domain X , the spectral norm of the Hessian ∥∇ f ( x ) ∥ has a maximizer. Define L = max x ∈X ∥∇ f ( x ) ∥ . Then f is L -smooth.

Lemma 1. For any f ∈ H , any x ∈ X and any D , we have the following inequality

Proof. This is a simple corollary of a standard result in meshless scattered data approximation [ e.g ., 28, Theorem 11.4]. The main idea is to express the estimation error as a linear functional, and then compute the operator norm of that linear functional.

Let λ = δ x D : H → R be the composition of the evaluation operator and differential operator, i.e . λf = Df ( x ) . Wendland [28, Theorem 11.4] provides a bound on ( λf -λµ D ) 2 :

where λ (1) applies λ to the first argument of k D and λ (2) applies λ to the second argument of k D .

Pick the linear functional λ : f ↦→ ∂ ∂x i f ( x ) where 1 ≤ i ≤ d . Then, the left hand side becomes ( λf -λµ D ) 2 = ( ∂ ∂x i f ( x ) -∂ ∂x i µ D ( x )) 2 . The right hand side λ (1) λ (2) k D ( · , · ) is exactly the i -th diagonal entry of ∇ k D ( x , x ) ∇ ⊤ . For each i , use the above inequality, and then summing over all coordinates finishes the proof.

A.2 Lemmas for Convergence on GP Sample Paths

In this section, we provide a few lemmas for the GP sample path f ∼ GP (0 , k ) . By Assumption 1, we have the follow lemma which asserts that f is smooth with high probability.

Lemma 3. For 0 < δ < 1 , there exists a constant L > 0 such that f is L -smooth w.p. at least 1 -δ .

Proof. The proof uses the Borell-TIS inequality. Let f ∼ GP (0 , k ) be a Gaussian process. Provided that sup x ∈X | f ( x ) | is almost surely finite, the Borell-TIS inequality states that

where u > 0 is an arbitrary positive constant and s 2 = sup x ∈X E | f ( x ) | 2 . Namely, if the supremum of f is almost surely finite then the supremum of f is bounded with high probability.

Since k is four time continuously differentiable, the second-order derivative ∂ 2 ∂x i ∂x j f exists and is almost surely continuous. On a compact domain X , the supremum sup x ∈X | ∂ 2 ∂x i ∂x j f ( x ) | is almost surely finite. By the Borell-TIS inequality, the expectation E sup x ∈X | ∂ 2 ∂x i ∂x j f ( x ) | is finite and the supremum sup x ∈X | ∂ 2 ∂x i ∂x j f ( x ) | is bounded with high probability. Thus, the Frobenius norm of the Hessian ∇ 2 f ( x ) is bounded with high probability by a union bound. Since the spectral norm of ∇ f ( x ) can be bounded by its Frobenius norm, the spectral norm of the Hessian ∥∇ f ( x ) ∥ is also bounded with high probability, which gives the smoothness constant.

The next two lemmas bound the gradient estimation error with high probability.

Lemma 10. Let u ∼ N ( 0 , Σ ) be a Gaussian vector. Then for any t > 0

Proof. This is a standard concentration inequality result, but we give a self-contained proof here for completeness. Denote the spectral decomposition Σ = QΛQ ⊤ . By Markov's inequality, we have

where s > 0 is an arbitrary positive constant. Let ϵ ∼ N ( 0 , I ) be a standard Gaussian variable. Then it is easy to see that ∥ u ∥ = ∥ Λ 1 2 ϵ ∥ . Then, replacing ∥ u ∥ with ∥ Λ 1 2 ϵ ∥ gives

where the first line plugs in ϵ ; the second line uses ∥·∥ 2 ≤ ∥·∥ 1 ; the third line is due to independence of ϵ i ; the forth line removes the absolute value resulting an extra factor of 2 ; the last lines uses the moment generating function of ϵ i . Optimizing the bound over s gives the desired result

Lemma 11. For any 0 < δ < 1 , let C t = 2log( π 2 t 2 6 δ ) . Then, the inequalities

hold for any t ≥ 1 with probability at least 1 -δ .

Proof. Since ∇ f ( x t ) ∼ N ( ∇ µ D t ( x t ) , ∇ k D t ( x t , x t ) ∇ ⊤ ) , applying Lemma 10 gives

The particular choice of C t makes the probability on the right hand side become 6 δ π 2 t 2 . Using the union bound over all t ≥ 1 and using the infinite sum ∑ ∞ t =1 1 t 2 = π 2 6 finishes the proof.

We provide an important remark. The probability in Lemma 11 is taken over the randomness of f and the observation noise. On the other hand, the posterior mean gradient ∇ µ D t is deterministic, since it is conditioned on the data D t .

B Bounds on the Error Function E d,k,σ

This section is devoted to bounding the error function E d,k,σ ( b ) in terms of the batch size b . The results in this section immediately give a bound on the posterior covariance trace by Lemma 8.

Before diving into the proofs, we present some immediate corollaries of Assumption 1 on the kernel. Because k is stationary, the kernel can be written as k ( x , x ′ ) = ϕ ( x -x ′ ) for some positivedefinite function ϕ . Observe that ∇ k ( x , x ′ ) = ∇ ϕ ( x -x ′ ) and ∇ k ( x , x ′ ) ∇ ⊤ = -∇ 2 ϕ ( x -x ′ ) . Denote the first-order partial derivative ∂ i ϕ ( x ) = ∂ ∂x i ϕ ( x ) and the second-order partial derivative

∂ 2 i ϕ ( x ) = ∂ 2 ∂x 2 i ϕ ( x ) . It is easy to see that ϕ is an even function and ∇ ϕ is an odd function. In addition, 0 is a maximum of ϕ . Therefore, ∇ ϕ ( 0 ) = 0 and the Hessian ∇ 2 ϕ ( 0 ) is negative semi-definite.

B.1 Noiseless Setting

The following is a bound for the error function E d,k, 0 for arbitrary kernels satisfying Assumption 1 in the noiseless setting σ = 0 .

Lemma 2. For σ = 0 , the error function is bounded by E d,k, 0 ( b ) ≤ C max { 0 , 1 + d -b } , where C = max 1 ≤ i ≤ d ∂ 2 ∂x i ∂x ′ i k ( 0 , 0 ) is the maximum of the Hessian's diagonal entries at the origin.

Proof. The bound holds trivially for b = 0 , 1 and thus a proof is only needed for b ≥ 2 , which we split into two cases 2 ≤ b ≤ d +1 and b > d +1 .

We first focus on the case 2 ≤ b ≤ d +1 . Let z 0 = 0 and z i = h e i where i = 1 , 2 , · · · b -1 , where e i is the i -th standard unit vector and h > 0 is a constant. Define Z = ( z 0 z 1 · · · z b -1 ) ⊤ . By the definition of the error function, we have

where we define A = ∇ k ( 0 , 0 ) ∇ ⊤ -∇ k ( 0 , Z ) k ( Z , Z ) -1 k ( Z , 0 ) ∇ ⊤ and use the inequality A ii ≤ -∂ 2 i ϕ ( 0 ) ≤ C for b ≤ i ≤ d .

Let us focus on the i -th diagonal entry A ii where 1 ≤ i ≤ b -1 . Then we have

where the first line is because conditioning on the subset z 0 and z i does not make the posterior smaller. Now let h → 0 and compute the limit by L'Hôpital's rule. We have

Thus letting h → 0 gives the inequality E d,k,σ ( b ) ≤ C (1 + d -b ) for 2 ≤ b ≤ d +1 .

When d > d +1 , note that E d,k,σ ( b ) is an decreasing function in b and thus E d,k,σ ( b ) ≤ E d,k,σ ( d + 1) = 0 . Both cases can be bounded by the expression C max { 0 , 1 + d -b } .

B.2 Noisy Setting

This section proves bounds on the error function E d,k,σ for the RBF kernel and the ν = 5 2 Matérn kernel in the noisy setting. The lemmas in this section will implicitly use the assumption that k ( 0 , 0 ) = 1 . This assumption is indeed satisfied by the RBF kernel and the Matérn kernel, which are of primary concern in this paper.

Before proving the bound on E d,k,σ , we need one more technical lemma:

Lemma 12 (Central Differencing Designs) . Consider the 2 md points Z ∈ R 2 md × d defined as

where 1 ≤ i ≤ d , 1 ≤ j ≤ 2 m and e i is the i -th standard unit vector. Define

Then, we have

for all 1 ≤ i ≤ d and thus

where α i = ϕ (2 h e i ) , β i = ∂ i ϕ ( -h e i ) and γ = σ 2 m .

Proof. Note that A is the posterior covariance at the origin 0 conditioned on Z . Denote Z ( i ) = ( z ( i ) 1 z ( i ) 2 · · · z ( i ) m z ( i ) m +1 · · · z ( i ) 2 m ) ⊤ the subset of 2 m points that lie on the i -th axis. Then, the i -th diagonal entry A ii can be bounded by

since conditioning on only a subset of points Z ( i ) would not make the posterior variances smaller ( e.g ., the posterior covariance is the Schur complement of a positive definite matrix). The remaining proof is dedicated to bounding the right hand side.

First, we need to compute the inverse of the 2 m × 2 m kernel matrix

where α i = ϕ (2 h e i ) is a nonnegative constant and 1 is a m dimensional vector (we drop the index i in the matrix ̂ K for notation simplicity). We compute the inverse analytically by forming its eigendecomposition

where Λ = diag( λ 1 , λ 2 , · · · , λ 2 m ) and Q = ( q 1 q 2 · · · q 2 m ) . Observe that:

where ⊗ denotes the Kronecker product. Because the eigenvalues (vectors) of a Kronecker product equal the Kronecker product of the individual eigenvalues (vectors), and because adding a diagonal shift simply shifts the eigenvalues, the top two eigenvalues of ̂ K are λ 1 = m (1 + α i ) + σ 2 and λ 2 = m (1 -α i ) + σ 2 . The remaining 2 m -2 eigenvalues are σ 2 . The top two eigenvectors are

Next, we cope with the term ∂ i k ( 0 , Z ( i ) ) , where the partial derivative is taken w.r.t. the first argument's i -th coordinate. Denote v ⊤ = ∂ i k ( 0 , Z ( i ) ) . Then it is easy to see that

where β i = ∂ i ϕ ( -h e i ) . Note that v happens to be an eigenvector of ̂ K as well, because v ∥ q 2 . As a result, Q ⊤ v has a simple expression Q ⊤ v = (0 - √ 2 mβ i 0 · · · 0) ⊤ . Thus, we have

Summing over the coordinates 1 ≤ i ≤ d finishes the proof.

With Lemma 12, we are finally ready to present the bounds for the RBF kernel and the Matérn kernel.

Lemma 4 (RBF Kernel) . Let k ( x 1 , x 2 ) = exp ( -1 2 ∥ x 1 -x 2 ∥ 2 ) be the RBF kernel. We have

where m ∈ N and W denotes the principal branch of the Lambert W function.

Proof. For any Z ∈ R 2 md × d , the following inequality holds by the definition of error function

Consider the following points z ( i ) j defined as

where 1 ≤ i ≤ d . The total number of points is exactly 2 md . By Lemma 12, we have

For the RBF kernel, the values of α i and β i are the same for each coordinate 1 ≤ i ≤ d since it is isotropic:

Plugging the value of α , β , γ and ∂ 2 i ϕ ( 0 ) into the bound on E d,k,σ , we have

where the second inequality replaces exp( -h 2 ) with exp( -2 h 2 ) in the numerator. Because the bound holds for arbitrary h , we can apply the transformation h ↦→ 1 √ 2 h , which gives the inequality

Our goal is to bound A ii in terms of m and σ 2 . Therefore, we minimize the right hand side over h . Define g ( x ) = 1 -me -x x m (1 -e -x )+ σ 2 , where x ≥ 0 . The derivative is given by:

The unique stationary point is x ∗ = 1 + W ( -m e ( m + σ 2 ) ) , where W is the principal branch of the Lambert W function. It is easy to see the stationary point x ∗ is the global minimizer of g ( x ) over R ≥ 0 . Plug x ∗ into the expression of g . Coincidentally, we have g ( x ∗ ) = 1 + W ( -m e ( m + σ 2 ) ) as well -x ∗ is a fixed point of g .

In summary, we have shown A ii ≤ 1 + W ( -m e ( m + σ 2 ) ) for each coordinate i . Summing A ii over all d coordinates proves the first inequality E d,k,σ (2 md ) ≤ d ( 1 + W ( -m e ( m + σ 2 ) ) ) . The second inequality is a direction implication of Lemma 7, which completes the proof.

Lemma 5 (Matern Kernel) . Let k ( · , · ) be the ν = 2 . 5 Matérn kernel. Then, we have

Proof. The proof is similar to Lemma 4. The difference is that we need to upper bound ∂ 2 i ϕ ( 0 ) + 2 β 2 i (1 -α i )+ γ by a rational function. Otherwise the expression is intractable to minimize.

The Matérn kernel with half integer ν can be written as a product of an exponential and a polynomial. In particular, for ν = 5 2 , we have

When x is in the nonnegative orthant, the gradient is

Since the Matérn kernel is isotropic, the α i and β i as in Lemma 12 are the same across different coordinate i , and their values are

In addition, -∂ 2 i ϕ ( 0 ) = 5 3 . By Lemma 12, we have

Next, we approximate the exponential function exp( -2 h ) by its Taylor polynomials. By Lemma 6, use the inequality exp( -2 h ) ≥ 1 -2 h for the numerator and the inequality exp( -2 h ) ≥ 1 -2 h + 2 h 2 -4 3 h 3 for the denominator. Applying these two inequalities gives

Let h = γ 1 4 and thus γ = h 4 . Then we have

where the third line drops h 2 and h 3 in the denominator; the forth line plugs in the value h = γ 1 4 back and drops the constants. Summing over the coordinates 1 ≤ i ≤ d gives the first inequality:

For large enough m , the bound is dominated by the first term σdm -1 2 , which completes the proof.

We end this section with a short summary. Lemma 4 and Lemma 5 happen to end up with the same rate E d,k,s (2 md ) = O ( σdm -1 2 ) . Replacing 2 md with the batch size b , we have shown that E d,k,s ( b ) = O ( σd 3 2 b -1 2 ) for the RBF kernel and ν = 2 . 5 Matérn kernel.

B.3 Discussion: Forward Differencing Designs

This section explores an alternative proof for the error function based on forward differencing designs, as opposed to the central differencing designs in Lemma 12. Similar to the previous section, we assume k ( 0 , 0 ) = 1 , which is indeed satisfied by the RBF kernel and Matérn kernel.

Lemma 13. Consider following ( d +1) m points Z ∈ R ( d +1) m × d defined as

where e i is the i -th standard unit vector. Define

Then, we have

where α i = ϕ ( h e i ) , β i = ∂ i ϕ ( -h e i ) and γ = σ 2 m .

Proof. The proof is similar to Lemma 12.

Denote Z ( i ) be the subset of points consisting of z ( i ) j and z (0) j , where j = 1 , 2 , · · · , m . Notice that the i -th diagonal entry A ii can be bounded by

We need to invert the 2 m × 2 m matrix

Again, we resort to the eigendecomposition of ̂ K = QΛQ ⊤ . The top two eigenvalues of ̂ K are λ 1 = m (1 + α i ) + σ 2 and λ 2 = m (1 -α i ) + σ 2 . The remaining 2 m -2 eigenvalues are σ 2 . The top two eigenvectors are

Denote v ⊤ = ∂ i k ( 0 , Z ( i ) ) . Note that v can be written as a linear combination of q 1 and q 2 :

Then, straightforward calculation gives Q ⊤ v = 1 2 β i √ 2 m (1 -1 0 · · · 0) ⊤ .

Then, we have

Summing over all coordinates finishes the proof.

Lemma 14. Let k ( x 1 , x 2 ) = exp ( -1 2 ∥ x 1 -x 2 ∥ 2 ) be the RBF kernel. The forward differencing designs give a decay rate of E d,k,σ ( b ) = O ( σdm -1 2 ) .

Proof. Define α = ϕ ( h e i ) and β = ∂ i ϕ ( -h e i ) . Plugging the values of α and β into the bounds in Lemma 13 yields

The bound holds for arbitrary h . Applying the transformation h ↦→ √ h gives

where the second line uses the inequality exp( -h ) ≥ 1 -h in the numerator and the denominator; the last line is because γ is nonnegative. Let h = γ 1 2 so that γ = h 2 . Then we have

where the first line plugs in γ = h 2 ; the third line drops the h 3 +2 h in the denominator; the forth lines plugs in h = γ 1 2 ; the fifth line is because γ 1 2 dominates the bound when m is large. Thus, we have shown E d,k,σ ( dm + m ) = O ( σdm -1 2 ) .

The above lemma shows that forward differencing designs achieve the same asymptotic decay rate as the central differencing designs for the RBF kernel. Though, the leading constant in the big O notation is slightly larger.

C Convergence Proofs

The following is a useful lemma for biased gradient updates, which bounds the gradient norm via the cumulative bias. The proof is adapted from a lemma by Ajalloeian and Stich [1, Lemma 2].

Lemma 15. Let f be L -smooth and bounded from below. Suppose the gradient oracle ˆ g t has bias bounded by ξ t in the t -th iteration. Namely, we have ∥ ˆ g t -g t ∥ 2 ≤ ξ t for all t ≥ 0 , where g t = ∇ f ( x t ) is the ground truth gradient. Then the update x t +1 = x t -η t ˆ g t with η t ≤ 1 L produces a sequence { x t } ∞ t =1 satisfying

Proof. By L -smoothness, we have

Plugging in the update formula x t +1 = x t -η t ˆ g t , we have

where the first inequality uses L -smoothness; the second inequality uses η t ≤ 1 L ; the fourth inequality expands the squared Euclidean norm; the last inequality uses the definition of bias. Summing the inequalities for t = 1 , 2 , · · · , T and rearranging the terms, we have

Dividing both sides by ∑ T t =1 η t gives

The left hand side is a weighted average, which is greater than the minimum over 1 ≤ t ≤ T , which completes the proof.

Next, we prove a variant of Lemma 15 for the projected update x t +1 = proj X ( x t -η t ∇ f ( x t )) . For the ground truth gradient g t = ∇ f ( x t ) , define the gradient mapping

For the approximate gradient ˆ g t = ∇ µ D t ( x t ) , define the gradient mapping

In the following, we introduce two lemmas characterizing the projection operator proj X ( · ) .

Lemma 16 ( e.g ., Lemma 3.1 of Bubeck et al. [2]) . Let X be convex and compact. Let x ∈ X and z ∈ R d . Then we have

As a result, ∥ x -z ∥ 2 ≥ ∥ proj X ( z ) -z ∥ 2 + ∥ x -proj X ( z ) ∥ 2 .

Lemma 17. The following holds:

Proof. (1-2): The first two inequalities are direct corollaries of Lemma 16. (3): This is proved as follows:

where the second line is because the projection operator is non-expansive.

Rearranging the terms finishes the proof.

Now we give a lemma proving biased gradient update with the projection operator. The proof is adapted from a lemma by Shu et al. [22].

Lemma 18. Let f be L -smooth over a convex compact set X . Moreover, assume the gradient norm ∥∇ f ( x ) ∥ is bounded by L ′ on X . Suppose the gradient oracle ˆ g t has bias bounded by ξ t in the t -th iteration: ∥ ˆ g t -g t ∥ 2 ≤ ξ t for all t ≥ 0 , where g t = ∇ f ( x t ) . Then the update x t +1 = proj X ( x t -η t ˆ g t ) with η t ≤ 1 L produces a sequence { x t } ∞ t =1 satisfying

where G ( x t ) = 1 η t ( x t -proj X ( x t -η t g t ) ) is the gradient mapping.

Proof. By L -smoothness, we have

where the second line plugs in the update x t +1 = x t -η t G ( x t ) and the third line is due to η t ≤ 1 .

̂ L

Now we analyze the two terms separately. For the first term, we have

where the second inequality uses Lemma 17. For the second term, we have

Summing them together, we have

where the second line uses Cauchy-Schwarz inequality.

A telescoping sum gives

which results in

The rest of this section proves all theorems and their corollaries in the main paper.

Theorem 1. Let f ∈ H whose smoothness constant is L . Running Algorithm 1 with constant batch size b t = b and step size η t = 1 L for T iterations outputs a sequence satisfying

Proof. By Lemma 1 and Assumption 3, we can bound the bias in the iteration t as

By Lemma 8, the trace in the RHS can be bounded by the error function E d,k,σ ( b ) . Thus, the gradient bias is B 2 E k,k,σ ( b ) . Applying Lemma 15 with η t = 1 L and ξ t = B 2 E k,k,σ ( b ) finishes the proof.

Corollary 1. Under the same assumptions of Theorem 1, using batch size b t = d +1 , we have

Therefore, the total number of samples n = O ( dT ) and the squared gradient norm ∥∇ f ( x t ) ∥ 2 converges to zero at the rate O ( d/n ) .

Proof. By Lemma 2, we have E d,k,σ ( d +1) = 0 . Plugging it into Theorem 1 gives the rate in the iteration number T . To get the rate in samples n , note that n = ∑ T t =1 ( d +1) = ( d +1) T . Plugging T = n d +1 into the rate finishes the proof.

Theorem 2. For 0 < δ < 1 , suppose f is a GP sample whose smoothness constant is L w.p. at least 1 -δ . Algorithm 1 with batch size b t and step size η t = 1 L produces a sequence satisfying

with probability at least 1 -2 δ , where C t = 2log ( ( π 2 / 6)( t 2 /δ ) ) .

Proof. By Lemma 11, we have

with probability at least 1 -δ . The trace on the RHS can be further bounded by the error function E d,k,σ ( b t ) by Lemma 8. Applying the union bound, with probability at least 1 -2 δ , the inequality

holds for all t ≥ 0 and f is L -smooth. Applying Lemma 15 with η t = 1 L and ξ = C t E d,k,σ ( b t ) finishes the proof.

Corollary 2. Let k ( · , · ) be either the RBF kernel or the ν = 2 . 5 Matérn kernel. Under the same conditions as Theorem 2, if

with probability at least 1 -2 δ . Here, T is the total number of iterations and n is the total number of samples queried.

Proof. By Theorem 2, with probability at least 1 -2 δ , we have

The proof boils down to bounding the average cumulative bias. The full details is in Appendix D.

Theorem 3. Under the same conditions as Corollary 2, without Assumption 2, using the projected update rule (7) , Algorithm 1 obtains the following rates:

with probability at least 1 -2 δ . Here, n is the total number of samples queried.

Proof. Since f is twice differentiable on a compact set X , its gradient norm ∥∇ f ( x ) ∥ attains a maximum. Thus, there exists a constant L ′ such that L ′ ≥ ∥∇ f ( x ) ∥ for all x ∈ X . By Lemma 18 and a similar argument in Theorem 2, with probability at least 1 -2 δ , we have

where C (1) t and C (2) t are constants growing in O (log t ) . By Lemmas 4 and 5, plug in the error function E d,k,σ ( b ) = O ( σd 3 2 b -1 2 ) . The rest of the proof follows a similar argument in Corollary 2, as shown in Appendix D.

Finally, we present a convergence result for GP sample path under noiseless assumption.

Theorem 4. For 0 < δ < 1 , suppose f is a GP sample whose smoothness constant is L with probability at least 1 -δ . Assuming σ = 0 , Algorithm 1 with batch size b t = d +1 and step size η t = 1 L produces a sequence satisfying

with probability at least 1 -2 δ .

with probability at least 1 -2 δ . By Lemma 2, E d,k,σ ( d +1) = 0 , the second term is essentially zero, which finishes the proof.

D Optimizing the Batch Size

In this section, we optimize the batch size in Theorem 2 and give explicit convergence rates. The discussion in this section will give a proof for Corollary 2.

For the RBF kernel and ν = 2 . 5 Matérn kernel, by Theorem 2, Lemma 4 and Lemma 5, we have shown the following bound

We discuss polynomially growing batch size b t = dt a , where a > 0 . Then we have

We discuss three cases: 0 < a < 2 , a = 2 and a > 2 .

Case 1. When 0 ≤ a < 2 , the infinite sum ∑ T t =1 t -1 2 a log t diverges. Its growth speed is on the order of O ( T 1 -1 2 a log T ) . The total number of samples n = ∑ T t =1 b t = O ( dT a +1 ) , and thus T = O ( d -1 a +1 n 1 a +1 ) . Thus, the rate is

where the last inequality uses the fact that the second term dominates the rate when 0 < a < 2 .

Case 2. When a = 2 , the infinite sum ∑ T t =1 t -1 log t diverges. Its growth speed is on the order of O (log 2 T ) . The total number of samples is n = d ∑ T t =1 t 2 = O ( dT 3 ) , and thus T = O ( d -1 3 n 1 3 ) . Then the rate is

Case 3. When a > 2 , the infinite sum ∑ T t =1 t -1 2 a log t converges to a constant when T →∞ . The total number of samples n = O ( dT a +1 ) . Thus, the rate is

Note that a = 2 achieves the fastest rate O ( n -1 3 ) in terms of samples n .

Now we discuss a batch size with logarithmic growth b t = d log 2 t . We have

E Additional Experiments

This section presents additional experimental details and additional numerical simulations.

Details of Figure 2. In Figure 2a, we plot the error function starting from b = 20 to make sure b ≥ 2 d so that Lemma 5 indeed applies. The decay rate O ( σd 3 2 b -1 2 ) has a (leading) hidden constant of 15 √ 2 2 inside the big O notation (see the proof of Lemma 5), and thus the bounds plotted in Figure 2 are multiplied by this constant. Otherwise, the expression σd 3 2 b -1 2 alone is not a valid upper bound.

ReLU. The ReLU function max { 0 , x } is non-differentiable at x = 0 . Nevertheless, thanks to convexity the subdifferential at x = 0 is defined as [0 , 1] . We estimate the 'derivative' at x = 0 by minimizing the acquisition function. The estimate µ ′ D (0) and queries are plotted in Figure 5. The estimated derivative µ ′ D (0) is always in [0 , 1] . Thus, the posterior mean gradient µ ′ D (0) produces a subgradient in this case.

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

0.0

0.2

0.4

0.6

0.8

1.0

ReLU

data

x

(a)

n

= 2

, µ

′

D

(0) = 2

.

06

×

10

-

5

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

0.0

0.2

0.4

0.6

0.8

1.0

ReLU

data

x

(b)

n

= 4

, µ

′

D

(0) = 0

.

49

Peer reviews:
comments: SUMMARY:
Local Bayesian optimization has been popular high-dimensional optimization methods, which achieves the state-of-the-art empirical performance over various benchmarks. This paper investigates the behavior of local Bayesian optimization (BO) methods. Under a prototype algorithm, the authors empirically  demonstrate superior performance of the stationary points found by local BO compared to vanilla BO and grid search. Under the assumption that the function generating parameter are known, they theoretically and experimentally analyze the convergence rate of local BO methods.

STRENGTHS:
1. I think the study problem in this paper is important and the theoretical experiment results partially explains the good performance of local BO.

2. The theoretical analysis is easy to follow, and gives insight of how to design local BO hyperparameters according to the problem dimension.

WEAKNESSES:
1. The prototype local BO algorithm samples points to reduce the gradient estimation unceratinty, and move search center according to the estimated mean gradient, which may not be representative of all local BO methods such as trust region Bayesian optimization (TuRBO) [1].

2. The assumptions in theoretical analysis may not hold in many real-world problems.

[1] Eriksson, David, et al. "Scalable global optimization via local Bayesian optimization." Advances in neural information processing systems 32 (2019).

QUESTIONS:
1. Many experiment in this paper use GP sample path as the objective function, with a dimension up to 100 (Figure 3). How to sample a function from GP under such high-dimensional input space? Any discretization is used during the function sampling?

LIMITATIONS:
The authors have addressed the limilations. I did not find potential negative societal impact of this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLocal Bayesian optimization methods are widely used to cope with the curse of dimensionality when optimizing high-dimensional black-box functions. Although these methods have very good performance empirically, virtually all theoretical results focus on the global optimization setting by proving bounds on the optimality gap. This paper shows that under typical assumptions a particular variant of local Bayesian optimization (GIBO) provably converges to a stationary point in polynomial time.\n\nSTRENGTHS:\nI greatly enjoyed this paper. It is well-written and easy to follow, and answers important questions with proof techniques that are novel to me. I am very happy to see that the authors included some empirical studies to verify the integrity of their results.\n\nWEAKNESSES:\nThis work is ready for publication.\n\nQUESTIONS:\n- based on the description in line 113 it seems the x axes in Figure 1 are mislabeled, is that correct?\n\n- I found the following statement on line 220 a bit vague: \"However, because observations of f are made with noise, the noise inherit too [sic] both evaluations of f is amplified by 1/2h as h -> 0.\" Can you please elaborate?\n\n- I found the second paragraph of the discussion intriguing. Do you think the noise in the BayesOpt gradient estimate may play a similar role as the minibatch gradient noise in SGD? My understanding of the argument is minibatch gradient noise tends to cause SGD to find wider basins in the loss surface, which some have argued is linked to better generalization. In this context we aren't concerned with generalization per se, but perhaps one could argue that wider basins of f are preferable since they will be more forgiving if the estimate of the stationary point is slightly off. \n\n- in practice it is quite common to estimate the GP surrogate hyperparameters through the log marginal likelihood, can you comment on how this might affect your analysis and empirical behavior?\n\n- how might one start thinking about \"convergence\" in the multi-objective case? Could we prove that local BayesOpt converges to a locally non-dominated point?\n\n- although this paper did not propose GIBO, can you comment on its numerical stability in practice? In my experience look-ahead acquisition functions tend to exhibit poor numerical behavior, which I imagine is only amplified by introducing an additional derivative operator.\n\nLIMITATIONS:\nI see two primary limitations\n\n- the analysis is restricted to GIBO. It would be interesting to know if any similar arguments can be made for TurBO [1], which is quite widely used. \n\n- the analysis relies on fairly strong (albeit common) assumptions on the nature of f and the specification of the surrogate. It seems plausible to me that (for example) BayesOpt could find a stationary point even if the surrogate is misspecified, assuming f is not too pathological.\n\n[1] Eriksson, David, et al. \"Scalable global optimization via local Bayesian optimization.\" Advances in neural information processing systems 32 (2019).", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the behavior and convergence property of the local Bayesian optimization approach, in particular, the method GIBO [16]. The paper provides the convergence rates of GIBO in both the noiseless and noisy settings. The paper also performs various empirical experiments to understand the tightness of the derived bounds. It also conducts some experiments to understand some other properties of local BO like multiple restarts and when the objective function is non-differentiable.\n\nSTRENGTHS:\n+ The paper tackles an important and difficult problem, which is to perform theoretical analysis for the local BO approach\n+ The paper writing is very clear and easy to understand. The paper is really technical; it has a lot of theory, but thanks to the clear writing, all the theoretical analysis is much easier to understand.\n+ The theoretical analysis on the convergence rates of GIBO seems to be sound to me (based on my overall understanding regarding the theoretical analysis of BO – I couldn’t dig into the proof in a very detailed manner).\n+ Some empirical experiments are also conducted to understand the tightness of the derived bounds.\n\nWEAKNESSES:\n+ The paper actually focuses only on one of the local BO approaches (the GIBO method) and this approach is actually quite different compared to some other popular local BO approaches like TurBO. The GIBO approach is based on gradient descent, but the other local BO approach like TurBO is still based on BO. The paper should make very clear regarding the focus of the theoretical analysis.\n+ Another weakness of the paper is that it still relies on the assumption that are used to analyze the global BO including the assumption that the objective function is a sample from a GP with known hyperparameters. Actually one of the main motivation of the local BO approach is based on the observation that the objective function can’t be model using one single global GP in the whole search domain. Therefore, the assumption of the theoretical analysis in this work seems to be quite strict, and therefore, the theoretical analysis (the bounds) may not shed too much light on the performance of the local BO approach.\n\nQUESTIONS:\nPlease answer my comments in the Weaknesses section, and the following additional questions:\n+ In the experiments (Section 6.1), how are the hyperparameters of the bounds or the algorithms set (like \\beta) ?\n\nLIMITATIONS:\nThe paper seems to not have a dedicated section describing the limitations of the work presented in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives theoretical considerations on local Bayesian optimization methods for black box optimization. In particular, for gradient-based local BO, when the objective function satisfies the smoothness assumption, the rate of convergence to the stationary point is derived for the noiseless and noisy cases, respectively. \nIn the noiseless case, local solutions can be found at a much faster rate with respect to the input dimension d than with conventional global BO. \nAlthough the convergence rate is worse in the presence of observation noise than in the noiseless case, it returns to the noiseless rate in the limit of zero noise variance, suggesting that it is a reasonable bound.\nThe tightness of the derived convergence rates is validated by numerical experiments.\n\nSTRENGTHS:\n- The rate of convergence of the gradient-based local BO algorithm to the local optimal solution is theoretically derived, and the rate is expected to be somewhat tight (empirically validated).\n\n- It is clear how the observational noise affects convergence in the local BO. Furthermore, by taking the limit of the noise variance parameter, the convergence rate connects the noiseless and noisy cases.\n\n- The convergence rate results for the noiseless case are given for both cases where the objective function is an element of the RKHS (in the main text) and a sample path of a Gaussian process (in the appendix).\n\nWEAKNESSES:\n- The local BO discussed here is a gradient method type algorithm called GIBO in particular, and approaches that construct the GP itself locally, such as TurBO [Eriksson+ 2019], have not been considered. \n\n- It would be nice to have a theory that connects convergence to a stationary point and regret analysis. In Section 3, the results suggest that local solutions are good enough compared to the global solution, and if this is a property that holds to some extent universally, then it seems reasonable to expect that the regrets of local search also have good properties (e.g., sublinear properties).\n\nQUESTIONS:\n- I think one advantage of local search is that it is compatible with parallel distributed processing (such as TurBO). Is it possible to extend the gradient-based local BO treated in this paper to parallel distributed processing as well? And if so, would some theory of  convergence for such methods be obtained by a direct extension of the results derived in this paper? \n\n- Theorem 3 derives the upper bound of the error function for the squared exponential kernel. Can we easily obtain the upper bound of the error function for other kernels (e.g. Matern kernel)?\n\n- In Lemma 2 and line 226, is $E_{d, k, \\sigma}(b_t, d)$ mistake for $E_{d, k, \\sigma}(b_t)$?\n\nLIMITATIONS:\nThe authors appear to have adequately addressed the limitations of the study and, where applicable, the potential adverse social consequences.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}