Title: Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Bayesian Optimization

Local Optimization

High dimensional

Minimizing upper confidence Bound
Abstract: Local Bayesian optimization is a promising practical approach to solve the high dimensional black-box function optimization problem. Among them is the approximated gradient class of methods, which implements a strategy similar to gradient descent. These methods have achieved good experimental results and theoretical guarantees. However, given the distributional properties of the Gaussian processes applied on these methods, there may be potential to further exploit the information of the Gaussian processes to facilitate the BO search. In this work, we develop the relationship between the steps of the gradient descent method and one that minimizes the Upper Confidence Bound (UCB), and show that the latter can be a better strategy than direct gradient descent when a Gaussian process is applied as a surrogate. Through this insight, we propose a new local Bayesian optimization algorithm, MinUCB, which replaces the gradient descent step with minimizing UCB in GIBO. We further show that MinUCB maintains a similar convergence rate with GIBO. We then improve the acquisition function of MinUCB further through a look ahead strategy, and obtain a more efficient algorithm LA-MinUCB. We apply our algorithms on different synthetic and real-world functions, and the results show the effectiveness of our method. Our algorithms also illustrate improvements on local search strategies from an upper bound perspective in Bayesian optimization, and provides a new direction for future algorithm design.

Minimizing UCB: a Better Local Search Strategy in Local Bayesian Optimization

Abstract

Local Bayesian optimization is a promising practical approach to solve high dimensional black-box function optimization problem. Among them is the approximated gradient class of methods, which implements a strategy similar to gradient descent. These methods have achieved good experimental results and theoretical guarantees. However, given the distributional properties of the Gaussian processes applied on these methods, there may be potential to further exploit the information of the Gaussian processes to facilitate the Bayesian optimization search. In this work, we develop the relationship between the steps of the gradient descent method and one that minimizes the Upper Confidence Bound (UCB), and show that the latter can be a better strategy than direct gradient descent when a Gaussian process is applied as a surrogate. Through this insight, we propose a new local Bayesian optimization algorithm, MinUCB, which replaces the gradient descent step with minimizing UCB in GIBO [22]. We further show that MinUCB maintains a similar convergence rate with GIBO. We then improve the acquisition function of MinUCB further through a look ahead strategy, and obtain a more efficient algorithm LA-MinUCB. We apply our algorithms on different synthetic and real-world functions, and the results show the effectiveness of our method. Our algorithms also illustrate improvements on local search strategies from an upper bound perspective in Bayesian optimization, and provides a new direction for future algorithm design.

1 Introduction

Bayesian Optimization [8] is one of the most well-known black box function optimization methods, where objectives can be extremely expensive to evaluate, noisy, and multimodal. The high efficiency of Bayesian Optimization in finding global optima leads to the widespread application in various research fields, such as hyperparameter tuning [10, 11, 28], neural architecture search [26], chemical experiment design [12], reinforcement learning [31], aerospace engineering [16]. However, the performance of Bayesian optimization is limited by the input dimension d , as the theoretical regret bound grows exponentially with input dimension [25]. This difficulty hinders the application of Bayesian optimization when the actual dimension of problem is rather high [8].

There are various methods that have been proposed to handle this difficult task, including works that rely on some assumptions on the model structure, such as the assumption that the majority of the variables have no effect [6] or the kernel satisfies an additive structure [9]. Local Bayesian optimization methods, which focus on finding a local optima (instead of the global one), have also been a popular (and less restrictive) compromise to manage the curse of dimensionality. Representative

1

methods of these include those based on local trust region methods [7], local latent space [20], and approximated gradient methods [22, 23, 36]. Among them the approximate gradient method has demonstrated strong performance in practical applications compared with other methods. The approximate gradient method can be described as a two-stage algorithm, which loops through the following two processes: first sample points to decrease the uncertainty of the local area according to a local exploration acquisition function, and then moving to the next point with a trustworthy high reward through a local exploitation acquisition function. Müller et al. [22] applied the idea of gradient descent and first proposed the GIBO algorithm, which was designed to alternate between sampling points to minimize the posterior variance of the gradient at a given location, and then moving in the direction of the expected gradient. Nguyen et al. [23] proposed MPD, which improved the local exploitation acquisition function through defining the descent direction by maximizing the probability of descent, and designed corresponding local exploration acquisition function to match this strategy. Wu et al. [36] further provided the detailed proof on the local convergence of GIBO with a polynomial convergence rate, for both the noiseless and noisy cases.

Although the approximated gradient method has been shown to be practical in dealing with highdimensional problems, there may be still some room to potentially improve on the current methods. We motivate the ideas for improvement with the following two questions.

1) We observe that GIBO only utilizes the posterior distributions of the gradient at a point, which will ignore most of the information provided by Gaussian process surrogate in the region, which may lead to an inefficient descent. MPD attempts to make better use of Gaussian processes by performing multi-step descent, but this strategy can exhibits numerical instability and may lead to suboptimal performance of the algorithm (as seen in Section 8). This motivates us to think: is there a better local exploitation acquisition function that can ensure the algorithm fully utilizes Gaussian process information, and also guarantee the convergence to local optima points?

2) Do these acquisition functions necessarily need to depend on accurate gradient estimates at a point, or are there other acquisition functions that can improve the efficiency and still ensure local convergence?

In this paper, we attempt to answer the above two questions through our two new local Bayesian optimization algorithms. To address the first question, we first develop the relationship between the step of the gradient descent method and minimizing the Upper Confidence Bound (UCB). When the Gaussian process is applied as the surrogate model, minimizing the UCB can usually achieve a point with a higher reward than simply doing gradient descent. Motivated by this idea, we propose our first algorithm, Local Bayesian Optimization through Minimizing UCB (MinUCB), which replaces gradient descent step with a step that minimizes the UCB in the GIBO algorithm. We show that MinUCBwill also converge to local optima with a similar convergence rate as GIBO. This discovery is also meaningful as it opens up possibilities for new designs on local Bayesian optimization algorithms. In this work we further apply the look ahead strategy to construct the local exploration acquisition function that is more compatible with minimizing the UCB, and propose our second algorithm, Look Ahead Bayesian Optimization through Minimizing UCB (LA-MinUCB). This algorithm is shown to be one step Bayesian optimal, and address our second question as a local efficient BO acquisition function does not require the additional step and accuracy of a gradient estimate. We implement extensive experiments to demonstrate the performance of our algorithms under different settings. The experimental results illustrate that our algorithms have better performance compared to other methods across many synthetic and real-world functions. We summarize our contributions as follows:

We develop the relationship between the gradient descent step and minimizing the UCB, and show that minimizing UCB is more efficient when the Gaussian process is the underlying surrogate.

We show that minimizing UCB is an efficient and accurate objective for local exploitation and propose MinUCB.

We improve the local exploration acquisition function of MinUCB and obtain a more efficient local Bayesian optimization algorithm LA-MinUCB.

We apply different synthetic and real-world function on our algorithm, and the results show the effectiveness of our methods.

0

0.25

0.5

0.75

1

1.25

1.5

x

-0.5

-0.25

0

0.25

0.5

0.75

1

y

Our method

Previous method

x

0

Real function

Approximated Gradient bound

Gradient bound

Upper confidence bound

-6

-4

-2

0

2

4

6

x

-1

0

1

2

3

4

y

Real function

Upper confidence bound

2 Literature Review

High dimensional optimization is a growing research area, where many different methods have been proposed to solve this problem, including Bayesian optimization methods. Among the BO methods, the most widely studied approach focuses on structured Gaussian processes, which imposes additional assumptions, such as low dimensional active spaces or additive spaces, on kernel function or data structure. This includes the Additive and ANOV A Models [4], in which the kernel of Gaussian process is defined through the summation of univariate kernels. This essentially decompose the original Gaussian process into a sum of low dimensional Gaussian processes, which facilitates computations and convergence [9, 37, 13]. Another approach along these lines assumes that the objective function depends only on low dimensional subspace, and examples include hyperparameter optimization for neural networks [3]. Most of these methods suppose a subspace is a linear subspace of the original Euclidean space. Wang et al.[34] apply random linear embeddings to approximate this subspace and propose REMBO. Letham et al. [18] improve on this and tries to fix over exploration of boundary and distortions in embedding through adaptive linear embeddings [18]. Other methods to learn the subspace structure include low-dimension matrix recovery [5] and nonlinear embeddings [21].

As mentioned in the Section 1, another line of research to address the computational and high dimensional challenges is to compromise and focus the BO to be more local, limiting the search region to facilitate computational feasibility and efficiency. Typical approaches along these lines look at incorporating the information about the local optimum [1], restricting the problem to a sequence of iteratively chosen one-dimensional sub-problems [14, 15], using trust regions [7, 32, 20], which expands and shrinks the size of its trust regions with information in each iteration, and approximated gradient methods [22, 23, 36]. Among them, the gradient based method MPD [23], proposed by Nguyen et al. has demonstrated very strong performance compared with other local methods, including gradient based method GIBO [22], trust region method Turbo [7] and Augmented Random Search (ARS) [19]. Given the promising performance of approximate gradient methods, we leverage and extend the works of GIBO and MPD, to provide a simplier and more general local BO approach that can perform as well.

3 Preliminaries

3.1 Bayesian Optimization

In this paper, we focus on the problem of minimizing a black-box function f ( · ) :

We assume that no higher order information can be obtained from the oracle (Zeroth-Order-Oracle), where only i.i.d noisy function evaluations y = f ( x ) + ε , ε ∼ N (0 , σ 2 ) can be observed. Here we focus on applying Bayesian optimization to solve the problem. BO is a surrogate based optimization approach where the objective function is typically modeled with a Gaussian process, and an acquisition function is constructed to sequentially determine next evaluation points in the optimization process.

3.2 Gaussian process and its derivatives

Currently Gaussian process (GP) is one of the most widely used surrogate model today as it has nice analytical form and is flexible to capture various functional forms. A GP ( m ( x ) , k ( x , x ′ )) is specified by its mean function m ( · ) and kernel function k ( · , · ) . Without loss of generality, we assume the mean function m ( x ) ≡ 0 . Suppose f is sampled from the GP (0 , k ( x , x ′ )) , and we already have a dataset D = { ( x 1 , y 1 ) , ..., ( x n , y n ) } , and set X D = [ x 1 , ..., x n ] , y D = [ y 1 , ..., y n ] T , then the posterior over f is also a GP ( µ D ( x ) , k D ( x , x ′ )) , where

where k ( x , X D ) = [ k ( x , x 1 ) , ..., k ( x , x n )] , k ( X D , x ) = k ( x , X D ) T , and k ( X D , X D ) is the positive definite kernel matrix [ k ( x i , x j )] i,j ≤ n .

It should be noted that if the kernel function k is differentiable, then the derivative of f is also a GP. Given the dataset D , The posterior of f satisfies:

In this work we only consider the noisy case, i.e. the standard derivation of white noise σ > 0 .

4 The relationship between gradient descent and minimizing UCB

We first review the traditional gradient descent approach. This approach is fundamentally based on the assumption of the smoothness of the function:

Definition 1. (Smoothness) A function f is L-smooth if and only if for all x 1 , x 2 ∈ X , we have

Suppose the initial point is x 0 , and function f ( · ) is L-smooth, then we will have the following inequality:

This provides a quadratic upper bound on f , and the minimum value of this upper bound is taken at x = x 0 -η ∇ f ( x 0 ) . In the gradient descent approach, the minimum value of this upper bound is taken as the descent step, where η = 1 L is used as the step size in traditional gradient descent analysis. In this view gradient descent can be treated as selecting the minimum point of this quadratic upper bound Eq. (1).

As direct gradient information is not observable in practice, the upper bound Eq. (1) cannot be obtained, and approximate gradient methods instead attempt to derive a looser upper bound based on Eq. (1). In GIBO [22, 36], they replace the gradient ∇ f ( x 0 ) with the derivative of the Gaussian process ∇ µ D ( x 0 ) , and apply the strategy x = x 0 -η ∇ µ D ( x 0 ) when the variance of this gradient is small enough. According to the proof of Lemma 15 in [36], GIBO is also a result of optimizing a different upper bound of f ( · ) . If x is chosen to be the form of x = x 0 -η ∇ µ D ( x 0 ) and η ≤ 1 L , this upper bound is:

If the approximation error of gradient ∥∇ µ D ( x 0 ) -∇ f ( x 0 ) ∥ 2 is small enough , then the optimal η is chosen as 1 L . This upper bound Eq.(2) is actually the local exploitation acquisition function of GIBO.

Although these two bounds have intuitive application in the gradient descent, they do have some limitations. The first is the obtainment of the L-smooth coefficient. Although it is possible to estimate the L-smooth coefficient through the Gaussian process, this estimation is expensive as it needs many samples, especially in high dimensional cases. The second is that these two bounds are relatively loose, and the minimum points of these two bounds tend to be too close to x 0 . Upper bound Eq. (1) is quadratic and increase very fast when the point x is far from x 0 , and upper bound Eq. (2) only allows the stepsize η to be less than 1 L . When L is unknown and we have to give it a large estimate to ensure convergence (usually in real case), the above phenomenon becomes more severe. Taking this view of gradient descent approaches as moving along the minimum of an upper bound of f ( · ) , then leads us to explore if it is possible to discover some tighter upper bounds, where the minimum point is lower than that in Eq. (1) and Eq. (2). This can lead us to find a point with a possible higher reward.

A commonly used concept in Bayesian Optimzation is the upper confiedence bound (UCB), which is defined as followed:

Previous work mainly focused on maximizing UCB to find the maximum value of a function [29]. However, it should be noted that UCB is also a natural bound for function f ( · ) . UCB fully utilizes the posterior distribution of f ( · ) , and give every point a probabilistic bound depending on the coefficient β . The standard deviation term σ D ( x ) has an upper bound and will not grow faster than the quadratic function, which means the UCB will not change drastically. This indicates that if we select x to be

Then the function value f ( x ∗ ) more likely to have a smaller value than the points obtained through optimizing the upper bound Eq. (1) and Eq. (2). This is because the points obtained by minimizing UCB can be further away from the initial point compared to simply gradient descent.

Fig (1) shows a simple 1-dimension illustrative example. In this example, we sample a function f from GP (0 , k ( x, x ′ )) , where k ( x, x ′ ) = exp ( -1 4 ( x -x ′ ) 2 ) , and the standard deviation of white noise σ = 0 . 05 . Here we illustrate the search of the next point from x 0 based on the upper bound perspective of three methods. Suppose we have already sampled two points, which are marked as black hollow circles. These two data points are selected through Central Finite Difference Approximations [27], which is aimed to better estimate the gradient of ∇ f ( x 0 ) . In the left figure, the green line represent the real quadratic upper bound Eq. (1) at x 0 , and the blue line is calculated through Eq. (2). The coefficient β in UCB bound Eq. (3) is set as 3, which means that for any point x , the probability of f ( x ) < UCB ( x ) is close to 99 . 9% . It can be seen from the left figure in Fig (1) that the UCB bound is much tighter than other two bounds, and the minimum point of the UCB bound has a much lower function value than the gradient based method. The right figure plots the UCB across the input space, and we see that the UCB changes relatively slowly and will not reach infinity. Further we observe that UCB is only small near the sampled point, indicating that minimizing the UCB can be viewed as a local strategy. This simple example illustrates that with a Gaussian process function, minimizing UCB can achieve a better point than the gradient methods, as UCB efficiently utilizes the information from Gaussian process. Based on these insights, we propose two new local Bayesian optimization algorithms, and demonstrate their performances in several numerical examples.

5 Local Bayesian Optimization through Minimizing UCB

The analysis the above section provides us with an important idea, that if we replace the gradient descent step with a step that minimizes a tighter upper bound such as UCB, we may be able to achieve a better result in local optimization. Our first algorithm, Local Bayesian Optimization through Minimizing UCB (MinUCB) (Algorithm 1), is developed with this idea, and we show that minimizing the UCB for the step move is an efficient objective for the local exploitation, that can guarantee the convergence with an appropriate local exploration acquisition function.

MinUCB can be viewed as a modified version of GIBO [22, 36] (we list GIBO algorithm in Appendix A for reference). In our approach, we adopt the same local exploration acquisition function to sample points as in GIBO (to keep that constant) (line 4 in Algorithm 1), and only set the objective of local exploitation acquisition function that drives the step move to minimizing the UCB, instead of gradient descent step x t +1 = x t -η t ∇ µ D t ( x t ) , as shown in line 8 in Algorithm 1. We first introduce some notations here for better illustration of our algorithm. We define k D∪ Z ( x t , x t ) = k D ( x t , x t ) -k D ( x t , Z )( k D ( Z , Z ) + σ 2 I ) -1 k D ( Z , x t ) , which is exactly the posterior variance of f ( x t ) conditioned on the dataset D and a new input Z . Because the estimation of variance does not require y , we have omitted the symbol here. The local exploration acquisition function for sampling is defined on this posterior variance:

which is the trace of the posterior covariance matrix of the ∇ f ( x t ) conditioned on the dataset and input. This trace quantifies the uncertainty of gradient ∇ f ( x t ) . With a large batch size b (2) t , minimizing this trace will result in a lower uncertainty on the estimation of gradient. Although UCB doesn't involve gradient descent, we keep this step constant and argue that the candidates selected through this local exploration acquisition function will still efficiently decrease the uncertainty on this local area, which will benefit the local exploitation move when minimizing the UCB.

In the local exploitation part, the β t controls the search area for each step. The larger the β t , the closer x t +1 will be to the existing data point. Minimizing the UCB can bring performance improvements, and we show later in Section 6 (Theorem 2) that MinUCB will have a similar convergence rate as GIBO with carefully selection on coefficients β t and batch size b (1) t , b (2) t . The results of MinUCB can provide inspiration on the design for more efficient local Bayesian optimization algorithm, as shown in Section 7.

It should be noted that in MinUCB we resample multiple times on the local exploitation result x t with a batch size b (1) t (line 3 in Algorithm 1). This step is mainly added to ensure the theoretical convergence of the algorithm. This step will typically be only a very small proportion of sampling points, especially in the high dimensional case.

6 Convergence Analysis of MinUCB

In this section, we establish a convergence analysis of MinUCB to demonstrate the effectiveness of using minimizing UCB as the objective of local exploitation. We prove that MinUCB has a polynomial convergence rate, and this rate also exhibits a polynomial relationship with the input dimension, indicating that MinUCB performs very well in high-dimensional case. For the whole convergence proof of MinUCB, please refer to Appendix B.1-B.4. In our convergence analysis, we set a mild assumption on kernel function k :

Assumption 1. The kernel k ( · , · ) is stationary, four times continuously differentiable, strictly positive definite, and bounded: max x ∈ X k ( x , x ) ≤ 1

Many common kernels such as RBF kernel and Matérn kernel with γ > 2 will satisfy this assumption. We also need the definition domain of Gaussian process is bounded:

Assumption 2. The Gaussian process f ( x ) is defined on a bounded closed set X , i.e. there exist a constant r > 0 that ∀ x 1 , x 2 ∈ X , ∥ x 1 -x 2 ∥ 2 ≤ r .

The next we borrow the definition of Error function from Wu et al. [36]. The Error function measures the maximum reduction of uncertainty about the gradient estimation at x = 0 when there are b data points Z without any extra dataset:

Definition 2. (Error function) Given input dimensionality d , kernel k and noise standard deviation σ , we define the following error function:

Based on the above assumptions, we develop the convergence theory for MinUCB:

Theorem 1. Suppose f is sampled from a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and k ( · , · ) satisfies Assumption 1. Then after t iterations of MinUCB algorithm, with the batch size b (1) t and b (2) t , it satisfies that

where ˜ β t and η t are both decreasing sequence. They satisfies ˜ β t = O ( β t ) and 1 η t = O ( d √ log t 2 d 2 δ + d 3 2 ) , and β t = √ 2 log π 2 t 2 δ

Proof Outline of Theorem 1 :

From the definition of x t +1 ,we can obtain f ( x t +1 ) ≤ min x ∈ R d µ D t ( x ) + β t σ D t ( x ) ≤ µ D t (ˆ x t +1 ) + β t σ D t (ˆ x t +1 ) , where ˆ x t +1 is a pseudo gradient descent step. ˆ x t +1 = x t -η t ∇ µ D t ( x t ) . The probability of the above inequality being true can be controlled by β t .

We try to build the relationship of µ D t ( x t ) + β t σ D t ( x t ) and µ D t (ˆ x t +1 ) + β t σ D t (ˆ x t +1 ) . We apply the local properties of the mean function µ D t ( x ) and standard deviation function σ D t ( x ) (proved in Appendix B.1 and B.2), which is:

We bring the above relationship into the initial inequality, and use the triangle inequality to establish the relationship between ∇ µ D t ( x t ) and ∇ f ( x t ) . After a few more simple steps of derivation, the final result can be obtained.

It can be seen that the convergence of gradient ∥∇ f ( x t ) ∥ 2 heavily depend on the batch size b (1) t and b (2) t . The large batch size will accelerate the convergence, with the additional cost of sampling. The upper bound of E d,k,σ ( b ) and its theoretical property can be referred in Wu et al. [36], and we also list them at Appendix B.3, lemma 6-9. Thus if we combine the above result, the convergence rate of MinUCB is obtained:

Theorem 2. Suppose f is sampled from a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , then if the kernel is RBF kernel or Matérn kernel with γ = 2 . 5 , and satisfy β t = √ 2 log π 2 t 2 , batch size

Then MinUCB will achieve the convergence rate of

where n is the number of samples. Our method achieves similar results to Wu et al. [36], except for some logarithmic term and the order of d . The dimension d in our boundary is larger than that in

Wu's work, which is because we also take the upper bound of the L-smooth coefficient of Gaussian process into consideration, while this upper bound also increases at the polynomial rate with the data dimension d , as seen in Theorem 3 in Appendix B.1. According to Theorem 2, we need to iteratively increase the UCB coefficient β t and batch size b (1) t , b (2) t to guarantee the convergence of MinUCB. This phenomenon can be explained that, when the algorithm approaches the local optima, the area around local optima will usually be flatter than other areas (as the gradient is near 0). The algorithm needs more detailed local exploration to ensure a better descent. The polynomial convergence rate demonstrates that our local exploitation strategy, minimizing UCB, is accurate and powerful. It can ensure the accuracy of the local search and fully utilize the information of Gaussian processes.

7 Look Ahead Bayesian Optimization through Minimizing UCB

Our proposed MinUCB enjoy good theoretical properties and provide an alternative idea of minimizing the UCB as a good way to progress the local search under a Gaussian process surrogate. However, there are still improvements that can be made to the local exploration in MinUCB. Specifically, UCB itself does not require any gradient information, and the local exploration in MinUCB still focuses on learning the information at a single current point. This under utilizes the Gaussian process surrogate, and the potential information in the local region. Based on the above, we focus here on selecting a better local exploration acquisition function for minimizing UCB, which can help to accelerate the local Bayesian optimization.

In this section we apply a look ahead strategy. The motivation here is to obtain desired candidates to improve the UCB bound, and help the next local exploitation achieve better results under an expectation view. With this idea we propose our second algorithm, Look Ahead Bayesian Optimization through Minimizing UCB (LA-MinUCB) (Algorithm 2).

Suppose we have the input Z and their labels y Z (this part is unknown before sampled), we define D Z as D Z = { ( z i , y i ) } , i = 1 , ..., b t as the dataset formed through Z and y Z . Then we define

This is the upper confidence bound when we already have the dataset D t -1 ∪ D Z , and we want to find the input Z to minimize the minimum point of UCB: min x UCB ( x , D t -1 , Z , y Z ) . However, as the label y Z is unknown, we can only choose to optimize it through its expectation. We adopt this look ahead predictive as the local exploration acquisition function in LA-MinUCB:

Although the local exploration in LA-MinUCB does not need to be specified around a certain point, it is still necessary to have an local exploitation step. Local exploitation step may find a point with current best reward, and will provide a better foundation for subsequent local exploration.

LA-MinUCB has a similar structure with traditional Knowledge Gradient [8] except for the standard derivation term σ D t ( x ) . This standard derivation term behaved as a regularization term, that force the sampled points to be not too far away from current area. This is because only when the selected points are closer to the current optimal point, are they more likely to learn which nearby area may

25D objective

50Dobjective

100D objective

2.5

2.5

eward

2.0

2.0

2.0

re

1.5

1.5

1.5

b

1.0

1.0

1.0

LA-MinUCB

urrer

0.5

MinUCB

0.5

0.5

MPD

0.0

GIBO

0.5

F0°0

0.0

TuRBO

0

100

200

400

500

0

100

200

300

400

500

0

100

200

300

400

500

#queries

4DCartPole

16DSwimmer

33DHopper

500

350

500

300

400

250

400

best

300

200

300

urrent

200

150

LA-MinUCB

200

100

100

R

MinUCB

MPD

100

50

GIBO

0

0

TurBO

0

0

10

20

OE

40

50

60

0

200

400

600

800

1000

0

500

1000150020002500300035004000

#queries

contain smaller values, thereby achieving a greater decrease in the next exploitation step. This extra standard derivation term restrict the over-exploration of the original Knowledge Gradient to help it perform better exploitation. Benefiting from this property, LA-MinUCB is able to focus on doing local search and quickly find local optima points.

The significant advantage of using LA-MinUCB is that its strategy has good theoretical properties:

Theorem 3. If only one iteration is left and we can observe the function value through sampling, then the local exploration in LA-MinUCB is Bayes-optimal among all feasible policies.

The proof of this theorem is listed in Appendix B.5. This indicates that LA-MinUCB is a greedy strategy, where in each step it use the optimal acquisition function to enhance the exploitation. From this perspective, LA-MinUCB is likely to be superior to MinUCB because LA-MinUCB is better prepared for the next step of exploitation. This is also reflected in our experiments results, which shows that LA-MinUCB have a very competitive performance in numerical experiments and practical applications. Meanwhile, this also illustrates that good local Bayesian optimization algorithms can be constructed without the need for approximate gradients.

8 Experiments

In this section we apply varies experimental settings to assess the efficacy of our proposed two algorithms, Min-UCB and LA-MinUCB, relative to two established methodologies. These include two approximated gradient methods, (1) GIBO [22] and (2)MPD [23], which has shown strong performance in local Bayesian optimization methods, and one trust region based method (3) TurBO [7]. Our code is based on Nguyen et al. [23], where they provide the program of GIBO, MPD, and various of objective functions. Our experimental settings on synthetic (Sect. 4.1) and reinforcement learning (Sect. 4.2) objectives are same as their papers. Each algorithm is executed a total of ten times for every objective function that we examine, initiating from an identical set of starting points sampled across the bounded domain via a Sobol sequence. We illustrate the results in Fig. (2) (synthetic objective) and (3) (reinforcement learning objective). The figures shows the mean of current best reward on the number of queries with an error bar (defined through the standard derivation). Our experimental framework was executed on a workstation of 20 Intel Xeon CPU cores, with a 32GB

of memory. The codes can be viewed on https://github.com/chinafzy1/Minimizing-UCB. For more computational details, please refer to Appendix A. For more experimental results and ablation study, please refer to Appendix C.

8.1 Synthetic Objectives

In our first experiments, we focused on optimizing synthetic objective functions within the ddimensional unit hypercube [0 , 1] d . These functions were generated by sampling from a Gaussian Process (GP) with a Radial Basis Function (RBF) kernel. This Synthetic objective is first mentioned in Müller et al. [22] , and for a more comprehensive understanding of this experimental setup, please refer to §4.1 in their work. Each experiment was allocated a budget of 500 function evaluations, under the dimension { 25 , 50 , 100 } separately. The error bar here is defined as plus or minus 0.2 standard derivation for better illustration, as the functions sampled by Gaussian processes have significant differences. Fig. (2) illustrates that our proposed LA-MinUCB and Min-UCB achieve highly competitive performance in both efficiency and accuracy, where LA-MinUCB is slightly better. The MPD ascent very rapidly at the beginning, but soon shows instability. Although each ascent applies the direction of maximum probability, it considers multiple ascent in one iteration in the algorithm. This strategy may instead lead to a small probability that the final point is greater than the original point, causing the algorithm to fall into suboptimal solutions. GIBO can achieve very close to the real local optimum, but its strategy is too conservative, resulting in slow convergence. Similarly, when the dimensionality is particularly high, TurBO performs relatively poorly and does not have much competitiveness.

8.2 Reinforcement Learning Objective

In this experiment, we turned to reinforcement learning, specifically MuJoCo-based locomotion tasks [30]. In these tasks, we try to learn a linear policy that maps states into actions in order to maximize the rewards given by the learning environment. We adopt the experiments similar with Nguyen et al. [23] including three environments: CartPole-v1 with 4 parameters, Swimmer-v1 with 16 parameters, and Hopper-v1 with 33 parameters. The only difference is that we apply the Hopper-v1 without the state normalization, as in Nguyen's work they show that this state normalization will cause a significant unstable change in the function value. The error bar here is defined as plus or minus one standard derivation. Our results, depicted in Fig. (3), show that LA-MinUCB has an excellent performance in all three cases. LA-MinUCB exhibits faster convergence speed and may achieve a better local optima than other methods. Although MinUCB has also shown slightly better performance than GIBO, it is still not that efficient as LA-MinUCB. This reflects the powerful performance of LA-MinUCB in real applications, and also indicates that efficient local Bayesian optimization algorithm can be designed without the estimation of gradient information.

9 Conclusion

In this paper we find the connection between gradient descent and minimizing the UCB, demonstrating that minimizing UCB is an efficient local exploitation strategy with the Gaussian Processes surrogate. We introduce MinUCB, an algorithm applying minimizing the UCB as the objective of local exploitation, that will converges to local optima within a polynomial rate. By enhancing the local exploration acquisition function of MinUCB, we have developed a more advanced local Bayesian optimization algorithm, LA-MinUCB. We have tested our algorithm on various synthetic and reinforcement learning objectives, and the results have confirmed the efficacy of our approaches.

In this article, we have not provided the convergence proof of the LA-MinUCB algorithm. One consideration is whether LA-MinUCB have a better theoretical convergence rate compared to other methods, such as GIBO? Additionally, are there any other better local exploitation strategy in local Bayesian optimization? These will become interesting research directions in the future.

Acknowledgments and Disclosure of Funding

Zheyi Fan and Qingpei Hu's work are partly supported by National Key Research and Development Program of China (2021YFA1000300 and 2021YFA1000301)

Szu Hui Ng's work is supported in part by the Ministry of Education, Singapore (grant: R-266-000149-114).

References

Riad Akrour, Dmitry Sorokin, Jan Peters, and Gerhard Neumann. Local bayesian optimization of motor skills. In International Conference on Machine Learning , pages 41-50. PMLR, 2017.

Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems , 33:21524-21538, 2020.

James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research , 13(2), 2012.

Mickael Binois and Nathan Wycoff. A survey on high-dimensional gaussian process modeling with application to bayesian optimization. ACM Transactions on Evolutionary Learning and Optimization , 2(2):1-26, 2022.

Josip Djolonga, Andreas Krause, and Volkan Cevher. High-dimensional gaussian process bandits. Advances in neural information processing systems , 26, 2013.

David Eriksson and Martin Jankowiak. High-dimensional bayesian optimization with sparse axis-aligned subspaces. In Uncertainty in Artificial Intelligence , pages 493-503. PMLR, 2021.

David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems , 32, 2019.

Peter I Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811 , 2018.

Jacob Gardner, Chuan Guo, Kilian Weinberger, Roman Garnett, and Roger Grosse. Discovering and exploiting additive structure for bayesian optimization. In Artificial Intelligence and Statistics , pages 1311-1319. PMLR, 2017.

Jacob R Gardner, Matt J Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cunningham. Bayesian optimization with inequality constraints. In ICML , volume 2014, pages 937-945, 2014.

Michael A Gelbart, Jasper Snoek, and Ryan P Adams. Bayesian optimization with unknown constraints. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence , pages 250-259, 2014.

Jeff Guo, Bojana Rankovi´ c, and Philippe Schwaller. Bayesian optimization for chemical reactions. Chimia , 77(1/2):31-38, 2023.

Eric Han, Ishank Arora, and Jonathan Scarlett. High-dimensional bayesian optimization via treestructured additive models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7630-7638, 2021.

Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe bayesian optimization in high dimensions via one-dimensional subspaces. In International Conference on Machine Learning , pages 3429-3438. PMLR, 2019.

Johannes Kirschner, Mojmir Mutn` y, Andreas Krause, Jaime Coello de Portugal, Nicole Hiller, and Jochem Snuverink. Tuning particle accelerators with safety constraints using bayesian optimization. Physical Review Accelerators and Beams , 25(6):062802, 2022.

Rémi Lam, Matthias Poloczek, Peter Frazier, and Karen E Willcox. Advances in bayesian optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic Approaches Conference , page 1656, 2018.

Armin Lederer, Jonas Umlauft, and Sandra Hirche. Uniform error bounds for gaussian process regression with application to safe control. Advances in Neural Information Processing Systems , 32, 2019.

Ben Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-examining linear embeddings for high-dimensional bayesian optimization. Advances in neural information processing systems , 33:1546-1558, 2020.

Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. Advances in neural information processing systems , 31, 2018.

Natalie Maus, Haydn Jones, Juston Moore, Matt J Kusner, John Bradshaw, and Jacob Gardner. Local latent space bayesian optimization over structured inputs. Advances in neural information processing systems , 35:34505-34518, 2022.

Riccardo Moriconi, Marc Peter Deisenroth, and KS Sesh Kumar. High-dimensional bayesian optimization using low-dimensional feature spaces. Machine Learning , 109:1925-1943, 2020.

Sarah Müller, Alexander von Rohr, and Sebastian Trimpe. Local policy search with bayesian optimization. Advances in Neural Information Processing Systems , 34:20708-20720, 2021.

Quan Nguyen, Kaiwen Wu, Jacob Gardner, and Roman Garnett. Local bayesian optimization via maximizing probability of descent. Advances in neural information processing systems , 35:13190-13202, 2022.

Beth A Reid, Will J Percival, Daniel J Eisenstein, Licia Verde, David N Spergel, Ramin A Skibba, Neta A Bahcall, Tamas Budavari, Joshua A Frieman, Masataka Fukugita, et al. Cosmological constraints from the clustering of the sloan digital sky survey dr7 luminous red galaxies. Monthly Notices of the Royal Astronomical Society , 404(1):60-85, 2010.

Shubhanshu Shekhar and Tara Javidi. Significance of gradient information in bayesian optimization. In International Conference on Artificial Intelligence and Statistics , pages 2836-2844. PMLR, 2021.

Yu Shen, Yang Li, Jian Zheng, Wentao Zhang, Peng Yao, Jixiang Li, Sen Yang, Ji Liu, and Bin Cui. Proxybo: Accelerating neural architecture search via bayesian optimization with zero-cost proxies. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 9792-9801, 2023.

Wei Shyy. A study of finite difference approximations to steady-state, convection-dominated flow problems. Journal of Computational Physics , 57(3):415-438, 1985.

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems , 25, 2012.

Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on Machine Learning , pages 1015-1022. Omnipress, 2010.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Matteo Turchetta, Andreas Krause, and Sebastian Trimpe. Robust model-free reinforcement learning with multi-objective bayesian optimization. In 2020 IEEE international conference on robotics and automation (ICRA) , pages 10702-10708. IEEE, 2020.

Xingchen Wan, Vu Nguyen, Huong Ha, Binxin Ru, Cong Lu, and Michael A Osborne. Think global and act local: Bayesian optimisation over high-dimensional categorical and mixed search spaces. In International Conference on Machine Learning , pages 10663-10674. PMLR, 2021.

Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics , pages 745-754. PMLR, 2018.

Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando De Feitas. Bayesian optimization in a billion dimensions via random embeddings. Journal of Artificial Intelligence Research , 55:361-387, 2016.

Jian Wu, Matthias Poloczek, Andrew G Wilson, and Peter Frazier. Bayesian optimization with gradients. Advances in neural information processing systems , 30, 2017.

Kaiwen Wu, Kyurae Kim, Roman Garnett, and Jacob Gardner. The behavior and convergence of local bayesian optimization. Advances in Neural Information Processing Systems , 36, 2024.

Juliusz Krzysztof Ziomek and Haitham Bou Ammar. Are random decompositions all we need in high dimensional bayesian optimisation? In International Conference on Machine Learning , pages 43347-43368. PMLR, 2023.

A Additional details of the algorithms

In this section we will provide additional details about our proposed algorithms. First we list the GIBO algorithm in the version of Wu et al.[36], as shown in Algorithm 3. MinUCB adopt the same local exploration acquisition function to sample points as in GIBO, and change the objective of local exploitation acquisition function into minimizing the UCB, instead of gradient descent step.

As for the computational problem of LA-MinUCB, the optimization for the local exploration acquisition function is quite hard, as it involves a nested optimization problem. To handle this task we use the idea of one-shot optimization in BoTorch [2], that transform the original problem into a deterministic optimization problem with fixed sampling. This method is first proposed to solve the optimization of Knowledge Gradient. However, considering that our method is very similar to Knowledge Gradient, their optimization method is also applicable to our problem. For more information about this one-shot optimization method, please refer to the Section 4.2 in Balandat et al. [2].

In our numerical experiment, which has been shown in Section 8, we apply the same experimental settings with Müller et al. [22] and Nguyen et al. [23]. The coefficients in GIBO and MPD are chosen as the optimal one, which are reported in their experiments or code. As for our MinUCB and LA-MinUCB, we use a fixed batch size in each experiment, i.e. b (1) t , b (2) t and b t are unchanged in each experiment. The UCB coeffieient β t in MinUCB and LA-MinUCB is fixed as 3, which has been shown to be able to balance the convergence speed and accuracy of the algorithm.

B Theoretical analysis for MinUCB and LA-MinUCB

In this section, we will present the theoretical results of the algorithms introduced in the article. Sections B.1-B.4 provide the theoretical convergence proof for MinUCB, while B.5 discusses the theoretical properties of LA-MinUCB. Sections B.1-B.3 serve as the preliminary theory for proving the convergence of MinUCB, mainly presenting the local smoothness properties of Gaussian processes, mean functions, and standard deviation functions, as well as the theoretical properties of approximate gradients. B.4 combines the above theories, connecting the minimization of UCB with gradient descent, and provides the convergence theorem for MinUCB. B.5 demonstrates that LA-MinUCB has one-step Bayesian optimality, illustrating the superiority of the algorithm.

B.1 The function smoothness in Gaussian Process and mean function

In this subsection we mainly build the theory for the smoothness of Gaussian process and mean function. Our convergence proof heavily depend on the smoothness analysis on the Gaussian process, mean function and standard derivation function. These proof rely on the upper bound of Gaussian processes proposed by Lederer et al.[17], which is related to the Lipshitz continues coefficient on kernel function:

Lemma 1 (Lederer et al.[17]) . Consider a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and let L k denote its Lipschitz constant on the set X with maximum extension r = max x , x ′ ∈ X ∥ x -x ′ ∥ 2 . Then, with probability of at least 1 -δ , the supremum of a sample function f ( x ) of this Gaussian process is bounded by:

We use this lemma to build the upper bound on the L-smoothness coefficient on the Gaussian process and mean function:

Theorem 3 (Smoothness of Gaussian process) . Consider a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and k ( · , · ) satisfies Assumption 1. We define the second-order partial derivative k ∂ ij ( · , · ) as

Let L ∂ ij k denote the Lipschitz constants of the second-order partial derivative k ∂ ij ( · , · ) on the set X with maximum extension r = max x , x ′ ∈ X ∥ x -x ′ ∥ 2 . Then, a sample function f ( · ) is β -smooth with probability of at least 1 -δ , and β satisfies:

where

Proof. We set the Hessian matrix of f ( x ) as

According to the equivalence of matrix norms, we have

Thus

Then through applying Lemma 2, we have the following equation holds with probability at least 1 -δ/d 2

Then if we combine the Eq.(8) and Eq.(9), we will get the result of Theorem 1.

Theorem 4. If sup x ∈ X | f ( x ) | < L , then with the probability of at least 1 -δ , for a dataset D , we have

Proof. For a dataset D , we have the mean function µ D ( x ) and variance function σ 2 D ( x ) .

For a random variable r ∼ N (0 , 1) , we have P ( | r | > c ) ≤ exp( -c 2 2 ) . Thus with the probability of at least 1 -δ 2 , we have

Combine the result of Eq(10) and Eq(11), we have the following results with the probability of at least 1 -δ :

Theorem 5 (Smoothness of mean function) . Consider a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and k ( · , · ) satisfies Assumption 1. Let L ∂ ij k denote the Lipschitz constants of the second-order partial derivative k ∂ ij ( · , · ) on the set X with maximum extension r = max x , x ′ ∈ X ∥ x -x ′ ∥ 2 . Then, given a dataset D , the mean function µ D ( x ) is β µ -smooth with probability of at least 1 -δ , and β satisfies:

and L ij = sup x ∈ X ∣ ∣ ∣ ∂ 2 ∂x i ∂x j f ( x ) ∣ ∣ ∣

Proof. We set the Hessian matrix of µ D ( x ) as

According to the equivalence of matrix norms, we have

Thus we can apply Theorem 3 and get the following result probability of at least 1 -δ :

Here we need an additional Theorem to give the upper bound on the L-smoothness coefficient on the posterior of f ( · ) , which is aim to bound the probability under a specific L-smoothness coefficient.

where

Lemma 2. Consider a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and let L k denote its Lipschitz constant on the set X with maximum extension r = max x , x ′ ∈ X ∥ x -x ′ ∥ 2 . Then, given a dataset D , we define the posterior of f ( · ) over dataset D as f D ( · ) . With probability of at least 1 -δ , the supremum of a sample function f D ( x ) of this Gaussian process is bounded by:

Proof. Note that for the covariance pseudo-metric d k ( x, x ′ ) = √ k ( x, x ) + k ( x ′ , x ′ ) -2 k ( x, x ′ ) , we have

Then this Lemma can be proved in a same way with Lemma B.1 in Lederer et al. [17].

Theorem 6 (Smoothness of the posterior Gaussian process) . Consider a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and k ( · , · ) satisfies Assumption 1. Let L ∂ ij k denote the Lipschitz constants of the second-order partial derivative k ∂ ij ( · , · ) on the set X with maximum extension r = max x , x ′ ∈ X ∥ x -x ′ ∥ 2 . Then, a sample function f D ( · ) is β D -smooth with probability of at least 1 -δ , and β satisfies:

where

Proof. We set the Hessian matrix of f D ( x ) as

The posterior f D ( · ) can be divided into two parts:

where ∂ 2 ∂x i ∂x j f D ( x ) -∂ 2 ∂x i ∂x j µ D ( x ) is a zero mean Gaussian process with covariance function k ∂ij D ( x , x ′ ) . According to Lemma 2, we have the following result with probability at least 1 -δ 2 d 2

Through the result of Lemma 4, with the probability of at least 1 -δ 2 d 2 ,

Combine the result of Eq (13), Eq (14), Eq (15) and Eq (16), this Theorem is proved.

B.2 Local smoothness in standard derivation function

In this subsection we attempt to build the smoothness theorem for standard derivation function σ D ( x ) . However, σ D ( x ) is not L-smoothness under some common kernels such as Gaussian kernel or Matérn kernel with γ = 2 . 5 . However, we can proof that σ D ( x ) may achieve a similar result with some small error term.

Here we define an event U L = { f ( · ) | f ( · ) is L-smooth } , and σ D ( x | L ) = √ π 2 E ( | f ( x ) -µ D ( x ) | ∣ ∣ D , f ( · ) ∈ U L )

Lemma 3. For any x ∈ X and dataset D , we have

Proof. According to the definition of σ L ( x ) , we have

Lemma 4. If P ( U L |D ) > 1 2 , there exist constants c 1 > 1 and c 2 > 0 independent of x and dataset D that satisfies

Proof. According to the Markov inequality, we have

For any event A and B , we have P ( A ∩ B ) = P ( A ) -P ( A ∩ B c ) ≥ P ( A ) -P ( B c ) . Thus we have

If we set a = σ D ( x ) , we will have the following inequality

So through the above analysis, we have

Lemma 5. If µ D ( · ) is L µ smooth and P ( U L |D ) > 1 2 , then we have

Proof.

As f ( · ) ∈ U L and µ D ( · ) is L µ smooth,

In a similar way, we can prove

So through the above analysis, we have the following results

The second to the last line is because of Cauchy-Schwarz inequality: E ( | X | ) = E ( | X | · 1) ≤ √ E ( X 2 ) · 1 = √ E ( X 2 ) .

B.3 Some properties about approximate gradient descent

In this subsection we list properties about approximate gradient descent, which is mainly from Wu et al.[36]. This part is also essential in our final convergence proof, as we will need to connect the previous approximate gradient descent method with our minimizing UCB to give the convergence speed of gradient.

We first borrow the definition of Error function from Wu's work. The Error function measures the maximum reduction of uncertainty about the gradient estimation at x = 0 when there are b data points Z without any extra dataset:

Definition 3. (Error function) Given input dimensionality d , kernel k and noise standard deviation σ , we define the following error function:

This Error function actually bounds the variance of the estimated gradient, which can be seen in the following lemma:

Lemma 6. In the t th iterations in MinUCB, we have

Proof. In the t th step of MinUCB, the sampled candidates can be divided into two parts. One of them are sampled through local exploration acquisition function:

Suppose their corresponding label is y , and we set D 2 t = ( X , y ) , then we can obtain:

The above inequality is mainly because the D 2 t is the subset of D t . Then through the same analysis with Lemma 8 in Wu et al.[36], we can prove that

which complete our proof.

To finish the convergence proof of MinUCB, we need the following lemmas from Wu et al.[36], this lemmas give the upper bound of the estimation error of gradient, and the upper bound of Error function under two common kernels RBF kernel and Matérn kernel with γ = 2 . 5 .

Lemma 7. For any 0 ≤ δ ≤ 1 , let C t = 2log ( π 2 t 2 6 δ ) . Then the inequalities

Lemma 8. let k ( x 1 , x 2 ) = exp( -∥ x 1 , x 2 ∥ 2 ) be the RBF kernel. We have

Lemma 9. let k ( · , · ) be the Matérn kernel. Then we have

B.4 Convergence proof of MinUCB

In this part we will show the convergence of MinUCB, which will use all the results in previous Subsections. The difficulty in proving the convergence is to build the relationship with previous approximate gradient methods and minimizing UCB. In this proof we try to connect the function value on the gradient descent point and minimizing UCB point, and will need some smoothness properties on the Gaussian process, mean function and standard derivation function, which are provided in Subsection B.1 and B.2. To give the accurate convergence rate, we need the upper bound on the Error function, which is provided in Subsection B.3. We first give the basic convergence theorem on the gradient for MinUCB:

Theorem 1. Suppose f is sampled from a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , and k ( · , · ) satisfies Assumption 1. Then after t iterations of MinUCB algorithm, with the batch size b (1) t and b (2) t , it satisfies that

where ˜ β t and η t are both decreasing sequence. They satisfies ˜ β t = O ( β t ) and 1 η t = O ( d √ log t 2 d 2 δ + d 3 2 ) , and β t = √ 2 log π 2 t 2 δ

Proof. According to the definition of x t +1 , x t +1 = arg min x ∈ X µ D t ( x ) + β t σ D t ( x ) .

Where ˆ x t +1 is a special point ˆ x t +1 = x t -η t ∇ µ D t ( x t ) , which is a pseudo gradient descent step. We will use this ˆ x t +1 to build the connection between gradient descent and minimizing UCB. The β t here is carefully chosen as β t = √ 2 log π 2 t 2 δ to guarantee

Now we try to give the relationship between the UCB bound on x t and ˆ x t +1 . Suppose the mean function µ D t ( · ) is L µ t -smoothness (this coefficient will be given in the subsequent parts), we will have

Here we apply the results in Subsection B.2, and give the upper bound for the σ D t (ˆ x t +1 ) . To simplify the symbols, we define tr ( ∇ k D t ( x t , x t ) ∇ T ) = γ t .

where the first line apply Lemma 3. The third line use Lemma 5 and the last line is achieved through Lemma 4. We now try to give the coefficient of L µ t and L t , and these coefficients are all based on the smoothness coefficient of Gaussian process. First, according to Theorem 3, with the probability of at least 1 -δ 6 , for any i, j = 1 , ..., d , we have

If we carefully choose L µ t and L t as

where

According to Theorem 5 and 6, this guarantees that

and we also have P ( U c L t | D t ) ≤ 6 δ π 2 t 2 . Thus for the standard derivation term, their approximate error term is tend to be a very small value:

If we combine Eq. (19), Eq. (20), Eq. (21) and Eq. (23), and we also set ˜ β t = c 1 β t and η t = √ 2 π 1 6 β t c 1 ( L t + L µt ) , we can obtain an upper bound for f ( x t +1 ) , and this upper bound is related to the gradient:

Through simple analysis, we have

and

Thus if we consider the Lemma 7, we can transform the previous upper bound into the following result with probability 1 -δ 6 for any t > 1

Based on the above result, we can further obtain the following results with probability 1 -δ 6 for any t > 1

Based on the above inequation, after t steps in MinUCB algorithm, we can have the following result with probability of at least 1 -δ :

By organizing the above results, we can obtain

Based on this inequality we can directly give the upper bound for the norm of gradient:

Notice that in each step of MinUCB, we do the repeated sampling at point x t , we denote D 1 t = { ( x t , y 1 t ) , ..., ( x t , y b (1) t t ) } as this resampling set. We will have

Through Lemma 6, we can achieve

We combine these two value into Eq. (24), which proves this theorem.

Theorem 2. Suppose f is sampled from a zero mean Gaussian process with a continuously differentiable convariance function k ( · , · ) , then if the kernel is Gaussian kernel or Matérn kernel with γ = 2 . 5 , and satisfy β t = √ 2 log π 2 t 2 δ , batch size

MinUCB will achieve the convergence rate of

Proof. According to the proof of Theorem 1, we have

Through Lemma 8 and 9, we can bound E d,k,σ ( b (2) t ) as E d,k,σ ( b (2) t ) = O ( σ √ b (2) t d ) . Thus if we combline the result of Theorem 1 and above error bounds, we will proof the results.

B.5 Theoretical property of LA-MinUCB

In this section we will provide the theoretical property for our LA-MinUCB algorithm, that LAMinUCB is Bayesian optimal if there is only one iteration left. Our result is proved in a similar way with the Proposition 2 in Wu et al. [35].

Theorem 3. If only one iteration is left and we can observe the function value through sampling, then the local exploration in LA-MinUCB is Bayes-optimal among all feasible policies.

Proof. Suppose that we are given a budget of N max samples, i.e. we may run the algorithm for N iterations. Thus, letting Π be the set of feasible policies π , we can formulate our problem as follows:

We analyze this problem under the DP framework. We define our state space as S n = ( µ D n ( · ) , k D n ( · , · )) after iteration n as it completely characterizes our belief on f . Under the DP framework, we will define the value function V n as follows:

for every s = ( µ, k ) . The Bellman equation tells us that the value function can be written recursively as

where

and X is the new input in the n +1 th step. At the same time, we also know that any policy π ∗ whose decisions satisfy

is optimal. If we were to stop at iteration n + 1 , then V n +1 ( S n +1 ) = min x ∈ X µ D n +1 ( x ) + β n +1 σ D n +1 ( x ) and Eq. (27) reduces to

which is exactly the result of local exploration acquisition function of LA-MinUCB. This proves that the local exploration acquisition function in LA-MinUCB is one-step Bayes-optimal.

C Additional experimental results

C.1 Experiements on other real-world objective functions and methods

We further evaluate our methods on other real-world objective functions, which are the same as Nguyen et al.[23]. The initial pair of functions deals with inverse challenges typically found in physics and engineering disciplines. The first one is related to electrical engineering, where the goal is to optimize the alignment of a theoretical model of an electronic circuit with actual data. This involves adjusting nine parameters, with a limit of 500 evaluations for the optimization process. The second function addresses a cosmological issue[24], where the objective is to fine-tune a cosmological model or physical simulator to match observational data from the cosmos. The focus is on maximizing the log likelihood of the model, which is influenced by various physical constants. We follow the setting in Eriksson et al.[7], which presents a harder optimization problem with 12 parameters and much larger bounds, and set the budget at 2000 evaluations. The third function concerns rover trajectory planning [33]. It involves optimizing the placement of 100 points in a 2D space to plot the rover's trajectory and reduce costs, creating a 200-dimensional optimization problem with a budget of 1000 function evaluations. The experimental results are shown in Fig 4. This experiment shows that LA-MinUCB remains the most competitive method, with its performance showing a significant improvement compared to other methods.

To demonstrate the performance difference between our method and the original UCB, we conducted a simple experiment on the synthetic functions introduced in Section 8.1, which is shown in Fig 5. The parameters of the traditional UCB method were consistent with ours, with the UCB parameter β set to 3. Under these settings, we compared the performance of these methods. The experiment showed that the performance of the traditional UCB was very limited across these three types of problems, struggling to find the maximum value of the function, and this issue became more severe as the data dimension increased. In contrast, our method exhibited good scalability with respect to dimensions, achieving favorable results across various dimensions. This indicates that through our proposed methods, we have transformed the traditional UCB concept into local optimization, achieving very good results and making a significant difference from the original UCB performance.

C.2 Ablation study

In the ablation study, we examine the impact of the core parameter β on MinUCB and LA-MinUCB. We consider three values for β , which are 1, 3, and 5. When β is set to 1, it indicates that the algorithm adopts a more aggressive strategy, allowing it to search for points with higher uncertainty during the descent step. Conversely, when β is set to 5, the algorithm tends to be more conservative, favoring the search for points that are very likely to have function values less than the current best point. We list the experimental results on synthetic function defined in Section 8.1 under these three β values in Figure 6,7 and 8. The results show that when β is small, both MinUCB and LA-MinUCB are more likely to achieve better results in the initial search phase. This is because a smaller beta value allows for a broader search range, increasing the chances of finding relatively good results early on. However, in the later stages of finding the optimal point, it may be weaker than when β is larger. This is because an overly aggressive strategy can cause the algorithm to hover around the optimal

9D map fitting

12D cosmo. constant

200Drovertrajectory

-20

800

-40

1000

700

600

-60

800

500

-80

600

400

-100

LA-MinUCB

MinUCB

400

300

120

MPD

200

140

TurBO

GIBO

200

0

50

100

150

200

250

300

0

200

400

600

800

1000

0

200

400

600

800

1000

#queries

25D objective

50Dobjective

100D objective

2.5

2.5

reward

2.0

2.0

2.0

1.5

1.5

1.5

LA-MinUCB

1.0

MinUCB

urrent

1.0

1.0

UCB

0.5

0.5

0.5

0.0

0.5

0.0

0.0

0

50

100

150

200

250

300

0

50

100

150

200

250

000

0

50

100

150

200

250

300

#queries

MinUCB25Dobjective

LA-MinUCB 25D objective

2.5

2.5

rewara

reward

2.0

2.0

Dest

1.5

best

1.5

1.0

1.0

urreilt

Current

0.5

0.5

β=1

β=1

0.0

β=3

0.0

β=3

0.5

β=5

0.5

β =5

0

50

100

150

200

250

300

0

50

100

150

200

250

300

#queries

#queries

MinUCB5oDobjective

LA-MinUCB50Dobjective

4

β=1

β=1

Current best reward

m

β=3

β=5

Currentbestreward

m

β=3

β =5

2

2

1

1

0

0

0

50

100

150

200

250

300

0

50

100

150

200

250

300

#queries

#queries

MinUCB100Dobjective

LA-MinUCB100Dobjective

3.0

β=1

3.0

β=1

TPM

2.5

β=5

β=3

tbestreward

2.5

β=3

β=5

2.0

2.0

UCSI

1.5

1.5

1.0

Current

1.0

Tn

二

0.5

0.5

F0°0

0.0

0

50

100

150

200

250

300

0

50

100

150

200

250

300

point without getting close. This can also be seen from the Theorem 2, which states that a larger β is a necessary factor for the algorithm to approach the extreme point. When β is set to 3, it represents a more balanced outcome, achieving a good convergence rate while the optimal point is also close to that when β is set to 5, making it a better choice.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Yes, please refer to our Section 9.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Our Appendix B provide the detailed proof of our algorithms.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to experiment settings in Section 8. The codes for the proposed algorithm and experiments will also be made publicly available upon publication.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The codes for the proposed algorithm and experiments will also be made publicly available upon publication.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [No]

Justification: Our experiment in the article is completely consistent with Nguyen et al. [23], and specific details can be found in their article.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to our experiments in Section 8.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to our experiments in Section 8.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This study purely contributes to the technical advancement of local Bayesian optimization algorithm and does not have any societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer to our experiment in Section 8.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method for local Bayesian Optimisation. Instead of relying on estimates of the gradient of the black-box functions like earlier methods, the algorithm proposed in the paper minimises an upper bound on the objective function. The paper shows a connection between the proposed method and gradient descent, whose update rule can also be thought of as minimising a certain upper bound on function. The authors study the convergence of the proposed algorithm theoretically and empirically, and show it can develop improvement over existing baselines. Additionally, the authors propose a variant of the algorithm with an improved exploration strategy.\n\nSTRENGTHS:\nThe paper is generally well-written and I found the connection between MinUCB and Gradient Descent very appealing and nicely presented. Minimising UCB seems to be an interesting alternative to performing gradient steps, as it does not require the specification of the stepsize.  Empirical results show that at least on a number of selected tasks, the algorithm delivers a non-negligible improvement in performance. Summing up the theoretical and empirical results of the paper, I believe their contribution is sufficient and I recommend accepting the paper.\n\nWEAKNESSES:\n1. I belive Theorem 1 could use a proof sketch, currently just by reading the main body, it is impossible to understand where the result is coming from. \n\n2. While authors acknowledge the existence of TuRBO and ARS, they do not empirically compare with them. It would be nice to see a comparison with those baselines, particularly with TuRBO, which is very popular.\n\n3. In the related work, the authors mention the methods using additive models in high-dimensional Bayesian Optimisation but only cite relatively old papers (the newest one is from 2017). Since then, additive methods have seen much development. To make the section more up-to-date, it would be great if the authors would cite more recent work, such as [1] or [2].\n\n[1] Ziomek, Juliusz Krzysztof, and Haitham Bou Ammar. \"Are random decompositions all we need in high dimensional Bayesian optimisation?.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Han, Eric, Ishank Arora, and Jonathan Scarlett. \"High-dimensional Bayesian optimization via tree-structured additive models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 9. 2021.\n\nQUESTIONS:\n1. In the paper, the authors explain the UCB is most likely to be “small” around already queried points and thus conclude the algorithm is mostly local, however, potentially allows to make bigger steps if needed. I wonder what would happen if we explicitly constrained the algorithm to be local, e.g. via the trust region strategy of TuRBO? Would it be possible for authors to conduct a simple ablation study on at least one of the problems? Thus could help gauge whether the improvement delivered by the algorithm comes from better local optimisation or from occasionally switching to a more global search.\n\nLIMITATIONS:\nThe authors explicitly state the lack of convergence guarantee for LA-MinUCB as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and analysis a new local Bayesian optimisation scheme. The main idea is to replace the update of the current solution by minimising the upper confidence bound of the GP estimate (in the minimisation setting). The strategy is shown to obtain similar convergence rates as previous works. In addition, a \"lookahead\" variant of the algorithm is introduced. Finally, the proposed strategies are evaluated on synthetic benchmarks and RL tasks, showing competitive performance.\n\nSTRENGTHS:\n*Originality*: \nThe proposed algorithm is a variant of a prior approach (GIBO), which updates the iterate by minimising the UCB of the function instead if a local quadratic upper bound. The individual ideas are not necessarily novel (e.g. pessimistic estimates are quite commonly used in bandit/RL algorithms), but the combination with the local Bayesian optimisation scheme appears to be novel.\n\n*Quality*: \n* The obtained bounds are slightly worse than prior works, but on the upside do not require the knowledge of a Lipschitz bound.\n* The experiments show good performance on the synthetic benchmarks and RL tasks\n\n*Clarity*:\n* The algorithm and the motivation is clearly presented and overall the paper is well written, but would still benefit from polishing. However some details could be explained better as outlined below.\n\n*Significance*: \nLocal Bayesian optimization schemes have been a very successful approach to zero-order (noisy) black box optimisation and the theoretical analysis of these algorithms is still lacking, although the ground work has been layed in the prior work by Wu et al (2024). The paper therefore contributes to this landscape and the results are relevant at least for the Bayesian optimisation community and may inform better optimisation algorithms more broadly; or encourage further improved bounds.\n\nWEAKNESSES:\nMy main concerns are the following:\n\n* The minimization of the UCB score in the step update appears to be global in nature. First, the range of min /argmin operations is never specified, e.g. in Algorithm 2, eq (4), (6), etc) which is confusing to the reader. Second, the authors claim that the acquisition step leads a \"local\" update (e.g. in lines 186-187, or the claim in 215 is not formally shown). What does it mean to be \"local\"? How is this formally proved? \n* While I agree with the general picture in Figure 1, and even if this is somewhat reflecting the general case, I believe one can construct kernels/examples where this \"local\" property is violated and the algorithm technically jumps to a different part of the domain (e.g. for linear kernel with near-zero slope, where observation data causes a sign switch and the step update moves to a different vertex of the domain). \n* Third, how can we efficiently find the minimizer of the UCB? We know that in general the UCB is not a convex function so finding the argmin is as hard as in global Bayesian optimization, without additional algorithmic constraints.\n* The discussion of prior works is lacking several works, e.g. the LineBO paper [1,2] which also provides local convergence guarantees based on a gradient descent scheme, and was shown in combination with trust regions.\n* The experiments lack several baselines (e.g. Turbo or LineBO) \n* The introduction of the Lookahead strategy in 7 is a bit out of the blue and not well connected to the remaining paper; and does not come with theoretical guarantees. How does this related/connected to Algorithm 1?\n* In general, the paper should better highlight the technical contributions and challenges related to the main result, and more formal/precise definitions of the problem setting and results.\n\nMinor remarks:\n* The abstract uses acronyms that have not been introduced (BO, GIBO).\n* The claim in line 28 that dimension 20 is a critical limit for Bayesian optimisation is somewhat arbitrary, even in lower dimensions convergence can be slow if the function is non-smooth etc.\n* Equation (2) and the surrounding discussion does not make sense to me: The right-hand side of (2) does not depend on x, and can therefore not used as acquisition function or to minimise over x.\n* The type setting of equation (1) looks odd - try using `\\langle` and `\\rangle` for the inner product.\n* Although not wrong, the discussion of UCB in line 164 is lightly miss-leading as the classical UCB approach is used in the maximization setting, and the corresponding quantity in the context of this paper would be the lower confidence bound (LCB).\n* 233: gamma is not defined\n* 241: What is n ?\n* 187: \"view\" -> \"viewed\". \n\n\n[1] Kirschner, J., Mutny, M., Hiller, N., Ischebeck, R., & Krause, A. (2019, May). Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces. In International Conference on Machine Learning (pp. 3429-3438). PMLR.\n\n[2] Kirschner, J., Mutný, M., Krause, A., Coello de Portugal, J., Hiller, N., & Snuverink, J. (2022). Tuning particle accelerators with safety constraints using Bayesian optimization. Physical Review Accelerators and Beams, 25(6), 062802.\n\nQUESTIONS:\nCan you define what it means to be \"local\" and how the update step satisfies such a property?\n\nLIMITATIONS:\nThe paper briefly discusses limitations, mainly the missing analysis of the lookahead approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

comments: SUMMARY:
The author proposed a local Bayesian optimization method using UCB to drive its iterates. The UCB step replaces the gradient descent step that is typically in such methods.

STRENGTHS:
The comparison of gradient-descent to UCB is very interesting. The author included analysis as well so there is enough content to the paper. The assumptions on kernels are interesting and not straightforward. Clearly, effort has been made to establish convergence, though I think some of the theorems in the appendix would be better labeled lemma.

WEAKNESSES:
The biggest concern I have is that the algorithm is too similar to Bayesian optimization using UCB. The author tried to add more technical development such as the look-ahead algorithm. But the main difference from the proposed algorithm and GIBO is that the next iterate is found via UCB. In doing so, it's hard to argue the proposed algorithm is a gradient-based method anymore. And the proposed algorithm becomes too familiar.

QUESTIONS:
1. Please proofread the paper to check grammar and spelling. For example, line 151, 161.
2. Line 171 "more likely to be lower than" should be "more likely to have a smaller value than".
3. Line 269 "minimize the minimum point?"
4. The gradient descent methods have a rich literature in how to choose the step size. It's not always $1/L$. The author should clarify on that point.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an extension of a local Bayesian optimization strategy, specifically targeting methods based on gradient information (GIBO). GIBO-type algorithms operate through two stages: an exploitation step and an exploration step. The paper introduces two novel algorithms within this framework.\n\nIn GIBO, the exploitation step is performed using gradient descent. The authors identify that gradient descent can be viewed as minimizing a quadratic upper bound. They further note that minimizing the tighter upper confidence bounds (UCB) can exploit more information from the GP surrogate model. Consequently, they replace the gradient descent step in GIBO with UCB minimization, resulting in a new algorithm named MinUCB. The paper provides a convergence analysis for MinUCB.\n\nThe second algorithm modifies the exploration phase of GIBO to optimally query points using a look-ahead strategy, aligning with the new exploitation approach. Both algorithms are empirically evaluated on some established benchmarks.\n\nSTRENGTHS:\n- The paper presents a clear, well-motivated, and relevant contribution to the field of local BO.\n- The related work is thoroughly discussed, situating the paper's contribution within the context of existing research.\n- The theoretical analysis is interesting and significantly strengthens the algorithmic contribution. \n- The proposed algorithms improve the state-of-the art for GIBO-style algorithms.\n\nWEAKNESSES:\n## Main aspects\n\n**Locality**: In contrast to GIBO and MDP, the exploitation step is standard UCB. Therefore $x_{t+1}$ does not need to be in the local region around $x_{t}$. In fact this will only be the case if the algorithm is initialized in a region that is lower then the mean. I hypothesize that the \"exploitation\" step of MinUCB will explore globally until it finds a \"promising region\". I don't see this as a fundamental problem with the algorithm and may even allow MinUCB to switch between local and global exploration, which could be beneficial and avoid getting stuck in local optima. However, the paper does not discuss this behavior or explain why we should expect MinUCB to exhibit local-only exploration behavior. Including such a discussion and empirical investigations on how local MinUCB actually is could significantly improve the paper.\n\n**Empirical evaluation**: The empirical evaluation is  limited, missing important details, and is generally not well executed.\n\n- The algorithm is evaluated on a subset of the problems in Nguyen et al. and does not introduce any new benchmarks. The low number of problems makes it hard to assess how efficient the algorithm really is, especially since the Hopper experiments seem problematic due to state normalization. No method is able to \"solve\" the hopper task, and the achieved reward is very low.\n- There are unexplained differences in the results on the synthetic benchmarks compared to those reported by Nguyen et al., where MDP and GIBO have similar performance after 500 evaluations. In the new results, GIBO outperforms MDP.\n- There are important details missing:\n  1. Batch sizes\n  2. Hyperpriors and hyperparameters\n- The code is not available for review, which hinders the evaluation of the experiments at the time of review.\n- The meaning of the error bars changes between plots.\n- A sensitivity study is missing. How sensitive is the algorithm to the choices of the newly introduced hyperparmeter, namely $\\beta$ in the UCB?\n\n\n**Limitations**: The paper does not discuss the limitations of the proposed method and its evaluation.\n\n\n## Minor\n\n- In Theorem 1, $n$ is never introduced.\n- The paper often cites arXiv preprints when peer-reviewed versions of the articles are available.\n- There are some grammar and spelling mistakes/inaccuracies:\n  - \"..solve _the_ high dimensional black-box.. problem\" should be plural: \"..solve high dimensional black-box.. problems\" (there are many such problems).\n  - \"Gradient descent\" should be lower case.\n  - \"large estimate to ensure _the_ convergence\" should be: \"large estimate to ensure convergence\" (we are talking about general convergence behavior).\n- The typesetting of \"L-smooth\" is off, perhaps due to being typed in math mode.\n- Some words/abbreviations are typeset in math mode (e.g., in equation 3, \"UCB\"; in equation 5, \"trace\").\n- Equation 1 is hard to parse due to the nested inequality and scalar product symbols.\n- Figure 3: \"LA-MinUCB has consistently optimal performance.\" In what sense is the performance optimal? This claim seems premature and needs clarification.\n\nQUESTIONS:\n1) In line 54 and later in the evaluation the authors claim \"MPD ... can exhibits numerical instability..\" but do not follow up on this claim. What numerical instabilities were observed?\n2) Why limit the experiments to a subset of the problems in Nguyen et al.?\n3) Why choose constant batch sizes and beta when the theory predicts choices with guaranteed convergence? Why deviate from the theoretically obtained values?\n4) Why are the results on the synthetic benchmarks significantly different from those reported in Nguyen et al.?\n5) How where the batch sizes chosen?\n\nLIMITATIONS:\nThe limitations of the algorithm are not discussed and need to be addressed. For instance, the paper derives convergence for specific choices of beta and batch sizes, but chooses different values in the empirical results. This discrepancy raises the question of whether there is a gap between theory and practice. Discussing these limitations and the rationale behind the chosen parameters in the experiments would provide a more comprehensive understanding of the algorithm's performance and applicability.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}