SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Roman Garnett
Washington University in St. Louis garnett@wustl.edu
A recent development in Bayesian optimization is the use of local optimization strategies, which can deliver strong empirical performance on high-dimensional problems compared to traditional global strategies. The "folk wisdom" in the literature is that the focus on local optimization sidesteps the curse of dimensionality; however, little is known concretely about the expected behavior or convergence of Bayesian local optimization routines. We first study the behavior of the local approach, and find that the statistics of individual local solutions of Gaussian process sample paths are surprisingly good compared to what we would expect to recover from global methods. We then present the first rigorous analysis of such a Bayesian local optimization algorithm recently proposed by Müller et al. (2021), and derive convergence rates in both the noiseless and noisy settings.
Bayesian optimization (BO) [11] is among the most well studied and successful algorithms for solving black-box optimization problems, with wide ranging applications from hyperparameter tuning [23], reinforcement learning [19, 27], to recent applications in small molecule [18, 26, 7, 13, 12] and sequence design [25]. Despite its success, the curse of dimensionality has been a long-standing challenge for the framework. Cumulative regret for BO scales exponentially with the dimensionality of the search space, unless strong assumptions like additive structure [15] are met. Empirically, BO has historically struggled on optimization problems with more than a handful of dimensions.
Recently, local approaches to Bayesian optimization have empirically shown great promise in addressing high-dimensional optimization problems [10, 19, 20, 27, 17]. The approach is intuitively appealing: a typical claim is that a local optimum can be found relatively quickly, and exponential sample complexity is only necessary if one wishes to enumerate all local optima. However, these intuitions have not been fully formalized. While global Bayesian optimization is well studied under controlled assumptions [24, 3], little is known about the behavior or convergence properties of local Bayesian optimization in these same settings — the related literature focuses mostly on strong empirical results on real-world applications.
In this paper, we begin to close this gap by investigating the behavior of and proving convergence rates for local BO under the same well-studied assumptions commonly used to analyze global BO. We divide our study of the properties of local BO in two questions:
Q1 Using common assumptions (e.g., when the objective function is a sample path from a GP with known hyperparameters), how good are the local solutions that local BO converges to?
Figure 1: The unreasonable effectiveness of locally optimizing GP sample paths. (Top row): Distributions of local solutions found when locally optimizing GP sample paths in various numbers of dimensions, with varying amounts of noise. (Bottom left): The minimum sample complexity of grid search required to achieve the median value found by GIBO ( $\sigma = 0$ ) in expectation. (Bottom middle, right): The performance of global optimization algorithms GP-UCB and random search in this setting. See §3 for details.
Q2 Under these same assumptions, is there a local Bayesian optimization procedure that is guaranteed to find a local solution, and how quickly can it do so?
For Q1, we find empirically that the behavior of local BO on "typical" high dimensional Gaussian process sample paths is surprising: even a single run of local BO, without random restarts, results in shockingly good objective values in high dimension. Here, we may characterize "surprise" by the minimum equivalent grid search size necessary to achieve this value in expectation. See Figure 1 and more discussions in §3.
For Q2, we present the first theoretical convergence analysis of a local BO algorithm, GIBO as presented in [19] with slight modifications. Our results bring in to focus a number of natural questions about local Bayesian optimization, such as the impact of noise and the increasing challenges of optimization as local BO approaches a stationary point.
1. We provide upper bounds on the uncertainty of the gradient when actively learned using the GIBO procedure described in [19] in both the noiseless and noisy settings.
2. By relating these uncertainty bounds to the bias in the gradient estimate used at each step, we translate these results into convergence rates for both noiseless and noisy functions.
3. We investigate the looseness in our bound by studying the empirical reduction in uncertainty achieved by using the GIBO policy and comparing it to our upper bound.
4. We empirically study the quality of individual local solutions found by a local Bayesian optimization, a slightly modified version of GIBO, on GP sample paths.
Bayesian optimization. Bayesian optimization (BO) considers minimizing a black-box function  $f$ :
$$
\underset {\mathbf {x} \in \mathcal {X}} {\text {m i n i m i z e}} f (\mathbf {x})
$$
through potentially noisy function queries  $y = f(\mathbf{x}) + \varepsilon$ . When relevant to our discussion, we will assume an i.i.d. additive Gaussian noise model:  $\varepsilon \sim \mathcal{N}(0,\sigma^2)$ . To decide which inputs to
be evaluated, Bayesian optimization estimates  $f$  via a surrogate model, which in turn informs a policy (usually realized by maximizing an acquisition function  $\alpha(\mathbf{x})$  over the input space) to identify evaluation candidates. Selected candidates are then evaluated, and the surrogate model is updated with this new data, allowing the process to repeat with the updated surrogate model.
Gaussian processes. Gaussian processes (GP) are the most commonly used surrogate model class in BO. A Gaussian process  $\mathcal{GP}(\mu, k)$  is fully specified by a mean function  $\mu$  and a covariance function  $k$ . For a finite collection of inputs  $\mathbf{X}$ , the GP induces a joint Gaussian belief about the function values,  $f(\mathbf{X}) \sim \mathcal{N}(\mu(\mathbf{X}), k(\mathbf{X}, \mathbf{X}))$ . Standard rules about conditioning Gaussians can then be used to condition the GP on a dataset  $\mathcal{D}$ , resulting in an updated posterior process reflecting the information contained in these observations:
$$
f _ {\mathcal {D}} \sim \mathcal {G P} \left(\mu_ {\mathcal {D}}, k _ {\mathcal {D}}\right). \tag {1}
$$
Existing bounds for global BO. Srinivas et al. [24] proved the first sublinear cumulative regret bounds of a global BO algorithm, GP-UCB, in the noisy setting. Their bounds have an exponential dependency on the dimension, a result that is generally unimproved without assumptions on the function structure like an additive decomposition [15]. This exponential dependence can be regarded as a consequence of the curse of dimensionality. In one dimension, Scarlett [21] has characterized the optimal regret bound (up to a logarithmic factor), and GP-UCB is near-optimal in the case of the RBF kernel. In the noiseless setting, improved convergence rates have been developed under additional assumptions. For example, De Freitas et al. [6] proved exponential convergence by assuming  $f$  is locally quadratic in a neighborhood around the global optimum, and Kawaguchi et al. [16] proved an exponential simple regret bound under an additional assumption that the optimality gap  $f(\mathbf{x}) - f^{*}$  is bounded by a semi-metric.
Gaussian process derivatives. A key fact about Gaussian process models that we will use is that they naturally give rise to gradient estimation. If  $f$  is Gaussian process distributed,  $f \sim \mathcal{N}(\mu, k)$ , and  $k$  is differentiable, then the gradient process  $\nabla f$  also is also (jointly) distributed as a GP. Noisy observations  $\mathbf{y}$  at arbitrary locations  $\mathbf{X}$  and the gradient measured at an arbitrary location  $\mathbf{x}$  are jointly distributed as:
$$
\left( \begin{array}{c} \mathbf {y} \\ \nabla f (\mathbf {x}) \end{array} \right) \sim \mathcal {N} \Bigg ((\mu (\mathbf {X}) \atop \nabla \mu (\mathbf {x}) \Bigg), \left( \begin{array}{c c} k (\mathbf {X}, \mathbf {X}) + \sigma^ {2} \mathbf {I} & k (\mathbf {X}, \mathbf {x}) \nabla^ {\top} \\ \nabla k (\mathbf {x}, \mathbf {X}) & \nabla k (\mathbf {x}, \mathbf {x}) \nabla^ {\top} \end{array} \right) \Bigg).
$$
This property allows probabilistic modeling of the gradient given noisy function observations. The gradient process conditioned on observations  $\mathcal{D}$  is distributed as a Gaussian process:
$$
\nabla f \mid \mathcal {D} \sim \mathcal {N} (\nabla \mu_ {\mathcal {D}}, \nabla k _ {\mathcal {D}} \nabla^ {\top}).
$$
3 How Good Are Local Solutions?
Several local Bayesian optimization algorithms have been proposed recently as an alternative to global Bayesian optimization due to their favourable empirical sample efficiency [10, 19, 20]. The common idea is to utilize the surrogate model to iteratively search for local improvements around some current input  $\mathbf{x}_t$  at iteration  $t$ . For example, Eriksson et al. [10] centers a hyper-rectangular trust region on  $\mathbf{x}_t$  (typically taken to be the best input evaluated so far), and searches locally within the trust region. Other methods like those of Muller et al. [19] or Nguyen et al. [20] improve the solution  $\mathbf{x}_t$  locally by attempting to identify descent directions  $\mathbf{d}_t$  so that  $\mathbf{x}_t$  can be updated as  $\mathbf{x}_{t + 1} = \mathbf{x}_t + \eta_t\mathbf{d}_t$ .
Before choosing a particular local BO method to study the convergence of formally, we begin by motivating the high level philosophy of local BO by investigating the quality of individual local solutions in a controlled setting. Specifically, we study the local solutions of functions  $f$  drawn from Gaussian processes with known hyperparameters. In this setting, Gaussian process sample paths can be drawn as differentiable functions adapting the techniques described in Wilson et al. [29]. Note that this is substantially different from the experiment originally conducted in Müller et al. [19], where they condition a Gaussian process on 1000 examples and use the posterior mean as an objective.
To get a sense of scale for the optimum, we first analyze roughly how well one might expect a simple grid search baseline to do with varying sample budgets. A well known fact about the order statistics of Gaussian random variables is the following:
Algorithm 1: A Local Bayesian Optimization Algorithm
Input: A black-box function  $f$  over a convex compact set  $\mathcal{X}$ .  
for  $t = 1,2,\dots ,T$  do  
     $\mathbf{X} = \operatorname{argmin}_{\mathbf{Z}} \alpha_{\mathrm{trace}}(\mathbf{x}_t,\mathbf{Z})$  where  $\mathbf{Z} \in \mathbb{R}^{b_t \times d}$  //  $b_t$  is the batch size  
    evaluate the black-box function  $f$  on  $\mathbf{X}$ , obtaining (possibly noisy) measurements  $\mathbf{y}$ $\mathcal{D}_t = \mathcal{D}_{t-1} \cup (\mathbf{X},\mathbf{y})$  // add  $(\mathbf{X},\mathbf{y})$  to the training data  
     $\mathbf{x}_{t+1} = \mathbf{x}_t - \eta_t\nabla \mu_{\mathcal{D}_t}(\mathbf{x}_t)$   
end
Remark 1 (e.g., [8, 14]). Let  $X_{1}, X_{2}, \dots, X_{n}$  be (possibly correlated) Gaussian random variables with marginal variance  $s^2$ , and let  $Y = \max \{X_i\}$ . Then the expected maximum is bounded by
$$
\mathbb {E} [ Y ] \leq s \sqrt {2 \log n}.
$$
This result directly implies an upper bound on the expected maximum (or minimum by symmetry) of a grid search with  $n$  samples. The logarithmic bound results in an optimistic estimate of the expected maximum (and the minimum): as  $n \to \infty$  the bound goes to infinity as well, whereas the maximum (and the minimum) of a GP is almost surely finite due to the correlation in the covariance.
With this, we now turn to evaluating the quality of local solutions. To do this, we optimize 50 sample paths starting from  $\mathbf{x} = \mathbf{0}$  from a centered Gaussian process with a RBF kernel (unit outputscale and unit lengthscale) using a variety of observation noise standard deviations  $\sigma$ . We then run iterations of local Bayesian optimization (as described later in Algorithm 1) to convergence or until an evaluation budget of 5000 is reached. In the noiseless setting  $(\sigma = 0)$ , we modify Algorithm 1 to pass our gradient estimates to BFGS rather than applying the standard gradient update rule for efficiency.
We repeat this procedure for  $d \in \{1, 5, 10, 20, 30, 50\}$  dimensions and  $\sigma \in \{0, 0.05, 0.2\}$  and display results in Figure 1. For each dimension, we plot the distribution of single local solutions found from each of the 50 trials as a box plot. In the noiseless setting, we find that by  $d = 50$ , a single run of local optimization (i.e., without random restarts) is able to find a median objective value of -12.9, corresponding to a grid size of at least  $n = 10^{36}$  to achieve this value in expectation!
For completeness, we provide results of running GP-UCB and random search also with budgets of 5000 and confirm the effectiveness of local optimization in high dimensions.
4 A Local Bayesian Optimization Algorithm
The study above suggests that, under assumptions commonly used to theoretically study the performance of Bayesian optimization, finding even a single local solution is surprisingly effective. The natural next question is whether and how fast we can find them. To understand the convergence of local BO, we describe a prototypical local Bayesian optimization algorithm that we will analyze in Algorithm 1, which is nearly identical to GIBO as described in [19] with one exception: Algorithm 1 allows varying batch sizes  $b_{t}$  in different iterations.
Define  $k_{\mathcal{D} \cup \mathbf{Z}}(\mathbf{x}_t, \mathbf{x}_t) = k_{\mathcal{D}}(\mathbf{x}_t, \mathbf{x}_t) - k_{\mathcal{D}}(\mathbf{x}_t, \mathbf{Z})(k_{\mathcal{D}}(\mathbf{Z}, \mathbf{Z}) + \sigma^2 \mathbf{I})^{-1} k_{\mathcal{D}}(\mathbf{Z}, \mathbf{x}_t)$ . Namely,  $k_{\mathcal{D} \cup \mathbf{Z}}(\cdot, \cdot)$  is the posterior covariance function conditioned on the data  $\mathcal{D}$  as well as new inputs  $\mathbf{Z}$ . Crucially, observe that the posterior covariance of a GP does not depend on the labels  $\mathbf{y}$ , hence the compact notation  $k_{\mathcal{D} \cup \mathbf{Z}}$ . The acquisition function  $\alpha_{\mathrm{trace}}$  of GIB0 is defined as
$$
\alpha_ {\text {t r a c e}} (\mathbf {x}, \mathbf {Z}) = \operatorname {t r} \left(\nabla k _ {\mathcal {D} \cup \mathbf {Z}} \left(\mathbf {x} _ {t}, \mathbf {x} _ {t}\right) \nabla^ {\top}\right), \tag {2}
$$
which is the trace of the posterior gradient covariance conditioned on the union of the training data  $\mathcal{D}$  and candidate designs  $\mathbf{Z}$ .
The posterior gradient covariance  $\nabla k_{\mathcal{D}}(\mathbf{x}_t,\mathbf{x}_t)\nabla^\top$  quantifies the uncertainty in the gradient  $\nabla f(\mathbf{x}_t)$ . By minimizing the acquisition function  $\alpha_{\mathrm{trace}}(\mathbf{x}_t,\mathbf{Z})$  over  $\mathbf{Z}$ , we actively search for designs  $\mathbf{Z}$  that minimizes the one-step look-ahead uncertainty, where the uncertainty is measured by the trace of the posterior covariance matrix. After selecting a set of locations  $\mathbf{Z}$ , the algorithm queries the (noisy) function values at  $\mathbf{Z}$ , and updates the dataset. The algorithm then follows the (negative) expected gradient  $\nabla \mu_{\mathcal{D}_t}(\mathbf{x}_t)$  to improve the current iterate  $\mathbf{x}_t$ , and the process repeats.
5 Convergence Results
In light of the strong motivation we now have for local Bayesian optimization, we will now establish convergence guarantees for the simple local Bayesian optimization routine outlined in the previous section, in both the noiseless and noisy setting. We first pause to establish some notations and a few assumptions that will be used throughout our results.
Notation. Let  $\mathcal{X} \subset \mathbb{R}^d$  be a convex compact domain (e.g.,  $\mathcal{X} = [0,1]^d$ ). Let  $\mathcal{H}$  be a reproducing kernel Hilbert space (RKHS) on  $\mathcal{X}$  equipped with a reproducing kernel  $k(\cdot, \cdot)$ . We use  $\|\cdot\|$  to denote the Euclidean norm and  $\|\cdot\|_{\mathcal{H}}$  to denote the RKHS norm. The minimum of a continuous function  $f$  defined on a compact domain  $\mathcal{X}$  is denoted as  $f^{*} = \min_{\mathbf{x} \in \mathcal{X}} f(\mathbf{x})$ .
In our convergence results, we make the following assumption about the kernel  $k$ . Many commonly used kernels satisfy this assumption, e.g., the RBF kernel and Matérn kernel with  $\nu > 2$ .
Assumption 1. The kernel  $k$  is stationary, four times continuously differentiable and positive definite.
The following is a technical assumption that simplifies proofs and convergence rates.
Assumption 2. The iterates  $\{\mathbf{x}_t\}_{t=1}^{\infty}$  stay in the interior of the domain  $\mathcal{X}$ .
Intuitively, with the domain  $\mathcal{X}$  large enough and an appropriate initialization, the iterates most likely always stay in the interior. In the case when the iterate  $\mathbf{x}_{t + 1}$  slides out of the domain  $\mathcal{X}$ , we can add a projection operator to project it back to  $\mathcal{X}$ , which will be discussed in §5.2.
We also define a particular (standard) smoothness measure that will be useful for proving convergence.
Definition 1 (Smoothness). A function  $f$  is  $L$ -smooth if and only if for all  $\mathbf{x}_1, \mathbf{x}_2 \in \mathcal{X}$ , we have:
$$
\left\| \nabla f \left(\mathbf {x} _ {1}\right) - \nabla f \left(\mathbf {x} _ {2}\right) \right\| \leq L \| \mathbf {x} _ {1} - \mathbf {x} _ {2} \|.
$$
Next, we define the notion of an error function, which quantifies the maximum achievable reduction in uncertainty about the gradient at  $\mathbf{x} = 0$  using a set of  $b$  points  $\mathbf{Z}$  and no other data, i.e., the data  $\mathcal{D}$  is an empty set in the acquisition function (2).
Definition 2 (Error function). Given input dimensionality  $d$ , kernel  $k$  and noise standard deviation  $\sigma$ , we define the following error function:
$$
E _ {d, k, \sigma} (b) = \inf  _ {\mathbf {Z} \in \mathbb {R} ^ {b \times d}} \operatorname {t r} \left(\nabla k (\mathbf {0}, \mathbf {0}) \nabla^ {\top} - \nabla k (\mathbf {0}, \mathbf {Z}) \left(k (\mathbf {Z}, \mathbf {Z}) + \sigma^ {2} \mathbf {I}\right) ^ {- 1} k (\mathbf {Z}, \mathbf {0}) \nabla^ {\top}\right). \tag {3}
$$
5.1 Convergence in the Noiseless Setting
We first prove the convergence of Algorithm 1 with noiseless observations, i.e.,  $\sigma = 0$ . Where necessary, the inverse kernel matrix  $k_{\mathcal{D}}(\mathbf{Z},\mathbf{Z})^{-1}$  should be interpreted as the pseudoinverse. In this section, we will assume the ground truth function  $f$  is a function in the RKHS with bounded norm. This assumption is standard in the literature [e.g., 3, 24], and the results presented here ultimately extend trivially to the other assumption commonly made (that  $f$  is a GP sample path).
Assumption 3. The ground truth function  $f$  is in  $\mathcal{H}$  with bounded norm  $\| f\|_{\mathcal{H}}\leq B$ .
Because  $k$  is four times continuously differentiable,  $f$  is twice continuously differentiable. Furthermore, on a compact domain  $\mathcal{X}$ ,  $f$  is guaranteed to be  $L$ -smooth for some constant  $L$  (see Lemma 9 for a complete explanation). To prove the convergence of Algorithm 1, we will show the posterior mean gradient  $\nabla \mu_{\mathcal{D}}$  approximates the ground truth gradient  $\nabla f$ . By using techniques in meshless data approximation [e.g., 28, 5], we prove the following error bound, which precisely relates the gradient approximation error and the posterior covariance trace.
Lemma 1. For any  $f \in \mathcal{H}$ , any  $\mathbf{x} \in \mathcal{X}$  and any  $\mathcal{D}$ , we have the following inequality
$$
\left\| \nabla f (\mathbf {x}) - \nabla \mu_ {\mathcal {D}} (\mathbf {x}) \right\| ^ {2} \leq \operatorname {t r} \left(\nabla k _ {\mathcal {D}} (\mathbf {x}, \mathbf {x}) \nabla^ {\top}\right) \| f \| _ {\mathcal {H}} ^ {2}. \tag {4}
$$
Since  $f$  has bounded norm  $\| f\|_{\mathcal{H}}\leq B$ , the right hand side of (4) is a multiple of the posterior covariance trace, which resembles the acquisition function (2). Indeed, the acquisition function (2) can be interpreted as minimizing the one-step look-ahead worst-case gradient estimation error in the RKHS  $\mathcal{H}$ , which justifies GIBO from a different perspective.
Next, we provide a characterization of the error function  $E_{d,k,0}$  under the noiseless assumption  $\sigma = 0$ . This characterization will be useful later to express the convergence rate.
Lemma 2. For  $\sigma = 0$ , the error function is bounded by  $E_{d,k,0}(b) \leq C \max \{0,1 + d - b\}$ , where  $C = \max_{1 \leq i \leq d} \frac{\partial^2}{\partial x_i \partial x_i'} k(\mathbf{0},\mathbf{0})$  is the maximum of the Hessian's diagonal entries at the origin.
Now we are ready to present the convergence rate of Algorithm 1 under noiseless observations.
Theorem 1. Let  $f \in \mathcal{H}$  whose smoothness constant is  $L$ . Running Algorithm 1 with constant batch size  $b_{t} = b$  and step size  $\eta_t = \frac{1}{L}$  for  $T$  iterations outputs a sequence satisfying
$$
\min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {1}{T} (2 L (f (\mathbf {x} _ {1}) - f ^ {*})) + B ^ {2} \cdot E _ {d, k, 0} (b). \tag {5}
$$
As  $T \to \infty$ , the first term in the right hand side of (5) decays to zero, but the second term may not. Thus, with constant batch size  $b$ , Algorithm 1 converges to a region where the squared gradient norm is upper bounded by  $B^2 E_{d,k,0}(b)$  with convergence speed  $\mathcal{O}\left(\frac{1}{T}\right)$ . An important special case occurs when the batch size  $b_t = d + 1$  in every iteration. In this case,  $E_{d,k,0}(b) = 0$  by Lemma 2 and the algorithm converges to a stationary point with rate  $\mathcal{O}\left(\frac{1}{T}\right)$ .
Corollary 1. Under the same assumptions of Theorem 1, using batch size  $b_{t} = d + 1$ , we have
$$
\min  _ {1 \leq t \leq T} \| \nabla f (\mathbf {x} _ {t}) \| ^ {2} \leq \frac {1}{T} \big (2 L \big (f (\mathbf {x} _ {1}) - f ^ {*} \big) \big).
$$
Therefore, the total number of samples  $n = \mathcal{O}(dT)$  and the squared gradient norm  $\| \nabla f(\mathbf{x}_t)\| ^2$  converges to zero at the rate  $\mathcal{O}(d / n)$ .
We highlight that the rate is linear with respect to the input dimension  $d$ . Thus, as expected, local Bayesian optimization finds a stationary point in the noiseless setting with significantly better non-asymptotic rates in  $d$  than global Bayesian optimization finds the global optimum. Of course, the fast convergence rate of Algorithm 1 is at the notable cost of not finding the global minimum. However, as Figure 1 demonstrates and as discussed in §3, a single stationary point may already do very well under the assumptions used in both analyses.
We note that the results in this section can be extended to the GP sample path assumption as well. Due to space constraint, we defer the result to Theorem 4 in the appendix.
5.2 Convergence in the Noisy Setting
We now turn our attention to the more challenging and interesting setting where the observation noise  $\sigma > 0$ . We analyze convergence under the following GP sample path assumption, which is commonly used in the literature [e.g., 24, 6]:
Assumption 4. The objective function  $f$  is a sample from a  $GP$ ,  $f \sim \mathcal{GP}(\mu, k)$  with known mean function, covariance function, and hyperparameters. Observations of the objective function  $y$  are made with added Gaussian noise of known variance,  $y = f(\mathbf{x}) + \varepsilon$ , where  $\varepsilon \sim \mathcal{N}(0, \sigma^2)$ .
Since the kernel  $k$  is four times continuously differentiable,  $f$  is almost surely smooth.
Lemma 3. For  $0 < \delta < 1$ , there exists a constant  $L > 0$  such that  $f$  is  $L$ -smooth w.p. at least  $1 - \delta$ .
Challenges. The observation noise introduces two important challenges. First, by the GP sample path assumption, we have: