Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes
Authors: Yunyue Wei
Vincent Zhuang
Saraswati Soedarmadji
Yanan Sui
Venue: NeurIPS 2024 poster (2024)
Keywords: Scalable Bayesian optimization
Sparse Gaussian process
Primary area: probabilistic_methods

Abstract: Bayesian optimization is an effective technique for black-box optimization, but its applicability is typically limited to low-dimensional and small-budget problems due to the cubic complexity of computing the Gaussian process (GP) surrogate. While various approximate GP models have been employed to scale Bayesian optimization to larger sample sizes, most suffer from overly-smooth estimation and focus primarily on problems that allow for large online samples.  In this work, we argue that Bayesian optimization algorithms with sparse GPs can more efficiently allocate their representational power to relevant regions of the search space. To achieve this, we propose focalized GP, which leverages a novel variational loss function to achieve stronger local prediction, as well as FocalBO, which hierarchically optimizes the focalized GP acquisition function over progressively smaller search spaces. Experimental results demonstrate that FocalBO can efficiently leverage large amounts of offline and online data to achieve state-of-the-art performance on robot morphology design and to control a 585-dimensional musculoskeletal system.

Scalable Bayesian Optimization via Focalized Sparse Gaussian Processes
Yunyue Wei', Vincent Zhuang", Saraswati Soedarmadji', Yanan Sui!
Tsinghua University
2 Google DeepMind
wei http000mmmsss.ssnnniiaoaa edu.
vincentzhhunggggoolle... com
chenxuying24dmails: www.guuu...iiii..t.o.m edu. ysui edu. cn
Abstract
Bayesian optimization is an effective technique for black box optimization, but its applicability is typically limited to low-dimensional and small- budget problems due to the cubic complexity of computing the Gaussian process (GP) surrogate. While various approximate GP models have been employed to scale Bayesian optimization to larger sample sizes, most suffer from overly- smooth estimation and focus primarily on problems that allow for large online samples. In this work, we argue that Bayesian optimization algorithms with sparse GPs can more efficiently allocate their representational power to relevant regions of the search space. To achieve this, propose focalized GP, which leverages a novel variational loss function to achieve stronger local prediction, well as FOCALBO, which hierarchically optimizes the focalized GP acquisition function over progressively smaller search spaces. Experimental results demonstrate that FOCALBO can efficiently leverage large amounts of offline and online data to achiove state-of-the-art performance on robot morphology design and to control 585-dimensional musculoskcletal system.
Introduction
Bayesian Optimization (BO) is powerful approach for solving black-box optimization prob- lems, demonstrating notable success in hyperparameter tuning]]] reinforcement learning[2] 3. and scientific discovery A The elficacy of BO is attributed to its ability to model the known objective function using a surrogate model and to strategically select the next sample position by optimizing an acquisition function. Among the surrogate models, Gaussian Processes (GPs) [ tification. However, the computation with the number of data points which can severely restrict the applicability of BO in handling large datascts. This poses significant challenge for real-world applications with high-dinensional and heterogeneous function landscapes such as those in robot control, which often necessitate a substantial amount of data to adequately explore the vast search space. To extend the scope of BO to accommodate larger dalasets (from horizooizzon online trials and/or pre-collected oflline datasets (i), is imperative employ surrogate models that offer enhanced computational efficiency.
are usually favored due to their flexibility and robust uncertainty quan-
the posterior GP covariance matrix scales as ((n3)
Using sparse GP models is popular method for reducing the computational cost of BO. Sparse GPs accomplish this by learning an approximation of the full GP, either by using
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

subset of data ensemble of local models[ or variational inference 9. However, classical sparse GP models are typically tailored for regression tasks, and therefore are designed to fit to the entire function landscape. Given limited representational resources, the resulting posterior is likely to be overly smooth, which may negatively impact the performance of BO. This issue is exacerbated in the high-dinensional setting, in which accurately fitting the entire domain a far more challenging task. As such, several works have proposed strategies to improve BO performance with sparse GP models by focusing promising regions [T0] [ or advanced sparse GP models] [2] However, most of their empirical evaluations are only conducted under large online sample setting in low-dimensional problems with fewer than 20 variables. It unclear whether existing methods can be generalized to large offline data or high-dimensional setting.
In this work, we explore the application of sparse Gaussian processes for optimizing high- dimensional problems with large offline (and optionally large online) datasets. We argue that by iteratively identifying key sub-regions of the input space and focusing the modeling capacity these areas, can enhance the modeling fidelity of the sparse GP in regions that are most relevant, thereby improving the overall performance of the Bayesian optimization algorithm. To this end, we propose a novel loss function to train variational sparse GP model (focalized GP) that emphasizes the fitting of local functional landscapes through weighting the training data. Along with focalized GP, we design hierachical algorithm, FOCALBO, to propose sample points via acquisition function optimization across varying scales of the search space. Experimental results demonstrate that FOCALBO can improve upon commonly used acquisition functions in optimizing heterogeneous functions and can effectively utilize large offline datasets for efficient high-dimensional optimization. Furthermore, we showcase that FOCALBO can efficiently optimize a policy with 585 paramcters to control a musculoskeletal system, leveraging both oflline and online data. To the best of our knowledge, FoCALBO is the first sparse GP-based Bayesian optimization algorithm capable of efficiently optimizing high-dimensional problems under both large online sample and large offline data settings.
Our main contributions: 1) We design FOCALBO, which employs hierarchical acquisition optimization strategy to achieve efficient optimization over digh-dimensional problems with heterogeneous structure with limited representation capability. 2) Experimental results demonstrate the superior performance of FOCALBO in leveraging large offline datasets for online optimization, and its capability to optimize high-dimensional musculoskcletal system control problems involving over 500 variables.
2
Related Work
2.1 Sparse Gaussian processes
Scaling Gaussian processes to large datasets is an important topic 13 It can be broadly divided into global approximation strategics and local approximation strategics.
Global sparse GPs perform distillation over the whole datasct to approximate the exponsive full covariance matrix with sparse represenlation. Several methods aim to choose subset of representative training points from the whole dataset, and use the corresponding covariance matrix in place of the full covariance 7 [5 LG Sparse kernels aim al removing Sparse approximation methods use inducing variables to learn low-rank representation of full
uncorrelated entries in the full covariance to obtain a compact matrix I7
IS
19 20
26 27. Stochastic variational Gaussian process (SVGP) is a popular sparse GP method which employs variational inference to learn inducing variables and kernel hyperparameters jointly and enable training using stochastic gradiont descent from mini-batch data 25 Recently, nearest neighbor information has also been used to further improve the scalability of sparse GP over massive amount of data 28 29
covariance matrix 21 22 23 24
In contrast, local sparse GPs divide the entire dataset and employ local GPs trained from different data subsets to approximate the full GP. For a given test. set, the prediction can be extracted from one of the local GPs 80
BI, mixture of GPs 82 [3) product of GPs
35
2

---

2.2 Scalable Bayesian optimization
Recent works have proposed modifications to sparse GPss for Bayesian optimization. Sparse GP has been used to determine the search region where local GPs are used to determine the next samples 36]. Weightted-ppdate online Gaussian processes (WOGP) was developed to select subset of training points to approximate high performing regions of the input space][] IMP-DPP is motivated by similar observation and uses a weighted Determinantal Point Process to select training points as inducing variables for the SVGPIII However, their proposed selection strategies require sequentially evaluating every training point, which can be computationally very expensive with large offline datasets. Combining SVGP with Thompson sampling has the same order regret as standard Thompson sampling methodB1 Online variational conditioning (OVC) was proposed to efficiently conditioning SVGPs in online setting, enabling using look-ahead acquisition functions B8 Vecchia approximation of GP was also applied 39 for Bayesian optimization, with improved performance compared to prior works 12 A concurrent work 40 aims at improving the acquisition optimization performance based target-aware Bayesian inference 41.
and random forest 44 can also be used as BO surrogate model to circumvent the cubic complexity of GP. Ensemble Bayesian optimization utilizes the addictive function structure and uses ensembles of addictive GPs parallel to achieve scalability [5] Trrst Region Bayesian optimizatiin (TuRBBO) and its variants uses exact GP to optimize over local regions, and employs a restart mechanism to achieve large mumber of evaluation, which is representative line of works in nigh-dimensional Bayesian optimization 46 47 48 TuRBO can also be combined with sparse GP models to further
Besides sparse GPs, Neural network 42 1
enhance the scalability 49 50
3 Background
3.1 Bayesian optimization
For unknown objective function f. Bayesian optimization aims to solve maXæex fax over input space X' E J0. 1j BO mainly consists of two components: surrogate model to approximate the objective function, and an acquisition function a to decide the next sample position based surrogate model.
Gaussian process a commonly used surrogate model. Consider a given datasct D (X, y) where X (I1, ..., Iτ) are input locations and (Y1,.., ...: :.) are associated noisy observations of f(X). We assume the observation noise to be independent Gaussian, i.c. f(;;) + n. ~ A0, o²). Using GP with kernel function K, the function distribution f. lest positions X. (I.,II...,,.., "². a multivariate Gaussian:
pf. X, y) = Nff, Xx.x|Xxx toll"'y. Xx.x- -x. XxxxxXxx + +oll-"Xxx.)
(1)
where K is the covariance matrix between subscript inputs. With the posterior distribution given D, the next sample point is the maximum position of the acquisition function: T++1 = max a()),M),, where M, is the GP model fitted on datasct collected at time step t. Common-used choice a includes upper confidence bound(UCB,, []]], expected improvement
(EI, 52 and Thompson sampling (TS, 53 The inner optimization problem is usually volved by grid search, evolutionary algorithms 5d or gradient- methods bsd When the
online sample budget is large, batch optimization is commonly used to evaluate multiple inputs in parallel56).
3.2 Variational Gaussian process
For predictive distribution conditioned on given dataset of size f, the computational complexity of exact. Gaussian process is O(1)) for each test. position due to the inverse the covariance matrix Kxx, which is expensive for large scale datasels with more than few thousand points. common used strategy to approximate full GP regression using sparse GPs. In sparse GP, < inducing variables (u,,... Um characterized by inducing inputs
3

---

Z = (Z1...., are introduced to approximate the covariance matrix of the full GP. In this section, we focus on sparse GP derived from variational inference.
Variational GP considers the joint latent prior
Kxx Kzx
Kx2 Kzz
pff, u)
0,
u
where f f=((()))...,.a.:.)))) A variational distribution g(u) Nu m, is used to approximate the posterior over inducing variables using the exact conditional distribution of given that is, (f, p(f u)g(u). The posterior of can be computed by marginalizing u with analytic form:
fax)))
q(f) - f p(f ()q(u)du = N(f Am,Kxx - (KZz S)A):
(3)
where A Zzz The variational parameters Z, m, S are optimized by maximizing the Evidence Lower Bound (ELBO) which can be written in the following formulation 2
6 - - Σ Bellelylloggp f(x))) KLlg(u) ll p(u)] LLL + LKL
where KL[- -] is the KL divergence between two distributions. The ELBO breaks into data likelihood term which factorizod over training data, and a KL divergence term which can be computed in closed form. The factorization over data allows optimization via stochastic gradient descent (SGD), reducing the computational complexity O(m²).
4 Focalized Gaussian Process for Bayesian Optimization
Prior studies about variational sparse GPs mainly designed for regression tasks, where the goal is to fit global training data distribution. In Bayesian optimization, the next sample is determined by the predic- tive function distribution over test positions. Gradient-basee and evolutionary-basee baased ac- quisition function optimization methods em- ploy local search from random starting points find local optimal of the ac- quisition function. Recent works also scale grid search-based optimization mensional space by restricting the search space within local sub-regions 46 Elr All the above procedure would benefit from accurate estimation over sub-region of the input space. Therefore, a sensible way to improve BO performance is to allocate limited computational resources to obtain better prediction over specific search regions instead of the entire input domain.
Training data True function Search region
SVGP
Focalizes
high di-
Figure 1: Performance comparison of focalized GP and SVGP over 1d GP functions. Posteri- shown mean + standard deviation.
We define the search region as the region where the acquisition function is optimized on, which is an axis-aligned hypercube with length 1 = "i,.""",aa""" centered at c:
Se, = (x 1 x c 5..
(5)
When -= (1,-.. ,1)T and c= (0.5, ... 0.5)T, the acquisition optimization is performed over vanilla BO algorithms. the rest of this section, we first. present the derivation of focalized loss function to improve GP prediction over the search region. Then we demonstrate how incorporate our proposed GP model into Bayesian optimization.
the entire input space X, as commonly-used

---

4.1 Focalized evidence lower bound
We recall eq and rewrite the mean estimation (e(x)) and variance estimation (x())) for each test position z.:
(r(x))) Σ k(r,; lix + +
(6)
0x(x))) = k(2,,,
Σ =1
kxx,, x; kishi,.( xzy):
where kij is the (i.j)-th entry of xXxx + = From eq. we can observe that the mean estimation at æ. is a linear combination of observation multiplied by k(x,, æ), and the reduction of variance is a quadratic form the covariance between æ, and training points. Both estimation can be written as linear summations of constant values with kernel function as weight. As mentioned in prior works [57, data points far from the test positions have vanishingly small influence on the predictive distribution with commonly used kernel functions. Utilizing this observation, we propose weight the data likelihood term using the kernel function to focus training over points that contribute to the prediction of the search region:
LWLL Σ wo Br Dogg(us f(x))) W/ max k( D.).
(7)
x. ES..
We use the maximum covariance of æ; to positions in the search region as the corresponding weight to filter out points that have marginally influence to the search region during GP training. In this way, the model can selectively utilize the training data to achieve good local prediction. When using a popular kernel functions such as RBF or Matern kernel, the maximum kernel value is equivalent to finding the nearest point in the search region, which can be easily calculated when the region boundary is axis-aligned as defined in eq. 5
We additionally regularize the sum of weights to make the model focus on improving prediction over search region:
X Sc,l
Ei w; E Scil
Creg
=
1),
X E Sc.1
where X E Sc,l -t Ei= La, ESg, is the number of training points in the search region. The proposed regularization term Lreg encourages accurate local prediction instead of blurred global cstimation, avoiding getting stuck on suboptimal of large kernel lengthscale. Combined with KL loss, our finalized new ELBO as follows:
L2 = LWLL + LKL - Creg.
(9)
Compared the original ELBO loss in SVGP, our proposed function maintains the same computational complexity and does not introduce additional hyperparameters. Our ELBO also reproduces oq. when considering to predict the entire input space X. During the model training, both GP hyperparameters and variational parameters are jointly optimized to obtain focalized GP for Bayesian optimization. Figure shows comparison of focalized GP and SVGP over 1d functions sampled from GP. While SVGP can ouly able to vaguely predict the function, localized GP accurately delineate the function landscape within search region by training with the focalized loss. Our proposed GP model is sensitive to high-performing positions within the search space which contribute to better acquisition optimization. We also systematically compare the GP prediction performance in Appendix [..]] where GP model trained from focalized ELBO consistently achieves good prediction on small size of scarch space.
Theoretical implications of focalized ELBO. Our focalized ELBO can interpreted as soft variant of training a local approximation over datapoints that lie within the search region. Here, we illustrate how local approximations can substantially reduce the KT. divergence of the approximate posterior over the search region, and discuss the effects of tighter
5

---

approximations on BO regret bounds. We focus on providing general theoretical intuition rather than deriving precise bounds due to the lack of existing convergence guarantees for ELBO maximization in the general setting.
Suppose that we know the optimal point lies in some small sub-region of X that contains N' << .N' training points, Corollary 19 in [Ss] shows that given a squared exponential kernel and some assumptions on the inducing point selection, for a fixed number of inducing points the KL-divergence upper bound scales quper-quaarattcallyy in the number of training points. Hence, fitting locally can yield much tighter approximations than fitting globally (e.g. SVGP).
Next, consider the impact of the KL approximation error the optimization regret. Proposition in [58] states that the gap between the means of the approximate and exact posteriors is upper bounded by ((ov7), where Y is an upper-bound on the approximation KI-ddvergence. This has an immediate impact on the regret- - for example, when GP-UCB M is combined with sparse GPs, the confidence bounds must be enlargened by an additive V factor to account for the approximation error. Because the regret bound scales with VTr where 3r is the maximum confidence interval coefficient, having a large approximation error can arbitrarily scale the regret incurred by the algorithm. In order to achieve no additional regret order, the additional approximation error noise must be uniformly bounded (Assumption 4 in 37) )- Although focalized GP cannot guarantee a constant bound, it still directly reduces the regret of the algorithm, where we empirically investigate in Appendix
4.2 Bayesian optimization with focalized GP
One advantage of focalized GP is that it can be easily integrated into existing BO algorithmss To further leverage the strong local modeling properties of focalized GP, we design FOCALBO, hierachical acquisition optimization framework described in Algorithm
Algorithm 1 FOCALBO
Input Initial Dataset Do. Inducing Vari- able Size m, Batch Size B
1: H +
At each BO iteration, FOCALBO iteratively op- timizes the acquisition function over a progres- sively smaller search region via focalized acqui- sition function (FaCALAca) as shown in Algo- rithm 2 The first depth of acquisition optimiza- tion starts with the entire input space X' with == (1,... , and e= (0.5,-. ... 0.5)* (line 1). We train specific focalized GP base on the search region at each round of acquisition optimization (line 4-5). Our framework is compatible with any acquisition function that extracts instant posterior information from the GP and is opti- mized within pre-defined scarch region. After one round of acquisition function optimization, the search space length 1 is halved to focus a smaller search region centered at current best position Øhest (line 6-7). In this way we can obtain more accurate model for decision making, and also relieve the oxereexploration problem when the problem dimension is high b59 One batch of inputs is proposed at each round of optimization, and the final decision is sampled from all proposed inputs via Softmax distribution over their corresponding acquisition func- tion values (line 9). Our hierarchical optimization strategy enables collecting candidates from both global sparse estimation and local focalized prediction, achicving balance between exploration and exploitation with constrained computation power.
2: for 1 = 1122.... do
3: (xr,) 1=1) ( FOCALACq(D (D H. m, B)
4: Observe [1, = (f + -11) 11
5:
D4+1 + (( b(.
6:
1est +- arg.ax,gLL....Bh
7:
if hipms < H then
8:
H H 1
9: else
H f- Н + 1
10: end if
11: end for
The optimization depth H in FOCALACQ controls the degree of utilizing local information from current. best position, where the GP estimate variance decreases with the shrinkage of search space. The best-performing optimization depth is likely problen-dependent (e.g. high- dimensional functions may require higher optimization depths). Therefore in FOCALBO, we

---

propose to antomatically adjust the optimization depth according to the instant optimization performance. At the beginning of the optimization, we initialize the optimization depth as 1, indicating global search of the input space (Algorithm [ line 1). Then we keep track of the depth where the proposed positions are sampled from. If the depth of the best point in this round is less than the current optimization depth H, we reduce H to encourage exploration of the input space, otherwise we increase for better exploitation of Øbest (line 6-10).
Our proposed framework is orthogonal to TuRBO-M |6) but bears some similarities in searching over multiple sub-regions and
A Algorithm 2 FOCALACQ
adaptively adjusting the search region. Our algorithm differs in that TuRBO-M con- structs equal-sized trust regions and fits inde- pendent Exact GP using separated dataset, aiming at searching for different local optima in the search space. By contrast, the search region in FOCALBO is constructed with dif- ferent sizes to make decision based on both global and local information. Our framework allows data sharing across search regions, and the use of focalized GP helps to accu- rately estimate local region with limited rep- resentation. Additionally, FOCALBO does not introduce extra hyperparameters. Finally, we demonstrate in Scction 5 that TuRBO complementary FOCALBO in optimizing high-dinensional problems.
Input Dataset D2-1, Optimization Depth H, Inducing Variable Size m, Batch Size B
1: < , e & (0.5... ... .0.5)²
2: Select current best point Bbest from Dz-1
3: for h h= 1.... .. do
4: Train M using L2 given Se,
5:
{al { & argmax ()
6:
I 1-1/2
7:
C + Tbest
8: end for
9: return
{ hm (he 1 P(i = 12)
"lant Laep""
exp
aa (
hmm m= exp
Experiments
In this section, we extensively evaluate FOCALBO over a variety of tasks. We first use synt hetic functions to showcase the compatibility of FOCALBO in improving commonly-used acquisition functions. Next, we consider the online optimization of robot morphology design that is additionally given a large offline dataset. We also show that FOCALBO is able to optimize very high-dimensional musculoskeletal system control with both a large offline dataset and a large number of online budget. Finally we dig deeper into FOCALBO to analyze how cach of its components contributes to superior optimization performance.
We compare FOCALBO with representative sparse GP models used for Bayesian optimization, including SVGP 25), [OGP[10, and Vecchia GP[]2.. We only run WOGP on senth functions due to its extremely low speed in dealing with the datasets in the remaining tasks. The number of inducing variables in sparse GP models is set as 50 for synthetic functions and as 200 for other tasks. The optimization performances are shown as mean +'r standard error for all considered problems over 10 independent trials.
5.1 Synthetic functions
We select Shekel and Michalewicz as the test functions, which are heterogeneous with both smooth and rigid regions. We also sample functions directly from Gaussian processes to evaluate algorithm performance under full BO assumption. For each function, we choose to use different acquisition functions to optimize: TS optimized by grid search, El optimized by analytic gradient, and probability of improvement (PI) optimized by Monte Carlo gradient [55]. Optimization performances are shown in Figure 2 We observe that FOCALBO significantly improves the performance of all acquisition functions compared to SVGP, and is able to consistently achiove top-tier performance over all problems. In Michalewicz function where a large fraction of the input space is flat, all basclines tend to increase the noise estimation to maintain a ionary prediction, while focalized GP is able to focus on the local search region and successfully optimize the function. Additional experiment with online samples as major data source is shown in Appendix B.2 where FOCALBO still maintains comparable or better performance against baselines.

---

Shekel-4d T5
EI
GP function-5d Pl
3.8
3.5
Function value
3.6
3.2
2.5
3.0
200
400
200
400
200
400
Evaluation number
Evaluation number
Evaluation number
SVGP
WOGP
Vecchia
Figure 2: Optimization performance under different synthetic function and acquisition function. Sparse GP models are trained with 50 inducing variables. The oflline dataset contains 2000 random data points and the online budget 500 with batch size of 10.
5.2 Robot morphology design
We compare FOCALBO to several baselines over robot morphology design task from Design-Bench, which provides large offline dataset with an exact function oracle]ā] The goal of the task is to optimize the morpho- logical structure D'Kitty robot 600 to im- prove the simulation performance under RL controller. While the benchmark is initially designed for oflline model-based optimiza- tion (MBO), it can also be used as an oflline- to-online BO benchmark. In this task, we use the training dataset with 10,000 points and additionally evaluate 128 points on-the- fly with batch size of 4. EI is used as the base acquisition function for better optimiz- ing with small batch size. We also try to combine FOCALBO with TuRBO optimize over the high-dimensional space, with the results shown in Figure 3
0.94
Function value
FocalBO EI
FocalBO TURBO
0.92
SVGP El
SVGP TuRBO
0.90
Vecchia GP
Vecchia GP TuRBO
Evaluation number
Figure 3: Optimization on robot morphology design. Function values are normalized by best and worst values the unseen full dataset.
We observe that
FOCALBO achieves significant. improvement from the initial data while other baselines struggle to obtain performance gain, even combined with TuRBO. FoCALBD with TuRBO effectively extracts information from large offline dataset and the first Ga-based method to achieve top-tier performance reported by prior MBO works 61
5.3 Human musculoskeletal system control
We further apply FOCALBO to control human arm musculoskeletal system 62 for the task of pouring liquid into a cup, as shown in Figure 4 a). To control the musculoskeletal system, we optimize linear policy и |A] x 10], where [ and | 117 the corresponding action and observation dimensions. The action dimension has been reduced from individual muscles to synergetic groups of muscles by applying principled component analysis to sampled action data from an RL agent (Appendix A.6). Although the original control dimension reduced, the remaining 585-dimensional input space is still very high for existing high- dimensional BO algorithms. Thercfore we consider a large offlino-online sctting, where we randomly sample 2000 points from the input space to serve as the offline datasot, and set the online budget as 3000 with batch size of 100. We use Thompson sampling as the base acquisition function. Figure 4bb demonsttrtes that FOCALBO outperforms other baselines, achieving higher maximum reward and faster convergence speed. Our supplementary video shows that. the optimized policy able to perform well on the task, demonstrating the successful application of FOCALBO to high-dimensional control problems.
8

---

FocalBO TS
FocalBO TuRBO
Function
TS
4000
TuRBO
Vecchia GP TS
Vecchia GP TuRBO
2000
start
end
1000
2000
3000
Evaluation number
(a)
(b)
Figure Optimization of musculoskeletal system control. (a) Task illustration of initial and larget state. Full video in supplementary. (b) Oplimization performance of algorithmss
5.4 Algorithm analysis
To understand the reasons behind FOCALBO's superior optimization performance, we investi- gate the optimization depth in FOCALBO, which the central component of the method. Figure 5(a) shows the evolution of optimization depth over different problems, where FOCALBO is able adapt the optimization depth according different function structure. For Shekel and musculoskeletal model control where the promising regions are distinct, the optimization exhibits an increasing trend exploit current best points, while for other problems the depth tends to converge at a fixed level. Figure bb) shows the sources of proposed batches musculoskeletal system control. Overall the samples exhibits clear
during the optimization trend from exploration to exploitation over high-dimensional input space. Our hierarchical optimization strategy enables flexibility between exploration and exploitation.
Optimization depth
15
Shekel
5 depth 4
9
100
Michalewicz
8
80
function
10
Dkitty
6
60
MS control
5
5
3
2
10
20
30
40
50
10
15
20
25
30
BO iteration
BO iteration
(a)
(b)
Figure 5: Algorithm analysis over optimization depth. (a) Depth evolution during optimiza- tion. (b) Samples source of each BO iteration during trial of musculoskeletal system control optimization. Color bar indicates the number of samples proposed by corresponding optimization depth.
Conclusion
In this paper, we propose FOCALBO, which uses a hierarchical acquisition optimization strategy equipped with focalized GP model to scale Bayesian optimization problems with large offline datasets and/or a large number of online samples. Despite limited representation capability, FOCALBO consistently improves various acquisition functions in optimizing heterogeneous functions, and adeptly leverages large offline dataset for efficient optimization over robot morphology. Under the large affline-toonlile optimization setting, FOCALBO achieves stable high-dimensional control of human musculoskeletal model with over 500 parameters. Ablation studies over the algorithm components further verify the principled design FOCALBO. Future work may include theoretically analyzing FOCALBO, and applying the method to more complex problems, such as large-scale parameter tuning and whole-body human musculoskeletal system control.

---

References
[1] Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Aduances in neural information processing systems, 25, 2012.
[2] Roberto Calandra, André Seyfarth, Jan Peters, and Mare Peter Deisenroth. Bayesian optimization for learning gaits under uncertainty: An experimental comparison on dynamic bipedal walker. Annals of Mathemalics and Artificial Intelligence, 16:5-23. 2016.
3 Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization alphago. ar preprint ar
2018.
[4] Rafacl K6mez-Bombarclli Jennifer N Wei, David Duvenaud, José Miguel Hernández- Lobato, Benjamín Sánchez- Longeli,,ngg,ingg, Dennis Sheberla, Jorge Aguilera-pparrauuing Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
[5] Carl Edward Rasmussen. Gaussian processes in machine lcarning. In Summer school on machine learning, pages 63-71. Springer, 2003.
[6] Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design- bench: Benchmarks for data-driven offline model-based optimization. In International Confer- PMLR, 2022.
ence on Machine Learning, pages
[7] Neil Lawrence, Matthias Seeger, and Ralf Herbrich. Fast sparse gaussian process methods: The informative vector machine. Advances in neural information processing systems, 15, 2002.
[8] Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. Advances in neural information processing systems, 18, 2005.
99] Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificiol intelligence and statistics, pages 567 574. PMLR, 2009.
[10] Mitchell McIntire, Daniel Ratner, and Stefano Ermon. Sparse gaussian processes for bayesian optimization. In UAT, volume 3, page 4, 2016.
[11] Henry B Moss, Sebastian W Ober, and Victor Picheny. Inducing point allocation for sparse gaussian processes in high-throughput bayesian optimisation. In International Conference on Arlificial Intelligence and Statislies, pages 5213 5230. PMLR, 2023.
[12] Felix Jimonez and Matthias Katzfuss. Scalable bayesian optimization using vecchia ap- proximations of gaussian processes. In International Conference on Artificial Intelligence and Statistics, pages 1492-1512. PMLR, 2023.
[13] Haitao Liu, Yew-Soon Ong, Xiaobo Shen, and Jianfei Cai. When gaussian process meets big data: A review of scalable gps. IEFF transactions neural networks and learning systems, 31111):4405-444 4423, 2020.
[14] Kohei Hayashi, Masaaki Imaizumi, and Yuichi Yoshida. On random subsampling of gaussian process regression: A graphon- graphon-based analysis. In International Conference Artificial Intelligence and Statistics, pages
PMI.R, 2020.
[15] Matthias Seeger. Pac-bayesian generalisation error bounds for gaussian process classifi- cation. Journal of machine learning research, 3(Oct):233 2293,269,,40 2002.
[16] Sathiya Keerthi and Wei Chu. A matching pursuit approach to sparse gaussian process regression. Advances in neural information processing systems, 18, 2005.
[17] Tilmann Gneiting. Compactly supported correlation functions. Journal of Mullivariate Analysis, (3(2):493 508, 2002.
10

---

[18] Arman Melkumyan and Fabio Tozeto Ramos. A sparse covariance function for exact gaussian process inference in large datasets. In Twenty-fire first international joint conference on artificial intelligence, 2009.
[19] Martin Buhmann. A new class of radial basis functions with compact support. Malhe- malics of Compulation, 70(233):307-333 318, 2001.
[20] Holger Wendland. Scattered data approzimation, volume 17. Cambridge university press, 2004.
[21] Joaquin Ciinoncro-Candeen and Carl Edward Rasmussen, A unifying view of sparse approximate gaussian process regression. The Journal of Machine Learning Research, 6:1939 1959, 2005.
[22] Alex Smola and Peter Bartlett. Sparse greedy gaussian process regression. Advonces in neural information processing systems, 13, 2000.
[23] Matthias W Seeger, Christopher KI Williams, and Neil D Lawrence. Fast forward selection to speed up sparse gaussian process regression. In International Workshop on Artificial Intelligence and Stalistics, pages 254 261. PMLR, 2003.
[24] Lehel Csató and Manfred Opper. Sparse on-line gaussian processes. Neural computation, 14(3):641 668, 2002.
[25] James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian processes for big data. Xiv preprint arxi 1069:1399.6880 6835, 2013.
[26] Lehel Csató and Manfred Opper. Sparse representation for gaussian process models. Advances in neurol information processing systems, 13, 2000.
[27] Andrew Wilson and Hannes Nickisch. Kernel interpolation for scalable structured gaussian processes (kiss-gp). In International conference on machine learning, pages 1775-1784. PMLR, 2015.
[28] Lubuan Wu, Geoff Pleiss, and John Cunningham, Variational nearest neighbor gaussian process. In International Conference on Machine Learning, pages 24114-24130. PMLR, 2022.
[29] Gia-Lac Tran, Dimitrios Milios, Pietro Michiardi, and Maurizio Filippone, Sparse within sparse gaussian processes using neighbor information. In International Conference on Machine Learning, pages 10369 10378. PMLR, 2021.
(30] Hyoung-Moon Kim, Bani K Mallick, and Chris C Holmes. Analyzing nonstationary spatial data using piecewise gaussian processes. Journal of the American Statistical Association, 100(470):653 -68, 2005.
[31] Abhirup Datta, Sudipto Banerjee, Andrew O Finley, and Alan E Gelfand. On nearest- neighbor gaussian process models for massive spatial data. Wiley Interdisciplinary Reviews:
Computational Statistics, 8(5):162 171, 2016.
(32] Seniha Esen Yuksel, Joseph N Wilson, and Paul D) Gader. Twenty years of mixture of experts. JEEE transactions on neural networks and learning systems, 1377 2012.
[33] Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey, Artificial Intelligence Review, 42:275-200 2014.
[34] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8) 2002.
[35] Samuel Cohen, Rendani Mbuvha, Tshilidzi Marwala, and Marc Deisenroth. Healing products of gaussian process experts. In International Conference on Machine Learning, pages 2068 2077. PMLR, 2020.
11

---

[36] Tipaluck Krityakierne and David Ginsbourger. Global optimization with sparse and local gaussian process models. In International Workshop on Machine Learning, Optimization and Big Dala, pages 185-196. Springer, 2015.
[37] Sattar Vakili, Henry Moss, Artem Artemev, Vincent Dutordoir, and Victor Picheny. Scalable thompson sampling using sparse gaussian process models. Advances in newral information processing systems, 34:5631-566 5643, 2021.
[38] Wesley J Maddox, Samuel Stanton, and Andrew G Wilson. Conditioning sparse varia- tional gaussian processes for online decision-making. Advances in Neural Information Processing Systems, 34:6365 -366655-379, 2021.
[39] Matthias Katzfuss, Joseph Guinness, Wenlong Gong, and Daniel Zilber. Vecchia approximations of gaussian-proeess predictions. Journal of Agriculbural, Biological and Enwironmental Stalislics, 25:383 011,,4142220 2020.
[40] Natalie Maus, Kyurae Kim, Geoff Pleiss, David Eriksson, John P Cunningham, and Jacob Gardner. Approximation-awar bayesian optimization. arXiv preprint arXiv:2406. 0430000408,.... 2024.
[41] Tom Rainforth, Adam Golinski, Frank Wood, and Sheheryar Zaidi. Target aware bayesian inference: how to beat optimal conventional estimators. Journal of Machine Learning Research, 288))) 54, 2020.
[42) Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian opti- mization using deep neural networks. In International conference on machine learning, pages 2171-2180 PMLR, 2015.
[43] Zhongkai Shangguan, Lei Lin, Wencheng Wu, and Beilei Xu. Neural process for black- box model optimization under bayesian framework. arXiv preprint arXiv:2104. 02487, 2021.
[44] Balaji Lakshminarayanan, Daniel M Roy, and Yee Whye Teh. Mondrian forests for large-scale regression when uncertainty matters. In Artificial Intelligence and Statistics, pages 1478-1487. PMLR, 2016.
[45] Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanic Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In International Conference on Artificial Intelligence and Statistics, pages 745-754. PMLR, 2018.
[46] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloezek. Scalable global optimization via local baycsian optimization. Advances in neural information processing systems, 32, 2019.
[47] Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimization using monte carlo tree scarch. Advances in Neural Information Processing Systems, 33:19511-19522, 2020.
[48] David Eriksson and Matthias Poloezek. Scalable constrained bayesian optimization. In International Conference on Artificial Intelligence and Statistics, pages 730-738. PMLR, 2021.
(49) Natalie Maus, Haydn Jones, Juston Moore, Matt J Kusner, John Bradshaw, and Jacob Gardner. Local latent space bayesian optimization over structured inputs. Advances in Neural Information Processing Systems, 35:34505 34518, 2022.
[50] Saulius vaisa and Julius Zilinskas. Scalable bayesian optimization with generalized product of experts. Journal of Global Optimization, 88(3):777-8D) 802, 2024.
[51] Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arxiv preprint ar XrXt::0912.3995, 2009.
12

---

[62] Donald R Jones, Marthias Schonlau, and William J Welch. Efficient global optimization of expensive black-bor functions. Journal of Global optimization, 13:455-499 492, 1998.
[53] Kirthevasan Kandasamy, Akshay Krishnamurthy, Jeff Schneider, and Barnabás Póczos. Parallelised bayesian optimisation via thompson sampling In International conference on artificial intelligence and statistics, pages 133-142 PMLR, 2018
[54] Nikolaus Hansen.
The ema cvolution strategy: a comparing review. Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, pages 75 102, 2006.
[55] Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorrch: A Framework for Efficient Monte- Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020.
[56] Javier González, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization via local penalization. In Artificial intelligence and statistics, pages 648 657. PMLR, 2016.
[57] Robert B Gramacy and Daniel W Apley. Local gaussian process approximation for large computer experiments. Journal of Computational and Graphical Statistics, 24(2):561- 578, 2015.
[58] David R Burt, Carl Edward Rasmussen, and Mark Van Der Wilk. Convergence of sparse variational inference in gaussian processes regression. Journal of Machine Learning Research, 21(131):1-66 63, 2020.
(59] ChangYong Oh, Efstratios Gavves, and Max Welling. Bock: Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning, pages 3868 3877. PML.R 2018.
[60] Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conference on robot learning, pages 1300-1313. PMLR, 2020.
[61] Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative ob- jective models for effective oflline model-based optimization. In International Conference on Machine Learning, pages 10358 10368. PMLR, 2021.
[62] Kaibo Hc, Chenhui Zuo, Jing Shao, and Yanan Sui. Self model for embodied intelli- gence: Modeling full-body human musculoskcletal system and locomotion control with hicrarchical low-dimensional representation. arXiv preprint orXiv:2312.05473, 2023.
[63] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXim:14122 6980, 2014.
[64] Ll'ya Meerovich Sobol'. On the distribution of points in cube and the approximate evaluation of integrals. Zharnal Vychislätel noi Matematiki i Matematicheskoi Fiziki, 7(4):784-802, 1967.
[65] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In 2012 IEEE/RSE international conference on intelligent robots and systems, pages 5026 5033. IEEE, 2012.
[66] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages PMLR, 2018
13

---

A Implementation Details
A.1 Implementation of FOCALBO
We implement FOCALBO with BoTTore which is a popular library for BO implementation with GPU acceleration. For acquisition optimization, directly use acquisition function implementation and corresponding optimizers from BoTorch. Our code for fully reproducing all experimental results is in the: https://githin ctm/ http//uuueeeeooaaagg Our muscu- loskeletal model will be released soon. In the meantime, the model can be accessed for research puppsses upon request segui@ttinghua.edu.com
A.2 Implementation of baselines
SVGP. We directly use approximated GP class in Gpytorch example
WOGP. We refer to the original implementation and write a Botorch GP wrapper with inducing point kernel to enable acquisition optimization using BoTorch. As the hyperparameter are unknown to the GP model, we first warm up WOGP using random of inducing points for 100 epochs, then perform weighted training point selection and continue hyperparameter fitting with the selected WOGP model.
Vecchia GP. We directly use the original implementation without much modification, as is also implemented in BoTorch.
TuRBO. We refer to the implementation in BoTorch tutorial and use the default setting in trust region length and success failure thresholds.
A.3 GP training details
For all GP, use Matern ÿ kernel with automatic relevance delermination, and do not restrict the lengthscale or noise range. For each round of GP training, we GP hyperparameters (and variational parameters for localized GP and SVGP) for 1000 epochs via Adam optimizer [3] with learning rate as 0.01. For focalized GP and SVGP, we initialize the inducing points using Sobol sampler [4]] over input space. all experiment are conducted Intel(R) Xeon(R) Gold 6348 CPU 2.60GHz, NVIDIA-A100 and 512Gb memory.
server with
A.4 Synthetic functions
For GP function, we directly sample from a exact 5d GP using Matern 5 kernel with length- scale as 0.5. For other synthetic functions, we directly use the test function implementation from BoTorch.
A.5 Robot morphology design
We use the dataset and function oracle from Design Bench We choose D'Kitty morphology design for its consistency in function values between oflline dataset and online function oracle, and its compatibility with python 3.8+.
A.6 Human musculoskeletal system control
We use the musculoskeletal system from 62, which enables foward simlation with Mujoco pa and environment customization. We design the following reward for each environment step:
https: httpss:/bbtorrch.orr org/
/ttps://gpytoorc..ai/
httpa:
14

---

r 50rpos pos 10rori + reach +
Fact
drdd
(10)
where "pos encourages the bottle near the target position, Fort encourages the bottle near the target orientation, Treach encourages the hand grab the bottle, Fitt encourages the hand
to lift the bottle, Fact penalize the overall muscle activation, Idone penalize the early ended episode due dropped bottle or hand outside of pre-defined range.
We trained a Soft Actor-Critic (SAV) p6 agent for 6M timesteps to collect task-related muscle activation data, and use principled component analysis to reduce the action dimension from 81 to
B Additional Experiments
B.1 Theoretical implications of sparse GP approximation.
also empirically measure our claim that Focalized GP can significantly reduce approximation error on the search region. We sampled 8000 training points from 2d GP functions to train focalized GP and SVGP. Over different size of the search region, we compare the KT. divergence of the GP posterior prediction over search region between sparse GPs and the exact GP. We observe that the KL divergence between focalized GP and exact GP is consistently smaller than that between SVGP and exact GP, implying tighter approximation the exact GP over local region.
In Figure 6
120
+
Focalized GP
KL divergence
100
SVGP
80
60
40
0.8
0.4
0.2
0.1
Search region size size
Figure 6: KL divergence between sparse GPs and exact GP. Results shows the mean and one standard error, averaged over independent trials.
While a rigorous regret bound hard derive, we conduct an empirical study where we directly compare the optimization performance between focalized GP and SVGP when combining with TuRBO. In this way we can eliminate the influence of hierarchical acquistion optimization. The optimization performances are shown in Figure 0 We observe that focalized GP outperforms SVGP on both high-dimensional problems, which empirically demonstrates theoretical implications that Focalized GP contributes to reducing regret.
Different way of centering the search region
We empirically investigate this in Figure (a), which compares different ways of selecting the search region center by measuring the distance from the search region center to the global optima. We observe that current. best point consistently is the closest to the global optimum, which validates this design choice.
5 kernel and For the experiment above, we sampled 2d functions from GPs with Matern 2 lengthscale of 0.05 (representing rigid functions), and selected the best point over unifromly sampled 10,000 points as the global optima.
A sparse GP is already more explorative than using the full GP, since the smaller representa- tional capacity leads to smoother posteriors. Figure (b), We demonstrate this empirically below, where we measure the pair-wise distance of 100 Thompson sampling points under exact and SVGP (with 50 inducing points). We observe that sparse GP actually samples more diverse sets compared to exact GPs, ie. exhibiting more exploration. Therefore,
15

---

Robot Morphology
MS Control
FocalGP TuRBO
5000
FocalGP TuRBO
Function value
0.91
SVGP TuRBO
Function value
SVGP TuRBO
4000
0.90
3000
0.89
2000
0
50
100
1000
2000
3000
Evaluation number
Evaluation number
Figure 7:
Optimization performance of focalized GP and SVGP when combining with
TuRBO.
Distance to best point
Best
Pair-wise distance
+
Exact GP
0.6
Random
0.45
SVGP
EI
0.4
UCB
Pl
0.40
0.35
50
100
500
1000
200
500
1000
Training data size
Training data size
Figure 8: (a) Distance of search region center to the global optima. (b) Pair-wise distance of Thompson sampling samples. Results shows the mean and one standard error, averaged over 50 independent trials.
using focalized GPs does not sacrifice exploration, and significantly helps exploitation by performing acquisition function optimization over smaller search regions.
B.2 Optimization on synthetic functions with large online data
We choose Ackley and Hartmann, which common-used test functions for BO community, We use the similar optimization setting in [2]. The optimization performances are shown in Figure g where FOCALBO is still able to achieve comparable better performance when online samples dominates the data source.
Ackley-5d init=20 inducing size=20 Hartmann-6d init=30 eval=120, inducing size=20 batch size=10, acqf=El
batch size=4, acqf=UCB
-1
2.5
Function
2.0
-3
1.5
100
200
0
25
50
75
100
Evaluation number
Evaluation number
Figure 9: GP predictive performance of specific search region on Ackley and Rastrigin function. Results show mean + one standard deviation over 10 random search regions.
16

---

B.3 GP predictive performance
We use two common-used syuthetic functions, Ackley and Rastrigin, to analyze the the GP predictive performance focalized GP compared with Exact GP and SVGP under different search region size I and different inducing variables number likelihood (NLL) and root mean squared error (RMSE) in Figure 10 The results shows that focalized GP outperforms both Exact GP and SVGP in terms of both NLL and MSE when the search space size is lower than 0.5. In Rastrigin function where Exact GP achieves similar performancc as SVGP, focalized GP is still able to accurately predict the local search region over different choice of inducing variable numbers. We also show in Figure [1 that the regularization term Freg is indispensable to the training of focalized GP to achieve good local prediction.
We show the negative log
Ackley 100.2
Ackley m= 128
Rastrigin 1=0.2
Rastrigin m=128
0.0
-0.
-0.50
NLI
0.75
-0.5
-0.5
-1.0
100
200
1.0
0.5
100
200
1.0
0.5
RMSE
0.02
RMSE
0.02
RMSE
0.05
RMSE
0.05
0.00
0.00
0.00
0.00
100
200
1.0
0.5
100
200
1.0
0.5
I
m
SVGP
FocalGP
Figure 10: GP predictive performance of specific search region on 2d Ackley and Rastrigin function. Results show mean one standard deviation over 10 random search regions.
Ackley 1=0.2
Ackley m= 12828
Rastrigin 1=0.2
Rastrigin m=128
0.0
0.0
0.50
-0.5
NLL
00.5
0.5
-0.5
-1.0
1.0
0.5
100
200
1.0
0.5
RMSE
0.025
RMSE
0.02
RMSE
0.05
RMSE
0.05
0.000
0.00
0.00
0.00
100
200
1.0
0.5
100
200
1.0
0.5
m
m
FocalGP
FocalGP w/o L_reg
Figure 11: Ablations on the reguralization loss Creg- Results show mean one standard deviation over 10 random search regions.
B.4 Comparison with TuRBO
We run the original TuRBO implementation (with exact GP and Thompson sampling) and TuRBO with nearest neighbor GO model on both robot morphology design and human musculoskeletal system control task (Figure 12). We observed that FocalBO outperforms TuRBO on both tasks with smaller computational cost. The reason of TuRBO's s poor performance may be that cannot quickly adapt over the search space when the online evaluation budget is small.
17

---

Robot Morphology
MS Control
0.94
Function value
Function value
6000
0.92
FocalBO
FocalBO
Original
2000
3000
Evaluation number
Evaluation number
Figure 12: Optimization performance of FocalBO and TuRBO.
C Lemmas Used for Theoretical Implications of Focalized ELBO
Lemma 1. (Corollary 19 in 188 Let ko be squared erponential kernel. Suppose that N real-valued (onedimensional) covariates are observed, with identical Gaussian marginal distribulions. Suppose the conditions of Theorem 13 salisfied for some R > O. Fix ony and on -(7/N2) such if inducing
E 1]. Then there erists an M = Ologg points distribuled according to an e-opprocimale M-DPP with kernel malrir KJ1;
Lemma 2. (Proposition 1 in 5 Suppose 2KL 7 For any x² E X, let μn denote the posterior mean of the variational approrimation at x¹ and μ2 denote the mean of the ezact posterior at x², Similarly, let oī,oz denote the variances of the approzimate and exact posteriors at x² Then,
11 S 0IV7 /3
and 1 < V
(11)
Lemma 3. (Assumplion in B (quality of the approsimate prediction). For the approrimate fu,, the exact pt and ot, and for all x E Ex
[+((x) - px(x)) (or(1),
(12)
where 0 S c all t 1 and some constant R
18

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, rescarch ethics, and socictal impact. Do not remove the chocklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information ion on how to answer these questions. For each question in the checklist:
You should answer [Yes] [No [NA]-
NA] means either that the question is Not Applicable for that particular paper or the relevant information Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
chairs, senior area chairs, and ethics reviewers. You will be
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to '[No] ', it is perfectly acceptable to answer '[No] provided a proper justification given (e.g., "error bars are not reported because it would be too computationally expensive' or "we were unable to find the license for the dataset we used'). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true is often more muanced, so please just use your best judgment and write justification to claborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We conduct comprchensive ovaluations on the proposed methods on both offlinc and online data setting, and compare with existing sparse GP-based BO basclines to support our main claims.
Guidelines:
The answer NA mcans that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
19

---

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We mention the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No mcans that the paper has limitations, but those are not discussed the paper.
The authors are encouraged to create a separate 'Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model sell--pcciiiiation asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closod captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algo- rithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assump- tions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not involve rigorous theoretical analysis about the proposed method.
Guidelines:
The answer NA means that the paper does not include thcoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
20

---

Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed reproduce the main experimental results of the paper to the extent that it affects the main claims /nr/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
J Justification:
Code to fully reproduce all experimental results has been attached.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible
verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same datasct, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g,, in the case of a large language model), releasing a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference docs require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other rescarchers to have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code to fully reproduce all experimental results has been attached.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipp
cc/ppu/..//uuuues///ddduiiisssss for more details.
21

---

While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submis- sion guidelines for more details.
The authors should provide instructions on data access and preparation, in- cluding how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why. At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All related experimental setting is stated in the main paper or the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental sctting should be presented in the core of the paper to level of detail that is nocessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All of the results are plotted with averaged performance with errorbar, and from the plot FOCALBO significantly outperforms baselines.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test /tstt split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it The authors should preferably report a 2-sigma error bar than state that they have a 96% CL, if the hypothesis of Normality of errors is not verified.
22

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
The related content has been stated in Appendix A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: We have conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special consideration due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We propose FOCALBO, which is able to optimize high-dimensional data with large offline/oline sample budgets. It can be used in high-dimensional robot control.
Justification:
Guidelines:
The answer NA means that there is socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance). fairness consid- erations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
the quality
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional
unintentional) misuse of the technology.
If there are negalive societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datascts)?
Question:
Answer: [NA]
Justification: There is risk about the possible misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasels, paperawithcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out to the assct's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification: he paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/ code /ooell as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Rescarch with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not include related topic in this question.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not include related topic in this question.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between insti- tutions and locations, and we expect authors to adhere to the NeurIPS Code of
Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "2iOPaLHiFW", "reviewer": "Reviewer_f3AD", "summary": "This paper focuses on scalable Bayesian optimization, where the training data size is large and the search space is high dimensional. This work proposes a new kind of sparse GP model by designing a variational loss function that allows for adaptively focusing on interesting regions throughout the optimization process. By coupling this GP model with common acquisition functions such as EI, TS, PI, the authors show in experiments the effectiveness of this newly developed GP model. The authors also provide comprehensive algorithm analysis to further decipher the exploration vs. exploitation behavior of the algorithm.", "questions": "- One concern is that the search region is always centered around the current best position. In the case when the best point from initially sampled points is far away from the true global optimum, and the landscape of objective function is not very smooth, then the algorithm will have a hard time to attain the global optimum quickly, because Focalized GP reduces to regular sparse GP in the broader search region and suffers inaccurate prediction. This issue might explain in Figure 3 why \"FocalBO TuRBO\" has a better performance than \"FocalBO EI\" (because TuRBO divided up the search space). I wonder if there is any better idea in setting the center of search region to encourage more exploration.", "limitations": "The discussion of limitations is absent in the paper, and it is better for the authors to discuss limitations of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The scalability issue of GP has greatly limited the application of BO to real-world problems. This paper introduces a novel sparse GP model and its complementary optimization strategy to solve this issue. The idea is intuitive and elegant, and with compelling empirical study, I think this is a valuable piece of work to the field.", "weaknesses": "Minor error:\n- L130, \"more that\" -> \"more than\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of this work! We address your question below. The rebuttal figures can be found by opening the rebuttal PDF file.\n\n**Better way to center the search region**\n\nA plausible alternative would be to instead center the search region at the maximum of the chosen acquisition function. We empirically investigate this in **Figure R1 (a)**, which compares different ways of selecting the search region center by measuring the distance from the search region center to the global optima. We observe that current best point consistently is the closest to the global optimum, which validates this design choice. \n\nFor the experiment above, we sampled 2d functions from GPs with Matern $\\frac{5}{2}$ kernel and lengthscale of 0.05 (representing rigid functions), and selected the best point over unifromly sampled 10,000 points as the global optima.\n\nA sparse GP is already more explorative than using the full GP, since the smaller representational capacity leads to smoother posteriors. We demonstrate this empirically in **Figure R1 (b)**, where we measure the pair-wise distance of 100 Thompson sampling points under exact and SVGP (with 50 inducing points). We observe that sparse GP actually samples more diverse sets compared to exact GPs, i.e. exhibiting more exploration. Therefore, using focalized GPs does not sacrifice exploration, and significantly helps exploitation by performing acquisition function optimization over smaller search regions.\n\nRegarding Figure 3, we believe that the better performance of \"FocalBO TuRBO\" may be because utilizing trust regions helps FocalBO to better exploit at early optimization depths, leading to faster convergence with limited evaluation budgets.\n\n**Limitations of our work**\n\nWhile FocalBO demonstrate superior performances against existing scalable BO algorithms, theoretical can be further improved to make it more rigorous. For instance, an analysis of the convergence of FocalBO would enhance our understanding of the algorithm’s mechanism. The currently-used hypercube search region may not align well with the underlying function landscape, potentially wasting computational resources on low-performing regions. A more sophisticated search region design such as [1] may help FocalBO to further improve the optimization performance.\n\n[1] Wang, Linnan, Rodrigo Fonseca, and Yuandong Tian. \"Learning search space partition for black-box optimization using monte carlo tree search.\" Advances in Neural Information Processing Systems 33 (2020): 19511-19522.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yUuuzqgqOP", "reviewer": "Reviewer_SUei", "summary": "Operating in the context of Bayesian optimization, the authors propose to train a surrogate model which focuses on a specific sub-region of the input space by weighting the log-likelihood contributions of each datapoint relative to their distance to that region. They also propose an algorithm for choosing this subregion over the course of optimization, which they demonstrate in numerical experiments.", "questions": "I understand that Lreq (Eq 8) is given such that it is divided by the number of points in the subregion and has one subtracted to it such that its minimum possible value is zero, and since the w's for any points inside S is constant, we are effectively minimizing the sum of kernel functions evaluated at extra-S training points with the projection of that point onto S. Why is this the right thing to do? It's going to bias lengthscale estimates to be small relative to a GP fit to all the data or even one fit only to data inside the subregion. Why is this desirable, and why is Lreg the right way to operationalize this? \n\nAm I correct that none of FocalBO Turbo, SVGP Turbo, or Vecchia TurBO represent the actual algorithm originally proposed my Eriksson et al, but rather are re-implementations of TuRBO's strategy in other approximate GP frameworks? \n\nI think there are some interesting ideas here, but I regretfully am presently suggesting rejection as a result of the limited scope of the competitors in the numerical experiments which in my view is required of a methodology paper with this medium level of conceptual novelty appearing in Neurips. I think this is a solid paper but does not yet meet the high bar of a Neurips article.", "limitations": "You say that you don't introduce any additional hyperparameters, but it might be more accurate to say that you provide defaults for additional hyperparameters: you set the relative strength of Lreg to the inverse of the number of points in the search region (which makes sense for making the minimum of this objective zero, but it's not clear what implications the scaling has) and you have a specific heuristic for choosing the optimization depth. Why is it reasonable to believe this would still be the right thing to do on some other black-box test function outside of the five you've presented here? \n\nI think the problems in Section 5.2 and 5.3 are really interesting problems.\nBut to help compare against prior work, it would be really important to compare against TuRBO itself: though TuRBO uses an \"exact\" GP, they use numerical analysis tricks to scale to large datasets; (glancing at the paper,they do 20,000 evaluations on a 60D problem). So it seems like TuRBO as originally presented by Eriksson et al should have been a competitor in this simulation study.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The reasoning and motivation at a conceptual level is clear and the article is well-written at a high level. \n\nI really like Section 5.4 which gives us an idea of how the algorithm is attacking the problem. \n\nThe test problems used in Section 5.2 and 5.3 of the numerical experiments are outstanding, as they represent nontrivial problems and are of high dimension.", "weaknesses": "The Lreg term of equation 8 seems to come out of left field, and the implications of adding it to the cost in equation 9 are insufficiently discussed; I say more about this in the Questions section. \n\nFigure 1 is not super convincing: it's clear that with a different hyperparameter/inducing point setting (perhaps with admittedly more inducing points), that SVGP would be able to interpolate as well.\n\nThe theoretical implications section is underdeveloped, containing neither new math nor computational experiments verifying that the claims made there hold up empirically.\n\nRegarding novelty level, I would assess it as medium as it falls within these existing frameworks of trust-region style optimization spawning from Erikkson et al as well as work about tailoring inducing point methods to consider specific parts of the input space (a different implementation but a similar spirit is [1]), though the combination to Bayesian optimization is in my view novel.  \n\n[1] Cole, D. Austin, Ryan B. Christianson, and Robert B. Gramacy. \"Locally induced Gaussian processes for large-scale simulation experiments.\" Statistics and Computing 31.3 (2021): 33.\n\n\nOnly for the benefit of the authors, I am attaching here some grammar/clarity issues which did not affect my scoring of this article:\n201) shows an comparison -> shows a comparison\n232) directly reduces\n246-248) sentence spanning these two lines needs attention.\n251) from GP -> from the GP\n298) both large offline dataset and large number -> both a large offline dataset and a large number\n304) inducing variable number -> number of inducing variables\n374) \"consistently augments\" does not make sense in context; I'm not sure what you're trying to say.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review. We are improving the paper based on your suggestions. We'd also like to clarify your concerns in the following paragraphs. Please find the rebuttal Figures by opening the rebuttal PDF file.\n\n**GP comparison in Figure 1**\n\nWe would like to highlight that our goal of designing focalized GP is to to **allocate a fixed set of limited computational resources to obtain better prediction over specific search regions instead of the entire input domain**. This would contribute to better acquisition function optimization in BO as local optimization are usually employed (line 147-165 in the paper). \n\nFor any function, there is some number of inducing points for which the sparse GP posterior converges to that of the full GP. In practice, however, this threshold is unknown, and may be prohibitively high for complex functions. Hence, Figure 1 attempts to illustrate that for a **fixed number of inducing points**, there is a failure mode in which attempting to model the full input space results in an overly-smooth posterior, and that one possible remedy (ours) is to allocate this representational budget towards a subregion of the input space.\n\n**The design of $\\mathcal{L}_{\\text{reg}}.$**\n\nWithout this regularization term, we observed that the ELBO loss can lead to arbitrarily high weights, leading to large lengthscales and poor fidelity within the search region. Hence, we designed $\\mathcal{L_{\\text{reg}}}$ to emphasize prediction within the search region, since we are only optimizing within those subregions in FocalBO. To better illustrate the formulation of $\\mathcal{L}_{\\text{reg}} $, we can rewrite it as\n\n\n\n$L_{\\text{reg}} = \\frac{\n\\sum_{i=1}^t \n w_i - |X \\in S_{c,l}|\n}{\n|X \\in S_{c,l}|\n} $\n$= \n\\frac{\n\\sum_{i=1}^t \n w_i -\\sum_{i=1}^t\n 1_{x\\in S_{c,l}} w_i\n}{\n\\sum_{i=1}^t 1_{x\\in S_{c,l}} w_i\n}\n= \\frac{\n \\sum_{i=1}^t\n 1_{x\\notin S_{c,l}} w_i\n}{\n\\sum_{i=1}^t 1_{x\\in S_{c,l}} w_i\n}\n$\n\nin which the numerator encourages reducing the weights of points outside search region (leading to better estimation within the search region), and the denominator scales the value of $\\mathcal{L}_{\\text{reg}}$ according to the amount of data points inside the search region. Using this relative quantity allows us to perform hyperparameter-free regularization over different search region sizes.\n\nIn our experiments, we observed that adding $\\mathcal{L}_{\\text{reg}}$ did help avoid overfitting to large lengthscales and enabled better local estimation compared to SVGP (Figure 7 and 8 in appendix).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTmGVms3dM", "reviewer": "Reviewer_BXny", "summary": "The paper presents a new method for scaling Bayesian Optimisation to large datasets by employing sparse GP in a way that facilitates BO.  The proposed method of Focal BO fits a sparse GP at different scales, optimises the acquisition function for each of them and selects the next point to query by sampling with probability proportional to its acquisition function value at its scale. The proposed algorithm is evaluated at a number of benchmarks, where a large number of data points is available.", "questions": "-  In Equation 8, why is there a $ - 1$ at the end? Since it is a constant it should not affect the optimisation process at all. \n\n- I cannot understand why authors refer to ELBO as \"ELBO Loss\", since due to how it is defined, I imagine you would rather want to maximise it (as it corresponds to expected likelihood). Are you maximising the loss function? In that case, I do not think it is proper to call it that name.", "limitations": "Within the paper authors do not discuss the limitations (or at least I cannot find them).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a very important and, in my opinion, understudied topic of scaling BO to large datasets. The proposed algorithm is elegant and does not introduce any new hyperparameters. It also seems to be delivering a significant improvement in performance over baselines.", "weaknesses": "- Since the paper is mostly empirical I would expect a bit more baselines. The simplest way to scale TuRBO to large datasets is simply to keep $N$ points closest to the optimal solution. It would be great to see how this naive \"keep closest N TuRBO\" baseline compares to FocalBO TuRBO.\n\n- \"FocalBO with TuRBO (...) is the first GP-based method to achieve top-tier performance reported by prior MBO works.\" - this is a strong statement without much justification. Please either directly compare with MBO baselines or quote the exact numbers achieved by prior work on the exact same setting as considered in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciations of our problem setting and algorithm design. We address your concerns below. Please find the rebuttal Figures by opening the rebuttal PDF file.\n\n**Baselines comparison**\n\nIn **Figure R4**, We run the \"keep closest N TuRBO\" baseline (denoted as NN GP TuRBO) on both robot morphology design and human musculoskeletal system control task, and find that FocalBO consistently outperforms this simple baseline. Thanks to reviewer SUei, we find that FocalBO also outperforms original TuRBO implementation on both high-dimensional benchmarks. The reason of TuRBO's poor performance may be that it cannot quickly adapt over the search space when the online evaluation budget is small. \n\nIn terms of the statement in section 5.2, we find that there is no statistically significant difference between FocalBO and the best performing algorithm reported in Design Bench baselines (p value=0.1168). We will extensively compare the optimization performances against MBO baselines in the new version.\n\n**The design of $\\mathcal{L}_{\\text{reg}}$**\n\nIndeed, the -1 does not affect the loss, and is primarily for illustrative purposes. Specifically, we can write\n\n$L_{\\text{reg}} = \\frac{\n\\sum_{i=1}^t \n w_i - |X \\in S_{c,l}|\n}{\n|X \\in S_{c,l}|\n} $\n$= \n\\frac{\n\\sum_{i=1}^t \n w_i -\\sum_{i=1}^t\n 1_{x\\in S_{c,l}} w_i\n}{\n\\sum_{i=1}^t 1_{x\\in S_{c,l}} w_i\n}\n= \\frac{\n \\sum_{i=1}^t\n 1_{x\\notin S_{c,l}} w_i\n}{\n\\sum_{i=1}^t 1_{x\\in S_{c,l}} w_i\n}\n$\n\nwhich motivates the regularization term as minimizing the weights of points outside of the search region, while normalizing by the size of the search region (to account for decreasing search region sizes in FocalBO).\n\n**ELBO statements**\n\nThanks to pointing out the confusing loss description - it's a typo on our end. We actually maximize the weighted data likelihood term and minimize the regularization term, therefore $\\mathcal{L}_{\\text{reg}}$ should be multiplied by $-1$. We will unify the definitions to the ELBO description to reduce confusion. \n\n**Limitations of our work**\n\nWhile FocalBO demonstrate superior performances against existing scalable BO algorithms, theoretical can be further improved to make it more rigorous. For instance, an analysis of the convergence of FocalBO would enhance our understanding of the algorithm’s mechanism. The currently-used hypercube search region may not align well with the underlying function landscape, potentially wasting computational resources on low-performing regions. A more sophisticated search region design such as [1] may help FocalBO to further improve the optimization performance.\n\n[1] Wang, Linnan, Rodrigo Fonseca, and Yuandong Tian. \"Learning search space partition for black-box optimization using monte carlo tree search.\" Advances in Neural Information Processing Systems 33 (2020): 19511-19522.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for helpful and insightful reviews. In addition to addressing the comments below, we will incorporate the suggestions and new figures into the camera-ready version. Our additional experimental results are attached in the rebuttal PDF below.

Meta-review: This paper proposes an approach for Bayesian Optimization in the large data regime that uses an SVGP with modified ELBO loss to improve quality of the surrogate in “promising” regions of the search space. Reviewers generally agree that the problem is relevant, and that the empirical performance of the approach compared to baselines is compelling (based also on the additional results provided by the authors during the rebuttal period). One of the main criticisms is that the methodology feels somewhat ad-hoc and that the theoretical analysis is underdeveloped. The former criticism is valid, but the approach is also simple, reasonably intuitive, and effective. There are some connections to other literature that I think are missing, in particular on the work of target-aware Bayesian inference (https://www.jmlr.org/papers/v21/19-102.html) whose goal is precisely to modify the inference procedure of the Bayesian model to improve its performance on the downstream task (in this case acquisition function optimization). I also should note that the following contemporary work on Approximation-Aware Bayesian Optimization using sparse GP approaches is very related https://arxiv.org/abs/2406.04308 (as this is only a recent preprint I am considering this as a miss of the authors, this is rather a heads up to help with improvements to the paper).

Overall, I think the quality of this paper is somewhat borderline - However, it’s one of the few that considers modifying the surrogate model for Bayesian Optimization in a way that targets the downstream task of better optimization performance, which in my opinion is under-studied and so this makes it a relevant contribution to the field. Hence I recommend it be accepted, but ask the authors to incorporate the constructive reviewer feedback and the additional empirical results.
Decision: Accept (poster)