CobBO: Coordinate Backoff Bayesian Optimization with Two-Stage Kernels

Abstract:
Bayesian optimization is a popular method for optimizing expensive black-box 1 functions. Yet it oftentimes struggles in high dimensions where the computation 2 could be prohibitively expensive and a sufficient estimation of the global landscape 3 requires more observations. We introduce Coordinate backoff Bayesian optimiza4 tion (CobBO) with two-stage kernels to alleviate this problem. In each iteration, a 5 promising subset of coordinates is selected in the first stage, as past observed points 6 in the full space are projected to the selected subspace adopting a simple kernel 7 that sacrifices the approximation accuracy for computational efficiency. Then in 8 the second stage of the same iteration a more sophisticated kernel is applied for 9 estimating the landscape in the selected low dimensional subspace where the com10 putational cost becomes affordable. Effectively, this second stage kernel refines the 11 approximation of the global landscape estimated by the first stage kernel through 12 a sequence of observations in the local subspace. This refinement lasts until a 13 stopping rule is met determining when to back off from a certain subspace and 14 switch to another coordinate subset. This decoupling significantly reduces the com15 putational burden in high dimensions, while the two-stage kernels of the Gaussian 16 process regressions fully leverage the observations in the whole space rather than 17 only relying on observations in each coordinate subspace. Extensive evaluations 18 show that CobBO finds solutions comparable to or better than other state-of-the-art 19 methods for dimensions ranging from tens to hundreds, while reducing the trial 20 complexity and computational costs. 21

Bayesian optimization is a popular method for optimizing expensive black-box1 functions. Yet it oftentimes struggles in high dimensions where the computation2 could be prohibitively expensive and a sufficient estimation of the global landscape3 requires more observations. We introduce Coordinate backoff Bayesian optimiza-4 tion (CobBO) with two-stage kernels to alleviate this problem. In each iteration, a5 promising subset of coordinates is selected in the first stage, as past observed points6 in the full space are projected to the selected subspace adopting a simple kernel7 that sacrifices the approximation accuracy for computational efficiency. Then in8 the second stage of the same iteration a more sophisticated kernel is applied for9 estimating the landscape in the selected low dimensional subspace where the com-10 putational cost becomes affordable. Effectively, this second stage kernel refines the11 approximation of the global landscape estimated by the first stage kernel through12 a sequence of observations in the local subspace. This refinement lasts until a13 stopping rule is met determining when to back off from a certain subspace and14 switch to another coordinate subset. This decoupling significantly reduces the com-15 putational burden in high dimensions, while the two-stage kernels of the Gaussian16 process regressions fully leverage the observations in the whole space rather than17 only relying on observations in each coordinate subspace. Extensive evaluations18 show that CobBO finds solutions comparable to or better than other state-of-the-art19 methods for dimensions ranging from tens to hundreds, while reducing the trial20 complexity and computational costs.21
1 Introduction22
Bayesian optimization (BO) has emerged as an effective zero-order paradigm for optimizing expen-23 sive black-box functions. The entire sequence of iterations rely only on the function values of the24 already queried points without information on their derivatives. Though highly competitive in low25 dimensions (e.g., the dimension D ≤ 20 [15]), Bayesian optimization based on Gaussian Process26 (GP) regression has obstacles that impede its effectiveness, especially in high dimensions.27
28 Approximation accuracy: GP regression assumes a class of random functions in a probability space29 as surrogates that iteratively yield posterior distributions by conditioning on the queried points. When30 suggesting new query points, for complex functions with numerous local optima and saddle points31 due to local fluctuations, always exactly using the values on the queried points as the conditional32 events may mismatch the function’s local landscape by overemphasizing the approximation accuracy33 of the global landscape.34
35 Curse of dimensionality: As a sample efficient method, Bayesian optimization often suffers from36 high dimensions. Fitting the GP model (estimating the parameters, e.g., length_scales [14]), comput-37 ing the Gaussian process posterior and optimizing the acquisition function in high dimensions all38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
incur large computational costs. It also results in statistical insufficiency of exploration [11, 65]. As39 the GP regression’s error grows with dimensions [8], more samples are required to balance that in40 high dimensions, which could cubically increase the computational costs in the worst case [45].41
To alleviate these issues, we design coordinate backoff Bayesian optimization (CobBO) with two-42 stage kernels, by challenging a seemingly natural intuition stating that it is always better for Bayesian43 optimization to have a more accurate approximation of the objective function at all times. We44 demonstrate that this is not necessarily true, by showing that smoothing out local fluctuations and45 using the estimated function values instead of the true observations to serve as the conditional events46 in selected subspaces can not only significantly reduce the computation time due to the curse of47 dimensionality but also help in capturing the large-scale properties of the objective function f(x).48
Specifically, CobBO introduces the two-stage kernels with a stopping rule. The first stage of each49 iteration adopts a simple kernel that sacrifices the approximation accuracy of f(x) for computational50 efficiency. For example, by using a universal radial basis function (RBF) approximation without51 learnable parameters [50], CobBO can eliminate the model fitting time in the full space. It captures52 a smooth approximation f̂(x) of the global landscape by interpolating the values of queried points53 projected to selected promising subspaces. These projected points serve as the conditional events54 for GP regression. In a selected coordinate subspace, the second stage of the same iteration applies55 a sophisticated kernel that can tolerate high computational cost in low dimensions. For example,56 CobBO uses the Automatic Relevance Determination (ARD) Matérn 5/2 kernel [40]. It refines the57 approximation of the local landscape by a sequence of observations determined by a stopping rule that58 backs off from a certain subspace and switches to another coordinate subset. In addition, computing59 the Gaussian process posterior and optimizing the acquisition function are both efficiently conducted60 in the low dimensional subspaces, bypassing the curse of dimensionality.61 For iteration t, instead of directly computing the Gaussian process posterior distribution62 { f̂(x)
∣∣∣Ht = {(xi, yi)}ti=1 , x ∈ Ω} by conditioning on the observations yi = f(xi) at queried63 points xi in the full space Ω ⊂ RD for i = 1, . . . , t, we change the conditional events, and consider64 {
f̂(x) ∣∣∣R (PΩt(x1, . . . , xt),Ht) , x ∈ Ωt,Ωt ⊂ Ω}
for a projection function PΩt(·) to a random subspace Ωt and an interpolation function R(·, ·), e.g.,65 using a RBF approximation without learnable parameters [50] as the simple kernel for the first66 stage. The projection PΩt(·) maps the queried points to virtual points on a subspace Ωt of a lower67 dimension [51]. The interpolation function R(·, ·) estimates the objective values at the virtual points68 using the queried points and their values as specified byHt. The second stage within the subspace Ωt69 uses the more sophisticated kernel, e.g., Matérn 5/2 kernel [40], which has a number of parameters70 that otherwise would be expensive to be learned in high dimensions.71
This method can be viewed as a variant of block coordinate ascent tailored to Bayesian optimization by72 applying backoff stopping rules for switching coordinate blocks. While similar work exists [43, 48],73 CobBO differs by introducing the two-stage kernels and addressing the following three issues:74
1. Selecting a block of coordinates for ascending requires determining the block size as well as75 the coordinates therein. CobBO selects the coordinate subsets by a multiplicative weights76 update method [2] to the preference probability associated with each coordinate. Thus, it77 samples more promising subspaces with higher probabilities.78
2. A coordinate subspace requires a sufficient number of query points acting as the conditional79 events for the GP regression. CobBO leverages all observations in the whole space by80 interpolating the values of queried points projected to selected promising subspaces, rather81 than simply starting from scratch in each subspace.82
3. Querying a certain subspace, under some trial budget, comes at the expense of exploring83 other coordinate blocks. Yet prematurely shifting to different subspaces does not fully84 exploit the full potential of a given subspace. Hence determining the number of consecutive85 function queries within a subspace makes a trade-off between exploration and exploitation.86 CobBO uses a stopping rule in each subspace to switch the selected coordinates. When87 consecutively querying data points in the same subspace, CobBO does not need to conduct88 the first-stage function approximation in the full space, which is far more efficient.89
Through comprehensive evaluations, CobBO demonstrates appealing performance for dimensions90 ranging from tens to hundreds. It obtains comparable or better solutions with fewer queries, in91 comparison with the state-of-the-art methods, for most of the problems tested in Section 4.2.92
2 Related work93
Certain assumptions are often imposed on the latent structure in high dimensions. Typical assumptions94 include low dimensional structures and additive structures. Their advantages manifest on problems95 with a low dimension or a low effective dimension. However, these assumptions do not necessarily96 hold for non-separable functions with no redundant dimensions.97
Low dimensional structure: The black-box function f is assumed to have a low effective dimen-98 sion [30, 58], e.g., f(x) = g(Φx) with some function g(·) and a matrix Φ of d ×D, d << D. A99 number of different methods have been developed, including random embedding [66, 11, 63, 36,100 44, 70, 5, 32], low-rank matrix recovery [11, 58], and learning subspaces by derivative informa-101 tion [11, 13]. In contrast to existing work on subspace selections, e.g., Hashing-enhanced Subspace102 BO (HeSBO) [44], Mahalanobis kernel for linear embeddings [33], DROPOUT [35] and LineBO [29]103 (which receives a special treatment in Appendix F), CobBO efficiently leverages all the observations104 in the whole space using the two-stage kernels and the stopping rule in each subspace for consecutive105 observations, rather than only relying on limited observations in each coordinate subspace. It exploits106 subspace structure from a perspective of block coordinate ascent, independent of the dimensions,107 different from some algorithms that are more suitable for low dimensions, e.g., BADS [1].108
Additive structure: A decomposition assumption is often made by f(x) = ∑k i=1 f
(i) (xi), with xi109 defined over low-dimensional components. In this case, the effective dimensionality of the model is110 the largest dimension among all additive groups [45], which is usually small. The Gaussian process111 is structured as an additive model [17, 28], e.g., projected-additive functions [36], ensemble Bayesian112 optimization (EBO) [61], latent additive structural kernel learning (HDBBO) [65] and group additive113 models [28, 36]. However, learning the unknown structure incurs a considerable computational114 cost [44], and is not applicable for non-separable functions, for which CobBO can still be applied.115
Trust regions and space partitions: Trust region BO has been proven effective for high-dimensional116 problems. A typical pattern is to alternate between global and local search regions. In the local117 trust regions, many efficient methods have been applied, e.g., local Gaussian models (TurBO [14]),118 adaptive search on a mesh grid (BADS [1]) or quasi-Newton local optimization (BLOSSOM [41]).119 TurBO [14] uses Thompson sampling to allocate samples across multiple regions. A related method is120 to use space partitions, e.g., LA-MCTS[60] on a Monte Carlo tree search algorithm to learn efficient121 partitions. CobBO differs by selecting low dimensional subspaces. It can also incorporate trust122 regions in the first-stage global approximation, as shown in the Appendix.123
3 Algorithm124
Without loss of generality, suppose that the goal is to solve a maximization problem x∗ =125 argmaxx∈Ωf(x) for a black-box function f : Ω → R. The domain is normalized Ω = [0, 1]D126 with the coordinates indexed by I = {1, 2, · · · , D}.127 For a sequence of points Xt = {x1, x2, · · · , xt} with t indexing the most recent iteration, we observe128 Ht = {(xi, yi = f(xi))}ti=1. A random subset Ct ⊆ I of the coordinates is selected, forming a129 subspace Ωt ⊆ Ω at iteration t. As a variant of coordinate ascent, the subspace Ωt contains a pivot130 point Vt, which presumably is the maximum point xMt = argmaxx∈Xtf(x) with Mt = f ( xMt ) .131 CobBO may set Vt different from xMt to escape local optima. Then, BO is conducted within Ωt while132 fixing all the other coordinates Cct = I \ Ct, i.e., the complement of Ct.133
Algorithm 1: CobBO(f, τ , T)
1 Hτ ← sample τ initial points and evaluate their values 2 Vτ ,Mτ ← Find the tuple with the maximal objective value inHτ 3 qτ ← 0 Initialize the number of consecutive failed queries 4 πτ ← Initialize a uniform preference distribution on the coordinates 5 for t← τ to T do 6 if switch Ωt−1 by the backoff stopping rule (Section 3.2) then 7 Ct ← Sample a promising coordinate block according to πt (Section 3.1) 8 Ωt ← Take the subspace of Ωt over the coordinate block Ct, such that Vt ∈ Ωt 9 else
10 Ωt ← Ωt−1 11 X̂t ← PΩt(Xt) [ Project Xt onto Ωt to obtain a set of virtual points (Eq. 1) ] 12 Ĥt ← R ( X̂t,Ht ) [ Smooth function values on X̂t by interpolation usingHt
] 13 p [ f̂Ωt(x)|Ĥt ] ← Compute the posterior distribution of the Gaussian process in Ωt
conditional on Ĥt 14 xt+1 ← argmaxx∈ΩtQf̂∼p(f̂ |Ĥt)(x|Ĥt) [ Suggest the next query in Ωt (Section 3) ] 15 yt+1 ← Evaluate the black-box function yt+1 = f(xt+1) 16 if yt+1 > Mt then 17 Vt+1 ← xt+1, Mt+1 ← yt+1, qt+1 ← 0 18 else 19 Vt+1 ← Vt, Mt+1 ←Mt, qt+1 ← qt + 1 20 πt+1 ← Update πt by a multiplicative weights update method (Eq. 2) 21 Ht+1 ← Ht ⋃ {(xt+1, yt+1)}, Xt+1 ← Xt ⋃ {xt+1} 22 end
For BO in Ωt, we use Gaussian processes as the random surrogates f̂ = f̂Ωt(x) to describe the134 Bayesian statistics of f(x) for x ∈ Ωt. At each iteration, the next query point is generated by solving135
xt+1 = argmaxx∈Ωt,Vt∈ΩtQf̂Ωt (x)∼p(f̂ |Ht)(x|Ht),
where the acquisition function Q(x|Ht) incorporates the posterior distribution of the Gaussian136 processes p(f̂ |Ht). Typical acquisition functions include the expected improvement (EI) [42, 27],137 the upper confidence bound (UCB) [3, 54, 55], the entropy search [24, 25, 64], and the knowledge138 gradient [16, 53, 69].139
Instead of directly computing the posterior distribution p(f̂ |Ht), we replace the conditional events140 Ht by141
Ĥt := R (PΩt (Xt) ,Ht) = {(x̂i, ŷi)} t i=1
with an interpolation function R(·, ·) and a projection function PΩt(·),142
PΩt(x) (j) = { x(j) if j ∈ Ct V
(j) t if j /∈ Ct
(1)
at coordinate j. It simply keeps the values of x whose corresponding coordinates are in Ct and143 replaces the rest by the corresponding values of Vt, as illustrated in Fig. 2.144
Applying PΩt(·) on Xt and discarding duplicates generate a new set of distinct virtual points X̂t =145 {x̂1, x̂2, x̂3, · · · , x̂t̂}, x̂i ∈ Ωt ∀ 1 ≤ i ≤ t̂ ≤ t. The function values at x̂i ∈ X̂t are interpolated as146 ŷi = R(x̂i,Ht) using the standard radial basis function [6, 7] and the observed points inHt. It not147 only significantly reduces the GP regression time due to the efficiency of RBF [6] and the acquisition148 function optimization in low dimensions [11], but also eventually improves the model accuracy using149 the more sophisticated kernel applied on Ωt.150
Note that only a fraction of the points in X̂t∩Xt151 directly observe the exact function values. The152 function values on the rest ones in X̂t\Xt are153 estimated by interpolation, which captures the154 landscape of f(x) by smoothing out the local155 fluctuations. To control the trade-off between156 the inaccurate estimations and the exact obser-157 vations in Ωt, we design a stopping rule that158 optimizes the number of consistent queries in159 Ωt. The more consistent queries conducted in a160 given subspace, the more accurate observations161
could be obtained, albeit at the expense of a smaller remaining budget for exploring other regions.162
The key features of CobBO are listed in Algorithm 1, with more details in the following sections.163 Several auxiliary components are utilized and presented in Appendix C to deal with a larger variety164 of problems and corner cases.165
3.1 Block coordinate ascent and subspace selection166
For Bayesian optimization, consider an infeasible assumption that each iteration can exactly maximize167 the function f(x) in Ωt. This is not possible for one iteration but only if one can consistently query168 in Ωt, since the points converge to the maximum, e.g., under the expected improvement acquisition169 function with fixed priors [59] and the convergence rate can be characterized for smooth functions170 in the reproducing kernel Hilbert space [8]. However, even with this infeasible assumption, it is171 known that coordinate ascent with fixed blocks can cause stagnation at a non-critical point, e.g., for172 non-differentiable [67] or non-convex functions [49]. This motivates us to select a subspace with a173 variable-size coordinate block Ct for each query. A good coordinate block can help the iterations174 to escape the trapped non-critical points. For example, one condition can be based on the result175 in [21] that assumes f(x) to be differentiable and strictly quasi-convex over a collection of blocks. In176 practice, we do not restrict ourselves to these assumptions.177
We induce a preference distribution πt over the coordinate set I , and sample a variable-size coordinate178 block Ct accordingly. This distribution is updated at iteration t through a multiplicative weights179 update method [2]. Specifically, the values of πt at coordinates in Ct starts off uniform and increase180 in face of an improvement or decrease otherwise according to different multiplicative ratios α > 1181 and β > 1, respectively,182
wt,j = wt−1,j ·  α if j ∈ Ct and yt > Mt−1 1/β if j ∈ Ct and yt ≤Mt−1 1 if j /∈ Ct ; w0,j = 1 D ; πt,j = wt,j∑D j=1 wt,j (2)
This update characterizes how likely a coordinate block can generate a promising search subspace.183 The multiplicative ratio α is chosen to be relatively large, e.g., α = 2.0, and β relatively small, e.g.,184 β = 1.1, since the queries that improve the best observations yt > Mt−1 happen more rarely than185 the opposite yt ≤Mt−1.186 How to dynamically select the size |Ct|? It is known that Bayesian optimization works well for low187 dimensions [15]. Thus, we specify an upper bound for the dimension of the subspace (e.g. |Ct| ≤ 30).188 In principle, |Ct| can be any random number in a finite set of possible block sizes C. This is different189 from the method that partitions the coordinates into fixed blocks and selects one according to, e.g.,190 cyclic order [68], random sampling or Gauss-Southwell [46].191
3.2 Backoff stopping rule for consistent queries192
Applying BO on Ωt requires a strategy to determine the number of consecutive queries for making a193 sufficient progress. This strategy is based on previous observations, thus forming a stopping rule. In194 principle, there are two different scenarios, exemplifying exploration and exploitation, respectively.195 Persistently querying a given subspace refrains from opportunistically exploring other coordinate196 combinations. Abruptly shifting to different subspaces does not fully exploit the potential of a given197 subspace.198
CobBO designs a heuristic stopping rule in compromise. It takes the above two scenarios into199 joint consideration, by considering not only the number of consecutive queries that fail to improve200 the objective function but also other factors including the improved difference Mt −Mt−1, the201 point distance ||xt − xt−1||, the query budget T and the problem dimension D. On the one hand,202 switching to another subspace Ωt+1 ( 6= Ωt) prematurely without fully exploiting Ωt incurs an203 additional approximation error associated with the interpolation of observations in Ωt projected to204 Ωt+1. On the other hand, it is also possible to over-exploit a subspace, spending high query budget205 on marginal improvements around local optima. In order to mitigate this, even when a query leads to206 an improvement, other factors are considered for sampling a new subspace.207
3.3 Theoretical Analysis208
One can view our block coordinate selection approach in section 3.1 as a combinatorial mixture of209 experts problem [10], where each coordinate is a single expert and the forecaster aims at choosing210 the best combination of experts in each step. Under this view, we bound the regret of our selection211 method with respect to the policy of selecting the best (unknown) block of coordinates at each step.212
Assume that there is a fixed optimal choice I∗ for the block of coordinates to pick at all steps. This213 block is characterized by improving the objective function for the largest number of times among214 all the possible coordinate blocks when performing Bayesian optimization over the corresponding215 subspaces. The following particular design of losses expresses this cause:216
`t,i =  − log(α̃) if i ∈ Ct and yt > Mt−1 log(β̃) if i ∈ Ct and yt ≤Mt−1 0 if i /∈ Ct ; α̃, β̃ > 1 (3)
as all the coordinates participating in the selected block incur the same loss that effectively rewards217 these coordinates for improving the objective and penalizes these for failing to improve the objective.218 All other coordinates that are not selected receive a zero loss and remain untouched.219
Note that α̃ and β̃ express the extent of reward and penalty, e.g. for α̃ = β̃ = e we have losses of220 `t,i ∈ {−1, 1, 0}. Yet, α̃ is better chosen to be larger than β̃, since the frequency of improving the221 objective is expected to be smaller.222
The loss received by the forecaster is to reflect the same motivation. This is done by averaging223 the losses of the individual coordinates in the selected block, so that the size of the block does not224 matter explicitly, i.e. a bigger block should not incur more loss just due to its size but only due to its225 performance. Such that for each coordinate block It ⊂ I = {1, · · · , D} selected at time step t, the226 loss incurred by the forecaster is Lt,It = 1 |It| ∑ i∈It `t,i. This is also the common loss incurred by227 all the coordinates participating in that block.228
In each step we have the following multiplicative update rule of the weights associated with each229 coordinate (setting α = α̃η and β = β̃η yields the update rule in Eq. 2):230
wt,i = wt−1,i · e−η`t,i = wt−1,i ·  α̃η if i ∈ Ct and yt > Mt−1 1/β̃η if i ∈ Ct and yt ≤Mt−1 1 if i /∈ Ct
(4)
The probability π̃t,It of selecting a certain coordinate block It is induced by πt as specified next. Thus the expected cumulative loss of the foreceaster is:
LT = T∑ t=1 ∑ c∈C ∑ It∈Sc π̃t,It · 1 |It| ∑ i∈It `t,i
Assume the best coordinate block is I∗, then the corresponding cumulative loss is:
L∗T = T∑ t=1 Lt,I∗ = T∑ t=1 1 |I∗| ∑ i∈I∗ `t,i
We hence aim at bounding the regret RegretT = LT − L∗T .231 Theorem 1. Sample from the combinatorial space of all possible coordinate blocks It ∈ ⋃ c∈C Sc232
with probability π̃t,It = ∏ i∈It w̃t,It/ ∑ c∈C ∑ Î∈Sc ∏ j∈Î w̃t,Î. Then the update rule in Eq. 2 with233
α = α̃η , β = β̃η and η = log(α̃β̃)−1 √ T−1|C|D log(D) yields:234
Regrett ≤ O ( (log(α̃β̃) · √ T |C|D log(D)) ) (5)
where w̃t,It = ∏ i∈It w 1/|It| t,i is the geometric mean of weights in block It. The upper bound in Eq. 5235
is tight, as the lower bound can be shown to be of Ω( √ T log(N)) [23] where the number of experts236
is N = ∑ c∈C Sc ≤ D|C|D in our combinatorial setup, as typically |C| D.237
In practice, the direct sampling policy introduced in Theorem 1 involves high computational costs due238 to the exponential growth of combinations inD. Thus CobBO suggests an alternative computationally239 efficient sampling policy with a linear growth in D.240 Theorem 2. Sample a block size c ∈ C with probability pc and c coordinates without replacement241 according to πt. Assume C ⊃ {1}, then the update rule in Eq. 2, with α = α̃η, β = β̃η and242 η = √ log(D)
T (log(α̃β̃)2−log(p1)) ≥ 1 yields:243
Regrett ≤ O (√ (log(α̃β̃)2 − log(p1)) · √ T log(D)) ) (6)
where pc > 0 for all c ∈ C and ∑ c∈C pc = 1, e.g., uniformly set pc ≡ |C|−1.The proof and detailed244 sampling policy are in Appendix A. The regret upper bound in Eq. 6 is tight, as the lower bound for245 an easier setup can be shown to be of Ω( √ T log(D)) [23]. The implication on η is valid only for246 settings of a very high dimensionality and low query budget. In particular, CobBO is designed for247 this kind of problems.248
Remark: Similar analysis and results follow when incorporating consistent queries from Section 3.2249 and sampling a new coordinate block once every several steps. This is done by effectively performing250 less steps of aggregated temporal losses, as shown in Appendix A.251
4 Numerical Experiments252
This section presents detailed ablation studies of the key components presented in Section 3 and253 comparisons with other algorithms.254
4.1 Empirical analysis and ablation study255 Ablation studies are designed to study the contributions of the key components in Algorithm 1 by256 experimenting with the Rastrigin function on [−5, 10]50 with 20 initial points. The best performing257 run out of 5 experiments for each configuration is presented in Figure 3.
RBF interpolation: RBF calculation is time efficient. Specifically, this is much beneficial in high259 dimensions. Figure 1 (left) shows the computation time of plain Bayesian optimization compared to260 CobBO’s. While the former applies GP regression using the Matérn kernel in the high dimensional261 space directly, the later applies RBF interpolation in the high dimensional space and GP regression262 with the Matérn kernel in the low dimensional subspace. This two-step composite kernel leads to a263 significant speed-up. Other time efficient alternatives are, e.g., the inverse distance weighting [26]264 and the simple approach of assigning the value of the observed nearest neighbour. Figure 3 (a) shows265 that RBF is the most favorable.266
Backoff stopping rule: CobBO applies a stopping rule to query a variable number of points in267 subspace Ωt (Section 3.2). To validate its effectiveness, we compare it with schemes that use a fixed268 budget of queries for Ωt. Figure 3 (b) shows that the stopping rule yields superior results.269 Coordinate blocks of a varying size: CobBO selects a block of coordinates of a varying size Ct270 (Section 3.1). Figure 3 (c) shows that a varying size is better than fixed.271
Preference probability over coordinates: For272 demonstrating the effectiveness of coordinate273 selection (Section 3.1), we artificially let the274 function value only depend on the first 25 coor-275 dinates of its input and ignore the rest. It forms276 two separate sets of active and inactive coordi-277 nates, respectively. We expect CobBO to refrain278 from selecting inactive coordinates. Figure 4279 shows the entropy of this preference probability280 πt over coordinates and the overall probability281 for picking active and inactive coordinate at each282 iteration. We see that the entropy decreases, as283 the preference distribution concentrates on the284 significant active coordinates.285
4.2 Comparisons with other methods286
The default configuration for CobBO is specified in the supplementary materials. CobBO performs287 on par or outperforms a collection of state-of-the-art methods across the following experiments. Most288 of the experiments are conducted using the same settings as in TurBO [14], where it is compared289 with a comprehensive list of baselines, including BFGS, BOCK [47], BOHAMIANN, CMA-ES [22],290 BOBYQA, EBO [61], GP-TS, HeSBO [44], Nelder-Mead and random search. To avoid repetitions,291 we only show TuRBO and CMA-ES that achieve the best performance among this list, and additionally292 compare CobBO with BADS [1], REMBO [63], Differetial Evolution (Diff-Evo) [56], Tree Parzen293 Estimator (TPE) [4] and Adaptive TPE (ATPE) [12].294
4.2.1 Low dimensional tests295 To evaluate CobBO on low dimensional problems, we use the lunar landing [38, 14] and robot296 pushing [62], by following the setup in [14]. Confidence intervals (95%) over 30 independent297 experiments for each problem are shown in Fig. 5.298
Lunar landing (maximization): This controller learning problem (12 dimensions) is provided by299 the OpenAI gym [38] and evaluated in [14]. Each algorithm has 50 initial points and a budget of300 1, 500 trials. TuRBO is configured with 5 trust regions and a batch size of 50 as in [14]. Fig. 5 (upper301 left) shows that, among the 30 independent tests, CobBO quickly exceeds 300 along some good302 sample paths.303
Robot pushing (maximization): This control problem (14 dimensions) is introduced in [62] and304 extensively tested in [14]. We follow the setting in [14], where TuRBO is configured with a batch305 size of 50 and 15 trust regions, each of which has 30 initial points. Each experiment has a budget of306 10, 000 evaluations. On average CobBO exceeds 10 within 5500 trials, as shown in Fig. 5 (lower307 left).308
4.2.2 High dimensional tests309 Since the duration of each experiment in this section is long, confidence intervals (95%) over repeated310 10 independent experiments for each problem are presented.311
Additive latent structure (minimization): As mentioned in Section 2, additive latent structures312 have been exploited in high dimensions. We construct an additive function of 56 dimensions, defined313 as f56(x) = Ackley(x1) + Levy(x2) + Rastrigin(x3) + Hartmann(x4) + Rosenbrock(x5) +314 Schwefel(x6), where the first three terms express the exact functions and domains described in315 Section 4.2.1, the Hartmann function on [0, 1]6 and the Rosenbrock and Schwefel functions on316 [−5, 10]10 and [−500, 500]10, respectively.317 We compare CobBO with TPE, BADS, CMA-ES and TuRBO, each with 100 initial points. Specifi-318 cally, TuRBO is configured with 15 trust regions and a batch size 100. ATPE is excluded as it takes319 more than 24 hours per run to finish. The results are shown in Fig. 5 (upper middle), where CobBO320 quickly finds the best solution among the algorithms tested.321
Rover trajectory planning (maximization): This problem (60 dimensions) is introduced in [62].322 The objective is to find a collision-avoiding trajectory of a sequence consisting of 30 positions in a323 2-D plane. We compare CobBO with TuRBO, TPE and CMA-ES, each with a budget of 20, 000324 evaluations and 200 initial points. TuRBO is configured with 15 trust regions and a batch size of325 100, as in [14]. ATPE, BADS and REMBO are excluded for this problem and the following ones,326 as they all take more than 24 hours per run. Fig. 5 (lower middle) shows that CobBO has a good327 performance.328
The 200-dimensional Levy and Ackley functions (minimization): We minimize the Levy and329 Ackley functions over [−5, 10]200 with 500 initial points. TuRBO is configured with 15 trust regions330 and a batch size of 100. These two problems are challenging and have no redundant dimensions. For331 Levy, in Fig. 5 (upper right), CobBO reaches 100.0 within 2, 000 trials, while CMA-ES and TuRBO332 obtain 200.0 after 8, 000 trials. TPE cannot find a comparable solution within 10, 000 trials in this333 case. For Ackley, in Fig. 5 (lower right), CobBO reaches the best solution among all of the algorithms334 tested.335
Regarding running times, for Ackley, CobBO runs for 12.8 CPU hours and TuRBO-1 run for more336 than 80 CPU hours or 9.6 GPU hours. Most other methods either cannot make any progress or find337 far worse solutions.338
5 Conclusion339
CobBO is a variant of coordinate ascent tailored for Bayesian optimization with a stopping rule to340 switch coordinate subspaces. The sampling policy of subspaces is proven to have tight regret bounds341 with respect to the best subspace in hindsight. Combining this projection on random subspaces with a342 two-stage kernels for function value interpolation and GP regression, we provide a practical Bayesian343 optimization method of affordable computational costs in high dimensions. Empirically, CobBO344 consistently finds comparable or better solutions with reduced trial complexity in comparison with345 the state-of-the-art methods across a variety of benchmarks.346
References347
[1] Luigi Acerbi and Wei Ji Ma. Practical bayesian optimization for model fitting with bayesian348 adaptive direct search. In Proceedings of the 31st International Conference on Neural Infor-349 mation Processing Systems, NIPS’17, page 1834–1844, Red Hook, NY, USA, 2017. Curran350 Associates Inc.351
[2] Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a352 meta-algorithm and applications. Theory of Computing, 8(6):121–164, 2012.353
[3] Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. J. Mach. Learn.354 Res., 3(null):397–422, Mar. 2003.355
[4] James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-356 parameter optimization. In J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira, and K. Q.357 Weinberger, editors, Advances in Neural Information Processing Systems 24, pages 2546–2554.358 Curran Associates, Inc., 2011.359
[5] Mickaël Binois, David Ginsbourger, and Olivier Roustant. On the choice of the low-dimensional360 domain for global optimization via random embeddings. Journal of Global Optimization,361 76(1):69–90, January 2020.362
[6] Martin D. Buhmann. Radial Basis Functions: Theory and Implementations. Cambridge363 Monographs on Applied and Computational Mathematics. Cambridge University Press, 2003.364
[7] Martin D. Buhmann and M. D. Buhmann. Radial Basis Functions. Cambridge University Press,365 USA, 2003.366
[8] Adam D. Bull. Convergence rates of efficient global optimization algorithms. Journal of367 Machine Learning Research, 12(88):2879–2904, 2011.368
[9] Roberto Calandra, André Seyfarth, Jan Peters, and Marc Peter Deisenroth. Bayesian optimiza-369 tion for learning gaits under uncertainty. Annals of Mathematics and Artificial Intelligence,370 76(1):5–23, 2016.371
[10] Nicolo Cesa-Bianchi and Gábor Lugosi. Prediction, learning, and games. Cambridge university372 press, 2006.373
[11] Josip Djolonga, Andreas Krause, and Volkan Cevher. High-dimensional gaussian process374 bandits. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger,375 editors, Advances in Neural Information Processing Systems 26, pages 1025–1033. Curran376 Associates, Inc., 2013.377
[12] ElectricBrain. Blog: Learning to optimize, 2018.378 [13] David Eriksson, Kun Dong, Eric Lee, David Bindel, and Andrew G Wilson. Scaling gaussian379 process regression with derivatives. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N.380 Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31,381 pages 6867–6877. Curran Associates, Inc., 2018.382
[14] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek.383 Scalable global optimization via local bayesian optimization. In Advances in Neural Information384 Processing Systems 32, pages 5496–5507. Curran Associates, Inc., 2019.385
[15] Peter I. Frazier. A tutorial on bayesian optimization, 2018.386 [16] Peter I. Frazier, Warren B. Powell, and Savas Dayanik. A knowledge-gradient policy for387 sequential information collection. SIAM J. Control Optim., 47(5):2410–2439, Sept. 2008.388 [17] Elad Gilboa, Yunus Saatçi, and John P. Cunningham. Scaling multidimensional Gaussian389 processes using projected additive approximations. In Proceedings of the 30th International390 Conference on International Conference on Machine Learning - Volume 28, ICML’13, page391 I–454–I–461. JMLR.org, 2013.392
[18] Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D393 Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM394 SIGKDD international conference on knowledge discovery and data mining, pages 1487–1495,395 2017.396
[19] Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato,397 Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparraguirre, Timothy D Hirzel,398 Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using a data-driven399 continuous representation of molecules. ACS central science, 4(2):268–276, 2018.400
[20] Javier Gonzalez, Zhenwen Dai, Philipp Hennig, and Neil Lawrence. Batch bayesian optimization401 via local penalization. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the402
19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings403 of Machine Learning Research, pages 648–657, Cadiz, Spain, 09–11 May 2016. PMLR.404
[21] Luigi Grippo and Marco Sciandrone. On the convergence of the block nonlinear gauss-seidel405 method under convex constraints. Operations Research Letters, 26(3):127–136, 2000.406
[22] Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolu-407 tion strategies. Evolutionary Computation, 9(2):159–195, June 2001.408
[23] David Haussler, Jyrki Kivinen, and Manfred K Warmuth. Tight worst-case loss bounds for409 predicting with expert advice. In European Conference on Computational Learning Theory,410 pages 69–83. Springer, 1995.411
[24] Philipp Hennig and Christian J. Schuler. Entropy search for information-efficient global412 optimization. J. Mach. Learn. Res., 13(1):1809–1837, June 2012.413
[25] José Miguel Henrández-Lobato, Matthew W. Hoffman, and Zoubin Ghahramani. Predictive414 entropy search for efficient global optimization of black-box functions. In Proceedings of the415 27th International Conference on Neural Information Processing Systems - Volume 1, NIPS’14,416 page 918–926, Cambridge, MA, USA, 2014. MIT Press.417
[26] IDW. https://en.wikipedia.org/wiki/Inverse_distance_weighting.418 [27] Donald R. Jones, Matthias Schonlau, and William J. Welch. Efficient global optimization of419 expensive black-box functions. Journal of Global optimization, 13(4):455–492, 1998.420 [28] Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. High dimensional bayesian421 optimization and bandits via additive models. In Proceedings of the 32nd International Confer-422 ence on International Conference on Machine Learning - Volume 37, ICML’15, page 295–304.423 JMLR.org, 2015.424
[29] Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause.425 Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces. In426 Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International427 Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research,428 pages 3429–3438, Long Beach, California, USA, 09–15 Jun 2019. PMLR.429
[30] H. J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in430 the presence of noise. Journal of Basic Engineering, 86(1):97–106, mar 1964.431
[31] Rémi Lam, Matthias Poloczek, Peter Frazier, and Karen E Willcox. Advances in bayesian432 optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic433 Approaches Conference, page 1656, 2018.434
[32] Ben Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-examining linear embed-435 dings for high-dimensional bayesian optimization. Advances in Neural Information Processing436 Systems, 33, 2020.437
[33] Ben Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-examining linear embed-438 dings for high-dimensional bayesian optimization. In H. Larochelle, M. Ranzato, R. Hadsell,439 M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, vol-440 ume 33, pages 1546–1558. Curran Associates, Inc., 2020.441
[34] Benjamin Letham, Brian Karrer, Guilherme Ottoni, Eytan Bakshy, et al. Constrained bayesian442 optimization with noisy experiments. Bayesian Analysis, 14(2):495–519, 2019.443
[35] Cheng Li, Sunil Gupta, Santu Rana, Vu Nguyen, Svetha Venkatesh, and Alistair Shilton.444 High dimensional bayesian optimization using dropout. In Proceedings of the Twenty-Sixth445 International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2096–2102, 2017.446
[36] Chun-Liang Li, Kirthevasan Kandasamy, Barnabas Poczos, and Jeff Schneider. High dimen-447 sional bayesian optimization via restricted projection pursuit models. In Arthur Gretton and448 Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial449 Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages450 884–892, Cadiz, Spain, 09–11 May 2016. PMLR.451
[37] Daniel J Lizotte, Tao Wang, Michael H Bowling, and Dale Schuurmans. Automatic gait452 optimization with gaussian process regression. In IJCAI, volume 7, pages 944–949, 2007.453
[38] LunarLander v2. https://gym.openai.com/envs/LunarLander-v2/.454 [39] Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies455 is competitive for reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,456 N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems457 31, pages 1800–1809. Curran Associates, Inc., 2018.458
[40] Matern kernel. https://scikit-learn.org/stable/modules/generated/sklearn.459 gaussian_process.kernels.Matern.html.460
[41] Mark McLeod, Michael A. Osborne, and Stephen J. Roberts. Optimization, fast and slow:461 Optimally switching between local and bayesian optimization. In ICML, 2018.462
[42] J. Močkus. On bayesian methods for seeking the extremum. In G. I. Marchuk, editor, Optimiza-463 tion Techniques IFIP Technical Conference Novosibirsk, July 1–7, 1974, pages 400–404, Berlin,464 Heidelberg, 1975. Springer Berlin Heidelberg.465
[43] Riccardo Moriconi, K. S. Sesh Kumar, and Marc Peter Deisenroth. High-dimensional bayesian466 optimization with projections using quantile gaussian processes. Optimization Letters, 14:51–64,467 2020.468
[44] Alexander Munteanu, Amin Nayebi, and Matthias Poloczek. A framework for Bayesian469 optimization in embedded subspaces. In Kamalika Chaudhuri and Ruslan Salakhutdinov,470 editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of471 Proceedings of Machine Learning Research, pages 4752–4761, Long Beach, California, USA,472 09–15 Jun 2019. PMLR.473
[45] Mojmir Mutny and Andreas Krause. Efficient high dimensional bayesian optimization with474 additivity and quadrature fourier features. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman,475 N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems476 31, pages 9005–9016. Curran Associates, Inc., 2018.477
[46] Julie Nutini, Mark Schmidt, Issam H. Laradji, Michael Friedlander, and Hoyt Koepke. Coordi-478 nate descent converges faster with the gauss-southwell rule than random selection. ICML’15:479 Proceedings of the 32nd International Conference on International Conference on Machine480 Learning, 37, July 2015.481
[47] ChangYong Oh, Efstratios Gavves, and Max Welling. BOCK : Bayesian optimization with482 cylindrical kernels. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th483 International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning484 Research, pages 3868–3877, Stockholm, Sweden, 10–15 Jul 2018. PMLR.485
[48] Rafael Oliveira, Fernando Rocha, Lionel Ott, Vitor Guizilini, Fabio Ramos, and Valdir Jr.486 Learning to race through coordinate descent bayesian optimisation. In IEEE International487 Conference on Robotics and Automation (ICRA), February 2018.488
[49] M.J.D. Powell. On Search Directions for Minimization Algorithms. AERE-TP. AERE, Theoret-489 ical Physics Division, 1972.490
[50] Radial basis function. https://docs.scipy.org/doc/scipy/reference/generated/491 scipy.interpolate.Rbf.html.492
[51] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. C. Platt,493 D. Koller, Y. Singer, and S. T. Roweis, editors, Advances in Neural Information Processing494 Systems 20, pages 1177–1184. Curran Associates, Inc., 2008.495
[52] Akshara Rai, Rika Antonova, Seungmoon Song, William Martin, Hartmut Geyer, and Christo-496 pher Atkeson. Bayesian optimization using domain knowledge on the atrias biped. In 2018497 IEEE International Conference on Robotics and Automation (ICRA), pages 1771–1778. IEEE,498 2018.499
[53] Warren Scott, Peter Frazier, and Warren Powell. The correlated knowledge gradient for500 simulation optimization of continuous parameters using gaussian process regression. SIAM501 Journal on Optimization, 21(3):996–1026, 2011.502
[54] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process503 optimization in the bandit setting: No regret and experimental design. In Proceedings of the504 27th International Conference on International Conference on Machine Learning, ICML’10,505 page 1015–1022, Madison, WI, USA, 2010.506
[55] N. Srinivas, A. Krause, S. M. Kakade, and M. W. Seeger. Information-theoretic regret bounds507 for gaussian process optimization in the bandit setting. IEEE Transactions on Information508 Theory, 58(5):3250–3265, 2012.509
[56] Rainer Storn and Kenneth Price. Differential evolution–a simple and efficient heuristic for510 global optimization over continuous spaces. Journal of global optimization, 11(4):341–359,511 1997.512
[57] Sonja Surjanovic and Derek Bingham. Optimization test problems, 2013.513 [58] Hemant Tyagi and Volkan Cevher. Learning non-parametric basis independent models from514 point queries via low-rank methods. Applied and Computational Harmonic Analysis, 37(3):389515 – 412, 2014.516
[59] Emmanuel Vazquez and Julien Bect. Convergence properties of the expected improvement517 algorithm with fixed mean and covariance functions. Journal of Statistical Planning and518 Inference, 140(11):3088 – 3095, 2010.519
[60] Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for520 black-box optimization using monte carlo tree search. ArXiv, abs/2007.00708, 2020.521
[61] Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian522 optimization in high-dimensional spaces. In International Conference on Artificial Intelligence523 and Statistics (AISTATS), 2018.524
[62] Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian525 optimization in high-dimensional spaces. In International Conference on Artificial Intelligence526 and Statistics, pages 745–754, 2018.527
[63] Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando De Freitas. Bayesian528 optimization in a billion dimensions via random embeddings. J. Artif. Int. Res., 55(1):361–387,529 Jan. 2016.530
[64] Zi Wang and Stefanie Jegelka. Max-value entropy search for efficient Bayesian optimization. In531 Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on532 Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3627–3635,533 International Convention Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.534
[65] Zi Wang, Chengtao Li, Stefanie Jegelka, and Pushmeet Kohli. Batched high-dimensional535 bayesian optimization via structural kernel learning. In Proceedings of the 34th International536 Conference on Machine Learning - Volume 70, ICML’17, page 3656–3664. JMLR.org, 2017.537
[66] Ziyu Wang, Masrour Zoghi, Frank Hutter, David Matheson, and Nando De Freitas. Bayesian538 optimization in high dimensions via random embeddings. In Proceedings of the Twenty-Third539 International Joint Conference on Artificial Intelligence, IJCAI ’13, page 1778–1784. AAAI540 Press, 2013.541
[67] J. Warga. Minimizing certain convex functions. Journal of the Society for Industrial and542 Applied Mathematics, 11(3):588–593, 1963.543
[68] Stephen J. Wright. Coordinate descent algorithms. Mathematical Programming: Series A and544 B, June 2015.545
[69] Jian Wu and Peter I. Frazier. The parallel knowledge gradient method for batch bayesian546 optimization. In Proceedings of the 30th International Conference on Neural Information547 Processing Systems, NIPS’16, page 3134–3142, Red Hook, NY, USA, 2016. Curran Associates548 Inc.549
[70] Miao Zhang, Huiqi Li, and Steven Su. High dimensional bayesian optimization via supervised550 dimension reduction. In Proceedings of the Twenty-Eighth International Joint Conference on551 Artificial Intelligence, IJCAI-19, pages 4292–4298. International Joint Conferences on Artificial552 Intelligence Organization, 7 2019.553
[71] Yichi Zhang, Daniel W Apley, and Wei Chen. Bayesian optimization for materials design with554 mixed quantitative and qualitative variables. Scientific reports, 10(1):1–13, 2020.555
Broader Impact556
As stated in [32], Bayesian optimization is a powerful optimization technique used in a wide range of557 industries and applications, such as robotics [37, 9, 52], internet tech companies [18, 34], designing558 novel molecules for pharmaceutics [19], material design for increasing efficiency of solar cells [71],559 and aerospace engineering [31]. All of these settings have high-dimensional optimization problems,560 and advances in BO will reflect on improved capabilities on these fields as well. We have fully open-561 sourced our code for CobBO using the MIT license to be available for researchers and practitioners562 in these fields, and many others. The ability to optimize a larger number of parameters than has563 previously been possible will bring further improvements to and further accelerate work in these564 areas.565
Checklist566
The checklist follows the references. Please read the checklist guidelines carefully for information on567 how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or568
[N/A] . You are strongly encouraged to include a justification to your answer, either by referencing569 the appropriate section of your paper or providing a brief inline description. For example:570
• Did you include the license to the code and datasets? [Yes] See Section ??.571 • Did you include the license to the code and datasets? [No] The code and the data are572 proprietary.573 • Did you include the license to the code and datasets? [N/A]574
Please do not modify the questions and only use the provided macros for your answers. Note that the575 Checklist section does not count towards the page limit. In your paper, please delete this instructions576 block and only keep the Checklist section heading above along with the questions/answers below.577
1. For all authors...578 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s579 contributions and scope? [Yes] See Section 3.580 (b) Did you describe the limitations of your work? [No]581 (c) Did you discuss any potential negative societal impacts of your work? [Yes] See582 Broader Impact.583 (d) Have you read the ethics review guidelines and ensured that your paper conforms to584 them? [Yes]585 2. If you are including theoretical results...586
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.3.587 (b) Did you include complete proofs of all theoretical results? [Yes] See the Appendix A.588
3. If you ran experiments...589 (a) Did you include the code, data, and instructions needed to reproduce the main ex-590 perimental results (either in the supplemental material or as a URL)? [Yes] In the591 supplemental material.592
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they593 were chosen)? [Yes] See Table 2 in the appendix, which contains the default hyperpa-594 rameters.595
(c) Did you report error bars (e.g., with respect to the random seed after running experi-596 ments multiple times)? [Yes] See Fig.5 and Fig.8-10.597
(d) Did you include the total amount of compute and the type of resources used (e.g., type598 of GPUs, internal cluster, or cloud provider)? [Yes] See page 9, line 324.599
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...600 (a) If your work uses existing assets, did you cite the creators? [N/A]601 (b) Did you mention the license of the assets? [Yes] MIT license; see the appendix.602 (c) Did you include any new assets either in the supplemental material or as a URL? [No]603 (d) Did you discuss whether and how consent was obtained from people whose data you’re604 using/curating? [Yes] See Section 4.2.605 (e) Did you discuss whether the data you are using/curating contains personally identifiable606 information or offensive content? [N/A] It does not contain personal identifiable607 information or offensive content.608
5. If you used crowdsourcing or conducted research with human subjects...609 (a) Did you include the full text of instructions given to participants and screenshots, if610 applicable? [N/A]611 (b) Did you describe any potential participant risks, with links to Institutional Review612 Board (IRB) approvals, if applicable? [N/A]613 (c) Did you include the estimated hourly wage paid to participants and the total amount614 spent on participant compensation? [N/A]615

Review summaries:
This paper aims at making Bayesian optimization scalable to problems with large dimensional inputs. The authors propose an approach that selects block coordinates of the input space to explore at each iteration. The authors provide some theoretical results to support the proposed approach, and report experimental results to demonstrate that the proposed approach can achieve better performance than existing approaches.

This paper proposes a new High-dim BO framework inspired by Block coordinate ascent. Although similar, this paper differs from other works which leverages a low dimensional structure assumption in that it utilizes all the dimension in the function space. Aside from the overall scheme, its other contributions include (1) an update scheme for the coordinate sampling distribution; (2) a heuristic stopping rule to balance between exploiting current coordinate block and exploring new coordinate blocks; and (3) a theoretical analysis that upper bounds the total regret on a surrogate loss function. The framework shows some promising results when compared with other high-dim BO works.

The authors have proposed a new algorithm for black box optimization called CobBO. At a fundamental level, the algorithm is the extension of classic block coordinate ascent to black box optimization using techniques from Bayesian Optimization. The algorithm focuses on a small subset of coordinates in every iteration. In the first step, it uses a simple kernel to interpolate the values of the function learnt at other points to the points in the subspace. In the next step, it uses a full fledged BO with a more sophisticated kernel in the subset to learn the function in this subspace. It balances exploration and exploitation between subspaces by using a backoff stopping rule. The authors have supported their case by extensive empirical studies and some theoretical results.