Monte Carlo Tree Descent for Black-Box Optimization
Authors: Yaoguang Zhai
Sicun Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Monte Carlo tree search
blackbox optimization
stochastic search
Bayesian optimization

Abstract: The key to Black-Box Optimization is to efficiently search through input regions with potentially widely-varying numerical properties, to achieve low-regret descent and fast progress toward the optima. Monte Carlo Tree Search (MCTS) methods have recently been introduced to improve Bayesian optimization by computing better partitioning of the search space that balances exploration and exploitation. Extending this promising framework, we study how to further integrate sample-based descent for faster optimization.  We design novel ways of expanding Monte Carlo search trees, with new descent methods at vertices that incorporate stochastic search and Gaussian Processes. We propose the corresponding rules for balancing progress and uncertainty, branch selection, tree expansion, and backpropagation. The designed search process puts more emphasis on sampling for faster descent and uses localized Gaussian Processes as auxiliary metrics for both exploitation and exploration. We show empirically that the proposed algorithms can outperform state-of-the-art methods on many challenging benchmark problems.

Monte Carlo Tree Descent for Black-Box Optimization
Yaoguang Zhai UCSD
Sicun Gao UCSD
Abstract
The key Black-Box Optimization is to efficiently search through input regions with potentially widely-varying numerical properties, to achieve low-regret descent and fast progress toward the optima. Monte Carlo Tree Search (MCTS) methods have recently been introduced to improve Bayesian optimization by computing better partitioning of the search space that balances exploration and exploitation. Extending this promising framework, we study how to further integrate sample- based descent for faster optimization. We design novel ways of expanding Monte Carlo search trees, with new descent methods at vertices that incorporate stochastic search and Gaussian Processes. We propose the corresponding rules for balancing progress and uncertainty, branch selection, tree expansion, and backpropagation. The designed search process puts more emphasis on sampling for faster descent and uses localized Gaussian Processes as auxiliary metrics for both exploitation and exploration. We show empirically that the proposed algorithms can outperform state-of-the tet-the-rttt art methods on many challenging benchmark problems.
1 Introduction
Black-Box Optimization (BBO), also referred to as Derivative-free or Zeroth-order Optimization, considers objective functions that are not known analytically and only be evaluated at various inputs, potentially at a high cost. The generality of the formulation makes BBO broadly applicable to wide range of challenging problems in machine learning
well many scientific and engineering problems [4] 5 (). BBO problems over compact domains are naturally NP-hard: in the number of local regions
worst case, we need exhaustively search through the combinatorially-Inn to find bigh-quality solutions. Thus, the goal of BBO algorithm design is to accelerate optimization progress with respect to the number of function evaluations.
Existing work on BBO can be categorized into model-based and model-free approaches. Most model- based approaches, typically in the framework of Bayesian Optimization (7 8), involve learning a surrogate function from samples of the unknown function and optimizing the surrogate rather than the original function. For highly nonlinear functions with high-dimensional input spaces, such methods are known to be costly because of the need for global modeling of the objective functions. Various Bayesian optimization approaches utilize ensembles of local surrogate models 2 to improve performance. Model-free approaches include simulated annealing [00. cross-entropy methods [. search gradient [12), as well as traditional direct search methods such
-Mad [13 14 The goal is to iteratively propose sampling distributions that can approach the optima. Such methods typically do not attempt to maintain global information about the objective and are challenged when the optimization landscape is highly non-convex [15]. In general, the lack of mechanisms for explicitly managing the search over the combinatorially-large number of local regions, both standard model-based and model-free BBO methods, has been major bottleneck of the field.
Recent advances in stochastic tree search methods [16] 17] offer new opportunities for balancing local search and modeling with more systematic global exploration in BBO problems. In particular, Monte Carlo Tree Search (MCTS) has recently been introduced for computing good partitioning of the search space for BBO 3 18 19 These approaches adaptively divide the input space into regions,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

balancing exploitation and exploration, to only perform Bayesian optimization at local regions and create better model ensembles. However, because the focus is still on modeling the objective, the ability of MCTS to quickly expand deep branches into promising search regions has not been fully utilized, As a result, the curse of dimensionality can still quickly stall the search, while model-free descent methods may be able to make more progress if they are also guided by MCTS.
We propose a new design of MCTS methods for BBO, with more emphasis on sample-efficient local descent, which can benefit the most from balanced exploitation and exploration. We use Bayesian optimization and local modeling as auxiliary metrics for guiding the search tree construction. At each node in our search tree, we iteratively collect samples in the neighborhood of some anchor point and also maintain a local Gaussian Process (GP) model for the neighborhood. The samples are chosen using they are also used to train the local GP models. These local models provide surrogate objectives to propose future samples without querying the ground truth function, and they also provide uncertainty metrics for exploration steps. We name our overall approach Monte Carlo Tree Descent (MCTD), because of the focus on faster descent led by samples that are managed by tree search, rather than using MCTS for explicit space partitioning. We evaluate the proposed methods with experiments on challenging benchmarks such as nonlinear optimization benchmarks (21, policy search for MuJoCo locomotion tasks [22). and neural architecture search (23). We compare our algorithm with state- of-the-art model- based [9] and MCTS- based methods [19] as well as model- [24] and direct
caapliing-bseed descent such as Stochastic Three Points methods (STP) [20), and
search methods [[3]. We observe clear benefits in the proposed designs for improving efficiency,
consistently outperforming existing methods on the tested benchmarks.
2 Related Work
Model-based methods. Bayesian optimization 7 25 typically uses Gaussian Processes to con- struct surrogate models of the objective functions 8 with samples selected by acquisition functions (c.g., confidence bounds, expected improvement, etc.) 126 27I. Model-based methods are known to suffer from the curse of dimensionality as the problem dimensionality and sample sizes grow quickly (28). Many approaches have been proposed to improve the scalability of Bayesian optimiza- tion methods in nigh-dimensional problems (29 30, 31. For instance, TuRBO is state-of-the of-thee-rt method that uses Thompson sampling with Expected Hypervolume Improvement (EHVI) [9] It samples in local trust regions and adjusts the trust regions after each sampling iteration, which has shown major benefits in improving the efficiency of model-based approaches for BBO.
Model-free methods. Model-free approaches focus on sampling inputs, either point-wise or population-based, that can incrementally approach optimal regions in the search space without explicitly maintaining models of the objective. Standard approaches include stochastic methods such as simulated annealing (SA) [10] and cross-entropy (CE) [1] and deterministic schemes such as Nelder-Mead (NM) [33. These methods have been successfully applied to a wide range of problems but they typically do not aim for optimizing efficiency, i.c., reducing the number of evaluations [3) They may still offer improvements faster than local methods that rely on gradient information [14 32 The Stochastic Three Points method 200 is a simple but effective way of direct search that compares function values at the base point, in one random direction, and in the opposite direction. Each step evaluates only two more points that lie in the opposite direction of the current point and moves towards the one with a better value. To improve sample efficiency, we attempt to combine The Stochastic Three Points method with model-based methods and carefully design the direction in which the method will try in each iteration.
Tree search methods. Various tree-search methods have been proposed improve partitioning of the search space in BBO, such as Deterministic and Simultancous Optimistic Optimization (DOO and SOO) [8]. and Hierarchical Optimistic Optimization (HOO) in B33. Specifically, DOO divides up the search domain into partitions, each of which is represented by a point within it, assuming known Lipschitz constants for the objective function. SOO and HOO extend DDO to stochastic versions but are mostly applicable to low- wiw-iimnniional problems because of the high cost involved in creating good partition cells. Voronoi Optimistic Optimization (VOO) 3 can be more efficient in high dimensions by combining Voronoi partitioning and tree search. LA-MCTS I9 introduces MCTS to manage the partitioning of the search space. It learns latent actions that define boundaries between good and bad regions in the search space and prioritizes the expansion of the search tree
2

---

around the boundaries. When continuing with such splitting, it sets a sampling preferential on every node in the tree. In every iteration, the search tree traversed from the root node leaf by selecting the highest approximated value based on Upper Confidence bounds applied to Trees (UCT) algorithm. The optimization is then performed from the subspace partition on the selected node. These methods successfully change the objective function modeling for global space to local regions. However, the partitioning of the state space, particularly when the space a high dimension, becomes a very challenging problem. The tree becomes extremely large when the optimization attempts to learn with high accuracy in local regions.
3 Preliminaries
We consider the problem of minimizing an objective function f(z) - R where the domain Q C IR" is compact. We assume the ability to evaluate /' fa( for arbitrary Q but do not have information about the analytic form of the function or its derivatives.
Gaussian Processes (GP) is commonly used in Bayesian optimization and is also used in our work to construct a surrogate model for the local model-based optimization. For finite collection of points T1, ... Th E Rd, GP constructs the mean vector μo from the function f at each Ty and the covariance matrix Eg by a kernel each pair of (2i, (. j = 1, 2, k. With po and En the prior distribution on is:
f(a1.....x .x) ~ NN (1. xx), Eo (21
......... Fy: #1 #1. 2x))
For any new point 2,, we can use Bayes' rule to compute the conditional distribution of fz))
f(l/11.... xx) NNpo(a1,..... Ik: x), 1. Tks I1 Ik. x))
(2)
The Stochastic Three Points (STP) method is model-free approach to BBO that uses only a small number of samples in each iteration to identify descent directions. At each time step t with a current sample F,, it generates a set D, {zr, 32 + 8t at, Zy St - ar} where s; is direction and a₁ is the step size at step t. When a₁ is small enough, the relationship between f(z, + St az), f(x;) and f(x; - St- aς) is monotonically non-increasing or non-decreasing if the gradient of the function is not zero in the direction of st. For the next step, F++11 argmin D. f()). In our method, the
STP-based local descent optimization will identify the best direction St with an optimized step size O₂ for improving its performance.
Monte Carlo Tree Search (MCTS) is leading framework for balancing exploration and exploitation in sampling-base tree search. It consists of four main steps: Selection, Expansion, Simulation, and Backpropagation. During Selection, a search tree is traversed from the root node to leaf node. This traversal is made by selecting the node with the highest value based on the UCT algorithm. For node n,, the UCT v is computed by:
v(ne) = R + C. Nn
in which R3 is the rewards on n2; N₁ and N₂ denote the number of visits on n, and its parent node ny, respectively; C is a constant to balance between exploitation and exploration. At each branch node n4. the child to select is the one with the highest // value among all of its immediate children. At Expansion, a new child node is then added to expand the tree. During Simulation, random simulation is run from the new child node until the terminal node is reached, and the simulation reward is approximated. Finally, the simulation reward is backpropagated through the selected nodes to update the tree. In our approach, we construct our Monte Carlo tree by assigning every leaf node to one optimization process. During each step, we use a modified UCT algorithm to select the node on which the optimization is launched.
4 Monte Carlo Tree Descent
Our MCTD algorithm iteratively constructs search tree over the domain of the objective function, and at each node of the tree we maintain a set of samples and a surrogate model learned from them. The balancing of exploration and exploitation takes into account several factors that will be explained in the subsequent sections. The overall algorithm is illustrated in Alg and we refer Fig. 4 in the Appendix that provides a visual illustration of the process.
3

---

Algorithm Monte Carlo Tree Descent (MCTD)
1: function MCTD(objective: f. domain: !2)
1:
function SELECT(node: n;;)
2: I < random sample in
2:
7h < fl;
3: Rg.sample_ < y) root node
3: while na has children do
4:
for step= do
4:
for child node Thee do
5:
+- Select(mo))
5:
Compute v(mia) by Eq
6: Optimize(n)
6:
end for
7: Backup(n)
7:
Compute v((bz) by Eq
8: end for
8:
max, (v(D))) < (naz) then
9: return 966
9:
return Expand(ns)
10: end function
10:
end if'
6 4 argmax, (((x))
12: function EXPAND(node: n1))
12:
Th < -,,b
if n2 leaf then
13:
end while
14:
T&0 & n. excluding parent/child
F.P(ms) in 8 satisfied then
Ty child list <- 74.0
15:
+- Expand(ms)
16:
end if
16: end if
lu <- level of n d x exp (-lv)
17: return 10b
18:
18: end function
19:
I f random sample in B(x
19:
20:
nim www.sample_ssee {zz, f())];;
20: function OPTIMIZE(node: nı;)
21:
n; m.chhldden.appeedd(r..
21: QD f
22:
return 7m
22: sa.ppamllee_ee >= NR then
23:
end function
23:
θ <- GP model of st.sample set 0D < O correlation length θ
24:
24:
25: function BACKUP(node: ni)
25:
else
26:
72 while n. has parent Rp do
26:
oracle + < None
27:
27:
end if
28:
Update (x yo) with E
28:
Descend on n. by O, f. ap from (a:
29:
Update dyp with Eq
29:
ni < Bayesian Optimize from { (x, y)
30:
f- nhp
30: Update (x;, v() and dy: by Eq and
31:
end while
31:
return
end function
32: end function
4.1 Overall Tree Search Strategy
We initialize our algorithm random sample in the domain the objective function, and the sampled points create the root node of the entire search tree. Unlike standard MCTS that considers finite and discrete actions at each node, for BBO over the continuous domains we can not expand the infinitely-many possible next samples child nodes of the root node. Consequently, already the root node, we need to decide between two choices. First, we could perform local descent on the current sample at this node. Second, we could explore a different region in the space by taking sample that is far from the current one, which will act as a new anchor point that forms a new child node of the tree, which expands the tree. When multiple child nodes have been expanded there is the third option of going down the tree along the most promising branch, and then focusing the next steps of search from there.
node,
Consequently, each iteration of the algorithm, we perform three operations sequentially. First, we perform branch selection starting from the root node, and then either land at some existing node or create a new anchor sample and node, from which we will perform local descent.
4.2 Branch Selection
In every step, we pick leaf for optimization. To balance exploration and exploitation, our algorithm uses UCT to determine the path between the root and the leaf, as shown in the function SELECT in We modified the UCT formula for fitting MCTD algorithm. For each child node nas
Alg line I

---

with the parent node ns, its UCT v((bs) is given by:
v(ma)) = -hi + Ca Σ dund + Cp- Vlog No/Nos
(4)
3=1
Here, Ca is a weight factor controlling the importance of recent improvements during optimization, Cp is a hyper-pare param-eereeeerr for the extent of exploration. N& and N& are the number of visits to the branch node n, and the child node nas, respectively. Vor is the current best function value in the sample set S = [zz 3)} which stores the samples during optimization on node nay::
(Tiss Mia) = argmin (z, v), (Ir, 2) E Ssi
(S)
and dyy,' is the most recent j's improvement at n&d after calling the objective function. When computing v, only the last J improvements function during the optimization, we record the improvement in the function value from this call. If the value from this call is worse than the optimal value before the call, we set the improvement to zero; otherwise, we set the improvement as the absolute difference between the optimal value before and after the call. That is, for you the time step as Vie (f)(
taken into account. For every call the objective
dube (t) max (y6 (t - j) yoe (t - j+
+ 1).0)
(6)
We similarly integrate the tree expansion as the UCT algorithm. At a branch node na. in addition to examining the UCT of all its child nodes we add an artificial exploration node Uhr that has the UCT value v(Maa) as following:
(((t)) = - (o + C. Vlog
(7)
where D, is the number of children of the node nhb Cp hyper-parameter for the extent of exploration but may be different from Cp. This exploration node is to determine whether to optimize in new domain because the existing children are not performing well enough. When the exploration node is selected, new child node under the branch node is created and returned.
If the path selects a leaf that is not newly created, we need to determine whether is worth optimizing on it. On leaf node n,, we expand the tree if the following condition is met:
EP((n)): -3 : Ca Edvj' < Cyj Voog N₁
(8)
Here, Cff is a weight factor for recent last J improvements and may be different from Ca. Cy also a hyper-parameter for the extent of exploration different from C, and C/ In the event the condition g is met, we will make a leaf expansion; otherwise, we descend on the selected leaf node nty.
4.3 Tree Expansion
When we need to take an exploration step at node, a new child node will be created. The new child node created at a random point lying within some distance from the selected node. The minimum and maximum distances are set to 10% and 50% of the domain's 's dimensional length, with exponential decay according to the node level. After the newly created child node is placed, it will be immediately selected as the node for optimization at the current step. When the selected node to explore is a leaf node ny, a new child node π 1 is created in the same way as above, making ny a branch node. At this time, new node fo, starting from the current best point at 27), is also created as the child 0 of node nt. This node no inherits batch of samples that are near its starting point x"y, as well as the latest improvement history on nly. The reduced number of samples forces the inheriting node yo focus on optimizing in the neighborhood of the starting point, while the newly expanded node nf1 is optimizing in distant region. Thus, the tree grows leaf node n while maintaining the possibility of further exploiting around the best point found on n2 at node nu These steps are in line 12 and 3 subplots in Fig show an example. As in Fig/4 (c), the expansion takes place on the root node rio. The node rigi is a new node for exploration, placed distant from no. Node Ao0 starts from xÿ. Similarly, in (d), node 7100 starts from 1h and node n011 is placed away from ni1,, but the distance between node non and no10 is much smaller than the distance between node rag and no1 at node creation. Fig. 4 (c) shows how new leaf node is created.
the function EXPAND in Alg. [
5

---

4.4 Local Optimization.
In every iteration, we use the STP method to attempt local descent and also use TuRBO-1 9 for local Bayesian optimization (BO). We tightly integrate the two methods. Samples obtained from local descent optimization are used to construct the surrogate GP regression model. The GP model only
oracle for the local descent optimization but also provides the correlation length according to which the local descent optimization scales its step sizes.
Local Descent. We use the STP method with the following changes. In STP, the direction &t at step é is usually selected from a sphere with uniform distribution direct search. Instead, we use the surrogate GP regression model to identify the point with the highest expected improvement. The steps of local descent optimization are as follows:
1. Choose node n; by SELECT. If the number of samples exceeds some threshold, we train Gaussian Process model that will be referred to the oracle for this node.
2. Compute the step size 0gy- In our case, we set O₁ to be inversely proportional to the square root of the product of node visits N; and the node level in the tree. We also rescale it according to the correlation length in the surrogate GP model when possible.
3. If the oracle is not available, get random direction 5t, and use 8; at for checking ground truth.
4. If the oracle is available, generate multiple samples in the box with edge length equaling the step size Or,, and choose the best point. The direction to the best point is 8t Ct..
5. Start one step of STP with the selected direction of a by calling the objective function.
Depending on the optimization progress, we may further optimize the objective function along the same direction with tuned step sizes in fine-grain descent approach.
The last step is used when the optimization comes fine-tuned phase with small variations in samples, so one can set a function threshold from which the search applies the fine-grain descent approach.
Local Bayesian Optimization. The TuRBO-1 [9] creates a hyper-rectangle Trust Region (TR) with volume L centered at the best sample. Afterward, it samples new candidates within the TR and queries the objective function for ground truth data. The length of L, will either increase after successive "successes" or decrease after consecutive "failures". We changes TuRBO-1 in three ways to fit it into our algorithm: 1) TuRBO-1 begins with collected samples of the node. Consequently, TuRBO-1 is compelled to optimize from the vicinity of the collected sample. 2) The trust region length has been preserved on the same node, the local BO continue from the previous epoch. 3) We do not perform restarts for TuRBO-1 in order to avoid TuRBO-1 restarting from random samples.
4.5
Back Up
In the BACKUP function, we backpropagate the updated best score found at leaf node and propagate it upwards to its parent nodes. This score update is important for informing future branch selections. This backup procedure is used in every step even if the best-found sample on the selected leaf node does not change after one iteration.
Experiments and Evaluation
5.1 Experiment Setup
Benchmarks We use several standard benchmark sets for testing BBO algorithms, from three synthetic functions for nonlinear optimization, reinforcement learning problems in MuloCo locomotion environments, and optimization problems in Neural Architecture Search (NAS). Synthetic functions are widely-used in nonlinear optimization benchmarks [21]. These functions usually have numerous local minima, valleys, and ridges in their landscapes which hard for normal optimization algorithms. MuJoCo locomotion environments [22] are popular for reinforcement learning tasks. NAS problems have practical significance, since many fields are using deep learning models, but implementing efficient neural networks requires substantial amount of time and effort.
categories:

---

caimplts
carpple
(a) Ackley-50d
(b) Ackley-1066 100d
(c) Michalewicz 100d
279
230
carplees
(d) Hopper-33d
(e) Walker- 10lker-102d
(I) Walker-204d
18000
41.3
12
LaMCTS
CMA
RandonSearch
MelderMend
1303
1180
2080
Evoluations
(g) HalfCheetah--O01
(h) CIFAR-10
(i) -IFAR-100
Figure 1: Overall performance of the baselines and our method. For Ackley and Michalewicz in (a),
(b), and (c), the goal optimize for the lowest function values; in MuJoCo tasks (d), (e), (f), and (g), we aim to maximize the rewards; and for CIFAR- 10 in (h) and CIFAR- 100 in (i) we want to find the architecture with the highest accuracy as quickly as possible
We select multiple problems from each set, and their input dimensions range from 33d to 204d. Details of benchmark problems can found in section B the appendix.
Baselines We selected TuRBO as one baseline from the BO algorithms. La-MCTS 19 is chosen as major comparator since this algorithm also constructs trees in similar manner. Moreover, CMA-ES [44] from the Evolutionary Algorithm category, Nelder-Mead [3] from Direct Search algorithms, well as the Random Search algorithm selected for comparison as baselines.
For CMA optimization. fimin2 from the CMA-ES package [24) is used with its default parameters.
We implement our own version of the Nelder-Mead algorithm as in [13], and set its expansion coefficient, contraction inside the simplex, contraction outside the simplex, and shrink coefficient as 2.0. 0.5, 0.5, and 0.5, respectively. TuRBO [9] is initialized with 20 random samples selected using Latin Hypercube sampling, and its Automatic Relevance Determination (ARD)
True. For La-MCTS 19), use different settings and include them in the supplementary material, well as our MCTD approach. Benchmarks are made mainly on Google Colab with Tesla P100 graphic card. Across all experiments, we set the number of evaluation calls to 3000.
5.2 Overall Performance
Evaluation Metries For each benchmark function, we run baselines and our algorithm by least five different random seeds. Due to the limit the computational power available to us, we set the number of calls to the objective function to 3000. Our study evaluates the best-found value at every step and computes the mean and standard deviation of all runs. As result, we can compare

---

Bud.Rat. 1:2
Bud.Rat. 5:1
15-
Bud Rat.
-20
-40
-45
O
Descent TURBO
500 1000
1500
2000
2500
3000
500
1500
2000
samples
samples
(a) Optimization by different budget ratio
(b) Queried value from local optimizers
Figure 2: (a) illustrates the optimization curves for Ackley- 100d when the computational budget is divided between local descent and local BO in the ratio of 1:2, 5:1, and 1:10; (b) shows the values of Michalewicz-100d from local descent and local BO at each query.
the best-found value the end of the run well the speed which each algorithm is capable of reaching the most optimal result. There is a possibility that some algorithms will find the optimal value before 3000 calls, which will result an early stop.
illlstraaes the comparison between our model and baselines on benchmark sets. It was found that general, random search, CMA, and NM methods performed poorly in these cases since they do not cooperate with any approach that may potentially improve the efficiency of the sample.
Efficiency Fig. [
According to Fig. (a), (b), and (c), MCTD significantly improves the speed of finding better results for the set of synthetic functions compared TuRBO and La-MCTS. In particular, the Ackley synthetic function exhibits a noticeable improvement when we balance local optimization exploitation and state space exploration. Michalewiez is improved moderately through descent optimization, and MCTS helps improve the optimization consistently.
The Mujoco benchmark problems are very difficult for global optimization. Our approach is com- petitive with TuRBO and La-MCTS on this set and has moderate improvement over the average value on functions Hopper-33d. Walker-204d, and Cheetah-1022 In particular, the combination of local BO and local descent optimization speeds up the optimization during its early stages. It is, however, difficult to balance local exploitation and space exploration by picking the correct weights to bring recent improvements, exploration terms, and objective function values into the same order of magnitude. This is because we use the absolute value of the objective function that varies significantly at different optimization steps. In light of this, we see a large variation from different runs in this set, as in Fig (d), (e), (f), and (g).
In CIFAR-10/CIFAR-O0R MCTD reaches the optimal solution by a small number of samples, which is critical for NAS searches. The combination of descent and modeling approaches facilitates the search for the optimal solution more quickly than if only one method was used.
Descent Optimizer and Bayesian Optimizer We examine the performance of our approach when the computational budget is divided between a local descent model and local Bayesian optimizer TuRBO. Fig. [(a) illustrates the optimization history of Ackley-100d when budget ratios are 1:2, 1: and 1:10. It is demonstrated that model with high budget for local descent suffers from a low optimization rate. In contrast, the model with a high budget in the local Bayesian optimizer may have difficulty escaping the local optimal point.
As shown in the case of Fig 2(a) when we use the budget that emphasizes local descent (budget 5:1), the performance is less compared with that of emphasizing local BO (budget 1:20) in term of optimization speed. Based on the budget ratio for every function in supplementary material Tab.2, is generally advantageous to use at least the same (or even more) amount of computational budget on local BO as on local descent. This may be one challenge for the local descent approach, since this indicates that local descent may require local BO the oracle when the function landscape is

---

Gr
Get0 CHISE
R 2213
(a) Michalewicz- 100d
(b) Walker-1022
(c) Michalewicz-1000
(d) Walker-100 102d
Cy-t0
0y-22
(e) Michalewicz-10l Walker-102d
(g) Michalewicz- 100d
(h) Walker- 102d
(i) Michalewicz-100d 100d
Walker- 102d
(k) Michalewiez- 100d
Walker-1024
Figure 3: Ablation studies on function Richalewicz-1000 with hyper parameters (a) Cp. (c) Ca.
(e) Opp (g) C% (i) C² and (k) switching to fine-grain STP at function value; ablation studies on function Walker- 102d with hyper parameters (b) Cp. (d) Ca. (f) Cp (h) Cp, (i) C and (I) switching
to fine-grain STP at function value.
difficult, such as for the complicated functions in MoloCo locomotion and non-continuous funetions in NAS sets.
However, the local descent approach still proves beneficial despite these factors. From Fig 2(b) we can see the optimization improvement of the local BO becomes insignificant when the process close to the local optimum. Local descent, the other hand, can contribute steadily the discovery of a superior solution. To conclude, using a balanced approach can yield better results than using each approach separately.
5.3 Ablation Studies
We also perform ablation studies to understand the effect of the hyperparameters used in the algorithm, in three categories. The first category includes C₁ (the weight in uctexp) Cfi' and C² (the weights leaf exploration in Eq( that control the expansion of the tree. The second set of values, Ca and Cp in Bq/4] balance local exploitation and space exploration. Lastly, the threshold value determines when fine-grain descent is required. We use the synthetic function Michalewicz- wiwwicczzOOddd and the locomotion Walker- 102d for the ablation study, and each case with least 3 different seeds. Please note that hyperparameters in results may be different than those presented in Section 5.2. We found that a wise choice on Cp. C and C² critical improving performance, while C. and C" are less significant. The switching threshold value is highly dependent upon the objective function's 's properties.
State Space Exploration The parameters Ca and Cp balance exploration and exploitation of the existing tree. As shown (() the moderate choice on Cp improves the overall performance slightly; however, this is not clearly observed in Fig From Fig and 3(d) we can see variation Ca may not help significant changes
overall performance. Even so, we can observe
a contribution from Ca and Cp: from Fig. 3(b) and Fig 3(d) we can see that path selection with low
values of Ca and Cp leads to little variation between runs since the path selection tends select the node where the current
value resides, thus limiting the path selection.

---

Tree Expansion The hyperparameters C. Cp and Cy are important for expanding the current tree. As result of setting the parameters Cp. C* to large values and O to a small value, it is likely that a new sibling leaf will be created at the selected node to explore the state space. Alternatively, the path will tend to select the node that has the current optimal value. Since a leaf always has zero children, even though Cj and C% have the same functionality as C% the criteria for tree exploration different for branch and leaf nodes. According to Fig 3()) and Fig 3(h) it is evident that good choice on C% and C" can improve the optimization performance by exploring new state space distant from the local optimal value. Conversely, when their values are set either too large (orange lines Fig ( and (gg) or too small (blue line Fig 3()). this would adversely affect the overall performance of the optimization process. The effect of Ca is less noticeable. However, small C'll results in a small variation between different runs a similar behavior as Ca-
Switching at Function Value The fine- aineggrinn& STP can be beneficial in certain cases, as the orange In these two lines, switching takes place at a late stage of
lines show in 3ik and optimization, which results in excessive use of normal STP. Generally, fine-grain STP can be used as soon as possible. However, in some experiments, the fine-grain STP exploits too much in small neighborhood at an early stage of optimization and led to low-quality GP models.
Conclusion
In this paper, we proposed novel designs for using the MCTS framework in BBO problems, with more emphasis on sample-efficient local descent, instead of using MCTS for explicit space partitioning. We design new descent methods at vertices of the search tree that incorporate stochastic search and Gaussian Processes. The local models provide surrogate objectives to propose future samples without querying the ground truth function, and they also provide uncertainty metrics for exploration steps. We propose the corresponding rules for balancing progress and uncertainty. branch selection, tree expansion, and backpropagation. We evaluated the proposed methods on challenging benchmarks and observed clear benefits in improving the efficiency of BBO methods.
Acknowledgement
This material is based work supported by DARPA Contract No. PA8750-18-C-0NP AFOSR YIP 199550-19-1-0044 NSF Career CCF 2047034, NSF CCF DASS 2217723, and Amazon Research Award. We appreciate the valuable feedback from Ya-Chien Chang, Chiaki Hirayama, Zhizhen Qin, Chenning Yu, Eric Yu, Hongzhan Yu, Ruipeng Zhang, and the anonymous reviewers.
References
[1] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint 1rXiv:1611.0157. 2016
[2] Lucian Buçoniu, Alexander Daniels, Rémi Munos, and Robert Babuška. Optimistic planning for continuous-actio deterministic systems. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pages 69-76. IEEE, 2013.
[3] Beomjoon Kim, Kyungjae Lee, Sungbin Lim, Leslie Kaelbling, and Tomas Lozano-Perez. Monte carlo tree search in continuous spaces using voronoi optimistic optimization with regret bounds. Proceedings of the AAAI Conference on Artificial Intelligence, 34(06)9991699924 Apr. 2020. doi: 10. 1l.1609aaai..v 3440..6644466644
[4] Garrett Vanderplaats. Very large scale optimization. In 8th Symposium on Multidisciplinary Analysis and Optimization, page 4809, 2002.
[5] Zhenyu Yang. Bernhard Sendhoff, Ke Tang, and Xin Yao. Target shape design optimization by evolving b-splines with cooperative coevolution. Applied Soft Computing, 48:672-682, 2016.
[6] Shuhei Kimura, Kaori Ide, Aiko Kashihara, Makoto Kano, Mariko Hatakeyama, Ryoji Masui, Noriko Nakagawa, Shigeyuki Yokoyama, Seiki Kuramitsu, and Akihiko Konagaya. Infer- ence of s-system models of genetic networks using a cooperative coevolutionary algorithm. Bioinformatics, 21(7):1154-11666 2005.
10

---

[7] Donald R Jones, Matthias Schonlau, and William Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455 1998.
[8] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning. pages 63-71. Springer, 2003.
[9] David Eriksson, Michael Pearce, Jacob Gardner, Ryan D Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in Neural Information Processing Systems, 32:5496-5507, 2019.
[10] Darrall Henderson, Sheldon H. Jacobson, and Alan W. Johnson. The Theory and Practice of Simulated Annealing, pages 287-319. Springer US, Boston, MA, 2003.
[11]
Pieter-Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. tutorial on the 2005.
cross-entropy method. Annals of operations research,
[12] Ralf Salomon. Evolutionary algorithms and gradient search: Similarities and differences. IEEE Transactions on Evolutionary Computation, 2(2):45-55, 1998.
[13] Fuchang Gao and Lixing Han. Implementing the nelder-meat simplex algorithm with adaptive parameters. Comput. Optim. Appl., 51(11:259-22) jan 2012.
[14] Tamara G Kolda, Robert Michael Lewis, and Virginia Torezon. Optimization by direct search: New perspectives on some classical and modern methods. SIAM review, 45(3):385-443 2003.
[15] Robert Michael Lewis, Virginia Torezon, and Michael W Trosset. Direct search methods: Then and now. Journal of computational and Applied Mathematics, 124(1-2):191-200) 2000.
[16] David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-, October 2017. /ttthttp://dd doi. org/ 1/g//1 10. 0038/nature24200
[17] Cameron Browne, Edward Jack Powley, Daniel Whitehouse, Simon M. Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez Liebana, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. JEEE Trans. Comput. Intellig. and Al in Games, 4(1): (1)):1-43, 2012. URL http:://tllp: uni uni-tuiieiieeeeeee de/ dt//bb/joornals/cciiiiig tciaig4. htm letssmm#BBoooee www.HHBBroooePPPCPPICIII
[18] Rémi Munos. Optimistic optimization of deterministic function without the knowledge of its smoothness. In J. Shawe- R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
[19] Linnan Wang. Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimization using monte carlo tree search. arXiv preprint arXiv:2007.00700 2020.
[20] El Houcine Bergou, Eduard Gorbunov, and Peter Richtárik. Stochastic three points method for unconstrained smooth minimization. SIAM Journal on Optimization, 30(4):2726-274) 2749, 2020.
[21] Giovanni Lavezzi, Kidus Guye, and Marco Ciarcià. Nonlinear programming solvers for unconstrained and constrained optimization problems: benchmark analysis, 2022. URL
htttps///arxvv.orr org/ abs/ 05297
[22] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEFF, 2012.
[23] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001 000366, 2020.
[24] Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. CMA-ES/pyema on Github. Zenodo,
101:10.528 (2238///2oooo25556666 February 2019. URI. httpthts:://o..rrgg//// http://oo...or/////.5/1/5/////// 5281/2eeodo 2559634
[25] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104
(1):148-175, 2016.
[26] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information- theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transac- tions on Information Theory, 58(5):350--325, May 2012.

---

[27] Peter Frazier. A tutorial on bayesian optimization. arXiv preprint arXiv: 1807.02811, 2018.
[28] Chang Oh, Efstratios Gavves, and Max Welling. Bock: Bayesian optimization with cylindrical kernels. In International Conference on Machine Learning, pages 3868-3877.
PMLR, 2018.
[29] Jacob Gardner, Chuan Guo, Kilian Weinberger, Roman Garnett, and Roger Grosse. Discovering and exploiting additive structure for bayesian optimization. In Artificial Intelligence and Statistics, pages 1311-1319. PMLR, 2017.
[30] Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. High-dimensional bayesian optimization via additive models with overlapping groups. In International conference on artificial intelligence and statistics, pages 298-307. PMLR, 2018.
[31] Mojmír Mutný and Andreas Krause. Efficient high dimensional bayesian optimization with additivity and quadrature fourier ffeatures. Advances in Neural Information Processing Systems
31. pages 9005-9016, 2019.
[32] Enrique Baeyens, Alberto Herreros, and José R Perán. A direct search algorithm for global optimization. Algorithms, 9(2):00, 2016.
[33] Sébastien Bubeck, Rémi Munos, Gilles Stoltz, and Csaba Szepesvári. X-armed bandits. Journal of Machine Learning Research, 12(5), 2011.
12

---

Appendix A
Tree expansion illustration
(a)
(b)
(c)
(d)
(e)
11
n
1gg
ngg
1100
noi
Roo
Topy Y
Ro2
R111
Top1 Mo11
Moro
Tp12
Figure 4: Top: Illustration of the nodes in the input domain; Bottom: Illustration of the tree expansion. (a) The root ni begins with a random sample. (b) Optimization is carried out on the root no- (c) Leaf exploration on the root / creates two nodes 7000 and no1 no0 starts from zij and no1 starts from point distant from zú- (d) Leaf exploration on the node Ro1 generating two new node no10 and 0011- noio starts from zú1 while Ro11 starts point away from zūj- (e) Branch exploration at root no creates a new child node no2.
Appendix B
Benchmark Sets
Synthetic Functions We chose Ackley and Michalewicz from the synthetic function set in the nonlinear optimization benchmark [2I]. Ackley is function with multiple local minima, and Michalewicz has steep valleys and many ridges. We use Ackley-50d, Ackley-1006 and Michalewicz- 100d as our benchmark.
MuJoCo Locomotion For reinforcement learning problems from MuJoCo locomotion environ- ments we chose Hopper, Walker, and HalfCheetah for tests. Hopper has dimensions action space and 11 in observation We choose linear policy matrix to search for maximizing the reward, therefore, the search space for Hopper-33d is in dimension of 3- 11 = 33. Similarly, we set linear policies in both Walker- 102d and HalfCheetah- 102d. In addition to the above linear policy, we double the weighting matrix space dimension in Walker from 102 to 204, such that Wis s + W2s where W1 and W2 are matrices in the dimension of 102. In this case, the optimization problem Walker-204d. Since our approach considers deterministic results, set the noise scale zero in all MuloCo environments avoid randomness in rewards.
W's in which W is the weighting
Neural Architecture Search For the NAS benchmark, we use two datasets CIFAR- 10 and CIFAR- 100 from TAS-Bench-201 (23). Each network in the datasets consists of three stacks of searching cells, and each cell has six positions where one can select one type of layer from five different types: (1) zeroize, (2) skip connection, (3) 1-by-1 convolution, (4) 3-by-3 convolution, and (5) 3-by-3 average pooling layer. Overall, there are 56 = 15625 different types of architectures, and each architecture is trained and evaluated on both CIFAR-10 and CIFAR-100. The accuracy of training and evaluation is recorded. To benchmark this set, we created the following functions in the real domain: replace each of the five types of layers with integer, and the real-valued input is rounded up to the nearest integer. The evaluation accuracy of the architecture is set as the function value, As an example, we set the input domain to { (0..5 5.5]6, }, and f[1111]) = f[[1])), where each 1 1- 5 corresponds to one type of the layers. It should be noted that in this method different inputs may refer to the same network architecture; therefore, the number of unique architectures examined is less than the number of functions called.
13


Reviews:
[{"review_id": "t4f9SIZLbXR", "reviewer": "Reviewer_6Fro", "summary": "The paper considers optimization of deterministic black-box functions. The proposed algorithm combines Monte Carlo tree search with local descent/bayesian optimization. Empirical work shows elicits strong performance of the proposed method compared to various baselines on a diverse set of domains.", "questions": "-", "limitations": "-", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is poorly written, and often it is difficult to figure out the intention of the authors. For instance, in Algorithm 1, the index i is undefined and probably is different from the index i in (2), which probably is j in Alg.1. dy is also undefined in the algorithm, although (2) gives sufficient hint about it. Other variables are undefined as well, but familiarity with MCTS is sufficient to have a reasonable guess. The authors often inform us what the algorithm is not, instead of focusing on a clear description of the algorithm, and explaining the differences subsequently. \n\nTherefore, while the algorithm could be interesting, it is fairly difficult to evaluate the strength of it.\n\nThe empirical results seem to indicate that the proposed algorithm performs well. Some of the domain choices seem a bit strange for black-box optimization (why would we want to optimize a linear policy on a MuJoCo domain this way, and not a more complex policy by reinforcement learning?), but ultimately it is possible to use these domains as well for testing. \n\n-------------------------\nAfter rebuttal:\n\nI appreciate the effort of the authors to improve the paper according to the suggestions of the reviewers. I hope, it will become a better paper after a few more iterations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "laAYALK9Hiw", "reviewer": "Reviewer_sxfi", "summary": "This paper proposes a new MCTS-based optimization method that combines descent methods and BO. Different from LA-MCTS, the proposed MCDescent does not use partition due to the limitation of high-dimensional space and uses a set of unique mechanisms in the MCTS framework. Expeirments show the MCDescent can outperform baselines and sota black-box optimization methods.", "questions": "How does MCDescent compare to other methods in terms of running time? The author should discuss this and shows it in the experiments.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Although the experimental results show the superior performance of the proposed method, I have the following concerns.\n\n1. The novelty of the proposed method is limited. Authors may need to highlight the motivation and make it reasonable, explain in detail why you combine the two approaches.\n\n2. The writing of this paper should be improved. For example,  Algorithm 1 should be rewritten for clarity, the notations used in the current version are very confusing and inconsistent.\n\n3. More experiments should be done to test the sensitivity of the hyper-parameters in the MCDescent, e.g., $C_p$. \n\n4. The number of samples is too small for the RL tasks. Although the authors want to examine the performance of different algorithms under a limited number of evaluations, the too-small budgets lead to inferior performance, and the policy does not converge on some tasks. For example, in the Walker environment, the number of samples used in LA-MCTS paper is 40000. However, it is only 3000 in this paper. \n\n5. The authors said, \" the performance of MCDescent may be limited in some high dimensional spaces that are inhospitable to descent\napproaches\". It is better to examine MCDescent on high-dimensional synthetic functions.\n\nThere are also some minor problems.\n\n1. More references are needed, such as lines 34-35.\n\n2. The experiments of Tree expansion and Optimization route can not be named \"Ablation study.\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bEjGmK4LDlp", "reviewer": "Reviewer_z8ZA", "summary": "This paper focuses on the problem of Black-Box-Optimization, optimizing a function while only being able to sample its value at different points.  The authors propose a novel combination of building a tree to organize different regions that are being searched, and uses modifications of STP and TuRBO to optimize from a given point.  The entire method balances exploring new regions for more promising spots with exploiting and optimizing in a good spot.  Modified UCT is used to guide traversal of the tree, which also includes an option to create a new child at each node.  This overall method is experimentally evaluated in many settings and shown to be competitive with previous approaches, while surpassing their performance on many problems. \n", "questions": "- On line 267 you state that some high dimensional spaces are inhospitable to descent approachs?  What are the characteristics of these inhospitable spaces? How can you tell if a space is inhospitable?\n", "limitations": "This was addressed to my satisfaction. \n", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- I quite liked this paper and felt that its approach to combining MCTS with BO was unique and makes a very solid contribution. \n- The evaluation is thorough and complete and is very convincing as to the strengths of the proposed methods. \n- I appreciated the ablation studies that were done.  Overall the evaluation feels solid. \n- The paper is well-written and clear.  \n\nWeaknesses: \n- The fact that the proposed methods are focused on deterministic functions wasn't made explicit until Line 216 on page 6, and later in the conclusion.  I think this would have been good to make clear from the very beginning. \n- The uncertainty regions in some of the results are very large.  Could more runs have been performed to bring down this uncertainty? \n- I am not sure that Figure 4 conveyed much meaning to me.  It was hard to understand the point you were trying to get across. \n\nMinor feedback: \n- Line 67-68 - This sentence doesn't make very much sense to me as written. \n- Line 122 - \"... have similar values...\" might better be: \"... have more similar values...\"\n- Algorithm 1 - \"GP.correlation lengh\" seems like it should be \"GP.correlation length\" \n- Line 176 - \"In the scene the ...\" might sound better as \"In the event that the ...\"\n- Line 227 - \"...we create following ...\" perhaps should be \"...we create the following...\"\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BuCCRbEp9c", "reviewer": "Reviewer_TrTs", "summary": "This paper proposes a method for Black-Box Optimization which incorporates the idea of UCT in sampling.\nThe proposed method, MCDescent, shows promising results in several benchmarks.\n", "questions": "There should be a discussion focusing on the hyperparameters.\nFor example, if possible, I would like to see the following explanation.\n1. How sensitive the performances are to the hyperparameters.\n1. How the authors chose the values in Table 2.\n1. Is there any guideline for tuning the hyperparameters?\n\nWhere is the definition of $C_d$?\nI may have overlooked it, but I could not find it in the main material.\n(It is mentioned in the supplementary as \"weight of recent improvement.\")\n\nI think there are better citations for CMA-ES.\nPlease explain the reason if it is meaningful to cite this specific implementation.\n\nThis is not a question, but I would like to see Figure 1 of the appendix\nin the main material if possible because it is very helpful for understanding the algorithm.", "limitations": "I could not think of any direct possibility of potential negative societal impact.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea is simple, and the algorithm is easy to understand.\nProbably the implementation is also not complex.\n\nWeaknesses:\nThe algorithm has eight hyperparameters, and their values vary for different benchmarks.\nAlso, there is no explanation about how they are tuned.\nI would be happy to give the paper a better score if this weakness is solved.\n\n---\nPost rebuttal.\nThank you very much for the detailed comments on my concerns.\nI changed my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept