SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MTE. When zero-shot generalization is unachievable due to significant task disparity, the Meta Transfer Efficiency (MTE) metric is proposed to evaluate the transfer learning capacity of a MetaBBO-RL approach. We begin by maintaining checkpoints of a MetaBBO-RL approach trained on a problem set  $B$ . We locate the checkpoint with the highest cumulative return, recording its index as  $T_{\mathrm{scratch}}$ . Next, we pre-train a checkpoint of the MetaBBO-RL approach on another problem set  $A$ , then load this checkpoint back and continue the training on the problem set  $B$  until it reaches the same best-accumulated return, recording the current index as  $T_{\mathrm{finetune}}$ . The  $MTE(A, B)$  is calculated by:
$$
M T E (A, B) = 100 \times \left(1 - \frac {T _ {\text {finetune}}}{T _ {\text {scratch}}}\right) \%, \tag{5}
$$
where a larger  $MTE(A, B)$  indicates that the knowledge learned in  $A$  can be easily transferred to solve  $B$ , while an MTE value less than or equal to zero indicates potential negative transfer issues. Similar to MGD, the MTE has neither symmetry nor transitivity properties.
4 Benchmarking Study
MetaBox serves as a valuable tool for conducting experimental studies, allowing researchers to 1) benchmark specific groups of MetaBBO algorithms, 2) tune their algorithms flexibly, and 3) perform in-depth analysis on various aspects including generalization and transfer learning abilities. In this section, we provide several examples to illustrate the use of MetaBox in experimental studies.
To initiate the fully automated Train-Test-Log process of MetaBox, users need to follow two steps: 1) indicate environment parameters, including problem-type, problem-dim and difficulty, to inform MetaBox about the problem set, its dimension and the train-test split used in the comparison study; and 2) specify the maximum number of training steps  $(M)$ , the number of independent test runs  $(N)$ , and the reserved function evaluation times (maxFEs) for solving a problem instance. Once these two steps are completed, users can trigger the runExperiment() to enjoy the full automation process. In the exemplary study below,  $M$ ,  $N$  and maxFEs are set to  $1.5 \times 10^{6}$ , 51, and  $2 \times 10^{4}$ , respectively, unless specified otherwise. All results presented are obtained using a machine of Intel i9-10980XE CPU with 32GB RAM. Note that MetaBox is also compatible with other platforms. We provide default control parameters for each baseline, which are listed in Appendix C.
4.2 Comparison of different baseline (Meta)BBO methods
We train all MetaBBO baselines (7 MetaBBO-RL and 1 MetaBBO-SL) in the Baseline Library on Synthetic and Protein-Docking testsuites, using both easy and difficult train-test split. We then test these baselines against the classic black-box optimizers available in our library and calculate the AEI of each algorithm. The results are depicted in Figure 4. On the Synthetic testsuites, two MetaBBO-RL methods, namely LDE and RLEPSO, show competitive performance against classic baselines. Specifically, LDE outperforms JDE21, the winner of IEEE CEC 2021 BBO Competitions [34], on the Synthetic-difficult testsuites, while RLEPSO outperforms both JDE21 and NL-SHADE-LBC, the winner of IEEE CEC 2022 BBO Competitions [55], on the Synthetic-easy testsuites. However, for this particular problem set, CMA-ES remains a strong baseline. As we shift to more realistic testsuites (Protein-docking of 280 instances), most baselines show varying degrees of performance drop. Notably, CMA-ES is inferior to a number of algorithms such as DEDQN, LDE, RLEPSO, and GLPSO. In general, we observe that MetaBBO-RL methods are more robust than classic hand-crafted
Figure 4: AEI scores of baselines, with error bars denoting the robustness across different tasks. Top: Results on Synthetic testsuites with easy or difficult difficulty. Bottom: Results on Protein-Docking testsuites with easy or difficult difficulty.
Figure 5: Hyper-tuning for the MetaBBO-RL approach LDE [21] using a  $2 \times 2$  grid search. Left: The average return during the training over 10 trials. Middle: The normalized cost over 51 independent runs during the test. Right: The corresponding AEI scores during the test.
optimizers when tested on different problem sets. For example, DEDQN generally performs the best for the Protein-Docking problems, ranking first and third in difficult and easy modes, respectively. This suggests that the classic optimizers might be only tailored to a specific family of problems, while MetaBBO-RL learns to enhance the low-level optimizer through meta-level learning, thereby exhibiting stronger robustness. Due to the space limitation, we leave additional comparison results, including those on other test suites, as well as the detailed sub-metrics such as running time, algorithm complexity, and per-instance optimization results in Appendix D.
4.3 Hyper-tuning a MetaBBO-RL approach
Tuning a MetaBBO-RL requires efforts in configuring both the meta-level RL Agent and the low-level black-box Optimizer. Fortunately, our MetaBox provides Template coding with convenient interfaces to assist users in accomplishing this task. To illustrate this, we conduct a  $2 \times 2$  grid search to hyper-tune the MetaBBO-RL approach LDE [21]. We noticed that LDE achieved lower AEI scores on Synthetic-easy test suites than some classic optimizers such as JDE21 and MadDE (see Figure 4). In our study, we investigate both REINFORCE [51] and PPO [52] RL agents for LDE, setting the population size (a hyper-parameter of DE) to either 30 or 50 (where the original version of LDE uses
Figure 6: Meta performance (MGD and MTE) of LDE [21] across different tasks, tested under the easy mode. Left: Logits on  $i$ -th row and  $j$ -th column is the  $MGD(i,j)$ , the smaller the better. Right: The average return over 10 trials is compared between the LDE models pre-trained and trained from scratch. The annotated MTE score shows that it saves 0.2x learning steps through fine-tuning.
REINFORCE and a population size of 50). Figure 5 depicts the average return along the training, the optimization curve during testing, and the AEI scores of these four LDE versions. It turns out that LDE with the PPO agent and a population size of 30 achieves higher scores and surpasses JDE21 and MadDE. This shows that, by leveraging the capabilities of MetaBox and conducting hyper-tuning studies, researchers are able to pursue the best configurations for their MetaBBO-RL methods.
4.4 Investigating generalization and transfer learning performance
Existing MetaBBO-RL methods [19-25] rarely assess their generalization and transfer ability, although they are two crucial aspects for evaluating the effectiveness of a learning-based method. To bridge this gap, we have introduced two explicit indicators (namely the MGD and MTE) in MetaBox to measure the level of learning achieved by a MetaBBO-RL algorithm. In Figure 6, we display the values of MGD and MTE for the MetaBBO-RL approach LDE.
Generalization performance. Results in the MGD plot (the left part of Figure 6) show that the pre-trained LDE model possesses a strong generalization ability. When the model is pre-trained on the Noisy-Synthetic testsuites, it exhibits a positive MGD on the Synthetic testsuites, with only a  $2.222\%$  drop in AEI, which is considered acceptable. Additionally, the LDE model trained on the Protein-Docking-easy or Synthetic-easy testsuites both exhibits exceptional generalization to the Noisy-Synthetic-easy testsuites, as evidenced by an MGD of  $-7.006\%$  and  $-7.049\%$ . Such robust generalization abilities are highly desirable in practical applications.
Transfer performance. We fine-tune the LDE pre-trained on the Noisy-Synthetic testsuites for the Synthetic testsuites. The progress of this fine-tuning process is depicted in the MTE plot (the right part of Figure 6), showing the average return over 10 trials. The results reveal that the pre-trained LDE requires fewer learning steps to arrive at the peak return level than the LDE trained from scratch, with the MTE value of  $20\%$ . Compared with the MGD value of  $2.222\%$  when zero-shooting the pre-trained LDE for the Synthetic testsuites, it can be noticed that the MGD and MTE capture two different aspects of the learning effectiveness. By considering both measures, researchers can gain deeper insights into the learning behavior and effectiveness of their MetaBBO-RL approaches.
We propose MetaBox, a benchmark platform for Meta-Black-Box Optimization with Reinforcement Learning (MetaBBO-RL). It intends to, 1) provide the first unified benchmark platform, 2) simplify coding towards efficient researching, 3) provide broad test suites and baselines for comprehensive comparison, and 4) provide novel evaluation metrics for in-depth analysis.
Our preliminary experimental investigations lead to the following key observations. First, although hand-crafted optimizers currently outperform learning-based optimizers (e.g., MetaBBO-RL), they are mostly designed to specific problem sets and become less effective when applied to other problem domains. In contrast, MetaBBO-RL exhibits adaptability across various optimization scenarios,
showing stronger robustness across different problem domains. This is a significant advantage, as it allows MetaBBO-RL to perform well on diverse problem sets without the need for much manual customization. Second, there is still room for further improvement in MetaBBO-RL, by discovering more effective designs in both meta-level agents and low-level optimizers. We find that even basic modifications, such as incorporating a PPO agent and adjusting the population size, can bring a significant performance boost. Third, interpreting the generalization and transfer effects in MetaBBO-RL can be challenging. While we were able to observe positive and negative effects in terms of MGD and MTE, rationally interpreting and understanding them is still difficult. Hence, additional studies are needed to interpret the learning effects of MetaBBO-RL in a more comprehensible manner.
While our MetaBox presents a significant advancement, we also acknowledge its potential limitations. It is important to note that the evaluation of BBO performance is not a one-size-fits-all endeavour. Different practical applications may have varying preferences and additional concerns, such as solution robustness, solution diversity, parallelization and scalability. These nuances extend beyond the scope of a single evaluation metric. In conjunction with the proposed AEI, MetaBox is committed to exploring more pragmatic and advanced evaluation criteria. Moreover, the problem sets in MetaBox will be expanded to include more diverse and eye-catching BBO tasks. Additionally, proactive maintenance and updates to MetaBox, including the extension of the baseline library, will also be pursued to ensure it is up-to-date with the latest methodologies in the field.
This work was supported in part by the National Natural Science Foundation of China under Grant 62276100, in part by the Guangdong Natural Science Funds for Distinguished Young Scholars under Grant 2022B1515020049, in part by the Guangdong Regional Joint Funds for Basic and Applied Research under Grant 2021B1515120078, and in part by the TCL Young Scholars Program.
[1] Matthias Feurer and Frank Hutter. Hyperparameter optimization. Automated Machine Learning: Methods, Systems, Challenges, pages 3-33, 2019.
[2] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 20(1):1997-2017, 2019.
[3] Graham R Smith and Michael JE Sternberg. Prediction of protein-protein interactions by docking methods. Current Opinion in Structural Biology, 12(1):28-35, 2002.
[4] John H Holland. Adaptation in natural and artificial systems. MIT press, 1975.
[5] Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9(2):159-195, 2001.
[6] Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary Computation, 11(1):1-18, 2003.
[7] Raymond Ros and Nikolaus Hansen. A simple modification in cma-es achieving linear time and space complexity. In Parallel Problem Solving from Nature-PPSN X: 10th International Conference, Dortmund, Germany, September 13-17, 2008. Proceedings 10, pages 296-305. Springer, 2008.
[8] James Kennedy and Russell Eberhart. Particle swarm optimization. In Proceedings of ICNN'95-International Conference on Neural Networks, pages 1942-1948. IEEE, 1995.
[9] Yue-Jiao Gong, Jing-Jing Li, Yicong Zhou, Yun Li, Henry Shu-Hung Chung, Yu-Hui Shi, and Jun Zhang. Genetic learning particle swarm optimization. IEEE Transactions on Cybernetics, 46(10):2277-2290, 2015.
[10] Rainer Storn and Kenneth Price. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 11(4):341, 1997.
MetaBox: A Benchmark Platform for Meta-Black-Box Optimization with Reinforcement Learning (Appendix)
A Details of Testsuites
A.1 Synthetic
There are 24 functions in the Synthetic testsuites. These noiseless synthetic functions, established by Hansen et al. [47], are taken from COCO [28]. The mathematical definitions and formulations of these functions can be found in Hansen et al. [47]. We have listed these functions in Table 2. To provide different levels of difficulty, we have divided the 24 functions into two parts:  $25\%$  and  $75\%$ . The  $25\%$  portion includes functions No.1, 5, 6, 10, 15, and 20, while the remaining 18 functions make up the  $75\%$  portion. In the easy mode, the  $75\%$  portion is used for training, and the  $25\%$  portion is used for testing. On the other hand, in the difficult mode, the  $25\%$  portion is used for training, and the  $75\%$  portion is used for testing. The maximum number of function evaluations (maxFEs) for problems in these test suites is set to  $2 \times 10^4$ .
Table 2: Summary of the Synthetic test functions.
No.	Functions	fi*
Separable functions	1	Sphere Function	700
2	Ellipsoidal Function	1900
3	Rastrigin Function	100
4	Buche-Rastrigin Function	1000
5	Linear Slope	2000
Functions with low or moderate conditioning	6	Attractive Sector Function	400
7	Step Ellipsoidal Function	300
8	Rosenbrock Function, original	1300
9	Rosenbrock Function, rotated	1300
Functions with high conditioning and unimodal	10	Ellipsoidal Function	600
11	Discus Function	900
12	Bent Cigar Function	2000
13	Sharp Ridge Function	2400
14	Different Powers Function	1500
Multi-modal functions with adequate global structure	15	Rastrigin Function (non-separable counterpart of F3)	1700
16	Weierstrass Function	1400
17	Schaffers F7 Function	200
18	Schaffers F7 Function, moderately ill-conditioned	700
19	Composite Griewank-Rosenbrock Function F8F2	1700
Multi-modal functions with weak global structure	20	Schwefel Function	2100
21	Gallagher's Gaussian 101-me Peaks Function	700
22	Gallagher's Gaussian 21-hi Peaks Function	2400
23	Katsuura Function	600
24	Lunacek bi-Rastrigin Function	1200
Default search range: [-5, 5]D
A.2 Noisy-Synthetic
There are 30 functions in the Noisy-Synthetic testsuites. These noiseless synthetic functions, established by Hansen et al. [47], are taken from COCO [28]. The mathematical definitions and formulations of these functions can be found in Hansen et al. [47]. We have listed these functions in
Table 2. To provide different levels of difficulty, we have divided the 24 functions into two parts:  $25\%$  and  $75\%$ . The  $25\%$  portion includes functions No.1, 5, 15, 16, 17, 19, 20, 25, while the remaining 22 functions make up the  $75\%$  portion. The maximum number of function evaluations (maxFEs) for problems in these test suites is set to  $2 \times 10^{4}$ .
Table 3: Summary of the Noisy-Synthetic test functions.
No.	Functions	f_i*
Functions with moderate noise	1	Sphere with moderate gaussian noise	700
2	Sphere with moderate uniform noise	1900
3	Sphere with moderate seldom cauchy noise	100
4	Rosenbrock with moderate gaussian noise	1000
5	Rosenbrock with moderate uniform noise	2000
6	Rosenbrock with moderate seldom cauchy noise	400
Functions with severe noise	7	Sphere with gaussian noise	100
8	Sphere with uniform noise	2000
9	Sphere with seldom cauchy noise	2000
10	Rosenbrock with gaussian noise	500
11	Rosenbrock with uniform noise	400
12	Rosenbrock with seldom cauchy noise	700
13	Step ellipsoid with gaussian noise	1900
14	Step ellipsoid with uniform noise	800
15	Step ellipsoid with seldom cauchy noise	2300
16	Ellipsoid with gaussian noise	200
17	Ellipsoid with uniform noise	100
18	Ellipsoid with seldom cauchy noise	200
19	Different Powers with gaussian noise	500
20	Different Powers with uniform noise	1600
21	Different Powers with seldom cauchy noise	1000
Highly multi-modal functions with severe noise	22	Schaffer's F7 with gaussian noise	2400
23	Schaffer's F7 with uniform noise	1400
24	Schaffer's F7 with seldom cauchy noise	2400
25	Composite Griewank-Rosenbrock with gaussian noise	2200
26	Composite Griewank-Rosenbrock with uniform noise	400
27	Composite Griewank-Rosenbrock with seldom cauchy noise	2500
28	Gallagher's Gaussian Peaks 101-me with gaussian noise	1000
29	Gallagher's Gaussian Peaks 101-me with uniform noise	600
30	Gallagher's Gaussian Peaks 101-me with seldom cauchy noise	2100
Default search range: [-5, 5]D
A.3 Protein-Docking
Ab initio protein docking poses a significant challenge in optimizing a noisy and costly black-box function [56]. The objective of ab initio protein docking, for a fixed basis protein conformation  $x_0$ , is to minimize the change in Gibbs free energy resulting from protein interaction between  $x_0$  and any other conformation  $x$ , denoted as  $E_{int}(x,x_0)$ . Following the approach of Moal and Bates [57], we formulate the objective as follows:
$$
\min  _ {x} E _ {i n t} (x, x _ {0}) = \sum_ {i} ^ {\text {a t o m s}} \sum_ {j} ^ {\text {a t o m s}} E \left(x ^ {i}, x _ {0} ^ {j}\right), \tag {6}
$$
where  $E(x^{i},x_{0}^{j})$  is the energy between any pair atoms of  $x$  and  $x_0$ , and is defined as:
$$
E _ {i, j} = \left\{ \begin{array}{l l} \frac {q _ {i} q _ {j}}{\epsilon r _ {i , j}} + \sqrt {\epsilon_ {i} \epsilon_ {j}} \left[ \left(\frac {R _ {i , j}}{r _ {i , j}}\right) ^ {1 2} - \left(\frac {R _ {i , j}}{r _ {i , j}}\right) ^ {6} \right], & \text {i f} r _ {i, j} <   7 \\ \left[ \frac {\left(r _ {\text {o f f} - f _ {i , j}}\right) ^ {2} \left(r _ {\text {o f f}} + 2 r _ {i , j} - 3 r _ {\text {o n}}\right)}{\left(r _ {\text {o f f}} - r _ {\text {o n}}\right) ^ {3}} \right] \left\{\frac {q _ {i} q _ {j}}{\epsilon r _ {i , j}} + \sqrt {\epsilon_ {i} \epsilon_ {j}} \left[ \left(\frac {R _ {i , j}}{r _ {i , j}}\right) ^ {1 2} - \left(\frac {R _ {i , j}}{r _ {i , j}}\right) ^ {6} \right] \right\}, & \text {i f} 7 \leq r _ {i, j} \leq 9 \\ 0 & \text {i f} r _ {i, j} > 9 \end{array} \right. \tag {7}
$$
All parameters and calculations are taken from the Charmm19 force field [58]. A switching function is employed, between 7 and 9, to disregard long-distance interaction energy. To construct the task
instance  $x_0$  for protein docking, we select 28 protein-protein complexes from the protein docking benchmark set 4.0 [29]. Each complex is associated with 10 different starting points, chosen from the top-10 start points identified by ZDOCK [59]. Consequently, the Protein-Docking testsuites comprise a total of 280 docking task instances. We split these instances into two parts, with approximately  $75\%$  and  $25\%$  of the instances serving as the difficulty levels within the testsuites. It is important to note that we parameterize the search space as  $\mathbb{R}^{12}$ , which is a reduced dimensionality compared to the original protein complex [56]. This reduction in dimensionality enables computational time savings while retaining the optimization nature of the problem. Due to the computationally expensive evaluations, the maximum number of function evaluations (maxFEs) is set to  $10^3$ . The default search range for the optimization is  $[-\infty, \infty]$ .
B Details of AEI Calculation
We describe the calculations of AEI as follows. Suppose we want to test a (Meta)BBO approach  $\Lambda$  to obtain its AEI on Testsuites  $A$ , i.e.,  $\mathrm{AEI}(\Lambda, A)$ . Suppose  $A$  has  $K$  problem instances, and we run  $\Lambda$  to optimize each problem in  $A$  for  $N$  times. In this testing process, we record 3 kinds of values:
Best objective value. The best objective value  $v_{obj}^{k,n}$  denotes the best objective found by  $\Lambda$  on the  $k$ -th problem during the  $n$ -th test run. To convert its monotonicity, we conduct an inverse transformation, i.e.,  $v_{obj}^{k,n} = \frac{1}{v_{obj}^{k,n}}$ .
Consumed function evaluations. We let  $v_{fes}^{k,n}$  denote the number of consumed function evaluations when the approach  $\Lambda$  terminates the optimization process for the  $k$ -th problem during the  $n$ -th test run, where  $\Lambda$  would terminate the optimization either upon locating a high-quality solution that fulfills a predetermined accuracy or when it reaches the maximum allowed function evaluations, i.e., maxFEs. We then pre-process this value by a min-max conversion for the consistency towards AEI maximization, i.e.,  $v_{fes}^{k,n} = \frac{\maxFEs}{v_{fes}^{k,n}}$ .
Runtime complexity. The calculation includes three steps following the conventions of IEEE CEC Competitions [34, 55]. We first pre-calculate a  $T_{0}$  which is a basic normalization term, calculating the computation time of some basic numpy operations (e.g., add, division, log, exp) as references. We record the time  $T_{1}^{k,n}$  for  $\Lambda$  to evaluate the  $k$ -th problem's objective during  $n$ -th run, and  $T_{2}^{k,n}$  as total running time for  $\Lambda$  to solve the  $k$ -th problem's objective during  $n$ -th run. The runtime complexity  $v_{\text{com}}^{k,n}$  is then calculated by  $\frac{T_{2}^{k,n} - T_{1}^{k,n}}{T_{0}}$ . Then we pre-process this value by a min-max conversion for consistency towards AEI maximization, i.e.,  $v_{\text{obj}}^{k,n} = \frac{1}{v_{\text{obj}}^{k,n}}$ .
C Baseline Setup
Classic optimizer. Random Search does not necessitate any control parameters. We establish the control parameters for CMA-ES, DE, and PSO in alignment with DEAP [53], while the control parameters for BO are set in accordance with Scikit-Optimizer [54]. For the classic optimizers implemented by us, the control parameters are delineated and can be accessed at https://github.com/GMC-DRL/MetaBox/blob/main/src/control_parameters_classic.md.
MetaBBO-SL. We briefly introduce the control parameters of RNN-OI here. It adopts an LSTM network to generate the next solution position on the observation of some present optimization status. This LSTM has input size as problem_dim + 2, hidden size as 32, and projection size as problem_dim. The Adam optimizer is used to train the network with a fixed learning rate  $10^{-5}$ . However, it can only optimize a problem for around maxFEs = 100 due to the inefficiency arising from utilizing RNNs to model an extended optimization horizon.
Figure 7: AEI scores with error bars of baselines on Noisy-Synthetic testsuites.
Figure 8: Average cost curve of baselines over Noisy-Synthetic-easy testsuites.
D Additional Results and Comparisons
D.1 Additional results
In addition to the AEI scores, MetaBox provides additional experimental results that complement the comprehensive analysis of MetaBBO-RL approaches. These results, along with AEI, are automatically generated and presented after the testing process. The currently available results for all baselines in Baseline Library can be accessed at https://github.com/GMC-DRL/MetaBox/blob/main/post_proteased_data/content.md. We briefly preview these additional results here:
Overall performance table. For each baseline in the Baseline Library, it presents the average final objective value  $(Obj)$ , the average Gap with respect to the performance of CMA-ES, and the average consumed function evaluation times (FEs). These values are obtained based on 51 independent runs.