SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zeyuan Ma $^{1}$ , Hongshu Guo $^{1}$ , Jiacheng Chen $^{1}$ , Zhenrui Li $^{1}$ , Guojun Peng $^{1}$ , Yue-Jiao Gong $^{1,*}$ , Yining Ma $^{2}$ , Zhiguang Cao $^{3}$
$^{1}$ South China University of Technology
$^{2}$ National University of Singapore
$^{3}$ Singapore Management University
{scut.crazynicolas, guohongshu369, jackchan9345}@gmail.com,
zhenrui.li@outlook.com, {pgj20010419, gongyuejiao}@gmail.com,
yiningma@u.nus.edu, zhiguangcao@outlook.com
Recently, Meta-Black-Box Optimization with Reinforcement Learning (MetaBBO-RL) has showcased the power of leveraging RL at the meta-level to mitigate manual fine-tuning of low-level black-box optimizers. However, this field is hindered by the lack of a unified benchmark. To fill this gap, we introduce MetaBox, the first benchmark platform expressly tailored for developing and evaluating MetaBBO-RL methods. MetaBox offers a flexible algorithmic template that allows users to effortlessly implement their unique designs within the platform. Moreover, it provides a broad spectrum of over 300 problem instances, collected from synthetic to realistic scenarios, and an extensive library of 19 baseline methods, including both traditional black-box optimizers and recent MetaBBO-RL methods. Besides, MetaBox introduces three standardized performance metrics, enabling a more thorough assessment of the methods. In a bid to illustrate the utility of MetaBox for facilitating rigorous evaluation and in-depth analysis, we carry out a wideranging benchmarking study on existing MetaBBO-RL methods. Our MetaBox is open-source and accessible at: https://github.com/GMC-DRL/MetaBox.
Black Box Optimization (BBO) is a class of optimization problems featured by its objective function that is either unknown or too intricate to be mathematically formulated. It has a broad range of applications such as hyper-parameter tuning [1], neural architecture searching [2], and protein-docking [3]. Due to the black-box nature, the optimizer has no access to the mathematical expression, gradients, or any other structural information related to the problem. Instead, the interaction with the black-box problem is primarily realized through querying inputs (i.e., a solution) and observing outputs (i.e., its objective value).
Traditional solvers for BBO problems include population-based optimizers such as genetic algorithms [4], evolutionary strategies [5-7], particle swarm optimization [8, 9], and differential evolution [10-13]. The Bayesian Optimization (BO) [14, 15] is also commonly used. However, with limited knowledge of the problem, these optimizers lean on carefully hand-crafted designs to strike a balance between exploration and exploitation when seeking the optimal solution.
To eliminate the burdensome task of manual fine-tuning, recent research has proposed the concept of Meta-Black-Box Optimization (MetaBBO), which aims to refine the black-box optimizers by
* Yue-Jiao Gong is the corresponding author.
Figure 1: Blueprint of our MetaBox platform. MetaBox offers template scripts for quick-start and, once completed by the user, it carries out an automatic Train-Test-Log process on the testsuites for comparison. Results such as performance scores, optimization curves, and comparative tables are automatically generated and made available for review.
identifying optimal configurations or parameters that boost the overall performance across various problem instances within a given problem domain. This leads to a bi-level optimization framework, where the meta-level enhances the performance of low-level black-box optimizers. The meta-level approaches include supervised learning (MetaBBO-SL) [16-18], reinforcement learning (MetaBBO-RL) [19-25], and self-referential search (MetaBBO-SR) [26, 27]. Among them, MetaBBO-RL models the optimizer fine-tuning as a Markov Decision Process (MDP) and learns an agent to automatically make decisions about the algorithmic configurations. By automating the tuning process, MetaBBO-RL not only significantly reduces the time and effort needed for customizing algorithms to specific unseen problems, but also potentially enhances the overall optimization performance.
Despite the success, MetaBBO-RL calls for a unified benchmark platform. Upon examining the recently proposed MetaBBO-RL approaches [19-25], we found that while several approaches claim state-of-the-art performance, they lack a comprehensive benchmark and comparison using a standardized, unified testbed. As a result, identifying which approach truly excels under specific conditions poses a significant challenge, thereby hindering further progress and advancement in this field.
To bridge the gap, we introduce MetaBox, the first benchmark platform for MetaBBO-RL. The blueprint of MetaBox is illustrated in Figure 1, aiming to offer researchers a convenient means to develop and evaluate the MetaBBO-RL approaches. Specifically, we have made the following efforts:
1. To simplify the development of MetaBBO-RL and ensure an automated workflow: 1) We introduce a MetaBBO-RL Template, depicted at the top left of Figure 1. It comprises two main components: the meta-level RL agent and the low-level optimizer, where we provided several implemented works (e.g., [19, 21, 24, 25]) following a unified interface protocol. 2) We automate the Train-Test-Log procedure in MetaBox, shown in the middle of Figure 1. The users simply need to complete their MetaBBO-RL template, specify the target problem set, and select several baselines for comparison. Initiating the entire automated process is as straightforward as executing the run experiment() command. This provides users with considerable flexibility in implementing different types of internal logic for their MetaBBO-RL algorithms while benefiting from the automated workflow.
2. To facilitate broad and standardized comparison studies: 1) We integrate a large-scale MetaBox testsuite consisting of over 300 benchmark problems with diverse landscape characteristics (such as single-/multi- modal, non-ill/ill- conditioned, strong/weak global structured, and noiseless/noisy). Showing in the left centre of Figure 1, the MetaBox testsuite inherits problem definitions from the well-known COCO [28] platform and the Protein-Docking benchmark (version 4.0) [29], with several modifications to adapt to the MetaBBO paradigms. 2) We develop a Baseline Library, located at the bottom left in
Figure 1. The library currently encompasses a wide range of classic optimizers [6, 8-14, 30, 31] and up-to-date MetaBBO-RL approaches [19-25]. We additionally integrate a MetaBBO-SL approach [16] to provide an extended comparison. Notably, all of the baselines are implemented by our MetaBBO-RL Template. This ensures consistency and allows for a fair and standardized comparison among the different approaches.
3. To comprehensively evaluate the effectiveness of MetaBBO-RL approaches: 1) We propose three Standardized Metrics to evaluate both optimization performance and learning effectiveness of a MetaBBO-RL approach, including a novel Aggregated Evaluation Indicator (AEI) that offers a holistic view of the optimization performance, a Meta Generalization Decay (MGD) metric that measures the generalization of a learned approach across different problems, and a Meta Transfer Efficiency (MTE) metric that quantifies the transfer learning ability. 2) We conduct a tutorial large-scale comparison study using Baseline Library, evaluate them on MetaBox testsuite by the proposed Standardized Metrics. Several key findings reveal that the pursuit of state-of-the-art performance continues to present challenges for current MetaBBO-RL approaches. Nonetheless, our MetaBox platform provides valuable insights and opportunities for researchers to refine and improve their algorithms.
To summarize, MetaBox provides the first benchmark platform for the MetaBBO-RL community (novel). It is fully open-sourced, offering template scripts that facilitate convenient development, training, and evaluation of MetaBBO-RL algorithms (automatic). Furthermore, MetaBox provides diverse benchmark problems and an extensive collection of integrated baseline algorithms, which will continue to expand through regular maintenance and updates (extendable).
As shown in 2, MetaBBO methods operate within a bi-level optimization framework designed to automate the fine-tuning process for a given BBO optimizer. Distinguishing themselves from conventional BBO techniques, MetaBBO methods introduce a novel meta-level as an automatic decision process. The purpose is to alleviate the need for labor-intensive manual fine-tuning of low-level BBO optimizers. Typically, they require the ability to generalize behaviors to address previously unseen problems through extensive training in a given problem distribution. Concretely: 1) At the meta level, the meta optimizer (e.g., an RL agent) dynamically configures the low-level optimizer based on the current optimization status at that particular time step. Then, the meta optimizer evaluates the performance of the low-level optimizer over the subsequent optimization steps, referred to as meta performance. The meta optimizer leverages this observed meta performance to refine its decision-making process, training itself through the maximization of accumulated meta performance, thereby advancing its meta objective. 2) At the lower level, the BBO optimizer receives a designated algorithmic configuration from the meta optimizer. With this configuration in hand, the low-level optimizer embarks on the task of optimizing the target objective. It observes the changes in the objective values across consecutive optimization steps and transmits this information back to the meta optimizer, thereby contributing to the meta performance sign
Figure 2: Illustration of the bi-level optimization procedure of MetaBBO.
MetaBBO-RL. It leverages reinforcement learning at the meta-level to configure the low-level black-box optimizer for boosting meta-performance [26] over a problem distribution. It involves three components: an RL agent (policy)  $\Pi_{\theta}$ , a backbone black-box optimizer  $\Lambda$ , and a dataset  $\mathbb{D}$  following certain problem distribution. The environment  $Env$  in MetaBBO-RL is formed by coupling the optimizer with a problem from the dataset, i.e.,  $Env := \{\Lambda, f = \text{sample}(\mathbb{D})\}$ , where  $f$  represents the sampled problem. Unlike traditional RL tasks [32, 33], the environments in MetaBBO-RL not only include the problem of optimizing but also include the low-level optimizer itself. At each step  $t$ , the agent queries the optimization status  $s_t$ , which includes the current solutions produced by  $\Lambda$  and their evaluated values by  $f$ . The policy  $\Pi_{\theta}$  takes  $s_t$  as input and suggests an action  $a_t = \Pi_{\theta}(s_t)$  to configure the optimizer  $\Lambda$ . Note that the action space of different MetaBBO algorithms can vary largely, as it could involve determining solution update strategy [19, 20, 23], generating hyperparameter [21, 24, 25], or both of the above two [22]. Subsequently,  $Env$  executes  $a_t$  to obtain the
Table 1: Comparison to BBO benchmarks. We report #Problem: the number of problems (#synthetic + #realistic); #Baseline: the number of baselines; Template: Template coding support; Automation: automated train/test workflow support; Customization: configurable settings; Visualization: visualization tools support; and RLSupport: Gym-style [33] RL benchmark.
#Problem	#Baseline	Template	Automation	Customization	Visualization	RLSupport
COCO [28]	54+0	2	✓	✓	×	✓	×
CEC [34, 36, 37]	28+0	0	×	×	×	×	×
IOHprofiler [38, 39]	24+0	0	✓	×	✓	✓	×
Bayesmark [40, 41]	0+228	10	✓	✓	×	×	×
Zigzag [42, 43]	4+0	0	×	×	✓	×	×
MetaBox	54+280	19	✓	✓	✓	✓	✓
next optimization status  $s_{t+1}$  and the reward  $r_t$  that measures the meta-performance improvement of  $\Lambda$  on problem  $f$ . The meta-objective of MetaBBO-RL is to learn a policy  $\Pi_\theta$  that maximizes the expectation of the accumulated meta-performance improvement  $r_t$  over the problem distribution  $\mathbb{D}$ ,  $\mathbb{E}_{f \sim \mathbb{D}, \Pi_\theta}[\sum_{t=0}^T r_t]$ , where  $T$  denotes the predefined evaluation budget, typically referred to the maxFEs parameter as in the existing BBO benchmarks [28, 34].
Other MetaBBO methods. Apart from MetaBBO-RL, two other paradigms include MetaBBO-SL [16-18] and MetaBBO-SR [26, 27], which all belong to the MetaBBO community but leverage different approaches at the meta-level. Existing MetaBBO-SL methods consider a recurrent neural network (RNN) that helps determine the next solutions at each step [16-18]. However, they often encounter a dilemma in setting the horizon of RNN training: a longer length improves the difficulty of training, whereas a shorter length requires breaking the entire optimization process into small pieces, often resulting in unsatisfactory results. MetaBBO-SR utilizes black-box optimizers (such as an evolution strategy [35]) at both the meta and low levels to enhance the overall optimization performance. Since the black-box optimizers themselves can be computationally expensive, the MetaBBO-SR approaches may suffer from limited efficiency due to their inherently nested structure.
Related BBO benchmarks. Existing benchmarks for MetaBBO are currently absent. However, there are several related BBO benchmark platforms, including COCO [28] and IOHprofiler [38, 39] for continuous optimization, ACLib [44] for algorithm configuration, Olympus [45] for planning tasks, and Bayesmark [40] for hyper-parameter tuning. These platforms provide testsuites, logging tools and some baselines for users to compare or refer to. In addition to these platforms, some BBO competitions provide a series of problems to evaluate specific aspects of algorithms, such as the GECCO BBOB workshop series (based on COCO [28]), the NeurIPS BBO challenge [41] (based on Bayesmark [40]), the IEEE CEC competition series [36], and the Zigzag BBO [42, 43] that consists of highly challenging problems. In contrast, our proposed MetaBox introduces a novel benchmark platform specifically tailored for MetaBBO-RL. It provides an algorithm development template, an automated execution philosophy, a wide range of testsuites, an extensive collection of baselines, specific MetaBBO performance metrics, and powerful visualization tools. In Table 1, we compare MetaBox and other related benchmarks to showcase the novelty of this work.
3 MetaBox: Design and Resources
3.1 Template coding and workflow automation
The core structure of MetaBox is presented on the left of Figure 3, shown in the form of a UML class diagram [46]. Drawing from the recent MetaBBO-RL, we abstract the MetaBBO-RL Template into two classes: the reinforcement Agent and the backbone Optimizer (both marked in pale orange). To develop or integrate a new MetaBBO-RL approach, users are required to specify their settings in the attributes Agent.config and Optimizer.config, and implement the following interfaces: 1) Agent.train EpisodeEnv for the training procedure, 2) Agent-rollout EpisodeEnv for the rollout procedure, and 3) Optimizer.update(action, problem) for the optimization procedure. Once these interfaces are set, the implemented templates seamlessly integrate with other components, requiring no extra adjustments. Except for the Agent and Optimizer classes, the other classes are hidden from users, simplifying coding tasks and ensuring code consistency.
Figure 3: The core structure and workflow of MetaBox. Left: UML Class diagram of MetaBox. Users can inherit from MetaBBO-RL templates highlighted in orange to enable polymorphism. Right: The automated Train-Test-Log workflow. The Agent directs the low-level (inner loop) optimization and trains itself on meta-level (outer loop), followed by testing and post-processing.
Given an implemented approach based on MetaBBO-RL Template, the right side of Figure 3 illustrates the entirely automated Train-Test-Log workflow in MetaBox. The process initiates when a user launches the function runexperiment(). Then, function Trainer.train() executes an outer-loop for meta-level iterations, where a runtime RL environment (env) is instantiated by pairing a problem from the trainset with an optimizer instance. The Agent.train Episode(env) function is then called to start an inner-loop for the BBO iterations, during which the optimizer in env follows the action informed by Agent and applies its update(action, problem) function to optimize the target BBO problem until the predefined function evaluation budget is exhausted. The meta-iteration ends when the Agent has trained its policy model for  $M$  steps. Next, the Tester.test() function is called to evaluate the learned policy on the testset through  $N$  independent runs, recording the statistic results. Finally, the Logger.log() function restores all saved training and testing results. These basic results are further post-processed into comprehensive metrics for in-depth analysis. The key to this automation resides in the universal internal-call bridging the MetaBBO-RL Template and the Train-Test-Log workflow, thereby obviating the need for users to grapple with complex coding tasks.
3.2 Testsuites
The testsuites integrated into MetaBox are briefly described as follows (more details in Appendix A):
Synthetic. It consists of 24 functions in five groups: separable, moderate, ill-conditioned, multimodal with global structure, and multi-modal with weakly structured. They are collected from the coco:bbb function set in the renowned COCO platform [28].
Noisy-Synthetic. It consists of 30 noisy functions from the coco:bbob-noisy function set [28], by extending the Synthetic set with different noise models and levels. The noises come from three different models: Gaussian noise, Uniform noise and Cauchy noise [47].
Protein-Docking. It is extracted from the Protein-Docking benchmark [29], with 280 instances of different protein-protein complexes. These problems are characterized by rugged objective landscapes and are computationally expensive to evaluate.
Besides, we make the following efforts to adapt the above testsuites, while enhancing usability and flexibility: 1) We treat each testsuite as a dataset and split it into training and testing sets according to a particular proportion, referred to as difficulty. This proportion determines the level of difficulty in generalizing or transferring the learned knowledge to unseen instances during testing. We provide two modes for controlling this aspect: easy mode, allocating  $75\%$  of the selected instances for training, and difficult mode, designating  $25\%$  of the selected instances for training. 2) Instead of providing predefined problem dimension candidates, such as  $\{2,3,5,10,20,40\}$  in COCO [28] and  $\{10,30\}$  in CEC2021 [34], we introduce an additional control parameter problem-dim to support customized problem dimension. 3) We also recognize that there is a potential user group (e.g., those in [16-18])
who require access to gradients of the objective during the training of their approaches. To cater to this need, we provide a PyTorch function interface with tensor calculation support, enabling users to incorporate back-propagation in their methods.
3.3 Baseline library
MetaBox leverages a total of 19 baselines, categorized into three types: 1) Classic Optimizer: Random Search (RS), Bayesian Optimization (BO) [14], Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [6], Differential Evolution (DE) [10] and its self-adaptive variants JDE21 [12], MadDE [11], NL-SHADE-LBC [13], Particle Swarm Optimization (PSO) [8] and its self-adaptive variants GLPSO [9], sDMSPSO [30], SAHLPSO [31]. 2) MetaBBO-RL: the Q-learning [48-50] styled approaches DEDDQN [19], DEDQN [20], RLHPSDE [22], QLPSO [23], and the Policy Gradient [51, 52] styled methodos LDE [21], RLPSO [24], RLEPSO [25]. 3) MetaBBO-SL: RNN-OI [16].
The majority of the classic baselines are reproduced by referring to the originally released codes (when available) or the respective original papers. We meticulously filled the Optimizer template (see Section 3.1) with their specific internal logic. However, there are a few exceptions: CMA-ES, DE and PSO are implemented by calling APIs of DEAP [53] (an evolutionary computation framework with LGPL-3.0 license), and BO is implemented by the Scikit-Optimizer [54] (a BO solver set with BSD-3-Clause license). This showcases the compatibility of MetaBox with existing open-sourced optimization codebases. For MetaBBO-RL, we first encapsulate the internal logic of the RL agent into the Agent template (see Section 3.1), and then encapsulate the internal logic of the backbone optimizer into the Optimizer template. This showcases the ease of integrating various MetaBBO-RL methods into MetaBox. In addition, we provide an example of a MetaBBO-SL approach, RNN-OI, to showcase that MetaBox is compatible with other MetaBBO methods.
3.4 Performance metrics
In MetaBox, we implement three standardized metrics to evaluate the performance of MetaBBO-RL approaches from aspects of both the BBO performance and the training efficacy.
AEI. The MetaBBO approaches can be evaluated using traditional BBO performance metrics, including the best objective value, the budget to achieve a predefined accuracy (convergence rate), and the runtime complexity [28, 36, 39]. In addition to analyzing them separately, we propose the Aggregated Evaluation Indicator (AEI), a unified scoring system to provide users with a comprehensive assessment. Aggregating these metrics together is challenging due to the significant variation in values across different metrics and problems. To address this, the AEI normalizes and combines the three metrics in the following way. We test a MetaBBO-RL approach on  $K$  problem instances for  $N$  repeated runs and then record the basic metrics (usually pre-processed by a min-max conversion for consistency towards AEI maximization, refer to Appendix B for details): best objective value  $v_{obj}^{k,n}$ , consumed function evaluation times  $v_{fes}^{k,n}$ , and runtime complexity  $v_{com}^{k,n}$ . To make the values more distinguishable and manageable, they have been first subjected to a logarithmic transformation:
$$
v _ {*} ^ {k, n} = \log \left(v _ {*} ^ {k, n}\right). \tag {1}
$$
Then, Z-score normalization is applied:
$$
Z _ {*} ^ {k} = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {v _ {*} ^ {k , n} - \mu_ {*}}{\sigma_ {*}}, \tag {2}
$$
where  $\mu_{*}$  and  $\sigma_{*}$  are calculated by using RS as a baseline. Finally, the AEI is calculated by:
$$
A E I = \frac {1}{K} \sum_ {k = 1} ^ {K} e ^ {Z _ {o b j} ^ {k} + Z _ {c o m} ^ {k} + Z _ {r e s} ^ {k}}, \tag {3}
$$
where Z-scores are first aggregated, then subjected to an inverse logarithmic transformation, and subsequently averaged across the test problem instances. A higher AEI indicates better performance of the corresponding MetaBBO-RL approach.
2Important to note that the gradients of the objective during testing are not allowed in the context of BBO.
MGD. We then introduce the Meta Generalization Decay (MGD) metric, so as to assess the generalization performance of MetaBBO-RL for unseen tasks. Given a model that has been trained on a problem set  $B$  and its AEI on the corresponding testset as  $AEI_B$ , we train another model on a problem set  $A$  and record its AEI on the testset  $B$  as  $AEI_A$ . The  $MGD(A,B)$  is computed by:
$$
M G D (A, B) = 100 \times \left(1 - \frac {A E I _ {A}}{A E I _ {B}}\right) \%, \tag{4}
$$
where a smaller  $MGD(A, B)$  indicates that the approach generalizes well from  $A$  to  $B$ . Note that MGD has neither symmetry nor transitivity properties.