SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wall-time table. For each baseline in the Baseline Library, it presents the average running time  $\frac{1}{K\times N}\sum_{k = 1}^{K}\sum_{n = 1}^{N}T_2^k$  and the average algorithm complexity  $Z$ -score  $\frac{1}{K}\sum_{k = 1}^{K}Z_{com}^{k}$ . These values are averaged over all the problems in the test suites and 51 independent runs.
Cost curve. For each baseline in the Baseline Library, the figure draws the averaged optimization curves (cost curves) over all the problems in the testsuites and 51 independent runs.
In this appendix, as preview examples, we present the experimental results of baselines on the Noisy-Synthetic testsuites, which includes AEI scores with error bars on both the Noisy-Synthetic-easy and the Noisy-Synthetic-difficult testsuites in Figure 7, the overall performance table on Noisy-Synthetic-easy testsuites in Table 4, the wall-time table on Noisy-Synthetic-easy testsuites in Table 5, and the cost curves of MetaBBO-RL baselines on Noisy-Synthetic-easy testsuites in Figure 8. Note that they are partially displayed due to space limitations, without loss of generality.
D.2 Investigation on RLEPSO
Hyper-tuning. Since RLEPSO [25] also shows competitive performance in all of the three test suites, we execute the same  $2 \times 2$  grid-search experiment for it and measure its MGD and MTE. Similarly, we
Table 4: Per-instance optimization results of baselines on Noisy-Synthetic-easy testsuites, showing the first 3 problems in the test set for illustration purpose.
Sphere-moderate-gauss	Rosenbrock-moderate-uniform	Step-Ellipsoidal-cauchy
Obj	Gap	FEs	Obj	Gap	FEs	Obj	Gap	FEs
Random	1.101E+1(2.739E+0)	1.000	2.000E+4(0.000E+0)	2.267E+3(1.023E+3)	1.000	2.000E+4(0.000E+0)	6.363E+2(2.709E+2)	1.000	2.000E+4(0.000E+0)
CMA-ES	7.689E-9(1.844E-9)	0.000	4.958E+3(1.931E+2)	1.147E+0(2.696E+0)	0.000	1.624E+4(2.174E+3)	5.743E+0(4.048E+1)	0.000	6.119E+3(3.629E+3)
PSO	1.906E+0(8.624E-1)	0.173	2.000E+4(0.000E+0)	2.679E+2(5.578E+2)	0.118	2.000E+4(0.000E+0)	2.950E+2(2.453E+2)	0.459	2.000E+4(0.000E+0)
SAHLPSO (-g)	4.477E+0(2.875E+0)	0.407	2.000E+4(0.000E+0)	9.651E+2(9.798E+2)	0.426	2.000E+4(0.000E+0)	4.189E+2(2.584E+2)	0.655	2.000E+4(0.000E+0)
sDMSPSO (v1)	1.661E+0(7.380E-1)	0.151	2.010E+4(0.000E+0)	1.080E+2(4.728E+1)	0.047	2.010E+4(0.000E+0)	1.395E+2(1.922E+2)	0.212	2.010E+4(0.000E+0)
Classic	1.292E-6(7.497E-7)	0.000	2.000E+4(0.000E+0)	6.867E+0(8.194E-1)	0.003	2.000E+4(0.000E+0)	3.322E+2(3.195E+2)	0.518	2.000E+4(0.000E+0)
DE (rand/1)	8.010E-9(1.636E-9)	0.000	4.601E+3(1.794E+2)	5.719E+0(1.373E+0)	0.002	2.000E+4(0.000E+0)	1.373E+2(2.041E+2)	0.209	2.000E+4(0.000E+0)
JDE21	5.302E-9(2.477E-9)	-0.000	6.283E+3(1.562E+3)	6.212E+0(5.923E+0)	0.002	2.001E+4(0.000E+0)	2.227E+2(2.440E+2)	0.344	2.001E+4(0.000E+0)
NL-SHADE-LBC	7.826E-9(1.494E-9)	0.000	1.444E+4(1.161E+2)	4.541E+0(8.124E-1)	0.001	2.000E+4(0.000E+0)	1.849E+2(2.153E+2)	0.284	2.000E+4(0.000E+0)
MadDE	7.988E-9(1.602E-9)	0.000	1.945E+4(1.438E+2)	5.361E+0(1.083E+0)	0.002	2.000E+4(0.000E+0)	1.613E+2(2.382E+2)	0.247	2.000E+4(0.000E+0)
Bayesian (BO)	9.144E-2(1.197E-1)	0.008	1.000E+2(0.000E+0)	4.506E+3(1.303E+4)	1.988	1.000E+2(0.000E+0)	1.024E+3(2.678E+1)	1.615	1.000E+2(0.000E+0)
Learnable	2.568E+0(2.915E+0)	0.233	2.000E+4(0.000E+0)	1.467E+2(2.788E+2)	0.064	2.000E+4(0.000E+0)	1.411E+2(2.387E+2)	0.215	2.000E+4(0.000E+0)
RLPSO	2.419E+0(1.277E+0)	0.220	2.000E+4(0.000E+0)	1.607E+2(1.164E+2)	0.070	2.000E+4(0.000E+0)	1.440E+2(1.843E+2)	0.219	2.000E+4(0.000E+0)
RLEPSO	7.521E-6(1.163E-5)	0.000	2.000E+4(1.707E+0)	7.244E+0(2.131E+0)	0.003	2.000E+4(1.473E+0)	1.580E+2(1.824E+2)	0.241	2.000E+4(1.695E+0)
DEDQN	3.369E+1(9.025E+0)	3.060	2.000E+4(0.000E+0)	2.152E+4(9.820E+3)	9.498	2.000E+4(0.000E+0)	1.097E+3(4.189E+1)	1.730	2.000E+4(0.000E+0)
DEDDDQN	7.907E-9(1.523E-9)	0.000	1.692E+4(5.373E+2)	2.913E+0(2.859E+0)	0.001	1.977E+4(7.700E+2)	1.022E+2(1.726E+2)	0.153	2.000E+4(0.000E+0)
LDE	8.127E-9(1.628E-9)	0.000	9.685E+3(3.920E+2)	2.681E+0(2.639E+0)	0.001	2.000E+4(0.000E+0)	1.292E+2(2.007E+2)	0.196	2.000E+4(0.000E+0)
RLHPSDE	1.118E-1(6.738E-2)	0.010	2.025E+4(0.000E+0)	4.248E+1(2.126E+1)	0.018	2.025E+4(0.000E+0)	6.269E+2(3.087E+2)	0.985	2.025E+4(0.000E+0)
RNN-OI	6.940E+1(6.824E-1)	6.304	1.000E+2(0.000E+0)	3.177E+4(6.347E+2)	14.021	1.000E+2(0.000E+0)	1.099E+3(1.198E+1)	1.734	1.000E+2(0.000E+0)
Table 5: Running time and algorithm complexity of baselines on Noisy-Synthetic-easy testsuits.
Baselines	CMA-ES	PSO	SAHLPSO	sDMSPSO	GLPSO	DE
T2 (ms)	3.337E+02	2.021E+03	4.506E+03	6.484E+01	5.508E+01	8.838E+02
Z-score	9.540E-01	8.093E-01	7.254E-01	1.047E+00	1.066E+00	8.420E-01
Baselines	JDE21	NL-SHADE-LBC	MadDE	QLPSO	RLPSO	RLEPSO
T2 (ms)	1.639E+02	2.036E+02	4.015E+02	4.442E+03	1.337E+04	3.540E+02
Z-score	9.588E-01	9.370E-01	8.827E-01	7.265E-01	6.530E-01	8.918E-01
Baselines	DEDQN	DEDDQN	LDE	RLHPSDE	RNN-OI	BO
T2 (ms)	2.309E+02	2.787E+04	3.303E+02	5.867E+02	4.846E+03	1.407E+07
Z-score	9.262E-01	6.101E-01	8.972E-01	8.517E-01	7.216E-01	3.542E-01
investigate the combination of different RL methods (REINFORCE or PPO) and different population sizes (100 or 50). The original RLEPSO is implemented in PPO with 100 population size. The grid search results are shown in Figure 9. The RLEPSO with the PPO agent and a population size of 50 has the highest average return during the training, however, this configuration shows the worst optimization results (considering the normalized cost) during the test. On the contrary, RLEPSO with the REINFORCE agent and a population size of 50 has the lowest average return during the training and the lowest normalized cost during the test. This may indicate that the design of the RLEPSO reward function may not be appropriate. Meanwhile, we note that for RLEPSO, changing its RL agent from PPO to REINFORCE may slightly improve the overall performance.
Generalization performance. We also present the MGD of RLEPSO on the testsuites with an easy difficulty level, displayed on the left side of Figure 10. It appears that RLEPSO exhibits poor generalization ability. For instance, when the RLEPSO model is pre-trained on the Protein-Docking-easy testsuites, there is a significant  $40.312\%$  drop in the AEI score. These results indicate that while RLEPSO achieves competitive AEI performance, its generalization ability is much worse compared
Figure 9: Hyper-tuning for the MetaBBO-RL approach RLEPSO [25] using a  $2 \times 2$  grid search. Left: The average return during the training over 10 trials. Middle: The normalized cost over 51 independent runs during the test. Right: The corresponding AEI scores during the test.
to LDE. The subpar MGD performance of RLEPSO may be attributed to its non-generalizable state design, which directly utilizes the consumed FEs as the state representation.
Transfer performance. We performed fine-tuning on the RLEPSO model that was pre-trained on the Noisy-Synthetic-easy test suites for solving the Synthetic-easy test suites. The progress of the average return over 10 trials is shown on the right side of Figure 10. However, the pre-trained RLEPSO exhibits negative transferability, with an MTE value of 0. This transfer failure suggests that some of the current MetaBBO-RL methods face overfitting issues, which in the case of RLEPSO may be attributed to its oversimplified state representation or other design elements that lack generalizability.
Figure 10: Meta performance (MGD and MTE) of RLEPSO [25] across different tasks, tested under the easy mode. Left: Logits on  $i$ -th row and  $j$ -th column is the  $MGD(i,j)$ , the smaller the better. Right: the average return over 10 trials is compared between the RLEPSO models pre-trained and trained from scratch, with a  $MTE = 0$ . This is a transfer failure.
E Used Assets
MetaBox is an open-sourced tool, and it can be accessed at: https://github.com/GMC-DRL/MetaBox. It is currently licensed under the BSD 3-Clause License. In Table 6, we provide a list of the resources or assets utilized in MetaBox, along with their respective licenses. It is important to note that we adhere strictly to these licenses during the development of MetaBox.
Table 6: Used assets and their licenses
Type	Asset	Codebase	License
Baseline	CMA-ES [5]	DEAP [53]	LGPL-3.0 License
DE [10]	DEAP [53]	LGPL-3.0 License
PSO [8]	DEAP [53]	LGPL-3.0 License
BO [14]	Scikit-Optimizer [54]	BSD-3-Clause License
Testsuites	Synthetic	COCO [28]	BSD 3-clause License
Noisy-Synthetic	COCO [28]	BSD 3-clause License
Protein-Docking	Protein-Docking 4.0 [29]	-
Algorithm 1: MetaBoxTrainer
Input: User-designed learnable agent  $A$  ,User-designed optimizer  $O$  ,User-specified problem set  $P_{train}$  with configuration  $C_{train}$    
Output: Trained Agent  $A$  , training records.   
Initialize problem set  $P_{train}$  with configuration  $C_{train}$  as the training dataset;   
while max learning steps Not reached do for each problem instance ins  $\in P_{train}$  do Construct optimization environment Env  $=$  Env_Construction(O,ins); A.train Episode(Env); Record training data; Plot training figures; end   
end Summarize and visualize the training records in Logger and return the trained agent;
Algorithm 2: TrainingEpisode
Input: User-designed learnable agent  $A$  , Constructed environment Env   
Output: Training records   
state  $=$  Env.reset();   
while termination condition Not achieved do action  $= A$  .get_action(state); next_state,reward,info  $=$  Env.step(action); Store transition  $<$  state,action,reward,next_state>; Update agent  $A$  Record training data and plot figures; state  $=$  next_state;   
end   
Summarize training records and return;
F Pseudo-code of MetaBox
The pseudo-code of Trainer is shown in Algorithm 1. Given the user-designed learnable agent  $A$  and optimizer  $O$ , our MetaBox framework firstly initializes the problem instance set (e.g., determines problem types, indexes file directories and loads data to construct problem instances according to the user-specified configuration  $C_{\text{train}}$ ). Then MetaBox iteratively performs training on each training instance until the max learning step is reached. For each instance, a Gym-style environment  $Env$  is constructed to merge  $O$  and the problem instance together, so as to provide a unified interface. Agent  $A$  then calls the train Episode() interface for interacting with  $Env$  and performing the actual training. All the generated logs during training are managed by the Logger as depicted in Figure 1.
We now turn to the details of train Episode(). Given the variance among numerous learning approaches, such as RL and SL, the internal workflow of train Episode() necessitates implementation aligned with specific designs. Within MetaBox, we offer a range of examples through the implementation of various baselines, serving as guides for users who wish to develop their own interfaces. Meanwhile, we note that our baseline library also covers the implementations of prevalent RL algorithms like PPO and REINFORCE. In Algorithm 2, we showcase a straightforward example of the workflow for implementing RL training algorithms within train Episode(). Starting from the Env initialization, in each step, the agent  $A$  provides Env the action according to the state, receives the next state, reward and other information, and uses them to update the policy. Within the env_STEP() interface, the action is translated into configurations which are then applied to the optimizer  $O$ . The optimizer then performs update rules to derive and evaluate new solutions. Following this, rewards and subsequent states are computed, with certain logging information being concurrently summarized.
As for the Tester shown in Algorithm 3, it first initializes the test problem set which is used to evaluate each algorithm in an algorithm set (including several baseline agents for comparison and the user's trained agent). For learning-based algorithms, respective agents and optimizers are then
Algorithm 3: MetaBoxTester
Input: User-specified algorithm set  $B$  including baselines and user's trained agent, User-specified problem set  $P_{\mathrm{test}}$  with configuration  $C_\mathrm{test}$    
Output: Testing results   
Initialize problem set  $P_{\mathrm{test}}$  with configuration  $C_\mathrm{test}$  as the test set;   
for each algorithm alg  $\in B$  do for each problem instance ins  $\in P_{\mathrm{test}}$  do if alg is a learning-based method then Initialize alg(agent and algOptimizer; end else Initialize algOptimizer as an optimizer; end Construct optimization environment Env  $\equiv$  Env_Construction(algOptimizer,ins); alg. rollout Episode(Env); Record testing data; Plot testing figures;   
end   
end Summarize testing results and call Logger for standardized metrics and visualization;
initialized, which may involve loading network models and initializing parameters. Conversely, classic optimizers are simply initialized as the optimizers. The rollout Episode() interface mirrors the train Episode() but removes the policy update procedures. After the testing, the recorded testing logs will be organized and presented within the Logger.