# A Deep Learning-Driven Black-Box Benchmark Generation Method via Exploratory Landscape Analysis
## 5 Experiment Setup
### Algorithm Performance Perspective  
The purpose of this experiment is to assess whether the optimization algorithms performed similarly on the original BBOB functions and their NNBBOB counterparts generated by our method. The aim is to compare both the ranking order of the algorithm performance and their convergence behavior, thereby validating the landscape fidelity from an algorithmic perspective. Landscape property analysis alone provides an inadequate basis for comparing imitated functions with their originals. Another way to analyze similarities is by observing the behavior of the algorithms across both function sets and checking if they perform similarly on the original and imitation counterparts. Therefore, a comprehensive benchmark test is conducted on BBOB using the Nevergrad platform framework [4]. The following comparison algorithms are selected: particle swarm optimization (PSO) [36] is a nature-inspired optimization algorithm that mimics the collective dynamics observed in swarms. The estimation of multivariate normal algorithm (EMNA) [37] is a type of Estimation of Distribution Algorithm (EDA) [38] used in optimization. The Nelder-Mead algorithm (ND) [39] is a derivative-free optimization approach used to minimize functions in continuous spaces. The diagonal covariance matrix adaptation evolution strategy (dCMAES) [40] is derived from the CMA-ES algorithm, which applies a diagonal covariance matrix rather than its dense counterpart. Random search (RS) [41] is a simple yet effective optimization algorithm used to search for the best solution in a given space randomly rather than following a specific pattern. Constrained optimization by linear approximation (COBYLA) [42] is a gradient-free optimization algorithm designed for nonlinear constrained problems. Differential evolution (DE) [43] is widely used to solve continuous and multimodal optimization functions. These algorithms enable a clear assessment of the accuracy with which the imitated functions replicate the behavior of the original BBOB functions. Each of these algorithms is under a budget constraint of \(10,000\cdot D\) function evaluations, where \(D\) is the dimension of the problem. Each algorithm is applied 51 times to every problem in \(2D\) and \(3D\). The performance rankings are determined by evaluating each optimizer with budget \(10,000\times D\) for each dimension separately between BBOB and NNBBOB (neural networks for black box optimization benchmark is referred to as NNBBOB) scenarios are tested. Tables 1 and 2 depict critical distance (CD) reports for the BBOB and NNBBOB problem in \(2D\) scenarios. The reports were derived from rankings based on the performance outcomes of each algorithm, with the results calculated as the median of 50 runs for each problem. To identify ties in the algorithm ranking, a non-parametric Friedman test combined with the Nemenyi post-hoc test is performed. Several statistical metrics are used to compare the algorithm's performance across the benchmark sets. The Mean Rank (MR) denotes the average rank position of each algorithm across the benchmark sets. The median deviation (MED) denotes the median loss value observed over the trials. The median absolute deviation (MAD) denotes the variation in the observed results. The confidence interval (CI) denotes the interval of expected performance difference. \(r\) denotes effect size, and Magnitude denotes the practical significance of effect size. The analysis is performed using the Python autorank (0.3.2 version) package [44]. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline
**Optimizer** & **MR** & **MED** & **MAD** & **CI** & \(\gamma\) & **Magnitude** \\ \hline DE & 1.938 & \(-1.122\) & 87.332 & \([-462.090,\,149.150]\) & 0.000 & negligible \\ COBYLA & 3.583 & \(-1.097\) & 87.797 & \([-462.090,\,149.151]\) & \(-0.000\) & negligible \\ PSO & 3.729 & 18.919 & 79.929 & \([-459.971,\,465.486]\) & \(-0.161\) & negligible \\ \hline \hline \end{tabular}
\end{table}