# An Analysis of the RBF Hyperparameter Impact on Surrogate-Assisted Evolutionary Optimization

Authors: Yoel Tenne
Journal: Scientific Programming
 (2022)

---

## Abstract

Computationally expensive optimization problems are often solved using surrogates and a common variant is the radial basis functions (RBF) model. It aggregates several basis functions which all depend on a hyperparameter affecting their individual outputs and consequentially the overall surrogate prediction. However, the optimal value of the hyperparameter is typically unknown and should therefore be calibrated. This raises the question how does the hyperparameter affect the overall optimization search effectiveness (and not just the stand-alone surrogate accuracy) and to what extent is such a calibration beneficial, which is an important consideration both for end-users and algorithm researchers alike. To rigorously address this issue this paper presents an analysis based on an extensive set of numerical experiments with an RBF surrogate-assisted evolutionary algorithm. It follows that the hyperparameter strongly affected performance and that the extent of its impact varied depending on the basis function, objective function modality, and problem dimension. Overall, calibration of the hyperparameter was typically highly beneficial to the search performance while dynamically optimizing the hyperparameter during the search yielded additional performance gains.


---

## Body

## 1. Introduction

Industrial optimization problems often involve functions whose evaluation is computationally expensive, for example, in engineering design optimization candidate products could be evaluated by a computer simulation (instead of real-world experiments) to improve the design process efficiency. This simulation-driven setup introduces several optimization challenges: (i) the simulation acts as a “black-box function”, namely, it assigns an output value to each input vector (candidate product) through involved numerical procedures or proprietary codes but there is no analytic expression for this functional relation thereby necessitating derivative-free optimization methods, (ii) each simulation run is typically computationally intensive so only a small number of vectors can be evaluated, and (iii) such “black-box functions” often have complicated features (such as multimodality) which exacerbate the search difficulty.A framework which has shown to be effective in such problems combines an evolutionary algorithm (EA) with a surrogate model (also termed in the literature as a “metamodel”) yielding the surrogate-assisted EA (SAEA). The evolutionary algorithm is a heuristic search algorithm which seeks an optimum by using operators inspired by the process of evolution and applies them to a population of candidate solutions, while the surrogate model approximates the true expensive black-box function and provides estimated function values at a much lower computational cost when compared to the true expensive function. A common surrogate variant is the radial basis functions (RBFs) model which uses a set of functions whose values are linearly aggregated. RBFs rely on a hyperparameter (HP) which affects their individual outputs and consequentially the surrogate prediction and the overall SAEA performance. The optimal HP value is typically unknown, and therefore it needs to be calibrated, which raises the following question:how significant is the impact of the HP on the search and does the numerical effort spent on such calibration yield significant performance gains?. The impact of the HP has been examined in the literature with respect to the stand-alone surrogate prediction accuracy but hardly so in the context of the overall surrogate-assisted search, which is a significantly more involved relation [1]. To address these open questions this study presents an extensive set of numerical experiments and analysis which examine how the HP affects the overall SAEA performance both when a fixed HP is selected a-priori and when it is dynamically calibrated during the search. Based on the large volume of results, the analysis clarifies in which instance does the HP strongly affect the SAEA performance. The remainder of this paper is organized as follows: Section 2 surveys the relevant literature, Section 3 explains the methods used, Section 4 presents the numerical experiments and their analysis, and lastly Section 5 concludes the paper.


## 2. Literature Survey

SAEAs have shown their effectiveness across a variety of problems ranging from drug formulation to aircraft design [2, 3]. The success of such algorithms stems from the combination of the EA, which is both derivative-free and capable of handling functions with complicated features, with the surrogate which provides approximate function values efficiently. In this setup the EA searches for an optimum of the surrogate while the latter is updated during the search. A basic SAEA is presented in Algorithm 1.Algorithm 1: A basic SAEA.
Input: initial sample size, max evaluations, EA parameters;Generate an initial set of vectors and evaluate with the true function;RepeatTrain a surrogate model by using the vectors evaluated so far;Search for an optimum of the surrogate by using an EA;Evaluate the predicted solution (and possibly additional vectors) with the true function;untilmax evaluations;Output: the best solution found;Various SAEA implementations have been explored in the literature, for example in [4], a multifidelity setup was proposed where several simulations of different accuracy were combined for design optimization of hydraulic turbines. In [5] the authors used an RBF-based SAEA for design optimization of a mechanical controller, while in [6] a multiobjective SAEA based on a Kriging surrogate was proposed. In [7] the authors used a SAEA to optimize the layout of a wind turbine farm with the goal of maximizing energy production while accounting for wind regimes and turbine wakes. More intricate frameworks include [8] which combined an EA with machine-learning techniques to handle vectors in which the simulation fails and [9, 10] in which ensembles of multiple surrogates were used concurrently to improve the prediction accuracy. Elaborate surrogate updating procedures were studied in [11, 12], while a dynamic in-search selection of the surrogate type was studied in [9, 13]. Recently in [14] the authors proposed a dual surrogate-assisted algorithm which uses a global surrogate and a group of local surrogates for approximation and a dual-population cooperative particle swarm algorithm to explore the surrogates to better handle highly multimodal functions. Other studies have proposed ensembles of surrogates [15] and more recently in [16] the authors proposed using an ensemble of local surrogates and selecting the subset that provides the most accurate approximation. Also related is [17] in which the authors proposed a surrogate-assisted particle swarm algorithm in which the sampled points are split into multiple subsets and a different surrogate is trained based on each sample. The surrogates are then managed to improve the global and local search capabilities of the algorithm. Also using multiple surrogates, in [18] the authors proposed a surrogate-assisted algorithm which used both a Kriging surrogate for localized modelling and a global quadratic model to approximate the overall function trend. The algorithm also uses two optimization algorithms (“grey wolf” and a multistart search) to identify good solutions from each surrogate.Various studies have also examined the impact of the HP on the stand-alone surrogate. In [19] a Kriging surrogate was used along with several HP calibration methods to achieve different balances between global and local prediction accuracies. In [20], the authors proposed estimating the optimal Kriging HP by using a multistart gradient search on the cross-validation error function, while in [21], calibrating the HP was found to affect the accuracy of various surrogates. In [22], the authors found that the size of the initial sample affected the calibration of the HP. However, the mentioned studies examined the impact of the HP only on the prediction accuracy of the stand-alone surrogate while this study analyzes the impact on the overall SAEA performance.


## 3. Methods

### 3.1. The RBF Surrogate

The RBF surrogate used in this study is a linear combination (superposition) of basis functions (BFs), namely,(1a)mx⟶=∑i=1nαiϕix⟶,(1b)ϕix⟶=ϕx⟶−x⟶i2,where mx⟶ is the surrogate prediction, while x⟶i, i=1,…,n, and ϕix⟶ are the sampled vectors and basis functions, respectively. The scalar weights αi are determined from the Lagrange interpolation conditions(2)mx⟶i=fx⟶i,i=1,…,n.Two widely used BFs are the Gaussian and multiquadric (MQ) [23] which are defined as(3)Gaussian:ϕix→=exp−σx→−x→i22,(4)MQ:ϕix→=1+σx→−x→i22,where σ is the HP which determines the BF shape and consequentially the overall surrogate prediction mx⟶. This relation is visualized in Figure 1.Figure 1
The impact of the hyperparameter on (a) univariate and (b) bivariate Gaussian and MQ basis functions.
(a)(b)Given its impact, various studies have examined how to calibrate the HP based on the stand-alone surrogate accuracy. In [24], the authors used an empirical fixed value derived from the mean distance between the sampled vectors but without accounting for the observed function values. In [25], the HP was calibrated by minimizing the root mean square error (RMSE) of the RBF prediction by using a gradient descent. A related leave-one-out cross-validation procedure was proposed in [26–29] with various modifications explored in [30–33]. In [34] the HP was calibrated based on a least-squares fit procedure, while in [35], MARS interpolating polynomials were used. More recently, [36] proposed using an EA for the HP calibration. In the numerical experiments performed in this study, both a fixed and a dynamically optimized HP were used as explained in Section 4.


### 3.2. The SAEA Algorithm

A representative SAEA was implemented for the numerical experiments. It begins by sampling an initial random population and training the RBF surrogate. The main loop then begins in which a real-coded EA seeks an optimum of the surrogate. Everyg generations the best q percent of the population members are evaluated with the true objective function and are added to the cache of sampled vectors. The surrogate is retrained based on the updated cache and the EA population is re-evaluated with the new surrogate. Following [37] the settings g=4 and q=20% were used. The loop repeats until either the true (expensive) function evaluations reach 50d (where d is the function dimension), or 1000 EA generations have occurred. Algorithm 2 provides the complete SAEA workflow.Algorithm 2: The implemented RBF-based SAEA.
Input: initial sample size, max evaluations, EA parameters, g and s parameters;generate and sample an initial random population;repeattrain an RBF surrogate based on the evaluated vectors;run one generation of the EA;for eachgthgenerationdoevaluate with the bestq% vectors with the true function;retrain the surrogate and re-evaluate the population;enduntilmax function evaluations or max EA generations;output: the best solution found;For the real-coded EA a representative variant with a population size of 20 was used which operates as follows: (i) vectors are ranked by fitness, (ii) parents are selected by stochastic universal selection (SUS) and recombined with a uniform crossover operator with a probability of 0.7 to produce offspring, and (iii) the latter are mutated with the breeder genetic algorithm operator with a probability of 0.1.


## 3.1. The RBF Surrogate

The RBF surrogate used in this study is a linear combination (superposition) of basis functions (BFs), namely,(1a)mx⟶=∑i=1nαiϕix⟶,(1b)ϕix⟶=ϕx⟶−x⟶i2,where mx⟶ is the surrogate prediction, while x⟶i, i=1,…,n, and ϕix⟶ are the sampled vectors and basis functions, respectively. The scalar weights αi are determined from the Lagrange interpolation conditions(2)mx⟶i=fx⟶i,i=1,…,n.Two widely used BFs are the Gaussian and multiquadric (MQ) [23] which are defined as(3)Gaussian:ϕix→=exp−σx→−x→i22,(4)MQ:ϕix→=1+σx→−x→i22,where σ is the HP which determines the BF shape and consequentially the overall surrogate prediction mx⟶. This relation is visualized in Figure 1.Figure 1
The impact of the hyperparameter on (a) univariate and (b) bivariate Gaussian and MQ basis functions.
(a)(b)Given its impact, various studies have examined how to calibrate the HP based on the stand-alone surrogate accuracy. In [24], the authors used an empirical fixed value derived from the mean distance between the sampled vectors but without accounting for the observed function values. In [25], the HP was calibrated by minimizing the root mean square error (RMSE) of the RBF prediction by using a gradient descent. A related leave-one-out cross-validation procedure was proposed in [26–29] with various modifications explored in [30–33]. In [34] the HP was calibrated based on a least-squares fit procedure, while in [35], MARS interpolating polynomials were used. More recently, [36] proposed using an EA for the HP calibration. In the numerical experiments performed in this study, both a fixed and a dynamically optimized HP were used as explained in Section 4.


## 3.2. The SAEA Algorithm

A representative SAEA was implemented for the numerical experiments. It begins by sampling an initial random population and training the RBF surrogate. The main loop then begins in which a real-coded EA seeks an optimum of the surrogate. Everyg generations the best q percent of the population members are evaluated with the true objective function and are added to the cache of sampled vectors. The surrogate is retrained based on the updated cache and the EA population is re-evaluated with the new surrogate. Following [37] the settings g=4 and q=20% were used. The loop repeats until either the true (expensive) function evaluations reach 50d (where d is the function dimension), or 1000 EA generations have occurred. Algorithm 2 provides the complete SAEA workflow.Algorithm 2: The implemented RBF-based SAEA.
Input: initial sample size, max evaluations, EA parameters, g and s parameters;generate and sample an initial random population;repeattrain an RBF surrogate based on the evaluated vectors;run one generation of the EA;for eachgthgenerationdoevaluate with the bestq% vectors with the true function;retrain the surrogate and re-evaluate the population;enduntilmax function evaluations or max EA generations;output: the best solution found;For the real-coded EA a representative variant with a population size of 20 was used which operates as follows: (i) vectors are ranked by fitness, (ii) parents are selected by stochastic universal selection (SUS) and recombined with a uniform crossover operator with a probability of 0.7 to produce offspring, and (iii) the latter are mutated with the breeder genetic algorithm operator with a probability of 0.1.


## 4. Experiments

### 4.1. Fixed HP

A first set of numerical experiments was performed to analyze if and to what extent different fixed HP values, namely, which are chosen a-priori, affect the overall SAEA performance. Following the Design-of-Experiments (DOE) framework [38] the numerical experiments were formulated by using a single variable, namely the HP, which was assigned different fixed values, and 3 factors (function type, function dimension, and BF type). Accordingly, in each combination of factors the numerical experiments were repeated with the different HP values so that the HP impact could be analyzed across a wide range of problem settings. The DOE setup is summarized in Table 1.Table 1
Design of the numerical experiments.
TypeNameValuesFactorFunctionAckley, Griewank, Powell, Rastrigin, RosenbrockDimension5, 10BFGaussian, MQVariableHyperparameter0.01, 0.1, 1, 10The established set of test functions of [39] was used as it includes functions with diverse features as shown in Figure 2.Figure 2
The test functions used.In each test case (namely, a combination of factors), the SAEA was ran with four different HP settings and with 30 runs per setting to obtain an adequate sample, thereby yielding a total of 2400 runs5functions×2dimensions×2BF×4HPsettings×30repeats.Tables2–5 present the resultant statistics (mean, median, standard deviation, min, and max) of the final function values for each batch of 30 runs across the different HP values and test cases. Also provided is the p-value of the Mann–Whitney statistical significance test. It was obtained by comparing the results corresponding to the HP associated with the best mean to the results associated with the 3 remaining HP settings. Statistical significance was measured at the 0.05 level, namely, for p≤0.05. As such in each of the 20 combinations of factors (function-dimension-BF), the number of significant performance differences observed (denoted s) varied between 0 (no significant differences observed) which implies that the HP had no significant impact to 3 (all comparisons had significant differences) which implies that the HP had a strong impact.Table 2
Statistics 5D, Gaussian.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-05Mean1.610e + 011.794e + 011.811e + 011.726e + 01SD4.476e + 002.037e + 001.810e + 002.225e + 00Median1.770e + 011.848e + 011.872e + 011.780e + 01Min6.060e + 001.199e + 011.276e + 011.231e + 01Max2.033e + 012.008e + 011.996e + 012.009e + 01p-value9.412e − 027.791e − 024.412e − 01Dixon-05Mean1.958e + 025.127e + 016.533e + 033.398e + 03SD2.114e + 024.044e + 016.211e + 033.428e + 03Median1.160e + 023.938e + 015.316e + 032.334e + 03Min1.566e + 002.819e + 001.831e + 026.586e + 00Max7.909e + 021.554e + 022.792e + 041.292e + 04p-value5.428e − 041.436e − 112.201e − 10Griewank-05Mean4.513e + 012.996e + 013.317e + 013.225e + 01SD2.180e + 011.440e + 011.911e + 011.855e + 01Median4.488e + 012.803e + 013.068e + 012.675e + 01Min5.703e + 006.611e + 003.571e + 008.263e + 00Max8.845e + 015.749e + 017.990e + 016.709e + 01p-value1.411e − 032.722e − 014.529e − 01Rastrigin-05Mean4.009e + 013.891e + 014.491e + 014.131e + 01SD9.180e + 001.245e + 011.033e + 019.774e + 00Median4.004e + 013.789e + 014.705e + 014.243e + 01Min2.457e + 011.497e + 012.215e + 011.978e + 01Max5.783e + 016.491e + 016.130e + 015.762e + 01p-value4.296e − 012.639e − 022.254e − 01Rosenbrock-05Mean2.000e + 019.075e + 006.598e + 011.609e + 02SD2.152e + 011.520e + 019.548e + 011.169e + 02Median1.455e + 014.323e + 001.706e + 011.353e + 02Min1.188e + 008.286e − 013.115e + 003.038e + 01Max1.023e + 026.534e + 013.413e + 025.017e + 02p-value2.247e − 059.656e − 062.886e − 11Table 3
Statistics 10D, Gaussian.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-10Mean2.018e + 012.014e + 011.971e + 011.977e + 01SD4.794e − 013.787e − 015.736e − 015.120e − 01Median2.026e + 012.025e + 011.975e + 011.995e + 01Min1.851e + 011.911e + 011.767e + 011.862e + 01Max2.070e + 012.080e + 012.054e + 012.047e + 01p-value1.229e − 043.187e − 042.436e − 01Dixon-10Mean3.845e + 031.043e + 051.234e + 056.127e + 04SD5.651e + 035.375e + 045.635e + 043.290e + 04Median2.158e + 039.943e + 041.128e + 055.259e + 04Min4.009e + 022.091e + 034.255e + 046.235e + 03Max2.456e + 042.012e + 052.567e + 051.335e + 05p-value1.027e − 101.436e − 112.614e − 11Griewank-10Mean1.546e + 021.171e + 021.129e + 021.170e + 02SD3.277e + 012.659e + 013.387e + 013.271e + 01Median1.510e + 021.137e + 021.129e + 021.159e + 02Min1.012e + 027.288e + 014.410e + 013.683e + 01Max2.346e + 021.883e + 021.624e + 021.920e + 02p-value2.551e − 053.503e − 012.973e − 01Rastrigin-10Mean1.053e + 021.187e + 021.220e + 021.109e + 02SD1.923e + 012.090e + 011.541e + 011.389e + 01Median1.097e + 021.154e + 021.229e + 021.102e + 02Min6.244e + 017.789e + 018.795e + 018.355e + 01Max1.325e + 021.546e + 021.598e + 021.370e + 02p-value2.066e − 025.718e − 042.166e − 01Rosenbrock-10Mean1.907e + 028.191e + 011.215e + 031.092e + 03SD1.141e + 026.346e + 014.582e + 024.177e + 02Median1.513e + 026.880e + 011.234e + 039.760e + 02Min5.047e + 019.250e + 003.200e + 025.115e + 02Max5.253e + 023.307e + 022.156e + 032.018e + 03p-value4.590e − 071.588e − 111.436e − 11Table 4
Statistics 5D, multiquadric.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-05Mean1.489e + 011.083e + 011.032e + 011.134e + 01SD3.841e + 004.928e + 002.644e + 002.648e + 00Median1.589e + 019.730e + 001.041e + 011.203e + 01Min5.115e + 003.771e + 005.193e + 005.197e + 00Max1.971e + 011.851e + 011.447e + 011.477e + 01p-value2.829e − 064.238e − 014.888e − 02Dixon-05Mean1.228e + 024.939e + 008.916e + 013.662e + 02SD1.155e + 027.272e + 001.376e + 026.810e + 02Median8.671e + 012.390e + 005.086e + 011.050e + 02Min7.622e + 004.486e − 011.659e + 001.686e + 00Max5.362e + 023.944e + 017.063e + 023.443e + 03p-value5.208e − 112.883e − 085.554e − 08Griewank-05Mean3.526e + 009.077e + 001.900e + 011.316e + 01SD2.545e + 008.885e + 001.387e + 019.310e + 00Median2.610e + 006.687e + 001.620e + 011.282e + 01Min6.409e − 012.856e − 011.048e + 002.213e + 00Max1.007e + 014.335e + 015.363e + 015.079e + 01p-value6.053e − 051.748e − 082.654e − 08Rastrigin-05Mean3.636e + 013.500e + 011.314e + 011.367e + 01SD1.156e + 011.030e + 018.408e + 008.820e + 00Median3.753e + 013.636e + 011.109e + 011.101e + 01Min1.281e + 018.895e + 002.029e + 002.426e + 00Max5.083e + 015.855e + 013.887e + 013.784e + 01p-value1.528e − 095.277e − 094.238e − 01Rosenbrock-05Mean2.763e + 016.443e + 003.542e + 001.604e + 01SD2.164e + 018.784e + 001.333e + 002.136e + 01Median1.593e + 013.464e + 003.613e + 006.037e + 00Min3.526e + 006.503e − 027.370e − 011.503e − 01Max8.159e + 014.343e + 016.196e + 009.069e + 01p-value6.146e − 104.122e − 011.257e − 05Table 5
Statistics 10D, multiquadric.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-10Mean1.648e + 011.224e + 011.470e + 011.480e + 01SD3.290e + 003.485e + 001.406e + 001.286e + 00Median1.751e + 011.268e + 011.464e + 011.487e + 01Min8.904e + 006.471e + 001.158e + 011.213e + 01Max1.993e + 011.722e + 011.766e + 011.691e + 01p-value5.271e − 061.014e − 025.497e − 03Dixon-10Mean2.079e + 031.294e + 031.114e + 042.838e + 04SD2.267e + 031.340e + 036.722e + 032.301e + 04Median1.320e + 039.800e + 029.620e + 032.347e + 04Min1.576e + 022.490e + 021.263e + 036.460e + 02Max1.097e + 046.601e + 032.647e + 049.430e + 04p-value3.450e − 021.244e − 103.866e − 10Griewank-10Mean1.929e + 017.998e + 011.000e + 021.073e + 02SD9.413e + 003.160e + 013.888e + 012.896e + 01Median1.962e + 017.887e + 019.844e + 011.087e + 02Min4.407e + 002.108e + 012.651e + 015.450e + 01Max4.322e + 011.444e + 021.720e + 021.720e + 02p-value1.822e − 106.329e − 111.436e − 11Rastrigin-10Mean1.026e + 029.370e + 014.610e + 015.373e + 01SD1.695e + 011.896e + 012.132e + 011.790e + 01Median1.036e + 029.604e + 014.194e + 015.095e + 01Min5.341e + 014.806e + 011.508e + 012.598e + 01Max1.338e + 021.277e + 028.833e + 011.011e + 02p-value7.685e − 112.000e − 093.926e − 02Rosenbrock-10Mean1.150e + 024.843e + 016.869e + 011.815e + 02SD6.459e + 013.006e + 013.263e + 011.160e + 02Median9.499e + 013.763e + 016.322e + 011.751e + 02Min2.865e + 018.866e + 002.229e + 014.291e + 01Max2.552e + 021.298e + 021.450e + 026.382e + 02p-value1.492e − 062.849e − 031.528e − 09The corresponding results are summarized in Table6 from which it follows that in 1 combination of factors the HP had no significant impact, in 3 cases it had only a minor impact, and in the remaining 16 cases it had a significant impact. The cases showing a lessened HP impact are low-dimensional and involve the Gaussian basis function. These results are attributed to the relative simplicity of the lower-dimensional cases which allowed for a good approximation across a range of HP values. Also, the output of a Gaussian BF surrogate approaches the sample mean faster than a MQ surrogate due to the Gaussian BF behaviour and therefore the RBF output was similar for different HP values which results in fewer performance differences, as observed. In contrast, with the MQ basis function and with the high-dimensional cases the HP consistently had a significant impact. This is attributed to the MQ basis function, whose output varies for a wider range of HP values. Also, in high-dimensional cases, generating an accurate surrogate is more challenging hence, different HP values resulted in significantly different prediction accuracies. These aspects are further addressed in length in the analysis which follows.Table 6
Test cases grouped by significant differencess.
sCaseTotal0G-Ack-0511G-Gri-05, G-Ras-05, G-Gri-1032G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-05, M-Ros-0553G-Dix-05, G-Ros-05, G-Dix-10, M-Dix-05, M-Gri-05, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1011Expanding on the preceding analysis an additional fine-grained analysis was also performed to highlight how each factor (BF, modality, dimension) affected performance and the results are summarized in Table7. It follows that the per-factor influence is as follows:Table 7
Significant performance differences by factor.
(a) Significant differences (s) by BFBFsTest caseGaussian0Ack-051Gri-05, Ras-05, Gri-102Ack-10, Ras-103Dix-05, Ros-05, Dix-10, Ros-10MQ012Ack-05, Ras-05, Ros-053Dix-05, Gri-05, Ack-10, Dix-10, Gri-10, Ras-10, Ros-10(b) Significant differences (s) by modalityModalitysTest caseHigh0G-Ack-051G-Gri-05, G-Ras-05, G-Gri-102G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-053M-Gri-05, M-Ack-10, M-Gri-10, M-Ras-10Low012M-Ros-053G-Dix-05, G-Ros-05, G-Dix-10, G-Ros-10, M-Dix-05, M-Dix-10, M-Ros-10(c) Significant differences by function dimension (d)DimensionsTest cases50G-Ack-051G-Gri-05, G-Ras-052M-Ack-05, M-Ras-05, M-Ros-053G-Dix-05, G-Ros-05, M-Dix-05, M-Gri-051001G-Gri-102G-Ack-10, G-Ras-103G-Dix-10, G-Ros-10, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-10


#### 4.1.1. Basis Function

For a Gaussian BF in 4 cases varying the HP had a negligible impact while the MQ BF variations were always significant, namely, the sensitivity to the HP was lower for a Gaussian BF than it was for a MQ one. This is attributed to the mathematical formulation of each BF, namely, for a Gaussian BF(5)φx⟶=exp−σx⟶−x⟶i22.the expression decays to 0 rapidly for a wide range of HP values which results in similar surrogate predictions for different HP values and consequentially a lower HP impact. In contrast, with a MQ BF,(6)φx⟶=1+σx⟶−x⟶i22.the output varies over a wider range of HP values which in turn affects the surrogate prediction and results in a more significant HP impact. As an example Figure3(a) shows a plot of the RMSE of the surrogate prediction when using Gaussian and MQ BFs with the Ackley-5D function. The error quickly rises for the Gaussian BF with HP values since the individual BFs decay to zero and the surrogate prediction approaches the sample mean. In contrast, there is a much higher sensitivity to the HP values with the MQ BFs as observed in the experiments.Figure 3
Variation of the RMSE by factors. (a) Prediction RMSE by BF (Ackley-5). (b) Prediction RMSE by modality (modified Rastrigin function,k). (c) Prediction RMSE by dimension (Rosenbrock, d).
(a)(b)(c)


#### 4.1.2. Modality

The HP impact was significant for the low modality functions (Dixon, Rosenbrock, 2-3 significant differences per test case) but its impact was diminished for the high modality functions (Ackley, Griewank, and Rastrigin). This is attributed to the fact that as the modality was increased the surrogate prediction accuracy degraded due to the more complicated function features up to a point where the prediction accuracy was negligibly affected by different HP values. To demonstrate this, numerical experiments were performed with a modified Rastrigin function(7)fx⟶=∑i=1dxi2−10coskπxi,where k is a prescribed parameter which determines the function modality (higher values increase modality). The surrogate prediction accuracy was examined for a 5D function and results are presented in Figure 3(b). It follows that the surrogate prediction RMSE increased with modality for all HP values as observed in the numerical experiments. Consequentially the impact of the HP was more pronounced for the lower modality settings and diminished as the modality was increased, as observed in the numerical experiments.


#### 4.1.3. Dimension

In the lower dimensional casesd=5, the HP had a significant impact in 7 cases, while in the higher dimensional cases d=10 in 9 cases. These results are attributed to the “curse of dimensionality,” namely in the lower dimensional problems a more accurate surrogate could be trained since the problems are simpler and hence the sensitivity to the HP was lower. In contrast, in the higher dimensional cases training an accurate surrogate was more challenging and required more precise HP values hence the higher sensitivity to the HP. To demonstrate this Figure 3(c) shows the variation of the prediction RMSE for different dimensions with the Rosenbrock function. It follows that for lower dimensions changing the HP affected the RMSE much less than for higher dimensions, as observed in the experiments.Beyond the above per-factor analysis, an additional evaluation was performed to examine if and to what extent the impact of the HP varied depending on the main algorithm parameter of the limit of true (expensive) function evaluations. Accordingly, beyond the baseline50d evaluations limit the full set of numerical experiments was repeated also for a limit of 100 d and 200 d. Table 8 summarizes the obtained results for the number of cases in which the HP had a significant impact. It follows that the impact was consistent across all sets and that the number of cases with a statistically significant performance difference and the test cases in which they occurred remained largely unchanged across the three settings. Accordingly, the remainder of the analysis is presented for the 50d setting.Table 8
Test cases grouped by significant differencess and evaluations limits.
EvaluationssCaseTotal50d0G-Ack-0511G-Gri-05, G-Ras-05, G-Gri-1032G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-05, M-Ros-0553G-Dix-05, G-Ros-05, G-Dix-10, M-Dix-05, M-Gri-05, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1011100d0Ack-0511G-Gri-05, G-Gri-10, M-Ack-05, M-Ack-1042G-Ras-10, M-Ras-05, M-Ros-05, M-Ros-1043G-Dix-05, G-Ras-05, G-Ros-05, G-Ack-10, G-Dix-10, G-Ros-10, M-Dix-05, M-Gri-05, M-Dix-10, M-Gri-10, M-Ras-1011200d001G-Gri-05, G-Gri-1022G-Ack-05, G-Ras-10, M-Ack-05, M-Ros-05, M-Ack-1053G-Dix-05, G-Ras-05, G-Ros-05, G-Ack-10, G-Dix-10, G-Ros-10, M-Dix-05, M-Gri-05, M-Ras-05, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1013Overall, the analysis shows that except for specific cases (Gaussian BF/low modality functions/low dimensions), the HP had a significant impact on the overall SAEA performance.


### 4.2. Dynamically Optimized HP

The preceding analysis examined cases where the HP value was unchanged. This raises the following question:would the overall SAEA performance vary if the HP is dynamically calibrated during the search?. For a rigorous evaluation an additional set of experiments was performed with a modified algorithm in which the HP was dynamically calibrated based on a cross-validation procedure with an 80-20 training-testing split ratio. The HP optimization was achieved by a golden search algorithm in the interval 10−4,1. Tests were conducted as in the preceding section and results are presented in Table 9 and compared to those from the best-performing variant (mean-wise) from the preceding section. It follows that optimizing the HP consistently improved performance as evident both from the mean and p-values where the variant with the optimized HP had a statistically significant performance advantage at 0.05 level (p-value<0.05) in 8 out of 10 cases (all except the Ackley-5D and Rosenbrock-5D functions).Table 9
Statistics, Gaussian.
FunctionStatistic5D10DOptimizedBest fixedOptimizedBest fixedAckleyMean1.452e + 011.610e + 011.400e + 011.971e + 01SD4.558e + 004.476e + 004.551e + 005.736e − 01Median1.497e + 011.770e + 011.435e + 011.975e + 01Min5.417e + 006.060e + 003.632e + 001.767e + 01Max2.049e + 012.033e + 012.046e + 012.054e + 01p-value1.099e − 011.693e − 07DixonMean1.445e + 015.127e + 011.422e + 013.845e + 03SD4.593e + 004.044e + 014.157e + 005.651e + 03Median1.434e + 013.938e + 011.498e + 012.158e + 03Min7.511e + 002.819e + 006.394e + 004.009e + 02Max2.032e + 011.554e + 022.021e + 012.456e + 04p-value3.950e − 051.436e − 11GriewankMean1.469e + 012.996e + 011.565e + 011.129e + 02SD4.021e + 001.440e + 014.288e + 003.387e + 01Median1.470e + 012.803e + 011.696e + 011.129e + 02Min6.577e + 006.611e + 006.330e + 004.410e + 01Max2.049e + 015.749e + 012.046e + 011.624e + 02p-value2.133e − 061.436e − 11RastriginMean1.618e + 013.891e + 011.653e + 011.053e + 02SD3.720e + 001.245e + 014.337e + 001.923e + 01Median1.738e + 013.789e + 011.862e + 011.097e + 02Min8.667e + 001.497e + 016.259e + 006.244e + 01Max2.050e + 016.491e + 012.057e + 011.325e + 02p-value1.165e − 091.436e − 11RosenbrockMean1.583e + 019.075e + 001.407e + 018.191e + 01SD4.182e + 001.520e + 014.255e + 006.346e + 01Median1.662e + 014.323e + 001.270e + 016.880e + 01Min8.272e + 008.286e − 015.874e + 009.250e + 00Max2.044e + 016.534e + 012.040e + 013.307e + 02p-value1.000e + 007.386e − 10Furthermore, the prediction accuracies of the optimized and fixed HP variants were compared and the results are presented in Figure4. It follows that the optimized HP variant had significantly smaller prediction errors and consequently achieved better final results. This trend was observed across different test cases, as shown. Overall, the analysis shows that dynamically optimizing the HP consistently improved the SAEA performance.Figure 4
Prediction error variation for a fixed vs. optimized HP.


## 4.1. Fixed HP

A first set of numerical experiments was performed to analyze if and to what extent different fixed HP values, namely, which are chosen a-priori, affect the overall SAEA performance. Following the Design-of-Experiments (DOE) framework [38] the numerical experiments were formulated by using a single variable, namely the HP, which was assigned different fixed values, and 3 factors (function type, function dimension, and BF type). Accordingly, in each combination of factors the numerical experiments were repeated with the different HP values so that the HP impact could be analyzed across a wide range of problem settings. The DOE setup is summarized in Table 1.Table 1
Design of the numerical experiments.
TypeNameValuesFactorFunctionAckley, Griewank, Powell, Rastrigin, RosenbrockDimension5, 10BFGaussian, MQVariableHyperparameter0.01, 0.1, 1, 10The established set of test functions of [39] was used as it includes functions with diverse features as shown in Figure 2.Figure 2
The test functions used.In each test case (namely, a combination of factors), the SAEA was ran with four different HP settings and with 30 runs per setting to obtain an adequate sample, thereby yielding a total of 2400 runs5functions×2dimensions×2BF×4HPsettings×30repeats.Tables2–5 present the resultant statistics (mean, median, standard deviation, min, and max) of the final function values for each batch of 30 runs across the different HP values and test cases. Also provided is the p-value of the Mann–Whitney statistical significance test. It was obtained by comparing the results corresponding to the HP associated with the best mean to the results associated with the 3 remaining HP settings. Statistical significance was measured at the 0.05 level, namely, for p≤0.05. As such in each of the 20 combinations of factors (function-dimension-BF), the number of significant performance differences observed (denoted s) varied between 0 (no significant differences observed) which implies that the HP had no significant impact to 3 (all comparisons had significant differences) which implies that the HP had a strong impact.Table 2
Statistics 5D, Gaussian.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-05Mean1.610e + 011.794e + 011.811e + 011.726e + 01SD4.476e + 002.037e + 001.810e + 002.225e + 00Median1.770e + 011.848e + 011.872e + 011.780e + 01Min6.060e + 001.199e + 011.276e + 011.231e + 01Max2.033e + 012.008e + 011.996e + 012.009e + 01p-value9.412e − 027.791e − 024.412e − 01Dixon-05Mean1.958e + 025.127e + 016.533e + 033.398e + 03SD2.114e + 024.044e + 016.211e + 033.428e + 03Median1.160e + 023.938e + 015.316e + 032.334e + 03Min1.566e + 002.819e + 001.831e + 026.586e + 00Max7.909e + 021.554e + 022.792e + 041.292e + 04p-value5.428e − 041.436e − 112.201e − 10Griewank-05Mean4.513e + 012.996e + 013.317e + 013.225e + 01SD2.180e + 011.440e + 011.911e + 011.855e + 01Median4.488e + 012.803e + 013.068e + 012.675e + 01Min5.703e + 006.611e + 003.571e + 008.263e + 00Max8.845e + 015.749e + 017.990e + 016.709e + 01p-value1.411e − 032.722e − 014.529e − 01Rastrigin-05Mean4.009e + 013.891e + 014.491e + 014.131e + 01SD9.180e + 001.245e + 011.033e + 019.774e + 00Median4.004e + 013.789e + 014.705e + 014.243e + 01Min2.457e + 011.497e + 012.215e + 011.978e + 01Max5.783e + 016.491e + 016.130e + 015.762e + 01p-value4.296e − 012.639e − 022.254e − 01Rosenbrock-05Mean2.000e + 019.075e + 006.598e + 011.609e + 02SD2.152e + 011.520e + 019.548e + 011.169e + 02Median1.455e + 014.323e + 001.706e + 011.353e + 02Min1.188e + 008.286e − 013.115e + 003.038e + 01Max1.023e + 026.534e + 013.413e + 025.017e + 02p-value2.247e − 059.656e − 062.886e − 11Table 3
Statistics 10D, Gaussian.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-10Mean2.018e + 012.014e + 011.971e + 011.977e + 01SD4.794e − 013.787e − 015.736e − 015.120e − 01Median2.026e + 012.025e + 011.975e + 011.995e + 01Min1.851e + 011.911e + 011.767e + 011.862e + 01Max2.070e + 012.080e + 012.054e + 012.047e + 01p-value1.229e − 043.187e − 042.436e − 01Dixon-10Mean3.845e + 031.043e + 051.234e + 056.127e + 04SD5.651e + 035.375e + 045.635e + 043.290e + 04Median2.158e + 039.943e + 041.128e + 055.259e + 04Min4.009e + 022.091e + 034.255e + 046.235e + 03Max2.456e + 042.012e + 052.567e + 051.335e + 05p-value1.027e − 101.436e − 112.614e − 11Griewank-10Mean1.546e + 021.171e + 021.129e + 021.170e + 02SD3.277e + 012.659e + 013.387e + 013.271e + 01Median1.510e + 021.137e + 021.129e + 021.159e + 02Min1.012e + 027.288e + 014.410e + 013.683e + 01Max2.346e + 021.883e + 021.624e + 021.920e + 02p-value2.551e − 053.503e − 012.973e − 01Rastrigin-10Mean1.053e + 021.187e + 021.220e + 021.109e + 02SD1.923e + 012.090e + 011.541e + 011.389e + 01Median1.097e + 021.154e + 021.229e + 021.102e + 02Min6.244e + 017.789e + 018.795e + 018.355e + 01Max1.325e + 021.546e + 021.598e + 021.370e + 02p-value2.066e − 025.718e − 042.166e − 01Rosenbrock-10Mean1.907e + 028.191e + 011.215e + 031.092e + 03SD1.141e + 026.346e + 014.582e + 024.177e + 02Median1.513e + 026.880e + 011.234e + 039.760e + 02Min5.047e + 019.250e + 003.200e + 025.115e + 02Max5.253e + 023.307e + 022.156e + 032.018e + 03p-value4.590e − 071.588e − 111.436e − 11Table 4
Statistics 5D, multiquadric.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-05Mean1.489e + 011.083e + 011.032e + 011.134e + 01SD3.841e + 004.928e + 002.644e + 002.648e + 00Median1.589e + 019.730e + 001.041e + 011.203e + 01Min5.115e + 003.771e + 005.193e + 005.197e + 00Max1.971e + 011.851e + 011.447e + 011.477e + 01p-value2.829e − 064.238e − 014.888e − 02Dixon-05Mean1.228e + 024.939e + 008.916e + 013.662e + 02SD1.155e + 027.272e + 001.376e + 026.810e + 02Median8.671e + 012.390e + 005.086e + 011.050e + 02Min7.622e + 004.486e − 011.659e + 001.686e + 00Max5.362e + 023.944e + 017.063e + 023.443e + 03p-value5.208e − 112.883e − 085.554e − 08Griewank-05Mean3.526e + 009.077e + 001.900e + 011.316e + 01SD2.545e + 008.885e + 001.387e + 019.310e + 00Median2.610e + 006.687e + 001.620e + 011.282e + 01Min6.409e − 012.856e − 011.048e + 002.213e + 00Max1.007e + 014.335e + 015.363e + 015.079e + 01p-value6.053e − 051.748e − 082.654e − 08Rastrigin-05Mean3.636e + 013.500e + 011.314e + 011.367e + 01SD1.156e + 011.030e + 018.408e + 008.820e + 00Median3.753e + 013.636e + 011.109e + 011.101e + 01Min1.281e + 018.895e + 002.029e + 002.426e + 00Max5.083e + 015.855e + 013.887e + 013.784e + 01p-value1.528e − 095.277e − 094.238e − 01Rosenbrock-05Mean2.763e + 016.443e + 003.542e + 001.604e + 01SD2.164e + 018.784e + 001.333e + 002.136e + 01Median1.593e + 013.464e + 003.613e + 006.037e + 00Min3.526e + 006.503e − 027.370e − 011.503e − 01Max8.159e + 014.343e + 016.196e + 009.069e + 01p-value6.146e − 104.122e − 011.257e − 05Table 5
Statistics 10D, multiquadric.
FunctionStatisticκ = 0.01κ = 0.1κ = 1κ = 10Ackley-10Mean1.648e + 011.224e + 011.470e + 011.480e + 01SD3.290e + 003.485e + 001.406e + 001.286e + 00Median1.751e + 011.268e + 011.464e + 011.487e + 01Min8.904e + 006.471e + 001.158e + 011.213e + 01Max1.993e + 011.722e + 011.766e + 011.691e + 01p-value5.271e − 061.014e − 025.497e − 03Dixon-10Mean2.079e + 031.294e + 031.114e + 042.838e + 04SD2.267e + 031.340e + 036.722e + 032.301e + 04Median1.320e + 039.800e + 029.620e + 032.347e + 04Min1.576e + 022.490e + 021.263e + 036.460e + 02Max1.097e + 046.601e + 032.647e + 049.430e + 04p-value3.450e − 021.244e − 103.866e − 10Griewank-10Mean1.929e + 017.998e + 011.000e + 021.073e + 02SD9.413e + 003.160e + 013.888e + 012.896e + 01Median1.962e + 017.887e + 019.844e + 011.087e + 02Min4.407e + 002.108e + 012.651e + 015.450e + 01Max4.322e + 011.444e + 021.720e + 021.720e + 02p-value1.822e − 106.329e − 111.436e − 11Rastrigin-10Mean1.026e + 029.370e + 014.610e + 015.373e + 01SD1.695e + 011.896e + 012.132e + 011.790e + 01Median1.036e + 029.604e + 014.194e + 015.095e + 01Min5.341e + 014.806e + 011.508e + 012.598e + 01Max1.338e + 021.277e + 028.833e + 011.011e + 02p-value7.685e − 112.000e − 093.926e − 02Rosenbrock-10Mean1.150e + 024.843e + 016.869e + 011.815e + 02SD6.459e + 013.006e + 013.263e + 011.160e + 02Median9.499e + 013.763e + 016.322e + 011.751e + 02Min2.865e + 018.866e + 002.229e + 014.291e + 01Max2.552e + 021.298e + 021.450e + 026.382e + 02p-value1.492e − 062.849e − 031.528e − 09The corresponding results are summarized in Table6 from which it follows that in 1 combination of factors the HP had no significant impact, in 3 cases it had only a minor impact, and in the remaining 16 cases it had a significant impact. The cases showing a lessened HP impact are low-dimensional and involve the Gaussian basis function. These results are attributed to the relative simplicity of the lower-dimensional cases which allowed for a good approximation across a range of HP values. Also, the output of a Gaussian BF surrogate approaches the sample mean faster than a MQ surrogate due to the Gaussian BF behaviour and therefore the RBF output was similar for different HP values which results in fewer performance differences, as observed. In contrast, with the MQ basis function and with the high-dimensional cases the HP consistently had a significant impact. This is attributed to the MQ basis function, whose output varies for a wider range of HP values. Also, in high-dimensional cases, generating an accurate surrogate is more challenging hence, different HP values resulted in significantly different prediction accuracies. These aspects are further addressed in length in the analysis which follows.Table 6
Test cases grouped by significant differencess.
sCaseTotal0G-Ack-0511G-Gri-05, G-Ras-05, G-Gri-1032G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-05, M-Ros-0553G-Dix-05, G-Ros-05, G-Dix-10, M-Dix-05, M-Gri-05, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1011Expanding on the preceding analysis an additional fine-grained analysis was also performed to highlight how each factor (BF, modality, dimension) affected performance and the results are summarized in Table7. It follows that the per-factor influence is as follows:Table 7
Significant performance differences by factor.
(a) Significant differences (s) by BFBFsTest caseGaussian0Ack-051Gri-05, Ras-05, Gri-102Ack-10, Ras-103Dix-05, Ros-05, Dix-10, Ros-10MQ012Ack-05, Ras-05, Ros-053Dix-05, Gri-05, Ack-10, Dix-10, Gri-10, Ras-10, Ros-10(b) Significant differences (s) by modalityModalitysTest caseHigh0G-Ack-051G-Gri-05, G-Ras-05, G-Gri-102G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-053M-Gri-05, M-Ack-10, M-Gri-10, M-Ras-10Low012M-Ros-053G-Dix-05, G-Ros-05, G-Dix-10, G-Ros-10, M-Dix-05, M-Dix-10, M-Ros-10(c) Significant differences by function dimension (d)DimensionsTest cases50G-Ack-051G-Gri-05, G-Ras-052M-Ack-05, M-Ras-05, M-Ros-053G-Dix-05, G-Ros-05, M-Dix-05, M-Gri-051001G-Gri-102G-Ack-10, G-Ras-103G-Dix-10, G-Ros-10, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-10


### 4.1.1. Basis Function

For a Gaussian BF in 4 cases varying the HP had a negligible impact while the MQ BF variations were always significant, namely, the sensitivity to the HP was lower for a Gaussian BF than it was for a MQ one. This is attributed to the mathematical formulation of each BF, namely, for a Gaussian BF(5)φx⟶=exp−σx⟶−x⟶i22.the expression decays to 0 rapidly for a wide range of HP values which results in similar surrogate predictions for different HP values and consequentially a lower HP impact. In contrast, with a MQ BF,(6)φx⟶=1+σx⟶−x⟶i22.the output varies over a wider range of HP values which in turn affects the surrogate prediction and results in a more significant HP impact. As an example Figure3(a) shows a plot of the RMSE of the surrogate prediction when using Gaussian and MQ BFs with the Ackley-5D function. The error quickly rises for the Gaussian BF with HP values since the individual BFs decay to zero and the surrogate prediction approaches the sample mean. In contrast, there is a much higher sensitivity to the HP values with the MQ BFs as observed in the experiments.Figure 3
Variation of the RMSE by factors. (a) Prediction RMSE by BF (Ackley-5). (b) Prediction RMSE by modality (modified Rastrigin function,k). (c) Prediction RMSE by dimension (Rosenbrock, d).
(a)(b)(c)


### 4.1.2. Modality

The HP impact was significant for the low modality functions (Dixon, Rosenbrock, 2-3 significant differences per test case) but its impact was diminished for the high modality functions (Ackley, Griewank, and Rastrigin). This is attributed to the fact that as the modality was increased the surrogate prediction accuracy degraded due to the more complicated function features up to a point where the prediction accuracy was negligibly affected by different HP values. To demonstrate this, numerical experiments were performed with a modified Rastrigin function(7)fx⟶=∑i=1dxi2−10coskπxi,where k is a prescribed parameter which determines the function modality (higher values increase modality). The surrogate prediction accuracy was examined for a 5D function and results are presented in Figure 3(b). It follows that the surrogate prediction RMSE increased with modality for all HP values as observed in the numerical experiments. Consequentially the impact of the HP was more pronounced for the lower modality settings and diminished as the modality was increased, as observed in the numerical experiments.


### 4.1.3. Dimension

In the lower dimensional casesd=5, the HP had a significant impact in 7 cases, while in the higher dimensional cases d=10 in 9 cases. These results are attributed to the “curse of dimensionality,” namely in the lower dimensional problems a more accurate surrogate could be trained since the problems are simpler and hence the sensitivity to the HP was lower. In contrast, in the higher dimensional cases training an accurate surrogate was more challenging and required more precise HP values hence the higher sensitivity to the HP. To demonstrate this Figure 3(c) shows the variation of the prediction RMSE for different dimensions with the Rosenbrock function. It follows that for lower dimensions changing the HP affected the RMSE much less than for higher dimensions, as observed in the experiments.Beyond the above per-factor analysis, an additional evaluation was performed to examine if and to what extent the impact of the HP varied depending on the main algorithm parameter of the limit of true (expensive) function evaluations. Accordingly, beyond the baseline50d evaluations limit the full set of numerical experiments was repeated also for a limit of 100 d and 200 d. Table 8 summarizes the obtained results for the number of cases in which the HP had a significant impact. It follows that the impact was consistent across all sets and that the number of cases with a statistically significant performance difference and the test cases in which they occurred remained largely unchanged across the three settings. Accordingly, the remainder of the analysis is presented for the 50d setting.Table 8
Test cases grouped by significant differencess and evaluations limits.
EvaluationssCaseTotal50d0G-Ack-0511G-Gri-05, G-Ras-05, G-Gri-1032G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-05, M-Ros-0553G-Dix-05, G-Ros-05, G-Dix-10, M-Dix-05, M-Gri-05, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1011100d0Ack-0511G-Gri-05, G-Gri-10, M-Ack-05, M-Ack-1042G-Ras-10, M-Ras-05, M-Ros-05, M-Ros-1043G-Dix-05, G-Ras-05, G-Ros-05, G-Ack-10, G-Dix-10, G-Ros-10, M-Dix-05, M-Gri-05, M-Dix-10, M-Gri-10, M-Ras-1011200d001G-Gri-05, G-Gri-1022G-Ack-05, G-Ras-10, M-Ack-05, M-Ros-05, M-Ack-1053G-Dix-05, G-Ras-05, G-Ros-05, G-Ack-10, G-Dix-10, G-Ros-10, M-Dix-05, M-Gri-05, M-Ras-05, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1013Overall, the analysis shows that except for specific cases (Gaussian BF/low modality functions/low dimensions), the HP had a significant impact on the overall SAEA performance.


## 4.1.1. Basis Function

For a Gaussian BF in 4 cases varying the HP had a negligible impact while the MQ BF variations were always significant, namely, the sensitivity to the HP was lower for a Gaussian BF than it was for a MQ one. This is attributed to the mathematical formulation of each BF, namely, for a Gaussian BF(5)φx⟶=exp−σx⟶−x⟶i22.the expression decays to 0 rapidly for a wide range of HP values which results in similar surrogate predictions for different HP values and consequentially a lower HP impact. In contrast, with a MQ BF,(6)φx⟶=1+σx⟶−x⟶i22.the output varies over a wider range of HP values which in turn affects the surrogate prediction and results in a more significant HP impact. As an example Figure3(a) shows a plot of the RMSE of the surrogate prediction when using Gaussian and MQ BFs with the Ackley-5D function. The error quickly rises for the Gaussian BF with HP values since the individual BFs decay to zero and the surrogate prediction approaches the sample mean. In contrast, there is a much higher sensitivity to the HP values with the MQ BFs as observed in the experiments.Figure 3
Variation of the RMSE by factors. (a) Prediction RMSE by BF (Ackley-5). (b) Prediction RMSE by modality (modified Rastrigin function,k). (c) Prediction RMSE by dimension (Rosenbrock, d).
(a)(b)(c)


## 4.1.2. Modality

The HP impact was significant for the low modality functions (Dixon, Rosenbrock, 2-3 significant differences per test case) but its impact was diminished for the high modality functions (Ackley, Griewank, and Rastrigin). This is attributed to the fact that as the modality was increased the surrogate prediction accuracy degraded due to the more complicated function features up to a point where the prediction accuracy was negligibly affected by different HP values. To demonstrate this, numerical experiments were performed with a modified Rastrigin function(7)fx⟶=∑i=1dxi2−10coskπxi,where k is a prescribed parameter which determines the function modality (higher values increase modality). The surrogate prediction accuracy was examined for a 5D function and results are presented in Figure 3(b). It follows that the surrogate prediction RMSE increased with modality for all HP values as observed in the numerical experiments. Consequentially the impact of the HP was more pronounced for the lower modality settings and diminished as the modality was increased, as observed in the numerical experiments.


## 4.1.3. Dimension

In the lower dimensional casesd=5, the HP had a significant impact in 7 cases, while in the higher dimensional cases d=10 in 9 cases. These results are attributed to the “curse of dimensionality,” namely in the lower dimensional problems a more accurate surrogate could be trained since the problems are simpler and hence the sensitivity to the HP was lower. In contrast, in the higher dimensional cases training an accurate surrogate was more challenging and required more precise HP values hence the higher sensitivity to the HP. To demonstrate this Figure 3(c) shows the variation of the prediction RMSE for different dimensions with the Rosenbrock function. It follows that for lower dimensions changing the HP affected the RMSE much less than for higher dimensions, as observed in the experiments.Beyond the above per-factor analysis, an additional evaluation was performed to examine if and to what extent the impact of the HP varied depending on the main algorithm parameter of the limit of true (expensive) function evaluations. Accordingly, beyond the baseline50d evaluations limit the full set of numerical experiments was repeated also for a limit of 100 d and 200 d. Table 8 summarizes the obtained results for the number of cases in which the HP had a significant impact. It follows that the impact was consistent across all sets and that the number of cases with a statistically significant performance difference and the test cases in which they occurred remained largely unchanged across the three settings. Accordingly, the remainder of the analysis is presented for the 50d setting.Table 8
Test cases grouped by significant differencess and evaluations limits.
EvaluationssCaseTotal50d0G-Ack-0511G-Gri-05, G-Ras-05, G-Gri-1032G-Ack-10, G-Ras-10, M-Ack-05, M-Ras-05, M-Ros-0553G-Dix-05, G-Ros-05, G-Dix-10, M-Dix-05, M-Gri-05, M-Ack-10, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1011100d0Ack-0511G-Gri-05, G-Gri-10, M-Ack-05, M-Ack-1042G-Ras-10, M-Ras-05, M-Ros-05, M-Ros-1043G-Dix-05, G-Ras-05, G-Ros-05, G-Ack-10, G-Dix-10, G-Ros-10, M-Dix-05, M-Gri-05, M-Dix-10, M-Gri-10, M-Ras-1011200d001G-Gri-05, G-Gri-1022G-Ack-05, G-Ras-10, M-Ack-05, M-Ros-05, M-Ack-1053G-Dix-05, G-Ras-05, G-Ros-05, G-Ack-10, G-Dix-10, G-Ros-10, M-Dix-05, M-Gri-05, M-Ras-05, M-Dix-10, M-Gri-10, M-Ras-10, M-Ros-1013Overall, the analysis shows that except for specific cases (Gaussian BF/low modality functions/low dimensions), the HP had a significant impact on the overall SAEA performance.


## 4.2. Dynamically Optimized HP

The preceding analysis examined cases where the HP value was unchanged. This raises the following question:would the overall SAEA performance vary if the HP is dynamically calibrated during the search?. For a rigorous evaluation an additional set of experiments was performed with a modified algorithm in which the HP was dynamically calibrated based on a cross-validation procedure with an 80-20 training-testing split ratio. The HP optimization was achieved by a golden search algorithm in the interval 10−4,1. Tests were conducted as in the preceding section and results are presented in Table 9 and compared to those from the best-performing variant (mean-wise) from the preceding section. It follows that optimizing the HP consistently improved performance as evident both from the mean and p-values where the variant with the optimized HP had a statistically significant performance advantage at 0.05 level (p-value<0.05) in 8 out of 10 cases (all except the Ackley-5D and Rosenbrock-5D functions).Table 9
Statistics, Gaussian.
FunctionStatistic5D10DOptimizedBest fixedOptimizedBest fixedAckleyMean1.452e + 011.610e + 011.400e + 011.971e + 01SD4.558e + 004.476e + 004.551e + 005.736e − 01Median1.497e + 011.770e + 011.435e + 011.975e + 01Min5.417e + 006.060e + 003.632e + 001.767e + 01Max2.049e + 012.033e + 012.046e + 012.054e + 01p-value1.099e − 011.693e − 07DixonMean1.445e + 015.127e + 011.422e + 013.845e + 03SD4.593e + 004.044e + 014.157e + 005.651e + 03Median1.434e + 013.938e + 011.498e + 012.158e + 03Min7.511e + 002.819e + 006.394e + 004.009e + 02Max2.032e + 011.554e + 022.021e + 012.456e + 04p-value3.950e − 051.436e − 11GriewankMean1.469e + 012.996e + 011.565e + 011.129e + 02SD4.021e + 001.440e + 014.288e + 003.387e + 01Median1.470e + 012.803e + 011.696e + 011.129e + 02Min6.577e + 006.611e + 006.330e + 004.410e + 01Max2.049e + 015.749e + 012.046e + 011.624e + 02p-value2.133e − 061.436e − 11RastriginMean1.618e + 013.891e + 011.653e + 011.053e + 02SD3.720e + 001.245e + 014.337e + 001.923e + 01Median1.738e + 013.789e + 011.862e + 011.097e + 02Min8.667e + 001.497e + 016.259e + 006.244e + 01Max2.050e + 016.491e + 012.057e + 011.325e + 02p-value1.165e − 091.436e − 11RosenbrockMean1.583e + 019.075e + 001.407e + 018.191e + 01SD4.182e + 001.520e + 014.255e + 006.346e + 01Median1.662e + 014.323e + 001.270e + 016.880e + 01Min8.272e + 008.286e − 015.874e + 009.250e + 00Max2.044e + 016.534e + 012.040e + 013.307e + 02p-value1.000e + 007.386e − 10Furthermore, the prediction accuracies of the optimized and fixed HP variants were compared and the results are presented in Figure4. It follows that the optimized HP variant had significantly smaller prediction errors and consequently achieved better final results. This trend was observed across different test cases, as shown. Overall, the analysis shows that dynamically optimizing the HP consistently improved the SAEA performance.Figure 4
Prediction error variation for a fixed vs. optimized HP.


## 5. Conclusion

Surrogate-assisted evolutionary algorithms (SAEA) are often used to solve simulation-driven optimization problems and a common surrogate is the radial basis functions (RBF) model. The latter relies on a hyperparameter (HP) which affects both the individual basis functions and the overall surrogate prediction. The HP needs to be set by some numerical procedure and this raises the question how significant is its impact on the overall SAEA performance and not only on the stand-alone surrogate, namely, is the computational effort spent on the HP calibration justified in terms of the overall SAEA performance. This is an important consideration both for end-users and algorithm researches.To address this issue, this paper has presented an extensive set of experiments involving a representative RBF-based SAEA. Analysis of the large volume of results shows that a fixed HP typically had a significant impact on performance except for specific cases (Gaussian basis function/low dimension/high modality). Analysis also shows that dynamically optimizing the HP during the search significantly improved performance. These results indicate that a careful a-priori selection of the HP can significantly improve the SAEA effectiveness while additional gains could be achieved with dynamic HP calibration.



---

*Source: 5175941-2022-12-20.xml*