# Sampling Effects on Algorithm Selection for Continuous Black-Box Optimization
## 6 Results  
### Performance of the Selector  
We now provide an answer to the questions formulated above. The results show that a larger value of \(n\) does not translate to higher selection accuracy, while greatly increasing the cost for easier problems. The remaining budget to find a solution decreases due to an increase of \(f_{\text{evals}}/D\) used to estimate the ELA features. Furthermore, changes in the input sample have minor effects on the performance. This implies that the selector is reliable even with noisy features, suggesting that their variance is small. Let us examine the results in detail. The performance of the best algorithm for a given problem is illustrated in Figure 5 for the ICARUS/HOPO and BBLL portfolios. We observe that BIPOP-CMA-ES is the dominant algorithm for \(\{10,20\}\) dimensions in both portfolios. The Nelder-Doerr algorithm replaces BFGS in the ICARUS/HOPO portfolio, resulting in deterioration of \(\widehat{T}\) for \(\{f_{1},f_{5}\}\). There are specialized algorithms for particular problems regardless of the number of dimensions, e.g., LSstep is the best for \(\{f_{3},f_{4}\}\). Table 3 shows the performance of the individual algorithms, the selectors during HOIO and HOPO. In addition, the results from an oracle, i.e., a method which always selects the best algorithm without incurring any cost, and a random selector are presented. The performance is measured as \(\overline{T}\), the 95% Confidence Interval (CI) of \(\widehat{T}\), \(SR\), \(SR\), \(BCSR\), \(WCSR\), \(TFR\) and \(\rho_{\text{RB}}\). In boldface are the best values of each performance measure over each validation method and \(n/D\). The table shows that only the oracles achieve \(SR=100\%\), with the ICARUS portfolio having the best overall performance with \(\rho_{\text{RB}}=2.136\) during HOPO. The table also shows that \(\overline{T}\) is always the lowest for \(n/D=100\), given that less of the budget is used to calculate the ELA features. Highest \(SR\) is achieved with \(n/D=562\) for the ICARUS portfolios and \(n/D=1000\) for the BBLL portfolio. However, the differences with smaller \(n\) may not justify the additional cost. For example, a \(SR\) of \(\{\text{HOIO}:99.7\%,\text{HOPO}:93.6\%\}\) and a \(WCSR\) of \(\{\text{HOIO}:0.5\%,\text{HOPO}:4.2\%\}\) is achieved with \(n/D=100\) for the ICARUS sets, a difference of \(\{\text{HOIO}:-0.1\%,\text{HOPO}:-1.7\%\}\) in \(SR\) and \(\{\text{HOIO}:0.1\%,\text{HOPO}:-1.6\%\}\) in \(WCSR\) for \(n/D=562\). Compared with total random selection, the selectors' \(WCSR\) and \(TFR\) are at least one order of magnitude smaller. Table 3 also shows on average a decrease on \(BCSR\) of \(\approx 40\%\), an increase of \(WCSR\) and \(TFR\) of \(\approx\)7% and \(\approx\)3% respectively; although the results for the ICARUS sets are below average for \(WCSR\) and \(TFR\), indicating better performance. Furthermore, \(SR\) is always above 90% for the ICARUS sets, while \(SR\) falls below 90% during HOPO for the BBLL set. Only BIPOP-CMA-ES with \(\rho_{\text{RB}}=1.823\) can match the overall performance of a selector. Figure 5: Performance of the best algorithm in terms of \(\hat{T}\) for each problem. The lines represent the dimension of the problem. The Nelder–Doerr algorithm replaces BFGS in the ICARUS/HOPO portfolio in most of the problems. Figure 6 illustrate the \(SR\) against \(\widehat{T}\) for the complete benchmark set. Figure 6a shows the results of individual algorithms, while Figure 6b shows the performance of such oracle as solid lines and the random selector as dashed lines for the ICARUS/HOIO, ICARUS/HOPO, and BBLL portfolios. As expected from Figure 5a, the ICARUS/HOPO oracle has a degradation in performance in the easier 10% of the problems. On the top 10% of the problems the performance of the three oracles is equivalent. The ICARUS/HOPO oracle has a small improvement on performance between 10% and 90% of the problems. Figure 6c,d illustrate the performance of the selectors for \(n/D=\{100,1000\}\).