# Sampling Effects on Algorithm Selection for Continuous Black-Box Optimization
## 6 Results
### Performance of the Selector
The shaded region indicates the theoretical area of performance, with the upper bound representing the ICARUS oracle and the lower bound random selection. Although it is unfeasible to match the performance of the oracle, due to the cost of extracting the ELA measures and a \(BCSR\) less than 100%, the figure shows the performance improvement against random selection. During HOIO, the ICARUS selector surpasses random selection at \(106f_{\rm evals}/D\) with \(n/D=100\), representing 5.9% of the problems; and at \(1349f_{\rm evals}/D\) with \(n/D=1000\), representing 31.2% of the problems. During HOPO, the ICARUS selector surpasses random selection at \(168f_{\rm evals}/D\) with \(n/D=100\), representing 9.9% of the problems; and at \(1620f_{\rm evals}/D\) with \(n/D=1000\), representing 40.3% of the problems. \begin{table}
\begin{tabular}{l l r r|r r r r r} \hline \hline  & \(\frac{n}{D}\) & \(\overline{T}\) & **95\% CI \(\widehat{T}\)** & \(SR\) & \(BCSR\) & \(WCSR\) & \(TFR\) & \(\rho_{RB}\) \\ \hline \((1+2)^{m}_{s}\)CMA-ES & & 2.852 & \([0.781,5.159]\) & 66.7\% & & & & 1.511 \\ BFGS & & 2.633 & \([0.349,4.648]\) & 43.8\% & & & & 1.074 \\ BIPOP-CMA-ES & & 3.397 & \([1.107,6.411]\) & **95.8\%** & & & & **1.823** \\ LSfminbnd & & 3.092 & \([1.105,5.172]\) & 27.1\% & & & & 0.566 \\ LSstep & & 3.306 & \([2.085,5.169]\) & 30.2\% & & & & 0.591 \\ Nelder-Mead & & **2.765** & \([0.694,5.455]\) & 59.4\% & & & & 1.388 \\ \hline  & 100 & **3.287** & \([2.011,6.409]\) & 99.7\% & 98.6\% & 0.5\% & 0.3\% & **1.960** \\  & 178 & 3.341 & \([2.257,6.409]\) & 99.6\% & 98.9\% & 0.3\% & 0.3\% & 1.926 \\  & 316 & 3.417 & \([2.503,6.410]\) & 99.7\% & 98.9\% & 0.4\% & **0.2\%** & 1.885 \\ HOIO & 562 & 3.509 & \([2.752,6.410]\) & **99.8\%** & **99.1\%** & **0.3\%** & 0.2\% & 1.837 \\  & 1000 & 3.622 & \([3.001,6.410]\) & 99.7\% & 99.1\% & 0.4\% & 0.3\% & 1.779 \\ \cline{2-10}  & ORACLE & 3.092 & \([0.410,6.410]\) & 100.0\% & 100.0\% & 0.0\% & 0.0\% & 2.090 \\ ICARUS & RAND & 3.178 & \([0.737,5.529]\) & 56.5\% & 33.3\% & 33.3\% & 18.5\% & 1.148 \\ \cline{2-10}  & 100 & **3.269** & \([2.025,6.409]\) & 93.6\% & 60.6\% & **4.2\%** & 2.0\% & **1.851** \\  & 178 & 3.293 & \([2.279,5.529]\) & 92.5\% & 55.9\% & 4.3\% & **0.7\%** & 1.814 \\  & 316 & 3.371 & \([2.508,5.529]\) & 93.8\% & **61.1\%** & 4.7\% & 1.9\% & 1.798 \\ HOPO & 562 & 3.510 & \([2.755,6.410]\) & **95.4\%** & 60.9\% & 5.9\% & 1.3\% & 1.756 \\  & 1000 & 3.618 & \([3.020,6.410]\) & 92.7\% & 57.4\% & 8.1\% & 2.3\% & 1.656 \\ \cline{2-10}  & ORACLE & 3.025 & \([0.770,6.410]\) & 100.0\% & 100.0\% & 0.0\% & 0.0\% & 2.136 \\  & RAND & 3.093 & \([0.847,5.468]\) & 63.2\% & 25.9\% & 26.2\% & 15.6\% & 1.320 \\ \hline  & 100 & **3.264** & \([2.011,6.409]\) & 99.4\% & 98.3\% & 0.7\% & 0.5\% & **1.968** \\  & 178 & 3.321 & \([2.257,6.409]\) & 99.5\% & 98.8\% & 0.6\% & 0.4\% & 1.936 \\ HOIO & 316 & 3.401 & \([2.503,6.410]\) & 99.5\% & 98.8\% & 0.5\% & 0.4\% & 1.891 \\  & 562 & 3.495 & \([2.752,6.410]\) & 99.7\% & **99.1\%** & **0.5\%** & **0.3\%** & 1.843 \\  & 1000 & 3.611 & \([3.001,6.410]\) & **99.7\%** & 99.0\% & 0.5\% & 0.3\% & 1.784 \\ BBLL & & 100 & **3.228** & \([2.052,5.402]\) & 87.7\% & 59.7\% & 10.2\% & 6.7\% & 1.755 \\  & 178 & 3.310 & \([2.281,5.402]\) & 90.0\% & 60.2\% & 9.6\% & 4.7\% & **1.757** \\ HOPO & 316 & 3.353 & \([2.517,5.76]\) & 88.5\% & 59.8\% & 9.6\% & 5.7\% & 1.707 \\  & 562 & 3.435 & \([2.759,5.177]\) & 88.5\% & 58.6\% & **8.2\%** & 5.4\% & 1.665 \\  & 1000 & 3.571 & \([3.024,5.436]\) & **90.5\%** & **61.2\%** & 10.5\% & **4.6\%** & 1.638 \\ \cline{2-10}  & ORACLE & 3.040 & \([0.410,6.410]\) & 100.0\% & 100.0\% & 0.0\% & 0.0\% & 2.126 \\ RAND & 3.170 & \([0.737,5.529]\) & 49.1\% & 25.0\% & 25.0\% & 14.3\% & 1.000 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance of each algorithm and the selector, in terms of \(\overline{T}\), 95% confidence interval of \(\widehat{T}\), \(SR\) and \(\rho_{\rm RB}\), which is the normalized performance score against random selection using the BBLL portfolio.