# Sampling Effects on Algorithm Selection for Continuous Black-Box Optimization
## 6 Results
### Sampling Effects on Selection  
Table 5 shows the 95% CI of \(\{\overline{T},SR,BCSR,WCSR,TFR\}\). The sub-index represents estimation bias, defined as the difference between the bootstrap mean and the original mean, which was presented in Table 3. A large bias implies that the estimated CI is less reliable. In boldface are those intervals whose range is the smallest for a given validation. The lowest bias is also in boldface. This table shows the smallest range is achieved with \(n/D=1000\), implying more stable predictions are made the selector with a large sample size. This also implies more stable ELA features. However, the differences are relatively minor. For example, the difference between the upper and lower bounds of \(\overline{T}\) represent \(65f_{\text{evals}}/D\) for \(n/D=100\), \(63f_{\text{evals}}/D\) for \(n/D=316\), and \(37f_{\text{evals}}/D\) for \(n/D=1000\). For \(SR\), the difference is 0.8% for \(n/D=100\), 0.6% for \(n/D=316\), and 0.4% for \(n/D=1000\). Hence, the ELA features are sufficiently stable with a small \(n\), to achieve reliable selection by the selector. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**HOIO**} & \multicolumn{4}{c}{**HOPO**} \\ \cline{2-10}  & \(\mathbf{SR}\) & \(\mathbf{BCSR}\) & \(\mathbf{WCSR}\) & \(\mathbf{TFR}\) & \(\mathbf{SR}\) & \(\mathbf{BCSR}\) & \(\mathbf{WCSR}\) & \(\mathbf{TFR}\) \\ \hline \(f_{1}\) & 100.0\% & **63.0\%** & 9.2\% & 0.0\% & 100.0\% & **45.5\%** & **15.0\%** & 0.0\% \\ \(f_{2}\) & 100.0\% & **69.8\%** & **13.7\%** & 0.0\% & 100.0\% & **55.0\%** & **19.5\%** & 0.0\% \\ \(f_{3}\) & 99.8\% & 91.0\% & 0.2\% & 0.2\% & 100.0\% & **86.6\%** & 0.1\% & 0.1\% \\ \(f_{4}\) & 98.0\% & **89.1\%** & 1.3\% & 1.0\% & 95.2\% & **82.2\%** & 2.4\% & 2.2\% \\ \(f_{5}\) & 100.0\% & **75.5\%** & **11.1\%** & 0.0\% & 100.0\% & **67.1\%** & **18.4\%** & 0.0\% \\ \(f_{6}\) & 98.7\% & **88.8\%** & 0.1\% & 0.1\% & 98.4\% & **86.9\%** & 0.2\% & 0.2\% \\ \(f_{7}\) & 98.7\% & 95.2\% & 0.4\% & 0.4\% & 98.3\% & 92.8\% & 0.9\% & 0.9\% \\ \(f_{8}\) & 98.6\% & **74.1\%** & 1.4\% & 1.4\% & 93.0\% & **59.6\%** & 7.0\% & 7.0\% \\ \(f_{9}\) & 100.0\% & **74.9\%** & 0.0\% & 0.0\% & 100.0\% & **66.2\%** & 0.0\% & 0.0\% \\ \(f_{10}\) & 99.5\% & **83.6\%** & 0.4\% & 0.4\% & 99.5\% & **76.2\%** & 0.4\% & 0.4\% \\ \(f_{11}\) & 99.0\% & **83.8\%** & 0.8\% & 0.8\% & 99.2\% & **76.8\%** & 0.8\% & 0.8\% \\ \(f_{12}\) & 94.0\% & **81.7\%** & 2.2\% & 0.0\% & 91.6\% & **76.4\%** & 1.8\% & 0.0\% \\ \(f_{13}\) & 97.1\% & 93.9\% & 1.7\% & 1.7\% & 95.3\% & 90.5\% & 2.8\% & 2.8\% \\ \(f_{14}\) & 95.7\% & **79.9\%** & 4.3\% & 4.3\% & 93.5\% & **70.7\%** & 6.5\% & 6.5\% \\ \(f_{15}\) & 99.7\% & 94.3\% & 1.0\% & 0.0\% & 99.0\% & 92.8\% & 1.2\% & 0.2\% \\ \(f_{16}\) & 99.1\% & 94.9\% & 0.6\% & 0.6\% & 99.2\% & 95.4\% & 0.5\% & 0.5\% \\ \(f_{17}\) & 91.8\% & **86.9\%** & 1.9\% & 1.9\% & 90.2\% & **83.1\%** & 1.5\% & 1.5\% \\ \(f_{18}\) & 97.3\% & 92.7\% & 2.5\% & 2.5\% & 97.0\% & 91.1\% & 2.8\% & 2.8\% \\ \(f_{19}\) & 99.7\% & 96.5\% & 0.3\% & 0.3\% & 95.6\% & 91.3\% & 2.5\% & 2.5\% \\ \(f_{20}\) & 90.0\% & **81.9\%** & 3.2\% & 0.0\% & **85.5\%** & **74.5\%** & 3.2\% & 0.0\% \\ \(f_{21}\) & 100.0\% & **67.9\%** & 0.0\% & 0.0\% & 100.0\% & **50.3\%** & 0.0\% & 0.0\% \\ \(f_{22}\) & 100.0\% & **75.7\%** & 0.0\% & 0.0\% & 100.0\% & **67.8\%** & 0.0\% & 0.0\% \\ \(f_{23}\) & **80.8\%** & **80.7\%** & 9.0\% & 9.0\% & **66.9\%** & **66.9\%** & **16.4\%** & **16.4\%** \\ \(f_{24}\) & **76.2\%** & **71.8\%** & 9.1\% & 9.1\% & **71.5\%** & **68.0\%** & **13.9\%** & **13.9\%** \\ \hline
2 & 96.2\% & **79.6\%** & 5.0\% & 2.0\% & 95.1\% & **73.2\%** & 7.0\% & 2.6\% \\ \(5\) & 96.0\% & **83.7\%** & 2.8\% & 1.2\% & 94.2\% & **77.2\%** & 4.2\% & 2.0\% \\ \(10\) & 96.6\% & **82.0\%** & 3.3\% & 1.4\% & 95.1\% & **74.4\%** & 4.6\% & 2.5\% \\ \(20\) & 96.8\% & **86.0\%** & 1.3\% & 1.0\% & 93.8\% & **77.4\%** & 3.8\% & 2.7\% \\ \hline \hline \end{tabular}
\end{table}