# An Exploratory Landscape Analysis-Based Benchmark Suite
## 3 Robustness of Exploratory Landscape Analysis Measures
### Results and Discussion
Table 1 contains the percentiles for the point of robustness over all the investigated benchmark functions. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**ELA Measure** & **10\%** & **25\%** & **50\%** & **75\%** & **90\%** & **95\%** & **99\%** & **100\%** \\ \hline disp.diff\_mean\_02 & 50 & 50 & 100 & 200 & 300 & 300 & 700 & 1000 \\ disp.diff\_mean\_05 & 50 & 50 & 100 & 200 & 300 & 400 & 800 & 1000 \\ disp.diff\_mean\_10 & 50 & 50 & 100 & 200 & 400 & 700 & 1000 & 1000 \\ disp.diff\_mean\_25 & 50 & 50 & 100 & 200 & 300 & 500 & 700 & 1000 \\ disp.diff\_median\_02 & 50 & 50 & 100 & 200 & 200 & 300 & 500 & 600 \\ disp.diff\_median\_05 & 50 & 50 & 100 & 200 & 200 & 400 & 800 & 1000 \\ disp.diff\_median\_10 & 50 & 50 & 100 & 200 & 300 & 600 & 900 & 1000 \\ disp.diff\_median\_25 & 50 & 50 & 100 & 200 & 300 & 400 & 900 & 1000 \\ disp.ratio\_mean\_02 & 50 & 50 & 100 & 200 & 300 & 300 & 700 & 1000 \\ disp.ratio\_mean\_05 & 50 & 50 & 100 & 200 & 300 & 400 & 700 & 1000 \\ disp.ratio\_mean\_10 & 50 & 50 & 100 & 200 & 400 & 700 & 1000 & 1000 \\ disp.ratio\_mean\_25 & 50 & 50 & 100 & 200 & 400 & 400 & 600 & 900 \\ disp.ratio\_median\_02 & 50 & 50 & 100 & 200 & 200 & 300 & 500 & 900 \\ disp.ratio\_median\_05 & 50 & 50 & 100 & 200 & 200 & 400 & 800 & 1000 \\ disp.ratio\_median\_10 & 50 & 50 & 100 & 200 & 300 & 500 & 900 & 1000 \\ disp.ratio\_median\_25 & 50 & 50 & 100 & 200 & 300 & 400 & 900 & 1000 \\ ela\_distr.kurtosis & 50 & 50 & 100 & 200 & 500 & 700 & 1000 & 1000 \\ ela\_distr.number\_of\_peaks & 50 & 50 & 50 & 50 & 500 & 700 & 900 & 1000 \\ ela\_distr.skewness & 50 & 50 & 100 & 200 & 500 & 800 & 1000 & 1000 \\ ela\_level.lda\_mda\_10 & 50 & 50 & 100 & 200 & 200 & 300 & 400 & 700 \\ ela\_level.lda\_mda\_25 & 50 & 50 & 100 & 200 & 400 & 600 & 1000 & 1000 \\ ela\_level.lda\_mda\_50 & 50 & 50 & 100 & 200 & 300 & 500 & 700 & 1000 \\ ela\_level.lda\_qda\_10 & 50 & 50 & 100 & 200 & 400 & 600 & 800 & 1000 \\ ela\_level.lda\_qda\_25 & 50 & 50 & 100 & 300 & 500 & 700 & 900 & 1000 \\ ela\_level.lda\_qda\_50 & 50 & 50 & 100 & 200 & 500 & 700 & 900 & 1000 \\ ela\_level.mmc\_lda\_10 & 50 & 50 & 100 & 200 & 200 & 400 & 700 & 1000 \\ ela\_level.mmc\_lda\_25 & 50 & 50 & 100 & 200 & 300 & 400 & 800 & 800 \\ ela\_level.mmc\_lda\_50 & 50 & 50 & 100 & 200 & 300 & 400 & 700 & 900 \\ ela\_level.mmc\_mda\_10 & 50 & 50 & 100 & 200 & 300 & 300 & 600 & 1000 \\ ela\_level.mmc\_mda\_25 & 50 & 50 & 100 & 200 & 300 & 400 & 900 & 1000 \\ ela\_level.mmc\_dda\_50 & 50 & 50 & 100 & 200 & 200 & 300 & 600 & 800 \\ ela\_level.mmc\_qda\_10 & 50 & 50 & 100 & 200 & 300 & 300 & 700 & 1000 \\ ela\_level.mmc\_qda\_25 & 50 & 50 & 100 & 200 & 400 & 500 & 800 & 1000 \\ ela\_level.mmc\_qda\_50 & 50 & 50 & 100 & 200 & 200 & 300 & 400 & 600 \\ ela\_level.qda\_mda\_10 & 50 & 50 & 100 & 200 & 300 & 400 & 700 & 900 \\ ela\_level.qda\_mda\_25 & 50 & 50 & 100 & 200 & 400 & 500 & 900 & 1000 \\ ela\_level.qda\_mda\_50 & 50 & 50 & 100 & 200 & 400 & 700 & 900 & 1000 \\ ela\_meta.lin\_simple.adj\_r2 & 50 & 50 & 100 & 200 & 300 & 500 & 900 & 1000 \\ ela\_meta.lin\_simple.coef.max & 50 & 50 & 100 & 200 & 300 & 400 & 900 & 1000 \\ ela\_meta.lin\_simple.coef.max\_by\_min & 50 & 50 & 50 & 50 & 600 & 700 & 900 & 1000 \\ ela\_meta.lin\_simple.coef.min & 50 & 50 & 100 & 200 & 400 & 500 & 800 & 900 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Percentiles for the point of robustness for all the investigated benchmark functions. The entries in the table represent sample sizes multiplied by \(D\), where \(D=10\). To select the appropriate percentile, the practitioner should consider the sensitivity and consequences that the choice of the sample size will have on the later stages of the application of landscape analysis. As noted earlier, there is a trade-off between accuracy and computational costs when calculating ELA measures. For example, if ELA measures are used in automated algorithm selection, a practitioner may be satisfied with lower accuracy to keep computational costs down.