# An Exploratory Landscape Analysis-Based Benchmark Suite
## 3 Robustness of Exploratory Landscape Analysis Measures
### Results and Discussion  
Figure 1 provides the distribution of the point of robustness for each of the investigated ELA measures on all the investigated benchmark functions. Figure 1 indicates that most of the ELA measures have two dominating points of robustness, with peaks at sample sizes \(50\times D\) and \(200\times D\). The features that provide the lowest point of robustness are **pca.expl_var.cov_x**, **pca.expl_var.cov_init**, **pca.expl_var.cov_x**, **pca.expl_var.cor_init**, **ela_meta.lin_simple.coef.max_by_min**, and **ela_distr.number_of peaks**, with large peaks at sample size \(50\times D\). The measures which have platykurtic distributions, i.e., wide and flat distributions, are the dispersion and level-set feature sets. These platykurtic distributions indicate that the number of sample sizes needed to produce robust ELA measures often differs for different benchmark functions. Figure 1 also shows that measures in a particular feature set tend to have the same distribution for the point of robustness. This observation is most prominent for the dispersion feature set. These similar distributions may indicate that measures within a feature set are highly correlated. Figure 2 contains the plots of the distribution of the point of robustness for all investigated benchmark suites. Figure 3 contains the plots of the distribution of the point of robustness for the combination of all investigated benchmark functions. These two figures combine the point of robustness across all investigated ELA measures for a particular benchmark suite. Thus, in Figure 1 the cynosure is the different ELA measures, and in Figures 2 and 3 the cynosure are the different benchmark suites. Figure 2 indicates that the distribution of the point of robustness is roughly the same for the BBOB and CEC benchmark suites. These distributions appear to follow a negative binomial distribution, and this is validated with a goodness-of-fit test. Figure 2f contrasts the robustness results of the BBOB and CEC benchmark suites, and indicates that the miscellaneous functions generally have a point of robustness at \(50\times D\). It is hypothesized that the oscillation functions used in the BBOB and CEC benchmark suites induces more phase transitions in the fitness landscapes, whereas the collection of miscellaneous functions are not oscillated. However, further research is required to determine the true cause. As seen in Figure 3, most benchmark functions provide robust ELA measures at sample sizes of \(50\times D\), \(100\times D\) and \(200\times D\). Since the different benchmark suites are defined for various search space dimensionalities, it is interesting to note that the distributions of the point of robustness do not change significantly between the benchmark suites. This implies that the improved Latin hypercube sampling algorithm is a good choice to generate samples for ELA, and is likely to provide good coverage of the function space, regardless of the size of the search space. Figure 1: Distribution of the point of robustness for each of the investigated ELA measures across all benchmark functions in \(D=10\) dimensions. As noted above, an ELA measure \(c(f,n)\) depends on both the function \(f\) and the sample size \(n\). Since the procedure to determine the point of robustness for an ELA  
Figure 3: Plot of the combined distribution of the point of robustness of all investigate benchmark functions in \(D=10\) dimensions. Figure 2: Plots of the distribution of the point of robustness for each of the investigated benchmark suites in \(D=10\) dimensions. measure holds the function constant and varies the sample size, a summary statistic is needed to generalize the robustness for a particular ELA measure across a collection of functions. For this purpose, percentiles may be used. A percentile describes the percentage of observations that fall below a particular value. For example, the median is the 50 th percentile. It implies that 50% of the observations in a data set lie above the median.