# An Exploratory Landscape Analysis-Based Benchmark Suite
## 3 Robustness of Exploratory Landscape Analysis Measures
### Determining Robustness  
The choice of the sample size for calculating ELA measures presents a trade-off between accuracy and computational costs. For smaller sample sizes, the computational cost will be low, but the accuracy of the resulting ELA measures is poor. For larger sample sizes, the computational costs will be high, and the accuracy of the resulting ELA measure will increase. As noted by [PERSON] et al. [35], an ELA measure \(c(f,n)\) which is calculated on an objective function \(f\) from a sample size \(n\) is a random variable. Therefore, \(c(f,n)\) has a probability distribution whose variance, \(\sigma^{2}\), should converge to zero when \(n\) tends to infinity, otherwise \(\sigma^{2}\) is dependent on \(f\) and \(n\). Several independent runs of a measure \(c(f,n)\) can be conducted to approximate the probability distribution. When the variance, \(\sigma^{2}\), is small then \(c(f,n)\) is said to be robust. Defining when the variance is small results in extra hyperparameters when using ELA, as a threshold needs to be defined for each ELA measure. Rather than defining an absolute threshold, this study makes use of a procedure that determines when the variance becomes small enough, relative to increasing sample sizes. This, coupled with the fact that the variance tends to zero as the sample size increases, allows one to determine the sample size needed to provide robust ELA measures. To determine the sample size needed to produce a robust measure, non-parametric statistical tests are needed, since the \(c(f,n)\) distributions are unknown and are unlikely to follow the normal distribution. In the literature, there are many hypothesis tests for equality of variance, also called homogeneity of variance, the most common being the \(F\)-test, which tests the hypothesis that two normal distributions have the same variance. The Levene test [36] is used to determine whether \(k\) samples have equal variances, and can be used as an alternative to the \(F\)-test when the population data does not follow the normal distribution. Consider \(k\) random samples, where the \(i\)-th sample has observations \(x_{i1},x_{i2},\ldots,x_{in_{i}}\). [PERSON] considers the absolute differences between each observation and its corresponding group mean, i.e., \(d_{ij}=|x_{ij}-\bar{x}_{i}|\), \(i=1,2,\ldots,k\), \(j=1,2,\ldots,n_{i}\), where \(n_{i}\) is the number of observations in the \(i\)-th group, and \(\bar{x}_{i. }\) is the sample mean for the \(i\)-th group. Then, the Levene test is defined as:  
\[H_{0}:\sigma_{1}^{2}=\sigma_{2}^{2}=\ldots=\sigma_{k}^{2} \tag{1}\]  
\[H_{1}:\sigma_{i}^{2}\
eq\sigma_{j}^{2}\text{for at least one pair}(i,j) \tag{2}\]  
The Levene test statistic is then defined as:  
\[L=\frac{N-k}{k-1}\frac{\sum_{i=1}^{k}n_{i}(\bar{d}_{i.}-\bar{d}_{.})^{2}}{\sum _{i=1}^{k}\sum_{j=1}^{n_{i}}(d_{ij}-\bar{d}_{i.})^{2}} \tag{3}\]  
where  
\[\bar{d}_{i.}=\sum_{j=1}^{n_{j}}\frac{d_{ij}}{n_{j}}\hskip 28.452756 pt\bar{d}_{..}=\sum_{i=1}^{k}\sum_{j=1}^{n_{i}}\frac{d_{ij}}{N}\hskip 28.452756 ptN=\sum_{i=1}^{ k}n_{i} \tag{4}\]  
[PERSON] transforms the population data by considering the absolute differences between each observation and its corresponding group mean, and therefore \(d_{ij}=|x_{ij}-\bar{x}_{i. }|\), \(i=1,2,\ldots,k\), \(j=1,2,\ldots,n_{i}\), and \(\bar{x}_{i. }\) is the sample mean for the \(i\)-th group in the above equations. [PERSON] and [PERSON] (1979) proposed a modification to the Levene test which provides more robust results, in which the absolute differences between each observation and its corresponding group median is calculated. That is, \(d_{ij}=|x_{ij}-\bar{x}_{i. }|\), where \(\bar{x}_{i. }\) is the median of the \(i\)-th group. However, it is only of interest whether the variance of the measures decreases as the sample size increases, and not if variances between sample sizes are equal or not. This is because a two-sided hypothesis test for the variance does not indicate if the variance of two samples is larger or smaller than the other.