# An Exploratory Landscape Analysis-Based Benchmark Suite
## 3 Robustness of Exploratory Landscape Analysis Measures
### Determining Robustness
For this purpose, Levene trend tests ([PERSON], 1980) can be used to determine if there is a monotonic increasing or decreasing trend in a group of variances. As described in ([PERSON], 1980), such a hypothesis test can be set up as follows:  
\[H_{0}:\sigma_{1}^{2}=\sigma_{2}^{2}=\ldots=\sigma_{k}^{2} \tag{5}\]  
\[H_{1}:\sigma_{1}^{2}\geq\sigma_{2}^{2}\geq\ldots\geq\sigma_{k}^{2} \tag{6}\]  
Then, all observations in a group \(i\) are assigned a score \(w_{i}\), for each group \(i=1,\ldots,k\). Now, regress the transformed data, \(d_{ij}\), on \(w_{i}\) and consider the regression slope  
\[\hat{\beta}=\frac{\sum_{i=1}^{k}n_{i}(w_{i}-\bar{w})(\bar{d}_{i.}-\bar{d}_{.} )}{\sum_{i=1}^{k}n_{i}(w_{i}-\bar{w})^{2}} \tag{7}\]  
where  
\[\bar{w}=\sum_{i=1}^{k}w_{i} \tag{8}\]  
Under the null hypothesis, \(\hat{\beta}=0\) and the test statistic follows a \(t\)-distribution with \((N-1)\) degrees of freedom, where \(N\) is the total number of observations from all groups. Scores can be assigned as either linear or non-linear functions, which respectively allows testing for linear or non-linear trends in the variances. In this study, linear scores are investigated. That is, \(w_{i}=1\forall i\). The lawstat package ([PERSON], 1980) in R is used to perform the Levene trend test. Now, to determine at what sample size a measure \(c(f,n)\), for a particular objective function \(f\), becomes robust, the following procedure is performed:  
1. Choose the sample sizes \(s=s_{1},...,s_{M}\) to be investigated. 2. For each sample size \(s_{i}\), calculate the measure \(c(f,s_{i})\) for \(r\) independent runs. 3. Perform the Levene trend test on the above samples, for each pair of sample sizes, \(s_{i}\) and \(s_{i+1}\). In this case, there \(k=2\) groups. Obtain the test statistic and \(p\)-value. 4. For each pair of sample sizes, if the resulting \(p\)-value is less than or equal to the predefined significance level, \(\alpha\), then the null hypothesis is rejected. This implies that it is likely that there is a monotonic decrease in the variance between the sample sizes. If the \(p\)-value is greater than \(\alpha\), then the null hypothesis cannot be rejected. It is then said that there is strong evidence that the variance between requivalencyhe different sample sizes is equal. When using the procedure described above, for a particular ELA measure, there are several possibilities with regards to the number of occurrences of \(p\)-values \(<\alpha\):  
1. Zero occurrences: This implies that there is no evidence that the variance is lower for any sample size. The smallest sample size is chosen as the point of robustness since there is no decrease in variance from increasing sample size. 2. One occurrence: The first sample size after the occurrence is chosen to be the point of robustness. 3. Two or more consecutive occurrences: The first sample size after the chain of consecutive occurrences is chosen as the point of robustness. 4. Two or more non-consecutive occurrences: The first sample size after the first chain of consecutive occurrences is chosen as the point of robustness. Please note that when the null hypothesis is rejected for a pair of sample sizes, it implies that the variance of the larger sample size is statistically likely to be lower than the variance of the smaller sample size. Therefore, the larger sample size is chosen as the point of robustness. Based on the observation of [PERSON] et al. (2015), the variance of a particular ELA measure tends to zero as the sample size increases. For the case of two or more non-consecutive occurrences of statistically significant pairs, [PERSON] et al. 's observation implies that the first chain of statistically significant pairs is more likely to provide practically significant differences in variance than the second, or later, chain of statistically significant pairs. Therefore, the first sample size after the first chain of statistically significant pairs is chosen as the point of robustness.