# An Exploratory Landscape Analysis-Based Benchmark Suite
## 4 Benchmark Suite Proposal
### Preprocessing  
To ensure the quality of the SOM, some preprocessing steps are needed:  
1. Determine the sample size used to sample ELA measures. This was determined in the previous section as \(800\times D\). 2. Identify ELA measures that do not provide useful information, in other words, measures that are not expressive [27]. 3. Identify ELA measures that are highly correlated to prevent multicollinearity. As defined in [27], an ELA measure is defined as expressive if it can distinguish different benchmark problems. If the variance of an ELA measure is low, it implies that the measure is not expressive. When analyzing the distribution of the ELA measures, it is noted that **pca.expl_var.cov_x** and **pca.expl_var.cor_x** generate the same value across all investigated benchmark functions. All investigated benchmark functions generated only two different values for **pca.expl_var.cor_init**. Furthermore, the variance of both **pca.expl_var.PC1.cov_x** and **pca.expl_var.PC1.cor_x** are both 0.000012. Therefore, the above ELA measures are removed from all further analysis in this study, as they are unlikely to provide useful information to the SOM. It is noted that these non-expressive measures all have peaks, in their distribution of point of robustness, at a sample size of \(50\times D\), as illustrated in Figure 1.  
Pearson's correlation coefficient can determine if a linear correlation between two ELA measures exists. However, a non-linear correlation relationship may exist between measures. In this case, [PERSON]'s correlation will not be able to detect the dependency. The maximal information coefficient (MIC) [41] is a measure that captures the strength of the association between two random variables. MIC has several useful properties:  
* It produces values in between 0 and 1, with 0 indicating that there is no association between the two variables, and 1 indicating that the variables have a perfect noiseless relationship. This allows for easy interpretation of the MIC score. * It captures a wide range of relationships, both functional and non-functional. * It is symmetric, which implies that \(MIC(X,Y)=MIC(Y,X)\). There is no rule of thumb, as with [PERSON]'s correlation, to determine thresholds of when associations are deemed to be strong. To be conservative in removing ELA measures, a threshold of 0.9 is selected. With a MIC score of larger than 0.9, an ELA measure can be said to explain a second ELA measure well, and therefore the second ELA measure is redundant. The minerva R package [42] is used to calculate the MIC. To calculate the MIC, the median value for each ELA measure of all the functions used in Section 3 is taken. Then, the MIC is calculated for each pair of ELA measures. To reduce the number of redundant ELA measures, groups of highly associated measures are found. This is done by creating a graph based on the MIC scores. Each ELA measure is represented as a node in this graph. If the MIC score is larger than 0.9, then an edge is added between the two ELA measures. Each group of correlated measures will then form an independent set. A representative ELA measure can be chosen from each group, and the remaining measures are not used as input for the SOM. The above procedure resulted in six groups of highly associated ELA measures. The graph generated by the above procedure is shown in Figure 4. Figure 11 in Appendix A indicates the MIC scores between all investigated ELA measures. From each group, the representative ELA measure is chosen as the measure which has the smallest point of robustness. As noted earlier, in this study, the 95 th percentile is chosen to summarize the point of robustness across functions. If there are ties in the smallest point of robustness, then the tie is broken arbitrarily. This ensures that the final set of ELA measures used as input to the SOM provides characteristics that are as reliable as possible.