# An Exploratory Landscape Analysis-Based Benchmark Suite
## 4 Benchmark Suite Proposal
### Self-Organizing Feature Map  
The Kohonen R package [43] was used to generate the SOM models in this study. The set of measures listed in the previous section are used as input for the SOM. These measures are calculated for 30 independent runs on the 340 benchmark functions described in Section 3. This results in a dataset of 10,200 training patterns. As mentioned in Section 3, this study focuses on 10-dimensional benchmark functions. To prevent any of the ELA measures from dominating the training of the SOM, the input is normalized to the range \([0,1]\). The stochastic training rule [32] was used to train the SOM. For the stochastic training rule, the learning rate decreases linearly from 0.1 to 0.01 as a function of the training epoch. The SOM was trained for 100,000 epochs. The positions of the codebook vectors in the grid are updated using a Gaussian neighborhood function. Euclidean distances are used when determining the winning codebook vector for the SOM. Several SOM models with differing grid sizes and dimensions were trained on the data. The U-matrix of the 75 by 75 grid provided the best visualization of the clustering structures [33]. Therefore, this map was chosen for the analysis. Figure 5a contains the U-matrix for the trained SOM. From this figure, it can be seen that the grid size is sufficiently large to see the fine grain clustering structures. This figure suggests that there are many natural clusterings. However, there are also several codebook vectors which are topologically similar. This is highlighted by the large grouping of nodes in the bottom left and right corners of the U-matrix. The benchmark suites from both the BBOB and CEC competitions typically have between 20 and 30 benchmark functions included. This is a reasonable number of benchmark problems, as this many functions are likely to provide a good representation of the problem space, provided that the functions are not similar. This number of functions is also not large enough for the computational costs to be excessive for the design and analysis of algorithms. The NbClust R package [44] was used to cluster the benchmark functions, as well as to validate the number of clusters. To cluster the codebook vectors in the SOM, the [PERSON] hierarchical clustering algorithm [45] was used. [PERSON]'s method works by minimizing the total within-cluster variance. The dissimilarity between the codebook vectors is calculated using Euclidean distance. Once the codebook vectors of the SOM were clustered, the validity of the clustering is quantified based on the Davies-Bouldin index [46]. The Davies-Bouldin index measures the cohesion of a cluster. A lower Davies-Bouldin score implies that the clusters are compact and well separated. Table 2 contains the corresponding Davies-Bouldin scores. From this table, it is observed that the optimal number of clusters is 24. Since all 30 independent runs of the ELA measures are used as input to the SOM, the quality of the clustering can be asserted by examining to which cluster each of the \begin{table}
\begin{tabular}{c c} \hline \hline
**Number of Clusters** & **Davies–Bouldin Score** \\ \hline
20 & 1.3568 \\
21 & 1.3758 \\
22 & 1.3566 \\
23 & 1.3481 \\
**24** & **1.3228** \\
25 & 1.3593 \\
26 & 1.3350 \\
27 & 1.3412 \\
28 & 1.3726 \\
29 & 1.3594 \\
30 & 1.3498 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Davies–Bouldin scores for differing number of clusters of the codebook vectors of the SOM, based on [PERSON]’s clustering method. independent runs are assigned. Ideally, all independent runs for a particular function should appear in the same cluster. For the clustering in Figure 5b, six of the 340 benchmark functions have independent runs assigned to three different clusters. However, upon inspection, these functions that have not been uniquely assigned to the clusters have been assigned to clusters that are topological neighbors in the SOM grid. This is likely since there are a different benchmark functions that have similar ELA measures, which affects the clustering of the codebook vectors.