# An Exploratory Landscape Analysis-Based Benchmark Suite
## 4 Benchmark Suite Proposal
### Selecting a Benchmark Suite  
As mentioned above, a SOM can be clustered based on the distances between the codebook vectors. The U-matrix of a SOM represents how similar the codebook vectors are to one another and thus can be used to visualize the clustering structures of the SOM. To find a minimal set of functions to be representative of the landscape characteristics, one representative benchmark function is selected from each cluster. Some suggestions are presented on how to select a suitable representative. However, note that many combinations of benchmark functions can be chosen - resulting in many possible benchmark suites. This is congruous since the training patterns presented to the SOM are the vectors of ELA measures for the benchmark functions and hence functions that have similar landscape characteristics will be clustered together.  
The results of the clustering allow for the benchmark suite to be used in situations such as competitions at conferences, or in creation of automated algorithm selection oracles, such as hyper-heuristics. Two sets of benchmark functions can be chosen as the training suite and the test suite, and the generalization abilities of algorithms can be observed. This will prevent overfitting algorithms on common benchmark functions and allow for better applicability for both unseen and real-world problems.  
Figure 6: Coverage of the problem space for each of the benchmark suites. Illustrated by highlighting in red which clusters contain functions from the particular benchmark suite.  
To select a benchmark function from a cluster, the following criteria are used:
* Functions from the miscellaneous group are preferable, as they do not require additional information such as rotation matrices and shift vectors, which is the case with the CEC and BBOB benchmark suites.
* Functions from the BBOB benchmark suite are preferred over functions from CEC benchmark suites, as there is a large amount of information, such as algorithm performance, for the BBOB benchmark suite.  
Using the above criteria, the proposed benchmark suite is composed of the following functions:  
* Schwefel 1 [7]
* Ripple 25 [7]
* Exponential [7]
* Needle Eye [9]
* Step Function N. 3 [7]
* Generalized Giunta [7]
* Generalized Paviani [7]
* Brown [7]
* Cosine Mixture [7]
* Mishra 7 [7]
* Mishra 1 [7]
* Generalized Price 2 [7]  
Generalized Egg Crate [7]
* Rosenbrock [7]
* Pinter 2 [7]
* Qing [7]
* BBOB FID 2 IID 1 [6]
* BBOB FID 6 IID 1 [6]
* BBOB FID 16 IID 1 [6]
* BBOB FID 17 IID 2 [6]
* Generalized Drop-Wave [21]
* Bonyadi-Michalewicz [21]
* Discus [21]
* Elliptic [21]Figure 7: Comparison of the distributions of the BBOB benchmark suite and the proposed benchmark suite for all of the ELA measures used as input to the SOM.  
Figure 8: Comparison of the distributions of the BBOB benchmark suite and the proposed benchmark suite for all of the ELA measures used as input to the SOM (continued).  
Figure 9: Comparison of the distributions of the BBOB benchmark suite and the proposed benchmark suite for all of the ELA measures used as input to the SOM (continued).