# An Exploratory Landscape Analysis-Based Benchmark Suite
## 2 Background
### Landscape Analysis  
In the context of the algorithm selection framework defined by [PERSON], the characteristics space can be defined using landscape analysis measures. Landscape analysis encompasses both fitness landscape analysis (FLA) [16] and exploratory landscape analysis (ELA) [17]. An individual FLA measure generally defines a single high-level characteristic of an optimization problem, whereas an individual FLA measure is generally associated with several high-level properties [17]. [PERSON] et al. [22] defined eight high-level properties for optimization problems, namely  
* Multi-modality, which refers to the number of local optima in the fitness landscape. * Global structure, which refers to the underlying structure of a fitness landscape when removing local optima. * Separability, which describes if an objective function can be decomposed into subproblems in which all the variables in each subproblem are independent of the variables in the other subproblems. * Variable scaling, which describes the effect that scale has on the behavior of algorithms in different dimensions. * Search space homogeneity, which describes the phase transitions between different areas of the fitness landscape, i.e., how the properties of the fitness landscape vary in different areas of the search space. * Basin size homogeneity, which describes the differences in the sizes of the basins of attractions. * Global to local optima contrast, which describes the difference in fitness values between local and global optima. * Plateaus, which refers to areas of a fitness landscape in which the fitness values do not fluctuate significantly. An individual FLA measure is interpretable in the sense that it can, on its own, describe a high-level property of a fitness landscape, as listed above. Individual ELA measures generally have less interpretability, and are designed to be used collectively and used as input for machine learning models. The distinction between ELA and FLA is not clear-cut, and often measures that were originally described as FLA measures are used in ELA. [PERSON] et al. [17] refer to groups of related ELA measures as feature sets. The flacco R package [23] provides an interface for calculating many landscape analysis measures in the literature. In this study, the following ELA feature sets from flacco are investigated:  
* Dispersion (_disp_): Defined by [PERSON] and [PERSON] [24], these measures describe the global structure of the objective function. * Information content (_ic_): Defined by [PERSON] et al. [25], these measures calculate the differences between points in the sampled fitness values to determine the ruggedness of the fitness landscape. * Level-set (_ela_level_): Defined by [PERSON] et al. [17], these measures split the initial sample into two groups, and then the performance of multiple classification algorithms is measured. * Meta-model (_ela_meta_): Defined by [PERSON] et al. [17], these measures determine how well the sampled fitness values fit linear and quadratic models. * Nearest better clustering (_nbc_): Defined by [PERSON] et al. [26], these measures calculate various statistics based on the comparison of the distances between the sample points' nearest neighbor and their nearest neighbor that has a better fitness value. * Principal component analysis measures (_pca_): Defined by [PERSON] and [PERSON] [23], these measures perform principal component analysis on the sampled values in both the decision variable and fitness spaces. * y-distribution features (_ela_distr_): Defined by [PERSON] et al. [17], these measures describe the distribution of the fitness values obtained by the sampling algorithm. These feature sets are chosen since they do not require any further objective function evaluations other than that of the initial sample. This simplifies the procedure needed to determine the point of robustness for the measures since the only variable with regards to the sampling is the size of the sample.