# Beyond Landscape Analysis: DynamoRep Features For Capturing Algorithm-Problem Interaction In Single-Objective Continuous Optimization

Gjorgjina Cenikj, Gašper Petelin, Carola Doerr, Peter Korošec, Tome Eftimov

<figure><img src="image_1.png" /></figure>

## To cite this version:

Gjorgjina Cenikj, Gašper Petelin, Carola Doerr, Peter Korošec, Tome Eftimov. Beyond Landscape Analysis: DynamoRep Features For Capturing Algorithm-Problem Interaction In Single-Objective Continuous Optimization. Evolutionary Computation, 2025, pp.1-28. 10.1162/evco_a_00370. hal-05171409

HAL Id: hal-05171409

https://hal.science/hal-05171409v1

Submitted on 20 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Beyond Landscape Analysis: DynamoRep Features For Capturing Algorithm-Problem Interaction In Single-Objective Continuous Optimization

**Gjorgjina Cenikj**
gjorgjina.cenikj@ijs.si
Computer Systems Department, Jožef Stefan Institute, Ljubljana, 1000, Slovenia;
Jožef Stefan International Postgraduate School, Ljubljana, 1000, Slovenia

**Gašper Petelin**
gasper.petelin@ijs.si
Computer Systems Department, Jožef Stefan Institute, Ljubljana, 1000, Slovenia;
Jožef Stefan International Postgraduate School, Ljubljana, 1000, Slovenia

**Carola Doerr**
carola.doerr@lip6.fr
LIP6, Sorbonne Université, Paris, 75252, France

**Peter Korošec**
peter.korosec@ijs.si
Computer Systems Department, Jožef Stefan Institute, Ljubljana, 1000, Slovenia

**Tome Eftimov**
tome.eftimov@ijs.si
Computer Systems Department, Jožef Stefan Institute, Ljubljana, 1000, Slovenia

## Abstract

The representation of optimization problems and algorithms in terms of numerical features is a well-established tool for comparing optimization problem instances, for analyzing the behavior of optimization algorithms, and the quality of existing problem benchmarks, as well as for automated per-instance algorithm selection and configuration approaches.

Extending purely problem-centered feature collections, our recently proposed DynamoRep features provide a simple and inexpensive representation of the algorithm-problem interaction during the optimization process. In this paper, we conduct a comprehensive analysis of the predictive power of the DynamoRep features for the problem classification, algorithm selection, and algorithm classification tasks. In particular, the features are evaluated for the classification of problem instances into problem classes from the BBOB (Black Box Optimization Benchmarking) suite, selecting the best algorithm to solve a given problem from a portfolio of three algorithms (Differential Evolution, Evolutionary Strategy, and Particle Swarm Optimization), as well as distinguishing these algorithms based on their trajectories. We show that, despite being much cheaper to compute, they can yield results comparable to those using state-of-the-art Exploratory Landscape Analysis features.

## Keywords

single objective numerical optimization, feature construction, problem classification, algorithm classification

## 1 Introduction

Numerical representations of black-box optimization instances, referred to as *problem landscape features*, are crucial for analyzing the similarity of optimization problem instances.
stances (Škvorc et al., 2020) and the representativeness and redundancy of existing
benchmark suites (Cenikj et al., 2022; Muñoz and Smith-Miles, 2020; Garden and En-
gelbrecht, 2014; Lang and Engelbrecht, 2021). They also serve as inputs for training
Machine Learning (ML) models for automated per-instance algorithm selection (Ker-
schke and Trautmann, 2019a) and configuration (Belkhir et al., 2017).

Several types of features have been proposed for single-objective continuous optimization problems, which are typically calculated using a set of solution candidates which are sampled from the decision space of the optimization problem instance via some non-adaptive approach such as uniform sampling or Latin Hypercube sampling and which are then evaluated. Therefore, current feature extraction methodologies generate a “static” depiction of the problem that is not tailored to a specific algorithm. These representations embody characteristics of the entire problem space, regardless of the sections of the problem space that have been encountered by the algorithm in question during its search procedure. It has been demonstrated that the characteristics of the problem space that an algorithm encounters during optimization can substantially differ from the global landscape of the problem being solved (Janković and Doerr, 2019).

Previous research (Mersmann et al., 2011; van Stein et al., 2023; Cenikj et al., 2024c; Seiler et al., 2024b) has focused on developing problem landscape features for single-objective optimization problems, which describe the characteristics of the problem instances. However, a recent work (Pulatov et al., 2022) in SAT solvers has shown that *algorithm features* can enhance the performance of ML models for algorithm selection by enabling them to predict the performance of multiple algorithms simultaneously, rather than training a separate model for each algorithm. Additionally, features that capture the *algorithm-problem interaction* can also benefit tasks such as dynamic algorithm selection, where the ML model is deciding whether it is beneficial to switch to a different algorithm instance online, during the run of an algorithm instance (Kostovska et al., 2022). Another potential application of features that capture the algorithm-problem interaction is the selection of algorithms for an algorithm portfolio, i.e., the set of algorithms the algorithm selector is asked to choose from. Finally, representing algorithm trajectories in terms of numerical features would also allow the analysis of novelty in newly proposed optimization algorithms by comparing their search behavior to existing algorithms.

Driven by the desire to capture the algorithm-problem interaction, in (Cenikj et al., 2023), we have proposed the DynamoRep features for problem representation using trajectory-based population dynamics. The primary goal of this approach is to utilize the simplest and least computationally demanding features that can be derived from the samples of the problem which the algorithm is already examining during its search process. The rationale behind using these features is that they do not add extra computational overhead to the algorithm's execution and do not necessitate additional function evaluations to generate the samples, as the solutions are already being evaluated as part of the algorithm's search. The DynamoRep features have been proposed for single-objective continuous optimization problems, although they have potential to be extended to other domains and settings, such as, for example, a multi-objective one. In the initial proof-of-concept study, we have demonstrated the feasibility of the proposed features for classifying 3-dimensional problem classes from the Black Box Optimization Benchmarking (BBOB) suite (Hansen et al., 2009) using samples from the trajectories of optimization algorithms. We have shown that four very simple statistical measures (*minimum*, *maximum*, *mean*, and *standard deviation*) are sufficient to capture properties of the algorithm search space, which can determine what problem the algorithm is
solving with classification accuracies above 90%. In the initial study, the applicability
of the features was only demonstrated on 3-dimensional problems. The features were
extracted from each of the iterations from the trajectory of the optimization algorithm
run, not taking care of the algorithm performance or its convergence.

**Our contribution:** In this paper, we extend our previous work by conducting a comprehensive analysis of the performance of the DynamoRep features for the tasks of **algorithm selection, problem classification, and algorithm classification**. Algorithm selection is the most common task used to evaluate the predictive power of problem representations, due to its highest practical utility (Kerschke and Trautmann, 2019a; Seiler et al., 2024a,b; Cenikj et al., 2024c). The problem classification task is well explored in several studies (Petelin et al., 2022; Renau et al., 2020) in which different groups of problem landscape features are used to test their utility to describe an optimization problem instance. This task has been performed to find good problem landscape features that in the future can help in the selection of diverse and unbiased problem instances that are involved in the experiments. However, the motivation for evaluating the DynamoRep features in the problem classification tasks covers other perspectives. First, we can detect problems on which the algorithm instance has similar search capabilities (if it is difficult for the classifier to distinguish between different problems in the classification task) or different search capabilities (if it is easy to distinguish between different problems in the classification task).

In addition, we explore the potential of the DynamoRep features for **algorithm classification**. The motivation behind this choice of a learning task is that it allows us to evaluate whether the features have enough predictive power to distinguish between the behavior of different algorithms. They could be further used for (i) dynamic algorithm selection, (ii) selecting algorithms with complementary behavior to be included in an algorithm portfolio used by an algorithm selector, or (iii) for analyzing the novelty of different algorithms by emphasizing the differences on which problems the algorithms have different search capabilities and on which problems they have similar search capabilities.

For all three tasks, we investigate the tradeoff between the model performance
and the budget used to calculate the features, and conduct a feature importance analy-
sis. We investigate the models’ dependency on the range of the objective functions by
observing the impact of scaling the objective function values. Additionally, we evalu-
ate the impact of performing differencing of the DynamoRep features with the goal of
eliminating the representation’s sensitivity to the initial population.

We compare our approach to the most commonly used Exploratory Landscape
Analysis (ELA) (Mersmann et al., 2011) features in two ways. Firstly, we use the classi-
cal way of computing the ELA features on a static sample of problem solutions gener-
ated using Latin Hypercube sampling (Menčík, 2016), which is independent from the
algorithm execution. Additionally, we compare our method to the ELA features com-
puted on the samples from the trajectories generated by the algorithm execution. As
opposed to our original work introducing the DynamoRep features (Cenikj et al., 2023)
and other applications of the ELA features on algorithm trajectory data (Jankovic et al.,
2021), where the ELA features were calculated by merging the individuals from all of
the algorithm iterations into a single sample, here we calculate the ELA features on
the samples from each iteration separately, and use the concatenated features from all
iterations to represent the trajectory. This has been previously done also in (Korošec
and Eftimov, 2024), and it is an approach to ensure fair ML benchmarking taking into
account the purpose of our comparative study, although it is not the main contribution
of this work.

**Outline:** The remainder of the paper is organized as follows. Section 2 provides an overview of related work. Section 3 describes the generation of the DynamoRep features, Section 4 explains the experimental design, while Section 5 provides a comparison of the running times to compute DynamoRep and iteration-based ELA features. Sections 6, 7, and 8 present the results for the algorithm selection, problem, and algorithm classification tasks, respectively, while Section 9 concludes the paper.

**Reproducibility:** The code for the experiments is available at https://github.com/gjorgjinac/DynamoRep_ext. The algorithm execution data and the results of the experiments are available at https://portal.ijs.si/nextcloud/s/ECaBkzj2QTx4Jkq.

## 2 Related Work

Most of the feature extraction methodologies for single-objective continuous optimization problems are based on a sample of solutions from the decision space of the optimization problem instance generated using some non-adaptive approach such as uniform sampling, random sampling or Latin Hypercube sampling. The most commonly used features are the Exploratory Landscape Analysis (ELA) features (Mersmann et al., 2011), which despite their prevalence, have some undesirable properties. For instance, the calculation of ELA features can be computationally demanding for high-dimensional problems (Morgan and Gallagher, 2014), and they have been shown to be sensitive to variations in the sample size and sampling method (Renau et al., 2020; Škvorc et al., 2021; Gallagher and Muñoz, 2024), as well as not being invariant to transformations such as scaling and shifting of the optimization problem (Škvorc et al., 2020, 2021). Therefore, in the last few years, other approaches to feature calculation have been explored. The recently proposed Topological Landscape Analysis (TLA) features (Petelin et al., 2022) characterize optimization problems using concepts from topological data analysis, capturing the persistence of different topological structures in point clouds. Representations based on deep learning techniques are also becoming popular, with different architectures being explored, such as convolutional neural networks (Seiler et al., 2022), transformers (Cenikj et al., 2024c; Seiler et al., 2024b,a), and autoencoders (van Stein et al., 2023). All of these features require additional function evaluations to be used before the algorithm execution to construct a sample of solutions which is then used to calculate the features. Unlike these studies, here we are exploring the possibility of extracting useful information from the samples which are already evaluated by the algorithm during its search procedure.

Related works aiming to capture features using dynamic sampling techniques, include features based on the Simultaneous Optimistic Optimization Tree (SOOT) (Derbel et al., 2019) and Search Trajectory Networks (STNs) (Ochoa et al., 2021). In contrast to SOOT features (Derbel et al., 2019), which proposes the extraction of features based on a predefined algorithm executed independently of the optimization algorithm used to solve the problem, our approach aims to use the solutions explored by the same algorithm, so that no additional function evaluations are used for feature calculation. Additionally, differently from STNs, whose primary purpose is visualization of algorithm behavior and require multiple runs of the algorithm, the primary goal of our work is to propose numerical representations of trajectories applicable to a single algorithm run, at low cost.
## 3 DynamoRep Representations

Given an optimization algorithm and an optimization problem instance, we aim to generate a vectorized representation of the interaction of the algorithm with the problem instance, which takes into account the algorithm trajectory's longitudinality. The main objective of the approach is to use the simplest, most computationally inexpensive features which can be extracted from the samples which the algorithm is already considering in its search procedure. The reasoning behind such features is that they do not introduce an additional overhead in the algorithm execution, and do not require an additional budget of function evaluations for generating the sample upon which they are calculated, since the solutions are anyway evaluated for the purposes of the algorithm's search.

For the following description, we assume that we are dealing with a $d$-dimensional search space $S \subseteq \mathbb{R}^d$, a single-objective problem instance $f: S \to \mathbb{R}$, and a population-based algorithm being executed on the problem $f$ with population size $\lambda$. Denoting by $(x_0^i, x_1^i, \dots, x_{d-1}^i, y^i)_{i=1,\dots,\lambda}$ the $\lambda$ decision variables $x^i = (x_0^i, x_1^i, \dots, x_{d-1}^i)$ of the population and their function values $y^i = f(x^i)$ obtained in a single iteration, we use as a representation a concatenation of the following expressions

$$(\min\{x_0^1, \dots, x_0^\lambda\}, \dots, \min\{x_{d-1}^1, \dots, x_{d-1}^\lambda\}, \min\{y^1, \dots, y^\lambda\}),$$
$$(\max\{x_0^1, \dots, x_0^\lambda\}, \dots, \max\{x_{d-1}^1, \dots, x_{d-1}^\lambda\}, \max\{y^1, \dots, y^\lambda\}),$$
$$(\text{mean}\{x_0^1, \dots, x_0^\lambda\}, \dots, \text{mean}\{x_{d-1}^1, \dots, x_{d-1}^\lambda\}, \text{mean}\{y^1, \dots, y^\lambda\}),$$
$$(\text{std}\{x_0^1, \dots, x_0^\lambda\}, \dots, \text{std}\{x_{d-1}^1, \dots, x_{d-1}^\lambda\}, \text{std}\{y^1, \dots, y^\lambda\}).$$

The representation concatenates the coordinate-wise minimum, maximum, mean, and standard deviation, respectively. In this way, we create a representation for a single iteration of the optimization algorithm. A representation for the entire trajectory is then generated by concatenating the representations of all of the iterations. Since we use four descriptive statistics, the representation for each iteration is of size $4(d + 1)$. If $n$ is the number of iterations for which the algorithm is run, the representation of the algorithm trajectory is hence of dimension $4n(d + 1)$.

We have selected these specific descriptive statistics for the aggregation of the dimensions of each population due to their low computational overhead, and the ability to obtain them without additional function evaluations, simply by utilizing the candidate solutions evaluated by the algorithm during its run. With these statistics, we aim to capture information about the nature of the candidate solutions generated by the algorithm (e.g., their range and diversity) in each iteration, as well as their quality (objective function values). Additionally, these statistics are computable for any sample size and can therefore be used for different population size, which is not the case for some ELA features for example, which produce invalid NaN values when the sample size is not sufficient. Furthermore, these statistics are widely utilized as aggregation operators in other ML domains such as computer vision (Gholamalinezhad and Khosravi, 2020) and graph neural networks (Corso et al., 2020).

## 4 Experimental Design

In this section, we outline the crucial components of the experimental design, including the *problem portfolio* (the set of problem instances that the algorithms are run on), the *algorithm portfolio* (the set of algorithms that are run on the problem portfolio), and the *machine learning model* (trained for algorithm selection, as well as problem and algorithm classification based on the proposed features). We also provide a detailed description of the models' evaluation.
## 4.1 Problem Portfolio

The Black-box Optimization Benchmarking (BBOB) suite (Hansen et al., 2009) from the Comparing Continuous Optimizers (COCO) environment (Hansen et al., 2020) contains 24 single-objective optimization problem classes. Each problem class can further have multiple problem instances, which represent a transformed (e.g., shifted or scaled) version of the original problem class. For our empirical evaluation, we use the first 100 instances of each problem class in dimensions $d \in \{3, 5, 10, 20\}$, respectively. However, due to space constraints, we mainly focus on dimension 5, while results for other dimensions are provided in the Supplementary Materials (SM).

## 4.2 Algorithm Portfolio

The *pymoo* (Blank and Deb, 2020) python library is used to run the Differential Evolution (DE), Particle Swarm Optimization (PSO), and Evolutionary Strategy (ES) algorithms on the problem instances from the BBOB benchmark. We run the algorithms in a fixed-budget scenario with a population size of 10*d*, and a budget of 50 iterations (including the initial population). A budget of 50 iterations therefore corresponds to 500*d* function evaluations. We use a budget of up to 30 iterations to calculate the features, and use the maximal budget of 50 iterations for algorithm selection. In future work, algorithms with smaller population sizes (less information in a single population regardless of the number of features) will be investigated. We need to point out here that the selection of the algorithms is only to test the proposed methodology. In the future, different algorithms can be used depending on the task that is being solved.

All algorithms use Latin Hypercube sampling (Menčík, 2016) to construct the initial population. The rest of the configuration properties of the algorithms are set to the default values provided in the *pymoo* library, version 0.6.0. We perform five executions of each algorithm on each of the 2,400 problem instances. To avoid a potential bias with respect to the initialization, the five executions are performed with five different random seeds, which are fixed before the algorithm execution. This means that the algorithms executed with the same random seed have the same initial population.

## 4.3 DynamoRep Feature Calculation

The feature calculation step involves the generation of the DynamoRep features as indicated in Section 3. With the goal of investigating the sensitivity of the machine learning model to the range of the objective function values and to the initial population considered by the algorithm, we perform experiments with two optional steps in the feature calculations: scaling and differencing.

**Scaling:** In order to investigate the sensitivity of the machine learning model to the range of the objective function values, we perform experiments when the objective function values (*y* values) are scaled in the range [0,1] as explored in (Prager and Trautmann, 2023). This is done using min-max scaling on a trajectory level, meaning that the values are scaled within each run of the algorithm on each problem instance, before calculating the DynamoRep features. We keep the decision variables (*x* values) unscaled in both cases, since in the COCO environment, they are already limited to be in the range [-5,5]. Note that if one would like to check the influence of using a different number of iterations for calculating the DynamoRep features, the scaling should be done separately for each feature calculation budget, since the features should not be scaled on the basis of the range of the objective function values in the entire algorithm trajectory, but only within the range of the observed iterations which are used to calculate the features.
**Differencing:** Differencing is a technique used to transform time series data by calculating the differences between successive observations (McGonigle et al., 2022). This method can help stabilize the mean of the series by removing changes in the level, thereby reducing or eliminating trends and seasonal patterns. In our case, the goal of performing differencing is to remove the DynamoRep features' sensitivity to the initial population which is dependent on the random seed used to execute the algorithm. The differencing of the DynamoRep features is performed by calculating the difference between each feature calculated with one statistic on one of the problem dimensions (decision variables or objective function value) and its corresponding feature from the previous iteration. This means that if we have features corresponding to the standard deviation of the objective function value in the first and second iteration `std_it_0_y` and `std_it_1_y`, these would be transformed into one feature `std_it_1_y - std_it_0_y`.

Note that the scaling step happens before the DynamoRep features are calculated, while the differencing step is performed after the feature calculation.

## 4.4 Evaluation Tasks

We evaluate the utility of the DynamoRep features for the tasks of algorithm selection, problem classification, and algorithm classification. Following the most common setup in the analysis of feature-based approaches to black-box optimization (Petelin et al., 2022; Belkhir et al., 2017; Mersmann et al., 2011; Jankovic et al., 2021), we use Random Forest (RF) models which we train for the three tasks, given the DynamoRep features extracted from the trajectory of the algorithm run. The models are evaluated using stratified 10-fold cross-validation. The RF model was selected since it offers good performance on tabular data (Shwartz-Ziv and Armon, 2022) and it provides feature importance values then help with interpretability. The way the data is split into training and testing sets depends on the task the model is trained for. For all tasks, the RF models are executed using the default configuration parameters in the *scikit-learn* (Pedregosa et al., 2011) python library version 1.2.2. We do not perform parameter tuning in order to evaluate the impact of the proposed features only, on a fixed model configuration.

### 4.4.1 Algorithm Selection

We treat algorithm selection as a regression task, where the RF model is trained to predict the performance of the three optimization algorithms on the basis of a single execution of one of the algorithms. In this algorithm selection setting, we aim to evaluate whether the trajectory features derived from a single algorithm provide enough information to predict its performance after additional iterations, as well as that of other algorithms, which will then be used to identify the most suitable one. This means that one of the algorithms is executed for some feature calculation budget of *b* iterations and the features are calculated up to this iteration. The performance of the algorithms is then predicted for a total budget of *B* > *b* iterations. A single RF model is trained for each problem dimension, each different *b* (feature calculation budget) and for each of the three algorithms. In our experiments, we only consider one total budget *B* of 50 iterations (for which we are prediction algorithm performance), with varying feature calculation budgets *b* ∈ {1, 3, 5, 10, 20, 30}.

To capture the algorithm performance, we use the normalized precision score introduced in (Cenikj et al., 2024a), and later used in (Seiler et al., 2024a; Cenikj et al., 2024b). The score is captured in the form of a real valued number in the range [0, 1] and is calculated as follows. Assume that $y_{arp}$ is the best solution (having the lowest
objective function value) found by algorithm a in run r for problem instance p. The value $y_{arp}$ is scaled based on the range of the solutions found by the other algorithms for the same problem instance and the same initial population (execution with the same random seed). Let $b_{rp}$ and $w_{rp}$ be the best solutions found by the best-performing and worst-performing algorithm found for problem instance p in run r, respectively. The scaled best objective function value achieved by algorithm a in run r on problem instance p is calculated as:

$$s_{arp} = \frac{y_{arp} - b_{rp}}{w_{rp} - b_{rp}}. \qquad (1)$$

This score captures how much the best solution found by algorithm a in run r was better than the other algorithms executed with the same initial population. In such a setting, the best-performing algorithm would be assigned a score of zero, and the worst performing one would get a score of one. We calculate the final value $s_{ap}$, capturing the performance of algorithm a on problem instance p across all algorithm runs, by taking the mean of all $s_{arp}$. An example of the score calculation can be found in Cenikj et al. (2024a). Note that in the case when all algorithms converge to the same solutions, they all obtain a score of zero.

We evaluate the AS model in two settings:

i) Instance-split - In this evaluation setting, the data is split into training and testing sets in such a way that instances from the same problem classes are present both in the training and in the test set. In each fold, the runs that belong to 10% of all problem instances from each problem class are removed from the training set and used for testing.

ii) Problem-split - In this evaluation setting, the data is split so that the training and testing data always consist of instances from different problem classes, i.e., instances from the same problem class cannot appear both in the training and in the testing data. In each fold, the runs of 10% of all problem classes are for testing, and the runs of the remaining 90% of the problem classes for training.

We evaluate the performance of the RF model by considering the true score of the algorithm which was selected by the model for each problem instance:

$$AS\_performance = \frac{1}{|P|} \sum_{p \in P} 1 - (s_{sp} - s_{bp}) \qquad (2)$$

Here, the true score of the algorithm selected by the RF model for problem instance p is denoted by $s_{sp}$, while $s_{bp}$ is the score of the actual best-performing algorithm for problem instance p. This performance metric is a maximization metric bounded between zero and one. A score of zero is obtained if the worst-performing algorithm is consistently selected as the best for all problem instances, whereas a value of one is attained when the algorithm selector correctly identifies the best algorithm for every problem instance. The score is meant to capture how much one is losing by selecting an algorithm which is not the ideal one for that problem instance.

### 4.4.2 Problem Classification

In the case of a problem classification task, the RF model gets as input the DynamoRep representation generated from the trajectory of one execution of an optimization algorithm and predicts to which of the 24 problem classes from the BBOB collection the representation belongs to. We train a separate RF model to classify problem instances from each problem dimension, and each of the three optimization algorithms (DE, ES,
and PSO). The split of the data into training and testing sets is done on an instance level. In each fold, the runs that belong to 10% of all problem instances from each problem class are removed from the training set and used for testing. We train each model on the trajectories obtained with a single algorithm execution (one random seed) and test it on trajectories obtained with 5 different seeds, to account for the fact that in practical applications, the algorithms might be executed with random seeds that are different from the one the model was trained on. That is, the models use 2,160 training instances (1 algorithm × 1 algorithm execution × 24 problems × 90 instances), and 1,200 testing instances (1 algorithm × 5 algorithm executions × 24 problems × 10 instances). One important thing to note is that for higher feature calculation budgets and high problem dimensions, the number of features can exceed the number of training instances. This can be further reduced using some dimensionality reduction techniques, however, in this case, we do not perform any dimensionality reduction so we can evaluate the importance of each feature separately. However, when used in practice, dimensionality reduction methods should be applied, especially in the case of a limited number of training instances or high problem dimensions.

### 4.4.3 Algorithm Classification

In the algorithm classification task, the input of the RF model is the same as for the problem classification task, however, the RF model is trained to predict to which of the three algorithms the trajectory belongs to. In this case, we train a single RF model for each problem dimension. For this task, the problem instances are split on a problem class level, i.e., each fold uses the runs of 10% of all problem classes for testing, and the runs of the remaining 90% of the problem classes for training. This means that in each fold, the runs of all problem instances from selected 2-3 problem classes (200-300 instances) will be removed from the training data and used for testing.

In the folds where 3 problem classes are used for testing, the models are trained on 6,300 instances (3 algorithms × 1 run × 21 problems × 100 instances) and evaluated on 4,500 instances (3 algorithms × 5 run × 3 problems × 100 instances).

The goal of such a split is to determine if the model can identify an optimization algorithm from its sampling dynamics on previously unseen problems. Consistent with the problem classification task, we train each model with trajectories obtained with one algorithm execution (single random seed) and testing with trajectories obtained with different executions (different random seeds).

## 4.5 Comparison to ELA features

We perform the comparison to the ELA features in two ways: i) calculating the ELA features on the samples generated in each iteration of the algorithm execution, and ii) calculating the ELA features in the classical way, on a "static" sample of solutions generated using Latin Hypercube sampling (Menčík, 2016). Both approaches are explained in detail in continuation.

### 4.5.1 Iteration-based ELA features

For all tasks, we compare the performance of the DynamoRep features to ELA features calculated on the population from each iteration of the optimization algorithms' run. The comparison with the ELA features calculated separately for each iteration in the section was included to ensure a fair experimental setup for feature comparison. While we recognize that ELA features were not originally intended for iterative use, from a machine learning perspective, two feature representations are considered comparable when they are derived from the same data.
We compute ELA features from the following categories using the *flacco* R library (Kerschke and Trautmann, 2019b): *basic*, *disp*, *ela_distr*, *ela_level*, *ela_meta*, *ic*, *nbc*, and *pca*. In this way, we calculate a total of 93 features. Section 5 in the SM contains the exact number of missing values for each feature. The group with the highest number of features containing missing values is the *ela_level* group, where three features contain no values at all. We discard all features that have more than 10% missing values in the training set, while for the remaining features, the missing values are imputed with the mean value of the feature in the training set. We need to point out here that, instead of the mean, the median value can be used as the most robust statistic, however, we have not performed it since the classifiers obtain good performance by using the mean value.

The ELA features calculated for each iteration are concatenated together to form a representation for the whole algorithm trajectory which is given as input to the RF model.

It may be important to note that when the DynamoRep features were originally introduced (Cenikj et al., 2023), the comparison to the ELA features was not performed in the same way. Namely, the samples from all iterations of the algorithm's trajectory were concatenated, and the ELA features were calculated on the entire trajectory as was previously done in (Jankovic et al., 2021). Here, instead, we calculate the ELA features on each iteration separately, and we then concatenate the representations for each iteration. This is due to the fact that in the original study, we only used 3d problems and a population size of 30, which would mean that if the ELA features were calculated on an iteration level, we would only have a sample size of 30.

Due to computational costs associated with calculating ELA features from each iteration, we perform this comparison only for 5d problems. The computational complexity of calculating the ELA features for higher dimensions is further increased because we are also testing the influence of scaling the objective function values. This is computationally expensive since the objective function values are scaled for each feature calculation budget separately, meaning that the ELA features produced for the same iteration of an algorithm trajectory can differ depending on the feature calculation budget.

### 4.5.2 Static ELA features

For the problem classification and algorithm selection tasks, we compare our results with the classical approach of computing ELA features on a collection of samples obtained using Latin Hypercube sampling (Menčík, 2016). We refer to this approach as the "Static ELA" features, since the samples they are calculated on do not have a dynamic component from the algorithm behaviour. We use the same feature groups as for the Iteration-based ELA features. The Static ELA features are calculated on samples of different sizes: 10d, 30d, 50d, 100d, 200d, and 300d, where *d* is the problem dimension. These samples sizes are equivalent in terms of function evaluations to running 1, 3, 5, 10, 20, or 30 iterations of an algorithm, since the population size is 10d. We additionally investigate the impact of scaling objective function values in the range [0,1] using min-max scaling performed before feature calculation.

Please note that for the algorithm classification task, we do not include a comparison to the classical ELA approach of calculating the ELA features on a sample obtained using Latin Hypercube sampling. This is due to the fact that the classical ELA features do not capture algorithm behavior, i.e., they represent the problem instance, regardless of the algorithm being executed. That is, the features remain the same regardless of
<figure><img src="image_3.png" /><figcaption>Figure 1: Mean time in seconds needed to calculate the DynamoRep features and the iteration-based ELA features for a single algorithm trajectory consisting of 30 iterations</figcaption></figure>

the algorithm; it is therefore not possible to train a classification model to predict the algorithm from such features. We then train the RF model using the same setup as the DynamoRep features, except here we do not use any algorithm-specific data, only features extracted from the artificially generated problem samples.

We need to highlight that feature selection techniques have not been applied neither to DynamoRep nor to ELA features. This has been done since we are testing the predictive power of different features and are not interested in finding the best classifier. Since the classifier results with all features are very promising and high, applying feature selection can only increase the accuracy by a few percentage points. Even more, since the RF algorithm is used as a classifier in both learning tasks, it is well known that it works by selecting only the important features from a given feature portfolio. Further, this will allow us to find the most important features together with the information from which part of the trajectory they have been calculated. However, in all experiments, the number of features is lower than the number of instances involved in the training data to avoid overfitting.

# 5 Computation Time of the DynamoRep and Iteration-based ELA features

In this section, we analyze the computation time for the DynamoRep and the iteration-based ELA features. Figure 1 compares the average time needed to generate the features for one algorithm trajectory consisting of 30 iterations. The running time of the feature calculation procedure was measured on the trajectories of the DE algorithm executed on one problem instance from each BBOB problem class for each of the problem dimensions 3, 5, 10, and 20. The ELA features were computed for each iteration independently, and the total time for a single trajectory is taken to be the sum of the time needed to compute all iterations.

For the DynamoRep features, we explore two implementations. In the first one, the features are extracted from the entire trajectory at once, while in the second one, the features are extracted from each iteration separately and are then concatenated together.
Both approaches produce the same results, they only differ in the implementation. In our further experiments, we use the first approach, since it is faster due to efficient calculations provided by the *pandas* python library, however, we still include the computation time of the second approach, as it is more similar to the way the iteration-based ELA features are computed and may thus be a fairer comparison. In Figure 1, the first approach is denoted as "DynamoRep (entire trajectory)", while the second approach is denoted as "DynamoRep (each iteration separately)". From the figure, we can see that the execution time of both implementations of the DynamoRep features is substantially lower than the iteration-based ELA features. Even for the 20d problems, the first approach to calculating the DynamoRep features takes 0.11 seconds, and it is 875 times faster than the 96.30 seconds needed for the iteration-based ELA features. The execution time for a single iteration can be approximated by dividing the values provided for the "DynamoRep (each iteration separately)" and "Iteration ELA" columns by 30, which represents the number of iterations.

One important thing to point out is that the DynamoRep features are implemented in python, while for the ELA features, we use the R implementation of the *flacco* package, so the execution time is also dependent on the programming language. We use the R implementation of the *flacco* package, since it has been the most utilized implementation in previous ELA studies.

# 6 Algorithm Selection Results

In this section, we present the results from the algorithm selection task. We start with an analysis of the performance achieved using increasing feature calculation budgets, which we compare to the performance of the iteration-based and of static ELA features. We then analyze the importance of the DynamoRep features for this task.

## 6.1 Algorithm Selection With Limited Feature Calculation Budget

Figures 2 and 3 show the results of the algorithm selection task, for the instance-split and problem-split evaluation settings, respectively. The feature calculation budget is denoted on the horizontal axis. The vertical axis shows the difference between the AS performance (as defined in equation 2) achieved by the feature-based models and a baseline model predicting the mean performance on the train set for each algorithm (this is equivalent to the single best solver). A score of zero indicates that the feature-based model is equivalent to the baseline, i.e., the features are not useful for algorithm selection. The larger the value, the better the model. The differently colored boxplots represent different models. The first three boxplots represent the results of the models using the DynamoRep features extracted from the trajectories of the three different algorithms. The second three boxplots represent the iteration ELA features calculated on the trajectories from the three algorithms. Finally, the last boxplot corresponds to the static ELA features, calculated on the Latin Hypercube sample. Please note here that the function evaluations used for calculating the static ELA features are equivalent to the function evaluations used for the calculation of the DynamoRep and iteration ELA features. Also note that the Static ELA features are only included in the subplots where differencing is not used, since it is not possible to perform differencing on the static sample. Additionally, in the subplots where differencing is used, there are no results for feature calculation budget of one iteration, since a minimum of two iterations is required to perform differencing.

Please note that we are presenting the difference between the feature-based models and the baseline, since there is a large variation between their performance of different
<figure><img src="image_4.png" /><figcaption>Figure 2: Algorithm selection results using the instance-split evaluation setting for 5d problems</figcaption></figure>

folds, and plotting the actual performance leads to reduced visibility. The variation in performance between folds is especially pronounced for the Problem-split evaluation, where the models are evaluated on previously unseen classes of problems. In this case, the baseline model achieves performances in the range 0.53-0.98 with a median of 0.85 and mean of 0.78. On the other hand, in the Instance-split evaluation, the baseline model has performances in the range 0.75-0.79 with a median of 0.77 and mean of 0.78.

Looking at the results from the instance-split evaluation setting in Figure 2, we can see that all of the models outperform the baseline. When differencing and scaling are not performed, the performance of the Static ELA and Iteration ELA features are nearly equivalent. With feature calculation budgets greater than one iteration, the DynamoRep features extracted from the ES algorithm trajectories also obtain comparable results. However, we can see that the results are slightly inferior when using the trajectories from DE and PSO in this setting. This is more pronounced in the setting where scaling is performed, but differencing is not. In this case, we can see that the DynamoRep features extracted from DE trajectories have deteriorated performance, which is not as pronounced for the ES and PSO trajectories. When using a feature calculation budget of a single iteration, however, the DynamoRep features outperform the baseline by only 0.05. This is expected, since the DynamoRep features are meant to capture the algorithm dynamics, not information from a single iteration. The iteration ELA features
<figure><img src="image_5.png" /><figcaption>Figure 3: Algorithm selection results using the problem-split evaluation setting for the BBOB problems in dimension 5.</figcaption></figure>

experience slight performance deterioration when scaling is applied, however, it is not as pronounced as the DynamoRep features. This can be attributed to the fact that some of the ELA features are not sensitive to scaling (Škvorc et al., 2020). The application of the differencing transformation does not have a large impact when scaling is not applied, however, when the two transformations are used together, they have a negative impact on the performance of all models. Interestingly, the impact of the joint use of differencing and scaling seems to have the least effect on the features calculated on the ES trajectories. Both the Iteration ELA and DynamoRep features calculated on the ES trajectories achieve similar performance, which is superior to the other trajectories.

Conversely, taking into account Figure 3, showing the results from the problem-split evaluation, we can see that most of the models do not outperform the baseline. For a few of the models, we can observe that the median is higher than zero, i.e., the model is sometimes outperforming the baseline by a small margin. This is occurring in the scenarios when differencing is not performed, and the feature calculation budget is higher than 20 iterations. However, in these cases, the improvement over the dummy model is negligible and although the median difference exceeds zero, the feature-based models do not consistently beat the baseline, since the performance in some folds is smaller than zero. These results provide a lot of open questions and potential future research directions related to zero-shot algorithm selection techniques.
## 6.2 Feature Importance Analysis

Figure 4 shows the feature importances assigned by the RF model to the DynamoRep features for different algorithms. The importance of each feature is obtained by taking the median feature importance given by the RF models in each fold in the 10-fold cross-validation. Please note that this analysis is focused on the instance-split evaluation setting, since the models evaluated in the problem-split setting did not outperform the baseline and it is therefore not useful to analyze how they do reasoning. Additionally, we do not analyze the scenarios when the differencing transformation is performed, since it had a negative effect on performance. Each column refers to a different algorithm. The top row refers to the importance assigned by models trained on the original trajectories, without scaling of the objective function values, while the bottom row refers to the importance assigned by models trained on the trajectories with scaled objective function values.

When the objective function values are not scaled, we can observe that the most important features are calculated using the standard deviation statistic. When trajectories from the DE algorithm are used, we can see that most of the selected features come from the later iterations (after iteration 20), or the first iteration. This could explain why the models using the DE trajectories have some predictive power when using a single iteration, however, they reach the performance of the competing models at iterations 20-30. Most of these features are calculated using the standard deviation statistic, which indicates that the RF model learns to do AS based on how the DE algorithm has converged in the final iterations. Considering the ES algorithm, we can see that the most important features are coming from the standard deviations in the initial ten iterations, or the last iteration, which could explain why the DynamoRep models using the ES trajectories are outperforming the DynamoRep models using trajectories from the other algorithms in the initial iterations. For the PSO algorithms, all of the most important features are calculated using the standard deviation statistic, mostly from the first or last ten iterations. On the contrary, when the objective function values are scaled, we can see diverse features having a high importance. For all algorithm trajectories, the top 2 most important features are the standard deviations of the fourth decision variable ($x_4$) from different iterations. The mean of the objective function values from the first and last iterations is also commonly appearing.

# 7 Problem Classification Results

This section presents the results from the problem classification task, starting with the inspection of the tradeoff between the feature calculation budget and the obtained accuracy, followed by a comparison with the iteration-based and static ELA features, and a feature importance analysis.

## 7.1 Problem Classification With Limited Feature Calculation Budgets

In this subsection, we present the results obtained for the problem classification task, when instead of using the entire algorithm trajectory, the training data is limited to a certain number of iterations. The purpose of this evaluation is to identify at which point of the optimization process, one can train a ML model to identify the problem that the algorithm is solving with sufficient confidence.

Figure 5a contains violin plots representing accuracies obtained for classifications of 5d problems, respectively. The figures representing the accuracies for 3d, 5d and 10d problems can be found in Section 3 from the supplementary material (SM).

The application of the differencing step did not result in improved performance
<figure><img src="image_6.png" /><figcaption>Figure 4: Top 20 DynamoRep features with highest feature importance assigned by the RF model for the algorithm selection task in the instance-split evaluation setting</figcaption></figure>
<figure><img src="image_7.png" /><figcaption>Figure 5: Accuracies obtained with the DynamoRep and iteration ELA features for the problem classification task of 5d problems</figcaption></figure>

for the problem classification task. For this reason, we include these results in the SM and do not investigate it further.

The subplots in the left column contain the accuracies obtained without scaling the objective function values, while the subplots in the right column contain the accuracies obtained with the scaling of the objective function values. The y-axis represents the obtained accuracy, while the x-axis represents the feature calculation budget (the number of iterations used to calculate the features that will be further used to train the RF model).

The violin plots represent the distribution of accuracy values obtained across different seeds used for running the optimization algorithm, and different folds in the 10-fold cross-validation of the RF model. The right half of each violin plot (in lime color) refers to the accuracies obtained when the RF model is evaluated on trajectories obtained by running the optimization algorithm with the same random seed which was used for the trajectories in the training data. On the other hand, the left part of the violin plot (in dark green color), represents the accuracies obtained when the RF model is evaluated on trajectories obtained with random seeds other than the one used for training.

The classification accuracy steadily increases by including more iterations for cal-
calculating the features, for all dimensions and all algorithms. We can observe that if the objective function values are not scaled, even using a feature calculation budget of one iteration (using the initial population generated using Latin Hypercube sampling, before the algorithm performs any operations), the RF model can already achieve accuracies above 0.5 for 5d problems. This means that features obtained from the initial sample of points have relatively high power to differentiate between problem classes. Since different problem classes can have different ranges for the objective function values, they are easily distinguished using simple features. While the BBOB benchmark handles the scaling of decision variables (x-values) because its initial goal was to compare methods that use this information (PSO in particular), the objective function values are not scaled with the same care.

When the objective function values are scaled, the RF model achieves accuracies around 0.2-0.4 when using data from a single iteration. However, increasing the feature calculation budget diminishes the negative effect of the scaling on the model accuracy, with the highest feature calculation budget of 30 iterations providing results that are less than 0.1 lower than the non-scaled model. Using the highest feature calculation budget of 30 iterations, accuracies above 0.8 are obtained in all cases. This result is also a consequence of the benchmark problem instances (BBOB) that are involved in our experiments, which are designed to cover different problem characteristics met in practice, and not to train ML models.

Another observation is that for the ES algorithm, there is a more sudden difference in the accuracies achieved using a single iteration, and using three iterations. Using data from three iterations, the non-scaled RF model already achieves an accuracy above 0.9 for the 5d problems.

## 7.2 Comparison to Iteration-based ELA Features

Figure 5b features the results obtained for the problem classification task using ELA features calculated from each iteration. Similarly to the previous figures, the y-axis represents the obtained accuracy, while the x-axis represents the feature calculation budget (the number of iterations used to train the RF model). The right half of each violin plot (in lime color) refers to the accuracies obtained when the RF model is evaluated on trajectories obtained using the random seed used for training, while the left part represents accuracies when the model is tested on trajectories obtained with a random seed that was not used for the training data.

From the figure, we can see that when the objective function values are not scaled, the ELA features provide accuracies above 0.80 even with a feature calculation budget of one iteration. However, the scaling of the objective function values disrupts the generalization of the model to different runs of the algorithm. This is evident from the fact that the accuracy indicated by the right parts of the violin plots differs substantially from its left counterpart.

In comparison to the DynamoRep features, we can see that the ELA features provide much better results when a lower feature calculation budget is used. However, when higher feature calculation budgets are used, the DynamoRep features provide comparable results. Additionally, when the objective function values are scaled and higher feature calculation budgets are used, the DynamoRep features generalize better to trajectories obtained with different random seeds, while for the ELA features, there is a larger discrepancy between the accuracy on trajectories obtained with the same random seed on which the model was trained, and the accuracy obtained on trajectories with different random seeds.
<figure><img src="image_8.png" /><figcaption>Figure 6: Accuracies obtained with the static ELA features for the problem classification task of 5d problems</figcaption></figure>

## 7.3 Comparison to Static ELA Features

Figure 6 shows the accuracies obtained for the problem classification task using the static ELA features. The dark green violin plots indicate the results obtained without scaling the objective function values, while the light green ones represent the results obtained when the objective function values are scaled before feature calculation. In this case, we can also observe that the scaling has a detrimental effect on performance, which is especially pronounced when a small sample size is used. As the sample size increases, the features become more robust to this transformation. Compared to the accuracies obtained using the DynamoRep features, we can see that when the DynamoRep features are extracted from trajectories of the ES algorithm, the accuracies are in the similar range, while when the features are extracted from trajectories of the other algorithms, the DynamoRep features provide inferior results. Comparing to the iteration ELA approach, we can see that the results are similar, regardless whether the ELA features were calculated on a Latin Hypercube sample, or on algorithm trajectory samples. However, we have to stress that these methods are not directly comparable, since the data on which the DynamoRep and iteration ELA features are calculated is completely different than the classical ELA approach. The primary goal of the DynamoRep features is to extract useful information using samples obtained during the algorithm execution, while the ELA features requires additional function evaluations which are not used in the algorithm execution.

## 7.4 Feature Importance Analysis

To further understand the value of the DynamoRep features, we analyzed the impurity-based feature importances produced by the RF model trained for the problem classification task. Figure 7 shows the 20 most important features. The importance of each
<figure><img src="image_9.png" /><figcaption>Figure 7: Top 20 DynamoRep features with highest feature importance assigned by the RF model for the problem classification task</figcaption></figure>

feature is obtained by taking the median feature importance produced by the RF models from all folds in the 10-fold cross-validation, across all runs of all three algorithms. In order to include the importance of the features from all iterations, we take the feature importances produced by the models trained with the highest feature calculation budget, i.e., 30 iterations. It is noteworthy to mention that a model trained with a lower feature calculation budget can have different feature importance, since in this case, some of the top 20 features are extracted from the last iterations.

When the values of the objective function are not scaled (left column of the figure), all of the most important features are calculated from the values of the objective function. The standard deviation of the objective function dominates the list of most important features for all dimensions. The mean of the objective function value in the first iteration is among the most important features. However, for higher dimensions, the 20 most important features come exclusively from the standard deviation of the objective function value.

The scaling of the objective function values completely changes the feature importance. As can be seen from the columns on the right, the standard deviation of the objective function does not appear for 5d problems, and appears once for 20d problems. For 5d problems, the standard deviation of the decision variables ($x$ values) from the final five iterations appear as 5 of the 20 most important features. When scaling is applied, the most important features are most commonly the means of the objective function values. These features appear among the top 20 features 11 times for 3d and 5d problems, 15 times for 10d problems, and 16 times for 20d problems. Interestingly, these features appear as the 20 most important features when they are calculated either from the first 10 iterations or the last 10 iterations. This indicates that the most important features are the ones that show how close is the population in terms of fitness values to the individual that is determined to be the best at the end of the optimization.
Figure 7 depicts the feature importances aggregated on a dimension level and con-
tains the results for 5d problems only. This summarization is provided for a more ro-
bust analysis and to be model-agnostic with regard to the optimization algorithm. For
a more fine-grained analysis we present a similar feature analysis on an algorithm level
for all dimensions in Section 3.2 of the SM.

An analysis of the predictive power of individual statistics, as well as the accuracy
obtained using features extracted solely from the decision variables or the objective
function values, can also be found in the SM.

8 Algorithm Classification Results

In this section, we present the results from the models trained for the algorithm classification task. The section follows the same organization as Section 7.

8.1 Algorithm Classification With Limited Feature Calculation Budgets

In this subsection, we present the results of the algorithm classification task, exploring
different number of iterations used for calculating the DynamoRep features.

Figure 8 presents the results from the algorithm classification with different pre-
processing steps applied, and using different feature calculation budgets. The rows
refer to results for different dimensions, while the columns refer to different combi-
nations of preprocessing steps applied. In particular, the first two columns show the
results without differencing the DynamoRep features, while the second two columns
contain the results obtained with differencing. The results in the first and third columns
are obtained with the unscaled objective function values, while the second and fourth
columns are using the scaled objective function values.

The violin plots are split into two parts. The right side of each violin plot (lime
color) refers to the accuracies obtained when the testing data contains trajectories ob-
tained with the same random seed as the trajectories used for training. On the other
hand, the left side contains the accuracies obtained on the testing set which contains
trajectories obtained with different random seeds than the ones used for training. The
y-axis depicts the achieved classification accuracy, while the x-axis shows the feature
calculation budget, i.e., the number of iterations used to calculate the DynamoRep fea-
tures. In the case of the differenced features, it is not possible to perform experiments
with a feature calculation budget of one iteration, since at least two iterations are re-
quired to perform the differencing of the features. For this reason, the results for fea-
ture calculation budget one are omitted. In the case of the undifferenced features, the
classification accuracy obtained with a budget of one iteration is 0.33, corresponding
to random guessing. This is due to the fact that all of the algorithms start their search
with the same initial population, i.e., they are represented by the same features, and
it is therefore impossible to distinguish between them if only one iteration is used to
calculate the features. The scaling and differencing preprocessing steps have a posi-
tive impact on the classification accuracy for all dimensions, which is in contrast to the
problem classification task, where lower accuracies were obtained with these prepro-
cessing steps. This is due to the fact that the scaling of the objective function values
makes the problem classification task more difficult since the model cannot rely on the
range of the objective function values to identify the problem. On the other hand, this
range is not so relevant for the algorithm classification task, so scaling the objective
function values might prevent the model from relying on this information. When the
differencing step is applied, the algorithms can be recognized with accuracies above
0.90 for 5d problems and nearly perfect accuracy for 20d problems with a feature cal-
<figure><img src="image_10.png" /><figcaption>Figure 8: Accuracies obtained for the algorithm classification task using the DynamoRep and Iteration ELA features, across different experimental settings for 5d problems</figcaption></figure>

calculation budget higher than 10 iterations. This result also indicates that by applying differencing, the algorithm dynamics is better captured.

## 8.2 Comparison to Iteration-based ELA Features

Figure 8b shows the results for the algorithm classification task obtained with the iteration-based ELA features for 5d problems. In this case, we can see that the objective function values are not scaled and the features are not differenced (upper left plot), the models do not generalize to trajectories obtained with different random seeds, in some cases achieving accuracy as low as 0.33, corresponding to random guessing. The scaling of the objective function slightly improves the generalization ability. However, the differencing provides a much more substantial improvement to the models' accuracy. When the features are differenced and a feature calculation budget lower than 10 iterations is used, the ELA features achieve comparable performance to the DynamoRep features, while using a higher feature calculation budget, the DynamoRep features provide slightly higher accuracies.

## 8.3 Feature Importance Analysis

Figure 9 presents the top 20 features with the highest median importance, for every combination of preprocessing steps. The subplots on the left contain the most important features when the differencing is not applied, while the subplots on the right contain the most important differenced features. The subplots in the first row refer to the features obtained without scaling of the objective function values, while the subplots in the second row refer to the features calculated with scaled objective function values. The feature importance values are aggregated over all dimensions, all folds of the 10-fold cross-validation, and all models trained on algorithm trajectories executed with a
<figure><img src="image_11.png" /><figcaption>Figure 9: Top 20 DynamoRep features with highest median importance for the algorithm classification task, across all four dimensions.</figcaption></figure>
different random seed. From the figure, it is evident that almost all of the most impor-
tant features are standard deviations of the decision variables, the exception being the
minimum and maximum of the objective function value in the 29th and 1st iteration, re-
spectively, which appear among the top 20 most important features when differencing
is not applied, and the objective function values are scaled.

A comprehensive analysis of the predictive efficacy of individual statistics, along-
side the accuracy derived from features extracted exclusively from decision variables
or objective function values, is detailed in the SM.

9 Conclusion

In this paper, we conduct a comprehensive analysis of the DynamoRep features for the
tasks of algorithm selection, problem classification, and algorithm classification. We
evaluate the features' predictive power for all tasks using different feature calculation
budgets. The results indicate that the DynamoRep features can be used to distinguish
problems and algorithms for problems of different dimensions. The explored prepro-
cessing steps of scaling the objective function values and differencing the DynamoRep
features are beneficial for the algorithm classification task, but have a negative impact
on the problem classification and algorithm selection tasks. The most informative Dy-
namoRep features for the problem classification task are the standard deviations of the
objective function values (when the objective function values are not scaled) and the
means of the objective function values (when the objective function values are scaled).
For the algorithm classification task, the most informative features are generally the
standard deviations of the decision variables. A comparison with the ELA features cal-
culated on each iteration of the algorithm execution indicates that ELA features can ob-
tain slightly better performance for the algorithm selection and problem classification
tasks, at the cost of larger computational complexity, while for the algorithm classifica-
tion task, the DynamoRep features provide slightly better results with feature calcula-
tion budgets higher than 10 iterations. On the algorithm selection task, the DynamoRep
features can outperform a single best solver in the instance-out evaluation, when the
testing data comes from the same distribution as the training data. In the problem-
split evaluation, neither the DynamoRep nor the Iteration ELA features outperform the
single best solver.

As future work, the DynamoRep features can be leveraged for per-run algorithm
selection and configuration, building on the work of (Kostovska et al., 2022). Their low
computational complexity makes them especially suitable for online, dynamic algo-
rithm selection. In future work, we intend to explore a scenario where the knowledge
obtained from one algorithm is used to warm-start others, enabling algorithm selection
in a per-run setting by dynamically switching between algorithms. Another direction
for future studies is the application of DynamoRep features to larger and more diverse
problem and algorithm collections. A particularly interesting possibility is the use of
the DynamoRep features for analyzing the novelty of different algorithms by identify-
ing similarities and differences in their search behavior on different problems. They can
also be used to select a subset of algorithms with complementary behavior from a given
algorithm portfolio used by an algorithm selector. Finally, we are interested in combin-
ing problem-focused features such as ELA with DynamoRep features, to exploit their
complementarity. We also plan to further investigate the potential of iteration-based
ELA, which is a nice by-product of this study, as it can provide useful insights into
algorithm behavior.
## Acknowledgements

Funding in direct support of this work: Slovenian Research Agency: research core funding No. P2-0098, young researcher grants No. PR-12393 to GC and No. PR-11263 to GP, projects No. J2-4460 to PK and No. GC-0001, and a bilateral project between Slovenia and France grant No. BI-FR/23-24-PROTEUS-001 (PR-12040). Our work is also supported by ANR-22-ERCS-0003-01 project VARIATION.

## References

* Belkhir, N., Dréo, J., Savéant, P., and Schoenauer, M. (2017). Per instance algorithm configuration of CMA-ES with limited budget. In *Proc. of Genetic and Evolutionary Computation Conference (GECCO)*, pages 681–688. ACM.

* Blank, J. and Deb, K. (2020). Pymoo: Multi-Objective Optimization in Python. *IEEE Access*, 8:89497–89509.

* Cenikj, G., Dieter Lang, R., Petrus Engelbrecht, A., Doerr, C., Korošec, P., and Eftimov, T. (2022). SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison. In *Proc. of Genetic and Evolutionary Computation Conference (GECCO)*.

* Cenikj, G., Petelin, G., Doerr, C., Korošec, P., and Eftimov, T. (2023). DynamoRep: Trajectory-Based Population Dynamics for Classification of Black-Box Optimization Problems. In *Proc. of Genetic and Evolutionary Computation Conference (GECCO)*, GECCO '23, page 813–821. Association for Computing Machinery.

* Cenikj, G., Petelin, G., and Eftimov, T. (2024a). A cross-benchmark examination of feature-based algorithm selector generalization in single-objective numerical optimization. *Swarm and Evolutionary Computation*, 87:101534.

* Cenikj, G., Petelin, G., and Eftimov, T. (2024b). Impact of Scaling in ELA Feature Calculation on Algorithm Selection Cross-Benchmark Transferability. In *2024 IEEE Congress on Evolutionary Computation (CEC)*. In Press.

* Cenikj, G., Petelin, G., and Eftimov, T. (2024c). TransOptAS: Transformer-Based Algorithm Selection for Single-Objective Optimization.

* Corso, G., Cavalleri, L., Beaini, D., Liò, P., and Veličković, P. (2020). Principal neighbourhood aggregation for graph nets. *Advances in Neural Information Processing Systems*, 33:13260–13271.

* Derbel, B., Lsiefooghe, A., Verel, S., Aguirre, H., and Tanaka, K. (2019). New Features for Continuous Exploratory Landscape Analysis based on the SOO Tree. In *FOGA 2019 - 15th ACM/SIGEVO Workshop on Foundations of Genetic Algorithms*, pages 72–86, Potsdam, Germany. ffhal-02282986.

* Gallagher, M. and Muñoz, M. A. (2024). Towards an Improved Understanding of Features for More Interpretable Landscape Analysis. In Li, X. and Handl, J., editors, *Proc. of Genetic and Evolutionary Computation Conference Companion*, pages 135–138. ACM.

* Garden, R. W. and Engelbrecht, A. P. (2014). Analysis and classification of optimisation benchmark functions and benchmark suites. In *2014 IEEE Congress on Evolutionary Computation (CEC)*, pages 1641–1649.
Gholamalinezhad, H. and Khosravi, H. (2020). Pooling methods in deep neural networks, a review. *arXiv preprint arXiv:2009.07485*.

Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., and Brockhoff, D. (2020). COCO: A platform for comparing continuous optimizers in a black-box setting. *Optimization Methods and Software*, pages 1–31.

Hansen, N., Finck, S., Ros, R., and Auger, A. (2009). Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions. Research Report RR-6829, INRIA.

Janković, A. and Doerr, C. (2019). Adaptive Landscape Analysis. In *Proc. of Genetic and Evolutionary Computation Conference (GECCO), GECCO '19*, page 2032–2035. Association for Computing Machinery.

Jankovic, A., Eftimov, T., and Doerr, C. (2021). Towards feature-based performance regression using trajectory data. In *Proc. of Applications of Evolutionary Computation (EvoApplications)*, LNCS, pages 601–617. Springer.

Kerschke, P. and Trautmann, H. (2019a). Automated algorithm selection on continuous black-box problems by combining exploratory landscape analysis and machine learning. *Evolutionary Computation*, 27(1):99–127.

Kerschke, P. and Trautmann, H. (2019b). *Comprehensive Feature-Based Landscape Analysis of Continuous and Constrained Optimization Problems Using the R-Package Flacco*, pages 93–123. Springer.

Korošec, P. and Eftimov, T. (2024). Opt2Vec-a continuous optimization problem representation based on the algorithm’s behavior: A case study on problem classification. *Information Sciences*, 680:121134.

Kostovska, A., Jankovic, A., Vermetten, D., De Nobel, J., Wang, H., Eftimov, T., and Doerr, C. (2022). *Per-run Algorithm Selection with Warm-starting using Trajectory-based Features*, volume 13398 of LNCS, pages 46–60. Springer.

Lang, R. D. and Engelbrecht, A. P. (2021). An Exploratory Landscape Analysis-Based Benchmark Suite. *Algorithms*, 14(3).

McGonigle, E. T., Killick, R., and Nunes, M. A. (2022). Modelling time-varying first and second-order structure of time series via wavelets and differencing. *Electronic Journal of Statistics*, 16(2):4398 – 4448.

Menčík, J. (2016). Latin Hypercube Sampling. In Mencik, J., editor, *Concise Reliability for Engineers*, chapter 16. IntechOpen, Rijeka.

Mersmann, O., Bischl, B., Trautmann, H., Preuss, M., Weihs, C., and Rudolph, G. (2011). Exploratory landscape analysis. In *Proc. of Genetic and Evolutionary Computation Conference (GECCO)*, pages 829–836.

Morgan, R. and Gallagher, M. (2014). Sampling techniques and distance metrics in high dimensional continuous landscape analysis: Limitations and improvements. *IEEE Trans. Evol. Comput.*, 18(3):456–461.

Muñoz, M. A. and Smith-Miles, K. (2020). Generating new space-filling test instances for continuous black-box optimization. *Evolutionary Computation*, 28:379–404.
Ochoa, G., Malan, K. M., and Blum, C. (2021). Search trajectory networks: A tool for analysing and visualising the behaviour of metaheuristics. *Applied Soft Computing*, 109:107492.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. *Journal of Machine Learning Research*, 12:2825–2830.

Petelin, G., Cenikj, G., and Eftimov, T. (2022). TLA: Topological Landscape Analysis for Single-Objective Continuous Optimization Problem Instances. In *Proceedings of IEEE Symposium Series On Computational Intelligence*, pages 1698–1705.

Prager, R. P. and Trautmann, H. (2023). Nullifying The Inherent Bias Of Non-Invariant Exploratory Landscape Analysis Features. In *Applications of Evolutionary Computation*, page 411–425, Berlin, Heidelberg. Springer-Verlag.

Pulatov, D., Anastacio, M., Kotthoff, L., and Hoos, H. (2022). Opening the black box: Automated software analysis for algorithm selection. In *Proceedings of the First International Conference on Automated Machine Learning*, volume 188 of *Proceedings of Machine Learning Research*, pages 6/1–18. PMLR.

Renau, Q., Doerr, C., Dreo, J., and Doerr, B. (2020). Exploratory Landscape Analysis is Strongly Sensitive to the Sampling Strategy. In *Proc. of Parallel Problem Solving from Nature (PPSN)*, pages 139–153.

Seiler, M., Škvorc, U., Cenikj, G., Doerr, C., and Trautmann, H. (2024a). Learned Features vs. Classical ELA on Affine BBOB Functions. In Affenzeller, M., Winkler, S. M., Kononova, A. V., Trautmann, H., Tušar, T., Machado, P., and Bäck, T., editors, *Parallel Problem Solving from Nature – PPSN XVIII*, pages 137–153. Springer Nature Switzerland.

Seiler, M. V., Kerschke, P., and Trautmann, H. (2024b). Deep-ELA: Deep Exploratory Landscape Analysis with Self-Supervised Pretrained Transformers for Single-and Multi-Objective Continuous Optimization Problems. *arXiv preprint arXiv:2401.01192*.

Seiler, M. V., Prager, R. P., Kerschke, P., and Trautmann, H. (2022). A collection of deep learning-based feature-free approaches for characterizing single-objective continuous fitness landscapes. In *Proc. of Genetic and Evolutionary Computation Conference (GECCO)*, GECCO '22, page 657–665, New York, NY, USA. Association for Computing Machinery.

Shwartz-Ziv, R. and Armon, A. (2022). Tabular data: Deep learning is not all you need. *Information Fusion*, 81:84–90.

van Stein, B., Long, F. X., Frenzel, M., Krause, P., Gitterle, M., and Bäck, T. (2023). DoE2Vec: Deep-Learning Based Features for Exploratory Landscape Analysis. In *Proceedings of the Companion Conference on Genetic and Evolutionary Computation, GECCO '23 Companion*, page 515–518, New York, NY, USA. Association for Computing Machinery.

Škvorc, U., Eftimov, T., and Korošec, P. (2020). Understanding the problem space in single-objective numerical optimization using exploratory landscape analysis. *Applied Soft Computing*, 90:106138.
Škvorc, U., Eftimov, T., and Korošec, P. (2021). The effect of sampling methods on the invariance to function transformations when using exploratory landscape analysis. In *2021 IEEE Congress on Evolutionary Computation (CEC)*, pages 1139–1146.