# Benchmarking CMA-ES with Basic Integer Handling on a Mixed-Integer Test Problem Suite

Tristan Marty, Yann Semet, Anne Auger, Sébastien Héron, Nikolaus Hansen

► To cite this version:

Tristan Marty, Yann Semet, Anne Auger, Sébastien Héron, Nikolaus Hansen. Benchmarking CMA-ES with Basic Integer Handling on a Mixed-Integer Test Problem Suite. GECCO 2023 Companion - Companion Conference on Genetic and Evolutionary Computation, Jul 2023, Lisbon, Portugal. pp.1628-1635, 10.1145/3583133.3596411. hal-04404108

HAL Id: hal-04404108

https://inria.hal.science/hal-04404108

Submitted on 24 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Benchmarking CMA-ES with Basic Integer Handling on a
Mixed-Integer Test Problem Suite

Tristan Marty

Thales Research and Technology,
Inria and Institut Polytechnique

Yann Semet

Thales Research and Technology

Anne Auger

Inria and Institut Polytechnique

Sébastion Héron

Thales Research and Technology

Nikolaus Hansen

Inria and Institut Polytechnique

## ABSTRACT

We compare the performances of one implementation of CMA-ES (pycma version 3.3.0) for optimizing functions with both continuous and integer variables. The implementation incorporates a lower bound on the variance along the integer coordinates to keep the optimization from stalling. This benchmark will serve as a baseline for further works on pycma. Results show substantial improvement since the last benchmarked version of pycma. Also this implementation is competitive with other mixed integer algorithms.

## CCS CONCEPTS

*Computing methodologies → Continuous space search;

## KEYWORDS

Benchmarking, Black-box optimization, Mixed-integer optimization

## ACM Reference format:

Tristan Marty, Yann Semet, Anne Auger, Sébastion Héron, and Nikolaus Hansen. 2023. Benchmarking CMA-ES with Basic Integer Handling on a Mixed-Integer Test Problem Suite. In Proceedings of The Genetic and Evolutionary Computation Conference 2023, Lisbon, Portugal, July 15–19, 2023 (GECCO ’23), 8 pages.

DOI: 10.1145/3583133.3596411

# 1 INTRODUCTION

Mixed optimization problems are of great importance in real-world applications. Typically, variables include real-valued degrees of freedom that correspond, directly or not, to physical measurements (e.g. angles, frequencies, forces, etc.) as well as discrete (integer or binary) settings that correspond to categorical choices (e.g. materials) or service activations (e.g. radar modes). Therefore, developing algorithmic variants that can handle several types of optimization variable at the same time without impairing the search process will be very useful in practice.

In the context of black box optimization for problems with continuous variables, evolutionary algorithms which update a covariance matrix, namely CMA-ES, have been studied for more than 20 years

[10]. While the CMA-ES was originally not designed to handle integer variables, it can be employed with integer variables by using the integer part of a continuous variable in the fitness evaluation. This is however expected to lead to integer variables getting stuck in the resulting fitness plateaus, and several versions have been proposed to improve the algorithm in this situation [4, 5].

In order to study mixed integer problems within the COCO framework [9], the bbob-mixint testbed was proposed in [13]. This testbed is based on the 24 functions of the bbob testbed with 80% of the variables to be integer with an affinity between 2 and 16.

In order to prevent variables getting stuck on the same integer level, for example in CMA-ES, a lower bound on the variance can be set such that there remains a reasonable probability for an integer variable to “jump” to another integer level, as for example in CMA-ES with margin (CMA-ESwM) [4]. The same principle is implemented in the pycma code with Integer Handling (cma-IH in the benchmark) with a more simplistic approach.

The purpose of this paper is to measure the performance of the current implementation of the widely distributed cma-IH code (version 3.3.0) [6] with different options and compare its performance with previous results and enhancements.

# 2 ALGORITHM PRESENTATION

The general idea behind cma-IH is to set a lower bound for the variance of the integer variables. It prevents integer variables from being trapped in one value. We formulate the bounding using the covariance matrix decomposition from the CMA-ES with diagonal decoding [1]. At each new iteration $t + 1$, new candidate solutions, $x^{(t+1)}$, are sampled according to a multivariate normal distribution.

$$x^{(t+1)} \sim N(m, \sigma^{(t)} C^{(t)}) \\ \sim m + \sigma^{(t)} \sqrt{C^{(t)}} N(0, I),$$

where C is a covariance matrix and σ is the step size. In order to bound the standard deviation of each integer variable, we introduce the diagonal scaling matrix D,

$$x^{(t+1)} \sim m + \sigma^{(t)} D^{(t)} \sqrt{C^{(t)}} N(0, I),$$

where $D^{(0)}$ is the identity. The update of σ and C are done as in the original CMA-ES algorithm. Then, the diagonal matrix D is updated based on the current standard deviation in each coordinate. The standard deviation of the $t^{th}$-th coordinate of $x$, $\sigma_{\text{std}(i)}^{(t)}$, is computed with the diagonal elements of C and D:

<footnote>Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).</footnote>

<footnote>GECCO ’23, Lisbon, Portugal</footnote>

<footnote>© 2023 Copyright held by the owner/author(s). 979-8-4007-0120-7/23/07...$15.00</footnote>

<footnote>DOI: 10.1145/3583133.3596411</footnote>
# Benchmarking CMA-ES with Basic Integer Handling on a
Mixed-Integer Test Problem Suite

Tristan Marty

Thales Research and Technology,
Inria and Institut Polytechnique

Yann Semet

Thales Research and Technology

Anne Auger

Inria and Institut Polytechnique

Sébastion Héron

Thales Research and Technology

Nikolaus Hansen

Inria and Institut Polytechnique

## ABSTRACT

We compare the performances of one implementation of CMA-ES (pycma version 3.3.0) for optimizing functions with both continuous and integer variables. The implementation incorporates a lower bound on the variance along the integer coordinates to keep the optimization from stalling. This benchmark will serve as a baseline for further works on pycma. Results show substantial improvement since the last benchmarked version of pycma. Also this implementation is competitive with other mixed integer algorithms.

## CCS CONCEPTS

*Computing methodologies → Continuous space search;

## KEYWORDS

Benchmarking, Black-box optimization, Mixed-integer optimization

## ACM Reference format:

Tristan Marty, Yann Semet, Anne Auger, Sébastion Héron, and Nikolaus Hansen. 2023. Benchmarking CMA-ES with Basic Integer Handling on a Mixed-Integer Test Problem Suite. In *Proceedings of The Genetic and Evolutionary Computation Conference 2023, Lisbon, Portugal, July 15–19, 2023 (GECCO ’23)*, 8 pages.

DOI: 10.1145/3583133.3596411

# 1 INTRODUCTION

Mixed optimization problems are of great importance in real-world applications. Typically, variables include real-valued degrees of freedom that correspond, directly or not, to physical measurements (e.g. angles, frequencies, forces, etc.) as well as discrete (integer or binary) settings that correspond to categorical choices (e.g. materials) or service activations (e.g. radar modes). Therefore, developing algorithmic variants that can handle several types of optimization variable at the same time without impairing the search process will be very useful in practice.

In the context of black box optimization for problems with continuous variables, evolutionary algorithms which update a covariance matrix, namely CMA-ES, have been studied for more than 20 years

[10]. While the CMA-ES was originally not designed to handle integer variables, it can be employed with integer variables by using the integer part of a continuous variable in the fitness evaluation. This is however expected to lead to integer variables getting stuck in the resulting fitness plateaus, and several versions have been proposed to improve the algorithm in this situation [4, 5].

In order to study mixed integer problems within the COCO framework [9], the bbob-mixint testbed was proposed in [13]. This testbed is based on the 24 functions of the bbob testbed with 80% of the variables to be integer with an affinity between 2 and 16.

In order to prevent variables getting stuck on the same integer level, for example in CMA-ES, a lower bound on the variance can be set such that there remains a reasonable probability for an integer variable to “jump” to another integer level, as for example in CMA-ES with margin (CMA-ESwM) [4]. The same principle is implemented in the pycma code with Integer Handling (cma-IH in the benchmark) with a more simplistic approach.

The purpose of this paper is to measure the performance of the current implementation of the widely distributed cma-IH code (version 3.3.0) [6] with different options and compare its performance with previous results and enhancements.

# 2 ALGORITHM PRESENTATION

The general idea behind cma-IH is to set a lower bound for the variance of the integer variables. It prevents integer variables from being trapped in one value. We formulate the bounding using the covariance matrix decomposition from the CMA-ES with diagonal decoding [1]. At each new iteration $t + 1$, new candidate solutions, $x^{(t+1)}$, are sampled according to a multivariate normal distribution.

$$x^{(t+1)} \sim N(m, \sigma^{(t)} C^{(t)}) \\ \sim m + \sigma^{(t)} \sqrt{C^{(t)}} N(0, I),$$

where C is a covariance matrix and σ is the step size. In order to bound the standard deviation of each integer variable, we introduce the diagonal scaling matrix D,

$$x^{(t+1)} \sim m + \sigma^{(t)} D^{(t)} \sqrt{C^{(t)}} N(0, I),$$

where $D^{(0)}$ is the identity. The update of σ and C are done as in the original CMA-ES algorithm. Then, the diagonal matrix D is updated based on the current standard deviation in each coordinate. The standard deviation of the $t^{th}$-th coordinate of $x, \sigma_{\text{std}(i)}^{(t)}$, is computed with the diagonal elements of C and D:

<footnote>Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).</footnote>

<footnote>GECCO ’23, Lisbon, Portugal</footnote>

<footnote>© 2023 Copyright held by the owner/author(s). 979-8-4007-0120-7/23/07...$15.00</footnote>

<footnote>DOI: 10.1145/3583133.3596411</footnote>
```python
import cocoex
import cma
suite = cocoex.Suite('bbob-mixint', '', '')
for problem in suite:
    up_bound = list(problem.upperBounds[
        :problem.number_of_integer_variables]) +
               [None]
    lo_bound = list(problem.lowerBounds[
        :problem.number_of_integer_variables]) +
               [None]
    cma.fmin2(problem, problem initial solution, 1,
               {'bounds': [lo_bound, up_bound]
                'CMA_stds': (problem.upperBounds
                             - problem.lowerBounds)/5,
                'integer_variables': list(range(
                    problem.number_of_integer_variables)),
                restarts =9)}
```

<caption>Figure 1: Calling code to benchmark cma.fmin2.</caption>

$$ \sigma_{\text{std}(i)}^{(t)} = \sigma^{(t)} D_{i,i}^{(t)} \sqrt{C_{i,i}^{(t)}} $$

This standard deviation has to be compared with a lower bound $\sigma_{LB}$. If the standard deviation in the $i^{th}$ coordinate is smaller than the threshold value, $\sigma_{\text{std}(i)}^{(t)} < \sigma_{LB}$, then the D matrix is scaled up in this coordinate.

$$ D_{i,i}^{(t+1)} \leftarrow \begin{cases} \frac{\sigma_{LB}}{\sigma^{(t)} \sqrt{C_{i,i}^{(t)}}} & \text{if } 1 \le i \le N_{\text{int}} \text{ and } \sigma_{\text{std}(i)}^{(t)} \le \sigma_{LB} \\ D_{i,i}^{(t)} & \text{otherwise} \end{cases} \quad (1) $$

where $N_{\text{int}}$ is the number of integer variables. In the version 3.3.0 of pycma with integer handling (cma-IH), the lower bound is set to:

$$ \sigma_{LB} = \frac{1}{2N_{\text{int}} + 1} \quad (2) $$

### 3 EXPERIMENTAL PROCEDURE

Figure 1 shows the calling code and options used to run the benchmark for cma-IH. A maximum of nine restarts may be performed and each restart doubles the population size starting from the default (IPOP-CMA-ES) [2]. The initial step size is chosen to sample solutions mainly within the upper and the lower bound of the problem. The bounds are enforced on the integer variables only. The maximum number of function evaluations allowed per problem is set to $10^4$ times the number of dimension of the problem.

This current implementation cma-IH is compared to performance data from the bbob-mixint COCO archive<fnref n="1" />. We show results of the algorithms given in Table 1.

The experiments were performed and the plots were produced with COCO [9], version 2.6.3.

### 4 CPU TIMING

In order to evaluate the CPU timing of the algorithm, we have run the cma-IH algorithm with restarts on the bbob-mixint test

<table><caption>Table 1: Compared algorithms</caption><tr><td>cma-IH</td><td>pycma version 3.3 (current) with the default integer handling option</td></tr><tr><td>CMA-ES-pycma</td><td>previous integer handling of pycma benchmarked in [13]</td></tr><tr><td>CMA-ESwM</td><td>CMA-ES with margin benchmarked in [3]</td></tr><tr><td>cma-NoIH</td><td>pycma version 3.3 (current) without any integer handling option</td></tr><tr><td>cmaIH1e-1</td><td>as cma-IH with a larger coordinate-wise lower bound on the sample standard deviation of σ<sub>LB</sub> = 10<sup>-1</sup></td></tr><tr><td>DE-scipy</td><td>Differential Evolution implemented in scipy benchmarked in [13]</td></tr><tr><td>RANDOMSEARCH</td><td>Random Search algorithm for baseline</td></tr></table>

suite [13] with a maximum budget of $10^4 \times$ dimensions function evaluations according to [11].

The Python code was run on a Linux machine with 32 cores, Intel® Xeon® E7 to T3 v4 processor. The time per function evaluation averaged over all functions in the test bed for dimensions 5, 10, 20, 40, 80 and 160 equals 3.7, 4.9, 5.1, 5.6, 6.4 and $8.4 \times 10^{-4}$ seconds respectively.

### 5 RESULTS

We show results from experiments according to [11] and [8] on the benchmark functions given in [13]. Figures 2 and 3 show summary results for the different function groups. Figure 4 shows the empirical distribution of function evaluations to reach 51 target values on the single functions [9]. Tables 2 and 3 present expected runtimes and statistical tests.<fnref n="2" />

The **expected runtime (ERT)**, used in the tables, depends on a given target precision, $I_{\text{target}} = f_{\text{opt}} + \Delta f$, and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach $f_t$, summed over all trials and divided by the number of trials that actually reached $f_t$ [11, 12]. **Statistical significance** is tested with the rank-sum test for a given target $\Delta f_t$ using, for each trial, either the number of needed function evaluations to reach $f_t$ (inverted and multiplied by -1), or, if the target was not reached, the best $\Delta f$-value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration.

Comparing two integer handling versions of pycma. We compare the performance of cma-IH (the current default implementation) versus CMA-ES-pycma (an older implementation). In comparison, the cma-IH performs increasingly better with increasing dimension and is overall about ten times faster than CMA-ES-pycma in dimension 80 (see Figure 3, lower right). The largest differences we observe on the ill-conditioned and multi-modal functions. The difference on the linear slope function is mainly due to the boundary handling which was applied also to the continuous variables in the CMA-ES-pycma experiment, as we confirmed in additional

<footnote n="1">¹See https://numbbo.github.io/data-archive</footnote>

<footnote n="2">²The entire data can be found at https://trmarty.github.io/MixedIntegerData/</footnote>
```python
import cocoex
import cma
suite = cocoex.Suite('bbob-mixint', '', '')
for problem in suite:
    up_bound = list(problem.upperBounds[
        :problem.number_of_integer_variables]) +
               [None]
    lo_bound = list(problem.lowerBounds[
        :problem.number_of_integer_variables]) +
               [None]
    cma.fmin2(problem, problem initial solution, 1,
               {'bounds': [lo_bound, up_bound]
                'CMA_stds': (problem.upperBounds
                             - problem.lowerBounds)/5,
                'integer_variables': list(range(
                    problem.number_of_integer_variables)),
                restarts =9)}
```

<caption>Figure 1: Calling code to benchmark cma.fmin2.</caption>

$$ \sigma_{\text{std}(i)}^{(t)} = \sigma^{(t)} D_{i,i}^{(t)} \sqrt{C_{i,i}^{(t)}} $$

This standard deviation has to be compared with a lower bound $\sigma_{LB}$. If the standard deviation in the $i^{th}$ coordinate is smaller than the threshold value, $\sigma_{\text{std}(i)}^{(t)} < \sigma_{LB}$, then the D matrix is scaled up in this coordinate.

$$ D_{i,i}^{(t+1)} \leftarrow \begin{cases} \frac{\sigma_{LB}}{\sigma^{(t)} \sqrt{C_{i,i}^{(t)}}} & \text{if } 1 \le i \le N_{\text{int}} \text{ and } \sigma_{\text{std}(i)}^{(t)} \le \sigma_{LB} \\ D_{i,i}^{(t)} & \text{otherwise} \end{cases} \quad (1) $$

where $N_{\text{int}}$ is the number of integer variables. In the version 3.3.0 of pycma with integer handling (cma-IH), the lower bound is set to:

$$ \sigma_{LB} = \frac{1}{2N_{\text{int}} + 1} \quad (2) $$

## 3 EXPERIMENTAL PROCEDURE

Figure 1 shows the calling code and options used to run the benchmark for cma-IH. A maximum of nine restarts may be performed and each restart doubles the population size starting from the default (IPOP-CMA-ES) [2]. The initial step size is chosen to sample solutions mainly within the upper and the lower bound of the problem. The bounds are enforced on the integer variables only. The maximum number of function evaluations allowed per problem is set to $10^4$ times the number of dimension of the problem.

This current implementation cma-IH is compared to performance data from the bbob-mixint COCO archive<fnref n="1" />. We show results of the algorithms given in Table 1.

The experiments were performed and the plots were produced with COCO [9], version 2.6.3.

## 4 CPU TIMING

In order to evaluate the CPU timing of the algorithm, we have run the cma-IH algorithm with restarts on the bbob-mixint test

<table><caption>Table 1: Compared algorithms</caption><tr><td>cma-IH</td><td>pycma version 3.3 (current) with the default integer handling option</td></tr><tr><td>CMA-ES-pycma</td><td>previous integer handling of pycma benchmarked in [13]</td></tr><tr><td>CMA-ESwM</td><td>CMA-ES with margin benchmarked in [3]</td></tr><tr><td>cma-NoIH</td><td>pycma version 3.3 (current) without any integer handling option</td></tr><tr><td>cmaIH1e-1</td><td>as cma-IH with a larger coordinate-wise lower bound on the sample standard deviation of σ<sub>LB</sub> = 10<sup>-1</sup></td></tr><tr><td>DE-scipy</td><td>Differential Evolution implemented in scipy benchmarked in [13]</td></tr><tr><td>RANDOMSEARCH</td><td>Random Search algorithm for baseline</td></tr></table>

suite [13] with a maximum budget of $10^4 \times$ dimensions function evaluations according to [11].

The Python code was run on a Linux machine with 32 cores, Intel® Xeon® E7 to T3 v4 processor. The time per function evaluation averaged over all functions in the test bed for dimensions 5, 10, 20, 40, 80 and 160 equals 3.7, 4.9, 5.1, 5.6, 6.4 and $8.4 \times 10^{-4}$ seconds respectively.

## 5 RESULTS

We show results from experiments according to [11] and [8] on the benchmark functions given in [13]. Figures 2 and 3 show summary results for the different function groups. Figure 4 shows the empirical distribution of function evaluations to reach 51 target values on the single functions [9]. Tables 2 and 3 present expected runtimes and statistical tests.<fnref n="2" />

The **expected runtime (ERT)**, used in the tables, depends on a given target precision, $I_{\text{target}} = f_{\text{opt}} + \Delta f$, and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach $f_t$, summed over all trials and divided by the number of trials that actually reached $f_t$ [11, 12]. **Statistical significance** is tested with the rank-sum test for a given target $\Delta f_t$ using, for each trial, either the number of needed function evaluations to reach $f_t$ (inverted and multiplied by -1), or, if the target was not reached, the best $\Delta f$-value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration.

Comparing two integer handling versions of pycma. We compare the performance of cma-IH (the current default implementation) versus CMA-ES-pycma (an older implementation). In comparison, the cma-IH performs increasingly better with increasing dimension and is overall about ten times faster than CMA-ES-pycma in dimension 80 (see Figure 3, lower right). The largest differences we observe on the ill-conditioned and multi-modal functions. The difference on the linear slope function is mainly due to the boundary handling which was applied also to the continuous variables in the CMA-ES-pycma experiment, as we confirmed in additional

<footnote n="1">¹See https://numbbo.github.io/data-archive</footnote>

<footnote n="2">²The entire data can be found at https://trmarty.github.io/MixedIntegerData/</footnote>
experiments (not shown). The cma-IH performs statistically significantly better on 6, 8, 10, 16 and 15 functions in dimension 5, 10, 20, 40 and 80, respectively. The CMA-ES-pycma performs better on the separable ellipsoid function f2 in dimension 20, 40 and 80.

*Comparing pycma with and without the integer handling.* The cma-IH and cma-NoIH perform very similar overall. The only statistically significant difference we observed is on the separable ellipsoid function f2 in dimension 5 (not shown), where cma-IH is about three times faster than cma-NoIH, probably due to preventing additional restarts before to reach better targets. The default lower bound on the sample standard deviations for integer variables has overall little impact on the performance on this test suite.

*A larger lower bound for the standard deviations.* In order to understand the impact of the choice of the standard deviation lower bound value, we have run experiments with a few different bound values and show the results for the lower bound set to $10^{-1}$. While in the smaller dimensions the effects are minor, in larger dimension the cmaIH1e-1 performs overall best and is three times faster than cma-IH on the ill-conditioned functions in larger dimension for budgets above 100 times dimension. On the 80-dimensional sphere function, cmaIH1e-1 solves the last target ten times quicker than cma-IH and also two times quicker than CMA-ESwM. The cmaIH1e-1 also solves the *Ellipsoid separable* whereas cma-IH gets stuck at around 20% of targets. The cmaIH1e-1 is slower than CMA-ESwM by a factor of approximately 3 on functions 5, 13 and 14, however it is significantly better than the current version cma-IH on functions 2, 5 or 12 while not being worse on all other functions. For this range of dimensions, bounding the standard deviation to $10^{-1}$ is a simple yet effective solution.

# 6 CONCLUSION

We have assessed the integer handling implemented in the current pycma module [7] (version 3.3.0). The performance with integer handling is overall superior to a previously benchmarked version, however it behaves quite similar to the version without integer handling. We attribute the latter observation to a suboptimal parameter setting: tests with a lower bound for the sample standard

deviations of 1/10, which is considerably larger than the original setting, reveal a remarkably improved performance on a number of functions. In particular in larger dimension, this variant performs overall best.

# REFERENCES

[1] Y. Akimoto and N. Hansen. 2020. Diagonal Acceleration for Covariance Matrix Adaptation Evolution Strategies. *Evolutionary Computation* 28, 3 (09 2020), 405–435. DOI: http://dx.doi.org/10.1162/evco.a.00260

[2] A. Auger and N. Hansen. 2005. A restart CMA evolution strategy with increasing population size. In *2005 IEEE Congress on Evolutionary Computation*, Vol. 2. 1769–1776 Vol. 2. DOI: http://dx.doi.org/10.1109/CEC.2005.1554902

[3] Ryoki Hamano, Shota Saito, Masahiro Nomura, and Shinichi Shirakawa. 2022. Benchmarking CMA-ES with Margin on the Bbob-Mixint Testbed. In *Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO '22)*. Association for Computing Machinery, New York, NY, USA, 1708–1716. DOI: http://dx.doi.org/10.1145/3520304.3534043

[4] Ryoki Hamano, Shota Saito, Masahiro Nomura, and Shinichi Shirakawa. 2022. CMA-ES with margin: lower-bounding marginal probability for mixed-integer black-box optimization. *Proceedings of the Genetic and Evolutionary Computation Conference (2022)*.

[5] Nikolaus Hansen. 2011. *A CMA-ES for Mixed-Integer Nonlinear Optimization*. Research Report RR-7751. INRIA. https://hal.inria.fr/inria-00629689

[6] Nikolaus Hansen. 2022. CMA-ES/pycma 3.3.0. https://github.com/CMA-ES/pycma/tree/development. (2022). commit e762ac0.

[7] Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. 2019. CMA-ES/pycma on Github. Zenodo. DOI:10.5281/zenodo.2559634. (Feb. 2019). DOI: http://dx.doi.org/10.5281/zenodo.2559634

[8] N. Hansen, A. Auger, D. Brockhoff, D. Tušar, and T. Tušar. 2016. COCO: Performance Assessment. *ArXiv e-prints* arXiv:1605.03560 (2016).

[9] N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. *Optimization Methods and Software* 36 (2021), 114–144. Issue 1. DOI: http://dx.doi.org/10.1080/10556788.2020.1808977

[10] N. Hansen and A. Ostermeier. 1996. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In *Proceedings of IEEE International Conference on Evolutionary Computation*. 312–317. DOI: http://dx.doi.org/10.1109/ICEC.1996.542381

[11] N. Hansen, T. Tušar, O. Mersmann, A. Auger, and D. Brockhoff. 2016. COCO: The Experimental Procedure. *ArXiv e-prints* arXiv:1603.08776 (2016).

[12] Kenneth Price. 1997. Differential evolution vs. the functions of the second ICEO. In *Proceedings of the IEEE International Congress on Evolutionary Computation*. IEEE, Piscataway, NJ, USA, 153–157. DOI: http://dx.doi.org/10.1109/ICEC.1997.592287

[13] Tea Tušar, Dimo Brockhoff, and Nikolaus Hansen. 2019. Mixed-Integer Benchmark Problems for Single- and Bi-Objective Optimization. In *Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '19)*. Association for Computing Machinery, New York, NY, USA, 718–726. DOI: http://dx.doi.org/10.1145/3321707.3321868
experiments (not shown). The cma-IH performs statistically significantly better on 6, 8, 10, 16 and 15 functions in dimension 5, 10, 20, 40 and 80, respectively. The CMA-ES-pycma performs better on the separable ellipsoid function f2 in dimension 20, 40 and 80.

*Comparing pycma with and without the integer handling.* The cma-IH and cma-NoIH perform very similar overall. The only statistically significant difference we observed is on the separable ellipsoid function f2 in dimension 5 (not shown), where cma-IH is about three times faster than cma-NoIH, probably due to preventing additional restarts before to reach better targets. The default lower bound on the sample standard deviations for integer variables has overall little impact on the performance on this test suite.

*A larger lower bound for the standard deviations.* In order to understand the impact of the choice of the standard deviation lower bound value, we have run experiments with a few different bound values and show the results for the lower bound set to $10^{-1}$. While in the smaller dimensions the effects are minor, in larger dimension the cmaIH1e-1 performs overall best and is three times faster than cma-IH on the ill-conditioned functions in larger dimension for budgets above 100 times dimension. On the 80-dimensional sphere function, cmaIH1e-1 solves the last target ten times quicker than cma-IH and also two times quicker than CMA-ESwM. The cmaIH1e-1 also solves the *Ellipsoid separable* whereas cma-IH gets stuck at around 20% of targets. The cmaIH1e-1 is slower than CMA-ESwM by a factor of approximately 3 on functions 5, 13 and 14, however it is significantly better than the current version cma-IH on functions 2, 5 or 12 while not being worse on all other functions. For this range of dimensions, bounding the standard deviation to $10^{-1}$ is a simple yet effective solution.

# 6 CONCLUSION

We have assessed the integer handling implemented in the current pycma module [7] (version 3.3.0). The performance with integer handling is overall superior to a previously benchmarked version, however it behaves quite similar to the version without integer handling. We attribute the latter observation to a suboptimal parameter setting: tests with a lower bound for the sample standard

deviations of 1/10, which is considerably larger than the original setting, reveal a remarkably improved performance on a number of functions. In particular in larger dimension, this variant performs overall best.

# REFERENCES

[1] Y. Akimoto and N. Hansen. 2020. Diagonal Acceleration for Covariance Matrix Adaptation Evolution Strategies. *Evolutionary Computation* 28, 3 (09 2020), 405–435. DOI: http://dx.doi.org/10.1162/evco.a.00260

[2] A. Auger and N. Hansen. 2005. A restart CMA evolution strategy with increasing population size. In *2005 IEEE Congress on Evolutionary Computation*, Vol. 2. 1769–1776 Vol. 2. DOI: http://dx.doi.org/10.1109/CEC.2005.1554902

[3] Ryoki Hamano, Shota Saito, Masahiro Nomura, and Shinichi Shirakawa. 2022. Benchmarking CMA-ES with Margin on the Bbob-Mixint Testbed. In *Proceedings of the Genetic and Evolutionary Computation Conference Companion (GECCO '22)*. Association for Computing Machinery, New York, NY, USA, 1708–1716. DOI: http://dx.doi.org/10.1145/3520304.3534043

[4] Ryoki Hamano, Shota Saito, Masahiro Nomura, and Shinichi Shirakawa. 2022. CMA-ES with margin: lower-bounding marginal probability for mixed-integer black-box optimization. *Proceedings of the Genetic and Evolutionary Computation Conference (2022)*.

[5] Nikolaus Hansen. 2011. *A CMA-ES for Mixed-Integer Nonlinear Optimization*. Research Report RR-7751. INRIA. https://hal.inria.fr/inria-00629689

[6] Nikolaus Hansen. 2022. CMA-ES/pycma 3.3.0. https://github.com/CMA-ES/pycma/tree/development. (2022). commit e762ac0.

[7] Nikolaus Hansen, Youhei Akimoto, and Petr Baudis. 2019. CMA-ES/pycma on Github. Zenodo. DOI:10.5281/zenodo.2559634. (Feb. 2019). DOI: http://dx.doi.org/10.5281/zenodo.2559634

[8] N. Hansen, A. Auger, D. Brockhoff, D. Tušar, and T. Tušar. 2016. COCO: Performance Assessment. *ArXiv e-prints* arXiv:1605.03560 (2016).

[9] N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. *Optimization Methods and Software* 36 (2021), 114–144. Issue 1. DOI: http://dx.doi.org/10.1080/10556788.2020.1808977

[10] N. Hansen and A. Ostermeier. 1996. Adapting arbitrary normal mutation distributions in evolution strategies: the covariance matrix adaptation. In *Proceedings of IEEE International Conference on Evolutionary Computation*. 312–317. DOI: http://dx.doi.org/10.1109/ICEC.1996.542381

[11] N. Hansen, T. Tušar, O. Mersmann, A. Auger, and D. Brockhoff. 2016. COCO: The Experimental Procedure. *ArXiv e-prints* arXiv:1603.08776 (2016).

[12] Kenneth Price. 1997. Differential evolution vs. the functions of the second ICEO. In *Proceedings of the IEEE International Congress on Evolutionary Computation*. IEEE, Piscataway, NJ, USA, 153–157. DOI: http://dx.doi.org/10.1109/ICEC.1997.592287

[13] Tea Tušar, Dimo Brockhoff, and Nikolaus Hansen. 2019. Mixed-Integer Benchmark Problems for Single- and Bi-Objective Optimization. In *Proceedings of the Genetic and Evolutionary Computation Conference (GECCO '19)*. Association for Computing Machinery, New York, NY, USA, 718–726. DOI: http://dx.doi.org/10.1145/3321707.3321868
<table><caption>Table 2: Expected runtime (ERT) to reach given targets, measured in number of f-evaluations, in dimension 40. For each function, the ERT and, in braces Δf-values as shown in the top row. #succ is the number of trials that reached the last target fopt + 10⁻⁸. The median number of conducted function evaluations is additionally given in italics, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with p = 0.05 or p = 10⁻⁴ when the number k following the star is larger than 1, with Bonferroni correction by the number of functions (24). Best results are printed in bold. Data produced with COCO v2.6.3</caption><thead><tr><th colspan="10">Δ<sub>opt</sub></th><th colspan="10">Δ<sub>opt</sub></th></tr><tr><th></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th><th></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th></tr></thead><tbody><tr><td>cma-H</td><td>818(133)</td><td>2349(137)</td><td>7769(8446)</td><td>8453(8858)</td><td>9602(9016)</td><td>1.2e4(9556)</td><td>1.5e4(144)</td><td>15/15</td><td>cma-H</td><td>7429(8778)</td><td>2.2e4(2e4)</td><td>2.6e4(2e4)</td><td>2.9e4(2e4)</td><td>3.2e4(3e4)</td><td>3.9e4(3e4)</td><td>4.0e4(3e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>1.5e4(2e4)</td><td>4.2e4(5e4)</td><td>4.2e4(6e4)</td><td>4.5e4(5e4)</td><td>5.3e4(5e4)</td><td>7.0e4(5e4)</td><td>8.7e4(5e4)</td><td>15/15</td><td>CMA-ES</td><td>7.1e4(7e4)</td><td>9.3e4(9e4)</td><td>1.1e5(9e4)</td><td>1.2e5(9e4)</td><td>1.4e5(8e4)</td><td>1.9e5(7e4)</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>2239(849)</td><td>3993(2022)</td><td>4522(2228)</td><td>5079(2058)</td><td>5992(1916)</td><td>8350(1843)</td><td>1.1e4(1698)</td><td>15/15</td><td>CMA-ESw</td><td>1.2e4(1e4)</td><td>1.4e4(1e4)</td><td>1.6e4(1e4)</td><td>1.9e4(1e4)</td><td>1.9e4(1e4)</td><td>2.3e4(1e4)</td><td>15/15</td></tr><tr><td>DE-scrip</td><td>8636(2819)</td><td>4.5e4(5327)</td><td>8.9e4(2e5)</td><td>9.2e4(2e5)</td><td>9.6e4(2e5)</td><td>1.0e5(2e5)</td><td>1.1e5(2e5)</td><td>13/15</td><td>DE-scrip</td><td>8.9e4(2e5)</td><td>9.9e4(2e5)</td><td>1.1e5(2e5)</td><td>1.2e5(2e5)</td><td>1.3e5(2e5)</td><td>1.4e5(2e5)</td><td>1.6e5(2e5)</td><td>13/15</td></tr><tr><td>cma-Nol</td><td>859(114)</td><td>1405(122)</td><td>8689(8511)</td><td>9428(8770)</td><td>9428(8770)</td><td>1.1e4(8925)</td><td>1.4e4(9846)</td><td>15/15</td><td>cma-Nol</td><td>1.0e4(1e4)</td><td>1.8e4(1e4)</td><td>2.1e4(3e4)</td><td>2.4e4(3e4)</td><td>2.8e4(3e4)</td><td>3.4e4(3e4)</td><td>4.1e4(3e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>866(140)</td><td>1532(493)</td><td>3957(3556)</td><td>4159(3350)</td><td>4477(3065)</td><td>5727(2310)*</td><td>6979(1526)*<sup>2</sup></td><td>15/15</td><td>cmaHle</td><td>3558(3540)</td><td>7104(7991)</td><td>1.2e4(8631)</td><td>1.7e4(9380)</td><td>2.6e4(9290)</td><td>2.9e4(1e4)</td><td>3.3e4(1e4)</td><td>15/15</td></tr><tr><td>f2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>4.4e4(2e4)</td><td>2.4e5(7e4)</td><td>3.6e5(3e5)</td><td>5.6e5(3e5)</td><td>5.6e5(6e5)</td><td>7.2e5(5e5)</td><td>1.2e6(7e5)</td><td>2/15</td><td>cma-H</td><td>578(136)</td><td>7193(8184)</td><td>1.5e4(2e4)</td><td>2.3e4(2e4)</td><td>2.8e4(2e4)</td><td>3.6e4(2e4)</td><td>4.2e4(3e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>1.1e4(1751)</td><td>6.4e4(2e4)</td><td>2.9e5(2e5)</td><td>4.5e5(3e5)</td><td>4.5e5(5e5)</td><td>4.5e5(4e5)</td><td>4.5e5(5e5)</td><td>8/15</td><td>CMA-ES</td><td>588(208)</td><td>6.2e4(2e4)</td><td>6.8e4(7e4)</td><td>1.2e5(1e5)</td><td>1.2e5(1e5)</td><td>2.5e5(9e4)</td><td>1.9e5(9e4)</td><td>1/15</td></tr><tr><td>CMA-ESw</td><td>3.5e4(2e4)</td><td>3.7e4(2e4)</td><td>4.3e4(3e4)</td><td>5.6e4(6e4)</td><td>5.8e4(6e4)</td><td>6.1e4(7e4)</td><td>6.0e4(6e4)</td><td>15/15</td><td>CMA-ESw</td><td>1643(629)</td><td>3018(658)</td><td>7015(1e4)</td><td>9529(1e4)</td><td>1.2e4(1e4)</td><td>1.9e4(1e4)</td><td>2.5e4(1e4)</td><td>1/15</td></tr><tr><td>DE-scrip</td><td>1.04(1948)</td><td>1.7e4(2747)</td><td>5.3e4(931)</td><td>8.9e4(2e5)</td><td>9.4e4(2e5)</td><td>1.0e5(2e5)</td><td>1.1e5(2e5)</td><td>15/15</td><td>DE-scrip</td><td>4850(1211)</td><td>2.3e4(6259)</td><td>3.1e5(4e5)</td><td>4.0e5(5e5)</td><td>4.2e5(4e5)</td><td>4.8e5(4e5)</td><td>5.7e5(4e5)</td><td>8/15</td></tr><tr><td>cma-Nol</td><td>4.2e4(2e4)</td><td>2.0e5(1e5)</td><td>3.7e5(8e4)</td><td>9.8e5(1e6)</td><td>1.2e6(8e5)</td><td>2.0e6(1e6)</td><td>6.0e6(7e6)</td><td>15/15</td><td>cma-Nol</td><td>652(182)</td><td>5705(8030)</td><td>2.0e4(4e4)</td><td>2.7e4(4e4)</td><td>3.1e4(4e4)</td><td>3.8e4(5e4)</td><td>4.6e4(5e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>1.2e3(940)</td><td>3.6e4(3e4)</td><td>5.1e4(3e4)</td><td>5.4e4(4e4)</td><td>5.5e4(4e4)</td><td>5.6e4(4e4)</td><td>6.0e4(4e4)</td><td>15/15</td><td>cmaHle</td><td>620(134)</td><td>1959(938)</td><td>5783(4326)</td><td>1.3e4(1e4)</td><td>2.2e4(4e4)</td><td>4.9e4(4e4)</td><td>5.6e4(4e4)</td><td>15/15</td></tr><tr><td>f3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1623(444)</td><td>1.2e5(9e4)</td><td>5.7e6(8e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>2.3e5(6e4)</td><td>2.3e5(6e4)</td><td>2.3e5(6e4)</td><td>2.4e5(6e4)</td><td>2.5e5(6e4)</td><td>2.6e5(6e4)</td><td>2.7e5(7e4)</td><td>14/15</td></tr><tr><td>CMA-ES</td><td>4467(4746)</td><td>6.0e5(5e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>2.9e6(3e6)</td><td>2.9e6(4e6)</td><td>2.9e6(4e6)</td><td>2.9e6(3e6)</td><td>2.9e6(2e6)</td><td>2.9e6(3e6)</td><td>2.9e6(4e6)</td><td>2/15</td></tr><tr><td>CMA-ESw</td><td>6105(1148)</td><td>2.1e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>1.7e4(430)</td><td>6.7e5(7e5)</td><td>5.7e6(6e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>3922(1500)</td><td>1.2e5(9e4)</td><td>2.9e6(4e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>1558(431)</td><td>2.5e4(2e4)*</td><td>5.6e6(8e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>ppdata/f4</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>8499(1e4)</td><td>2.0e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>4.0e4(5e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>1.2e4(1e4)</td><td>5.9e6(5e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>1.8e4(1e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>2.5e4(2e4)</td><td>3.8e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>8.7e5(8e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>2.2e4(3099)</td><td>5.2e5(5e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>1.2e4(1e4)</td><td>2.3e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>2921(654)</td><td>1.6e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f5</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1270(184)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>15/15</td><td>cma-H</td><td>9577(1e4)</td><td>1.5e4(3e4)</td><td>1.7e4(3e4)</td><td>2.0e4(3e4)</td><td>2.4e4(3e4)</td><td>2.9e4(3e4)</td><td>3.6e4(4e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>1.8e4(2e4)</td><td>7.5e4(8e4)</td><td>7.5e4(8e4)</td><td>7.6e4(8e4)</td><td>7.9e4(8e4)</td><td>8.8e4(6e4)</td><td>1.0e4(5e4)</td><td>15/15</td><td>CMA-ES</td><td>1.0e5(1e5)</td><td>1.7e5(1e5)</td><td>2.0e5(1e5)</td><td>2.2e5(1e5)</td><td>2.8e5(1e5)</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>887(370)*<sup>2</sup></td><td>1451(780)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>12/15</td><td>CMA-ESw</td><td>3.1e4(3e4)</td><td>1.1e5(2e5)</td><td>1.1e5(2e5)</td><td>1.2e5(2e5)</td><td>1.2e5(2e5)</td><td>1.2e5(2e5)</td><td>1.3e5(2e5)</td><td>13/15</td></tr><tr><td>DE-scrip</td><td>4.2e4(7980)</td><td>1.8e5(2e5)</td><td>2.0e5(2e5)</td><td>2.2e5(2e5)</td><td>2.4e5(2e5)</td><td>2.8e5(2e5)</td><td>3.3e5(2e5)</td><td>15/15</td><td>CMA-ESw</td><td>7.0e4(3e4)</td><td>2.5e5(3e5)</td><td>2.6e5(4e5)</td><td>2.7e5(4e5)</td><td>2.8e5(3e5)</td><td>3.0e5(4e5)</td><td>3.2e5(2e5)</td><td>10/15</td></tr><tr><td>cma-Nol</td><td>1429(230)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>15/15</td><td>cma-Nol</td><td>1.1e4(1e4)</td><td>2.5e4(3e4)</td><td>2.8e4(3e4)</td><td>3.2e4(3e4)</td><td>3.6e4(3e4)</td><td>4.2e4(3e4)</td><td>5.1e4(3e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>1318(226)</td><td>3429(4218)</td><td>3429(4218)</td><td>3429(4218)</td><td>3429(4218)</td><td>3429(4218)</td><td>3429(4218)</td><td>15/15</td><td>cma-Nol</td><td>6473(8852)</td><td>2.5e4(2e4)</td><td>2.9e4(2e4)</td><td>3.1e4(2e4)</td><td>3.4e4(3e4)</td><td>4.8e4(2e4)</td><td>4.4e4(2e4)</td><td>15/15</td></tr><tr><td>f6</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1283(816)</td><td>5.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>1434(678)</td><td>2.7e4(3e4)</td><td>4.7e4(4e4)</td><td>5.2e4(4e4)</td><td>5.6e4(4e4)</td><td>6.5e4(4e4)</td><td>7.5e4(4e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>4.5e4(7e4)</td><td>2.9e6(4e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>1533(335)</td><td>4.0e5(5e5)</td><td>4.1e5(3e5)</td><td>4.3e5(3e5)</td><td>4.5e4(5e5)</td><td>4.9e5(3e5)</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>887(370)*<sup>2</sup></td><td>1451(780)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>12/15</td><td>CMA-ESw</td><td>2825(1204)</td><td>6.3e5(7e5)</td><td>8.2e5(1e6)</td><td>8.3e5(8e6)</td><td>8.3e5(1e6)</td><td>8.4e5(8e6)</td><td>8.4e5(1e6)</td><td>5/15</td></tr><tr><td>DE-scrip</td><td>1.7e4(6864)</td><td>5.9e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>2.5e4(2e4)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>3/15</td></tr><tr><td>cma-Nol</td><td>1316(620)</td><td>4.3e5(8e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>1260(378)</td><td>4.0e4(5e4)</td><td>5.2e4(6e4)</td><td>5.6e4(6e4)</td><td>6.0e4(6e4)</td><td>6.9e4(7e4)</td><td>7.7e4(7e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>4796(7773)</td><td>6.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>1451(568)</td><td>4.2e4(5e4)</td><td>5.2e4(6e4)</td><td>5.6e4(7e4)</td><td>6.0e4(7e4)</td><td>6.8e4(7e4)</td><td>7.7e4(8e4)</td><td>15/15</td></tr><tr><td>f7</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>5.2e5(4e5)</td><td>8.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>6/15</td><td>cma-H</td><td>2.9e6(3e6)</td><td>6.0e6(5e6)</td><td>6.0e6(5e6)</td><td>6.0e6(9e6)</td><td>6.0e6(6e6)</td><td>6.0e6(7e6)</td><td>6.0e6(7e6)</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>4.8e5(4e5)</td><td>1.4e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>4/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>4.0e5(4e5)</td><td>5.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>7/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>4.0e5(5e5)</td><td>4.1e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>9/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f8</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>2340(2820)</td><td>4.0e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>867(340)</td><td>1012(274)</td><td>1.4e5(8e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>5.3e4(6e4)</td><td>1.9e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>979(370)</td><td>2.4e4(2e4)</td><td>4.4e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>6995(4116)</td><td>1.5e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>4501(6348)</td><td>4868(6366)</td><td>1.8e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>4.1e4(4962)</td><td>3.9e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>4935(1332)</td><td>6037(2072)</td><td>1.6e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>3682(8742)</td><td>3.8e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1/15</td><td>cma-Nol</td><td>872(187)</td><td>994(242)</td><td>2.5e5(1e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>1487(282)</td><td>6.0e4(9e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>875(148)</td><td>997(238)</td><td>5.2e4(5e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f9</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1.4e4(9322)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>4.4e4(4e4)</td><td>9.4e4(2e5)</td><td>1.3e5(2e5)</td><td>1.4e5(3e5)</td><td>1.9e5(2e5)</td><td>1.9e5(2e5)</td><td>1.9e5(3e5)</td><td>11/15</td></tr><tr><td>CMA-ES</td><td>6.9e5(8e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>1.5e4(3e5)</td><td>3.4e5(4e5)</td><td>7.4e5(1e5)</td><td>7.4e5(4e5)</td><td>7.5e5(6e5)</td><td>7.5e5(7e5)</td><td>7.5e5(7e5)</td><td>6/15</td></tr><tr><td>CMA-ESw</td><td>4.3e4(4112)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>1.4e5(2e5)</td><td>1.9e5(2e5)</td><td>2.0e5(2e5)</td><td>2.2e5(4e5)</td><td>2.2e5(3e5)</td><td>2.2e5(3e5)</td><td>2.2e5(3e5)</td><td>1/15</td></tr><tr><td>DE-scrip</td><td>8627(9834)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>3.6e4(3882)</td><td>7.9e4(7e5)</td><td>6.3e5(6e5)</td><td>1.1e6(1e5)</td><td>1.1e6(1e5)</td><td>1.2e6(2e5)</td><td>1.2e6(1e5)</td><td>1/15</td></tr><tr><td>cma-Nol</td><td>6811(2e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>1.0e5(2e5)</td><td>1.5e5(3e5)</td><td>2.3e5(3e5)</td><td>2.4e5(3e5)</td><td>2.4e5(3e5)</td><td>2.4e5(3e5)</td><td>2.5e5(2e5)</td><td>11/15</td></tr><tr><td>cmaHle</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>f10</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1.5e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>2.0e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>1.5e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f11</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>68(48)</td><td>5.8e6(7e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>2.8e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>76(96)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>112(102)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>177(226)</td><td>1.3e5(8e4)*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>0/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>77(48)</td><td>1.4e6(1e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>2.8e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>64(52)</td><td>1.1e6(9e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>6.6e5(7e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f12</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1.1e4(1e4)</td><td>1.2e4(1e4)</td><td>1.3e4(1e4)</td><td>1.5e4(1e4)</td><td>1.6e4(1e4)</td><td>1.9e4(1e4)</td><td>2.2e4(1e4)</td><td>15/15</td><td>cma-H</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>7.1e4(7e4)</td><td>7.8e4(7e4)</td><td>8.4e4(6e4)</td><td>9.1e4(6e4)</td><td>9.5e4(8e4)</td><td>1.1e5(6e4)</td><td>1.3e5(6e4)</td><td>15/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>3.7e4(3e4)</td><td>4.0e4(3e4)</td><td>4.3e4(3e4)</td><td>4.5e4(3e4)</td><td>4.6e4(3e4)</td><td>4.9e4(3e4)</td><td>5.2e4(3e4)</td><td>15/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>1.7e5(2e5)</td><td>1.7e5(3e5)</td><td>1.8e5(2e5)</td><td>1.9e5(2e5)</td><td>1.9e5(2e5)</td><td>2.0e5(2e5)</td><td>2.1e5(2e5)</td><td>11/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>5842(9760)</td><td>6491(9902)</td><td>7584(1e4)</td><td>8730(1e4)</td><td>1.0e4(1e4)</td><td>1.3e4(1e4)</td><td>1.6e4(1e4)</td><td>15/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>2875(2073)</td><td>3260(1858)</td><td>4092(1654)</td><td>4841(1462)</td><td>5596(1310)</td><td>7218(1566)</td><td>8826(1664)*<sup>2</sup></td><td>15/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr></tbody></table>
<table><caption>Table 2: Expected runtime (ERT) to reach given targets, measured in number of *f*-evaluations, in dimension 40. For each function, the ERT and, in braces Δ*f*-values as shown in the top row. #succ is the number of trials that reached the last target $f_{opt} + 10^{-8}$. The median number of conducted function evaluations is additionally given in *italics*, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with $p = 0.05$ or $p = 10^{-k}$ when the number $k$ following the star is larger than 1, with Bonferroni correction by the number of functions (24). Best results are printed in bold. Data produced with COCO v2.6.3</caption><thead><tr><th>Δf<sub>opt</sub></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th><th>Δf<sub>opt</sub></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th></tr></thead><tbody><tr><td>cma-H</td><td>818(133)</td><td>2349(137)</td><td>7769(8446)</td><td>8453(8858)</td><td>9602(9016)</td><td>1.2e4(9556)</td><td>1.5e4(144)</td><td>15/15</td><td>cma-H</td><td>7429(8778)</td><td>2.2e4(2e4)</td><td>2.6e4(2e4)</td><td>2.9e4(2e4)</td><td>3.2e4(3e4)</td><td>3.9e4(3e4)</td><td>4.0e4(3e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>1.5e4(2e4)</td><td>4.2e4(5e4)</td><td>4.2e4(6e4)</td><td>4.5e4(5e4)</td><td>5.3e4(5e4)</td><td>7.0e4(5e4)</td><td>8.7e4(5e4)</td><td>15/15</td><td>CMA-ES</td><td>7.1e4(7e4)</td><td>9.3e4(9e4)</td><td>1.1e5(9e4)</td><td>1.2e5(8e4)</td><td>1.4e5(8e4)</td><td>1.9e5(7e4)</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>2239(849)</td><td>3993(2022)</td><td>4522(2228)</td><td>5079(2058)</td><td>5992(1916)</td><td>8350(1843)</td><td>1.1e4(1698)</td><td>15/15</td><td>CMA-ESw</td><td>1.2e4(1e4)</td><td>1.4e4(1e4)</td><td>1.6e4(1e4)</td><td>1.9e4(1e4)</td><td>1.9e4(1e4)</td><td>2.3e4(1e4)</td><td>15/15</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>8636(2819)</td><td>4.5e4(5327)</td><td>8.9e4(2e5)</td><td>9.2e4(2e5)</td><td>9.6e4(2e5)</td><td>1.0e5(2e5)</td><td>1.1e5(2e5)</td><td>13/15</td><td>DE-scrip</td><td>8.9e4(2e5)</td><td>9.9e4(2e5)</td><td>1.1e5(2e5)</td><td>1.2e5(2e5)</td><td>1.3e5(2e5)</td><td>1.4e5(2e5)</td><td>1.6e5(2e5)</td><td>13/15</td></tr><tr><td>cma-Nol</td><td>859(114)</td><td>1405(122)</td><td>8689(8511)</td><td>9428(8770)</td><td>9428(8770)</td><td>1.1e4(8925)</td><td>1.4e4(9846)</td><td>15/15</td><td>cma-Nol</td><td>1.0e4(1e4)</td><td>1.8e4(1e4)</td><td>2.1e4(3e4)</td><td>2.4e4(3e4)</td><td>2.8e4(3e4)</td><td>3.4e4(3e4)</td><td>4.1e4(3e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>866(140)</td><td>1532(493)</td><td>3957(3556)</td><td>4159(3350)</td><td>4477(3065)</td><td>5727(2310)*</td><td>6979(1526)*<sup>2</sup></td><td>15/15</td><td>cmaHle</td><td>3558(3540)</td><td>7104(7991)</td><td>1.2e4(8631)</td><td>1.7e4(9380)</td><td>2.6e4(9290)</td><td>2.9e4(1e4)</td><td>3.3e4(1e4)</td><td>15/15</td></tr><tr><td>f2</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f14</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>4.4e4(2e4)</td><td>2.4e5(7e4)</td><td>3.6e5(3e5)</td><td>5.6e5(3e5)</td><td>5.6e5(6e5)</td><td>7.2e5(5e5)</td><td>1.2e6(7e5)</td><td>2/15</td><td>cma-H</td><td>578(136)</td><td>7193(8184)</td><td>1.5e4(2e4)</td><td>2.3e4(2e4)</td><td>2.8e4(2e4)</td><td>3.6e4(2e4)</td><td>4.2e4(3e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>1.1e4(1751)</td><td>6.4e4(2e4)</td><td>2.9e5(2e5)</td><td>4.5e5(3e5)</td><td>4.5e5(5e5)</td><td>4.5e5(4e5)</td><td>4.5e5(5e5)</td><td>8/15</td><td>CMA-ES</td><td>1.8e4(1e4)</td><td>6.2e4(7e4)</td><td>6.8e4(7e4)</td><td>1.2e5(1e5)</td><td>1.2e5(1e5)</td><td>2.5e5(7e4)</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>3.5e4(2e4)</td><td>3.7e4(2e4)</td><td>4.3e4(3e4)</td><td>5.6e4(6e4)</td><td>5.8e4(6e4)</td><td>6.1e4(7e4)</td><td>6.0e4(6e4)</td><td>15/15</td><td>CMA-ESw</td><td>1643(629)</td><td>3018(658)</td><td>7015(1e4)</td><td>9529(1e4)</td><td>1.2e4(1e4)</td><td>1.9e4(1e4)</td><td>2.5e4(1e4)</td><td>15/15</td></tr><tr><td>DE-scrip</td><td>1.04(1948)</td><td>1.7e4(2747)</td><td>5.3e4(931)</td><td>8.9e4(2e5)</td><td>9.4e4(2e5)</td><td>1.0e5(2e5)</td><td>1.1e5(2e5)</td><td>15/15</td><td>DE-scrip</td><td>4850(1211)</td><td>2.3e4(6259)</td><td>3.1e5(4e5)</td><td>4.0e5(5e5)</td><td>4.2e5(4e5)</td><td>4.8e5(4e5)</td><td>5.7e5(4e5)</td><td>8/15</td></tr><tr><td>cma-Nol</td><td>4.2e4(2e4)</td><td>2.0e5(1e5)</td><td>3.7e5(8e4)</td><td>9.8e5(1e6)</td><td>1.2e6(8e5)</td><td>2.0e6(1e6)</td><td>6.0e6(7e6)</td><td>15/15</td><td>cma-Nol</td><td>652(182)</td><td>5705(8030)</td><td>2.0e4(4e4)</td><td>2.7e4(4e4)</td><td>3.1e4(4e4)</td><td>3.8e4(5e4)</td><td>4.6e4(5e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>1.2e3(3940)</td><td>3.6e4(3e4)</td><td>5.1e4(3e4)</td><td>5.4e4(4e4)</td><td>5.5e4(4e4)</td><td>5.6e4(4e4)</td><td>6.0e4(4e4)</td><td>15/15</td><td>cmaHle</td><td>620(134)</td><td>1959(938)</td><td>5783(4326)</td><td>1.3e4(1e4)</td><td>2.2e4(4e4)</td><td>4.9e4(4e4)</td><td>5.6e4(4e4)</td><td>15/15</td></tr><tr><td>f3</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f15</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1623(444)</td><td>1.2e5(9e4)</td><td>5.7e6(8e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>2.3e5(6e4)</td><td>2.3e5(6e4)</td><td>2.3e5(6e4)</td><td>2.4e5(6e4)</td><td>2.5e5(6e4)</td><td>2.6e5(6e4)</td><td>2.7e5(7e4)</td><td>14/15</td></tr><tr><td>CMA-ES</td><td>4467(4746)</td><td>6.0e5(5e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>2.9e6(3e6)</td><td>2.9e6(4e6)</td><td>2.9e6(4e6)</td><td>2.9e6(3e6)</td><td>2.9e6(2e6)</td><td>2.9e6(3e6)</td><td>2.9e6(4e6)</td><td>2/15</td></tr><tr><td>CMA-ESw</td><td>6105(1148)</td><td>2.1e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>1.7e4(430)</td><td>6.7e5(7e5)</td><td>5.7e6(6e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>3922(1500)</td><td>1.2e5(9e4)</td><td>2.9e6(4e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>1558(431)</td><td>2.5e4(2e4)*</td><td>5.6e6(8e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>ppdata/f4</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f16</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>8499(1e4)</td><td>2.0e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>4.0e4(5e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>1.2e4(1e4)</td><td>5.9e6(5e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>1.8e4(1e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>2.5e4(2e4)</td><td>3.8e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>8.7e5(8e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>2.2e4(3909)</td><td>5.2e5(5e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>1.2e4(1e4)</td><td>2.3e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>2921(654)</td><td>1.6e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f5</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f17</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1270(184)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>3429(2040)</td><td>15/15</td><td>cma-H</td><td>9577(1e4)</td><td>1.5e3(3e4)</td><td>1.7e3(3e4)</td><td>2.0e3(3e4)</td><td>2.4e3(3e4)</td><td>2.9e3(3e4)</td><td>3.6e3(4e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>1.8e4(2e4)</td><td>7.5e4(8e4)</td><td>7.5e4(8e4)</td><td>7.6e4(8e4)</td><td>7.9e4(8e4)</td><td>8.8e4(6e4)</td><td>1.0e4(5e4)</td><td>15/15</td><td>CMA-ES</td><td>1.0e5(1e5)</td><td>1.7e5(1e5)</td><td>2.0e5(1e5)</td><td>2.2e5(1e5)</td><td>2.8e5(1e5)</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>887(370)*<sup>2</sup></td><td>1451(780)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>1458(792)</td><td>12/15</td><td>CMA-ESw</td><td>3.1e4(3e4)</td><td>1.1e5(2e5)</td><td>1.1e5(2e5)</td><td>1.2e5(2e5)</td><td>1.2e5(2e5)</td><td>1.2e5(2e5)</td><td>1.3e5(2e5)</td><td>13/15</td></tr><tr><td>cma-Nol</td><td>4.2e4(7980)</td><td>1.8e5(2e5)</td><td>2.0e5(2e5)</td><td>2.2e5(2e5)</td><td>2.4e5(2e5)</td><td>2.8e5(2e5)</td><td>3.3e5(2e5)</td><td>15/15</td><td>CMA-ESw</td><td>7.0e4(3e4)</td><td>2.5e5(3e5)</td><td>2.6e5(4e5)</td><td>2.7e5(4e5)</td><td>2.8e5(3e5)</td><td>3.0e5(4e5)</td><td>3.2e5(2e5)</td><td>10/15</td></tr><tr><td>cmaHle</td><td>1429(230)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>4171(4692)</td><td>15/15</td><td>cma-Nol</td><td>1.1e4(1e4)</td><td>2.5e4(3e4)</td><td>2.8e4(3e4)</td><td>3.2e4(3e4)</td><td>3.6e4(3e4)</td><td>4.2e4(3e4)</td><td>5.1e4(3e4)</td><td>15/15</td></tr><tr><td>f6</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f18</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1283(816)</td><td>5.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>1434(678)</td><td>2.7e3(3e4)</td><td>4.7e4(4e4)</td><td>5.2e4(4e4)</td><td>5.6e4(1e4)</td><td>6.5e4(1e4)</td><td>7.5e4(4e4)</td><td>15/15</td></tr><tr><td>CMA-ES</td><td>4.5e4(7e4)</td><td>2.9e6(4e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>1533(335)</td><td>4.0e5(5e5)</td><td>4.1e5(3e5)</td><td>4.3e5(3e5)</td><td>4.5e5(3e5)</td><td>4.9e5(3e5)</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>1.7e4(6864)</td><td>5.9e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>2825(1204)</td><td>6.3e5(7e5)</td><td>8.2e5(1e6)</td><td>8.3e5(8e6)</td><td>8.3e5(1e6)</td><td>8.4e5(8e6)</td><td>8.4e5(1e6)</td><td>5/15</td></tr><tr><td>DE-scrip</td><td>1.7e4(6864)</td><td>5.9e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>2.5e4(2e4)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>1.7e6(2e6)</td><td>3/15</td></tr><tr><td>cma-Nol</td><td>1316(620)</td><td>4.3e5(8e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>1260(378)</td><td>4.0e4(5e4)</td><td>5.2e4(6e4)</td><td>5.6e4(6e4)</td><td>6.0e4(6e4)</td><td>6.9e4(7e4)</td><td>7.7e4(7e4)</td><td>15/15</td></tr><tr><td>cmaHle</td><td>4796(7773)</td><td>6.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>1451(568)</td><td>4.2e4(5e4)</td><td>5.2e4(6e4)</td><td>5.6e4(7e4)</td><td>6.0e4(7e4)</td><td>6.8e4(7e4)</td><td>7.7e4(8e4)</td><td>15/15</td></tr><tr><td>f7</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f19</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>5.2e5(4e5)</td><td>8.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>6/15</td><td>cma-H</td><td>2.9e6(3e6)</td><td>6.0e6(5e6)</td><td>6.0e6(5e6)</td><td>6.0e6(9e6)</td><td>6.0e6(6e6)</td><td>6.0e6(7e6)</td><td>6.0e6(7e6)</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>4.8e5(4e5)</td><td>1.4e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>4/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>4/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>4.0e5(4e5)</td><td>5.2e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>7/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>4.0e5(5e5)</td><td>4.1e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>9/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f8</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f20</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>2340(2820)</td><td>4.0e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>6/15</td><td>cma-H</td><td>867(340)</td><td>1012(274)</td><td>1.4e5(8e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>5.3e4(6e4)</td><td>1.9e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>979(370)</td><td>2.4e4(2e4)</td><td>4.4e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>6995(4116)</td><td>1.5e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>4501(6348)</td><td>4868(6366)</td><td>1.8e5(2e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>4.1e4(4962)</td><td>3.9e5(6e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>DE-scrip</td><td>4935(1332)</td><td>6037(2072)</td><td>1.6e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>3682(8742)</td><td>3.8e5(4e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>1/15</td><td>cma-Nol</td><td>872(187)</td><td>994(242)</td><td>2.5e5(1e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>1487(282)</td><td>6.0e4(9e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>875(148)</td><td>997(238)</td><td>5.2e4(3e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f9</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f21</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1.4e5(9322)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>4.4e4(4e4)</td><td>9.4e2(5e5)</td><td>1.3e5(2e5)</td><td>1.4e5(3e5)</td><td>1.9e5(2e5)</td><td>1.9e5(2e5)</td><td>1.9e5(3e5)</td><td>11/15</td></tr><tr><td>CMA-ES</td><td>6.9e5(8e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>1.5e5(3e5)</td><td>3.4e5(4e5)</td><td>7.4e5(1e5)</td><td>7.4e5(6e5)</td><td>7.5e5(6e5)</td><td>7.5e5(7e5)</td><td>7.5e5(7e5)</td><td>6/15</td></tr><tr><td>CMA-ESw</td><td>4.3e4(4112)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>1.4e5(2e5)</td><td>1.9e5(2e5)</td><td>2.0e5(2e5)</td><td>2.2e5(4e5)</td><td>2.2e5(3e5)</td><td>2.2e5(3e5)</td><td>2.2e5(3e5)</td><td>1/15</td></tr><tr><td>DE-scrip</td><td>8627(9834)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>7.9e4(3e4)</td><td>6.3e5(6e5)</td><td>6.3e5(6e5)</td><td>1.1e6(1e5)</td><td>1.1e6(1e5)</td><td>1.2e6(2e5)</td><td>1.2e6(1e5)</td><td>1/15</td></tr><tr><td>cma-Nol</td><td>6811(2e4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>1.0e5(2e5)</td><td>1.5e5(3e5)</td><td>2.3e5(3e5)</td><td>2.4e5(3e5)</td><td>2.4e5(3e5)</td><td>2.4e5(3e5)</td><td>2.5e5(2e5)</td><td>11/15</td></tr><tr><td>cmaHle</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f22</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1.5e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>2.0e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>1.5e6(2e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f11</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f23</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>68(48)</td><td>5.8e6(7e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-H</td><td>2.8e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>76(96)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>112(102)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>177(226)</td><td>1.3e5(8e4)*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>∞*<sup>3</sup></td><td>0/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>77(48)</td><td>1.4e6(1e6)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cma-Nol</td><td>2.8e5(3e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>64(52)</td><td>1.1e6(9e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>cmaHle</td><td>6.6e5(7e5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>f12</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>f24</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>cma-H</td><td>1.1e4(1e4)</td><td>1.2e4(1e4)</td><td>1.3e4(1e4)</td><td>1.5e4(1e4)</td><td>1.6e4(1e4)</td><td>1.9e4(1e4)</td><td>2.2e4(1e4)</td><td>15/15</td><td>cma-H</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ES</td><td>7.1e4(7e4)</td><td>7.8e4(7e4)</td><td>8.4e4(6e4)</td><td>9.1e4(6e4)</td><td>9.5e4(8e4)</td><td>1.1e5(6e4)</td><td>1.3e5(6e4)</td><td>15/15</td><td>CMA-ES</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>CMA-ESw</td><td>3.7e4(3e4)</td><td>4.0e4(3e4)</td><td>4.3e4(3e4)</td><td>4.5e4(3e4)</td><td>4.6e4(3e4)</td><td>4.9e4(3e4)</td><td>5.2e4(3e4)</td><td>15/15</td><td>CMA-ESw</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>DE-scrip</td><td>1.7e5(2e5)</td><td>1.7e5(3e5)</td><td>1.8e5(2e5)</td><td>1.9e5(2e5)</td><td>1.9e5(2e5)</td><td>2.0e5(2e5)</td><td>2.1e5(2e5)</td><td>11/15</td><td>DE-scrip</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cma-Nol</td><td>5842(9760)</td><td>6491(9902)</td><td>7584(1e4)</td><td>8730(1e4)</td><td>1.0e4(1e4)</td><td>1.3e4(1e4)</td><td>1.6e4(1e4)</td><td>15/15</td><td>cma-Nol</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>cmaHle</td><td>2875(2073)</td><td>3260(1858)</td><td>4092(1654)</td><td>4841(1462)</td><td>5596(1310)</td><td>7218(1566)</td><td>8826(1664)*<sup>2</sup></td><td>15/15</td><td>cmaHle</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr></tbody></table>
<figure><img src="image_1.png" /><figcaption>Figure 2: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 40-D.</figcaption></figure>
<figure><img src="image_2.png" /><figcaption>Figure 2: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 40-D.</figcaption></figure>
<figure><img src="image_2.png" /><figcaption>Figure 3: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 80-D.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 3: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 80-D.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>Figure 4: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets $10^{[-8..2]}$ in dimension 80.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 4: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of *f*-evaluations, divided by dimension (FEvals/DIM) for the 51 targets $10^{[-8..2]}$ in dimension 80.</figcaption></figure>