<figure><img src="image_1.png" /></figure>

HAL
open science

# Comparing Boundary Handling Techniques of CMA-ES on the bbob and sbox-cost Test Suites

Dimo Brockhoff

► To cite this version:

Dimo Brockhoff. Comparing Boundary Handling Techniques of CMA-ES on the bbob and sbox-cost Test Suites. GECCO '23 Companion: Companion Conference on Genetic and Evolutionary Computation, Jul 2023, Lisbon, Portugal. pp.2318-2325, 10.1145/3583133.3596413. hal-04403658

HAL Id: hal-04403658

https://inria.hal.science/hal-04403658v1

Submitted on 18 Jan 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Comparing Boundary Handling Techniques of CMA-ES on the
bbob and sbox-cost Test Suites

Dimo Brockhoff

Inria, CMAP, CNRS, Ecole Polytechnique, Institut Polytechnique de Paris
91120 Palaiseau, France

## ABSTRACT

Bound constraints on the variables are the most basic constraints in an optimization problem formulation and, thus, among the most common. It is therefore essential to understand the impact of different boundary handling techniques on algorithm performance. Equally, it is important to understand the practical impact of using bound constraint handling in an algorithm on principally unconstrained problems but where the user has a good indication of the domain of the (sought) optimum. Both questions will be investigated in this paper on the newly introduced box-constrained version sbox-cost of the well-known, unconstrained test suite bbob and for the example of the two boundary handling techniques, implemented in the CMA-ES python module pycma. The numerical experiments performed with the COCO platform show that there is (i) only a minor difference in performance between the two test suites and (ii) a slight performance reduction for the (default) BoundTransform boundary handling compared to the BoundPenalty version of CMA-ES.

## CCS CONCEPTS

*Computing methodologies → Continuous space search;

## KEYWORDS

Benchmarking, black-box optimization, bound constraints

**ACM Reference format:**

Dimo Brockhoff. 2023. Comparing Boundary Handling Techniques of CMA-ES on the bbob and sbox-cost Test Suites. In *Proceedings of Genetic and Evolutionary Computation Conference Companion*, Lisbon, Portugal, July 15–19, 2023 (GECCO ’23 Companion), 8 pages.

DOI: 10.1145/3583133.3596413

## 1 INTRODUCTION

Very often, real-world optimization problems have bound constraints on their variables, i.e., the variables are restricted to stay within (partly open) intervals. For those cases, optimization algorithms have to provide boundary handling options that restrict the sampling of solutions to the feasible domain. The well-known

CMA-ES [7], implemented in the python module pycma, has two of such options: the default BoundTransform and the BoundPenalty option.

In this paper, we investigate the performance of those two boundary handling techniques on the recently introduced sbox-cost test suite—a slightly modified version of the well-known, unconstrained bbob suite that returns infinity as function value outside of the region $[-5, 5]^n$ with $n$ being the search space dimension and thus imitating bound constraints on the variables. The sbox-cost test suite furthermore changes the instances of the original bbob suite by placing the optima (pseudo-)randomly within the entire feasible space, compared to the interval $[-4, 4]^n$ for the bbob suite. The only exception is the linear slope function $f_5$ in which the optimum within $[-5, 5]^n$ stays on one of the corners in both suites.<fnref n="1" /> Given that the optima lie within the search domain $[-5, 5]^n$ (with the exception of the mentioned linear slope function), we do not expect any (large) performance difference between algorithms as long as the search is local and does not sample outside of the feasible domain.

Numerical experiments on the COCO implementation of the test suites [4] support this. However, some unexpected performance differences between the two test suites and between the two boundary handling methods within CMA-ES can be observed on a few test functions as we will see later on—ultimately resulting in the need to further investigate the reason for those differences in future work.

## 2 SBOX-COST VS. BBOB

We already mentioned the main differences between the original bbob test suite and its bound constrained sbox-cost counterpart: unconstrained vs. bound constrained to $[-5, 5]^n$ and the optima being sampled within $[-4, 4]^n$ vs. within $[-5, 5]^n$ (with the exception of the linear slope function where there is no difference within $[-5, 5]^n$). Both suites are implemented within the COCO platform [4] and results can be easily compared with its postprocessing module. When doing so, we use 15 instances and the default 51 target precisions $100, ..., 10^{-8}$ prescribed for both the bbob and the sbox-cost test suite.

## 3 ALGORITHM PRESENTATION

The Covariance Matrix Adaptation Evolution Strategy (CMA-ES, [7]) is a state-of-the-art solver for unconstrained, nonlinear black-box optimization. It samples, in each iteration, $\lambda$ solutions from a multivariate Normal distribution which is adapted throughout

©Dimo Brockhoff 2023. This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in GECCO '23 Companion, http://dx.doi.org/10.1145/3583133.3596413.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

GECCO '23 Companion, Lisbon, Portugal

© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.

979-8-4007-0120-7/23/07...$15.00

DOI: 10.1145/3583133.3596413

<footnote n="1">¹Note that with the unboundedness of the bbob suite, the optimal function value can furthermore be reached in a subspace of $\mathbb{R}^n$ outside the box $[-5, 5]^n$ in bbob (with a flat objective function value) while the single search point with optimal function value lies in a corner of the bounded search space for sbox-cost.</footnote>
the run in terms of its mean, stepsize, and covariance matrix, hence the name. Its python implementation offers two different boundary handling techniques, the default BoundTransform and the BoundPenalty. In the following, we assume that both lower and upper bounds on the variables are given by the (feasibility) interval [lb, ub] for each variable.<fnref n="2" />

**BoundTransform.** The BoundTransform is CMA-ES' default technique to handle bound constraints, mapping an arbitrary variable value $x_i$, periodically back to the feasible interval [lb, ub].

The transformation is thereby the identity in about 90% of the originally feasible interval (i.e., within [lb + al, ub - au] for a choice of al and au that, in practice, depend on the absolute values of lb and ub). The transformation is quadratic in the intervals [lb - 3·al, lb + al] and [ub - au, ub + 3·au] and “is periodically expanded beyond the limits [...] with a period of 2·(ub - lb + al + au)” (see the pycma API documentation here and here). This construction allows for a smooth transformation close to the domain boundary and a robust handling of “arbitrary” bound values.

In the remainder of this paper, we denote the version of CMA-ES, using the BoundTransform boundary handling, as CMA-bt.

**BoundPenalty.** With this boundary handling method, CMA-ES evaluates an infeasible solution $x$ as

$$f(x^{\text{feas}}) + \sum w_i \cdot (x_i - x_i^{\text{feas}})^2 \quad (1)$$

where $x^{\text{feas}}$ is the feasible (in-bounds) solution that is closest to $x$ (in Euclidean space) and the weights $w_i$ for each variable $i$ are updated in each iteration. For more details we refer to section IV.B in [6].

In the following, we denote this version of CMA-ES as CMA-bp.

# 4 EXPERIMENTAL PROCEDURE

We compare both CMA-bt and CMA-bp on the two test suites bbob and sbox-cost as mentioned. The experiments are performed with COCO [4], in its version from the suite-sbox-cost branch of the sbox-cost organization at Github in its version from the 13th of February 2023. The plots are produced with an adapted version of COCO 2.6.3.<fnref n="3" /> To distinguish the experiments on the two test suites from each other, the name of the test suite is appended to the algorithm name of the data sets in the plots. For better readability, the prefix “cma-” is removed in the tables. For the CMA-ES algorithm, we used the python module pycma, version 3.3.0.

Initial values for the CMA-ES variants (besides defaults) are an initial step size of 2, the search space origin as initial mean in the first run and a maximum of 9 restarts. The initial mean for a restart is chosen according to the initial solution proposal functionality of COCO. All output from CMA-ES is suppressed for speed (and lower environmental impact of the experiment) and the bounds are chosen as -5 and +5 respectively for each variable. The length of a single experiment is bounded above by $10^4$ times dimension many function evaluations and stopped as soon as this

amount of function evaluations is reached or if COCO detected a function value within a precision of $10^{-8}$ from the optimal function value and, in addition, any of the many (default) CMA-ES-internal stopping criteria is reached.

The **expected runtime** (ERT), used in the figures and tables below, depends on a given target precision, $I_{\text{target}} = f_{\text{opt}} + \Delta f$, and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach $f_t$, summed over all trials and divided by the number of trials that actually reached $f_t$ [3, 9]. **Statistical significance** is tested with the rank-sum test for a given target $\Delta f_t$ using, for each trial, either the number of needed function evaluations to reach $f_t$ (inverted and multiplied by -1), or, if the target was not reached, the best $\Delta f$-value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration. For a detailed explanation of the COCO plots, please refer to the COCO documentation [4].

**Parameter Tuning.** No parameter tuning has been performed before the experiments.

**Environmental Impact.** Including all preliminary experiments (for smaller budgets but with essentially the same setup), we used less than 64 CPU hours of computing time overall. With a maximal power consumption of 120W for the used processor type, this results in less than 8kWh (plus cooling).

# 5 CPU TIMING

In order to evaluate the CPU timing of the algorithm, we have run the different CMA-ES versions with maximally nine restarts on the entire bbob [5] and sbox-cost test suites for $10^4 n$ function evaluations according to [8]. The Python code was run on a Linux machine with 32 Intel Xeon E5-2683 v4 processors with 2.10GHz and python 3.7.4 but without using the batch parallelization code of COCO. The (average) time per function evaluation for dimensions 2, 3, 5, 10, and 20 on the bbob suite equaled 0.14, 0.15, 0.16, 0.17, and 0.19 milliseconds respectively for the CMA-ES with default BoundTransform and 0.15, 0.16, 0.18, 0.21, and 0.26 milliseconds for the version with BoundPenalty boundary handling. For the sbox-cost experiments, the timings were slightly faster: 0.14, 0.14, 0.15, 0.17, and 0.19 milliseconds for the CMA-ES with BoundTransform and 0.13, 0.13, 0.16, 0.19, and 0.23 milliseconds for the BoundPenalty version.

# 6 RESULTS

Results from experiments according to [8] and [2] on the benchmark functions given in [1] are presented in Figures 1, 2, 3, and 4 and Tables 1 and 2. In addition to the displayed results in this paper, we provide all postprocessed results together with the original data, in particular for different dimensions, at the supplementary material webpage<fnref n="4" />.

With the two algorithm variants, ran on two test suites, also two main investigations can be made: on the impact of the different instance sampling on problem difficulty and on the impact on algorithm performance of the boundary handling itself.

<footnote n="2">²Of course, the code allows to specify different bounds and to also omit one of the two or both bounds independently for each variable.</footnote>

<footnote n="3">³Originally, the two test suites could not be postprocessed together although they are, in principle, compatible with each other.</footnote>

<footnote n="4">⁴see https://brockho.github.io/pycmaonsboxcost/</footnote>
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Expected running time (ERT in number of *f*-evaluations as $\log_{10}$ value), divided by dimension for target function value $10^{-8}$ versus dimension. Slanted grid lines indicate quadratic scaling with the dimension. Different symbols correspond to different algorithms given in the legend of $f_1$ and $f_{24}$. Light symbols give the maximum number of evaluations from the longest trial divided by dimension. Black stars (if present) indicate a statistically better result compared to all other algorithms with $p < 0.01$ and Bonferroni correction number of dimensions (six). The ERT of the (artificial) best algorithm from BBOB 2009 is shown as thick line with diamond markers. Legend: ◦: cma-bp-sboxcost-Brockhoff, ◇: cma-bt-sboxcost-Brockhoff, ⋆: cma-bt-bbob, +: cma-bt-sboxcost.</figcaption></figure>
<figure><img src="image_4.png" /><figcaption>Figure 2: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 5-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>

## 6.1 Impact of bound constraints: bbob vs. sbox-cost

Let us first have a look at the question whether the addition of bound constraints and the changed optima sampling makes the test problems harder to solve. Of course, here, this question can only be answered with respect to the CMA-ES but preliminary experiments showed globally the same observation for a simple random search in $[-5, 5]^n$. Over all functions, we see only slight differences for each CMA-ES version between the two test suites with a difference of maximally 50% for budgets larger than 500n function evaluations in 5-D (Figure 2) and with slightly larger gaps for the larger budgets in 20-D (Figure 3). For the easiest targets and thus the smallest budgets (for example below 100n function evaluations in 20-D) no difference in the ECDF plot is visible.

When looking at the single functions, we observe that (significant) differences appear only on few functions. The largest differences can be observed for CMA-bt on the Rastrigin (in low dimension), Sharp Ridge, non-separable Ellipsoid, Discus, Rosenbrock, Gallagher, and Weierstrass functions, and to a lesser extend on the Step Ellipsoid and for CMA-bp on the non-separable Rastrigin function, Sharp Ridge, Rosenbrock (in the end) and on the multimodal Schaffer and Gallagher functions. Most of the time, the bbob versions are easier to solve but some examples with the opposite observation exists. The most consistent differences over all dimensions are observed on the non-separable Ellipsoid and Discus functions for CMA-bt and on the Sharp Ridge and Rastrigin

functions for CMA-bp. Most of the observed differences are not statistically significant. The only exceptions are two target precisions on the sphere function for CMA-bt (one in dimension 10 and one in dimension 20) and the harder targets for the non-separable Ellipsoid and Discus functions in 20-D for CMA-bp—in all four cases in favor of the bbob function (for the results of the statistical tests, we refer to the supplementary material).

## 6.2 Impact of boundary handling in CMA-ES

A larger effect than the definition of the functions can be observed when the two boundary handling techniques are compared with each other. While aggregated over all functions also no large differences occur on the bbob test suite, the BoundPenalty shows slight advantages over the BoundTransform in particular for larger budgets and in higher dimension. The tendency towards a slightly better performance of BoundPenalty over BoundTransform is the same on the sbox-cost test suite but with less clarity in terms of a dependency on budget and/or dimension.

On the single functions, no visual differences occur on 11 out of the 24 functions (in 10-D and 20-D). On the other functions, it is mostly the BoundPenalty version outperforming the variant with BoundTransform, but exceptions (in some dimensions) exist. Worth to mention are the differences on the linear slope function (statistically significantly better performance for BoundPenalty on both test suites for the harder targets in all dimensions). The differences are more pronounced in smaller dimension with a factor of larger than 5 in dimension 2 (and, in comparison, a factor
<figure><img src="image_5.png" /><figcaption>Figure 3: Bootstrapped empirical cumulative distribution of the number of *f*-evaluations divided by dimension (FEvals/DIM) for 51 targets with target precision in $10^{[-8..2]}$ for all functions and subgroups in 20-D. As reference algorithm, the best algorithm from BBOB 2009 is shown as light thick line with diamond markers.</figcaption></figure>

of around 1.5 in 20-D). Other functions with notable differences are the non-separable Ellipsoid and Discus functions. Here, the differences become more pronounced with larger dimension for the easier targets (the differences are only significant for the harder targets in 20-D for both functions and in 10-D for the Discus function). Other functions with similar behavior (statistically significant only in larger dimension and for the harder targets in favor of CMA-bp) are the Step Ellipsoid, non-separable Rastrigin, and the two Schaffer functions. Statistically significant differences in favor of BoundTransform cannot be observed. For the results of the statistical tests, we refer once again to the supplementary material.

## 7 CONCLUSIONS

We investigated the performance of two variants of CMA-ES handling bound constraints on both the original bbob and the newly introduced box-constrained sbox-cost test suite. Two main observations can be made. First, the performance differences (on both test suites) are larger between the two boundary handling methods compared to the observed differences for the same algorithm, when run on the two test suites. Second, on about half of the 24 test functions, the BoundPenalty boundary handling of CMA-ES performs (slightly) better than the default BoundTransform setting. The observed performance degradation for CMA-bt especially on the Ellipsoid and Discus functions and on some additional functions for the hardest targets suggest to revisit the algorithm in future work.

## ACKNOWLEDGMENTS

This research was conducted with support of the consortium in Applied Mathematics CIROQUO, gathering partners in technological research and academia in the development of advanced methods for Computer Experiments.

## REFERENCES

[1] S. Finck, N. Hansen, R. Ros, and A. Auger. 2009. *Real-Parameter Black-Box Optimization Benchmarking 2009: Presentation of the Noiseless Functions*. Technical Report 2009/20. Research Center PPE. http://coco.lri.fr/downloads/download15.03/bbobdocfunctions.pdf Updated February 2010.

[2] Nikolaus Hansen, Anne Auger, Dimo Brockhoff, and Tea Tušar. 2022. Anytime Performance Assessment in Blackbox Optimization Benchmarking. *IEEE Transactions on Evolutionary Computation* 26, 6 (2022), 1293–1305.

[3] N. Hansen, A. Auger, S. Finck, and R. Ros. 2012. *Real-Parameter Black-Box Optimization Benchmarking 2012: Experimental Setup*. Technical Report. INRIA. http://numbbo.github.io/gforge/bbob2012-downloads

[4] N. Hansen, A. Auger, R. Ros, O. Mersmann, T. Tušar, and D. Brockhoff. 2021. COCO: A Platform for Comparing Continuous Optimizers in a Black-Box Setting. *Optimization Methods and Software* 36 (2021), 114–144. Issue 1. https://doi.org/10.1080/10556788.2020.1808977

[5] N. Hansen, S. Finck, R. Ros, and A. Auger. 2009. *Real-Parameter Black-Box Optimization Benchmarking 2009: Noiseless Functions Definitions*. Technical Report RR-6829. INRIA. http://hal.inria.fr/inria-00362633/en/

[6] N. Hansen, S.P.N. Niederberger, L. Guzzella, and P. Koumoutsakos. 2009. A Method for Handling Uncertainty in Evolutionary Optimization with an Application to Feedback Control of Combustion. *IEEE Transactions on Evolutionary Computation* 13, 1 (2009), 180–197.

[7] N. Hansen and A. Ostermeier. 2001. Completely derandomized self-adaptation in evolution strategies. *Evolutionary Computation* 9, 2 (2001), 159–195.

[8] N. Hansen, T. Tušar, O. Mersmann, A. Auger, and D. Brockhoff. 2016. COCO: The Experimental Procedure. *ArXiv e-prints* arXiv:1603.08776 (2016).
<figure><img src="image_6.png" /><figcaption>Figure 4: Empirical cumulative distribution of simulated (bootstrapped) runtimes, measured in number of f-evaluations, divided by dimension (FEvals/DIM) for the 51 targets 10[−8..2] in dimension 10.</figcaption></figure>

<footnote>[9] Kenneth Price. 1997. Differential evolution vs. the functions of the second ICEO. In *Proceedings of the IEEE International Congress on Evolutionary Computation*. IEEE, Piscataway, NJ, USA, 153–157. https://doi.org/10.1109/ICEC.1997.592287</footnote>
<table><caption>Table 1: Expected runtime (ERT in number of f-evaluations) divided by the respective best ERT measured during BBOB-2009 (when finite) in dimension 5. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of bootstrapped run lengths appear for each algorithm and target, the corresponding reference ERT in the first row. The different target Δf-values are shown in the top row. #succ is the number of trials that reached the (final) target f<sub>opt</sub> + 10<sup>-8</sup>. The median number of conducted function evaluations is additionally given in *italics*, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with *p* = 0.05 or *p* = 10<sup>-k</sup> when the number *k* following the star is larger than 1, with Bonferroni correction by the number of functions (24). A ↑ signifies the number of trials that were worse than the ERT of the best algorithm from BBOB 2009 shown only when less than 10 percent were worse and the ERT was better. Best results are printed in bold. Data</caption><thead><tr><th>Δf<sub>opt</sub></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th><th>Δf<sub>opt</sub></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th></tr></thead><tbody><tr><td></td><td>11</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td><td>15/15</td><td>f13</td><td>132</td><td>195</td><td>250</td><td>319</td><td>1310</td><td>1752</td><td>2255</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>2.0(2)</td><td>8.5(3)</td><td>14(3)</td><td>21(4)</td><td>27(5)</td><td>39(5)</td><td>53(5)</td><td>15/15</td><td>bp-bbob</td><td>2.0(5)</td><td>3.9(1)</td><td>4.1(1)</td><td>3.9(10)</td><td>1.1(0.2)</td><td>1.1(0.2)</td><td>1.1(0.1)</td><td>15/15</td></tr><tr><td>bt-box</td><td>2.1(1)</td><td>9.3(4)</td><td>15(4)</td><td>22(4)</td><td>29(5)</td><td>42(3)</td><td>54(5)</td><td>15/15</td><td>bt-box</td><td>2.9(1)</td><td>3.4(1)</td><td>3.6(1.0)</td><td>4.2(0.6)</td><td>1.2(0.2)</td><td>1.1(0.2)</td><td>1.1(0.1)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>1.9(2)</td><td>10(4)</td><td>16(5)</td><td>24(6)</td><td>31(7)</td><td>42(7)</td><td>55(5)</td><td>15/15</td><td>bt-bbob</td><td>2.8(1)</td><td>3.7(1)</td><td>4.0(1)</td><td>3.8(0.7)</td><td>1.1(0.2)</td><td>1.1(0.2)</td><td>1.0(0.2)</td><td>15/15</td></tr><tr><td>bt-box</td><td>2.3(2)</td><td>9.1(4)</td><td>16(5)</td><td>23(6)</td><td>30(5)</td><td>42(5)</td><td>56(6)</td><td>15/15</td><td>bt-box</td><td>3.2(1)</td><td>4.1(0.8)</td><td>4.7(3)</td><td>5.8(6)</td><td>2.0(2)</td><td>6.1(14)</td><td>6.7(11)</td><td>12/15</td></tr><tr><td>f2</td><td>83</td><td>87</td><td>88</td><td>89</td><td>90</td><td>92</td><td>94</td><td>15/15</td><td>f14</td><td>10</td><td>41</td><td>58</td><td>90</td><td>139</td><td>251</td><td>476</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>7.3(3)</td><td>9.4(3)</td><td>11(2)</td><td>12(1)</td><td>13(2)</td><td>14(2)</td><td>15(2)</td><td>15/15</td><td>bp-bbob</td><td>7.3(3)</td><td>8.4(0.8)</td><td>2.2(0.8)</td><td>3.4(1)</td><td>3.6(0.8)</td><td>3.3(0.6)</td><td>3.2(0.5)</td><td>2.5(0.4)</td><td>15/15</td></tr><tr><td>bt-box</td><td>9.1(5)</td><td>11(5)</td><td>13(5)</td><td>13(5)</td><td>14(5)</td><td>16(5)</td><td>17(6)</td><td>15/15</td><td>bt-box</td><td>10.1(1)</td><td>2.8(2)</td><td>3.7(1)</td><td>3.7(0.7)</td><td>4.0(1)</td><td>3.6(0.8)</td><td>2.6(0.4)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>7.3(2)</td><td>9.0(1)</td><td>10(2)</td><td>11(2)</td><td>12(2)</td><td>13(2)</td><td>14(2)</td><td>15/15</td><td>bt-bbob</td><td>0.87(1)</td><td>2.4(1)</td><td>3.7(1)</td><td>3.5(1)</td><td>3.4(0.8)</td><td>3.4(0.5)</td><td>2.5(0.2)</td><td>15/15</td></tr><tr><td>bt-box</td><td>7.8(2)</td><td>9.0(2)</td><td>10(1)</td><td>11(1)</td><td>12(1)</td><td>13(1)</td><td>15(1)</td><td>15/15</td><td>bt-box</td><td>1.7(2)</td><td>2.5(1)</td><td>3.3(1)</td><td>3.8(0.6)</td><td>3.7(0.8)</td><td>3.5(0.7)</td><td>3.1(1)</td><td>15/15</td></tr><tr><td>f3</td><td>716</td><td>1622</td><td>1637</td><td>1642</td><td>1646</td><td>1650</td><td>1654</td><td>15/15</td><td>f15</td><td>511</td><td>9310</td><td>19369</td><td>19743</td><td>20073</td><td>20769</td><td>21359</td><td>14/15</td></tr><tr><td>bp-bbob</td><td>0.99(1)</td><td>14(20)</td><td>96(143)</td><td>96(99)</td><td>96(87)</td><td>96(121)</td><td>129(166)</td><td>3/15</td><td>0.99(1)</td><td>1.0(0.5)</td><td>1.4(2)</td><td>0.92(0.8)</td><td>0.92(0.8)</td><td>0.93(0.8)</td><td>0.93(0.8)</td><td>0.94(0.8)</td><td>15/15</td></tr><tr><td>bt-box</td><td>1.3(2)</td><td>21(22)</td><td>96(82)</td><td>96(137)</td><td>96(89)</td><td>96(131)</td><td>96(108)</td><td>4/15</td><td>bt-box</td><td>1.6(2)</td><td>0.92(1)</td><td>1.1(0.8)</td><td>1.1(0.8)</td><td>1.2(0.8)</td><td>1.1(0.8)</td><td>1.2(0.8)</td><td>14/15</td></tr><tr><td>bt-bbob</td><td>1.1(1)</td><td>15(16)</td><td>71(72)</td><td>71(79)</td><td>71(63)</td><td>72(65)</td><td>72(109)</td><td>5/15</td><td>bt-box</td><td>1.7(2)</td><td>1.8(2)</td><td>2.0(2)</td><td>2.0(2)</td><td>2.0(3)</td><td>2.0(2)</td><td>2.0(2)</td><td>11/15</td></tr><tr><td>bt-box</td><td>1.2(1)</td><td>29(41)</td><td>213(295)</td><td>212(222)</td><td>212(177)</td><td>212(231)</td><td>212(296)</td><td>2/15</td><td>bt-box</td><td>1.8(2)</td><td>3.4(4)</td><td>3.2(3)</td><td>3.2(3)</td><td>3.1(3)</td><td>3.1(4)</td><td>3.0(5)</td><td>8/15</td></tr><tr><td>f4</td><td>809</td><td>1633</td><td>1688</td><td>1758</td><td>1817</td><td>1886</td><td>1903</td><td>15/15</td><td>f16</td><td>120</td><td>612</td><td>2662</td><td>10163</td><td>10449</td><td>11644</td><td>12095</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>1.9(1)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>bp-bbob</td><td>1.5(1)</td><td>2.1(3)</td><td>1.7(3)</td><td>0.50(0.7)</td><td>0.51(0.7)</td><td>0.49(0.7)</td><td>0.61(0.8)</td><td>15/15</td></tr><tr><td>bt-box</td><td>1.5(1)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>bt-box</td><td>1.6(1)</td><td>2.7(2)</td><td>2.2(3)</td><td>0.77(0.8)</td><td>0.77(0.7)</td><td>0.74(0.7)</td><td>0.75(0.7)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>1.8(3)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>bt-box</td><td>1.7(2)</td><td>3.2(6)</td><td>2.0(2)</td><td>0.62(0.6)</td><td>0.75(0.8)</td><td>0.71(0.7)</td><td>0.72(0.7)</td><td>15/15</td></tr><tr><td>bt-box</td><td>1.7(2)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td><td>bt-box</td><td>2.0(4)</td><td>1.9(3)</td><td>1.6(3)</td><td>0.88(0.9)</td><td>0.96(0.9)</td><td>0.96(0.8)</td><td>0.95(0.8)</td><td>14/15</td></tr><tr><td>f5</td><td>10</td><td>10</td><td>10</td><td>10</td><td>10</td><td>10</td><td>10</td><td>15/15</td><td>f17</td><td>5.0</td><td>215</td><td>899</td><td>2861</td><td>3669</td><td>6351</td><td>7934</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>7.4(4)</td><td>24(19)</td><td>29(21)</td><td>30(19)</td><td>30(19)</td><td>30(19)</td><td>30(19)</td><td>15/15</td><td>bp-bbob</td><td>1.3(1)</td><td>0.90(0.3)</td><td>0.70(0.3)↑1</td><td>0.53(0.5)</td><td>0.97(0.5)</td><td>0.96(0.6)</td><td>1.1(0.1)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>5.2(3)</td><td>21(13)</td><td>26(16)</td><td>26(16)</td><td>26(16)</td><td>26(16)</td><td>26(16)</td><td>15/15</td><td>bt-bbob</td><td>2.3(2)</td><td>0.97(0.7)</td><td>1.1(2)</td><td>0.61(0.6)</td><td>0.94(0.5)</td><td>0.93(0.5)</td><td>1.4(0.7)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>12(7)</td><td>23(9)</td><td>32(10)</td><td>40(10)</td><td>47(8)</td><td>63(10)</td><td>78(10)</td><td>15/15</td><td>bt-bbob</td><td>0.92(1)</td><td>1.7(0.4)</td><td>0.85(1)</td><td>0.55(0.5)</td><td>0.72(0.5)</td><td>0.92(0.5)</td><td>1.0(0.5)</td><td>15/15</td></tr><tr><td>bt-box</td><td>11(4)</td><td>23(7)</td><td>31(8)</td><td>37(8)</td><td>45(7)</td><td>60(9)</td><td>75(10)</td><td>15/15</td><td>bt-box</td><td>3.3(4)</td><td>2.4(0.7)</td><td>1.2(2)</td><td>0.91(0.7)</td><td>0.89(0.6)</td><td>0.81(0.1)↑1</td><td>0.80(0.4)</td><td>15/15</td></tr><tr><td>f6</td><td>114</td><td>214</td><td>281</td><td>404</td><td>580</td><td>1038</td><td>1332</td><td>15/15</td><td>f18</td><td>103</td><td>378</td><td>3968</td><td>8451</td><td>9280</td><td>10905</td><td>12469</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>1.5(0.7)</td><td>1.7(0.5)</td><td>1.9(0.3)</td><td>1.7(0.2)</td><td>1.5(0.2)</td><td>1.2(0.1)</td><td>1.2(0.1)</td><td>15/15</td><td>bp-bbob</td><td>0.88(0.7)</td><td>1.0(0.7)</td><td>0.59(0.9)</td><td>0.50(0.4)↑1</td><td>0.51(0.4)↑1</td><td>0.84(0.6)</td><td>0.93(0.5)</td><td>15/15</td></tr><tr><td>bt-box</td><td>2.1(1)</td><td>2.0(1)</td><td>2.2(0.9)</td><td>2.0(0.8)</td><td>1.8(0.7)</td><td>1.3(0.4)</td><td>1.3(0.4)</td><td>15/15</td><td>bt-box</td><td>0.77(0.5)</td><td>3.0(6)</td><td>1.0(1)</td><td>0.67(0.7)</td><td>0.89(1)</td><td>1.1(0.8)</td><td>1.0(0.7)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>1.8(0.7)</td><td>1.8(0.5)</td><td>2.0(0.4)</td><td>1.8(0.3)</td><td>1.6(0.3)</td><td>1.2(0.2)</td><td>1.2(0.1)</td><td>15/15</td><td>bt-box</td><td>0.78(0.6)</td><td>3.0(6)</td><td>1.0(1)</td><td>0.93(0.7)</td><td>0.93(0.6)</td><td>0.96(0.5)</td><td>1.1(0.8)</td><td>15/15</td></tr><tr><td>bt-box</td><td>1.5(0.8)</td><td>1.7(0.5)</td><td>1.9(0.4)</td><td>1.7(0.2)</td><td>1.5(0.2)</td><td>1.2(0.2)</td><td>1.2(0.1)</td><td>15/15</td><td>bt-box</td><td>1.1(0.7)</td><td>4.3(5)</td><td>1.4(2)</td><td>1.1(0.9)</td><td>1.0(0.8)</td><td>1.0(0.5)</td><td>1.4(0.6)</td><td>14/15</td></tr><tr><td>f7</td><td>24</td><td>324</td><td>1171</td><td>1451</td><td>1572</td><td>1572</td><td>1597</td><td>15/15</td><td>f19</td><td>1</td><td>1</td><td>242</td><td>1.05</td><td>1.26</td><td>1.26</td><td>1.26</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>3.3(2)</td><td>1.2(1)</td><td>0.70(0.4)</td><td>0.74(0.3)</td><td>0.76(0.5)</td><td>0.76(0.5)</td><td>0.85(0.6)</td><td>15/15</td><td>bp-bbob</td><td>10(1)</td><td>1(0)</td><td>145(226)</td><td>6.7(9)</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-box</td><td>4.3(2)</td><td>1.3(0.8)</td><td>0.65(0.4)</td><td>0.65(0.4)↑1</td><td>0.63(0.3)↑1</td><td>0.63(0.3)↑1</td><td>0.64(0.3)↑1</td><td>15/15</td><td>bt-box</td><td>10(1)</td><td>1(0)</td><td>144(170)</td><td>6.8(7)</td><td>5.9(6)</td><td>5.9(7)</td><td>5.8(5)</td><td>1/15</td></tr><tr><td>bt-bbob</td><td>3.9(2)</td><td>1.0(1)</td><td>0.91(0.8)</td><td>0.83(0.7)</td><td>0.90(0.7)</td><td>0.90(0.7)</td><td>0.97(0.7)</td><td>15/15</td><td>bt-box</td><td>10(1)</td><td>1(0)</td><td>142(158)</td><td>1.3(1)</td><td>2.0(1)</td><td>3.0(4)</td><td>3.0(3)</td><td>2/15</td></tr><tr><td>bt-box</td><td>4.2(2)</td><td>0.94(0.5)</td><td>0.82(0.5)</td><td>0.78(0.5)</td><td>0.74(0.4)</td><td>0.74(0.4)</td><td>0.77(0.4)</td><td>15/15</td><td>bt-box</td><td>10(1)</td><td>1(0)</td><td>154(263)</td><td>2.1(3)</td><td>6.1(5)</td><td>6.1(7)</td><td>6.0(5)</td><td>1/15</td></tr><tr><td>f8</td><td>73</td><td>273</td><td>336</td><td>372</td><td>391</td><td>410</td><td>422</td><td>15/15</td><td>f20</td><td>16</td><td>851</td><td>38111</td><td>51362</td><td>54470</td><td>54861</td><td>55313</td><td>14/15</td></tr><tr><td>bp-bbob</td><td>2.4(0.8)</td><td>3.5(1)</td><td>3.9(1)</td><td>4.1(1)</td><td>4.2(1)</td><td>4.5(1)</td><td>4.7(1.0)</td><td>15/15</td><td>bp-bbob</td><td>1.9(1)</td><td>3.7(3)</td><td>1.6(2)</td><td>1.3(0.9)</td><td>1.3(0.9)</td><td>1.4(1)</td><td>1.4(2)</td><td>8/15</td></tr><tr><td>bt-box</td><td>3.0(2)</td><td>3.0(2)</td><td>3.7(1)</td><td>3.8(1)</td><td>3.9(1)</td><td>4.1(1)</td><td>4.4(1)</td><td>15/15</td><td>bt-box</td><td>2.0(2)</td><td>8.1(6)</td><td>1.6(1)</td><td>1.2(1)</td><td>1.2(0.8)</td><td>1.2(1)</td><td>1.3(1)</td><td>8/15</td></tr><tr><td>bt-bbob</td><td>3.0(1)</td><td>4.2(4)</td><td>4.5(3)</td><td>4.6(3)</td><td>4.8(3)</td><td>5.0(3)</td><td>5.2(3)</td><td>15/15</td><td>bt-box</td><td>1.9(2)</td><td>5.5(4)</td><td>1.2(0.9)</td><td>1.0(0.8)</td><td>0.98(1)</td><td>0.99(0.7)</td><td>1.00(0.7)</td><td>10/15</td></tr><tr><td>bt-box</td><td>3.4(2)</td><td>3.3(1)</td><td>3.9(0.7)</td><td>4.1(0.9)</td><td>4.2(0.9)</td><td>4.4(0.9)</td><td>4.6(0.9)</td><td>15/15</td><td>bt-box</td><td>1.7(2)</td><td>6.3(6)</td><td>1.3(1)</td><td>1.0(0.8)</td><td>0.98(0.8)</td><td>1.00(0.8)</td><td>1.1(1)</td><td>10/15</td></tr><tr><td>f9</td><td>35</td><td>127</td><td>214</td><td>263</td><td>300</td><td>335</td><td>369</td><td>15/15</td><td>f21</td><td>41</td><td>1157</td><td>1674</td><td>1692</td><td>1705</td><td>1729</td><td>1757</td><td>14/15</td></tr><tr><td>bp-bbob</td><td>3.9(2)</td><td>6.4(2)</td><td>5.7(1)</td><td>5.2(1)</td><td>4.9(0.9)</td><td>5.0(0.7)</td><td>4.9(0.6)</td><td>15/15</td><td>bp-bbob</td><td>1.5(1)</td><td>4.6(3)</td><td>4.6(6)</td><td>4.7(6)</td><td>4.7(6)</td><td>4.9(7)</td><td>5.0(7)</td><td>15/15</td></tr><tr><td>bt-box</td><td>3.6(1)</td><td>6.5(2)</td><td>5.7(1)</td><td>5.3(1.0)</td><td>5.0(0.9)</td><td>4.9(0.8)</td><td>4.9(0.8)</td><td>15/15</td><td>bt-box</td><td>1.4(0.9)</td><td>6.1(6)</td><td>8.1(12)</td><td>8.1(12)</td><td>8.2(12)</td><td>8.3(12)</td><td>8.4(13)</td><td>14/15</td></tr><tr><td>bt-bbob</td><td>4.6(2)</td><td>5.0(1)</td><td>4.7(0.7)</td><td>4.6(0.6)</td><td>4.4(0.6)</td><td>4.5(0.6)</td><td>4.5(0.5)</td><td>15/15</td><td>bt-bbob</td><td>1.7(1)</td><td>5.2(5)</td><td>16(26)</td><td>16(27)</td><td>15(21)</td><td>15(20)</td><td>15(23)</td><td>11/15</td></tr><tr><td>bt-box</td><td>4.4(1)</td><td>6.7(4)</td><td>5.8(2)</td><td>5.3(2)</td><td>5.1(1)</td><td>5.1(1)</td><td>5.1(1.0)</td><td>15/15</td><td>bt-box</td><td>3.8(17)</td><td>11(9)</td><td>15(18)</td><td>15(18)</td><td>15(22)</td><td>15(17)</td><td>16(18)</td><td>12/15</td></tr><tr><td>f10</td><td>349</td><td>500</td><td>574</td><td>607</td><td>626</td><td>829</td><td>880</td><td>15/15</td><td>f22</td><td>71</td><td>386</td><td>938</td><td>980</td><td>1008</td><td>1040</td><td>1068</td><td>14/15</td></tr><tr><td>bp-bbob</td><td>1.7(0.5)</td><td>1.7(0.4)</td><td>1.7(0.2)</td><td>1.7(0.2)</td><td>1.8(0.2)</td><td>1.5(0.2)</td><td>1.6(0.2)</td><td>15/15</td><td>bp-bbob</td><td>9.3(22)</td><td>21(25)</td><td>120(164)</td><td>115(95)</td><td>112(155)</td><td>109(158)</td><td>107(118)</td><td>5/15</td></tr><tr><td>bt-box</td><td>2.0(0.5)</td><td>1.8(0.4)</td><td>1.8(0.4)</td><td>1.8(0.2)</td><td>1.9(0.2)</td><td>1.6(0.1)</td><td>1.7(0.2)</td><td>15/15</td><td>bt-box</td><td>3.6(9)</td><td>19(16)</td><td>67(113)</td><td>64(75)</td><td>63(68)</td><td>61(65)</td><td>59(76)</td><td>7/15</td></tr><tr><td>bt-bbob</td><td>3.1(2)</td><td>3.1(3)</td><td>2.8(2)</td><td>2.8(2)</td><td>2.9(2)</td><td>2.3(2)</td><td>2.4(1)</td><td>15/15</td><td>bt-box</td><td>1.2(0.7)</td><td>23(45)</td><td>86(150)</td><td>82(103)</td><td>80(90)</td><td>78(123)</td><td>76(99)</td><td>6/15</td></tr><tr><td>bt-box</td><td>2.1(0.6)</td><td>2.4(0.9)</td><td>2.9(3)</td><td>3.1(3)</td><td>3.3(3)</td><td>2.8(2)</td><td>2.8(2)</td><td>15/15</td><td>bt-box</td><td>2.1(0.7)</td><td>19(17)</td><td>58(58)</td><td>56(106)</td><td>67(74)</td><td>76(88)</td><td>74(98)</td><td>6/15</td></tr><tr><td>f11</td><td>143</td><td>202</td><td>763</td><td>977</td><td>1177</td><td>1467</td><td>1673</td><td>15/15</td><td>f23</td><td>3.0</td><td>518</td><td>14249</td><td>27890</td><td>31654</td><td>33030</td><td>34256</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>3.5(1)</td><td>3.3(1)</td><td>1.0(0.2)</td><td>0.89(0.2)</td><td>0.80(0.1)↑0</td><td>0.75(0.1)↑0</td><td>0.75(0.1)↑0</td><td>15/15</td><td>bp-bbob</td><td>3.7(5)</td><td>6.9(5)</td><td>1.8(2)</td><td>1.3(2)</td><td>1.2(1)</td><td>1.1(1)</td><td>1.1(1)</td><td>12/15</td></tr><tr><td>bt-box</td><td>3.9(1)</td><td>3.5(0.7)</td><td>1.1(0.2)</td><td>0.92(0.2)</td><td>0.84(0.1)↑1</td><td>0.77(0.1)↑1</td><td>0.77(0.1)↑1</td><td>15/15</td><td>bt-box</td><td>3.7(4)</td><td>7.4(7)</td><td>5.2(7)</td><td>2.7(3)</td><td>2.4(3)</td><td>2.3(4)</td><td>2.3(2)</td><td>7/15</td></tr><tr><td>bt-bbob</td><td>5.4(4)</td><td>4.5(3)</td><td>1.3(0.8)</td><td>1.1(0.6)</td><td>0.99(0.5)</td><td>0.90(0.4)</td><td>0.89(0.3)</td><td>15/15</td><td>bt-box</td><td>2.5(3)</td><td>7.0(7)</td><td>4.0(5)</td><td>2.3(2)</td><td>2.1(2)</td><td>2.0(1)</td><td>2.0(2)</td><td>8/15</td></tr><tr><td>bt-box</td><td>4.4(2)</td><td>4.0(1)</td><td>1.4(0.3)</td><td>1.2(0.5)</td><td>1.6(1)</td><td>1.7(2)</td><td>1.7(2)</td><td>15/15</td><td>bt-box</td><td>2.9(2)</td><td>9.3(7)</td><td>2.8(4)</td><td>1.5(1)</td><td>1.3(1.0)</td><td>1.3(1)</td><td>1.2(1.0)</td><td>11/15</td></tr><tr><td>f12</td><td>108</td><td>268</td><td>371</td><td>413</td><td>461</td><td>1303</td><td>1494</td><td>15/15</td><td>f24</td><td>1622</td><td>2.2(65)</td><td>6.46e6</td><td>9.66e6</td><td>9.66e6</td><td>1.3e7</td><td>1.3e7</td><td>3/15</td></tr><tr><td>bp-bbob</td><td>4.6(0.7)</td><td>3.9(4)</td><td>4.3(4)</td><td>4.9(4)</td><td>5.5(4)</td><td>2.7(2)</td><td>2.9(2)</td><td>15/15</td><td>bp-bbob</td><td>1.4(2)</td><td>0.56(0.5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-box</td><td>5.2(3)</td><td>3.6(3)</td><td>4.2(4)</td><td>4.8(3)</td><td>5.2(3)</td><td>2.5(1)</td><td>2.5(1)</td><td>15/15</td><td>bt-box</td><td>1.6(1)</td><td>3.4(3)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-bbob</td><td>7.6(5)</td><td>5.4(7)</td><td>5.6(5)</td><td>6.0(4)</td><td>6.2(4)</td><td>2.8(2)</td><td>2.8(2)</td><td>15/15</td><td>bt-box</td><td>2.2(2)</td><td>3.4(3)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-box</td><td>4.7(0.6)</td><td>4.7(5)</td><td>6.8(5)</td><td>7.9(5)</td><td>8.0(6)</td><td>3.5(2)</td><td>3.5(2)</td><td>15/15</td><td>bt-box</td><td>1.4(1)</td><td>3.4(3)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr></tbody></table>
<table><caption>Table 2: Expected runtime (ERT in number of *f*-evaluations) divided by the respective best ERT measured during BBOB-2009 (when finite) in dimension 20. This ERT ratio and, in braces as dispersion measure, the half difference between 10 and 90%-tile of bootstrapped run lengths appear for each algorithm and target, the corresponding reference ERT in the first row. The different target $\Delta f$-values are shown in the top row. #succ is the number of trials that reached the (final) target $f_{opt} + 10^{-8}$. The median number of conducted function evaluations is additionally given in *italics*, if the target in the last column was never reached. Entries, succeeded by a star, are statistically significantly better (according to the rank-sum test) when compared to all other algorithms of the table, with $p = 0.05$ or $p = 10^{-k}$ when the number $k$ following the star is larger than 1, with Bonferroni correction by the number of functions (24). A ↑ signifies the number of trials that were worse than the ERT of the best algorithm from BBOB 2009 shown only when less than 10 percent were worse and the ERT was better. Best results are printed in bold. Data</caption><thead><tr><th>Δf<sub>opt</sub></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th><th>Δf<sub>opt</sub></th><th>1e1</th><th>1e0</th><th>1e-1</th><th>1e-2</th><th>1e-3</th><th>1e-5</th><th>1e-7</th><th>#succ</th></tr><tr><th></th><th>f1</th><th>43</th><th>43</th><th>43</th><th>43</th><th>43</th><th>43</th><th></th><th>f1</th><th>f14</th><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td>bp-bbob</td><td>6.5(1)</td><td>13(1)</td><td>19(1)</td><td>25(1)</td><td>32(1)<sup>*</sup></td><td>45(3)</td><td>57(4)</td><td></td><td>bp-bbob</td><td>3.2(3)</td><td>3.3(2)</td><td>3.6(2)</td><td>3.5(2)</td><td>0.72(0.4)</td><td>2.4(3.9)</td><td>1.4(0.5)</td><td>15/15</td></tr><tr><td>bp-sbx</td><td>7.1(1)</td><td>14(2)</td><td>20(2)</td><td>26(1)</td><td>33(1)</td><td>46(2)</td><td>59(4)</td><td></td><td>bp-sbx</td><td>3.5(2)</td><td>12(8)</td><td>10(6)</td><td>14(29)</td><td>3.8(6)</td><td>4.2(5)</td><td>3.6(4)</td><td>15/15</td></tr><tr><td>bt-bbob</td><td>8.0(2)</td><td>15(3)</td><td>21(3)</td><td>28(3)</td><td>34(3)</td><td>47(4)</td><td>60(4)</td><td></td><td>bt-bbob</td><td>3.8(3)</td><td>2.6(2)</td><td>4.5(3)</td><td>4.4(2)</td><td>0.93(0.4)</td><td>1.1(0.7)</td><td>1.2(0.6)</td><td>15/15</td></tr><tr><td>bt-sbx</td><td>8.4(2)</td><td>16(2)</td><td>23(4)</td><td>29(3)</td><td>37(5)</td><td>51(6)</td><td>64(8)</td><td></td><td>bt-sbx</td><td>3.1(1)</td><td>3.0(2)</td><td>5.3(4)</td><td>5.1(4)</td><td>1.1(0.8)</td><td>1.7(1)</td><td>1.7(1)</td><td>14/15</td></tr><tr><td></td><td>f2</td><td>385</td><td>386</td><td>387</td><td>388</td><td>390</td><td>391</td><td>393</td><td></td><td>f14</td><td>75</td><td>239</td><td>304</td><td>451</td><td>932</td><td>1648</td><td>15661</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>20(4)</td><td>25(3)</td><td>28(3)</td><td>30(2)</td><td>31(2)</td><td>33(1)</td><td>34(2)</td><td></td><td>bp-bbob</td><td>2.4(1)</td><td>2.3(0.4)</td><td>3.3(0.4)</td><td>3.7(0.4)</td><td>3.0(0.2)</td><td>3.8(0.2)</td><td>0.66(0.0)<sup>†</sup></td><td>15/15</td></tr><tr><td>bp-sbx</td><td>21(4)</td><td>25(4)</td><td>27(3)</td><td>29(3)</td><td>30(2)</td><td>30(2)</td><td>32(2)</td><td></td><td>bp-sbx</td><td>3.0(1)</td><td>2.5(0.6)</td><td>3.3(0.3)</td><td>3.7(0.4)</td><td>3.1(0.2)</td><td>3.8(0.3)</td><td>0.65(0.1)<sup>†</sup></td><td>15/15</td></tr><tr><td>bt-bbob</td><td>22(1)</td><td>26(3)</td><td>29(3)</td><td>30(2)</td><td>31(2)</td><td>32(2)</td><td>34(2)</td><td></td><td>bt-bbob</td><td>3.0(1)</td><td>2.9(0.7)</td><td>3.7(0.4)</td><td>4.0(0.6)</td><td>3.3(0.2)</td><td>3.9(0.2)</td><td>0.69(0.0)<sup>†</sup></td><td>15/15</td></tr><tr><td>bt-sbx</td><td>22(2)</td><td>25(2)</td><td>29(2)</td><td>30(2)</td><td>31(2)</td><td>33(2)</td><td>34(2)</td><td></td><td>bt-sbx</td><td>3.8(1)</td><td>2.8(0.6)</td><td>3.6(0.3)</td><td>3.9(0.4)</td><td>3.2(0.3)</td><td>3.9(0.7)</td><td>0.73(0.1)<sup>†</sup></td><td>15/15</td></tr><tr><td></td><td>f3</td><td>5066</td><td>7626</td><td>7635</td><td>7637</td><td>7643</td><td>7646</td><td>7651</td><td></td><td>f15</td><td>30378</td><td>1.5e5</td><td>3.1e5</td><td>3.2e5</td><td>3.2e5</td><td>4.5e5</td><td>4.6e5</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>21(22)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bp-bbob</td><td>0.98(0.5)</td><td>1.3(0.9)</td><td>1.4(1)</td><td>1.4(2)</td><td>1.7(2)</td><td>2.0(3)</td><td>2.0(2)</td><td>3/15</td></tr><tr><td>bp-sbx</td><td>26(30)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bp-sbx</td><td>1.3(0.5)</td><td>6.2(8)</td><td>4.5(4)</td><td>4.5(5)</td><td>4.4(5)</td><td>3.2(3)</td><td>3.1(3)</td><td>2/15</td></tr><tr><td>bt-bbob</td><td>25(29)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bt-bbob</td><td>1.2(0.8)</td><td>20(22)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-sbx</td><td>41(48)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bt-sbx</td><td>0.97(0.8)</td><td>20(16)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td></td><td>f4</td><td>4722</td><td>7628</td><td>7666</td><td>7686</td><td>7700</td><td>7758</td><td>1.4e5</td><td>0/15</td><td>f4</td><td>1384</td><td>2.27265</td><td>77015</td><td>1.4e5</td><td>1.9e5</td><td>2.0e5</td><td>2.2e5</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bp-bbob</td><td>1.2(0.3)</td><td>0.30(0.2)<sup>†</sup></td><td>0.66(0.5)</td><td>0.74(0.6)</td><td>1.0(2)</td><td>2.0(2)</td><td>1.8(2)</td><td>6/15</td></tr><tr><td>bp-sbx</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bp-sbx</td><td>1.3(0.3)</td><td>0.42(0.2)<sup>†</sup></td><td>0.60(0.6)</td><td>0.76(0.5)</td><td>1.2(1)</td><td>1.3(1)</td><td>1.2(0.7)</td><td>9/15</td></tr><tr><td>bt-bbob</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bt-bbob</td><td>4.1(3)</td><td>0.58(0.4)<sup>†</sup></td><td>1.2(1)</td><td>2.0(2)</td><td>2.1(2)</td><td>2.6(4)</td><td>2.4(2)</td><td>5/15</td></tr><tr><td>bt-sbx</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>2e5</td><td>0/15</td><td>bt-sbx</td><td>3.6(3)</td><td>0.58(0.5)</td><td>0.96(1)</td><td>1.1(1)</td><td>1.4(1)</td><td>3.4(4)</td><td>3.1(3)</td><td>4/15</td></tr><tr><td></td><td>f5</td><td>41</td><td>41</td><td>41</td><td>41</td><td>41</td><td>41</td><td>41</td><td>15/15</td><td>f17</td><td>63</td><td>1030</td><td>4005</td><td>12242</td><td>30677</td><td>56288</td><td>80472</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>30(24)</td><td>51(33)</td><td>56(35)</td><td>57(35)</td><td>57(35)</td><td>57(35)</td><td>57(35)</td><td></td><td>15/15</td><td>bp-bbob</td><td>0.98(0.6)</td><td>0.90(0.3)</td><td>1.1(2)</td><td>1.1(0.7)</td><td>0.76(0.3)</td><td>0.77(0.0)<sup>†</sup></td><td>0.78(0.3)</td><td>15/15</td></tr><tr><td>bp-sbx</td><td>30(18)</td><td>45(21)</td><td>50(24)</td><td>52(24)</td><td>52(24)</td><td>52(24)</td><td>52(24)</td><td></td><td>15/15</td><td>bp-sbx</td><td>1.4(0.9)</td><td>1.0(0.4)</td><td>1.4(2)</td><td>1.2(0.7)</td><td>0.79(0.4)</td><td>1.1(0.4)</td><td>1.1(0.3)</td><td>14/15</td></tr><tr><td>bt-bbob</td><td>22(3)</td><td>33(8)</td><td>41(8)</td><td>48(8)</td><td>55(8)</td><td>69(9)</td><td>87(7)</td><td></td><td>15/15</td><td>bt-bbob</td><td>0.97(0.6)</td><td>1.1(0.2)</td><td>2.0(2)</td><td>1.6(0.5)</td><td>1.0(0.9)</td><td>1.1(0.4)</td><td>0.98(0.3)</td><td>15/15</td></tr><tr><td>bt-sbx</td><td>23(8)</td><td>37(9)</td><td>44(10)</td><td>52(9)</td><td>59(8)</td><td>73(9)</td><td>87(11)</td><td></td><td>15/15</td><td>bt-sbx</td><td>1.8(1)</td><td>0.95(0.2)</td><td>1.1(2)</td><td>1.7(1)</td><td>1.9(3)</td><td>1.6(2)</td><td>1.6(1)</td><td>13/15</td></tr><tr><td></td><td>f6</td><td>1296</td><td>2343</td><td>3413</td><td>4255</td><td>5220</td><td>6728</td><td>8409</td><td></td><td>f18</td><td>621</td><td>3972</td><td>19561</td><td>28555</td><td>67569</td><td>1.3e5</td><td>1.5e5</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>1.5(0.8)</td><td>1.2(0.6)</td><td>1.1(0.5)</td><td>1.1(0.4)</td><td>1.1(0.3)</td><td>1.1(0.3)</td><td>1.1(0.3)</td><td></td><td>15/15</td><td>bp-bbob</td><td>0.85(0.3)</td><td>0.9(0.2)</td><td>0.87(0.8)</td><td>1.3(0.4)</td><td>0.80(0.3)</td><td>0.72(0.5)</td><td>0.76(0.4)</td><td>14/15</td></tr><tr><td>bp-sbx</td><td>1.9(0.9)</td><td>1.5(0.6)</td><td>1.4(0.5)</td><td>1.4(0.5)</td><td>1.3(0.4)</td><td>1.3(0.4)</td><td>1.3(0.3)</td><td></td><td>15/15</td><td>bp-sbx</td><td>0.88(0.2)</td><td>0.46(0.1)<sup>†</sup></td><td>0.86(0.6)</td><td>1.3(0.9)</td><td>1.8(2)</td><td>3.0(3)</td><td>3.5(5)</td><td>5/15</td></tr><tr><td>bt-bbob</td><td>1.5(0.4)</td><td>1.2(0.2)</td><td>1.1(0.1)</td><td>1.1(0.1)</td><td>1.1(0.1)</td><td>1.1(0.1)</td><td>1.1(0.2)</td><td></td><td>15/15</td><td>bt-bbob</td><td>0.96(0.2)</td><td>0.97(0.1)<sup>†</sup></td><td>1.2(1.0)</td><td>3.4(4)</td><td>2.3(2)</td><td>2.1(2)</td><td>2.0(2)</td><td>7/15</td></tr><tr><td>bt-sbx</td><td>1.4(0.3)</td><td>1.2(0.2)</td><td>1.1(0.2)</td><td>1.1(0.1)</td><td>1.1(0.1)</td><td>1.1(0.2)</td><td>1.1(0.2)</td><td></td><td>15/15</td><td>bt-sbx</td><td>1.1(0.3)</td><td>0.55(0.2)<sup>†</sup></td><td>0.89(0.5)</td><td>2.0(2)</td><td>2.6(3)</td><td>2.1(2)</td><td>2.9(2)</td><td>5/15</td></tr><tr><td></td><td>f7</td><td>1351</td><td>4274</td><td>9503</td><td>16523</td><td>16524</td><td>16524</td><td>16969</td><td></td><td>f19</td><td>1</td><td>1</td><td>3.4e5</td><td>4.7e6</td><td>6.2e6</td><td>6.7e6</td><td>6.7e6</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>0.72(0.2)<sup>†</sup></td><td>2.8(2)</td><td>1.6(0.6)</td><td>1.0(0.4)</td><td>1.0(0.4)</td><td>1.0(0.4)</td><td>1.0(0.4)</td><td></td><td>15/15</td><td>bp-bbob</td><td>1.0(1)</td><td>1.0(1)</td><td>2.8(3)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bp-sbx</td><td>0.96(0.3)</td><td>2.8(2)</td><td>1.9(0.6)</td><td>1.2(0.4)</td><td>1.2(0.4)</td><td>1.2(0.4)</td><td>1.2(0.4)</td><td></td><td>15/15</td><td>bp-sbx</td><td>1.0(1)</td><td>1.0(1)</td><td>8.7(9)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-bbob</td><td>1.1(0.4)</td><td>4.6(3)</td><td>3.2(2)</td><td>2.0(1)</td><td>2.0(1)</td><td>2.0(1)</td><td>1.9(1)</td><td></td><td>15/15</td><td>bt-bbob</td><td>1.0(1)</td><td>1.0(1)</td><td>8.5(9)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-sbx</td><td>1.7(2)</td><td>5.4(2)</td><td>4.3(3)</td><td>2.8(2)</td><td>2.9(1)</td><td>2.9(1)</td><td>2.8(1)</td><td></td><td>15/15</td><td>bt-sbx</td><td>1.0(1)</td><td>1.0(1)</td><td>8.5(9)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td></td><td>f8</td><td>2039</td><td>3871</td><td>4040</td><td>4148</td><td>4219</td><td>4371</td><td>4484</td><td></td><td>f20</td><td>82</td><td>46150</td><td>3.1e6</td><td>5.5e6</td><td>5.5e6</td><td>5.6e6</td><td>5.6e6</td><td>14/15</td></tr><tr><td>bp-bbob</td><td>3.4(1.0)</td><td>3.4(0.5)</td><td>3.6(0.5)</td><td>3.7(0.5)</td><td>3.7(0.5)</td><td>3.8(0.4)</td><td>3.8(0.5)</td><td></td><td>15/15</td><td>bp-bbob</td><td>3.8(0.7)</td><td>5.5(4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bp-sbx</td><td>3.4(0.8)</td><td>3.9(2)</td><td>4.1(2)</td><td>4.2(2)</td><td>4.3(2)</td><td>4.3(2)</td><td>4.3(2)</td><td></td><td>15/15</td><td>bp-bbob</td><td>3.5(1)</td><td>11(13)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-bbob</td><td>3.4(0.8)</td><td>3.6(0.6)</td><td>3.8(0.5)</td><td>3.9(0.5)</td><td>4.0(0.5)</td><td>4.0(0.5)</td><td>4.0(0.5)</td><td></td><td>15/15</td><td>bp-bbob</td><td>4.0(2)</td><td>6.2(7)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-sbx</td><td>3.6(0.7)</td><td>3.4(0.3)</td><td>3.7(0.2)</td><td>3.8(0.2)</td><td>3.8(0.2)</td><td>3.8(0.2)</td><td>3.9(0.2)</td><td></td><td>15/15</td><td>bt-sbx</td><td>4.4(2)</td><td>14(14)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td></td><td>f9</td><td>1716</td><td>3102</td><td>3277</td><td>3379</td><td>3455</td><td>3594</td><td>3727</td><td></td><td>f21</td><td>561</td><td>6541</td><td>14103</td><td>14318</td><td>14643</td><td>15567</td><td>17589</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>3.8(0.9)</td><td>5.3(4)</td><td>5.6(3)</td><td>5.6(3)</td><td>5.7(3)</td><td>5.6(3)</td><td>5.6(3)</td><td></td><td>15/15</td><td>bp-bbob</td><td>1.4(1)</td><td>30(31)</td><td>29(25)</td><td>29(46)</td><td>28(51)</td><td>26(33)</td><td>23(40)</td><td>5/15</td></tr><tr><td>bp-sbx</td><td>4.0(0.7)</td><td>5.2(3)</td><td>5.5(3)</td><td>5.6(3)</td><td>5.6(3)</td><td>5.6(3)</td><td>5.5(3)</td><td></td><td>15/15</td><td>bp-sbx</td><td>5.4(11)</td><td>25(39)</td><td>23(39)</td><td>23(18)</td><td>22(33)</td><td>21(20)</td><td>19(32)</td><td>6/15</td></tr><tr><td>bt-bbob</td><td>3.9(0.8)</td><td>4.0(0.4)</td><td>4.3(0.4)</td><td>4.4(0.4)</td><td>4.4(0.4)</td><td>4.4(0.4)</td><td>4.4(0.4)</td><td></td><td>15/15</td><td>bt-bbob</td><td>7.9(24)</td><td>45(73)</td><td>40(50)</td><td>39(35)</td><td>38(48)</td><td>36(26)</td><td>32(34)</td><td>4/15</td></tr><tr><td>bt-sbx</td><td>3.9(0.8)</td><td>4.5(0.6)</td><td>4.8(0.6)</td><td>4.9(0.6)</td><td>4.9(0.6)</td><td>4.9(0.6)</td><td>4.9(0.6)</td><td></td><td>15/15</td><td>bt-sbx</td><td>4.1(5)</td><td>16(26)</td><td>10(10)</td><td>10(8)</td><td>10(7)</td><td>9(16)</td><td>8(13)</td><td>9/15</td></tr><tr><td></td><td>f10</td><td>7413</td><td>8661</td><td>10735</td><td>13641</td><td>14920</td><td>17073</td><td>17476</td><td></td><td>f22</td><td>467</td><td>5580</td><td>23491</td><td>24163</td><td>24948</td><td>26847</td><td>1.3e5</td><td>12/15</td></tr><tr><td>bp-bbob</td><td>1.1(0.1)</td><td>1.1(0.1)</td><td>1.0(0.1)</td><td>0.87(0.1)<sup>†</sup></td><td>0.82(0.0)<sup>†</sup></td><td>0.76(0.0)<sup>†</sup></td><td>0.77(0.0)<sup>†</sup></td><td></td><td>15/15</td><td>bp-bbob</td><td>16(28)</td><td>42(35)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bp-sbx</td><td>1.1(0.2)</td><td>1.2(0.1)</td><td>1.1(0.1)</td><td>0.90(0.1)<sup>†</sup></td><td>0.85(0.0)<sup>†</sup></td><td>0.79(0.0)<sup>†</sup></td><td>0.81(0.0)<sup>†</sup></td><td></td><td>15/15</td><td>bp-sbx</td><td>13(30)</td><td>49(59)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-bbob</td><td>1.6(0.6)</td><td>1.6(0.5)</td><td>1.3(0.3)</td><td>1.1(0.3)</td><td>1.0(0.2)</td><td>0.95(0.2)</td><td>0.96(0.2)</td><td></td><td>15/15</td><td>bt-bbob</td><td>33(6)</td><td>57(63)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-sbx</td><td>1.5(0.4)</td><td>1.9(1)</td><td>1.8(0.9)</td><td>1.6(0.7)</td><td>1.6(0.6)</td><td>1.5(0.5)</td><td>1.5(0.5)</td><td></td><td>15/15</td><td>bt-sbx</td><td>4.6(0.8)</td><td>43(39)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td></td><td>f11</td><td>1002</td><td>2228</td><td>6278</td><td>8586</td><td>9762</td><td>12285</td><td>14831</td><td></td><td>f23</td><td>3.0</td><td>1614</td><td>67457</td><td>3.7e5</td><td>4.9e5</td><td>8.1e5</td><td>8.4e5</td><td>15/15</td></tr><tr><td>bp-bbob</td><td>4.2(0.3)</td><td>2.2(0.2)</td><td>0.85(0.1)<sup>†</sup></td><td>0.67(0.0)<sup>†</sup></td><td>0.62(0.0)<sup>†</sup></td><td>0.55(0.0)<sup>†</sup></td><td>0.50(0.0)<sup>†</sup></td><td></td><td>15/15</td><td>bp-bbob</td><td>1.6(2)</td><td>22(20)</td><td>1.8(2)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bp-sbx</td><td>4.3(0.4)</td><td>2.2(0.1)</td><td>0.88(0.1)<sup>†</sup></td><td>0.69(0.0)<sup>†</sup></td><td>0.64(0.0)<sup>†</sup></td><td>0.57(0.0)<sup>†</sup></td><td>0.51(0.0)<sup>†</sup></td><td></td><td>15/15</td><td>bp-sbx</td><td>2.4(3)</td><td>56(67)</td><td>2.6(2)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-bbob</td><td>7.4(2)</td><td>4.4(2)</td><td>1.6(0.7)</td><td>1.3(0.5)</td><td>1.1(0.4)</td><td>0.96(0.3)</td><td>0.84(0.3)</td><td></td><td>15/15</td><td>bt-bbob</td><td>2.3(4)</td><td>90(119)</td><td>5.4(5)</td><td>7.1(11)</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-sbx</td><td>10(6)</td><td>5.5(3)</td><td>2.2(1.0)</td><td>2.0(1.0)</td><td>2.0(0.9)</td><td>1.8(1)</td><td>1.6(0.9)</td><td></td><td>15/15</td><td>bt-sbx</td><td>2.0(2)</td><td>179(252)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td></td><td>f12</td><td>1042</td><td>1938</td><td>2740</td><td>3156</td><td>4140</td><td>12407</td><td>13827</td><td></td><td>f24</td><td>1.3e6</td><td>7.5e6</td><td>5.2e7</td><td>5.2e7</td><td>5.2e7</td><td>5.2e7</td><td>5.2e7</td><td>3/15</td></tr><tr><td>bp-bbob</td><td>2.6(2)</td><td>3.5(3)</td><td>4.0(2)</td><td>4.2(2)</td><td>3.7(1)</td><td>1.5(0.5)</td><td>1.6(0.4)</td><td></td><td>15/15</td><td>bp-bbob</td><td>0.50(0.5)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bp-sbx</td><td>2.5(0.9)</td><td>2.4(2)</td><td>2.9(2)</td><td>3.3(1)</td><td>3.1(1)</td><td>1.3(0.3)</td><td>1.4(0.2)</td><td></td><td>15/15</td><td>bp-sbx</td><td>0.48(0.4)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-bbob</td><td>3.3(3)</td><td>3.3(3)</td><td>3.7(2)</td><td>3.9(2)</td><td>3.5(1)</td><td>1.5(0.3)</td><td>1.5(0.3)</td><td></td><td>15/15</td><td>bt-bbob</td><td>0.37(0.3)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr><tr><td>bt-sbx</td><td>2.2(0.3)</td><td>2.5(3)</td><td>3.0(3)</td><td>3.8(2)</td><td>3.7(2)</td><td>1.6(0.6)</td><td>1.6(0.6)</td><td></td><td>15/15</td><td>bt-sbx</td><td>0.30(0.2)</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>∞</td><td>0/15</td></tr></tbody></table>