On the Impact of Active Covariance Matrix Adaptation in
the CMA-ES With Mirrored Mutations and Small Initial
Population Size on the Noiseless BBOB Testbed
Dimo Brockhoff

Anne Auger

Nikolaus Hansen

INRIA Lille - Nord Europe
Dolphin team
59650 Villeneuve d’Ascq
FranceProjet TAO, INRIA
Saclay—Ile-de-France
LRI, Bât 490, Univ. Paris-Sud
91405 Orsay Cedex, France

Projet TAO, INRIA
Saclay—Ile-de-France
LRI, Bât 490, Univ. Paris-Sud
91405 Orsay Cedex, Franceanne.auger@inria.fr

nikolaus.hansen@inria.fr

dimo.brockhoff@inria.fr
ABSTRACT(IPOP-CMA-ES, [2]). Two other improvements, are the active covariance matrix update [9] and mirrored mutations[1]. In an accompanying paper [3], it is shown that both active covariance matrix adaptation and mirrored mutations
can be easily combined in the standard IPOP-CMA-ES.
Here we pursue the question how strong is the inﬂuenceof the active covariance matrix adaptation on the overall
performance. To this end, we compare the IPOP-CMA-ES
with mirrored mutations and a small initial population sizewith the algorithm enhanced by active covariance matrix
adaptation on the noiseless BBOB test bed. We ﬁnd thatthe additional active covariance matrix adaptation improves
the algorithm signiﬁcantly on about one third of all functions
while no detrimental eﬀect can be observed.

Mirrored mutations as well as active covariance matrix adaptation are two techniques that have been introduced into thewell-known CMA-ES algorithm for numerical optimization.
Here, we investigate the impact of active covariance matrix
adaptation in the IPOP-CMA-ES with mirrored mutation
and a small initial population size. Active covariance matrix adaptation improves the performance on 8 of the 24
benchmark functions of the noiseless BBOB test bed. The
eﬀect is the largest on the ill-conditioned functions with thelargest improvement on the discus function where the expected runtime is more than halved. On the other hand, no
statistically signiﬁcant adverse eﬀects can be observed.

Categories and Subject Descriptors

2. ALGORITHM PRESENTATION

G.1.6 [Numerical Analysis]: Optimization—global optimization, unconstrained optimization; F.2.1 [Analysis of
Algorithms and Problem Complexity]: Numerical Algorithms and Problems

General Terms
Algorithms

Keywords
Benchmarking, Black-box optimization

1.

INTRODUCTION

The covariance matrix adaptation evolution strategy (CMAES) is considered as one of the standard stochastic optimization algorithms for continuous domain. In recent years, several algorithmic improvements of the original version havebeen made. One is the idea of increasing the population
size (by a factor of two) after each restart of the algorithm

The baseline algorithm for our comparison is the IPOPCMA-ES with mirrored mutations and an initial population
size of (4 + 3 log(D))/2, i.e., a population size that ishalf as large as in the original IPOP-CMA-ES where D is
the problem dimension. The reason is that in initial experiments, mirrored mutations showed a larger impact when
the population size is small [4]. We denote this baseline
algorithm as CMAmh 1 . On the other hand, we have the algorithm CMAmah with additional active covariance matrix
adaptation which is implemented according to [8].
Both algorithms are run for a maximum of 2 · 105 · D
function evaluations and restarted up to 10 times. Besides
the reduced initial population size, all parameters are set
as recommended in the original CMA-ES. The MATLAB
source code is version 3.54.beta.mirrors of the CMA-ES andcan be found at http://canadafrance.gforge.inria.fr/
mirroring/.

3. TIMING EXPERIMENTS
In order to see the dependency of the algorithms on the
problem dimension, the requested BBOB’2012 timing experiment has been performed for the two algorithms CMAmh
and CMAmah on an Intel Core2 Duo T9600 laptop with
2.80GHz, 4.0GB of RAM, and MATLAB R2008b on Windows Vista SP2. The algorithms have been restarted for
up to 2 · 105 N function evaluations until 30 seconds have

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
GECCO’12 Companion, July 7–11, 2012, Philadelphia, PA, USA.
Copyright 2012 ACM 978-1-4503-1178-6/12/07 ...$10.00.

1
The index ’m’ stands for mirrored mutations and the index’h’ for half the standard population size.

291

passed. The per-function-evaluation-runtimes were 22; 19;
12; 8.3; 5.8; 5.8 and 12 times 10−4 seconds for the CMAmh
and 21; 19; 11; 8.3; 6.1; 5.7 and 11 times 10−4 seconds for
the CMAmah in 2, 3, 5, 10, 20, 40, and 80 dimensions respectively.

4.

5.

REFERENCES[1] A. Auger, D. Brockhoﬀ, and N. Hansen. Mirrored
Sampling in Evolution Strategies With Weighted
Recombination. In Genetic and Evolutionary
Computation Conference (GECCO 2011), pages
861–868. ACM, 2011.
[2] A. Auger and N. Hansen. A Restart CMA Evolution
Strategy With Increasing Population Size. In Congress
on Evolutionary Computation (CEC 2005), volume 2,
pages 1769–1776. IEEE Press, 2005.
[3] D. Brockhoﬀ, A. Auger, and N. Hansen. On the Eﬀect
of Mirroring in the IPOP Active CMA-ES on the
Noiseless BBOB Testbed. In GECCO (Companion)
workshop on Black-Box Optimization Benchmarking(BBOB’2012). ACM, 2012.
[4] D. Brockhoﬀ, A. Auger, N. Hansen, D. V. Arnold, andT. Hohm. Mirrored Sampling and Sequential Selection
for Evolution Strategies. In Conference on Parallel
Problem Solving from Nature (PPSN XI), pages
11–21. Springer, 2010.
[5] S. Finck, N. Hansen, R. Ros, and A. Auger.
Real-parameter black-box optimization benchmarking
2009: Presentation of the noiseless functions.
Technical Report 2009/20, Research Center PPE,2009. Updated February 2010.
[6] N. Hansen, A. Auger, S. Finck, and R. Ros.
Real-parameter black-box optimization benchmarking
2012: Experimental setup. Technical report, INRIA,
2012.
[7] N. Hansen, S. Finck, R. Ros, and A. Auger.
Real-parameter black-box optimization benchmarking
2009: Noiseless functions deﬁnitions. Technical Report
RR-6829, INRIA, 2009. Updated February 2010.
[8] N. Hansen and R. Ros. Benchmarking a weighted
negative covariance matrix update on the BBOB-2010
noiseless testbed. In Genetic and Evolutionary
Computation Conference (GECCO 2010), pages
1673–1680, New York, NY, USA, 2010. ACM.
[9] G. Jastrebski and D. Arnold. Improving evolution
strategies through active covariance matrix
adaptation. In IEEE Congress on Evolutionary
Computation (CEC 2006), pages 2814–2821, 2006.
[10] K. Price. Diﬀerential evolution vs. the functions of thesecond. In Proceedings of the IEEE International
Congress on Evolutionary Computation (ICEO), pages
153–157, 1997.

RESULTS

Results from experiments according to [6] on the benchmark functions given in [5, 7] are presented in Figures 1,
2 and 3 and in Tables 1. The expected running time(ERT), used in the ﬁgures and table, depends on a given
target function value, ft = fopt + Δf , and is computed over
all relevant trials as the number of function evaluations executed during each trial while the best function value did notreach ft , summed over all trials and divided by the numberof trials that actually reached ft [6, 10]. Statistical signiﬁcance is tested with the rank-sum test for a given target Δft(10−8 as in Figure 1) using, for each trial, either the numberof needed function evaluations to reach Δft (inverted andmultiplied by −1), or, if the target was not reached, the bestΔf -value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial
under consideration.
The main observation is that the IPOP-CMA-ES variant
with additional covariance matrix adaptation is never statistically worse than the one without active covariance matrix
adaptation. The strongest positive eﬀect of the additional
covariance matrix adaptation can be observed for the illconditioned functions (Fig. 3) where for all test functionsin this class the CMAmah statistically signiﬁcantly outperforms the CMAmh for a target of 10−8 (Fig. 1). The factor of improvement in terms of ERT for these functions lies
between 1.3 (for the ellipsoid function) and 2.24 (for the
discus function) in 40D, see Table 1. Also on the separable
ellipsoid, the CMAmah statistically signiﬁcantly outperforms
the CMAmh for all tested dimensions. Also on other functions, statistically signiﬁcant improvements can be observed,
mainly found for larger dimensions: for 20D and 40D on f6 ,for 10D, 20D, and 40D on f7 , for 20D on f8 , for 20D and
40D on f12 , and for 40D on f18 . Overall, there are only two
functions where the CMAmh is (slightly) better in 20D fordiﬃcult targets than the CMAmah . Furthermore, within thegiven budget, for f3 , f4 , f19 and for all weakly structured
functions, unsuccessful runs occur for both algorithms witha slight advantage for the CMAmah (with 6 successful runs
more than the CMAmh ).
In summary, active covariance matrix adaptation regularly improves the performance of IPOP-CMA-ES with mirrored mutations, also when the initial population size issmaller than the standard population size.

292Figure 1: Expected running time (ERT in number of f -evaluations) divided by dimension for target function
value 10−8 as log10 values versus dimension. Diﬀerent symbols correspond to diﬀerent algorithms given in
the legend of f1 and f24 . Light symbols give the maximum number of function evaluations from the longest
trial divided by dimension. Horizontal lines give linear scaling, slanted dotted lines give quadratic scaling.
Black stars indicate statistically better result compared to all other algorithms with p < 0.01 and Bonferroni
correction number of dimensions (six). Legend: ◦:CMA mh, :CMA mah.

293

Figure 2: Expected running time (ERT in log10 of number of function evaluations) of CMA mh (x-axis) versus
CMA mah (y-axis) for 46 target values Δf ∈ [10−8 , 10] in each dimension on functions f1 –f24 . Markers on theupper or right edge indicate that the target value was never reached. Markers represent dimension: 2:+,
3:, 5:, 10:◦, 20:2, 40:3.

294

17 Schaﬀer F7, cond.10

19 Griewank-Rosenbrock20 Schwefel x*sin(x)23 Katsuuras

24 Lunacek bi-Rastrigin

22 Gallagher 21 peaks 18 Schaﬀer F7, cond.1000

21 Gallagher 101 peaks

12 Bent cigar

11 Discus

15 Rastrigin

16 Weierstrass

10 Ellipsoid

9 Rosenbrock rotated

14 Sum of diﬀ. powers

13 Sharp ridge

8 Rosenbrock original

7 Step-ellipsoid

6 Attractive sector

5 Linear slope

4 Skew Rastrigin-Bueche

3 Rastrigin separable

2 Ellipsoid separable

1 Sphere20-D

weak structure fcts

multi-modal fcts

ill-conditioned fcts

moderate fcts

separable fcts

all functions

5-D

Figure 3: Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in 5-D (left) and 20D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D)
to reach a target value fopt +Δf with Δf = 10k , where k ∈ {1, −1, −4, −8} is given by the ﬁrst value in the legend,
for CMA mh (◦) and CMA mah (). Light beige lines show the ECDF of FEvals for target value Δf = 10−8of all algorithms benchmarked during BBOB-2009. Right sub-columns: ECDF of FEval ratios of CMA mh
divided by CMA mah, all trial pairs for each function. Pairs where both trials failed are disregarded, pairs
where one trial failed are visible in the limits being > 0 or < 1. The legends indicate the number of functions
that were solved in at least one trial (CMA mh ﬁrst).

295

5-D
Δf
1e+1
f1
11
1: CMA 1.8(1)2: CMA 1.7(1)f2
83
1: CMA16(5)2: CMA13(3)f3716
1: CMA 1.7(2)2: CMA 2.4(2)f4809
1: CMA 1.8(2)2: CMA 2.5(2)f5
10
1: CMA 2.9(1)2: CMA 3.3(2)f6
114
1: CMA 1.8(0.9)
2: CMA 1.7(1.0)
f7
24
1: CMA 4.5(5)2: CMA 3.9(3)f8
73
1: CMA 2.5(1)2: CMA 1.8(0.6)
f935
1: CMA 5.0(2)2: CMA 4.4(1)f10349
1: CMA 4.1(1)2: CMA 2.9(1)f11
143
1: CMA12(4)2: CMA 6.5(2) 2
f12
108
1: CMA 8.3(10)2: CMA 6.5(8)f13132
1: CMA 4.7(5)2: CMA 4.0(4)f14
10
1: CMA 1.4(2)2: CMA 1.4(1)f15
511
1: CMA 3.2(3)2: CMA 2.0(2)f16120
1: CMA 2.9(1)2: CMA 2.3(1)f17
5.2
1: CMA 2.5(2)2: CMA 3.1(2)f18
103
1: CMA 0.51(0.3) ↓2: CMA 6.2(2)f19
1
1: CMA12(15)2: CMA17(12)f20
16
1: CMA 2.6(1)2: CMA 2.3(2)f21
41
1: CMA 3.3(1)2: CMA 3.6(4)f22
71
1: CMA 5.3(8)2: CMA 6.3(14)f23
3.0
1: CMA 4.5(10)2: CMA 1.9(2)f24
1622
1: CMA 1.4(1)2: CMA 1.6(1)

20-D

1e+1
1e-1
1e-3
1e-5
1e-7
#succ
#succ Δf
f1
43
43
43
43
43
15/15
15/15
1:
CMA
3.8(0.6)
10(1)
15(1)
22(2)
28(2)
15/15
15/15
10(1)
15(1)
21(1)
27(1)
15/15
15/15 2: CMA 3.8(0.8)
f2
385
387
390
391
393
15/15
15/15
38(2)
40(2)
41(2)
41(2)
15/15
15/15 1: CMA 32(5)
3
3
3
3
3
28(2)
30(1)
31(1)
32(1)
15/15
15/15 2: CMA 23(5)
f3
5066
7635
7643
7646
7651
15/15
1637
1646
1650
1654
15/15
∞
∞
∞
∞2.7e60/151004(1285)
999(1349)
997(1410)
995(1303)
5/15 1: CMA 13(16)4950(5828)
4945(5805)
4944(5660)
4941(5716)
1/15
3145(3852)
3129(3730)
3121(3743)
3114(3868)
2/15 2: CMA 13(7)
f4
4722
7666
7700
7758
1.4e5
9/15
1688
1817
1886
1903
15/15
∞
∞
∞
∞
∞2.5e60/15∞∞∞∞8.7e5
0/15 1: CMA∞∞∞∞∞2.5e60/15∞∞∞∞8.7e5
0/15 2: CMA
f
41
41
41
41
41
15/15
5
10
10
10
10
15/15
3.8(1)
3.8(1)
3.8(1)
3.8(1)
15/15
4.0(1)
4.1(1)
4.1(1)
4.1(1)
15/15 1: CMA 3.2(1)4.1(1)
4.1(1)
4.1(1)
4.1(1)
15/15
4.7(2)
4.8(2)
4.8(2)
4.8(2)
15/15 2: CMA 3.1(0.6)
f6
1296
3413
5220
6728
8409
15/15
281
580
1038
1332
15/15
1:
CMA
1.2(0.4)
1.1(0.3)
1.3(0.4)
1.8(0.5)
2.3(1)
15/15
1.8(0.9)
1.5(0.4)
1.1(0.2)
1.1(0.2)
15/15
1.0(0.2)
1.1(0.3)
1.2(0.3)
1.3(0.3) 2
15/15
1.7(0.5)
1.3(0.4)
0.94(0.3)
0.92(0.2) 15/15 2: CMA 1.2(0.3)
f7
1351
9503
16524
16524
16969
15/15
1171
1572
1572
1597
15/15
1:
CMA
2.9(1)
2.6(1)
1.6(0.6)
1.6(0.6)
1.6(0.6)
15/15
1.4(0.9)
1.4(0.6)
1.4(0.6)
1.4(0.7)
15/15
1.5(0.6)2
0.95(0.3) 2
0.95(0.3) 2
0.93(0.3) 2
15/15
1.4(0.7)
1.2(0.6)
1.2(0.6)
1.2(0.6)
15/15 2: CMA 1.7(1)
336
391
410
422
15/15
f8
2039
4040
4219
4371
4484
15/15
5.1(4)
5.3(3)
5.5(3)
5.6(3)
15/15 1: CMA 3.2(1)4.7(3)
4.8(3)
4.8(3)
4.8(3)
15/15
4.0(3)
4.3(2)
4.4(2)
4.5(2)
15/15 2: CMA 2.6(0.5)
3.6(2)
3.7(2)
3.6(2)
3.6(2)
15/15
214
300
335
369
15/15
f9
1716
3277
3455
3594
3727
15/15
4.5(0.9)
4.6(0.9)
4.6(0.8)
4.6(0.8)
15/15
9.0(5)
7.6(4)
7.4(3)
7.0(3)
15/15 1: CMA 3.1(2)4.9(3)
5.0(3)
4.9(3)
4.9(2)
15/15
7.5(5)
6.3(3)
6.1(3)
5.8(3)
15/15 2: CMA 2.9(0.9)
f10
7413
10735
14920
17073
17476
15/15
574
626
829
880
15/15
3.3(0.3)
3.2(0.3)
2.5(0.2)
2.5(0.2)
15/15 1: CMA 1.6(0.2)
1.4(0.1)
1.1(0.0)
0.95(0.0)
0.94(0.0)
15/15
2.4(0.3)3
2.3(0.3) 3
1.9(0.2)3
1.9(0.2) 3 15/15 2: CMA 1.2(0.2) 3
1.0(0.1)3
0.79(0.1) 3↓4
0.71(0.0) 3↓4
0.71(0.0) 3↓4 15/15
763
1177
1467
1673
15/15
f11
1002
6278
9762
12285
14831
15/15
3.0(0.3)
2.1(0.2)
1.7(0.1)
1.6(0.1)
15/15 1: CMA 11(0.9)
2.1(0.1)
1.5(0.1)
1.3(0.0)
1.1(0.0)
15/15
1.8(0.3)3
1.3(0.2) 3
1.1(0.1)3
1.0(0.1) 3 15/15 2: CMA 4.8(0.5) 3
0.92(0.1) 3
0.66(0.0) 3↓4
0.56(0.0) 3↓4
0.49(0.0) 3↓4 15/15
1e-1
12
9.2(3)
8.1(2)
88
21(3)15(2)3

1e-3
12
16(3)
16(2)
90
21(2)
16(2)3371
10(9)8.1(6)
250
8.9(2)
5.6(3)
58
2.5(0.9)
2.4(0.6)
19369
1.1(0.6)
1.1(0.7)
2662
2.5(1)
2.4(2)
899
2.6(3)
2.1(2)
3968
1.1(1)
1.5(1)
242
421(494)
518(491)38111
1.7(2)
1.6(2)
1674
40(62)
5.7(10)938
288(414)289(412)14249
38(70)
52(71)
6.4e6
2.2(2)∞

461
10(9)8.8(5)
1310
2.2(0.8)
1.7(0.8)
139
4.1(2)
3.4(0.9)20073
1.1(0.6)
1.1(0.7)
10449
1.2(1)
0.94(0.7)3669
1.5(1.0)
1.3(0.8)9280
1.3(0.6)
1.2(0.6)
1.2e5
2.5(2)
2.4(2)
54470
1.3(1)
1.2(1)
1705
41(63)
5.9(11)
1008
436(590)346(449)31654
23(32)
23(33)
9.6e6
1.5(2)
∞

1e-5
12
24(4)
23(2)
92
22(2)
17(2)31303
4.5(4)
3.9(2)
1752
2.5(1)
1.7(0.6)251
6.4(0.8)
4.1(1)3

20769
1.1(0.6)
1.1(0.7)11644
1.1(1.0)0.90(0.6)
6351
1.4(0.6)
1.1(0.5)10905
1.3(0.6)
1.1(0.7)
1.2e5
2.5(2)
2.4(2)
54861
1.3(1)
1.2(1)
1729
42(70)
6.0(11)
1040
424(548)336(512)33030
22(31)
22(31)
1.3e7
1.1(1)
∞

1e-7
12
31(4)
30(3)
94
23(2)
17(2)3

1494
15/15
f12
1042
4.5(4)
15/15 1: CMA 4.5(5)3.8(2)
15/15 2: CMA 1.2(1)
2255
15/15
f13
652
2.8(0.9)
15/15 1: CMA 3.7(3)2
1.6(0.5)
15/15 2: CMA 3.6(3)
476
15/15
f14
75
5.1(0.6)
15/15 1: CMA 2.3(1)3.5(0.4) 3 15/15 2: CMA 2.0(0.6)
21359
14/15
f15
30378
1.1(0.6)
15/15 1: CMA 0.98(0.6)
1.1(0.7)
15/15
2: CMA 0.92(0.6)
12095
15/15
f16
1384
1.1(1.0)
15/15
1: CMA 3.0(3)0.90(0.6) 15/15
2: CMA 2.3(3)
7934
15/15
f17
63
1.5(0.6)
15/15
1: CMA 1.8(0.8)
1.0(0.4)
15/15
2: CMA 2.0(1)
12469
15/15
f18
621
1.3(0.6)
15/15 1: CMA
3.3(8)
1.1(0.6)
15/15 2: CMA 0.95(0.5)
1.2e5
15/15
f19
1
2.5(2)
15/15 1: CMA
82(52)
2.3(2)
15/15 2: CMA 95(46)
55313
14/15
f20
82
1.3(1)
15/15 1: CMA
2.6(0.8)
1.2(1)
15/15 2: CMA 2.5(0.8)
1757
14/15
f21
561
42(69)
14/15 1: CMA
8.7(7)
6.1(11)
15/15 2: CMA 25(22)
1068
14/15
f22
467
414(526)
7/15 1: CMA
6.9(8)
329(385)
8/15 2: CMA187(29)
34256
15/15
f23
3.2
21(31)
9/15 1: CMA
2.8(3)
22(29)
9/15 2: CMA 2.9(3)
1.3e7
3/15
f24
1.3e6
1.1(1)
1/15 1: CMA
42(42)∞1.0e6
0/15 2: CMA 42(48)2740
6.4(5)
2.6(2)
2751
6.4(6)
4.2(3)
304
1.9(0.3)
1.9(0.3)
3.1e5
0.66(0.3)
0.64(0.3)
77015
1.1(0.7)
0.90(0.6)4005
7.7(6)
4.0(2)
19561
1.2(0.7)
1.2(0.7)
3.4e5
3.3(4)
2.5(4)
3.1e6
1.3(1)
0.87(0.4)14103
75(95)
76(86)
23491
∞
∞
67457
519(602)516(590)
5.2e7
∞
∞

4140
5.6(3)
2.5(1)
18749
1.8(0.7)
1.3(0.4)932
2.8(0.3)
2.3(0.3)2
3.2e5
0.67(0.3)
0.65(0.3)
1.9e5
1.0(0.8)
0.92(0.5)
30677
1.4(0.8)
0.92(0.4)
67569
1.1(0.7)
0.79(0.3)
6.2e6
1.8(2)
0.91(0.8)
5.5e6
10(12)
2.4(3)
14643
73(118)73(84)
24948
∞
∞
4.9e5
∞
∞
5.2e7
∞
∞

12407
2.4(1)
1.1(0.4)224455
2.4(1.0)
1.4(0.6)
1648
5.7(0.7)
3.4(0.4)3
4.5e5
0.50(0.3)↓2
0.48(0.2)↓2
2.0e5
1.1(0.8)
1.3(1)
56288
1.3(0.7)
0.96(0.4)
1.3e5
1.1(0.3)
0.88(0.4)
6.7e6
2.9(3)
1.3(1)
5.6e6
10(11)
3.2(4)
15567
69(111)69(109)
26847
∞
∞
8.1e5
∞
∞
5.2e7
∞
∞

13827
15/15
2.4(1)
15/15
1.1(0.3) 3
15/15
30201
15/15
2.7(0.7)
15/15
1.8(0.9) 
15/15
15661
15/15
1.2(0.1)
15/15
0.66(0.0) 3↓4 15/15
4.6e5
15/15
0.50(0.3) ↓2
15/15↓2
0.49(0.2)
15/15
2.2e5
15/15
1.0(0.8)
15/15
1.2(0.9)
15/15
80472
15/15
1.3(0.6)
15/15
0.94(0.3)
15/15
1.5e5
15/15
1.1(0.3)
15/15
0.91(0.3)
15/15
6.7e6
15/15
2.8(3)
3/15
1.3(1)
6/15
5.6e6
14/15
10(12)
1/15
3.2(4)
3/15
17589
15/15
61(72)
8/15
61(72)
8/15
1.3e5
12/15
∞1.1e60/15∞1.1e6
0/15
8.4e5
15/15
∞2.4e60/15∞2.4e6
0/15
5.2e7
3/15
∞4.0e60/15∞4.0e6
0/15

Table 1: ERT in number of function evaluations divided by the best ERT measured during BBOB-2009 givenin the respective ﬁrst row with the central 80% range divided by two in brackets for diﬀerent Δf values.
#succ is the number of trials that reached the ﬁnal target fopt + 10−8 . 1:CMA is CMA mh and 2:CMA is
CMA mah. Bold entries are statistically signiﬁcantly better compared to the other algorithm, with p = 0.05
or p = 10−k where k ∈ {2, 3, 4, . . . } is the number following the  symbol, with Bonferroni correction of 48. A ↓indicates the same tested against the best BBOB-2009.

296