BI-population CMA-ES Algorithms with Surrogate Models
and Line Searches
Ilya Loshchilov

Marc Schoenauer

Michèle Sebag

Laboratory of Intelligent
Systems (LIS)
EPFL, Lausanne, Switzerland

TAO, INRIA Saclay
Univ. Paris-Sud, Orsay, France

CNRS, LRI UMR 8623
Univ. Paris-Sud, Orsay, France

ilya.loshchilov@epfl.chmarc.schoenauer@inria.fr

michele.sebag@lri.fr

ABSTRACT

Keywords

In this paper, three extensions of the BI-population Covariance Matrix Adaptation Evolution Strategy with weighted
active covariance matrix update (BIPOP-aCMA-ES) are investigated. First, to address expensive optimization, we
benchmark a recently proposed extension of the self-adaptive
surrogate-assisted CMA-ES which benefits from more intensive surrogate model exploitation (BIPOP-saACM-k). Second, to address separable optimization, we propose a hybridof BIPOP-aCMA-ES and STEP algorithm with coordinatewise line search (BIPOP-aCMA-STEP). Third, we propose
HCMA, a hybrid of BIPOP-saACM-k, STEP and NEWUOA
to benefit both from surrogate models and line searches. All
algorithms were tested on the noiseless BBOB testbed using
restarts till a total number of function evaluations of 106 n
was reached, where n is the dimension of the function search
space.
The comparison shows that BIPOP-saACM-k outperforms
its predecessor BIPOP-saACM up to a factor of 2 on illconditioned problems, while BIPOP-aCMA-STEP outperforms the original BIPOP-based algorithms on separable
functions. The hybrid HCMA algorithm demonstrates thebest overall performance compared to the best algorithmsof the BBOB-2009, BBOB-2010 and BBOB-2012 when running for more than 100n function evaluations.

Benchmarking, black-box optimization, evolution strategy,
CMA-ES, self-adaptation, surrogate models, surrogate-assisted
optimization, separable optimization

Categories and Subject Descriptors
G.1.6 [Numerical Analysis]: Optimization—global optimization, unconstrained optimization; F.2.1 [Analysis of
Algorithms and Problem Complexity]: Numerical Algorithms and Problems

General Terms
Algorithms

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.
GECCO’13, July 6–10, 2013, Amsterdam, The Netherlands.
Copyright 2013 ACM 978-1-4503-1963-8/13/07 ...$10.00.

1. INTRODUCTION
The Covariance Matrix Adaptation Evolution Strategy(CMA-ES) [5] has become a popular tool for continuous
black-box optimization. However, CMA-ES is a rather local
optimization algorithm and can be ineffective when dealing with multi-modal optimization problems. In order toavoid premature convergence to local optima, several restart
strategies for CMA-ES have been proposed [1, 3, 14]. These
strategies, typically based on restarting and increasing (moregenerally, on varying) the population size and initial mutation step-size each time at least one stopping criterion of
CMA-ES is met, implement the simplest niching approach
but rather in the time than in space [21]. Thanks to its
population-based nature, the CMA-ES benefits from usinglarger population sizes on multi-modal functions with someglobal structure and, thus, typically outperforms quasi-Newton
methods such as BFGS [23]. The latter, however, often outperforms CMA-ES on a special but quite popular class ofnoiseless unimodal functions, when gradient information isuseful. This drawback of CMA-ES becomes especially undesirable when dealing with expensive optimization, where
each function evaluation takes some time and/or costs somemoney.
A common way to reduce the cost of optimization within
Evolutionary Algorithms is to periodically learn a surrogate
model of the expensive function and exploit it during thesearch by pre-filtering promising solutions and/or by optimizing the surrogate directly instead of the expensive function. These approaches have been adopted within Evolution
Strategies and CMA-ES by using different surrogate learning approaches: Radial Basis Functions network [7], Gaussian Processes [25], Artificial Neural Network [10], Support
Vector Regression [12], Local-Weighted Regression [11, 2],
Ranking Support Vector Machine (Ranking SVM) [11, 13,8]. More recently it has been shown [17] that key invarianceproperties of CMA-ES can be preserved even in surrogateassisted scenario by using Ranking SVM approach. Moreover, hyper-parameters used to build surrogate models canbe also adapted during the search allowing the user to define
only the range of these hyper-parameters, while their actual
values will be adapted by the s∗aACM-ES algorithm [17].

In this paper, we benchmark the recently proposed extension of s∗aACM-ES, referred to as s∗aACM-ES-k, where thesurrogate model is more intensively exploited by increasing
the population size used for its optimization. This extension
further improves the performance of the surrogate-assisted
CMA-ES on uni-modal functions, however, almost does not
improve the performance of multi-modal functions. This isespecially true for separable multi-modal functions, where
CMA-ES is outperformed by Evolutionary Algorithms which
explicitly or implicitly exploit separability. In order to investigate that would be the effect of such an exploitation
by CMA-ES, we present a hybrid version of CMA-ES algorithm and select the easiest point (STEP) [24, 19] algorithm, which performs coordinate-wise search and is quite
efficient on separable functions [24, 19]. Finally, to benefit
both from surrogate models and line searches, we propose ahybrid CMA (HCMA) which combines s∗aACM-ES-k, STEP
and NEWUOA algorithm [20].

2.

THE ALGORITHMS

for direct optimization. The use of Ranking SVM allows
to preserve invariance properties of CMA-ES w.r.t. rankpreserving transformations of the objective function and orthogonal transformations of the search space. The main loopof s∗ACM-ES: the surrogate model fˆ is optimized for n̂ generations by the CMA-ES, then the expensive function f is
optimized for one generation. To adjust the number of generations n̂ for the next time, the model error is computed as afraction of incorrectly predicted comparison relations that isobserved after ranking the last λ evaluated points according
to f and fˆ. The s∗ACM-ES performs an online optimizationof the surrogate model hyper-parameters during the optimization of the objective function by performing a searchin a space of model hyper-parameters, and by generating
λhyp surrogate models in each iteration. The information
about the fittest models with the smaller prediction error
on the last λ evaluated solutions is used to adjust the surrogate hyper-parameters for the next iteration. The detailed
description of s∗ACM-ES is given in [17].

2.4 The BIPOP-s∗ACM-ES-k

2.1 The (µ/µw , λ)-CMA-ES
In each iteration t, (µ/µw , λ)-CMA-ES [5] samples λ new
solutions xi ∈ Rn , where i = 1, . . . , λ, and selects the best
µ among them. These µ points update the distribution ofparameters of the algorithm to increase the probability ofsuccessful steps in iteration t + 1. The sampling is defined2by a multi-variate normal distribution, N (mt , σ t C t ), withtcurrent mean of distribution m , n × n covariance matrix C t
and step-size σ t . Most of algorithms which will be benchmarked in this paper are based on an active version of the
CMA-ES (in short, aCMA-ES) with negative covariance matrix update [9, 6].

2.2 The BIPOP-aCMA-ES
In BIPOP-CMA-ES [3] after the first single run (when atleast one of stopping criteria is met) with default population
size, the CMA-ES is restarted in one of two possible regimes
by taking into account the budget of function evaluations
spent in the corresponding regime. Each time the algorithm
is restarted, the regime with smallest budget used so far is
used.
Under the first regime the population size is doubled as
λlarge = 2irestart λdef ault in each restart irestart and uses
00some fixed initial step-size σlarge= σdefault .
Under the second regime the CMA-ES is restarted with0some small population size λsmall and step-size σsmall, whereλsmall is set toU [0,1]2 
λλsmall = λdef ault 12 λ large,
def ault(1)

Here U [0, 1] denotes independent uniformly distributed
numbers in [0, 1] and λsmall ∈ [λdef ault , λ/2]. The initial
00−2U [0,1]
step-size is set to σsmall= σdef.
ault × 10
The active version of BIPOP-CMA-ES (BIPOP-aCMAES) has been proposed in [14].

2.3 The BIPOP-s∗ACM-ES
The s∗ACM-ES [17] is the surrogate-assisted version of the(µ/µw , λ)-CMA-ES, where Ranking SVM-based surrogate
model is used periodically instead of the expensive function

A more intensive exploitation of a surrogate model can be
useful if the model is sufficiently precise. An extension of s∗
ACM-ES, based on this idea, referred to as s∗ACM-ES-k, hasbeen proposed in [18]. In s∗ACM-ES-k, fˆ is optimized for n̂
generations by CMA-ES with population size λ = kλ λdef ault
and number of parents µ = kµ µdef ault , where kλ ≥ 1 andkµ ≥ 1. Then, f is optimized for 1 generation by CMA-ES
with population size λ = λdef ault and number of parentsµ = µdef ault . The original s∗ACM-ES corresponds to s∗
ACM-ES-k with kλ = 1 and kµ = 1. Thus, larger valuesof kλ and kµ lead to a more intensive exploitation of fˆ. To
prevent the algorithm from a potential divergence when n̂
oscillates around 0 and 1, kλ > 1 is used only if n̂ ≥ n̂kλ ,where n̂kλ is the number of generations which corresponds
to an ”accurate enough” model to be intensively exploited.

2.5 The BIPOP-aCMA-STEP
The select the easiest point (STEP) [24, 19] is a simple
line search method which is based on iterative interval division. The STEP determines the next point to evaluate byanalyzing the usefulness of evaluation the function at the
middle of an interval [xi , xi−1 ] with respect to the intervaldifficulty computed as [24]D=p4ŷ − 2∆y + 4 ŷ 2 − ŷ∆y
,
∆x2

(2)

where ∆x = xi − xi−1 , ∆y = f (xi ) − f (xi−1 ), ŷ = f ∗ −f (xi−1 ) + l, f ∗ is the best solution found so far by the STEP
and l is the tolerance (i.e., the precision required for thevalue of the optimum).
The STEP can be applied to n-dimensional optimization
problem by performing n coordinate-wise iterative or parallel searches. A variant of iterative search: optimize f along
the coordinate i for NST EP function evaluations or until astopping criterion is met, then fix the best xi , increment iand repeat the procedure again while i ≤ n. This approachis quite straightforward but requires NST EP to be set a priori. However, NST EP can be initialized by a small value andthen incremented (e.g., doubled) in a similar way as the population size in BIPOP-aCMA-ES each time STEP finishes
the search on n coordinates. This approach might lead to aloss of a factor of about 2 in terms of number of function evaluations compared to an optimal setting of NST EP . Moreover, some step function-like improvement of the objectivefunction can be observed due to switch between variables,
this also means that objective values of tested solutions are
far away in the objective space during the coordinate-wise
search on all but n-th coordinate.
We propose a parallel STEP search where at each iteration one step per coordinate is performed, then x∗i whichcorresponds to the best solution so far along the coordinate
i is used to construct a recommended solution x∗ combinedof x∗i , i = 1, . . . , n. The objective value of this solution isestimated and is used for two purposes. First, if the value of
this solution is worse (up to some precision p) than the oneof a solution recommended in the previous iteration, thenf is non-separable or/and noisy. In this case, it is not recommended to use the STEP. Second, the new recommended
solution can be of the same quality as the previous one even
if the problem is separable. In this case, the STEP still canrun while its best recommended solution is better than the
one found by its competitor algorithm running in parallel
with, e.g., the same budget of function evaluations.
In BIPOP-aCMA-STEP algorithm, a fraction ρST EP offunction evaluations is allocated to the STEP, while 1-ρST EP
is allocated to the BIPOP-aCMA-ES. At each iteration after
the first nM inIterST EP iterations, the best solutions of twoalgorithms are compared and the STEP is stopped if its bestsolution is worse or if its current solution is worse than its
previously recommended solution.

2.6 The HCMAThe HCMA algorithm is a hybrid of BIPOP-s∗aACM-ESk and STEP algorithms coupled as described in the previous
section. Additionally, the NEWUOA algorithm [20] is used
for the first 10n function evaluations to insure good convergence of the HCMA on simple functions which can be
efficiently approximated by quadratic surrogate models. All
solutions generated by STEP and NEWUOA are not used by
BIPOP-s∗aACM-ES-k for surrogate learning since the sampling/test distribution of CMA-ES might not correspond to
the training distribution defined by these solutions.

2.7 The Benchmarked Algorithms
For benchmarking we consider five CMA-ES algorithms
in BIPOP restart scenario [3]: BIPOP-aCMA-ES [16, 3],
BIPOP-s∗aACM-ES [17], BIPOP-s∗aACM-ES-k [18], BIPOPaCMA-STEP and HCMA1 . For all but BIPOP-s∗aACM-ESk [18] and BIPOP-aCMA-STEP algorithms the BBOB results are taken from the literature. The setting of parameters of BIPOP-s∗aACM-ES-k is given in [18], the maximum
number of function evaluations is set to 106 n. However,surrogate models are used only for the first 2 · 105 (104 n
for n = 20) function evaluations for n = 2, 3, 5, 10, 20 and
2 · 103 n evaluations for n = 40 in order to reduce the overallrunning time and to fit to the BBOB-2013 focus on expensive optimization with the budget of up to 103 n function
evaluations. The value of kλ is set to 1, 1, 1, 10, 100, 1000
for n = 2, 3, 5, 10, 20, 40, respectively; kµ = 1 for all dimensions; n̂kλ = 4. The value of l (respectively, p) of the STEP
is set to 10−10 (respectively, 10−8 ), nM inIterST EP = 10 andρST EP = 0.5. The following parameters of NEWUOA are
1
For the sake of reproducibility the source code is available
at http://sites.google.com/site/bbobgecco2013/used [22]: m = 2n + 1 interpolation points, the initial radiusρbeg of the search region is set to 10, the final radius ρend is
set to 10−16 , the maximum number of function evaluations
is set to 10n.

3. RESULTS
Results from experiments according to [4] on the benchmark functions given in [4] are presented in Figures 1 and
2 and in Table 1. The expected running time (ERT),
used in the figures and table, depends on a given target function value, ft = fopt + ∆f , and is computed over all relevant
trials (on the first 15 instances) as the number of function
evaluations executed during each trial while the best function value did not reach ft , summed over all trials and divided by the number of trials that actually reached ft [4].
Statistical significance is tested with the rank-sum test
for a given target ∆ft (10−8 as in Figure 1) using, for eachtrial, either the number of needed function evaluations toreach ∆ft (inverted and multiplied by −1), or, if the target
was not reached, the best ∆f -value achieved, measured only
up to the smallest number of overall function evaluations forany unsuccessful trial under consideration.
The BIPOP-s∗aACM-ES and BIPOP-s∗aACM-ES-k perform similarly for n = 2, 3, 5 and before 104 n function evaluations because kλ = 1 is used, after 104 n evaluations BIPOPs∗aACM-ES-k represents BIPOP-aCMA-ES. For larger n,
BIPOP-s∗aACM-ES-k outperforms BIPOP-s∗aACM-ES on
unimodal ill-conditioned problems, e.g., for 20-dimensional
ones the target function values 10−8 can be reached faster(see Figure 1, Table 1): by a factor of 1.8 on f2 Separable
Ellipsoid and f10 Rotated Ellipsoid functions, by a factorof 1.5 on f11 Discus, by a factor of 1.1 on f12 Bent cigar,
by a factor of 1.8 on f13 Sharp ridge and by a factor of 1.2
on f14 Sum of different powers. On f8 and f9 Rosenbrock’s
function the speedup of a factor of about 1.3 is obtained.
A relative loss of performance (compared to BIPOP-s∗aACM-ES) on multi-modal functions can be explained by
not using surrogate models after 104 n and thus using theoriginal BIPOP-aCMA-ES which usually is outperformed by
BIPOP-s∗aACM-ES. A loss of performance is observed on f6
Attractive sector (by a factor of 1.2), f17 Schaffer (by a factorof 1.2) and f23 Katsuuras (by a factor of 1.5). However, thereason of this loss remains unclear for f17 and f23 functions
because a similar loss of performance is also observed on thesame problems in dimension 5 where kλ = 1 for both algorithms (i.e., there is no intensive surrogate model exploitation). This difference might be explained by the stochasticity of multi-modal optimization or by fact of adapting an
additional hyper-parameter - a stopping criterion of SVM
surrogate learning procedure in BIPOP-s∗aACM-ES-k [18],whose values were fixed in BIPOP-s∗aACM-ES [17]. The
adaptation of this hyper-parameter probably is not efficientenough compared to the offline tuned value, but this adaptation makes the algorithm more parameter-free. While there
is no data to assess the effect of more intensive surrogate
model exploitation for budgets larger than 104 n, the loss of
performance on f6 is clearly due to kλ > 1. A simple way to
reduce this loss would be to increment n̂kλ , but this might
decrease the speedup on ill-conditioned problems back toward the level of BIPOP-s∗aACM-ES.
The results shown in Figure 2 suggest that separable multimodal functions such as f3 Separable Rastrigin and f4 Separable Skew Rastrigin-Bueche can be easily solved by BIPOP-

aCMA-STEP and HCMA thanks to its STEP part. Thus,
this kind of hybridization is beneficial if the main target is tosolve the maximum number of problems within a relatively
large budget of function evaluations.
The HCMA demonstrates the best performance among
the tested algorithms after 100n function evaluations. The
hybridization with STEP allows to exploit the separability
when it is detected, i.e., when STEP generates better solutions than BIPOP-s∗aACM-ES-k. In a few cases whenseparability was not detected successfully 3-dimensional f3
Rastigin and 3-, 5-dimensional f4 Skew Rastrigin-Bueche
separable functions, it seems that 10n function evaluations(nM inIterST EP = 10) sometimes is not sufficient to see thedifference between STEP and BIPOP-s∗aACM-ES-k on these
problems.

4.

CPU TIMING EXPERIMENTThe time complexity of BIPOP-aCMA-STEP is essentially the same as the one of BIPOP-aCMA-ES since the
STEP has linear time complexity and its relatively small”constant factor” (see Eq.(2)) depends only on a particular
implementation. The time complexity of BIPOP-s∗aACMES-k was measured after 15 runs on 20-dimensional f8 , thecost per function evaluation is about 0.45 seconds (runs were
performed on different machines with 2.2 and 2.4 GHz cores
under Ubuntu 10.04 using Octave 3.2.3). These results are
similar to the ones of [15], the time complexity can be reduced by a factor of λhyp = 20 by switching off the adaptation of surrogate hyper-parameters.

5.

CONCLUSION

In this paper, we have compared the recently proposed
self-adaptive surrogate-assisted BIPOP-s∗aACM-ES-k withintensive surrogate model exploitation and its predecessor
BIPOP-s∗aACM-ES with the BIPOP-aCMA-ES. The comparison shows that the performance of surrogate-assisted
CMA-ES can be further improved on ill-conditioned functions, e.g., on 20-dimensional f10 Rotated Ellipsoid the BIPOPs∗aACM-ES-k algorithm is about 6 times faster than BIPOPaCMA-ES and about 1.8 times faster than BIPOP-s∗aACMES. However, the performance on multi-modal functions should
be further analyzed for a larger maximum number of function evaluations.
The proposed BIPOP-aCMA-STEP as expected shows
good results on separable multi-modal functions by exploitation when it is detected (when STEP performs better than
BIPOP-aCMA-ES). The proposed hybrid HCMA algorithm
demonstrates the best overall performance when running for
more than 100n function evaluations on 10-, 20- and 40dimensional problems, its overall performance for this budget of function evaluations and for these dimensions is better
than of all algorithms tested during the BBOB-2009, BBOB2010 and BBOB-2012.
It would be interesting to further develop HCMA and
BIPOP-aCMA-STEP and make these algorithms better suited
to partially-separable functions and/or adjust the algorithm
to work in a sampling space of CMA-ES defined by its covariance matrix. A natural perspective is to improve theresults of HCMA on multi-modal function by applying therecently proposed alternative restart strategies for CMA-ES[16].

6. REFERENCES[1] A. Auger and N. Hansen. A Restart CMA Evolution
Strategy With Increasing Population Size. In IEEE
Congress on Evolutionary Computation, pages
1769–1776. IEEE Press, 2005.
[2] Z. Bouzarkouna, A. Auger, and D. Y. Ding.
Investigating the Local-Meta-Model CMA-ES for
Large Population Sizes. In EvoApplications (1), pages
402–411, 2010.
[3] N. Hansen. Benchmarking a BI-population CMA-ES
on the BBOB-2009 function testbed. In GECCO
Companion, pages 2389–2396, 2009.
[4] N. Hansen, A. Auger, S. Finck, and R. Ros.
Real-Parameter Black-Box Optimization
Benchmarking 2012: Experimental Setup. Technicalreport, INRIA, 2012.
[5] N. Hansen, S. Müller, and P. Koumoutsakos. Reducing
the time complexity of the derandomized evolution
strategy with covariance matrix adaptation(CMA-ES). Evolutionary Computation, 11(1):1–18,
2003.
[6] N. Hansen and R. Ros. Benchmarking a weighted
negative covariance matrix update on the BBOB-2010
noiseless testbed. In GECCO ’10: Proceedings of the
12th annual conference comp on Genetic andevolutionary computation, pages 1673–1680, New
York, NY, USA, 2010. ACM.
[7] F. Hoffmann and S. Holemann. Controlled Model
Assisted Evolution Strategy with Adaptive
Preselection. In International Symposium on Evolving
Fuzzy Systems, pages 182–187. IEEE, 2006.
[8] H. Ingimundardottir and T. Runarsson. Sampling
Strategies in Ordinal Regression for Surrogate Assisted
Evolutionary Optimization. In 11th International
Conference on. Intelligent Systems Design andApplications (ISDA), pages 1158–1163, 2011.
[9] G. A. Jastrebski and D. V. Arnold. Improving
Evolution Strategies through Active CovarianceMatrix Adaptation. In IEEE Congress on
Evolutionary Computation, pages 2814–2821, 2006.
[10] Y. Jin. A Comprehensive Survey of Fitness
Approximation in Evolutionary Computation. Soft
Computing, 9(1):3–12, 2005.
[11] S. Kern, N. Hansen, and P. Koumoutsakos. Local
Meta-Models for Optimization Using Evolution
Strategies. In Th. Runarsson et al., editor, PPSN IX,
pages 939–948. LNCS 4193, Springer Verlag, 2006.
[12] O. Kramer. Covariance Matrix Self-Adaptation and
Kernel Regression - Perspectives of Evolutionary
Optimization in Kernel Machines. Fundam. Inf.,
98:87–106, 2010.
[13] I. Loshchilov, M. Schoenauer, and M. Sebag.
Comparison-Based Optimizers Need
Comparison-Based Surrogates. In R. S. et al., editor,
Parallel Problem Solving from Nature (PPSN XI),
volume 6238 of LNCS, pages 364–373. Springer,
September 2010.
[14] I. Loshchilov, M. Schoenauer, and M. Sebag.
Alternative Restart Strategies for CMA-ES. In V. C.
et al., editor, Parallel Problem Solving from Nature(PPSN XII), LNCS, pages 296–305. Springer, 2012.

Figure 1: Expected running time (ERT in number of f -evaluations) divided by dimension for target function
value 10−8 as log10 values versus dimension. Different symbols correspond to different algorithms given in
the legend of f1 and f24 . Light symbols give the maximum number of function evaluations from the longest
trial divided by dimension. Horizontal lines give linear scaling, slanted dotted lines give quadratic scaling.
Black stars indicate statistically better result compared to all other algorithms with p < 0.01 and Bonferroni
correction number of dimensions (six). Legend: ◦:BIPOP-aCMA, ▽:BIPOP-saACM, ⋆:BIPOP-saACM-k,
2:BIPOP-aCMA-STEP, △:HCMA.

separable fcts

moderate fcts
HCMA

BIPOP-s∗ aACM

BIPOP-aCMA-STEP

best 2009

best 2009

BIPOP-s∗ aACM-k

BIPOP-aCMA

HCMABIPOP-s∗ aACM-k

BIPOP-aCMA

BIPOP-s∗ aACM

BIPOP-aCMA-STEP

ill-conditioned fcts

multi-modal fcts
HCMA

BIPOP-s∗ aACM

BIPOP-s∗ aACM-k

BIPOP-s∗ aACM-k

BIPOP-s∗ aACM

best 2009

BIPOP-aCMA-STEPHCMA

BIPOP-aCMA

BIPOP-aCMA-STEP

best 2009

BIPOP-aCMA

weakly structured multi-modal fcts

all functionss∗

BIPOP-

aACM

best 2009

best 2009

BIPOP-aCMA-STEP

BIPOP-s∗ aACM-k

HCMABIPOP-aCMA-STEP

BIPOP-s∗ aACM

BIPOP-aCMA

BIPOP-s∗ aACM-k

HCMABIPOP-aCMA

Figure 2: Bootstrapped empirical cumulative distribution of the number of objective function evaluations
divided by dimension (FEvals/D) for 50 targets in 10[−8..2] for all functions and subgroups in 20-D. The “best2009” line corresponds to the best ERT observed during BBOB 2009 for each single target.

∆fopt 1e1
1e0
1e-1
1e-3
1e-5
1e-7
#succ
f1
43
43
43
43
43
43
15/15
BIPOP-a 7.5(2)
14(2)
20(2)
33(2)
46(4)
59(4)
15/15
BIPOP-s 4.0(0.2)
5.1(0.4)
6.6(0.7)10(0.7)13(0.8)16(1)15/15
BIPOP-s 3.7(0.1)
4.9(0.8)
6.2(0.5)
8.9(0.5)12(0.7)15(1)15/15
5.2(0)
7.3(0)10(0)13(0.5)16(0)15/15BIPOP-a 4.2(0.5)
HCMA 1.00(0.0) ⋆41.0(0.0) ⋆4 1.0(0.0) ⋆4 1.0(0.0)⋆4 1.0(0.0)⋆4 1.0(0.0)⋆4 15/15∆fopt 1e1
f13652
BIPOP-a 4.1(3)BIPOP-s 1.1(0.9)
BIPOP-s 1.1(0.7)
BIPOP-a 5.0(5)HCMA 1.3(0.1)

1e0
2021
3.3(3)
0.89(0.7)
0.74(0.5)
2.7(2)
0.90(0.5)

1e-1
2751
3.7(3)
1.4(1.0)0.85(0.4)
4.4(2)
0.99(0.7)

1e-3
1e-5
1e-7
#succ
18749
24455
30201
15/15
0.85(0.4)
1.1(0.8)
1.4(0.6)
15/15
0.38(0.1)↓4 0.42(0.2)↓4 0.40(0.1)↓4 15/15
0.17(0.1) ↓40.21(0.1) ↓40.22(0.1) ↓415/15
0.89(0.3)
1.1(0.6)
1.1(0.6)
15/15
0.25(0.1)↓4 0.27(0.1)↓4 0.24(0.1)↓4 15/15∆fopt 1e1
f2
385
BIPOP-a23(4)BIPOP-s 6.8(1)BIPOP-s 4.3(0.9)
BIPOP-a 1.3(0.1)
HCMA
1.3(0.1)∆fopt 1e1
f14
75
BIPOP-a 3.2(1)BIPOP-s 3.2(1)1e0239
2.6(0.5)
1.8(0.6)

1e-1304
3.3(0.4)
1.9(0.4)

1e-3932
3.2(0.3)
1.5(0.2)

1e-5
1648
3.8(0.2)
1.4(0.2)

BIPOP-s 3.1(0.7)

1e0
386
27(3)
8.0(1)
4.6(0.8)
1.5(0.1)
1.5(0.1)

1e-1
387
29(3)
8.9(1)
4.9(0.8)
1.7(0.1)
1.7(0.0)

1e-3
390
31(2)
10(1)
5.4(0.7)
2.1(0.2)
2.1(0.2)

1e-5
391
33(1)
10(1)
5.7(0.7)
2.4(0.3)
2.4(0.3)

1e-7
393
34(1)
10(1)
6.1(0.6)
2.8(0.2)
2.8(0.2)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
1e0
1e-1
1e-3
1e-5
1e-7
#succ
f3
5066
7626
7635
7643
7646
7651
15/15
BIPOP-a 9.0(5)1.7e4(2e4) ∞
∞
∞
∞ 2e7
0/15
BIPOP-s 10(7)
∞
∞
∞
∞
∞ 2e7
0/5
BIPOP-s 20(23)
∞
∞
∞
∞
∞ 2e7
0/15
BIPOP-a 0.26(0.0) ↓40.37(0.1) ↓40.46(0.1) ↓40.49(0.1)↓40.51(0.1)↓40.53(0.1)↓415/15
HCMA
0.26(0.0) ↓40.37(0.1)↓40.46(0.1) ↓40.49(0.1) ↓40.51(0.1) ↓40.53(0.1) ↓415/15
1e0
1e-1∆fopt 1e1
f4
4722
7628
7666
∞
BIPOP-a 3.0e4(3e4) ∞BIPOP-s ∞∞∞BIPOP-s 1.9e4(2e4) ∞∞BIPOP-a0.42(0.1) ↓40.67(0.1)↓40.90(0.1)
HCMA 0.42(0.1) ↓40.67(0.1) ↓40.90(0.1)

1e-3
7700
∞
∞
∞
1.5(0.2)
1.5(0.2)

1e-5
7758
∞
∞
∞
1.8(0.2)
1.8(0.2)

1e-7
#succ
1.4e5
9/15
∞ 2e7
0/15
∞ 2e7
0/5
∞ 2e7
0/15
0.11(1e-2) 15/15
0.11(1e-2) 15/15∆fopt 1e1
f5
41
BIPOP-a 5.5(0.9)
BIPOP-s 4.7(0.7)
BIPOP-s 4.5(0.9)
BIPOP-a1.0(0) ⋆4

1e0
41
6.6(0.8)
5.3(0.7)
5.2(0.6)
1.0(0) ⋆4

1e-1
41
6.7(0.9)
5.4(0.7)
5.3(0.8)
1.0(0)⋆41e-3
41
6.7(0.9)
5.4(0.7)
5.4(0.6)
1.0(0) ⋆4

1e-5
41
6.7(0.9)
5.4(0.7)
5.4(0.6)
1.0(0) ⋆4

1e-7
41
6.7(0.9)
5.4(0.7)
5.4(0.6)
1.0(0)⋆4

1.2(0.1)

1.4(0.3)

1.4(0.3)

1.4(0.3)

1.4(0.3)

1.4(0.3)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
f6
1296
BIPOP-a 1.5(0.2)
BIPOP-s 1.4(0.3)
BIPOP-s 1.6(0.5)
BIPOP-a 1.6(0.2)
HCMA 1.7(0.3)

1e0
2343
1.2(0.1)
1.2(0.2)
1.3(0.4)
1.3(0.1)
1.4(0.2)

1e-1
3413
1.1(0.1)
1.1(0.2)
1.2(0.3)
1.1(0.1)
1.3(0.2)

1e-35220
1.1(0.1)
1.1(0.2)
1.4(0.3)
1.1(0.1)
1.4(0.2)

1e-56728
1.1(0.1)
1.3(0.3)
1.5(0.3)
1.1(0.1)
1.6(0.3)

1e-78409
1.1(0.1)
1.4(0.3)
1.7(0.4)
1.1(0.1)
1.7(0.4)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

1e-1
9503
2.3(1)
0.84(0.3)
0.65(0.3)
2.5(1)
0.84(0.4)

1e-3
1e-5
1e-7
#succ
16524
16524
16969
15/15
1.5(0.6)
1.5(0.6)
1.4(0.6)
15/15
0.61(0.1) ↓3 0.61(0.1)↓3 0.60(0.1)↓3 15/15
0.56(0.2) 0.56(0.2) 0.55(0.2) 15/15
1.5(0.7)
1.5(0.7)
1.5(0.7)
15/15
0.57(0.2)
0.57(0.2)
0.57(0.2)
15/15

HCMA

1e0
∆fopt 1e1
f7
1351
4274
BIPOP-a 1.3(1)2.9(2)BIPOP-s 1.0(0.9)
1.6(0.6)
BIPOP-s 0.52(0.2) ↓21.2(0.9)
3.2(2)BIPOP-a 1.5(1)HCMA 0.98(0.9) 1.1(0.2)∆fopt 1e1
f8
2039
BIPOP-a 3.3(0.9)
BIPOP-s 1.3(0.2)
BIPOP-s 1.0(0.1) ⋆BIPOP-a 3.7(1.0)
HCMA 1.3(0.2)

1e0
1e-1
1e-3
1e-5
1e-7
#succ
3871
4040
4219
4371
4484
15/15
4.0(3)
4.3(3)
4.4(3)
4.4(3)
4.5(3)
15/15
1.5(0.9)
1.5(0.9)
1.6(0.8)
1.6(0.8)
1.6(0.8)
15/15
0.97(0.1) ⋆21.0(0.1)⋆2 1.1(0.1) ⋆2 1.1(0.1) ⋆2 1.1(0.1)⋆2 15/15
4.1(3)
4.4(3)
4.5(3)
4.5(3)
4.5(3)
15/15
1.1(0.1)
1.2(0.1)
1.2(0.1)
1.2(0.1)
1.2(0.1)
15/15∆fopt 1e1
f9
1716
BIPOP-a 3.8(1.0)
BIPOP-s 1.5(0.3)
BIPOP-s 1.3(0.2)
BIPOP-a 3.9(0.6)
HCMA 1.6(0.3)

1e0
3102
4.4(0.6)
1.7(0.2)
1.3(0.2) ⋆4.1(0.4)
1.5(0.3)

1e-13277
4.7(0.6)
1.7(0.2)
1.4(0.2)⋆
4.3(0.3)
1.6(0.3)

1e-33455
4.8(0.5)
1.8(0.2)
1.4(0.2) ⋆4.5(0.3)
1.6(0.3)

1e-53594
4.8(0.5)
1.8(0.2)
1.4(0.2) ⋆4.5(0.3)
1.6(0.3)

1e-7
3727
4.8(0.5)
1.7(0.2)
1.4(0.2)
4.5(0.3)
1.6(0.3)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
1e0
1e-1
1e-3
1e-5
1e-7
#succ
f10
7413
8661
10735
14920
17073
17476
15/15
1.2(0.2)
1.1(0.1)
0.83(0.0) ↓4 0.76(0.0)↓4 0.77(0.0)↓4 15/15
BIPOP-a 1.2(0.2)
BIPOP-s 0.36(0.1) ↓4 0.35(0.0)↓4 0.31(0.0)↓4 0.24(0.0) ↓4 0.23(0.0)↓4 0.23(0.0)↓4 15/15⋆⋆⋆
BIPOP-s 0.21(0.0) ⋆2↓40.20(0.0) ↓40.17(0.0) ↓40.13(0.0) ↓40.13(0.0) ↓40.13(0.0) ↓415/15
1.2(0.1)
1.0(0.1)
0.80(0.1) ↓4 0.74(0.0)↓4 0.76(0.0)↓4 15/15
BIPOP-a 1.2(0.2)
HCMA 0.25(0.0) ↓4 0.23(0.0)↓4 0.20(0.0)↓4 0.15(0.0) ↓4 0.14(0.0)↓4 0.15(0.0)↓4 15/15∆fopt 1e1
f11
1002
BIPOP-a 4.5(0.3)
BIPOP-s 2.5(0.4)
BIPOP-s 1.6(0.4) ⋆

1e0
1e-1
1e-3
1e-5
1e-7
#succ
2228
6278
9762
12285
14831
15/15
2.3(0.1)
0.87(0.0)
0.64(0.0) ↓4 0.56(0.0)↓4 0.51(0.0)↓4 15/15
1.2(0.2)
0.44(0.1)
0.30(0.0) ↓4 0.26(0.0)↓4 0.23(0.0)↓4 15/15
0.77(0.2) ⋆ 0.29(0.1) ↓30.20(0.0) ↓40.17(0.0) ⋆↓40.15(0.0) ↓415/15

BIPOP-a 4.8(0.4)
HCMA 1.7(0.2)

2.4(0.2)
0.81(0.1)

0.91(0.1)
0.66(0.0) ↓4 0.57(0.0)↓4 0.52(0.0)↓4 15/15
0.30(0.0)↓4 0.21(0.0) ↓4 0.18(0.0)↓4 0.16(0.0)↓4 15/15∆fopt 1e1
f12
1042
BIPOP-a 3.3(3)BIPOP-s 0.99(0.9)
BIPOP-s 0.83(0.1)
BIPOP-a 3.1(3)HCMA 1.7(1)

1e0
1938
3.6(3)
1.1(1)
0.93(0.9)
3.6(3)
1.7(1)

1e-1
2740
4.0(2)
1.2(0.9)
1.2(0.9)
3.8(3)
1.6(1)

1e-3
4140
3.7(1)
1.2(0.9)
1.1(0.6)
3.5(2)
1.4(0.7)

1e-5
1e-7
#succ
12407
13827
15/15
1.5(0.4)
1.5(0.4)
15/15
0.55(0.3)↓ 0.59(0.3)↓2 15/15
0.51(0.2) ↓20.53(0.2) ↓215/15
1.5(0.6)
1.5(0.6)
15/15
0.60(0.2)↓ 0.61(0.2)↓2 15/15

1.8(0.3)

2.1(0.5)

1.4(0.2)

1.3(0.1)

BIPOP-a 5.4(1)3.9(0.7)

3.3(0.4)

3.9(0.4)

HCMA

3.3(0.8)
1.3(0.3) ⋆3 2.7(1)2.9(0.3)

1.7(0.1)

1.5(0.1)

1e-3
3.2e5
1.2(0.6)
0.89(0.6)
0.97(0.5)
1.0(0.6)
0.90(0.7)

1e-7
#succ
15661
15/15
0.68(0.0)↓4 15/15
0.23(0.0)↓4 15/15
0.19(0.0) ⋆3↓415/150.68(0.0)↓4 15/15

0.21(0.0)↓4 15/15
1e-5
1e-7
#succ
4.5e5
4.6e5
15/15
0.89(0.5)
0.89(0.4)
15/15
0.66(0.5) 0.65(0.5) 15/15
0.71(0.3)↓2 0.73(0.3)↓2 15/15
0.76(0.4)↓ 0.76(0.4)↓ 15/15
0.66(0.5) ↓ 0.66(0.5)↓ 15/15∆fopt 1e1
f15
30378
BIPOP-a 0.88(0.4)
BIPOP-s 0.65(0.6)
BIPOP-s 0.62(0.4)
BIPOP-a 0.90(0.8)
HCMA 1.0(2)

1e0
1.5e5
1.6(0.6)
1.3(0.6)
1.8(0.4)
1.5(0.6)
1.5(0.8)

1e-1
3.1e5
1.2(0.6)
0.91(0.7)
0.96(0.5)
1.0(0.6)
0.90(0.7)∆fopt 1e1
f16
1384
BIPOP-a 1.7(0.6)
BIPOP-s 1.9(0.6)
BIPOP-s 1.3(0.3)
BIPOP-a 2.3(0.9)
HCMA 1.8(0.8)

1e0
27265
0.87(0.5)
0.74(0.4)
0.58(0.3)
0.67(0.6)
0.66(0.5)

1e-1
1e-3
1e-5
77015
1.9e5
2.0e5
0.78(0.5)
0.92(0.4)
1.2(0.8)
0.51(0.3)
0.60(0.5)
0.84(0.5)
0.46(0.2) ↓ 0.41(0.2) ↓20.61(0.5)
1.0(1.0)
1.1(1)
1.3(1)
0.52(0.2)↓ 0.49(0.3)
0.69(0.5)

1e-7
2.2e5
1.1(0.8)
0.83(0.5)
1.0(0.6)
1.2(0.9)
1.1(1)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
f17
63
BIPOP-a 3.0(1)BIPOP-s 3.2(2)BIPOP-s 2.8(1)BIPOP-a 4.2(2)HCMA 2.7(4)

1e0
1030
1.1(0.3)
1.2(0.4)
3.3(4)
2.0(0.3)
2.8(1)

1e-1
4005
1.7(2)
2.7(3)
3.8(3)
1.00(2)
3.1(3)

1e-3
30677
1.0(0.4)
1.2(0.7)
1.5(0.6)
0.94(0.3)
1.6(0.9)

1e-5
56288
0.90(0.5)
1.2(0.5)
1.7(0.5)
1.0(0.5)
1.8(0.7)

1e-7
80472
1.1(0.4)
1.4(0.8)
1.6(0.5)
1.1(0.4)
1.8(0.8)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
f18
621
BIPOP-a0.94(0.2)
BIPOP-s 1.0(0.3)
BIPOP-s 1.1(0.5)
BIPOP-a 1.2(0.2)
HCMA 1.4(0.4)

1e0
3972
0.77(0.1)
1.5(1)
3.2(3)
1.0(2)
2.7(3)

1e-1
19561
1.5(0.7)
0.92(0.4)
1.9(0.6)
1.2(0.7)
1.3(0.5)

1e-3
67569
1.4(0.5)
0.96(0.4)
1.4(0.8)
1.2(0.6)
1.1(0.6)

1e-5
1.3e5
1.2(0.7)
1.6(0.6)
1.4(0.7)
1.2(0.6)
1.1(0.6)

1e-7
1.5e5
1.5(0.7)
1.6(0.5)
1.6(0.8)
1.3(0.6)
1.2(0.6)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

1e-3
1e-5
6.2e6
6.7e6
1.0(0.4)
1.0(0.4)
0.72(0.4) ↓ 0.73(0.4)
0.76(0.2) 0.77(0.3)
0.86(0.3) 0.94(0.3)
0.95(0.4) 0.96(0.3)

1e-7
6.7e6
1.0(0.4)
0.73(0.4)
0.77(0.3)
0.94(0.3)
0.96(0.3)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
f19
1
BIPOP-a184(66)
BIPOP-s 143(52)
BIPOP-s 148(44)
BIPOP-a259(90)49(6)⋆3
HCMA

1e0
1e-1
1
3.4e5
2.9e4(2e4) 1.4(1)2.5e4(1e4)0.42(0.3)
3.3e4(3e4) 0.72(0.8)
2.7e4(2e4) 1.2(1)3.5e4(3e4) 0.72(0.7)∆fopt 1e1
1e0
f20
82
46150
BIPOP-a 4.9(1)4.8(2)BIPOP-s 2.9(0.5)
2.1(1)BIPOP-s 2.8(0.5)
2.4(2)BIPOP-a 6.3(1)4.7(2)HCMA 0.77(0.4) ⋆41.9(1)

1e-1
3.1e6
1.4(0.6)
0.97(0.7)
1.1(0.6)
1.0(0.5)
1.2(0.6)

1e-3
5.5e6
0.95(0.4)
0.87(0.4)
0.93(0.3)
0.95(0.3)
0.88(0.3)

1e-5
5.6e6
0.95(0.3)
0.86(0.4)
0.93(0.3)
0.95(0.3)
0.88(0.3)

1e-7
5.6e6
0.95(0.3)
0.85(0.4)
0.94(0.3)
0.95(0.3)
0.88(0.3)

#succ
14/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
f21
561
BIPOP-a 7.8(12)BIPOP-s 2.6(4)BIPOP-s 2.1(2)BIPOP-a 4.8(5)HCMA 0.76(2)

1e0
6541
110(57)1.5(1)
3.7(5)
80(95)
3.5(5)

1e-1
14103
72(89)
6.0(11)20(6)53(109)49(50)

1e-3
14643
69(86)
5.8(11)19(6)51(105)47(48)

1e-5
15567
65(81)
5.5(10)19(6)48(99)
44(45)

1e-7
17589
58(71)
4.8(9)
17(5)43(88)
39(40)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1
f22
467
BIPOP-a37(59)
BIPOP-s 7.7(9)BIPOP-s 7.6(10)BIPOP-a63(74)HCMA 11(16)

1e0
5580
260(240)100(96)221(549)
220(507)135(194)

1e-1
23491
273(450)
178(320)311(450)288(396)486(638)

1e-3
24948
257(401)173(301)
293(436)271(373)458(479)

1e-5
26847
239(394)
168(274)273(399)
252(347)425(558)

1e-7
1.3e5
48(75)
35(54)
54(74)
50(69)
85(114)

#succ
12/15
13/15
15/15
13/15
14/15
11/15

∆fopt 1e1
f23
3.2
BIPOP-a 4.7(9)BIPOP-s 3.0(6)BIPOP-s 3.2(4)BIPOP-a 2.3(4)HCMA 12(10)

1e0
1614
43(44)
21(13)
29(36)
23(26)
23(15)

1e-3
4.9e5
1.4(1)
1.4(1)
2.7(5)
2.3(2)
1.8(2)

1e-5
8.1e5
0.85(0.7)
1.3(1)
1.8(3)
1.4(1)
1.1(1)

1e-7
8.4e5
0.90(0.6)
1.3(1)
1.9(3)
1.5(1)
1.1(1)

#succ
15/15
15/15
15/15
15/15
15/15
15/15

∆fopt 1e1f24
1.3e6
BIPOP-a 1.00(1)BIPOP-s 0.99(1)
BIPOP-s 1.3(1)BIPOP-a 1.7(2)HCMA 0.88(1)

1e0
7.5e6
0.94(0.9)
0.88(0.7)
0.93(1)0.90(0.9)
1.2(1)

1e-3
5.2e7
2.8(3)
0.80(0.8)
1.2(1)
2.7(3)
5.7(6)

1e-5
5.2e7
2.8(3)
0.80(1.0)
1.2(1)
2.7(3)
5.7(6)

1e-7
5.2e7
2.8(3)
0.79(0.8)
1.2(1)
2.7(3)
5.7(6)

#succ
3/15
2/15
6/15
4/15
2/15
1/15

1e-1
67457
1.2(1)
0.61(0.3)
0.74(0.8)
0.69(0.6)
0.96(0.7)
1e-1
5.2e7
2.8(3)
0.80(0.8)
1.2(1)
2.7(3)
5.7(6)

Table 1: Expected running time (ERT in number of function evaluations) divided by the respective best ERT
measured during BBOB-2009 (given in the respective first row) for different ∆f values in dimension 20. The
central 80% range divided by two is given in braces. The median number of conducted function evaluations
is additionally given in italics, if ERT(10−7 ) = ∞. #succ is the number of trials that reached the final target
fopt + 10−8 . Best results are printed in bold.

[15] I. Loshchilov, M. Schoenauer, and M. Sebag.
Black-box Optimization Benchmarking of
IPOP-saACM-ES and BIPOP-saACM-ES on the
BBOB-2012 Noiseless Testbed. In T. Soule and J. H.
Moore, editors, Genetic and Evolutionary
Computation Conference (GECCO Companion), pages
175–182. ACM Press, July 2012.
[16] I. Loshchilov, M. Schoenauer, and M. Sebag.
Black-box Optimization Benchmarking of
NIPOP-aCMA-ES and NBIPOP-aCMA-ES on the
BBOB-2012 Noiseless Testbed. In T. Soule and J. H.
Moore, editors, Genetic and Evolutionary
Computation Conference (GECCO Companion), pages
269–276. ACM Press, July 2012.
[17] I. Loshchilov, M. Schoenauer, and M. Sebag.
Self-Adaptive Surrogate-Assisted Covariance Matrix
Adaptation Evolution Strategy. In T. Soule and J. H.
Moore, editors, Genetic and Evolutionary
Computation Conference (GECCO), pages 321–328.
ACM Press, July 2012.
[18] I. Loshchilov, M. Schoenauer, and M. Sebag. Intensive
Surrogate Model Exploitation in Self-adaptive
Surrogate-assisted CMA-ES (saACM-ES). In C. Blumand E. Alba, editors, Genetic and Evolutionary
Computation Conference (GECCO), page to appear.
ACM Press, July 2013.
[19] P. Pošı́k and W. Huyer. Restarted local searchalgorithms for continuous black box optimization.
Evolutionary Computation, 20(4):575–607, 2012.
[20] M. Powell. The NEWUOA software for unconstrained
optimization without derivatives. In G. Pillo,
M. Roma, and P. Pardalos, editors, Large-Scale
Nonlinear Optimization, volume 83 of Nonconvex
Optimization and Its Applications, pages 255–297.
Springer US, 2006.
[21] M. Preuss. Niching the cma-es via nearest-better
clustering. In Proceedings of the 12th annual
conference companion on Genetic and evolutionary
computation, pages 1711–1718. ACM, 2010.
[22] R. Ros. Benchmarking the newuoa on the bbob-2009function testbed. In Proceedings of the 11th Annual
Conference Companion on Genetic and Evolutionary
Computation Conference: Late Breaking Papers, pages
2421–2428. ACM, 2009.
[23] D. F. Shanno. Conditioning of Quasi-Newton Methods
for Function Minimization. Mathematics ofComputation, 24(111):647–656, 1970.
[24] S. Swarzberg, G. Seront, and H. Bersini. Step: The
easiest way to optimize a function. In Evolutionary
Computation, 1994. IEEE World Congress on
Computational Intelligence., Proceedings of the FirstIEEE Conference on, pages 519–524. IEEE, 1994.
[25] H. Ulmer, F. Streichert, and A. Zell. Evolution
Strategies assisted by Gaussian Processes with
Improved Pre-Selection Criterion. In IEEE Congress
on Evolutionary Computation, pages 692–699, 2003.