Comparing Black-Box Differential Evolution andClassic Differential Evolution:
Revisited Investigation of Invariance to Rotation
Technical Report IJS-DP 12585
Delovno poročilo IJS-DP 12585

1

Aljoša Vodopija1,2 , Tea Tušar1 , Bogdan Filipič1,2
Department of Intelligent Systems, Jožef Stefan InstituteJamova cesta 39, SI-1000 Ljubljana, Slovenia
2
Jožef Stefan International Postgraduate School
Jamova cesta 39, SI-1000 Ljubljana, Slovenia{aljosa.vodopija, tea.tusar, bogdan.filipic}@ijs.si
September 2018

Abstract
We revisit our previous work on comparing black-box differential evolution (BBDE) andclassic differential evolution, where we observed an unexpected residual sensitivity to rotation of BBDE. A detailed analysis of the results shows that it stems from a constraint
handling technique that favours separability. Equipped with these findings, we re-designed
our experiments by using only unbounded decision spaces. The new results show that for
unbounded decision spaces BBDE is indeed invariant to rotation. In addition, we corrected
the pseudocode of the ‘target-to-best’ variants, since we found out that they were written
incorrectly in the original paper.

Contents
1 Introduction

3

2 Observations on the original paper
2.1 Invariance to rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 The ‘target-to-best’ strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

33
4

3 Revisited experimental setup and results
3.1 Invariance to rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4444 Conclusions

5

References91

Introduction

This report presents a revisited study of our previous work [1], where we compared black-box
differential evolution (BBDE) [2] with classic differential evolution (DE) [3] using the COCO
platform [4]. Reading the original paper is a precondition for understanding the following text.
In the original work, the experiments on separable and rotated variants of the ellipsoid problem (problems f2 and f10 of the bbob test suite [5]) have demonstrated that while the BBDE
variants are less sensitive to rotation than the DE variants, some sensitivity to this transformation still persists. The reason for this sensitivity remained unexplained. In addition, we realizedthat in the original paper the pseudocode describing DE/target-to-best and BBDE/target-tobest variants were incorrect.
In this study, we re-investigate the invariance to rotation of the BBDE variants and correct
the pseudocode representing DE/targe-to-best and BBDE/target-to-best. By examining theresults obtained in the previous work and experimenting with different settings we discover thatthe sensitivity to rotation was a consequence of using bounded decision spaces and a constraint
handling technique favoring separability.
The rest of this report is organized as follows. Section 2 provides two observations on theoriginal paper, while Section 3 presents the revisited experiments and their results. Section 4
concludes the report.

2

Observations on the original paper

2.1

Invariance to rotation

In the original experiments, decision spaces were bounded with box constraints. A feasible
solution p = (x1 , . . . xD )T satisfied the following constraints: −5 ≤ xi ≤ 5 for all i ∈ {1, . . . , D}.
In order to satisfy these constraints, a constraint handling technique, replacing any infeasible
component of the solution with a randomly selected feasible value, was used.
Figure 1 shows an example describing how an infeasible solution (red point) is replaced with afeasible one (black point). In this two-dimensional example the infeasible solution violates the x1 constraint only. The constraint handling technique replaces x11 with a randomly selected feasible
value from [−5, 5] keeping the value of x12 intact. This specific constraint handling techniqueimplicitly imposes an exploration bias toward searching in directions aligned with coordinate
system axes. In other words, an optimizer using a similar constraint handling technique would
show superior performance on separable problems.
x2

x12 = x22

x21

x11

x1

Figure 1: The constraint handling technique replacing the infeasible solution (red point) with a feasible
one (black point). The direction of replacing the infeasible solutions is aligned with the x1 -axis.

3Figure 2 shows the results of the original experiments obtained for the two ellipsoid problems
in 20 dimensions. The BBDE performance is slightly better on the separable problem. Examining the results in more detail we find that about 26% of generated solutions for the ellipsoid
problem have at least one infeasible value. In other words, in 26% of the cases the constraint
handling technique imposes a bias towards searching in separable directions.
51 targets: 100..1e-08
15 instances

best 2009

Fraction of function,target pairs

Fraction of function,target pairs

2 Ellipsoid separable

1.0 bbob f2, 20-D
0.8

DE-rand0.6
0.4

BBDE-N

0.2
0.0

BBDE

v2.2.1

0

2

4

6

8

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8

BBDE-N

0.6
0.4

BBDE

0.2
0.0

DE-randv2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)(a) Separable ellipsoid,
f2 , dimension 20(b) Rotated ellipsoid,
f10 , dimension 20

8

Figure 2: ECDFs of simulated (bootstrapped) runtimes of the ‘rand’ strategy for separable (left) androtated (right) ellipsoid and dimension 20. The yellow line indicates DE/rand, the purple line BBDE-N,
and the blue line BBDE.

2.2

The ‘target-to-best’ strategies

The general form of the DE used in our work is shown in Algorithm 1. In the original paper, thepseudocode of the DE/target-to-best and BBDE/target-to-best variants was incomplete. The
paper incorrectly stated that the trial vector is created as
ptrial = p + F · (pbest − p)in DE/target-to-best and as
ptrial = p + F · (pbest − p2 )in BBDE/target-to-best. Is this report, we present the correct pseudocode in Algorithms 2 and3, respectively.

3

Revisited experimental setup and results

3.1

Invariance to rotation

In this revisited study, decision spaces are unbounded in all the experiments. Thus, any solution
generated by any algorithm is feasible, and therefore no constraint handling technique needs
to be used. However, the initial population is generated in such a way that its solutions lie in[−5, 5]D . All other settings are equal to those from the original experiments.

3.2

Results

The overall performance of the DE and BBDE variants on bounded and unbounded decision
spaces is shown in Figures 3, 5, and 7. We can observe that the results of the revisited experiments are almost identical to those in the original experiments. For this reason, no additionalcomments are provided here (the interested reader is referred to the original paper [1]).
4Algorithm 1 DE
Input: population size and stopping criterion;
Output: population P of solutions;
1: create the initial population P of random solutions;
2: evaluate the solutions in P;
3: while stopping criterion not met do
4:
Pnew ← ∅;5:
for all p ∈ P do
6:create a trial vector ptrial ;7:
ensure feasibility of ptrial ;8:evaluate ptrial ;9:
if ptrial is better than p then10:p ← ptrial ;11:
end if
12:
Pnew ← Pnew ∪ {p};13:
end for
14:
P ← Pnew ;15: end while
16: return P;
Algorithm 2 Trial vector creation by DE/target-to-best
Input: population P and target vector p ∈ P;
Output: trial vector ptrial ;1: randomly select two different solutions p1 , p2 ∈ P;
2: create a trial vector ptrial = p + F · (pbest − p) + F · (p1 − p2 );3: alter ptrial by crossover with p;
4: return ptrial ;Algorithm 3 Trial vector creation by BBDE/target-to-best
Input: population P and target vector p ∈ P;
Output: trial vector ptrial ;1: randomly select two different solutions p1 , p2 ∈ P;
2: sample a scaling factor F from XF ∼ exp(N (0, 1));
3: create a trial vector ptrial = p + F · (pbest − p) + F · (p1 − p2 );4: return ptrial ;In contrast, from Figures 4, 6, and 8 it is evident that while the DE variants are highly
sensitive to rotation, the performance of the BBDE variants is equal on separable and rotatedellipsoid problems when the decision space is unbounded. We can conclude that the observed
sensitivity to rotation in our original experiments was a consequence of using a constraint handling technique favoring separability.

4

Conclusions

In this study, we re-investigated the rotational invariance of BBDE. We found that the rotational
sensitivity observed on the ellipsoid problem is due to a constraint handling technique favoring
separability. By re-designing our experiments so that unbounded decision spaces are used insteadof bounded ones, we showed that for unbounded decision spaces the performance of BBDE is5

uDE-rand

0.6
0.4

uBBDE

0.2

uBBDE-Nv2.2.1

0

2

4

6

0.8

uDE-rand

0.6
0.4

uBBDE-N

0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

uBBDE

v2.2.10

2

4

6

1.0 bbob f1-f24, 5-D
0.8

uDE-rand

0.6
0.4

uBBDE

0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

uBBDE-Nv2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(a) Dimension 2(b) Dimension 3(c) Dimension 5

best 2009

51 targets: 100..1e-08
15 instances

0.8

uBBDE-N

0.6
0.4

uDE-rand

0.2

uBBDE

v2.2.10

2

4

6

1.0 bbob f1-f24, 20-D
0.8

uBBDE-N

0.6
0.4

uBBDE

0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

Fraction of function,target pairs

1.0 bbob f1-f24, 10-D

0.0

1.0 bbob f1-f24, 3-D

Fraction of function,target pairs

Fraction of function,target pairs

0.8

0.0

Fraction of function,target pairs

best 2009

51 targets: 100..1e-08
15 instances

Fraction of function,target pairs

Fraction of function,target pairs

1.0 bbob f1-f24, 2-D

uDE-randv2.2.10

2

4

6

1.0 bbob f1-f24, 40-D

best 2009

51 targets: 100..1e-08
15 instances

0.8

uBBDE-N

0.6
0.4

uBBDE

0.2
0.0

8

8

uDE-randv2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(d) Dimension 10(e) Dimension 20(f) Dimension 40

8

Figure 3: ECDFs of simulated (bootstrapped) runtimes of the ‘rand’ strategy for different problem
dimensions (see [1] for more details), where the yellow line indicates the unbounded DE/rand, the purple
line the unbounded BBDE-N, and the blue line the unbounded BBDE.

DE-rand0.6
0.4

BBDE-N

0.2
0.0

BBDE

v2.2.10

2

4

6

2 Ellipsoid separable

8

1.0 bbob f2, 10-D

51 targets: 100..1e-08
15 instances

2 Ellipsoid separable

best 2009

0.8

uDE-rand

0.6
0.4

uBBDE-N0.2
0.0

log10(# f-evals / dimension)

uBBDE

v2.2.10(a) Boundeddecision space

2

4

6

8

1.0 bbob f2, 20-D

51 targets: 100..1e-08
15 instances

DE-rand0.6
0.4

BBDE-N

0.2
0.0

BBDE-N

0.6
0.4

BBDE

0.2
0.0

DE-randv2.2.10246log10(# f-evals / dimension)(e) Boundeddecision space

86(c) Boundeddecision space

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

uBBDE-N

0.6
0.4

uBBDE

0.2

uDE-randv2.2.12

4

6

2 Ellipsoid separable

8

1.0 bbob f2, 20-D

51 targets: 100..1e-08
15 instances

best 2009

0.8

uDE-rand

0.6
0.4

uBBDE-N

0.2
0.0

uBBDE

v2.2.1

0

2

4

6

8

log10(# f-evals / dimension)(d) Unboundeddecision space

Rotated ellipsoid, f10 , dimension 20

0.8

0

4

(b) Unboundeddecision space

1.0 bbob f10, 10-D

0.0

2log10(# f-evals / dimension)

Fraction of function,target pairs

best 2009

Fraction of function,target pairs

Fraction of function,target pairs

10 Ellipsoid

51 targets: 100..1e-08
15 instances

0.8

BBDE

v2.2.10log10(# f-evals / dimension)

Rotated ellipsoid, f10 , dimension 10
1.0 bbob f10, 10-D

best 2009

0.8

8

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8

BBDE-N

0.6
0.4

BBDE

0.2
0.0

DE-randv2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)(f) Unboundeddecision space(g) Boundeddecision space

8

Fraction of function,target pairs

best 2009

Fraction of function,target pairs

51 targets: 100..1e-08
15 instances

0.8

Fraction of function,target pairs

Fraction of function,target pairs

2 Ellipsoid separable

1.0 bbob f2, 10-D

Separable ellipsoid, f2 , dimension 20
Fraction of function,target pairs

Separable ellipsoid, f2 , dimension 10

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8

uBBDE-N

0.6
0.4

uBBDE

0.2
0.0

uDE-randv2.2.1

0

2

4

6

8

log10(# f-evals / dimension)(h) Unboundeddecision space

Figure 4: ECDFs of simulated (bootstrapped) runtimes of the ‘rand’ strategy for the separable (topline) and rotated (bottom line) ellipsoid in dimensions 10 (first two columns) and 20 (last two columns).
The yellow line indicates DE/rand, the purple line BBDE-N, and the blue line BBDE (see [1] for moredetails). Plots (a), (c), (e) and (g) show the performance of the algorithms on the bounded decision
space, while the remaining plots (b), (d), (f) and (h) on the unbounded decision space.

6

0.6

uDE-best
0.4
0.2

uBBDE-bes

v2.2.1

0

2

4

6

0.8
0.6

uBBDE-bes
0.4
0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

uDE-best

v2.2.10

2

4

6

1.0 bbob f1-f24, 5-D
0.8
0.6

uDE-best
0.4
0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

uBBDE-bes

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(a) Dimension 2(b) Dimension 3(c) Dimension 5

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uDE-best
0.4
0.2

uBBDE-bes

v2.2.10

2

4

6

1.0 bbob f1-f24, 20-D
0.8
0.6

uDE-best
0.4
0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

Fraction of function,target pairs

1.0 bbob f1-f24, 10-D

0.0

1.0 bbob f1-f24, 3-D

Fraction of function,target pairs

Fraction of function,target pairs

0.8

0.0

Fraction of function,target pairs

best 2009

51 targets: 100..1e-08
15 instances

Fraction of function,target pairs

Fraction of function,target pairs

1.0 bbob f1-f24, 2-D

uBBDE-bes

v2.2.10

2

4

6

1.0 bbob f1-f24, 40-D

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uDE-best
0.4
0.2
0.0

8

8

uBBDE-bes

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(d) Dimension 10(e) Dimension 20(f) Dimension 40

8

Figure 5: ECDFs of simulated (bootstrapped) runtimes of the ‘best’ strategy for different problem
dimensions (see [1] for more details), where the purple line indicates the unbounded DE/best, and theblue line the unbounded BBDE/best.

best 20090.6

DE-best0.4
0.2

BBDE-best

v2.2.1

0

2

4

6

8

51 targets: 100..1e-08
15 instances

2 Ellipsoid separable

best 2009

0.8
0.6

uDE-best
0.4
0.2
0.0

uBBDE-bes

v2.2.1

0

2

4

6

8

1.0 bbob f2, 20-D

51 targets: 100..1e-08
15 instances

0.6

DE-best
0.4
0.2
0.0

0

4

6

8

51 targets: 100..1e-08
15 instances

0.6

uDE-best
0.4
0.2
0.0

uBBDE-bes

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)(a) Boundeddecision space(b) Unboundeddecision space(c) Boundeddecision space(d) Unboundeddecision space

best 20090.6

DE-best0.4
0.2

BBDE-best

v2.2.12

4

6

8

1.0 bbob f10, 10-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uDE-best
0.4
0.2
0.0

uBBDE-bes

v2.2.1

0

2

4

6

8

Rotated ellipsoid, f10 , dimension 20
Fraction of function,target pairs

10 Ellipsoid

51 targets: 100..1e-08
15 instances

0.8

best 20090.8

log10(# f-evals / dimension)

Fraction of function,target pairs

Fraction of function,target pairs

2

2 Ellipsoid separable

1.0 bbob f2, 20-D

log10(# f-evals / dimension)

1.0 bbob f10, 10-D

0

BBDE-best

v2.2.1

Rotated ellipsoid, f10 , dimension 10

0.0

best 2009

0.8

8

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

DE-best
0.4
0.2
0.0

BBDE-best

v2.2.1

0

2

4

6

8

Fraction of function,target pairs

0.0

2 Ellipsoid separable

1.0 bbob f2, 10-D

Fraction of function,target pairs

51 targets: 100..1e-08
15 instances

0.8

Fraction of function,target pairs

Fraction of function,target pairs

2 Ellipsoid separable

1.0 bbob f2, 10-D

Separable ellipsoid, f2 , dimension 20
Fraction of function,target pairs

Separable ellipsoid, f2 , dimension 10

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uDE-best
0.4
0.2
0.0

uBBDE-bes

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(e) Boundeddecision space(f) Unboundeddecision space(g) Boundeddecision space(h) Unboundeddecision space

8

Figure 6: ECDFs of simulated (bootstrapped) runtimes of the ‘best’ strategy for the separable (topline) and rotated (bottom line) ellipsoid in dimensions 10 (first two columns) and 20 (last two columns).
The yellow line indicates DE/rand, the purple line BBDE-N, and the blue line BBDE (see [1] for moredetails). Plots (a), (c), (e) and (g) show the performance of the algorithms on the bounded decision
space, while the remaining plots (b), (d), (f) and (h) on the unbounded decision space.

7

0.6

uDE-ttb
0.4
0.2

uBBDE-ttb

v2.2.1

0

2

4

6

0.8
0.6

uBBDE-ttb
0.4
0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

uDE-ttb

v2.2.10

2

4

6

1.0 bbob f1-f24, 5-D
0.8
0.6

uBBDE-ttb
0.4
0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

uDE-ttb

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(a) Dimension 2(b) Dimension 3(c) Dimension 5

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uDE-ttb
0.4
0.2

uBBDE-ttb

v2.2.10

2

4

6

1.0 bbob f1-f24, 20-D
0.8
0.6

uBBDE-ttb
0.4
0.2
0.0

8

best 2009

51 targets: 100..1e-08
15 instances

Fraction of function,target pairs

1.0 bbob f1-f24, 10-D

0.0

1.0 bbob f1-f24, 3-D

Fraction of function,target pairs

Fraction of function,target pairs

0.8

0.0

Fraction of function,target pairs

best 2009

51 targets: 100..1e-08
15 instances

Fraction of function,target pairs

Fraction of function,target pairs

1.0 bbob f1-f24, 2-D

uDE-ttb

v2.2.10

2

4

6

1.0 bbob f1-f24, 40-D

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uDE-ttb
0.4
0.2
0.0

8

8

uBBDE-ttb

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)

log10(# f-evals / dimension)(d) Dimension 10(e) Dimension 20(f) Dimension 40

8

Figure 7: ECDFs of simulated (bootstrapped) runtimes of the ‘target-to-best’ strategy for different
problem dimensions (see [1] for more details), where the purple line indicates the unbounded DE/targetto-best, and the blue line the unbounded BBDE/target-to-best.

0.6

DE-ttb0.4
0.2
0.0

BBDE-ttb

v2.2.10

2

4

6

2 Ellipsoid separable

8

1.0 bbob f2, 10-D

51 targets: 100..1e-08
15 instances

2 Ellipsoid separable

best 2009

0.8
0.6

uDE-ttb
0.4
0.2
0.0

log10(# f-evals / dimension)uBBDE-ttb

v2.2.10(a) Boundeddecision space

2

4

6

8

1.0 bbob f2, 20-D

51 targets: 100..1e-08
15 instances

0.6

DE-ttb
0.4
0.2
0.0

0

0.6

BBDE-ttb
0.4
0.2
0.0

DE-ttb

v2.2.10246log10(# f-evals / dimension)(e) Boundeddecision space

86(b) Unboundeddecision space(c) Boundeddecision space

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.6

uBBDE-ttb
0.4
0.2

uDE-ttb

v2.2.12

4

6

2 Ellipsoid separable

8

1.0 bbob f2, 20-D

51 targets: 100..1e-08
15 instances

best 2009

0.8
0.6

uDE-ttb
0.4
0.2
0.0

uBBDE-ttb

v2.2.1

0

2

4

6

8

log10(# f-evals / dimension)(d) Unboundeddecision space

Rotated ellipsoid, f10 , dimension 20

0.8

0

4

log10(# f-evals / dimension)

1.0 bbob f10, 10-D

0.0

2log10(# f-evals / dimension)

Fraction of function,target pairs

best 2009

Fraction of function,target pairs

Fraction of function,target pairs

10 Ellipsoid

51 targets: 100..1e-08
15 instances

0.8

BBDE-ttb

v2.2.1

Rotated ellipsoid, f10 , dimension 10
1.0 bbob f10, 10-D

best 2009

0.8

8

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

BBDE-ttb
0.4
0.2
0.0

DE-ttb

v2.2.10246log10(# f-evals / dimension)

log10(# f-evals / dimension)(f) Unboundeddecision space(g) Boundeddecision space

8

Fraction of function,target pairs

best 2009

Fraction of function,target pairs

51 targets: 100..1e-08
15 instances

0.8

Fraction of function,target pairs

Fraction of function,target pairs

2 Ellipsoid separable

1.0 bbob f2, 10-D

Separable ellipsoid, f2 , dimension 20
Fraction of function,target pairs

Separable ellipsoid, f2 , dimension 10

1.0 bbob f10, 20-D

10 Ellipsoid

best 2009

51 targets: 100..1e-08
15 instances

0.8
0.6

uBBDE-ttb
0.4
0.2
0.0

uDE-ttb

v2.2.1

0

2

4

6

8

log10(# f-evals / dimension)(h) Unboundeddecision space

Figure 8: ECDFs of simulated (bootstrapped) runtimes of the ‘target-to-best’ strategy for the separable(top line) and rotated (bottom line) ellipsoid in dimensions 10 (first two columns) and 20 (last twocolumns). The yellow line indicates DE/rand, the purple line BBDE-N, and the blue line BBDE (see[1] for more details). Plots (a), (c), (e) and (g) show the performance of the algorithms on the boundeddecision space, while the remaining plots (b), (d), (f) and (h) on the unbounded decision space.

8

indeed invariant to rotation on the ellipsoid problem. In addition, we corrected the pseudocode
describing ‘target-to-best’ variants, since they were written incorrectly in the original work.

Acknowledgment
We acknowledge financial support from the Slovenian Research Agency (young researcher program, project no. Z2-8177, and research core funding no. P2-0209). This work is also partof a project that has received funding from the European Union’s Horizon 2020 research andinnovation program under grant agreement no. 692286.

References[1] A. Vodopija, T. Tušar, and B. Filipič, “Comparing black-box differential evolution and classic
differential evolution,” in GECCO ’18 Companion: Genetic and Evolutionary Computation
Conference Companion. ACM, 2018, to appear.
[2] K. V. Price, “How symmetry constrains evolutionary optimizers,” in Proceedings of the IEEE
Congress on Evolutionary Computation (CEC 2017). IEEE, 2017, pp. 1712–1719.
[3] K. V. Price, R. M. Storn, and J. A. Lampinen, Differential Evolution.
139–156.

Springer, 2005, pp.

[4] N. Hansen, A. Auger, O. Mersmann, T. Tušar, and D. Brockhoff, “COCO: A platform for comparing continuous optimizers in a black-box setting,” ArXiv e-prints, vol.
arXiv:1603.08785, 2016.
[5] S. Finck, N. Hansen, R. Ros, and A. Auger, “Real-parameter black-box optimization benchmarking 2010: Noiseless functions definitions,” INRIA, Saclay, France, Tech. Rep. RR-6829,
2014.

9