Benchmarking MOEAs for Multi- and Many-objective
Optimization Using an Unbounded External Archive
Ryoji Tanabe

Akira Oyama

Japan Aerospace Exploration Agency (JAXA)
Sagamihara City, Japanrt.ryoji.tanabe@gmail.com

Japan Aerospace Exploration Agency (JAXA)
Sagamihara City, Japanoyama@flab.isas.jaxa.jp

ABSTRACT
While a large number of multi-objective evolutionary algorithms(MOEAs) for many-objective optimization problems (MaOPs) havebeen proposed in the past few years, an exhaustive benchmarking
study has never been performed. Moreover, most previous studies evaluated the performance of MOEAs based on nondominatedsolutions in the final population at the end of the search. In thispaper, we exhaustively investigate the convergence performanceof 21 MOEAs using an unbounded external archive that stores
all nondominated solutions found during the search process. Surprisingly, the experimental results for the WFG functions with up
to six objectives indicate that several recently proposed MOEAs
perform significantly worse than classical MOEAs. Moreover, theperformance rank among the 21 MOEAs significantly depends on
the number of function evaluations. Thus, the previously reported
performance of MOEAs on MaOPs as well as the widely used benchmarking methodology must be carefully reconsidered.

CCS CONCEPTS•Mathematics of computing → Evolutionary algorithms;

KEYWORDS
Benchmarking, Evolutionary Multi-objective Optimization
ACM Reference format:
Ryoji Tanabe and Akira Oyama. 2016. Benchmarking MOEAs for Multiand Many-objective Optimization Using an Unbounded External Archive.
In Proceedings of the Genetic and Evolutionary Computation Conference 2017,
Berlin, Germany, July 15–19, 2017 (GECCO ’17), 8 pages.
DOI: 10.1145/nnnnnnn.nnnnnnn

1

INTRODUCTION

An unconstrained (bound-constrained) multi-objective continuous
optimization problem (MOP) can be formulated as follows:
minimize f (x ) = ( f 1 (x ), ..., f M (x )) T
subject to x ∈ S ⊆ RD

(1)

where, f : S → RM is an objective function vector that consistsof M potentially conflicting objective functions, and RM is theobjective function space. Here, x = (x 1 , ...,x D ) T is a D-dimensional
solution vector, and S = Π D
j=1 [a j ,b j ] is the bound-constrained
search space, where a j ≤ x j ≤ b j for each index j ∈ {1, ...,D}.
We say that x 1 dominates x 2 and denote x 1 ≺ x 2 iff fi (x 1 ) ≤fi (x 2 ) for all i ∈ {1, ...,M } and fi (x 1 ) < fi (x 2 ) for at least one index
GECCO ’17, Berlin, Germany
2016. 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
DOI: 10.1145/nnnnnnn.nnnnnnn

i. Here, x ∗ is a Pareto optimal solution if there exists no x ∈ Ssuch that x ≺ x ∗ . Moreover, f (x ∗ ) is a Pareto optimal objectivefunction vector. The set of all x ∗ in S is the Pareto optimal solution
set (PS), and the set of all f (x ∗ ) is the Pareto frontier (PF). The
goal of an MOP is to find a set of nondominated solutions that are
well-distributed and close to the PF in the objective function space.
A multi-objective evolutionary algorithm (MOEA) is an efficientapproach for solving MOPs [16]. However, while classical MOEAs(e.g., NSGA-II [5] and SPEA2 [40]) perform relatively well for MOPs
with M ≤ 3, their performance significantly degrades for MOPs
with M ≥ 4 [30]. An MOP with M ≥ 4 is referred to as a manyobjective continuous optimization problem (MaOP). In the past
few years, researchers in the evolutionary computation community
have attempted to design new MOEAs that can handle a large
number of objectives [16]. Table 1 shows 21 MOEAs, including
recently proposed methods for MaOPs (e.g., NSGA-III [6]) andclassical methods which were originally designed only for MOPs
with M ≤ 3 (e.g., NSGA-II [5]).
Which MOEA is best? While a number of MOEAs for MaOPs
have been proposed in 2014 – 2016 as shown in Table 1, as far
as we know, an exhaustive benchmarking study has never been
performed. In almost all previous studies, a proposed MOEA wascompared with only a few algorithms/heuristics selected by theauthors. Although a number of previous studies have examinedbenchmarking MOEAs for MaOPs (e.g., [23, 25, 30]), the number ofmethods for comparison was limited to 8 ∼ 9.
Moreover, most importantly, in the previous benchmarking studies [23, 25, 30] and in all of the papers listed in Table 1, MOEAs werecompared based on nondominated solutions in the final population
at the end of the search. Note that the performance rank decided
by the above traditional benchmarking procedure strongly depends
on the maximum number of function evaluations (FEvalsmax ). In
other words, for example, the conclusion “MOEA1 performs better
than MOEA2 ” obtained by an experiment with FEvalsmax = 7 × 105
cannot be applied when FEvalsmax is set to 2 × 104 . However, as far
as we know, the standard setting of FEvalsmax for the end-of-run
results-based traditional comparison has never been defined in theevolutionary computation community, except for some competitions (e.g., [38]), and thus researchers set FEvalsmax to an arbitrary
number1 . Table 1 shows the FEvalsmax used in each of the previous
studies. These values of FEvalsmax differ significantly from eachother. For example, FEvalsmax was set to 1 × 104 in the HypE paper[2], whereas, in the recent studies, FEvalsmax was set to 5.5 × 105
1 We do not have an intention to claim that researchers set FEvalsmax to an arbitrary
number for all possible problems. Note that FEvalsmax cannot be set to an arbitrary num-

ber in practice since some real-world problems require the execution of a simulationthat takes a long time to evaluate the solution [15, 26].

GECCO ’17, July 15–19, 2017, Berlin, Germany

Ryoji Tanabe and Akira Oyama

Table 1: Properties of 21 MOEAs benchmarked in this paper:
C1 Pareto-dominance based MOEAs, C2 relaxed-dominance based
MOEAs, C3 decomposition based MOEAs, C4 indicator based
MOEAs, and C5 reference-vector based MOEAs. We also show
FEvalsmax used in each paper and its publication year. MOEA/D-07
and MOEA/D-09 correspond to the original names “MOEA/D” and“MOEA/D-DE”, respectively.
MOEAs

C1

NSGA-II [5]X

NSGA-III [6]X

SPEA2 [40]X

SPEA2+SDE [21]X

GrEA [33]

C2

C3

C4

X

θ -DEA [34]X

Nonconvex

X

X
X

Deceptive

X

Biased

WFG7

Nonconvex

2001

1.0 × 106

WFG8

Nonconvex

Biased2014

1.0 × 105

WFG9

Nonconvex

Deceptive, Biased

2013

1.0 × 105X

2016

5.5 × 105X

2016

5.5 × 1052009

3.0 × 105

MOEA/D-DRA [37]

X

2009

3.0 × 105

MOEA/D-STM [20]

X

2014

3.0 × 105X

2015

5.5 × 105X

2016

2.6 × 105X

2015

1.4 × 106

EFR-RR [35]

X

2016

2.6 × 105

RVEA [4]X2016

2.8 × 105X

IBEAϵ + [39]

X

2004

2.0 × 104

IBEAH D [39]

X

2004

2.0 × 104X

2011

1.0 × 104

BiGE [22]X

2015

1.0 × 105

MOMBI-II [8]

X

2015

4.4 × 105

X

Nonconvex

WFG5

5.5 × 105

2007

HypE [2]Degenerate

WFG4

Biased

X

2014

X

X

WFG3

OthersX

Nonconvex

X

I-DBEA [1]Discontinuous

Separability

WFG6

MOEA/D-09 [18]MOEA/D-DU [35]WFG2Multimodality

2.5 × 104

MOEA/D-07 [36]

X

Mixed

Year

7.5 × 104

MOEA/DD [19]Shape of PF

WFG1

2002

C5

X

X

Function

FEvalsmax

X

VaEA [32]Table 2: Properties of the WFG functions.

[6, 19, 32, 34]. A number of previous studies (e.g., [19–21, 33, 34])
have reported the poor performance of HypE. However, HypE was
originally designed for optimization for FEvalsmax = 1 × 104 , and aperformance comparison with a larger FEvalsmax may seem unfair.
In addition, most previous studies used nondominated solutionsin the population (at the end of the search) for calculating the hypervolume (HV) value. In general, MOEAs maintain (nondominated)
solutions obtained during the search process in the population, thesize of which is limited. When using nondominated solutions in the
population for the HV calculation, a monotonic increase of the HVover time (which is equal to the number of function evaluations)cannot be ensured [7, 10, 24, 27]. Thus, we cannot exactly evaluate
the performance of MOEAs using such a traditional evaluation
methodology.
Taking into account the above considerations, in this paper, we
exhaustively investigate the convergence performance of the 21
MOEAs listed in Table 1 on MaOPs with up to six objectives using
an unbounded external archive. The unbounded external archive
stores all nondominated solutions found during the search process and can be introduced into any MOEAs without any changes
in their original algorithms [3, 7, 10, 24, 27]. When using the unbounded external archive, the issue described above can be addressed [24, 27]. Since, in practice, users of MOEAs want to know

nondominated solutions whenever possible, we believe that thebenchmark methodology using the unbounded external archive ismore practical. The unbounded external archive was also adopted
in the recently proposed BBOB-biobj benchmark suite [29] in the
COCO framework2 . When the number of obtained nondominatedsolutions in the unbounded external archive is too large, a methodthat selects only representative solutions (e.g., [13]) should be applied to them.
This paper is inspired by two recent studies [3, 29]. In [3], Brockhoff et al. established the data profile-based methodology for visualizing the convergence performance of MOEAs, and their work was
extended to the recently designed BBOB-biobj functions [29] in the
COCO platform. We believe that the COCO with the BBOB-biobj
suite [29] is one of most reliable benchmarking platforms. However,
unfortunately, COCO currently provides only two-objective MOPs
and the performance of MOEAs for MOPs with M ≥ 3 cannot be
evaluated using the current COCO platform. Although some previous studies evaluated the convergence performance of MOEAs[9, 26], these studies did not use the unbounded external archive.
This paper will be helpful to both users and algorithm designersof MOEAs. In practice, users want to apply a high-performance
MOEA to real-world problems in order to obtain well-approximatednondominated solutions. Algorithm designers develop novel, stateof-the-art MOEAs by analyzing and improving well-performing
MOEAs. Our benchmarking study will be useful in both cases.
Moreover, all of the experimental data presented herein are available on the Website3 . Thus, researchers can easily compare their
newly designed MOEAs to the 21 MOEAs.
The remainder of this paper is organized as follows. Section
2 introduces the experimental settings used in this study. The
experimental results are described and discussed in Section 3. We
further discuss the experimental results and some previous work in
Section 4. Finally, in Section 5, we conclude this paper and discuss
our future work.

2

EXPERIMENTAL SETTINGSIn this benchmarking study, we used the nine WFG functions [11]with M ∈ {2, 3, 4, 5, 6}. Table 2 shows the properties of the WFG
functions. As suggested in [11], the position parameter k was set
2 http://coco.gforge.inria.fr/

3 https://sites.google.com/site/benchmarkingmoeas/

Benchmarking MOEAs for Multi- and Many-objective Optimization
WFG1 (M = 2)

0.7
0.6

Median hypervolume values across all 21 runs

0.5
0.4
0.3
0.2
0.1
0.0

0

10000

20000

30000

40000

50000

WFG1 (M = 4)

0.7

0.5
0.4
0.3
0.2
0.1

0

10000

20000

30000

40000

50000

0.5
0.4
0.3
0.2
0.1

0

10000

20000

30000

40000

0.50
0.45
0.40

0

10000

50000

IBEA+
MOEA/D-07
MOEA/D-DRA
IBEAHD
MOEA/D-09
MOEA/D-STM
θ-DEA
SPEA2+SDE
BiGE
HypE
EFR-RR
VaEA
RVEA
GrEA
MOMBI-II
NSGA-III
NSGA-III-DBEA
MOEA/D-DU
SPEA2
MOEA/DD

20000

30000

40000

50000

0.8

0.7

0.6

0

10000

20000

30000

40000

50000

MOEA/D-DU
HypE
EFR-RR
IBEA+BiGE
SPEA2+SDE
IBEAHD
MOEA/D-STM
MOMBI-II
NSGA-II
GrEA
SPEA2
θ-DEA
NSGA-III
VaEA
MOEA/DDRVEA
I-DBEA
MOEA/D-09
MOEA/D-DRA
MOEA/D-07

WFG2 (M = 6)

0.8

0.7

0.6

0

10000

20000

30000

40000

0.50

0.45

0.40

0.35

0

10000

50000

HypE
MOEA/D-DU
SPEA2+SDE
MOEA/D-STM
BiGE
EFR-RR
NSGA-II
IBEA+MOMBI-II
MOEA/D-DRA
GrEA
VaEA
SPEA2
θ-DEA
NSGA-III
MOEA/DDRVEA
I-DBEAIBEAHD
MOEA/D-09
MOEA/D-07

20000

30000

40000

50000

0.65
0.60
0.55
0.50
0.45
0.40

0

10000

20000

30000

40000

50000

MOEA/D-DRA
MOEA/D-09
MOEA/D-07
BiGE
IBEA+
MOEA/D-STM
HypE
MOMBI-II
RVEA
SPEA2+SDE
NSGA-II
MOEA/D-DU
EFR-RR
θ-DEA
IBEAHD
NSGA-III
GrEA
I-DBEA
SPEA2
MOEA/DDVaEA

WFG3 (M = 6)

0.70
0.65
0.60
0.55
0.50
0.45
0.40

IBEAHD
IBEA+SPEA2+SDE
BiGE
MOEA/D-STM
GrEA
MOEA/D-DU
VaEA
MOEA/DD
EFR-RR
θ-DEA
MOEA/D-DRA
MOEA/D-09
NSGA-II
MOEA/D-07
SPEA2
MOMBI-II
NSGA-IIIHypE
RVEA
I-DBEA

WFG3 (M = 4)

0.75

0.9

0.5

0.55

0.70

0.9

0.5

EFR-RRRVEA
SPEA2+SDE
MOEA/D-STM
BiGE
GrEA
VaEA
MOMBI-II
SPEA2
θ-DEA
IBEA+
NSGA-III
MOEA/D-09
IBEAHD
MOEA/D-DU
NSGA-II
MOEA/D-DRA
HypE
MOEA/DD
MOEA/D-07
I-DBEAWFG2 (M = 4)

1.0

0.6

0.0

IBEA+
MOEA/D-07
MOEA/D-DRA
MOEA/D-STM
MOEA/D-09
IBEAHD
EFR-RR
SPEA2+SDE
θ-DEA
HypE
NSGA-II
BiGE
GrEA
MOEA/D-DU
VaEA
RVEA
MOMBI-II
NSGA-IIII-DBEA
SPEA2
MOEA/DDWFG1 (M = 6)

0.7

0.55

0.35

WFG3 (M = 2)

0.60

0.60

1.0

0.6

0.0

WFG2 (M = 2)

0.65
MOEA/D-09
IBEA+
MOEA/D-STM
MOEA/D-DRA
MOEA/D-DU
NSGA-IIIBEAHD
EFR-RR
MOEA/DD
MOEA/D-07
SPEA2
VaEA
NSGA-III
θ-DEA
SPEA2+SDE
GrEA
HypE
BiGE
I-DBEARVEA
MOMBI-II

GECCO ’17, July 15–19, 2017, Berlin, Germany

0

10000

20000

30000

40000

50000

BiGE
MOEA/D-09
MOEA/D-DRA
IBEA+HypE
MOEA/D-07
MOMBI-II
GrEA
SPEA2+SDE
MOEA/D-STM
I-DBEARVEA
NSGA-II
MOEA/D-DU
θ-DEA
EFR-RR
NSGA-III
VaEA
IBEAHD
MOEA/DD
SPEA2

Number of function evaluations
Figure 1: Convergence performance of the 21 MOEAs on the normalized WFG1, WFG2, and WFG3 functions with M ∈ {2, 4, 6}.

to k = 2 (M − 1), and the distance parameter l was set to l = 20,
where the number of variables D is D = k + l 4 .
On all the WFG functions, the scale of each objective function
is different. In general, an MOEA with an efficient normalization
strategy for handling differently scaled objective function values
performs well for such scaled MOPs [6]. Since objective function
values of real-world problems are differently scaled, normalization
strategies are mandatory in practice. However, in the 21 MOEAs,
while some MOEAs adopt sophisticated normalization strategies(e.g., NSGA-III [6] and RVEA [4]), the others do not use any normalization strategies (e.g., NSGA-II [5] and MOEA/D-07 [36]). Since wewant to evaluate the performance of MOEAs as fairly as possible, weused the normalized WFG functions to remove the effect of the normalization strategies as [14, 35]. For the normalized WFG functions,each objective function value fi (x ),i ∈ {1, ...,M } was normalizedusing the ideal point (0, ..., 0) T and the nadir point (2, ..., 2M ) T as
in [14, 35].
4 In order to maintain consistency of properties of the WFG functions, k must be set

to a number divisible by (M − 1) . Thus, M and D are dependent on each other, and D
could not be constant like the BBOB-biobj functions [29]. The scalability of MOEAs
with respect to D will be investigated in our future work.

As suggested in [14, 35], the reference point for calculating theHV value was set to (1.1, ..., 1.1) T . Note that the nadir point of
the normalized WFG functions is (1, ..., 1) T . In this setting, theHV range for all of the WFG functions is [0, 1.1M ]. We further
normalized HV values ∈ [0, 1.1M ] to the range [0, 1] by divided by
1.1M . The computational cost of the HV calculation exponentially
increases both with M and the number of nondominated solutionsin the unbounded external archive. In the COCO software with thetwo-objective BBOB-biobj functions [29], when a newly generated
solution enters the unbounded external archive Aunb , the HV valueof the nondominated solutions in Aunb is recalculated immediately.
However, for MaOPs with M ≥ 4, since the computational costof the HV calculation is very high, on-the-fly HV calculation likethe COCO software with the two-objective BBOB-biobj functions
is almost impossible. In order to address this issue, in this paper
we calculated the HV value of the nondominated solutions in Aunb
only when FEvals ∈ {1 × 103 , 2 × 103 , ..., 4.9 × 104 , 5 × 104 }.
We used the SBX crossover and polynomial mutation for all 21
MOEAs. As suggested in [6], we set the control parameters of thevariation operators as follows: pc = 1.0, ηc = 30, pm = 1/D, andηm = 20. For most of the 21 MOEAs, we used the source codes

GECCO ’17, July 15–19, 2017, Berlin, Germany
WFG4 (M = 2)

0.36

0.32

Median hypervolume values across all 21 runs

0.30
0.28
0.26
0.24
0.22
0

10000

20000

30000

40000

50000

0.6

0.5

0.4

0

10000

20000

30000

40000

50000

0.7
0.6
0.5
0.4

0

10000

20000

30000

40000

0.26
0.24
0.22
0.20
0.18
0.16
0

10000

50000

20000

30000

40000

50000

0.6
0.5
0.4
0.3

0

10000

20000

30000

40000

50000

IBEAHD
IBEA+RVEA
MOEA/D-DU
MOEA/D-DRA
MOEA/D-07
HypE
BiGE
EFR-RR
MOEA/D-09
MOEA/D-STM
SPEA2+SDE
MOMBI-II
GrEA
θ-DEA
MOEA/DD
NSGA-III
VaEA
I-DBEA
NSGA-II
SPEA2

WFG5 (M = 6)

0.7
0.6
0.5
0.4

0

10000

20000

30000

40000

0.25

0.20

0.15

0.10

0

10000

50000

IBEAHD
RVEA
BiGE
IBEA+
MOEA/D-DRA
SPEA2+SDE
MOEA/D-07
MOEA/D-09
HypE
MOEA/D-STM
MOEA/D-DU
EFR-RR
MOMBI-II
GrEA
θ-DEA
VaEA
NSGA-IIII-DBEA
MOEA/DDSPEA2
NSGA-II

20000

30000

40000

50000

0.7
0.6
0.5
0.4
0.3
0.2

0

10000

20000

30000

40000

50000

IBEAHD
IBEA+
MOEA/D-07
MOEA/D-DU
MOEA/D-DRA
MOEA/D-09
RVEA
SPEA2+SDE
BiGE
EFR-RR
GrEA
HypE
MOMBI-II
MOEA/D-STM
θ-DEA
MOEA/DD
NSGA-III
VaEA
I-DBEA
NSGA-II
SPEA2

WFG6 (M = 6)

0.8
0.7
0.6
0.5
0.4
0.3
0.2

SPEA2+SDE
IBEA+
GrEA
VaEA
θ-DEA
MOEA/DDBiGE
MOEA/D-STM
IBEAHD
NSGA-II
SPEA2
MOEA/D-DU
HypE
MOMBI-II
MOEA/D-07
EFR-RR
MOEA/D-DRA
RVEA
MOEA/D-09
NSGA-IIII-DBEAWFG6 (M = 4)

0.9

0.8

0.3

0.30

0.8

0.7

0.2

HypE
BiGE
NSGA-II
IBEA+IBEAHDSPEA2+SDE
VaEA
EFR-RR
GrEA
NSGA-III
SPEA2
θ-DEA
MOMBI-III-DBEARVEA
MOEA/D-DU
MOEA/D-DRA
MOEA/D-STM
MOEA/D-09
MOEA/DD
MOEA/D-07

WFG5 (M = 4)

0.9
MOEA/D-DU
BiGE
IBEA+SPEA2+SDE
HypE
MOEA/D-DRA
MOEA/D-STM
MOEA/D-09
RVEA
EFR-RR
MOEA/D-07
MOMBI-II
GrEA
θ-DEA
VaEA
NSGA-IIII-DBEA
MOEA/DDSPEA2
NSGA-IIIBEAHD

0.8

0.3

IBEAHD
MOEA/D-DU
MOEA/D-07
IBEA+
MOEA/D-DRA
MOEA/D-09
RVEA
EFR-RRBiGE
SPEA2+SDE
HypE
MOEA/D-STM
MOMBI-II
GrEA
MOEA/DD
θ-DEA
NSGA-III
VaEA
I-DBEA
NSGA-II
SPEA2

WFG4 (M = 6)

0.9

0.28

0.14

WFG6 (M = 2)

0.35

0.30

0.8

0.7

0.3

IBEA+IBEAHDGrEA
SPEA2+SDE
MOEA/D-STM
BiGE
MOEA/D-DU
EFR-RR
MOEA/D-DRA
MOEA/D-09
MOMBI-II
NSGA-III
MOEA/DD
MOEA/D-07
θ-DEA
NSGA-II
VaEA
SPEA2
HypE
RVEA
I-DBEA

WFG4 (M = 4)

0.8

WFG5 (M = 2)

0.32

0.34

0.20

Ryoji Tanabe and Akira Oyama

0

10000

20000

30000

40000

50000

IBEAHDSPEA2+SDE
IBEA+BiGE
MOEA/D-DU
RVEA
EFR-RR
MOMBI-II
MOEA/D-STM
MOEA/D-DRA
MOEA/D-09
MOEA/D-07
HypE
GrEA
θ-DEA
I-DBEA
VaEA
NSGA-III
MOEA/DD
NSGA-II
SPEA2

Number of function evaluations
Figure 2: Convergence performance of the 21 MOEAs on the normalized WFG4, WFG5, and WFG6 functions with M ∈ {2, 4, 6}.

downloaded from websites as listed in Table S.1 in the supplementary file. We replaced the DE operator with the SBX crossover in
MOEA/D-09, MOEA/D-DRA, and MOEA/D-STM. A slightly modified code of MOEA/D-09 was used as the source code of MOEA/D-07.
The bug-fixed jMetal source code of IBEAH D was used in this study.
We implemented IBEAϵ + by modifying the source code of IBEAH D .
For each M ∈ {2, 3, 4, 5, 6}, we set the population size µ to 100, 92,
220, 212, and 184 respectively for GrEA, BiGE, SPEA2+SDE, and
MOMBI-II because µ in their codes must be set as a multiple offour. For the remaining MOEAs, µ was set to 100, 91, 220, 210, and
182 respectively. The µ values used in this paper have been widelyused in previous studies such as [1, 6, 19, 35]. For the decomposition and reference vector based MOEAs, weight/reference vectors
were generated using simplex-lattice design (for only M = 6, its
two-layered version [6] was used). For GrEA, we set the div value
as suggested in [4, 35]. The maximum number of function evaluations (FEvalsmax ) was set to 5 × 104 , and 21 independent runs were
performed.

3

EXPERIMENTAL RESULTS

In this section, we investigate the performance of the 21 MOEAs
listed in Table 1 on the WFG functions [11]. The experimental
results for each function are described in Section 3.1. Then, in
Section 3.2, we compare the MOEAs for all of the WFG functions.
We also discuss the experimental results in Section 4.

3.1

Results for each WFG function

Figures 1, 2, and 3 show the results for the 21 MOEAs for each
WFG function instance. Due to space constraints, we show only
the results for the WFG functions with M ∈ {2, 4, 6}, but the resultsfor M ∈ {3, 5} are presented in Figures S.1 ∼ S.3 in the supplemental
file. Figures S.4 ∼ S.6 in the supplemental file also show the detailed
results for FEvals ∈ {3 × 104 , 3.1 × 104 , ...5 × 104 }. In the following,
we discuss the results for each WFG function:• WFG1: IBEAϵ + and MOEA/D variants perform well in M ∈{2, 4, 6}. For M = 2, MOEA/D-STM achieves the highest HV value
until FEvals = 3 × 104 , but MOEA/D-09 and IBEAϵ + catch up with
MOEA/D-STM for FEvals > 3 × 104 . For M = 4 and 6, the best
performer is MOEA/D-07 for FEvals < 3 × 104 and < 1.3 × 104
respectively. After that, IBEAϵ + outperforms the other MOEAs.

Benchmarking MOEAs for Multi- and Many-objective Optimization
WFG7 (M = 2)

0.40

0.35

Median hypervolume values across all 21 runs

0.30

0.25

0.20

0.15

0

10000

20000

30000

40000

50000

WFG7 (M = 4)

0.8

0.6

0.5

0.4

0

10000

20000

30000

40000

50000

0.8
0.7
0.6
0.5
0.4

0

10000

20000

30000

40000

0.26
0.24
0.22
0.20
0.18
0.16
0

10000

50000

BiGE
IBEA+SPEA2+SDE
MOEA/D-DU
EFR-RR
MOEA/D-STM
RVEA
MOEA/D-DRA
MOEA/D-09
HypE
MOMBI-II
MOEA/D-07
GrEA
IBEAHD
VaEA
θ-DEA
MOEA/DD
NSGA-IIII-DBEASPEA2
NSGA-II

20000

30000

40000

50000

0.60
0.55
0.50
0.45
0.40
0.35
0.30
0

10000

20000

30000

40000

50000

MOEA/D-DU
IBEAHD
IBEA+SPEA2+SDE
BiGE
EFR-RR
MOEA/D-07
MOEA/D-DRA
MOEA/D-09
MOMBI-II
GrEA
MOEA/DD
HypE
RVEA
θ-DEA
NSGA-III
MOEA/D-STM
I-DBEA
VaEA
NSGA-II
SPEA2

WFG8 (M = 6)

0.6
0.5
0.4
0.3

0

10000

20000

30000

40000

0.25

0.20

0.15

0.10

0

10000

50000

BiGE
MOEA/D-DU
IBEA+SPEA2+SDE
MOEA/D-DRA
MOMBI-II
HypE
MOEA/D-STM
MOEA/D-09
EFR-RR
MOEA/D-07
RVEA
IBEAHD
θ-DEA
NSGA-III
GrEA
MOEA/DD
I-DBEA
VaEA
NSGA-II
SPEA2

20000

30000

40000

50000

0.65
0.60
0.55
0.50
0.45
0.40
0.35
0.30
0.25

0

10000

20000

30000

40000

50000

RVEA
IBEA+SPEA2+SDE
IBEAHDBiGE
EFR-RR
GrEA
MOEA/D-DU
MOMBI-II
MOEA/D-09
MOEA/D-DRA
HypE
MOEA/D-07
θ-DEA
VaEA
MOEA/D-STM
NSGA-III
MOEA/DD
NSGA-III-DBEASPEA2

WFG9 (M = 6)

0.7
0.6
0.5
0.4
0.3
0.2

SPEA2+SDE
BiGE
MOEA/D-DU
θ-DEA
MOEA/DD
NSGA-III
VaEA
SPEA2
MOMBI-II
MOEA/D-DRA
RVEA
MOEA/D-09
HypE
IBEAHD
MOEA/D-07
IBEA+GrEA
MOEA/D-STM
EFR-RR
NSGA-III-DBEAWFG9 (M = 4)

0.8

0.7

0.2

0.30

0.70

0.65

0.25

MOEA/D-DU
MOEA/D-STM
MOEA/D-07
SPEA2+SDE
MOEA/D-DRA
MOEA/DDIBEAHD
MOEA/D-09
NSGA-III
MOMBI-II
IBEA+EFR-RR
GrEA
VaEA
SPEA2
BiGE
NSGA-II
θ-DEA
HypE
RVEA
I-DBEAWFG8 (M = 4)

0.8

0.9

0.3

IBEAHD
MOEA/D-DU
IBEA+EFR-RRRVEA
SPEA2+SDE
BiGE
MOEA/D-DRA
MOEA/D-07
GrEA
MOEA/D-09
HypE
MOMBI-II
MOEA/D-STM
θ-DEA
MOEA/DD
NSGA-III
VaEA
I-DBEA
NSGA-II
SPEA2

WFG7 (M = 6)

1.0

0.28

0.14

WFG9 (M = 2)

0.35

0.30

0.70

0.7

0.3

WFG8 (M = 2)

0.32
IBEAHD
IBEA+SPEA2+SDE
GrEA
BiGE
MOEA/D-DU
MOEA/D-STM
EFR-RR
MOEA/D-09
MOEA/D-DRA
NSGA-III
HypE
MOEA/DD
θ-DEA
MOEA/D-07
NSGA-II
VaEA
SPEA2
MOMBI-IIRVEA
I-DBEAGECCO ’17, July 15–19, 2017, Berlin, Germany

0

10000

20000

30000

40000

50000

SPEA2+SDE
IBEA+IBEAHD
RVEA
BiGE
GrEA
HypE
MOEA/D-STM
MOMBI-II
MOEA/D-DRA
MOEA/D-DU
MOEA/D-07
EFR-RR
VaEA
MOEA/D-09
θ-DEA
NSGA-III
SPEA2
I-DBEA
MOEA/DD
NSGA-II

Number of function evaluations
Figure 3: Convergence performance of the 21 MOEAs on the normalized WFG7, WFG8, and WFG9 functions with M ∈ {2, 4, 6}.

• WFG2: For M = 2, EFR-RR outperforms the other MOEAs afterFEvals = 2 × 104 . For M = 4 and M = 6, HypE and MOEA/D-DU
achieve high HV values, whereas MOEA/D-07 and MOEA/D-09
perform poorly.
• WFG3: For M = 2, IBEAH D performs relatively well after FEvals= 1×104 . However, with increasing M, the performance of IBEAH D
deteriorates gradually. For M = 4 and M = 6, while MOEA/D09, MOEA/D-DRA, IBEAϵ + , and BiGE perform well for the larger
FEvals, the best MOEA for the smaller FEvals is HypE.
• WFG4: For M = 2 and M = 4, IBEAH D exhibits good performance at all times, while for M = 6, its performance becomes
inferior to the other MOEAs. On the other hand, for M = 6, BiGE
and IBEAϵ + achieve high HV values, and MOEA/D-DU outperforms
the remaining MOEAs at approximately FEvals = 5 × 104 .
• WFG5: For M = 2 and = 4, the best MOEA is RVEA within
FEvals = 8 × 103 and = 2.1 × 104 respectively. For M = 6, IBEAH D
exhibits the best performance on the six-objective WFG5 function.
• WFG6: For M = 2, IBEAH D and IBEAϵ + perform well for thesmaller FEvals. The two IBEA variants also outperform the remaining MOEAs for M = 4. IBEAH D is also the best performer for
M = 6.

• WFG7: The two IBEA variants show the best anytime performance for M = 2. For M = 4, IBEAH D and IBEAϵ + also perform
well, and MOEA/D-DU and EFR-RR catch up them for larger FEvals.
For M = 6, IBEAϵ + still performs well while the performance of
IBEAH D is degraded. Moreover, BiGE and SPEA2+SDE are competitive with IBEAϵ + .
• WFG8: MOEA/D-STM outperforms the other MOEAs for M = 2
at all times. For M = 4, the best MOEA is IBEAH D within FEvals =2 × 104 , and MOEA/D-DU outperforms the other MOEAs. Similarly,
for M = 6, HypE achieves the highest HV value until FEvals =1 × 104 , and BiGE performs best after that.
• WFG9: For M = 2, SPEA2+SDE performs relatively well while
the evolution of the five MOEAs (GrEA, MOEA/D-STM, EFR-RR,
NSGA-II, and I-DBEA) clearly stagnates just after the beginning of
the search. For M = 4, RVEA exhibits the good performance at all
times and is clearly the best MOEA. For M = 6, IBEAH D performs
well, but SPEA2+SDE and IBEAϵ + outperform IBEAH D for larger
FEvals.

GECCO ’17, July 15–19, 2017, Berlin, Germany
M =2

20

APS values

M =4

18

10

5

0

10000

20000

30000

40000

50000

0

M =6

20

I-DBEA
RVEA
16
MOEA/D-07
HypE
14
MOMBI-II
SPEA2
θ-DEA
12
NSGA-II
MOEA/DD
10
VaEA
NSGA-III
MOEA/D-09
8
MOEA/D-DRA
EFR-RR
6BiGE
GrEA
MOEA/D-DU
4
MOEA/D-STM
SPEA2+SDE
2
IBEAHD
IBEA+

15

0

Ryoji Tanabe and Akira Oyama

SPEA2
I-DBEA
VaEA
NSGA-II
MOEA/DD
15
NSGA-III
θ-DEA
GrEA
MOMBI-II
MOEA/D-STM
HypE
10
RVEA
EFR-RR
MOEA/D-07
SPEA2+SDE
BiGE
5
MOEA/D-09
MOEA/D-DRA
MOEA/D-DU
IBEAHD
IBEA+
0

10000

20000

30000

40000

50000

Number of function evaluations

0

SPEA2
MOEA/DD
NSGA-III-DBEA
NSGA-III
VaEA
θ-DEA
GrEA
MOMBI-II
EFR-RR
MOEA/D-07
RVEA
IBEAHD
MOEA/D-DU
MOEA/D-09
MOEA/D-STM
HypE
MOEA/D-DRA
SPEA2+SDE
BiGE
IBEA+
0

10000

20000

30000

40000

50000

Figure 4: Results for the 21 MOEAs based on the APS for all WFG functions with M ∈ {2, 4, 6} (lower is better).

3.2

Overall performance

While the previous section described the results for each WFG function, here, we describe the overall performance of the 21 MOEAs
for all nine WFG functions with each M. For this comparison, weused the average performance score (APS) [2]. Suppose that n algorithms {A1 , ...,An } are compared for a given problem instanceusing the HV values obtained in multiple runs. For each i ∈ {1, ...,n}and j ∈ {1, ...,n}\{i}, let δi,j = 1, if A j significantly outperforms
Ai using the Wilcoxon rank-sum test with p < 0.05, otherwise
δi,j = 0. Then, the performance score P (Ai ) is defined as follows:P
P (Ai ) = nj∈ {1,...,n }\{i } δi,j . P (Ai ) represents the number of algorithms outperforming Ai . The APS is the average of the P (Ai )values for all problem instances. In other words, the APS valueof Ai represents how good (relatively) the performance of Ai is
among the n algorithms for all problem instances (lower is better).
Figure 4 shows the results for the 21 MOEAs for all the nine
WFG functions with M ∈ {2, 4, 6}. We calculated the APS value forevery FEvals ∈ {1, 1 × 103 , ..., 5 × 104 }. The results for M ∈ {3, 5} canbe found in Figure S.7 in the supplemental file. Below, we discuss
the results for each algorithm category:• Seven Pareto dominance-based MOEAs: Interestingly, in thiscategory, the worst and best performers are SPEA2 and SPEA2+SDE
respectively. SPEA2+SDE is an improved version of SPEA2 thatincorporates the SDE strategy [21]. Based on these results, theSDE strategy appears to contribute significantly to the outstanding
performance of SPEA2+SDE. For almost all M, the convergence
speed of NSGA-III is lower than that of NSGA-II, and NSGA-II
performs better than NSGA-III for the smaller FEvals, even whenM = 6. NSGA-III is an improved version of NSGA-II for MaOPs by
replacing the crowding distance-based selection with the reference
vectors-based niching selection. The newly introduced niching selection appears to make the convergence speed of NSGA-III slow.
The two relaxed-dominance based MOEAs (GrEA and θ -DEA) perform similarly to each other for the WFG functions with M ≥ 4.
• Nine decomposition-based MOEAs: For M = 2, MOEA/DSTM clearly performs better than the remaining decompositionbased MOEAs. However, within FEvals = 2 × 103 , the classical
MOEA/D-07 outperforms its improved variants. For M ∈ {4, 5, 6},
RVEA performs best for much smaller FEvals, followed by MOEA/D07. For M = 4, after some FEvals, MOEA/D-DRA performs very
well, but MOEA/D-DU outperforms MOEA/D-DRA after FEvals= 4.3 × 104 . For M = 6, the best performer is also MOEA/DDRA, followed by MOEA/D-STM and MOEA/D-09. I-DBEA andMOEA/DD clearly perform worse than the other decompositionbased MOEAs including MOEA/D-07. In this category, only I-DBEA
and MOEA/DD use the Pareto-dominance for the selection, and it
might cause their poor performance. However, with increasing M,
the APS value of I-DBEA decreases gradually, and so it is possiblethat I-DBEA performs well for MaOPs with a large number ofobjectives.
• Five indicator-based MOEAs: In this category, MOMBI-II (anR2-indicator based MOEA) is the worst performer, but it outperforms some MOEAs in the other categories. For M ∈ {2, 3, 4, 5}, thebest performer is clearly IBEAϵ + , followed by IBEAH D . For M = 6,
HypE performs very well within FEvals = 1 × 104 . However, as thesearch progresses, the APS value of HypE gradually deteriorates
for all M. For FEvals > 104 , IBEAϵ + outperforms the remaining
indicator-based MOEAs, but the APS value of BiGE is gradually
improved. Taking into account the results, when M is increasing,
BiGE might perform well for a larger FEvals.
• All 21 MOEAs: For M ∈ {2, 3}, the best performer is IBEAϵ + ,followed by IBEAH D . For M ∈ {4, 5, 6}, RVEA performs best formuch smaller FEvals. For M = 4, IBEAϵ + still outperforms theremaining MOEAs. For M ∈ {5, 6}, among the 21 MOEAs, the best
MOEA is HypE for the smaller FEvals, followed by IBEAϵ + .

4

DISCUSSION

Here, we discuss the results described in Section 3. Unsurprisingly,
the results show that the performance rank of the 21 MOEAs depends significantly for (i) the function type, (ii) M, and (iii) the
number of function evaluations (FEvals). Thus, determining a best
MOEA among the 21 MOEAs is difficult, even though for sameproblem instance. For example, for the six-objective WFG8 function, HypE performs best within FEvals = 1 × 104 , followed by BiGE.
Keeping in mind this fact, in the following, we roughly discuss theperformance of the 21 MOEAs based on the APS value (Figure 4).
The performance degradation of NSGA-II and SPEA2 with increasing M is described in Section 3. This tendency is consistent
with the previous study [30]. On the other hand, some recently
proposed MOEAs (e.g., NSGA-III, MOEA/DD, and VaEA) perform
poorly for the WFG functions with M ≤ 4. The reason seems thatthese MOEAs were originally designed for MaOPs with a large
number of objectives. The poor performance of MOEAs that wereIBEA+NSGA-III

1.0

1.0

0.2
0.0

0.8

f3

0.8
0.6
0.4

0.4

0.2

0.2

0.0
0.0

0.2

0.4
0.2

0.4

0.6

f10.6

0.8
0.8

1.0

0.6

0.0

f2

0.0
0.0

0.4
0.2

0.4

1.0

0.6

f10.6

0.8
0.8

1.0

f2

1.0

(a) Nondominated solutions in the population
IBEA+

NSGA-III

1.0
0.8

0.6

0.2
0.0

0.4
0.2

0.4

0.6

f10.6

0.8
0.8

1.0

1.0

f2

0.6

0.4

0.4

0.2

0.2

0.0
0.0

0.2
0.0

f3

1.0
0.8

f3

specially designed for MaOPs for MOPs with a small number ofobjectives is consistent with the results of the previous study [23].
IBEAϵ + , which was proposed in 2004, is the first indicator based
MOEA and was not specially designed for MaOPs. The poor performance of IBEAϵ + for MaOPs as well as MOPs with M < 4 has
been reported in numerous articles (e.g., [8, 12, 17, 30, 39]). However, surprisingly, in this paper, the classical IBEAϵ + shows thehighest performance among the 21 MOEAs, including recently proposed state-of-the-art MOEAs, for the functions with two to six
objectives. Moreover, for M = 6, RVEA performs best immediately
after the start of the search, and HypE is the best MOEA up to
FEvals = 1 × 104 . HypE was proposed in 2011 and was speciallydesigned for MaOPs with up to 50 objectives, but it is the mostclassical algorithm among the 21 MOEAs listed in Table 1. HypE is
considered as a standard, benchmark algorithm and has frequently
been compared to newly proposed MOEAs [19, 21, 33, 34]. In otherwords, it is widely believed that recently proposed MOEAs performbetter than HypE. However, such conventional wisdom is clearly
contradicted by the experimental results in this paper. In summary,the benchmarking results presented in Section 3 are not consistent
with the results of the previous studies.
One reason for this may be the difference in the evaluation
methodology. We evaluated the performance of the 21 MOEAs
using the unbounded external archive, whereas previous studies
evaluated MOEAs using the final population at the end of the search.
In general, the population maintains the good (nondominated) solutions obtained during the search process of an MOEA, but the
population size is limited. Thus, MOEAs require a carefully designed
environmental selection method so that the solutions in the population are uniformly distributed. According to this policy, recently
proposed MOEAs for MaOPs (e.g., NSGA-III, MOEA/DD, and VaEA)
were designed. On the other hand, when using the unbounded
external archive, MOEAs do not require such well-designed environmental selection method because all nondominated solutions
obtained during the search process are automatically maintained in
the external archive independent from the population. We believethat the unbounded external archive is beneficial for an MOEA thatcannot maintain well-distributed solutions in the population.
For further investigation, we show the distribution of nondominated solutions in the population and the unbounded externalarchive obtained by IBEAϵ + and NSGA-III for the three-objective
WFG4 function in Figure 5. As shown in Figure 5(a), while NSGA-III
maintains the uniformly distributed solutions in the population,
the solutions obtained by IBEAϵ + are biased to specific regions (therim and the center of the PF). The biased distribution of IBEAϵ + , asshown in Figure 5(b), is also reported in a previous study [8, 17, 30].
On the other hand, as shown in Figure 5(b), both MOEAs successfully maintain the densely distributed solutions covering the entire
PF in the unbounded external archive. Moreover, the density ofthe distribution of the solutions obtained by NSGA-III is not sohigh as that of IBEAϵ + . It is possible that IBEAϵ + can generate
well-distributed solutions but cannot maintain them in the population. When using the unbounded external archive, this issue of
IBEAϵ + is addressed. In fact, recent studies [17, 31] have reported
the promising performance of IBEAϵ + variants with a (bounded)
external archive for maintaining obtained nondominated solutions.

GECCO ’17, July 15–19, 2017, Berlin, Germany

f3

Benchmarking MOEAs for Multi- and Many-objective Optimization

0.0
0.0

0.4
0.2

0.4

0.6

f10.6

0.8
0.8

1.0

f2

1.0

(b) Nondominated solutions in the unbounded external archive

Figure 5: Distribution of nondominated solutions in (a) the population and (b) the unbounded external archive at FEvals = 5 × 104
for the three-objective WFG4 function. Data for a single run with a
median HV value are shown (IBEAϵ + and NSGA-III).

Another reason for this might be the setting of the maximum
number of function evaluations (FEvalsmax ). As shown in Table 1,
we do not know why, but FEvalsmax used for comparative studies isincreasing yearly. For example, while FEvalsmax used in the HypE
paper [2], published in 2011, was only 1 × 104 , FEvalsmax used inrecent papers (e.g., NSGA-III [6], MOEA/DD [19], θ -DEA [34], and
VaEA [32]) was 5.5 × 105 . The latter is 55 times larger than the former.
Note that some real-world problems require the execution of a very
expensive simulation in order to evaluate the solution [15, 26], andthus, in practice, users of MOEAs cannot always set FEvalsmax to alarge number (e.g., 5.5 × 105 ). As mentioned above, many articles
report the poor performance of HypE [19, 21, 33, 34], but we believethat this is because of the different settings of FEvalsmax . HypE wasdesigned for optimization for FEvalsmax ≤ 1 × 104 , but the settingsof FEvalsmax used in other previous studies (e.g., [19, 32, 34]) wereclearly much larger. In fact, as shown in Figure 4, HypE performs
very well within FEvals = 1×104 for M = 6. Although most previous
studies determined the performance rank of MOEAs based only on
the end-of-run results as described above, the end-of-run results do
not provide sufficient information, as pointed out in [3], and mighthide an important fact (i.e., the excellent performance of HypE forsmall FEvals).

5

CONCLUSION

We have investigated the convergence performance of the 21 MOEAs
listed in Table 1 using the unbounded external archive that maintains all nondominated solutions obtained during the search process. Although the performance rank of the 21 MOEAs depend
significantly on (i) the function type, (ii) M, and (iii) FEvals, it issignificantly inconsistent with the results reported in previous studies. While the performance of some recently proposed MOEAs isGECCO ’17, July 15–19, 2017, Berlin, Germany
poor, some classical MOEAs perform very well (e.g., the two IBEA
variants [39] and HypE [2]) in this experimental study. We also
discussed why the performance rank obtained in this paper is significantly different from that in previous studies. Although a large
number of MOEAs for MaOPs have been proposed, we conclude
that their performance and the benchmarking methodology must
be carefully reconsidered in light of the experimental results.
We used the (normalized) WFG functions [11] for benchmarking the 21 MOEAs, but it is pointed out that they have some unnatural, exploitable features [3, 14]. For example, MOEAs withsystematically, uniformly generated reference and weight vectors(e.g., NSGA-III, RVEA, and MOEA/D variants) can successfully findwell-approximated nondominated solutions for the WFG functions
because the shape of the distribution of the reference and weightvectors is the same as the shape of the PF [14]. The comparisonof MOEAs for other benchmark functions and the original WFG
functions is an area for future research. Due to the expensive costof the exact HV calculation, running an MOEA for MaOPs withM ≥ 7 for FEvals > 5×104 is very difficult. In order to address theseissues, we will use a hypervolume approximation method [2]. Since
the main objective of this paper was to benchmark MOEAs, we onlyused the HV indicator. Analyzing the performance of MOEAs using
various performance indicators is an interesting future direction.
The performance of MOEAs depends on control parameter settings and can be improved by parameter tuning [27]. For example,[28] reveals that a simple parameter tuning significantly improves
NSGA-III. Although the results show that NSGA-III with the default
parameter setting [6] performs poorly, NSGA-III with tuned parameters may perform well. Its investigation is our future direction.
Due to space constraints, in this paper, the performance of
MOEAs was only qualitatively discussed (e.g., MOEA1 performs
better than MOEA2 ), not quantitatively discussed (e.g., MOEA1 is
two times faster than MOEA2 ) as in [3]. Our future work will quantitatively analyze the performance of the 21 MOEAs by measuring
the run-lengths to reach a predefined HV target value as in COCO.

ACKNOWLEDGMENT
This research is supported by the HPCI System Research Project “Researchand development of multiobjective design exploration and high-performancecomputing technologies for design innovation” (Project ID:hp160203).

REFERENCES[1] M. Asafuddoula, T. Ray, and R. A. Sarker. 2015. A Decomposition-Based Evolutionary Algorithm for Many Objective Optimization. IEEE TEVC 19, 3 (2015),
445–460.
[2] J. Bader and E. Zitzler. 2011. HypE: An Algorithm for Fast Hypervolume-Based
Many-Objective Optimization. Evol. Comput. 19, 1 (2011), 45–76.
[3] D. Brockhoff, T. Tran, and N. Hansen. 2015. Benchmarking Numerical Multiobjective Optimizers Revisited. In GECCO. 639–646.
[4] R. Cheng, Y. Jin, M. Olhofer, and B. Sendhoff. 2016. A Reference Vector Guided
Evolutionary Algorithm for Many-Objective Optimization. IEEE TEVC 20, 5(2016), 773–791.
[5] K. Deb, S. Agrawal, A. Pratap, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE TEVC 6, 2 (2002), 182–197.
[6] K. Deb and H. Jain. 2014. An Evolutionary Many-Objective Optimization Algorithm Using Reference-Point-Based Nondominated Sorting Approach, Part I:
Solving Problems With Box Constraints. IEEE TEVC 18, 4 (2014), 577–601.
[7] J. E. Fieldsend, R. M. Everson, and S. Singh. 2003. Using unconstrained elitearchives for multiobjective optimization. IEEE TEVC 7, 3 (2003), 305–323.
[8] R. H. Gómez and C. A. Coello Coello. 2015. Improved Metaheuristic Based on
the R2 Indicator for Many-Objective Optimization. In GECCO. 679–686.

Ryoji Tanabe and Akira Oyama[9] D. Hadka and P. M. Reed. 2012. Diagnostic Assessment of Search Controls andFailure Modes in Many-Objective Evolutionary Optimization. Evol. Comput. 20,
3 (2012), 423–452.
[10] T. Hanne. 1999. On the convergence of multiobjective evolutionary algorithms.
EJOR 117, 3 (1999), 553–564.
[11] S. Huband, P. Hingston, L. Barone, and R. L. While. 2006. A review of multiobjective test problems and a scalable test problem toolkit. IEEE TEVC 10, 5 (2006),
477–506.
[12] H. Ishibuchi, N. Akedo, and Y. Nojima. 2015. Behavior of Multiobjective Evolutionary Algorithms on Many-Objective Knapsack Problems. IEEE TEVC 19, 2(2015), 264–283.
[13] H. Ishibuchi, Y. Sakane, N. Tsukamoto, and Y. Nojima. 2009. Selecting a small
number of representative non-dominated solutions by a hypervolume-based
solution selection approach. In FUZZ-IEEE. 1609–1614.
[14] H. Ishibuchi, Y. Setoguchi, H. Masuda, and Y. Nojima. 2017. Performance ofDecomposition-Based Many-Objective Algorithms Strongly Depends on Pareto
Front Shapes. IEEE TEVC 21, 2 (2017), 169–190.
[15] Y. Jin. 2011. Surrogate-assisted evolutionary computation: Recent advances andfuture challenges. Swarm and Evol. Comput. 1, 2 (2011), 61–70.
[16] B. Li, J. Li, K. Tang, and X. Yao. 2015. Many-Objective Evolutionary Algorithms:
A Survey. ACM C. S. 48, 1 (2015), 13:1–13:35.
[17] B. Li, K. Tang, J. Li, and X. Yao. 2016. Stochastic Ranking Algorithm for ManyObjective Optimization Based on Multiple Indicators. IEEE TEVC 20, 6 (2016),
924–938.
[18] H. Li and Q. Zhang. 2009. Multiobjective Optimization Problems With Complicated Pareto Sets, MOEA/D and NSGA-II. IEEE TEVC 13, 2 (2009), 284–302.
[19] K. Li, K. Deb, Q. Zhang, and S. Kwong. 2015. An Evolutionary Many-Objective
Optimization Algorithm Based on Dominance and Decomposition. IEEE TEVC
19, 5 (2015), 694–716.
[20] K. Li, Q. Zhang, S. Kwong, M. Li, and R. Wang. 2014. Stable Matching-Based
Selection in Evolutionary Multiobjective Optimization. IEEE TEVC 18, 6 (2014),
909–923.
[21] M. Li, S. Yang, and X. Liu. 2014. Shift-Based Density Estimation for Pareto-Based
Algorithms in Many-Objective Optimization. IEEE TEVC 18, 3 (2014), 348–365.
[22] M. Li, S. Yang, and X. Liu. 2015. Bi-goal evolution for many-objective optimization
problems. Artif. Intell. 228 (2015), 45–65.
[23] M. Li, S. Yang, X. Liu, and R. Shen. 2013. A Comparative Study on Evolutionary
Algorithms for Many-Objective Optimization. In EMO. 261–275.
[24] M. López-Ibáñez, J. D. Knowles, and M. Laumanns. 2011. On Sequential Online
Archiving of Objective Vectors. In EMO. 46–60.
[25] J. Maltese, B. M. Ombuki-Berman, and A. P. Engelbrecht. 2017. A Scalability
Study of Many-Objective Optimization Algorithms. IEEE TEVC (2017), (in press).
[26] A. J. Nebro, J. José Durillo, C. A. Coello Coello, F. Luna, and E. Alba. 2008. A Studyof Convergence Speed in Multi-objective Metaheuristics. In PPSN. 763–772.
[27] A. Radulescu, M. López-Ibáñez, and T. Stützle. 2013. Automatically Improving
the Anytime Behaviour of Multiobjective Evolutionary Algorithms. In EMO.
825–840.
[28] R. Tanabe and A. Oyama. 2017. The Impact of Population Size, Number ofChildren, and Number of Reference Points on the Performance of NSGA-III. In
EMO. 606–621.
[29] T. Tusar, D. Brockhoff, N. Hansen, and A. Auger. 2016. COCO: The Bi-objective
Black Box Optimization Benchmarking (bbob-biobj) Test Suite. CoRR (2016).
[30] T. Wagner, N. Beume, and B. Naujoks. 2007. Pareto-, Aggregation-, and IndicatorBased Methods in Many-Objective Optimization. In EMO. 742–756.
[31] H. Wang, L. Jiao, and X. Yao. 2015. Two Arch2: An Improved Two-Archive
Algorithm for Many-Objective Optimization. IEEE TEVC 19, 4 (2015), 524–541.
[32] Y. Xiang, Y. Zhou, M. Li, and Z. Chen. 2017. A Vector Angle-Based Evolutionary
Algorithm for Unconstrained Many-Objective Optimization. IEEE TEVC 21, 1(2017), 131–152.
[33] S. Yang, M. Li, X. Liu, and J. Zheng. 2013. A Grid-Based Evolutionary Algorithm
for Many-Objective Optimization. IEEE TEVC 17, 5 (2013), 721–736.
[34] Y. Yuan, H. Xu, B. Wang, and X. Yao. 2016. A New Dominance Relation-Based
Evolutionary Algorithm for Many-Objective Optimization. IEEE TEVC 20, 1(2016), 16–37.
[35] Y. Yuan, H. Xu, B. Wang, B. Zhang, and X. Yao. 2016. Balancing Convergenceand Diversity in Decomposition-Based Many-Objective Optimizers. IEEE TEVC
20, 2 (2016), 180–198.
[36] Q. Zhang and H. Li. 2007. MOEA/D: A Multiobjective Evolutionary Algorithm
Based on Decomposition. IEEE TEVC 11, 6 (2007), 712–731.
[37] Q. Zhang, W. Liu, and H. Li. 2009. The performance of a new version of MOEA/Don CEC09 unconstrained MOP test instances. In IEEE CEC. 203–208.
[38] Q. Zhang, A. Zhou, S. Zhao, P. N. Suganthan, W. Liu, and S. Tiwari. 2008. Multiobjective optimization test instances for the CEC 2009 special session and competition.
Technical Report. Univ. of Essex.
[39] E. Zitzler and S. Künzli. 2004. Indicator-Based Selection in Multiobjective Search.
In PPSN. 832–842.
[40] E. Zitzler, M. Laumanns, and L. Thiele. 2001. SPEA2: Improving the Strength
Pareto Evolutionary Algorithm. Technical Report. ETHZ.