The Impact of Population Size, Number ofChildren, and Number of Reference Points on
The Performance of NSGA-III
Ryoji Tanabe and Akira Oyama
Institute of Space and Astronautical Science, Japan Aerospace Exploration Agency,
Kanagawa, Japan
tanabe@flab.isas.jaxa.jp, oyama@flab.isas.jaxa.jp

Abstract. We investigate the impact of three control parameters (thepopulation size µ, the number of children λ, and the number of reference points H) on the performance of Nondominated Sorting Genetic
Algorithm III (NSGA-III). In the past few years, many efficient MultiObjective Evolutionary Algorithms (MOEAs) for Many-Objective Optimization Problems (MaOPs) have been proposed, but their control parameters have been poorly analyzed. The recently proposed NSGA-III
is one of most promising MOEAs for MaOPs. It is widely believed that
NSGA-III is almost parameter-less and requires setting only one control
parameter (H), and the value of µ and λ can be set to µ = λ ≈ H as
described in the original NSGA-III paper. However, the experimental results in this paper show that suitable parameter settings of µ, λ, and H
values differ from each other as well as their widely used parameter settings. Also, the performance of NSGA-III significantly depends on them.
Thus, the usually used parameter settings of NSGA-III (i.e., µ = λ ≈ H)
might be unsuitable in many cases, and µ, λ, and H require a particularparameter tuning to realize the best performance of NSGA-III.

1

Introduction

A Multi-Objective continuous Optimization Problem (MOP) which frequently
appears in engineering problems can be formulated as follows:
minimize f (x) = (f1 (x), ..., fM (x))Tsubject to x ∈ S ⊆ RD

(1)

where, f : S → RM is an objective function vector which consists of M conflicting
objective functions, and RM is the objective function space. x = (x1 , ..., xD )T isD
a D-dimensional solution vector, and S = Πj=1
[aj , bj ] is the bound-constrained
search space, where aj ≤ xj ≤ bj for each variable index j ∈ {1, ..., D}.
We say that x1 dominates x2 and denote x1 ≺ x2 if and only if fi (x1 ) ≤fi (x2 ) for all i ∈ {1, ..., M } and fi (x1 ) < fi (x2 ) for at least one index i. x∗ is a
Pareto optimal solution if there exists no x ∈ S such that x ≺ x∗ . f (x∗ ) is also
called a Pareto optimal objective function vector. The set of all x∗ in S is thePareto optimal solution Set (PS), and the set of all f (x∗ ) is the Pareto Frontier (PF). The goal of MOPs is finding a set of well-distributed nondominatedsolutions which close to the PF in the objective function space.
A Multi-Objective Evolutionary Algorithm (MOEA) is one of most promising approaches for solving MOPs [14]. Since MOEAs use a set of individuals(solutions of a given problem) for the search, they can find good nondominatedsolutions in one single run. NSGA-II [5] proposed in the early 2000’s is a representative MOEA. However, NSGA-II performs relatively well on MOPs withM ≤ 3, but its performance significantly degrades on MOPs with M ≥ 4 [22,13],
where an MOP with M ≥ 4 is called a Many-Objective continuous Optimization Problem (MaOP). Since MaOPs frequently appear in real-world applications[14], the poor performance of MOEAs is a critical problem. Thus, in the past few
years, researchers in the Evolutionary Computation community have studied on
the development of MOEAs which can handle a large number of objectives [14].
To the best of our knowledge, while many efficient MOEAs for MaOPs havebeen proposed in previous work, their control parameters have been poorly analyzed. Almost all previous work only proposed a novel MOEA and evaluatedits performance on some benchmark problems, but analysis of their control parameters have been minimal. In particular, effects of the population size µ andthe number of children λ on the performance of MOEAs are poorly understood.
It is well-known that the performance of EAs and MOEAs significantly depends
on control parameter settings [8]. Therefore, users of MOEAs need to appropriately set their parameter values for a target application. General rules of
thumb for parameter settings (e.g., the population size should be set to about
20 on six-objective multimodal MOPs) help users with such tedious tasks. In
addition, since appropriate parameter settings might differ from each MOEA,
the performance comparison between MOEAs with inappropriate ones is unfair[10,12]. An analysis of the impact of control parameter settings is also usefulfor understanding the performance behavior of MOEAs and designing a novel,
efficient MOEA. For example, a small population size leads NSGA-II to the fast
convergence [3] and is effective for expensive optimization problems.
In this paper, we analyze the impact of three control parameters (the population size µ, the number of children λ, and the number of reference points H) on
the performance of Nondominated Sorting Genetic Algorithm III (NSGA-III) [6].
We exhaustively investigate the performance of NSGA-III with various parameter settings of µ, λ, and H on the WFG benchmarks [9] with M ∈ {2, 4, 6, 8}.
NSGA-III, one of most promising MOEAs for MaOPs, is an improved version
of NSGA-II [5] by replacing the crowding distance-based selection with the reference vectors-based niching selection. NSGA-III can be categorized into the(µ + λ)-MOEA framework. Therefore, NSGA-III essentially has the three control parameters (µ, λ, and H), except for settings of variation operators (e.g.,
the crossover probability of SBX), and µ, λ, and H can be set independently ofeach other. However, it is widely believed that NSGA-III is almost parameterless and requires setting only one control parameter (H), and the value of µ andλ can be set to µ = λ ≈ H as described in [6]. In fact, µ, λ, and H of NSGA-III

were set to same values as µ = λ ≈ H in almost all previous work (exceptions will be discussed in Section 3). Thus, the performance of NSGA-III with
µ 6= λ 6= H has never been examined while parameter studies of MOEAs areimportant for practical uses as well as algorithm designs as discussed above. In
this paper, we show that suitable parameter settings of µ, λ, and H values differ
from each other as well as widely used parameter settings, and the performanceof NSGA-III significantly depends on them.
This paper is organized as follows: Section 2 introduces the NSGA-III algorithm. We describe related work in Section 3. Section 4 describes experimental
settings, and Section 5 presents the experimental analysis of µ, λ, and H of
NSGA-III. Finally, Section 6 concludes this paper and discusses our future work.

2

NSGA-III

This section briefly introduces NSGA-III [6]. A NSGA-III population P t =i,t T{x1,t , ..., xµ,t } is a set of individuals xi,t = (xi,t1 , ..., xD ) , i ∈ {1, ..., µ}. Aft
ter the initialization of the population P , the following steps are repeated until
the termination criteria are met.
At the beginning of each iteration t, children Qt = {u1,t , ..., uλ,t } are generated by applying variation operators (e.g., crossover and mutation operai,t Ttors) to randomly selected individuals from P t , where ui,t = (ui,t1 , ..., uD ) ,ti ∈ {1, ..., λ}. After the generation of Q , µ individuals composing the population in the next iteration P t+1 are selected from a union Rt = P t ∪ Qt , where|Rt | = µ + λ. The nondominated sorting [5] is applied to Rt , and the individuals
in Rt are grouped as {F 1,t , F 2,t , ...} according to their nondomination levels.
Starting from i = 1, the individuals in F i,t is added to a new temporal population S t , and the index counter i is incremented as i = i + 1. This procedure
is repeated until the size of S t equals or exceeds µ. The index counter i at theend of the nondomination levels-based selection is recorded as l. When |S t | = µ,P t+1 = S t , and this environmental selection procedure is finished. Otherwise(i.e., |S t | > µ), all individuals in S t \F l,t are added to P t+1 , and remaining K
individuals (K = µ− |P t+1 |) are selected from F l,t so that |P t+1 | = µ according
to the following reference vectors-based niching selection.
First, for each fi (i ∈ {1, ..., M }), for each x ∈ S t , the objective function
value fi (x) is normalized into [0, 1] by an originally defined normalization rule in[6]. After the normalization, the niching procedure with the reference vectors isapplied to eachPx ∈ S t . The reference vector consists of the reference point w =MT(w1 , ..., wM ) , j=1 wj = 1 and the ideal point which is identical to the origin(0, ..., 0)T . NSGA-III requires a set of the reference points W = {w1 , ..., wH }.
W can be generated by using any method, but it is desirable that W consistof uniformly distributed reference points when a user’s preference information
cannot be utilized for the search.
For each x ∈ S t \F l,t and each reference vector, the perpendicular distance
between the normalized objective function vector f 0 (x) and the reference vector

is calculated. Then, x is associated with a reference vector which has the minimum distance. For each wi (i ∈ {1, ..., H}), the number of associated individuals
is counted as ρi,t . After the association procedure of all members in S t \F l,t , theother members in F l,t ∈ S t are also associated in the same manner, but theniche count ρ is not counted at this time.
Here, K individuals in F l,t (K = µ − |P t+1 |) is selected for the population in
the next iteration P t+1 . First, wmin,t having the minimum niche count ρmin,t =minH
i=1 {ρi,t } is selected. When there are more than two such reference points,
one of them is randomly selected as wmin,t . Let I t be a set of the individuals
in F l,t associated with wmin,t . In this case, the following three situations (i)–(iii) can be considered: (i) |I t | = 0, (ii) |I t | ≥ 1 and ρmin,t = 0, and (iii)|I t | ≥ 1 and ρmin,t ≥ 1. In the case of (i), wmin,t is temporarily removed from
W at the iteration t. In the case of (ii), an individual having the minimum
perpendicular distance between the normalized objective function vector f 0 (x)and the reference vector is selected from I t and added to P t+1 . In the case of (iii),
a randomly selected individual in I t is added to P t+1 . In any of the situations(i)–(iii), the selected individual is removed from I t , and ρmin,t = ρmin,t + 1. The
niching procedure described above is repeatedly applied until |P t+1 | = µ.

3Previous parameter studies of NSGA-III

Here, we review previous parameter studies of three control parameters (thepopulation size µ, the number of children λ, and the number of reference points
H) of NSGA-III. In [6], Deb and Jain visually verified that NSGA-III withsmall µ and H values can find good nondominated solutions on three-objectivefunctions. Seada and Deb evaluated the convergence performance of U-NSGAIII with µ > H on two-objective MOPs [18], where U-NSGA-III is a unified
version of NSGA-III for single, multi, and many-objective optimization. The
experimental results in [18] show that the larger µ value than H is suitablefor U-NSGA-III when solving a multi-modal MOP. Andersson et. al. presented
a parameter tuning study of NSGA-II and NSGA-III [1]. µ (≈ H) and four
control parameters of the SBX crossover and polynomial mutation (i.e., thecrossover/mutation probability pc and pm , the distribution index ηc and ηm ) of
NSGA-II and NSGA-III were tuned using a meta-GA. However, the work [1]mainly focused on the latter four control parameters (pc , pm , ηc , and ηm ) of
the variation operators, and the discussion on the tuned µ (≈ H) value wasminimal. The experiments in the two previous work [18,1] were performed only
on two-objective MOPs, and the impact of µ, λ, and H was not investigated on
MaOPs with M ≥ 4. Also, λ was set as λ = µ in [6,18], and three parameters
were set as µ = λ ≈ H in [1].
Very recently, Ishibuchi et. al. [12] examined the performance of various
MOEAs, including NSGA-III, with various µ values on the three- and fiveobjective functions. Their experimental results show that the performance ofthe MOEAs significantly depends on the settings of µ, and a suitable µ value
differs from each MOEA. However, they set the three parameter values of NSGA-III as µ = λ ≈ H and did not examine the performance of NSGA-III (and theother MOEAs) with µ 6= λ 6= H in the same way as [1].
In summary, the performance of NSGA-III with different µ, λ, and H values
has been poorly understood. Note that the situation is not unique to NSGA-III.
Whereas there is much previous work on parameter studies of variation operators[21,25], the impact of µ and λ on the performance of MOEAs has been hardly
investigated. While there are several parameter studies on µ such as [11,23,17,3],
they set µ and λ to the same value, and the performance of MOEAs with µ 6= λhas been poorly discussed (the only one exception [7] will be discussed in Section
5.2).

4

Experimental settings

We used the nine WFG functions [9] with M ∈ {2, 4, 6, 8}. The shapes of thePFs of the WFG1, WFG2, and WFG3 functions are complicated, discontinuous, and degenerate respectively, and those of the others WFG functions arenonconvex. The WFG4 and WFG9 functions are multimodal, and the WFG2,
WFG6, WFG8, and WFG9 functions are also nonseparable. As suggested in [9],the position parameter k was set to k = 2(M − 1), and the distance parameterl was set to l = 20, where the number of variables D is D = k + l.
We used the source code of NSGA-III implemented by Yuan1 . We used the
SBX crossover and polynomial mutation as in the original NSGA-III [6]. As
suggested in [6], we set the control parameters of the variation operators as
follows: pc = 1.0, ηc = 30, pm = 1/D, ηm = 20. The number of independent runs was set to 31 for M ∈ {2, 4, 6} and only 11 for M = 8 due to
expensive computational cost of the hypervolume calculation. The number ofmaximum function evaluations was set to 1 × 104 as [2]. We set the population size µ, the number of children λ, and the number of reference points H
as follows: µ ∈ {10, 20, 40, 80, 160, 320}, λ ∈ {1, 10, 20, 40, 80, 160, 320, 640}, andH ∈ {10, 20, 40, 80, 160} (i.e., 6 × 8 × 5 = 240 configurations were verified).
We used the hypervolume (HV) indicator [28] for evaluating the quality ofa set of obtained nondominated solutions A2 . Before calculating the HV value(as well as the MinSum and SumMin values [13] used in Section 5.3), the objective function vector f (x) of each x ∈ A was normalized using the ideal
point (0, ..., 0)T and the nadir point (2, ..., 2M )T as in [26]. The reference point
for calculating HV was set to (1.1, ..., 1.1)T . Almost all previous work (e.g.,[27,2,25,18,26]) used nondominated solutions in the bounded archive (= thepopulation P ) for calculating the HV value. The bounded archive maintains(nondominated) solutions obtained during the search process, but its size is limited. As pointed out in [10], when using the bounded archive and the HV as
the performance indicator, the large µ values are more beneficial than the smallµ values. Furthermore, a monotonic increase of the hypervolume over time (=1
2

The code was downloaded from http://www.cs.bham.ac.uk/~xin/journal_papers.html
Since the IGD indicator [28] used in [6] is unsuitable for comparing nondominatedsolution sets of different size as pointed out in [10], we did not use it.

Algorithm 1: Generation method of a set of reference points W
tmp

1
2
3
4
5
6

Generate W tmp = {wtmp,1 , ..., wtmp,H} using SLD so that H tmp  H;W ← ∅, wc = (1/M, ..., 1/M )T , W ← W ∪ {wc };for i = 1 to H tmp do
di ← EuclideanDistance(wtmp,i , wc ), Vi ← T RU E;
while |W | < H dok ← arg max dj , Vk ← F ALSE, W ← W ∪ {wtmp,k };j|Vj =T RU E

7
8

for l = 1 to H tmp do
If Vl = T RU E, then dl ← min(EuclideanDistance(wtmp,l , wtmp,k ), dl );the number of function evaluations) cannot be ensured when using the boundedarchive [15,17]. Thus, we cannot perform a fair comparison of MOEAs withvarious µ values using the bounded archive. As suggested in [15,17,1,3,12], forthe fair comparison of NSGA-III with various µ values, we used the unboundedarchive that stores all nondominated solutions found during the search process.
As described in Section 2, NSGA-III requires a set of reference points W ={w1 , ..., wH }. In the original NSGA-III paper [6], Simplex-Lattice Design (SLD)[4] and Two-layered SLD (TSLD) [6] were used for generating W . SLD and
TSLD are the widely used weight/reference points generation methods, but theycannot generate W with arbitrary size [20,24]. That is, they are not suitable forour scale-up studies of H and M . In addition to SLD and TSLD, generation
methods of W with arbitrary size were proposed in [27,20,16]. In this paper, weused a generation method of W inspired by the proposed methods in [27,16].
Algorithm 1 shows the procedure. First, a set of candidate reference pointstmp
W tmp = {wtmp,1 , ..., wtmp,H } are generated using SLD so that H tmp  H.
For each M ∈ {2, 4, 6, 8}, we set the division number of SLD to 105 , 83, 24, and
14 respectively (i.e., H tmp = 100 001, 102 340, 118 755, and 116 280). Then, after
adding a central point wc = (1/M, ..., 1/M )T to W , a reference point in W tmp
having the largest Euclidean distance value from ones already belonging to W is
selected and added to W (lines 2–4 and 5–8 in Algorithm 1 respectively). This
trial is repeated until the size of W is equal to H.

5

Experimental results and discussion

We analyze the impact of the parameter settings of H, λ, and µ on the performance of NSGA-III. The results are shown in Section 5.1, Section 5.2, andSection 5.3 respectively. We also discuss how much the performance of NSGA-III
can be improved by tuning the µ, λ, and H values in Section 5.4.
5.1

Influence of H and µ values on the performance of NSGA-III

Figure 1 shows the influence of the H values on the performance of NSGA-III
with the various µ values on the WFG1, WFG2, and WFG4 functions with

M ∈ {2, 4, 6, 8} at the number of function evaluations = 1 × 104 . We can seethat the choice of µ and H values significantly affects the performance of NSGAIII, and their appropriate values also depend on the function, M , and the otherparameter values. Due to space constraints, we show only the results on theselected functions, but the results on the WFG3, 7, 9 functions and the WFG5,6, 8 functions are similar to the WFG1 and WFG4 functions respectively. Notethat some HV values in Figure 1 exceed 1 since the reference point for the HV
calculation was set to (1.1, ..., 1.1)T .
First, we discuss the impact of the µ values on the performance of NSGAIII. NSGA-III with µ ∈ {40, 80} and {80, 160} achieves the relatively higher HVvalues for the WFG1 and WFG2 functions respectively. Large µ values might
be suitable for NSGA-III when solving a MOP with the disconnected PF (e.g.,
WFG2 function). The results on the multimodal WFG4 function with M ∈ {2, 4}
is counterintuitive – NSGA-III with µ = 10 performs best. With an increasing
M , an appropriate µ value is also increased as µ ∈ {20, 40} for M = 6 andµ ∈ {40, 80} for M = 8. The reason for this is simply that the objective function
space is increasing according to M , and large µ values become suitable for NSGAIII. The reason why NSGA-III with the smallest population size µ = 10 performs
well on the WFG4 function with M ∈ {2, 4} can be considered as follows: TheWFG4 function is multimodal, but its shape of the PF is not so complex such asthat of the WFG1 and WFG2 functions. Therefore, NSGA-III with the smallestµ value can find good nondominated solutions. NSGA-III with µ = 320 performs
poorly in many cases but outperforms NSGA-III using the small µ values when
M increases. Therefore, large µ values might be suitable for solving MaOPs witha very large M (i.e., M ≥ 10).
Then, we analyze the impact of the H values on the performance of NSGA-III.
As shown in Figure 1, for the WFG functions with M ∈ {2, 4}, the increase of the
H values deteriorates the performance of NSGA-III with the small population
size µ ∈ {10, 20} while it does not significantly influence the performance of
NSGA-III with the large µ values. On the other hand, for the WFG functions
with M ∈ {6, 8}, there exists a particular H value which is suitable for NSGA-III
with each µ value. For example, NSGA-III with µ = 80 achieves the highest HVvalues on the eight-objective WFG1, WFG2 and WFG4 functions when using
H = 160. Since large H values increase the number of niches in the environmental
selection and promote diversity in the population (see Section 2), they might
be suitable for NSGA-III when solving MaOPs as shown Figure 1. Seada andDeb [18] reports that the larger µ value than H is suitable for U-NSGA-III
when solving a two-objective multimodal MOP (see Section 3). Although theexamined version of NSGA-III is slightly different from [18], the experimental
results of this study show that the use of the smaller µ value than H can improve
the performance of NSGA-III on MaOPs with M ≥ 6.
5.2

Influence of λ and µ values on the performance of NSGA-III

Figure 2 shows the influence of the λ values on the performance of NSGA-III withthe various µ values. For M = 2, the large λ values deteriorate the performance

10
0.20
0.18
0.16
0.14
0.12
0.10
0.08
0.06
0.04
0.02

1
10
80
1

0.25
0.20

WFG1 (M =2)
20
20
10

20

1

10
1

10
10

20
10
20

10
11
40
160

80
20
40
10

1

1
160

0.15

20
10

10
40

0.30

20

11
40

10
1
20
10
80

WFG1 (M =4)
160
1
20
160
1

1

320

80

HV values

0.10
0.05

10

0.35

0.30
0.20

1
1
40
320

0.15

160

0.25

0.10
0.05
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10

80
10

1
10
40

20

40

80

1
40
1
160
1

10
1
10
10
1

10
1
40

WFG1 (M =6)

320
20

160

40

40
160
10
20

1
10
160
80

320
20
10

20
20

10
80
10
160
20
160
40

160

1
20
20
20
20
20
160

1

80

1
1
10
11
10

80

160

10
10

10
1

WFG1 (M =8)
20

0.73
0.72
0.71
0.70
1 0.69
10
0.68
0.67
0.66
1 0.6520
10 0.64

1

10
10
160

1
10
160

1.35
1.30
1.25
1.20
1.15
1.10
1.05
1.00
0.95
1.60
1.55
1.50
1.45
1.40
1.35
1.30
1.25
1.20
1.15

80

80
160

40
20
40
160
10
160

1.9
1.8
1.7
1.5
1.4

80

WFG2
(M =2) 10
1
10
160
160
20

10
20
80

320
20

160

40
160

40
40 0.42
160

320
10

320
10 0.40

0.41

160

0.39

160
640

40

0.37

20

40

80

160

1
11

20
1
320

20
40
1

10
10
320

10 1.00
10320

WFG2 (M =4)

640
80
160
160

160
10
40

10
10

1
1

20

1.05

0.95
0.90
640
0.85
0.80
10
0.75
1 0.70
0.65

10

20

40

80

160

1
320
10

40
1

1
320
20

1
80
10

1.4

10
1
320
640

640

1.2

80
160
10

160
40
1
80
320
640
10

WFG2 (M =6)

640
160
160
20

320
20
20
40

320
1
160
80

WFG2 (M =8)
1
160
10

10
640
10

10
160
160

40
20
160

40
1
640
10
1

80
160
320

80
160
160

320
640
40

20

40

80

320
160

320
20
20
10
1
1

WFG4 (M =2)
20
10
1
1
1
80

0.9

1.6
1.5
1.4
1.3
1.2
1.1
1.0
0.9

Number of reference points H

160
10

1
80

20

40

80
20
10
40
1
1

20
20
20
1
160
1

20
1
11
40

80
80

WFG4 (M =4)

1
40
10
10
160

160
10

1.1

0.8

20
20
1

10

1.3

1.0

10
40
1

80

0.38

10

2.0

1.6

320

40

20

40

160
160
20
160
20

80
10
80
10

160
80

WFG4 (M =6)

40

1
20
1
160
1

20
320

640
10
320
160
40
320
10

11
40
10
40
1
160

10
1
10
320
640
160

1
10
1
10
320

320
20

40

80

WFG4 (M =8)

320
640
320
40
160

320
640
640
40
640

10

20

320
320
640
320
640
160

40

160
160
20
40

320
160
80
40
10
160
20

160
320
80

160

Fig. 1: Influence of the H values on the performance of NSGA-III with the various µ
values on the WFG1, WFG2, and WFG4 functions with M = {2, 4, 6, 8}. We show theresults of NSGA-III with the best configuration of µ, H, and λ values which achieved
the best median HV value at the number of function evaluations = 1×104 . The numbersin the figure represent the corresponding λ values.

of NSGA-III. This is consistent with the results reported in the previous work[7]. In [7], Durillo et. al. investigated the performance of the steady state version
of NSGA-II and SPEA2 (i.e., λ = 1) on two-objective MOPs, and the results
show their promising performance. This trend can also be seen in the results of
NSGA-III with the large µ values (µ ∈ {160, 320}) on the WFG functions withM ∈ {4, 6, 8}. On the other hand, for the WFG functions with M ∈ {6, 8}, there
is a particular λ value for each µ value which makes NSGA-III with the small µvalues (µ ∈ {10, 20, 40, 80}) efficient. For example, NSGA-III with µ = 10 and
λ = 80 performs best on the WFG4 function with M = 4.
In [7], the performance comparison of (µ + µ)-MOEAs and (µ + 1)-MOEAs
was performed only on MOPs with M = 2. Our results show that the larger
λ value than µ might be suitable for NSGA-III with small µ values on MaOPs

10
0.20

10
20
10

0.15
0.10

20

10
20
40
40

20

40

WFG1 (M =2)
20
20
10
10

20
10
10
20

10
10
10
10

160 80

10
10
10

80
20

0.72 401020800.70 20
80

80
20
20
80
10

0.68

WFG2
(M =2)
160
10
40
80
10

10
20 40
160
20
20

10
10 0.66
20
0.05
20 20
20 20
40 10 160 10 40
20
10 0.6410 10
0.00
1 10 20 40 80 160 320 640
1 10 20

HV values

80
0.28 160
0.26 10
0.24
10
0.22
0.20 40
0.18
0.16
10
0.14
0.12
1

80
10
40
20

10

WFG1 (M =4)
40
20
80
80
10

40
80
10
10
40

10
80 160 160
80
10
20

40

10

0.35 160
80

80
10
40
10

20

10
80
80

40

1.35
1.30
40
1.25
10
80
10 80 1.20
160
80 1.1520
10 1.1040 40
1.05
160 1.00

80 160 320 640

WFG1 (M =6)

80
40
10
10
20
10

40
40
40
10
10
40

160
40
20
10
10
80

0.42
10
20
40
80
0.41 4080 160 320 640

1

10

20

40

80

40

80
80

1

160
80
160
40
160
40

80
160
160
160
40
40
40 40 40
80
40 80
40
40
40
40
40

40

80 160 320 640

8040

160 40
1 10 20

80
40
40
160
80

1

2.0
1.9
1.8
1.7
1.6
1.5
1.4
1.3
1.2
1.1

40

10

10

20

40

80 160 320 640

20
20
80
40
160
10

20
40
40
40
10
160

20
20
40
10
160
160

10
20
40
80
160
160

10

20

10

20

40

80 160 320 640

80
80
80 40
80 40
80 160 160
80 160 80
80
160

40
40
160
40
160

40
80
20
160
80
20

1.05

40

80 160 320 640

0.75

WFG2 (M =6)

10

20

40

80 160 320 640

WFG2 (M =8)

1.6
160 40 160 160 160 160
1.5160
10 160160 10 80
20 16040 1.420 160 160
160
40
40 10
1.3
160
20
80
40 40 40
80 80 80160
1.2
160 80 160 160 10 40 80
80 40
20 10 1.1
80
1.0
80
0.9
80
0.8
0.7

1

10

20

40

80 160 320 640

Number of children λ

40
20
80
10
20
10

1

40 16040 1601.00
80 160 160 160 16020160
160 80 80
80 0.95 4040160
80 0.90 40
160 16010 40
40 10 10
160
10 40
10 40
10 20 1010
10 10 0.85 20
10 10 10 20
10 10 10
10 0.801.6

80 160 320 640

WFG1 (M =8)160
160 80 160160
160 40 40
160 160 40

40

10 20
20
80
40
10 40
80
20
20 1601080
40
40 160
20

0.37
0.36

1.4
160 160 160 160 160 160
40 160160
80 160 160
40 16040 80
80
80 80 1.3
160 16040
0.30 160 160
1.5 160 160808080
160 160
160 160 1.2
20 20 160
40 40
80
80
160
80
40 80 80 160 160
0.25 40 40 160
1.4 16040
40 160 1.1
160 160 40
160 160 160
40
40
20
20
80
1.0
20
80 80 40 20160
20
40 1.30.20
20 0.9160
40 40 40 10 10 100.8
0.15 160 160 160 160 80 40 80
1.2
0.7
40
160 40
0.10
1.1
0.6
0.45
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.05

10
20
40
40

10
20
40
10
40

0.38

WFG2 (M =4)

40
80
10

WFG4 (M =2)
10
20
80
40
40

20
10
0.40
20
20 0.39 10
10
20

40

320

1

WFG4 (M =4)

20
10
40
80
160
160

20
10
40
80
160
160

WFG4 (M =6)

160 40

40

1

20

10

20
10
40
80
160
160

20
40
40
20
160
80

40
20
80
20
160
80

40
80
20
160
20
80

40
40

80 160 320 640

WFG4 (M =8)160 160 160 160 80
160 160 160 160 160 160
80 80
160 160
160
160
160 160 160 160 160 160 160 40
80
80
160 40401608080
160 160160
160 160
80 80
80 40
80
40
1 10 20 40 80 160 320 640

Fig. 2: Influence of the λ values on the performance of NSGA-III with the various µ
values on the WFG1, WFG2, and WFG4 functions with M = {2, 4, 6, 8}. We show theresults of NSGA-III with the best configuration of µ, H, and λ values which achieved
the best median HV value at the number of function evaluations = 1×104 . The numbersin the figure represent the corresponding H values.

with M ≥ 4. Unfortunately, we cannot explain the reason why NSGA-III withλ > µ performs well on MaOPs. A further investigation of the impact of λ on
the performance of NSGA-III is our future work.
5.3

Convergence behavior of NSGA-III with various µ values

We here analyze the convergence behavior of NSGA-III with the various µ values.
For the analysis, in addition to the HV indicator, we also measured two convergence metrics (MinSum and SumMin [13]) and the number of nondominatedsolutions stored in the unbounded archive A3 . MinSum indicates only the con3

Since, as far as we know, there is no good diversity indicator for the unboundedarchive, we could not measure the diversity of the obtained nondominated solutions.

µ =10µ =20µ =40
WFG4 (M =4)

1.1
1.4
1.3
1.0
0.40
1.2
0.9
1.1
0.35
0.8
1.0
0.9
0.7
0.30
0.8
0.6
0.7
0.25
0.5
0.6
0.20
0.4
0.5
0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0WFG4 (M =2)
WFG4 (M =4)
1.25
1.40
1.45
1.40
1.35
1.20
1.35
1.30
1.30
1.25
1.15
1.25
1.20
1.20
1.10
1.15
1.15
1.10
1.10
1.05
1.05
1.05
1.00
1.00
1.00
0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0WFG4 (M =2)
WFG4 (M =4)
0.40
0.5
0.6
0.35
0.5
0.4
0.30
0.4
0.25
0.3
0.20
0.3
0.2
0.15
0.2
0.10
0.1
0.1
0.05
0.00
0.0
0.0
0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0WFG4 (M =2)
WFG4 (M =4)
6.0e2
6.0e3
9.0e3
8.0e3
5.0e2
5.0e3
7.0e3
4.0e2
4.0e3
6.0e3
5.0e3
3.0e2
3.0e3
4.0e3
2.0e2
2.0e3
3.0e3
2.0e3
1.0e2
1.0e3
1.0e3
0
0
0
0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0µ =80µ =160µ =320

WFG4 (M =6)

WFG4 (M =8)
1.6
1.4
1.2
1.0
0.8
0.6
0.4
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG4 (M =6)
WFG4 (M =8)
1.5
1.4
1.3
1.2
1.1
1.0
0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG4 (M =8)
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0.0
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4 0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG4 (M =6)
WFG4 (M =8)
1.0e4
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG4 (M =6)

Size of archive

SumMin values

MinSum values

HV values

0.45

WFG4 (M =2)

8.0e3
6.0e3
4.0e3
2.0e3
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4

Number of function evaluations0
0 2.0e3 4.0e3 6.0e3 8.0e3 1.0e4

Fig. 3: Convergence behavior of NSGA-III with the various µ values. The figures show
the HV, MinSum, SumMin, and |A| over the number of function evaluations on the
WFG4 function with M ∈ {2, 4, 6, 8}. We plot the median and quartile values of eachindicator across all runs. We show the data of NSGA-III with the best configuration ofµ, H, and λ values which achieved the best median HV value at the number of function
evaluations = 1 × 104 .

vergence quality of A to the center of theP
PF. The MinSum value of A is calcuMlated as follows: MinSum(A) = minx∈A
j=1 fj (x) . Also, SumMin evaluates
only the convergence performance of A toward the PF around
PM its M edges. The
SumMin value of A is obtained as follows: SumMin(A) = j=1 minx∈A {fj (x)}.
We also report the number of nondominated solutions stored in the unboundedarchive A.
Figure 3 shows the HV, MinSum, SumMin, and |A| over the number offunction evaluations on the WFG4 function with M ∈ {2, 4, 6, 8}. We here only
discuss the results on the WFG4 function, but note that the obtained results
highly depend on the function instance. The analysis of the convergence behaviorof NSGA-III on the other WFG functions is our future work.

Figure 3 shows that the large µ values make the convergence speed of NSGAIII slow for M ∈ {2, 4, 6}. For example, NSGA-III with µ = 320 achieves theworst HV value at any time, and its convergence speed towards the PF is slow asshown in the MinSum and SumMin values. The results are consistent with theprevious work by Brockhoff et. al. [3] which investigated the anytime behaviorof NSGA-II with various µ values on two-objective MOPs. NSGA-III with thesmall µ values (µ ∈ {10, 20}) show the good anytime behavior on the WFG4
function with M ∈ {2, 4} and the strong convergence to the PF (see the MinSum
and SumMin values). On the other hand, the performance of NSGA-III with
µ = 10 is poor on the WFG4 function with M ∈ {6, 8}, and µ = 80 seems tobe suitable. For M = 8, NSGA-III with µ = 10 performs worse than NSGA-III
with µ = 320. Although NSGA-III with µ = 40 achieves the best MinSum andSumMin value, its HV value is worst. This might be due to its low number ofobtained nondominated solutions |A|. In summary, the results show that largeµ values make the convergence speed of NSGA-III slow but might be suitablefor MaOPs with M ≥ 6.
5.4

Generalist vs. Specialist: How much can the performance of
NSGA-III be improved by tuning µ, λ, and H values?

So far in this paper, we have analyzed the impact of parameter settings on theperformance of NSGA-III for each WFG function instance. We here investigate
how much the performance of NSGA-III can be improved by tuning the µ, λ, andH values. We consider the following two parameter tuning scenarios: (1) finding
the parameter settings that robustly perform well on a given set of functions,and (2) finding ones that perform well only on a specific function. We say thatthe parameter settings obtained by the parameter tuning on the scenarios (1)and (2) are generalist and specialist respectively, where they were named by Smit
and Eiben in [19].
First, we deal with the parameter tuning scenario (1) for finding the generalist. We used the Average Performance Score (APS) [2]. Recall that we verified
the 240 configurations of µ, λ, and H in this paper (see Section 4). Suppose that
we compare the performance of n algorithms {A1 , ..., An } on a given problem
instance using the HV values obtained in multiple runs (n = 240 in this study).
For each i ∈ {1, ..., n} and j ∈ {1, ..., n}\{i}, let δi,j = 1, if Aj outperforms Ai
based on the Wilcoxon rank-sum test with p < 0.05, otherwise
Pn δi,j = 0. Then,the performance score P (Ai ) is defined as follows: P (Ai ) = j∈{1,...,n}\{i} δi,j .
P (Ai ) represents the number of compared algorithms that outperforms Ai . The
APS is an average of the P (Ai ) values on all problem instances.
Table 1 provides the combination of µ, λ, and H values which achieved thesmallest APS value for each M on all WFG functions, i.e., the generalist for each
M . We have no intention of claiming that NSGA-III with the generalist shown in
Table 1 performs best on all possible problems (it performs well only on the WFG
benchmarks). As discussed in [19], obtaining such “best” parameter settings is
impossible in practice. We can see that µ = 40 is relatively suitable for NSGA-III
for M ∈ {2, 4, 6}. On the other hand, NSGA-III with µ = 80 performs well onTable 1: Generalist for each M : Combination of (µ, λ, H) values that achieved thesmallest APS value for each M at the number of function evaluations = 1 × 104 .
M =2

M =4

M =6

M =8

(40, 10, 20) (40, 10, 40) (40, 80, 40) (80, 40, 80)
Table 2: Specialist for each WFG function with M = 8: Combination of (µ, λ, H) values
that achieved the highest median HV value for each eight-objective WFG function at
the number of function evaluations = 1 × 104 .
WFG1

WFG2

WFG3

WFG4

WFG5

WFG6

WFG7

WFG8

WFG9

(80, 10, 160) (160, 10, 40) (160, 80, 20) (80, 80, 160) (40, 80, 40) (40, 10, 40) (40, 80, 40) (40, 80, 40) (80, 20, 40)

the WFG functions with M = 8. The results also show that the appropriate H
values depend on M as discussed in Section 5.1. Unfortunately, general rules of
thumb for the appropriate parameter settings of λ cannot be found in Table 1.
Then, we deal with the parameter tuning scenario (2) for finding the specialist. Table 2 shows the combination of (µ, λ, H) values which achieved the best
median HV value for each eight-objective WFG function, i.e., the specialist foreach WFG function with M = 8. The results show that the appropriate parameter settings of µ, λ, and H significantly depend on a given function. When solving
the WFG1 and WFG4 functions, the large H value is suitable, but NSGA-III
with the small H value (H = 40) performs best on the other WFG functions,
except for the WFG3 function. The large µ value is also suitable for the WFG3
and WFG4 functions.
Although the parameter tuning is not the main concern of this paper, it isinteresting to compare the performance of NSGA-III with different parameter
configurations. Figure 4 shows the convergence behavior of NSGA-III with thegeneralist, specialist, and default parameter settings of (µ, λ, H) on the WFG
functions with M = 8. For the default parameter settings, we set µ = λ =H = 156 as suggested in the original NSGA-III paper [6]. We also show theresults of NSGA-III with default parameter settings using a set of reference
points generated by TSLD (see Section 4), i.e., the original NSGA-III. For thegeneralist and specialist, we used the parameter settings shown in Table 1 and
2 respectively.
Figure 4 shows that NSGA-III with the specialist clearly outperforms NSGAIII with the three other parameter settings on all WFG functions with M = 8.
NSGA-III with the generalist performs better than NSGA-III with the default
settings, except for the WFG2 and WFG3 functions. Since the shapes of the PFsof the WFG2 and WFG3 functions are complex, NSGA-III requires particularparameter settings (see Table 2). Due to this reason, we consider that NSGA-III
with the generalist did not work well on these WFG functions. We cannot find aclear winner on the comparison between NSGA-III with the default parametersettings using a set of reference points W generated by Algorithm 1 and TSLD

Generalist
0.5

WFG1 (M =8)

2.0

0.4

1.8

0.3

1.6

0.2

1.4

0.1

1.2

0.0
0
1.6

2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG4 (M =8)

1.0
0
1.8

Default

1.2
1.1
1.0
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG5 (M =8)

0.6
0
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0

0.6
0

1.5
1.4
1.3
1.2
1.1
1.0
0.9
0.8
0.7
0.6
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
0

2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG6 (M =8)

1.2
1.0
0.8

0.8
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG7 (M =8)

1.8
1.6

1.0

0.8

0.9
0

1.4

1.2
1.0

WFG3 (M =8)
1.3

1.4

1.2

Default-TSLDWFG2 (M =8)

1.6

1.4

HV values

Specialist

0.6
2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG8 (M =8)

0.4
0
1.6

2.0e3 4.0e3 6.0e3 8.0e3 1.0e4
WFG9 (M =8)

1.4
1.2
1.0
0.8
0.6
0.4
2.0e3 4.0e3 6.0e3 8.0e3 1.0e40Number of function evaluations

2.0e3 4.0e3 6.0e3 8.0e3 1.0e4

Fig. 4: Convergence behavior of NSGA-III with the generalist, specialist, and default
settings of (µ, λ, H) on the WFG functions with M = 8. We plot the median andquartile HV values across all 11 runs. For the default settings, we set µ = λ = H = 156
as suggested in the original NSGA-III paper [6]. We also show the results of NSGAIII with default settings using a set of reference points generated by TSLD, i.e., theoriginal NSGA-III. For the generalist and specialist, see Table 1 and 2 respectively.

(labeled as “Default” and “Default-TSLD” in Figure 4 respectively). A further
performance evaluation of reference points generation methods is an avenue for
future work. However, Algorithm 1 has an advantage compared to TSLD – itcan generate a set of reference points W with arbitrary size.
In summary, the results show that (i) the performance of NSGA-III can begreatly improved by tuning the parameter settings of µ, λ, and H individually,and (ii) their appropriate settings depend on the function instance.

6

Conclusion

We analyzed the impact of the population size µ, the number of children λ,and the number of reference points H on the performance of NSGA-III [6].

It was widely believed that NSGA-III was almost parameter-less and requires
setting only one control parameter (H), and the value of µ and λ could be set
to µ = λ ≈ H [6]. In contrast, our experimental results showed that appropriate
parameter settings of µ, λ, and H differ from each other as well as widely usedparameter settings. Also, the performance of NSGA-III significantly depends on
the settings of µ, λ, and H. In fact, NSGA-III with a carefully tuned parametersetting (i.e., the specialist) performs significantly better than NSGA-III withthe widely used settings. Thus, the usually used parameter settings of NSGA-III(i.e., µ = λ ≈ H) might be unsuitable in many cases, and NSGA-III requires aparticular parameter tuning to realize its best performance.
Future work will analyze the behavior of NSGA-III with different values ofµ, λ, and H. We will also investigate the relationship between good parameter
combinations of NSGA-III and the shape of the PF by using some statistical
approaches (e.g., main effect plots). While the focus of this study was limited
to the bound-constrained WFG functions, an analysis on constrained MOPs isan avenue for future work. We believe that the effect of the parameter settingsof µ, λ, and H is applicable to other reference vectors-based MOEAs such as
U-NSGA-III [18], θ-DEA [26], and VaEA [24], and its investigation is also our
future direction.
Acknowledgments. This research is supported by the HPCI System Research
Project “Research and development of multiobjective design exploration andhigh-performance computing technologies for design innovation” (Project ID:hp160203).

References
1. M. Andersson, S. Bandaru, A. H. C. Ng, and A. Syberfeldt. Parameter Tuning of
MOEAs Using a Bilevel Optimization Approach. In EMO, pages 233–247, 2015.
2. J. Bader and E. Zitzler. HypE: An Algorithm for Fast Hypervolume-Based ManyObjective Optimization. Evol. Comput., 19(1):45–76, 2011.
3. D. Brockhoff, T. Tran, and N. Hansen. Benchmarking Numerical Multiobjective
Optimizers Revisited. In GECCO, pages 639–646, 2015.
4. I. Das and J. E. Dennis. Normal-Boundary Intersection: A New Method for Generating the Pareto Surface in Nonlinear Multicriteria Optimization Problems. SIAM
J. Optimiz, 8(3):631–657, 1998.
5. K. Deb, S. Agrawal, A. Pratap, and T. Meyarivan. A fast and elitist multiobjective
genetic algorithm: NSGA-II. IEEE TEVC, 6(2):182–197, 2002.
6. K. Deb and H. Jain. An evolutionary many-objective optimization algorithm usingreference-point-based nondominated sorting approach, part I: solving problems
with box constraints. IEEE TEVC, 18(4):577–601, 2014.
7. J. J. Durillo, A. J. Nebro, F. Luna, and E. Alba. On the Effect of the Steady-State
Selection Scheme in Multi-Objective Genetic Algorithms. In EMO, pages 183–197,
2009.
8. A. E. Eiben, R. Hinterding, and Z. Michalewicz. Parameter control in evolutionary
algorithms. IEEE TEVC, 3(2):124–141, 1999.
9. S. Huband, P. Hingston, L. Barone, and R.L.While. A review of multiobjective test
problems and a scalable test problem toolkit. IEEE TEVC, 10(5):477–506, 2006.

10. H. Ishibuchi, H. Masuda, and Y. Nojima. Comparing solution sets of different sizein evolutionary many-objective optimization. In IEEE CEC, pages 2859–2866,
2015.
11. H. Ishibuchi, Y. Sakane, N. Tsukamoto, and Y. Nojima. Evolutionary ManyObjective Optimization by NSGA-II and MOEA/D with Large Populations. In
IEEE SMC, pages 1758–1763, 2009.
12. H. Ishibuchi, Y. Setoguchi, H. Masuda, and Y. Nojima. How to compare manyobjective algorithms under different settings of population and archive sizes. In
IEEE CEC, pages 1149–1156, 2016.
13. H. Ishibuchi, N. Tsukamoto, and Y. Nojima. Evolutionary many-objective optimization: A short review. In IEEE CEC, pages 2419–2426, 2008.
14. B. Li, J. Li, K. Tang, and X. Yao. Many-Objective Evolutionary Algorithms: A
Survey. ACM Comput. Surv., 48(1):13, 2015.
15. M. López-Ibáñez, J. D. Knowles, and M. Laumanns. On Sequential Online Archiving of Objective Vectors. In EMO, pages 46–60, 2011.
16. S. Martı́nez, H. E. Aguirre, K. Tanaka, and C. A. C. Coello. On the low-discrepancy
sequences and their use in MOEA/D for high-dimensional objective spaces. In
IEEE CEC, pages 2835–2842, 2015.
17. A. Radulescu, M. López-Ibáñez, and T. Stützle. Automatically Improving theAnytime Behaviour of Multiobjective Evolutionary Algorithms. In EMO, pages
825–840, 2013.
18. H. Seada and K. Deb. A unified evolutionary optimization procedure for single,
multiple, and many objectives. IEEE TEVC, 2016 (in press).
19. S. K. Smit and A. E. Eiben. Parameter Tuning of Evolutionary Algorithms: Generalist vs. Specialist. In EvoApplications, pages 542–551, 2010.
20. Y. Tan, Y. Jiao, H. Li, and X. Wang. MOEA/D + uniform design: A new versionof MOEA/D for optimization problems with many objectives. Computers & OR,
40(6):1648–1660, 2013.
21. T. Tusar and B. Filipic. Differential Evolution versus Genetic Algorithms in Multiobjective Optimization. In EMO, pages 257–271, 2006.
22. T. Wagner, N. Beume, and B. Naujoks. Pareto-, Aggregation-, and Indicator-Based
Methods in Many-Objective Optimization. In EMO, pages 742–756, 2007.
23. S. Wessing, N. Beume, G. Rudolph, and B. Naujoks. Parameter Tuning Boosts
Performance of Variation Operators in Multiobjective Optimization. In PPSN,
pages 728–737, 2010.
24. Y. Xiang, Y. Zhou, M. Li, and Z. Chen. A vector angle based evolutionary algorithm
for unconstrained many-objective optimization. IEEE TEVC, 2016 (in press).
25. Y. Yuan, H. Xu, and B. Wang. An Experimental Investigation of Variation Operators in Reference-Point Based Many-Objective Optimization. In GECCO, pages
775–782, 2015.
26. Y. Yuan, H. Xu, B. Wang, and X. Yao. A new dominance relation-based evolutionary algorithm for many-objective optimization. IEEE TEVC, 20(1):16–37,
2016.
27. Q. Zhang, W. Liu, and H. Li. The performance of a new version of MOEA/D on
CEC09 unconstrained MOP test instances. In IEEE CEC, pages 203–208, 2009.
28. E. Zitzler, L. Thiele, M. Laumanns, C. M. Fonseca, and V. G. da Fonseca. Performance assessment of multiobjective optimizers: an analysis and review. IEEE
TEVC, 7(2):117–132, 2003.