## 4 Experimental Study
### Benchmark and Comparison Methodology
#### 4.2.2 Multi-Objective Algorithms  
To perform this experimental study, we selected three well-known algorithms that are representative of the three categories of multi-objective evolutionary algorithms: NSGA-II [23] (Pareto-based), MOEA/D [24] (decomposition-based), and SMS-EMOA [25] (indicator-based). We remark that the purpose of this experimental study was to expose how to properly conduct it using representative techniques, not to try find the best algorithm, nor the most accurate configuration of the three we selected.  
All of them were configured with a population size of 50 individuals and a stopping condition of computing 300,000 function evaluations. They shared the same crossover and mutation operators (see Table 3 below). MOEA/D was set with a neighbor selection probability of 0.9, a neighbor size of 20, a maximum number of replaced solutions of 2, and a Tschebyscheff aggregation scheme.  
We implemented a crossover operator for integer encoding that works as follows: after selecting two parent solutions, an array of the solution (makespan, velocity, idle) is chosen randomly. After that, two random positions are selected again randomly, and they are interchanged between the two parents, leading to two children. The crossover probability was set to 90%.  
The mutation operator works as follows: first, as in the crossover, a random selection is performed between one of the three arrays composing the encoded solution (makespan, velocity and idle); second, a random interchange position is performed between two positions. The mutation probability was set to 100% (i.e., we always applied the mutation operator).  
Should the goal of the experimental study be the determination of which algorithm performs best for every problem instance, these parameter values should be tuned by resorting to any of the parameter tuning strategies known in the field of metaheuristics (e.g., Bayesian optimization, extensive search over a grid of values, and other similar techniques [105]). However, since the purpose of this experimental study was to exemplify how to develop a principled performance comparison conforming to standards and good practices, such parameters were left to their default values set in the reference implementation of every algorithm in jMetal. Table 3 summarizes the values of the parameters for every algorithm.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multicolumn{2}{c}{**NSGA-II**} & \multicolumn{2}{c}{**MOEA/D**} & \multicolumn{2}{c}{**SMS-EMOA**} \\ \hline
**Parameter** & **Value** & **Parameter** & **Value** & **Parameter** & **Value** \\ \hline Selection & Binary tournament & Neighbor selection probability & 0.9 & Selection & Random \\  & & Neighbor size & 20 & & \\  & & Maximum number of replaced solutions & 2 & & \\  & & Aggregation scheme & Tschebyscheff & & \\ \hline \multicolumn{2}{l}{_Common Parameters:_} & & & \\ Population size: 50 individuals & & & & \\ Number of function evaluations: \(3\times 10^{5}\) & & & & \\ \multicolumn{2}{l}{Crossover: 2-parent position exchange in uniformly selected array of the chromosome} & & & \\ \multicolumn{2}{l}{Crossover probability: 0.9} & & & \\ Mutation random exchange in uniformly selected array of the chromosome & & & \\ Mutation probability: 1.0 & & & \\ \hline \hline \end{tabular}
\end{table}
Table 3: Summary of the operators and parameter values set in the multi-objective algorithms considered in our experiments.  
Figure 7: Example of an encoded solution to the energy-efficient JSP addressed in the experimental part of this survey.