## 5 Experiment
### Performance Metrics  
Three popular metrics were employed to evaluate the performance of the proposed method for MOJSSP: Relative Error (RE), Coverage of two sets (Cov) [39], and Spacing [40]. They can be expressed as follows:  
1. Relative Error (RE) Extending the method of [PERSON] and [PERSON] [41], we analyzed the performance of the two acquired objectives using the relative error (RE) metric. The formulation is as follows: \[RE=\frac{\overline{F}-F_{\text{best}}}{F_{\text{best}}}\times 100\] (7) where \(\overline{F}\) is the mean value of makespan or total tardiness and \(F_{\text{best}}\) is the best makespan or total tardiness obtained among the iterations.
2. Coverage of Two Sets (Cov) This indicator was used to measure the dominance between two sets of solutions. The definition is as follows: \[Cov(X,Y)=\frac{\left|\left|y\in Y;\exists x\in X:x<=y\right|\right|}{\left|Y \right|}\] (8)  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**IP** & **IT** & & **NSGA-II** & **NSGA-II + SA** \\ \hline \multirow{4}{*}{25} & \multirow{2}{*}{100} & best & 933 & 853 \\  & & average & 974 & 909 \\ \cline{2-5}  & & \multirow{2}{*}{300} & best & 895 & 848 \\  & & average & 947 & 910 \\ \hline \multirow{4}{*}{50} & \multirow{2}{*}{100} & best & 895 & 854 \\  & & average & 941 & 912 \\ \cline{2-5}  & & \multirow{2}{*}{300} & best & 865 & 848 \\  & & average & 931 & 915 \\ \hline \multirow{4}{*}{100} & \multirow{2}{*}{100} & best & 898 & 848 \\  & & average & 938 & 919 \\ \cline{2-5}  & & \multirow{2}{*}{300} & best & 861 & 848 \\  & & average & 912 & 943 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Makespan of the NSGA-II and NSGA-II + SA.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**IP** & **IT** & & **NSGA-II** & **NSGA-II + SA** \\ \hline \multirow{4}{*}{25} & \multirow{2}{*}{100} & best & \(-\)3860 & \(-\)4529 \\  & & average & \(-\)3350 & \(-\)4338 \\ \cline{2-5}  & & \multirow{2}{*}{300} & best & \(-\)4001 & \(-\)4524 \\  & & average & \(-\)3757 & \(-\)4537 \\ \hline \multirow{4}{*}{50} & \multirow{2}{*}{100} & best & \(-\)4241 & \(-\)4630 \\  & & average & \(-\)3743 & \(-\)4400 \\ \cline{2-5}  & & \multirow{2}{*}{300} & best & \(-\)4279 & \(-\)4815 \\  & & average & \(-\)3982 & \(-\)4569 \\ \hline \multirow{4}{*}{100} & \multirow{2}{*}{100} & best & \(-\)4273 & \(-\)4698 \\  & & average & \(-\)3703 & \(-\)4495 \\ \cline{1-1} \cline{2-5}  & & \multirow{2}{*}{300} & best & \(-\)4539 & \(-\)4698 \\ \cline{1-1}  & & average & \(-\)4229 & \(-\)4587 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Total tardiness of the NSGA-II and NSGA-II + SA.  
where \(X\) and \(Y\) are two non-dominated sets to be compared. \(x\) and \(y\) are subsets of \(X\) and \(Y\), respectively. \"\(x<=y\)\" represents \(x\) dominates \(y\) or \(x=y\). The value \(Cov(X,Y)=1\) means that all solutions in \(Y\) are dominated by or equal to solutions in \(X\). The opposite, \(Cov(X,Y)=0\) represents that no solutions in \(Y\) are dominated by the set \(X\). It should be mentioned that there is not necessarily a relationship between \(Cov(X,Y)\) and \(Cov(Y,X)\). Therefore, these two values should be calculated independently. \(Cov(X,Y)>Cov(Y,X)\) means the set \(X\) is better than the set \(Y\).
3. Spacing It measures the standard deviation of the minimum distance from each solution to other solutions. The smaller the Spacing value is, the more uniform the solution set is. The expression is as follows: \[Spacing=\sqrt{\frac{1}{n-1}{\sum\limits_{1}^{n}{(\overline{d}-d_{i})}^{2}}}\] (9) where \(n\) is the number of solutions in the obtained Pareto frontier, \(d_{i}\) is the minimum distance between the solution and its nearest solution, and \(\overline{d}\) is the average value of all \(d_{i}\).