# Efficient Multi-Objective Simulation Metamodeling for Researchers
## 4 Results
### Dual Sourcing Simulation Case Study  
As mentioned in Section 3.1.4, the goal of this case study is to minimize costs and penalties for a manufacturer present in multiple locations, involving procurement from regular or expedited suppliers with varied costs and delays. The Dual Sourcing experiments provide an important contrast with those of the Continuous News Vendor, as the proportion of total input combinations that would be on the Pareto front is much higher (861 of 1681). While solution Pareto fronts have between 383 to 694 points across all experiments, the hypervolume evaluation plot in Figure 18 shows that all MO pairs perform very well, similar to what was observed in the Student Services experiments. Table 4 summarizes the results from the Dual Sourcing experiments based on the mean normalized hypervolume performance (\(\mu_{norm}\)s) of each MO pair with the best-performing one at each time limit. In all time limits (240 s, 300 s, 360 s and 420 s), GR-B outperforms the rest of the MO pairs, and this performance variation is statistically significant with \(p\)-values  
Figure 16: Continuous News Vendor experiments—optimal and best NN-GA Pareto front. Figure 17: Continuous News Vendor experiments—crowding distance evaluation—time series. less than 0.05. Regardless of the given time limit, all methods produce a high \(\mu_{norm}\) over 0.99. The \(\mu_{norm}\)s produced by GR-B increase slightly from 0.9954 to 0.9997 even if the time limit is increased by a factor of 1.75. However, the Hausdorff distance evaluation plotted in Figure 19 suggests a large relative separation of solution qualities, which is not observed when considering the normalized hypervolumes. As the Hausdorff distances are normalized, an outlier with a high pre-normalized value will compress the distribution of normalized values towards 0. In fact, the Student Services experiments may exhibit the compression of normalized values issue the most, as the highest plotted point in the Hausdorff distance evaluation plot in Figure 5 which corresponds to the upper confidence interval for NN-GA at the 50 s time limit is close to 0.3, in contrast to the highest plotted point in the Hausdorff distance evaluation for the Dual Sourcing case which is 0.8. The definition of the Hausdorff distance can lead to it being very sensitive to the possibility of a \"very bad\" solution that is part of a solution Pareto front leading to a very high distance, even though the corresponding point in objective space would not greatly affect the front's measured hypervolume. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Time Limit (s)** & **MO Pair** & \(\mu_{norm}\) & \(\sigma\) & \(p\)**-Value** \\ \hline
240 & GR-B & 0.995415 & \(6.93\times 10^{-4}\) & Best \\  & GR-GA & 0.993214 & \(7.18\times 10^{-4}\) & \(1.10\times 10^{-4}\) \\  & NN-B & 0.992700 & \(5.32\times 10^{-4}\) & \(3.49\times 10^{-4}\) \\  & NN-GA & 0.991999 & \(1.08\times 10^{-3}\) & \(6.89\times 10^{-5}\) \\ \hline
300 & GR-B & 0.997616 & \(2.22\times 10^{-4}\) & Best \\  & GR-GA & 0.995695 & \(3.17\times 10^{-4}\) & \(8.17\times 10^{-5}\) \\  & NN-B & 0.994081 & \(5.68\times 10^{-4}\) & \(8.71\times 10^{-7}\) \\  & NN-GA & 0.994431 & \(3.89\times 10^{-4}\) & \(2.18\times 10^{-7}\) \\ \hline
360 & GR-B & 0.998792 & \(1.51\times 10^{-4}\) & Best \\  & GR-GA & 0.995610 & \(7.16\times 10^{-4}\) & \(6.25\times 10^{-8}\) \\  & NN-B & 0.994595 & \(8.22\times 10^{-4}\) & \(9.88\times 10^{-8}\) \\  & NN-GA & 0.996224 & \(2.34\times 10^{-4}\) & \(1.82\times 10^{-8}\) \\ \hline
420 & GR-B & 0.999778 & \(7.88\times 10^{-5}\) & Best \\  & GR-GA & 0.997865 & \(2.35\times 10^{-4}\) & \(8.83\times 10^{-10}\) \\  & NN-B & 0.996043 & \(5.52\times 10^{-4}\) & \(9.75\times 10^{-9}\) \\  & NN-GA & 0.996886 & \(2.57\times 10^{-4}\) & \(7.44\times 10^{-9}\) \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Dual Sourcing experiments—normalized hypervolume performance comparison of each MO pair with the best-performing one at each time limit.