# Efficient Multi-Objective Simulation Metamodeling for Researchers
## 5 Discussion  
In the experimental investigation of four simulation optimization problems, it turned out that two of them (Student Services and Dual Sourcing) can be considered straightforward in that all the MO pairs evaluated performed similarly and produced high-quality Pareto fronts. In the other two (Telecom and Continuous News Vendor), there was more of a separation of quality with different patterns between the MO pairs evaluated, which comes from the way in which the metamodels estimate objective values for the inputs being explored by the optimizers. One can form more intuitive relationships from MO pair performance patterns to response surface properties than starting from the simulation components or domains, as there are only a limited number of MO pair performance patterns. Ideally, even a problem with a computationally intensive simulation model of interest would have well-behaved response surfaces, allowing a metamodel to identify areas of interest quickly and with confidence. This would enable the use of expensive FEs more sparingly, focusing on those areas of interest identified by the metamodel to maximize \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Simulation** & **MO Time (min)** & **FE Time (min)** & **FE Record Time** & **FE Record Size** \\  & & & **(min)** & **(KB)** \\ \hline Student Services & 352 & 672 & 3.6 & 13 \\ Telecom & 205 & 557 & 3.7 & 21 \\ Continuous & 462 & 1541 & 24.2 & 13 \\ News Vendor & 337 & 3981 & 10.7 & 40 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Experimental computational costs. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Case Study** & **Best MO Pair (s)** & \(n_{best}\) \\ \hline Student Services & NN-B \& NN-GA & 3 \\ Telecom & GR-B \& NN-B & 3 \\ Continuous News Vendor & GR-B & 4 \\ Dual Sourcing & GR-B & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Best-performing MO pairs per case study. the precision of the optimization. However, being \"well-behaved\" is not necessarily a simple property that would hold for all MO pairs that might be employed. As seen, some MO pairs may perform better than others for certain problems of interest. Additionally, multi-domain problems may or may not behave similarly to the problems of any particular component domain [40]. The application of metamodeling to ABS is currently a topic of great interest and ongoing research. In the Student Services case study, we showed that despite the agent-based behavior of the model, it was still able to be solved to a high standard across all evaluated MO pairs. In fact, for this problem, the agents' decision-making did not lead to a complex response surface. In contrast, a more challenging problem like the one investigated in the Continuous News Vendor case study has a simple simulation model but a response surface that is not equally responsive to different MO pairs. It is important to note that the FE times are based on a fixed record. However, the time limit stopping condition treats a second of FE time as equivalent to a second of metamodel or optimizer time. Therefore, if one runs experiments on problems where records were generated on a slower system, the faster system will always obtain better quality solutions within the same time limit, as the faster system has more computational resources. It will be possible to replicate previous results by creating a different version of the record with scaled decreased FE times. This will better match what the faster system is experiencing. Additionally, a smaller time limit can be used in the same proportion. This approach still allows the user to use records, but avoids having to set up numerous simulation models. While handy for experimental work like that detailed here, the downside of using an FE recording approach is that it does introduce an issue if there is a desire to create records for simulation models with a large input space due to memory constraints. Furthermore, referencing the records will take longer as they grow in size. This can lead to a memory-computation cost tradeoff.