# Efficient Multi-Objective Simulation Metamodeling for Researchers
## 2 Test Environment and Methods  
The primary feature of the test environment is to enable researchers to run experiments on discrete multiple objective optimization problems using a simulation model and metamodel. These experiments evaluate the relative performance of MO pairs on a time and solution quality basis. As many such experiments may need to be conducted and the main goal of employing a metamodel is to economize on simulation model costs, the test environment utilizes a specific method for generating and storing simulation model Full Evaluations (FFs), or FE, for all future users. For each problem, corresponding to a simulation model, an exhaustive list of input-output data (including evaluation times) allows for faster-than-FE lookup, and reproducible FEs. In addition, this mechanism takes a stochastic simulation model and provides all future users with a deterministic response to any set of inputs, namely the mean of the simulation repetitions for that specific set of inputs.  
The components of the test environment interact as illustrated in Figure 1. Notably, while the external simulation models are evaluated on another platform (AnyLogic or SimOpt), users will use the FE record, meaning that only R needs to be actively executed for experiments. This record is separate from what the optimizer has in \"memory\" for each experiment run, which only consists of FEs it has already requested and \"paid for\" in terms of time costs and any Quick Evaluations (QEs) it has made via the latest updated metamodel.  
The overall sequence followed to complete an experiment run is as follows:  
1. Optimizer requests FEs;
2. Requested FEs are delivered to memory;
3. All FEs in memory are used to update the metamodel;
4. Optimizer updates its current Pareto front, using all FEs in memory;
5. Optimizer requests QEs from metamodel (this is an exploration step);
6. Metamodel delivers QEs to the optimizer;
7. If any of the QEs would be part of the Pareto front, they will be listed and requested by returning to step 1.  
For each experiment run, the first time step 1 occurs, the FEs requested correspond to the initial sample. Step 4 can be considered the exploitation step each time it occurs except the first time, as this is when the Pareto front actually improves. Although QEs are used to identify potential additions to the Pareto front, the objective values from a QE must be \"confirmed\" with FEs for those inputs.  
If an experiment run reaches the time limit while going through a list of requested FEs, only the FEs that would have been done before the time limit can be part of the final Pareto front. Otherwise, reaching the time limit means that the most recent Pareto front is the one that is stored and evaluated.  
For each problem investigated, a set of experiments is run on the simulation model, using different combinations of metamodel, optimizer and time limits. Each experiment run produces a Pareto front as output, which will be stored for later use. Once the experiment is completed, all Pareto fronts can be evaluated using the metrics detailed below. This allows the user to compare the evolution of solution quality against the time limit for each MO pair.