# Evolutionary Statistical System Based on Novelty Search:
## 4 Experimentation and Results
### Results  
The results of the quality assessment are presented in Figures 7-11. For each of the five maps, a set of three related figures is presented. In each set of figures, a graphical representation of the fires at different time instants appears for reference (Figures 7a, 8a, 9a, 10a and 11a). The \(x\)- and \(y\)-axes represent the terrain in meters, while the colors show which areas are reached by the fire at different time steps. Below each map (in Figures 7b, 8b, 9b, 10b and 11b), the average fitness prediction values (over 30 repetitions) are shown for each prediction step for the respective fire. The \(x\)-axis shows the prediction steps (where the first prediction step corresponds to the third time step of the fire), and the \(y\)-axis shows the average fitness values. Each method is shown in a different color and shape. At any given step, a higher average fitness represents a better prediction for that step. Lastly, the box plots in Figures 7c, 8c, 9c, 10c and 11c show the distribution of fitness values over the 30 repetitions for each method. In each of these box plot figures, the subplots titled with numbers correspond to individual prediction steps for the fire identified by the main title. Each box shows the fitness distribution over 30 repetitions for one method at that step. For example, the leftmost boxes (in black) show the distribution for ESS-NS. In order to assess the time efficiency of the new method, we computed the average execution times of the 30 seeds for each case, which are shown in Table 4.  
Figure 7: Case 520: (**a**) map of the real fire spread; (**b**) fitness averages; and (**c**) fitness distributions. Figure 8: Case 533: (**a**) map of the real fire spread; (**b**) fitness averages; and (**c**) fitness distributions. Figure 9: Case 751: (**a**) map of the real fire spread; (**b**) fitness averages; and (**c**) fitness distributions. Figure 10: Case 519: (**a**) map of the real fire spread; (**b**) fitness averages; and (**c**) fitness distributions. Figure 11: Case 534: (**a**) map of the real fire spread; (**b**) fitness averages; and (**c**) fitness distributions. As for the fitness distribution in Figures 7c, 8c, 9c, 10c and 11c, ESS-NS presents a much narrower distribution of predictions compared to ESS and ESSIM-EA. ESSIM-DE presents more variability in this respect, with the lowest distribution of fitness values for Cases 533 and 534 throughout all steps and in some of the steps for Cases 520, 751, and 519. In general, the best method regarding fitness distribution is ESS-NS, which is most likely related to its strategy of keeping a set of solutions of high fitness found throughout the search instead of returning a final population as the other methods. This provides a guarantee of the robustness of the method, which yields similar results regardless of the initial population. The runtimes for ESS-NS (Table 4) are considerably faster than the other methods for Cases 533, 751, 519, and 534. In these cases, the times for ESS-NS are proportional to those of ESSIM-DE. For Case 520, the runtimes of ESS-NS are longer than ESSIM-DE, and comparable to ESS, but still better than ESSIM-EA. Overall, ESS-NS is the fastest method for these experiments. Regarding time complexity, the main bottleneck for these methods is the simulation time; therefore, this complexity is determined by the population size, the number of iterations, and the number of individuals that are different in each population (assuming that all implementations avoid repeating simulations that have already been performed when the same individual remains in a subsequent iteration). Then, the improvements seen with NS are mainly due to its exploration ability, which often allows it to reach the fitness threshold earlier than other methods. Changing the fitness threshold also affects the time complexity, providing a parameter for the trade-off between speed and quality.