## 4 Experiments and Results
In the first three simulations, each algorithm is run using 300 iterations. Since this is a large number of iterations, we correlate an algorithm with this configuration with a strong individual optimizer. The other three simulations represent the run of each proposed ensemble strategy. A base optimizer is run with five iterations in a single ensemble iteration, and we correlate it with a weak optimizer. Therefore, a single ensemble iteration is equivalent to 15 cumulative iterations from three weak optimizers. We choose 20 ensemble iterations for each ensemble strategy to obtain a cumulative number of 300 iterations (i.e., the same number of iterations as a strong individual optimizer). Our reasoning is to perform a fair comparison between strong individual optimizers and ensembles of weak optimizers. We run all six simulations with thirty independent runs to obtain statistically meaningful results. Figure 8 shows the convergence curve graphs of all six simulations obtained on the training and test samples. The curve graphs of the individual optimizers ICA, TLBO, and SHLO are shown with continuous lines, while the curve graphs of the ensemble strategies are shown with dashed lines. We illustrate on the \(x\)-axis the current number of an ensemble iteration. In the case of individual optimizers, one ensemble iteration is equivalent to fifteen individual iterations. Therefore, we collect performance statistics from individual optimizers every 15 iterations. On the \(y\)-axis, we show the RMSE obtained at each ensemble iteration (i.e., at every 15 iterations of the individual ICA, TLBO, and SHLO algorithms) averaged over 30 independent runs. We can see in Figure 8 that the convergence curve graphs for the training and test samples are very similar in shape but have different scales. During the first ten ensemble iterations, all three ensemble strategies outperformed the individual optimizers in conver \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Optimization Algorithm** & **Mean RMSE** & **Mean No. EVALUATIONS** & **Best NN Structure** \\ \hline ICA (**best**) & 0.00581 & 37,202 & [3-11-3-3] \\ TLBO & 0.00612 & 40,424 & [3-7-3] \\ SHLO & 0.00643 & 40,033 & [3-7-2-3] \\ SLO & 0.00663 & 60,998 & [3-9-2-3] \\ FGA & 0.01020 & 42,8156 & [3-8-2-3] \\ VS & 0.01179 & 188 & [3-10-3-3] \\ VOA (**worst**) & 0.01265 & 5558 & [3-11-3-3] \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The algorithms sorted according to their performance. Figure 8: The convergence curve graphs on: (**a**) train samples; (**b**) test samples. gence speed. It is interesting to note that hybrid cascade ensemble has better convergence than hybrid single elite solution. However, these ensembles have similar performances after the 10 th ensemble iteration. The best overall performance is observed for the hybrid multiple elite solutions ensemble, outperforming all individual algorithms and the other two ensemble strategies. A hybrid multiple elite solutions ensemble is an extension of hybrid single elite solution because it uses a larger population of elites. Therefore, we argue that the increased performance of the hybrid multiple elite solutions ensemble comes from creating a more diverse improved population at the end of each ensemble iteration. The individual ICA optimizer has a slow convergence, but after 300 iterations, it achieves a performance similar to that of the hybrid multiple elite solutions ensemble (at ensemble iteration 20). However, for some problems, 300 iterations might be a large value, and choosing the multiple elite hybrid solution strategy is preferred. In Table 2, we present the detailed simulation results for the individual ICA, TLBO, and SHLO optimizers at iteration 300 and the results of the proposed ensemble strategies at ensemble iteration 20. We can see that the hybrid multiple elite solutions ensemble provides the smallest errors with a mean RMSE train of 0.00582 and a mean RMSE test of 0.01029. The same ensemble provides the best solutions with an RMSE train of 0.00517 and an RMSE test of 0.00932.