## 4 Experiments and Results  
In this section, we present the experiments performed and evaluate the MLP training efficiency of strong individual optimizers and ensembles of weak optimizers. Performance evaluation is completed using the root mean square error (RMSE) on the training and testing samples data set. We determine the RMSE at specific iterations to obtain a convergence curve graph during the optimization process. Our original implementation for MLP allows us to access parameter information from the neural network easily. The C# implementation of the algorithms from paper [1] is used, and for the scope of this paper, we extend this implementation to training MLPs with these algorithms. We also implement the three ensemble strategies proposed in this work: hybrid cascade, hybrid single elite solution, and hybrid multiple elite solutions. The experimental simulations are performed on machines with different specifications, and the execution time cannot be fairly evaluated. For this reason, we consider the number of evaluations as an index of the processing power used for a given individual optimizer or ensemble of optimizers. An evaluation counter is incremented each time an optimizer uses the objective function. We observed that the objective function is the most time-consuming procedure, with \(\sim\)98% processing time, which is similar to the timing profile of NEAT [48]. In this function, a solution array is decoded into an MLP structure, and then the RMSE of the MLP outputs is calculated using the training or testing samples. The data set consists of 3217 samples split into 75% (2412) training samples and 25% (805) testing samples. Our experimental setup consists of two main steps, as illustrated in Figure 7. In the first step, we randomly search the parameter values for each algorithm. The search procedure is used to identify the three best-performing algorithms, and the best parameters found for these algorithms will be used in all subsequent simulations. We choose a fixed number of 300 iterations for each algorithm, while the other parameters can vary. The configuration of the search procedure is \(n_{\mathit{conf}}\) = 50 random sets of parameter values for each algorithm and \(n_{\mathit{runs}}\) = 2 independent runs to average the results. Since the search procedure is time-consuming, we do not use a larger number for \(n_{\mathit{runs}}\). The parameter limits for an algorithm are appropriately chosen to avoid inconsistent combinations of parameter values. For example, in the ICA algorithm, _no. empires_ must not exceed _pop. size_, but the chosen limits do not allow such an inconsistent combination to be generated during the search procedure. Tables A1-A7 in the Appendix A show the limits we choose for each parameter of an algorithm and the results of the search procedure: the best parameter values, the average RMSE calculated on the training samples, the average number of evaluations, and the structure of the best neural networks. Based on Tables A1-A7, we show in Table 1 the algorithms sorted (from best to worst) according to their performance using the RMSE value averaged over \(n_{\mathit{runs}}\) = 2 runs. One can see that the first three best-performing candidates are ICA, TLBO and SHLO. Although the algorithms presented in Table 1 show a different number of evaluations, in the present paper we do not investigate the causes that lead to these obtained values (e.g., parameter settings or early stopping conditions). It is not the purpose of this work to present a detailed comparison between the individual algorithms. Therefore, we choose  
Figure 7: The experiment setup. three algorithms for which we obtain the lowest mean RMSE value (using the settings presented in Tables A1-A7) and compare their individual performance with the ensembles created with the same algorithms. In the second step of our experiments, we choose ICA, TLBO, and SHLO as the base optimizers. In Figure 7, we illustrate that we perform six main simulations.