## 3 Numerical Results
### Design of Experiment--Hybrid Algorithm Parameters Optimization
#### 3.2.1 Single-Population Genetic Algorithm and Evolution Strategy Results  
The factors that were found to have the highest impact on the resultant optimization capabilities are the mutation rate, penalty exponent, and penalty multiplier--C, D, and E from Table 1. Other factors are assumed to be less important, i.e., their effect on the objective function minimization is statistically insignificant. The main effect plot (for averaged responses) is shown in Figure 9, using the fitted full-quadratic response surface (RSM).  
Figure 8: Training and validation loss as a function of epochs.  
The factor associated with the population layout, as well as the crossover rate, show a nearly linear and barely noticeable effect on the algorithm efficiency. On the other hand, a very significant impact is seen for the strongly nonlinear effect of the mutation rate. Both factors connected to the penalty function are nonlinear in their effects on the mean objective function, but their impact is not as strong as the mutation rate in a given example (even though they are still considered significant factors).  
The penalty function, which is studied in the literature significantly less often, compared to other evolution algorithm parameters, shows high mutual interactions between the multiplier and the exponent, as shown below. The increasing multiplier (from 10 to 1000) may diminish the efficiency of the algorithm--when the exponent is equal to 3 or improves the efficiency when the exponent is equal to 1. This shows that penalization may be one of the critical aspects of constrained evolutionary optimization (Figure 10).  
In order to utilize the maximum efficiency of the hybrid evolution algorithm, the optimization of the parameters of the factors was conducted in Minitab 21.1 [43]. The effect of the parameters adjustment is presented in Figure 11. A small-sized population of 16 individuals with a relatively high number of generations (100) is chosen in the optimal set. Only 50% of individuals in new generations are created using crossover, while the mutation rate of each gene is fixed at 5.9%. A highly nonlinear penalty function (with an exponent of 3) is used with a multiplication factor of 10, indicating stronger penalization of severely unfitted individuals or weaker penalization of barely unfitted individuals.  
Figure 10: The interaction between the penalty multiplier and the penalty exponent and its effect on the objective function.  
Figure 9: The main effect plot for the hybrid optimization using GA + EVOL approach (mean value of the fitted RSM shown in the Y axis).  
Once again, 10 repetitions of the new scenario are run in order to verify the performance of the algorithm. Indeed, the average of best-fitted individuals from every 10 runs is characterized by the lowest value of the objective function among all of the analyzed hybrid algorithm parameters, in accordance with RSM-based prediction. Even though the average objective function is relatively low, there is a visible spread in the results (min and max values are 0.257 and 0.356, respectively, with an average of 0.284), as shown in Figure 12.