# A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy
## 4 Case Studies
### Numerical Examples  
Six numerical examples varying from two-dimensional (2-D) to 12-dimensional (12-D) [42; 44] were chosen to test the performance of RLS-EM: (1) Branin-Hoo function; (2) Camelback function; (3) Hartmann-3 function; (4) Hartmann-6 function; (5) extended Rosenbrock function (9-D); and (6) Dixon-Price function (12-D). A description of each test is given in Appendix B.  
LHS was used to generate the training and testing sets, the MATLAB routine \"lhsdesign\" with \"maximin\" criterion and 100 iterations were used to generate the (\(N+N_{add}\)) samples and \(N_{t}\) tests. The summary of the sampling in the numerical cases is provided in Table 1.  
Table 2 lists the setup details of the individual models, which were used to develop the ensemble model based on different variable dimensions and nonlinearities. Each individual model has significant differences between variables with different dimensions and different degrees of nonlinearity, e.g., for the low-dimensional variables such as variable with numbers two and four, constant regression can satisfy the accuracy requirements of a KRG model, while the high-dimensional variables require quadratic regression to obtain a more accurate model. Similarly, the kernel parameters and regularization parameters of different dimensional variables with different degrees of nonlinearity are different for the SVR model. Thus, the KRG, RBF, and SVR model setting information for different dimensional variables are listed in detail, as shown in Table 2.  
To validate the performance of the different surrogate models, 100 runs were executed for each of the numerical examples. The MATLAB routine \"boxplot\" was used for easy visualization and comparison. The three-dimensional surface plots of the Branin-Hoo and Camelback functions are shown in Figures 1 and 2, respectively. The nine surface plots in Figure 1 show that each surrogate model fits the Branin-Hoo function well. However, Figure 2 shows that the different surrogate models have considerable differences in the Camelback function fitting. Despite the two functions being highly nonlinear, the RLS-EM can accurately approximate the actual functions. The boxplots of RMSE, AAE, and R\({}^{2}\) for the different test functions are shown in Figures 3-5; the mean and standard deviations of the different surrogate models for the performances are listed in Table 3. After 100 runs were executed, the mean and standard deviation of the RMSE, AAE, and R\({}^{2}\) metrics for the numerical examples are shown in Table 3. For each metric of the numerical examples, the values to the left of the symbol \"/\" are the mean of the different models, and the values below are the standard deviations corresponding to the models. For the RMSE and AAE metrics, the smaller mean values indicate the better model accuracy, and the smaller standard deviation values show the better robustness. The R\({}^{2}\) metric with a mean value is closer to one and a smaller standard deviation indicate a more accurate and more robust model. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Function** & _ndrb_1  & \(N\) & \(N_{add}\) & \(N_{t}\) \\ \hline Branin-Hoo & 2 & 20 & 6 & 400 \\ Camelback & 2 & 20 & 6 & 400 \\ Hartman-3 & 3 & 30 & 9 & 1000 \\ Hartman-6 & 6 & 100 & 30 & 1000 \\ Extended & 9 & 150 & 45 & 1000 \\ Rosenbrock & 10 & 200 & 60 & 1000 \\ Dixon–Price & 10 & 200 & 60 & 1000 \\ \hline \hline \end{tabular}  
\end{table} 
Table 1: Summary specifications for numerical cases.