# A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy
## 4 Case Studies
### Numerical Examples
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Function** & **Metric** & **KRG** & **RBF** & **SVR** & **BP** & **PWS** & **NPWS** & **OWS** & **RLS-EM** \\ \hline \multirow{3}{*}{Branin-Hoo} & RMSE 1  & **11.858**/4.452 & 19.127/4.461 & 18.981/4.591 & 13.869/5.920 & 15.455/14.372 & 15.582/4.333 & 15.184/4.475 & 12.008/4.679 \\  & AAE & **6.032**/1.971 & 11.278/2.020 & 11.315/2.154 & 7.577/3.312 & 8.670/1.942 & 8.771/1.902 & 8.460/2.063 & 6.436/2.317 \\  & R2  & **0.941**/**0.045** & 0.859/0.068 & 0.860/0.071 & 0.917/0.068 & 0.905/0.055 & 0.904/0.055 & 0.908/0.056 & 0.939/0.050 \\ \hline \multirow{3}{*}{Camel back} & RMSE & 19.490/4.823 & 7.855/5.755 & 130.05/4.262 & 10.961/5.970 & 11.190/3.312 & 11.441/**0.027** & 10.842/4.059 & **7.812**/3.662 \\  & AAE & 11.367/2.764 & **4.591**/2.666 & 7.136/2.147 & 6.389/4.683 & 6.519/1.676 & 6.651/1.851 & 6.329/2.001 & **4.832**/1.248 \\  & R2  & 6.698/0.157 & 0.930/1.97 & 0.859/0.091 & 0.867/0.221 & 0.898/0.073 & 0.895/0.064 & 0.899/0.097 & **0.943**/**0.062** \\ \hline \multirow{3}{*}{Hart mann-3} & RMSE & 0.235/0.060 & 0.417/0.049 & 0.372/0.076 & 0.253/0.083 & 0.273/0.045 & 0.278/**0.044** & 0.265/0.048 & **0.233**/**0.044** \\  & AAE & 0.159/0.036 & 0.281/0.030 & 0.221/0.035 & 0.170/0.051 & 0.176/0.024 & 0.179/**0.023** & 0.171/0.027 & 0.157/0.034 \\  & R2  & 0.929/0.038 & 0.788/0.049 & 0.825/0.077 & 0.914/0.061 & 0.907/0.022 & 0.905/0.032 & 0.913/0.033 & **0.931**/**0.030** \\ \hline \multirow{3}{*}{Hart mann-6} & RMSE & 0.239/0.034 & 0.192/0.019 & 0.214/0.025 & 0.193/0.021 & 0.196/0.023 & 0.196/0.023 & 0.195/0.023 & **0.190**/**0.019** \\  & AAE & 0.156/0.022 & 0.115/**0.008** & 0.115/**0.008** & 0.116/0.011 & 0.112/0.010 & 0.112/0.010 & **0.111**/0.009 & 0.114/**0.008** \\  & R2  & 0.588/0.114 & 0.739/0.046 & 0.677/0.055 & 0.734/0.057 & 0.728/0.050 & 0.727/0.051 & 0.731/0.049 & **0.743**/**0.045** \\ \hline \multirow{3}{*}{Extended Rosen-Price} & RMSE (10\({}^{5}\)) & 0.201/0.023 & 0.185/0.021 & 0.201/0.027 & 0.187/0.022 & **0.180**/0.021 & 0.181/0.022 & **0.180**/**0.022** & **0.184**/**0.020** \\  & AAE (10\({}^{5}\)) & 0.154/0.017 & 0.142/0.015 & 0.154/0.019 & 0.144/0.017 & 0.138/0.016 & 0.138/0.016 & 0.138/0.016 & 0.141/**0.015** \\  & R2  & 0.765/0.028 & 0.801/0.024 & 0.764/0.036 & 0.796/0.027 & 0.811/0.023 & 0.811/0.023 & **0.811**/**0.022** & **0.810**/**0.022** & 0.808/0.023 \\ \hline \multirow{3}{*}{Dixon-Price} & RMSE (10\({}^{6}\)) & 0.159/**0.017** & 0.195/0.022 & 0.230/0.029 & 0.161/0.019 & 0.175/0.020 & 0.177/0.020 & 0.171/0.019 & **0.158**/0.019 \\  & AAE (10\({}^{6}\)) & 0.238/0.034 & **0.151/0.017** & 0.174/0.022 & **0.151**/0.019 & 0.166/0.020 & 0.168/0.020 & 0.162/0.019 & 0.160/0.019 \\ \cline{1-1}  & R2  & 0.835/**0.025** & 0.753/0.034 & 0.760/0.045 & 0.831/0.033 & 0.802/0.026 & 0.797/0.026 & 0.810/**0.025** & **0.841**/0.057** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Comparison of root mean squared error (RMSE), average absolute error (AAE), and R2 for different surrogate models. BP: BestPRESS, PRESS: prediction sum of squares, PWS: PRESS weighted surrogate, NPWS: non-parametric PRESS weighted surrogate, OWS: optimal weighted surrogate, RLS-EM: regularized least squares ensemble model. The BP, PWS, NPWS, and OWS use GMSE as error metric, and they require more computation time than the individual surrogate models, especially for high-dimensional problems. The GMSE error metric takes a relatively longer time to repeatedly construct the individual surrogate models, and the computation time is also affected by the number of divisions. In the RLS-EM, the individual surrogate models are constructed based on the initial samples, the weight factors are obtained by the regularization least squares method, which helps avoid the time spent on repetitively constructing the individual surrogate models. Figure 6 shows the computational cost of Hartmann-6, Extended-Rosenbrock, and Dixon-Price problems, which are represented by the subscript numbers of 1, 2 and 3, respectively.