# A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy
## 2 Background of Ensemble Methods  
Usually, the surrogate model technique is utilized to construct several different surrogates and select the best one. However, this scenario has two major shortcomings [41]. It is wasteful to discard the so-called inaccurate models, and the accuracy of the surrogate model is affected by the sample points. This is because the surrogate model may exhibit different precisions for different data sets. To overcome these drawbacks, ensemble methods are proposed.  
An ensemble surrogate model is a weighted combination of several individual surrogate models [42], which is defined as:  
\[\hat{f_{e}}(x)=\sum\limits_{i=1}^{M}w_{i}(x)\hat{f_{i}}(x),\ \ \ \ \ \sum\limits_{i=1}^{M}w_{i}(x)=1 \tag{1}\]  
where \(\hat{f_{e}}(x)\) is the prediction of the ensemble, \(M\) is the number of surrogate models used, and \(w_{i}\) is the weight factor for the \(i\)th surrogate \(\hat{f_{i}}(x)\). Evidently, the larger weights are assigned to the more accurate surrogate models, and vice versa.  
[PERSON] et al. [33] proposed the evaluation of the weight factors \(w_{i}\) in a linear ensemble as:  
\[w_{i}=\frac{1}{V_{i}}/\sum\limits_{j=1}^{M}\frac{1}{V_{j}} \tag{2}\]  
where \(V_{i}\) is the prediction variance of the \(i\)th surrogate model.  
[PERSON] et al. [31] considered PRS, KRG, and RBF, and proposed an ensemble scheme to estimate the weight factors in a WAS, including the BestPRESS (BP), the PRESS weighted surrogate (PWS), and the non-parametric PRESS weighted surrogate (NPWS).  
Taking the prediction sum of squares (PRESS) as the error measure, the NPWS is given as:  
\[w_{i}=\sum\limits_{j=1,j\
eq i}^{M}E_{i}/\left[(M-1)\sum\limits_{j=1}^{M}E_{j}\right] \tag{3}\]  
where \(E_{j}\) is the GMSE of the \(i\)th surrogate model calculated from:  
\[E_{j}=\frac{1}{N}\sum\limits_{i=1}^{N}\left[y(x_{i})-\hat{y}_{j}^{(-i)}(x_{i}) \right]^{2} \tag{4}\]  
where \(y(x_{i})\) is the true response at the \(i\)th data point \(x_{i}\), and \(\hat{y}_{j}^{(-i)}(x_{i})\) is the corresponding prediction from the surrogate model constructed using all except the \(i\)th data point \(x_{i}\), and \(N\) is the number of sample points.  
The model with the least PRESS error is assigned a weight factor of one, and all the other models are assigned zero weight factors; this strategy is called the BP model [31].  
The PWS uses the GMSE as a global error metric to select the weight factors using a heuristic formulation, which is formulated as follows:  
\[w_{i}=w_{i}^{*}/\sum\limits_{j=1}^{M}w_{j}^{*} \tag{5}\] \[w_{i}^{*}=(E_{i}+\alpha E_{avg})^{\beta},\ \beta<0,\ \alpha<1\] \[E_{avg}=\frac{1}{M}\sum\limits_{i=1}^{M}E_{i}\]The weighting scheme requires the user to specify parameters \(\alpha\) and \(\beta\), which control the contribution of the individual surrogates; \(\alpha\) and \(\beta\) are assumed to be 0.05 and \(-1\), respectively [31].  
[PERSON] and [PERSON] [32] used GMSE as the global error metric and proposed an optimization algorithm to calculate the weight factors; the algorithm is expressed as follows:  
\[\begin{array}{ll}\text{Find }w_{i}\\ \text{Min }GMSE\\ \text{s.t. }\sum\limits_{i=1}^{M}w_{i}=1\end{array} \tag{6}\]  
[PERSON] et al. [43] proposed an ensemble surrogate model called optimal weighted surrogate (OWS); the OWS is represented as follows:  
\[\begin{array}{ll}\min\limits_{w_{i}}&MSE(f_{e}(x))=w^{\text{T}}\text{C}w\\ \text{s.t. }&\sum\limits_{j=1}^{M}w_{j}=1\end{array} \tag{7}\]  
The correlation matrix of the error from the individual surrogate models that are used to constitute the ensemble surrogate model is expressed as follows:  
\[C_{ij}=\frac{1}{N}e_{i}^{\text{T}}e_{j} \tag{8}\]  
where \(e_{i}\) and \(e_{j}\) are the vectors of cross-validation errors (i.e., PRESS) for the \(i\)th and \(j\)th surrogate models, respectively. The application of the ensemble models can be found in [44; 45; 46; 47].