# A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy
## 3 Proposed Regularized Least Squares Ensemble Model
### Samples Adding by the Augmentation Strategy  
The RLS-EM proposed in this paper seeks to simultaneously capture the global and local accuracy. Since the PRS may exhibit lower accuracy in some nonlinear applications, RLS-EM only combines KRG, RBF, and SVR to meet the local accuracy and global trend requirements. The predicted values of the KRG and RBF models at the training points are equal to the actual function values, so the collinearity is unavoidable. To solve the collinearity problem, an augmentation strategy and a regularization strategy were developed. The augmentation strategy is used to reduce the influence of the collinearity on one hand; on the other hand, the augmentation strategy helps to improve the accuracy of the model in the unexplored area.  
The \(N\) samples obtained by Latin hypercube sampling (LHS) technique are used to construct the KRG, RBF, and SVR surrogate models, and the corresponding prediction values at the samples are \(\hat{\mathbf{Y}}=[\hat{y}_{1}(x_{i}),\ \hat{y}_{2}(x_{i}),\ \hat{y}_{3}(x_{i})]\), \(i=1\), \(2\), \(\ldots\), \(N\), \(\hat{y}_{1}(x_{i})\), \(\hat{y}_{2}(x_{i})\), \(\hat{y}_{3}(x_{i})\) represents the KRG, RBF, and SVR prediction values at \(x_{i}\), respectively, the corresponding actual function values \(Y=[y_{1},y_{2},\ldots,y_{N}]^{\rm T}\). The augmentation strategy is implemented to add additional samples in the exploration regions that are far from the \(N\) original samples. The number of \(N_{add}\) points (the set is \(X_{add}\)) is obtained from Algorithm 1.  
```
Input:\(\mathbf{X}=[\ x_{1},\ x_{2},\ldots,\ x_{N}]\).
1: Set \(\mathbf{S}_{add}^{pri}\) empty, \(\mathbf{S}=\mathbf{X}\).
2: Obtain \(3\times N_{add}\) samples by LHS, put them in \(X_{\rm lhs}\).
3: For \(i=1\): \(3N_{add}\) do
4: Calculate the distance of all the members in \(\mathbf{X}_{\rm lhs}\) to the samples in \(\mathbf{S}\).
5: Move the sample with the largest distance from \(\mathbf{X}_{\rm lhs}\) to \(\mathbf{S}_{add}^{pri}\) and \(\mathbf{S}\).
6: End for
7: Construct KRG, RBF by \(\mathbf{S}\), calculate the uncertainties with (13) at the sample set \(\mathbf{S}_{add}^{pri}\), storage the difference values in \(P_{kr}\).
8: Sort \(P_{kr}\) from the largest to the least, choose the top \(N_{add}\) values of the corresponding samples, and put them into \(\mathbf{X}_{add}\).
9:\(\mathbf{X}_{add}\).
```  
**Algorithm 1** Pseudo code of augmentation strategy for adding samples  
\[P_{kr}(x_{i})=\left|\hat{y}_{1}(x_{i})-\hat{y}_{2}(x_{i})\right| \tag{13}\]  
where the \(P_{kr}(x_{i})\) is the absolute value of the difference between the KRG and RBF model at the \(i\)th adding points. The prediction values of the KRG, RBF, and SVR surrogate models on \(\mathbf{X}_{add}\) are represented as \(\hat{\mathbf{Y}}_{add}=[\hat{y}_{1}(x_{iadd}),\hat{y}_{2}(x_{add}),\hat{y}_{3}(x _{iadd})]\), \(iadd=1\), \(2\), \(\ldots\), \(N_{add}\), and the actual functions values at the \(N_{add}\) points are \(Y=[y_{N+1},y_{N+2},\ldots,y_{N+Nadd}]^{\rm T}\). Then, the augmentation matrix system of \(\hat{\mathbf{Y}}_{expand}\) and \(\mathbf{Y}\) are represented as:\[\hat{\mathbf{Y}}_{expand}=\left[\begin{array}{cccc}\hat{y}_{1}(x_{1})&\hat{y}_{2}(x_ {1})&\hat{y}_{3}(x_{1})\\ \vdots&\vdots&\vdots\\ \hat{y}_{1}(x_{N})&\hat{y}_{2}(x_{N})&\hat{y}_{3}(x_{N})\\ \hat{y}_{1}(x_{N+1})&\hat{y}_{2}(x_{N+1})&\hat{y}_{3}(x_{N+1})\\ \vdots&\vdots&\vdots\\ \hat{y}_{1}(x_{N+N_{add}})&\hat{y}_{2}(x_{N+N_{add}})&\hat{y}_{3}(x_{N+N_{add }})\\ \end{array}\right]\mathbf{Y}=\left[\begin{array}{c}y_{1}\\ \vdots\\ y_{N}\\ y_{N+1}\\ \vdots\\ y_{N+N_{add}}\\ \end{array}\right] \tag{14}\]