# A Least Squares Ensemble Model Based on Regularization and Augmentation Strategy
## 3 Proposed Regularized Least Squares Ensemble Model
### The Regularization Strategy in the Least Squares System  
A regularization term is added to further reduce the impact of collinearity. Due to the interpolation properties of the KRG and RBF models, \(\hat{y}_{1}(x_{i})\) and \(\hat{y}_{2}(x_{i})\) are equal to the actual function values at the \(N\) samples. The relatively high precision of KRG, RBF, and SVR surrogate models may also predict approximately equal values at some samples in the set \(\mathbf{X}_{add}\). By adding a regularization term multiplying an identity matrix, the matrix coefficients \(\hat{\mathbf{w}}_{expand}\) can be estimated from the augmented matrix inversion system as follows:  
\[\hat{\mathbf{w}}_{expand}=\left(\hat{\mathbf{Y}}_{expand}^{\mathrm{T}}\hat{\mathbf{Y}}_{ expand}+\lambda\mathbf{I}\right)^{-1}\hat{\mathbf{Y}}_{expand}^{\mathrm{T}}\mathbf{Y} \tag{15}\]  
where \(\mathbf{I}_{3\times 3}\) is an identity matrix, and \(\lambda\) is the regularization parameter. Since the linear correlation in \(\hat{\mathbf{Y}}_{expand}^{\mathrm{T}}\) is expected to be lower than that in \(\hat{\mathbf{Y}}\), and the regularization item further reduces the linear correlation, the accuracy and robustness on evaluating \(\mathbf{w}\) by means of \(\hat{\mathbf{w}}_{expand}\) is expected to be better. The weight factors for the ensemble model are calculated as:  
\[w_{i}^{*}=\frac{\hat{w}_{expand}^{i}}{\sum\limits_{i=1}^{3}\hat{w}_{expand}^{ j}},i=1,\ 2,\ 3 \tag{16}\]  
The regularized least squares ensemble method can be expressed as follows:  
1. Random sampling \(N\) samples, the \(N_{add}\) samples are obtained by the augmentation strategy, and the actual function values \(Y=[y_{1},\,y_{2},\,\ldots,y_{N+Nadd}]\) are calculated by expensive simulations. 2. Choose \(N\) samples to construct the KRG, RBF, and SVR surrogate models, as the prediction values of the KRG and RBF at the \(N\) samples are equal to the corresponding actual function values of \(y_{i}\), \(i=1,\,2,\,\ldots,\,N\); calculate \(\hat{y}_{3}(x_{i})\) of the SVR model at each of the \(N\) samples. 3. Evaluate \(\hat{y}_{1}(x_{N+i})\), \(\hat{y}_{2}(x_{N+i})\), and \(\hat{y}_{3}(x_{N+i})\) for the KRG, RBF, and SVR surrogate models, where \(i=1,\,2,\ldots,\,N_{add}\) and \(N_{add}\) is the number of adding samples. Construct the matrix \(\hat{\mathbf{Y}}_{expand}\) and \(\mathbf{Y}\) as in Equation (14). 4. Calculate the inverse of the augmented matrix system for \(\hat{w}_{expand}\) by Equation (15), and the standardized weight factors by Equation (16). However, the regularization parameter \(\lambda\) should be confirmed before using Equation (15); a search algorithm was developed to obtain the optimal regularization parameter value \(\lambda^{*}\), and the detailed pseudo codes are summarized in Algorithm 2. ```
1:A constant array \(\prod=\left\{\xi_{1},\xi_{2},\cdots,\xi_{q}\right\}(0<\xi_{1}<\xi_{2}<\cdots<\xi_{q }<1)\) is set for \(\lambda\), and \(l=\min=1\), \(r=\max=q\). 2:While \(\lambda_{\min}<\lambda_{\max}\), \(\text{mid}=\lfloor(l+r)/2\rfloor\), go to step 3, else go to step 8. 3:Randomly divide the predicted values of the KRG, RBF, and SVR surrogate models of the \(N+N_{add}\) samples into \(k\) (we use \(k=5\) in this paper) equal parts. 4:The \(\hat{\mathbf{Y}}_{\text{train}}\) matrix is made up by the predicted values of three individual surrogate models in the \(k-1\) group, and by singular value decomposition (SVD), which can be expressed as

\[\hat{\mathbf{Y}}_{\text{train}}=\mathbf{U}\mathbf{D}\mathbf{V}^{\text{T}} \tag{17}\]

where \(\hat{\mathbf{Y}}_{\text{train}}=[\hat{\mathbf{Y}}_{1},\hat{\mathbf{Y}}_{2}, \hat{\mathbf{Y}}_{3}]\), \(\mathbf{U}=(\mathbf{U}_{1},\mathbf{U}_{2},\mathbf{U}_{3})\) is a \(m_{l}\times 3\) orthogonal matrix, and \(m_{l}\) is the number of the \(k-1\) parts of the samples. \(D=\text{diag}(d_{1},d_{2},d_{3})\) is a \(3\times 3\) diagonal matrix. \(\mathbf{V}=(\mathbf{V}_{1},\mathbf{V}_{2},\mathbf{V}_{3})\) is a \(3\times 3\) orthogonal matrix. 5:After the SVD, \(\hat{w}_{e}\) is calculated for \(\lambda_{l}\) and \(\lambda_{r}\) by:

\[