## 2 Introduction of LOO-CV  
It has been proven that no single surrogate model always performs the best for all engineering practice [21]. In the model building process of E-AHF, we first construct several single surrogate models without any prior information about the true model and then use the LOO-CV method to select the best model and filter out worse ones. For hybrid surrogate model, instead of randomly determining library of surrogate models, a filtering process, e.g., the LOO-CV method, which is a common modeling selection approach, could be performed first to eliminate poorly-performing individual surrogates and select the best one. The basic idea of the LOO-CV method is to take one sample point from the data set containing \(n\) sample points as the validation set, and then use the remaining \(n-1\) sample points as the training set to build a model. Obtain the prediction error at the validation point and repeat this process \(n\) times sequentially until all \(n\) sample points have been validated once. Finally, \(n\) prediction errors are obtained, and the average value of \(n\) prediction errors is taken as the LOO-CVerror of the model constructed from \(n\) sample points. The LOO-CVerror is shown in Equation (1).  
\[LOO-CVerror=\frac{1}{n}{\sum_{j=1}^{n}}{\left({y_{j}}{\left({x_{j}}\right)- \hat{y}_{j(-j)}}{\left({x_{j}}\right)}\right)}^{2} \tag{1}\]  
where LOO-CVerror means the error generated by LOO-CV, \(n\) is the number of training points, \(y_{j}{\left({x_{j}}\right)}\) is the true response at \(x_{j}\) and \(\hat{y}_{j(-j)}{\left({x_{j}}\right)}\) stands for the prediction at \(x_{j}\), which is calculated using the \(n-1\) training points except the \(j\)th training point (validation point). In general, the model with the smallest LOO-CVerror is the most accurate, and vice versa.  
However, it may occur some problems when using the LOO-CV method. LOO-CV cannot always filter out inaccurate models well and select accurately the best one without enough samples in some cases. We use four common surrogate models, i.e., polynomial regression surface (PRS) [22], radial basis function with multiquadric kernel function (RBF-MQ) [23], radial basis function with thin plate spline basis function (RBF-TPS) [23] and kriging (KRG) [24], to fit the true function. Details about those four surrogate techniques see References [22; 23; 24]. In this work, we use MOA-CVerror and LOO-CVerror to select the most accurate of those four surrogate models. \(R^{2}\) is considered as the calibration criterion of MOA-CVerror and LOO-CVerror. That is, we assume that the results of \(R^{2}\) is reliable, then compare the results of MOA-CVerror and LOO-CVerror with \(R^{2}\) to get the accurate rate of MOA-CV and LOO-CV. A higher \(R^{2}\) means an accurate model. On the contrary, a lower MOA-CVerror or LOO-CVerror indicates a better model.  
Take a 1-dimensional (1D) function as an example to explain the problem occur for LOO-CV as shown in Figure 1. Five sample points are generated (as black ball shows), and four surrogate models are built based on five sample points. The true function is shown as the black line. \(R^{2}\) of the four models are listed in Table 1. We can see that from \(R^{2}\) the most accurate model is RBF-TPS, RBF-MQ performs the second, KRG performs the third and PRS performs the worst. However, from LOO-CVerror, the best model is KRG, the second one is PRS, the third one is RBF-MQ and the last one is RBF-TPS. That is, RBF-TPS is actually the most accurate model, however, in the construction of E-AHF, the step of using LOO-CV method to filter worse surrogate models means will remove the RBF-TPS model from the library of surrogate models and remain the PRS model which is the worst one.