# Kernel Parameter Optimization for Kriging Based on Structural Risk Minimization Principle

Authors: Hua Su; Chunlin Gong; Liangxian Gu
Journal: Mathematical Problems in Engineering
 (2017)

---

## Abstract

An improved kernel parameter optimization method based on Structural Risk Minimization (SRM) principle is proposed to enhance the generalization ability of traditional Kriging surrogate model. This article first analyses the importance of the generalization ability as an assessment criteria of surrogate model from the perspective of statistics and proves the applicability to Kriging. Kernel parameter optimization method is used to improve the fitting precision of Kriging model. With the smoothness measure of the generalization ability and the anisotropy kernel function, the modified Kriging surrogate model and its analysis process are established. Several benchmarks are tested to verify the effectiveness of the modified method under two different sampling states: uniform distribution and nonuniform distribution. The results show that the proposed Kriging has better generalization ability and adaptability, especially for nonuniform distribution sampling.


---

## Body

## 1. Introduction

Computer-intensive optimization problem becomes more and more as the increasing requirement for high-fidelity model in industry area, especially for aerospace engineering. It is very urgent to improve computation efficiency [1, 2]. Surrogate model is the comprehensive application of the Design of Experiment (DOE), mathematical statistics, and optimization techniques. It approximates the complicated and time-consuming physical model by building some analytical mathematical model, to reduce the analysis process and smooth the design space. It becomes one of the most effective methods for computer-intensive problem and has been widely applied to high-fidelity design and optimization [3].The fitting precision of surrogate model for physical model is one of the most important indicators. For computational resource limitation, it is not rational to assess the entire design space of the physical model directly; using the error between surrogate model and physical model at the sample points to estimate the fitting precision is the most effective way. Fitting precision is a function of the sample capacity. It converges to the true precision only when the sample capacity satisfies the large number theorem. Actually sample capacity and fitting precision are conflicting. We hope less sample points with higher fitting precision. Vapnik proposed the statistical learning theory (SLT) in the late 1970s [4, 5] and became a machine learning method for limiting sample points in the 1990s. The core idea is to control the generalization ability of the learning machine by control the capacity of the machine. That is to improve the fitting precision of the surrogate model by applying SRM principle.There are many surrogate models, such as Response Surface Model (RSM), Kriging, Radial Basis Function (RBF), and Support Vector Regression (SVR). Kriging is an interpolation method based on statistical theory; the main idea is to evaluate the approximate function of the object based on the dynamic construction of design space to predict the information of unknown points [6, 7]. For Kriging, linear weighted combination of the information of nearby points is also needed for predicting unknown points, which can be determined by minimizing the variance of the estimated value error, which means Kriging is also a best linear unbiased estimator problem [8]. Kriging surrogate model is simple and stable compared to other methods and has been widely used in many fields [9–12]. Some improved Kriging models have also been studied, such as Gradient-Enhanced Kriging [13], CoKriging [14], and Hierarchical Kriging [15].Kriging is sensitive to sample points. It has worse predicting result when with less sample points. The quantity of information of the correlation matrix and the parameters of the basis function have obvious influence to fitting precision. It is very necessary to optimize the parameters of the basis function to improve fitting capacity. SRM is the most important theory to evaluate the generalization ability of the surrogate model, which has been widely used in kernel parameter optimization of SVM [16], fuzzy model [17], and chaotic system [18]. But there are a few studies for other surrogate models. Zhu [19] proposed an automatic method to optimize basis function for RBF using the SRM principle. Chen [20] applied to SRM principle to RBF network study to improve the generalization ability. These previous researches indicate that using SRM principle to optimize the parameters of the surrogate model really can improve the fitting precision of surrogate model.This paper proposes an anisotropic basis function parameter optimization method for Kriging based on SRM principle. The parameters of the correlation functions of Kriging are optimized using the smoothness measure as objective function. Some benchmarks with different scale are used to verify the proposed methods in this paper. The influence of the distribution pattern of sample points with uniform distribution and nonuniform distribution is also studied.


## 2. Assessment Criteria of Surrogate Model

The purpose of machine learning is to find the internal dependencies by learning given data to predict unknown data or estimate model characteristic. From the perspective of statistics, machine learning can be viewed as to predict the unknown output relation by some given training samples as accurate as possible. To minimize the risk functional, we should find the optimal functionf(x,α∗) in function set fx,α:(1)Rα=∫y-fx,α2dFx,y.fx,α is the prediction function set; α is the parameter of the prediction function. So the machine learning is designed to minimize the risk functional of the squared error loss function.


### 2.1. Empirical Risk Minimization Principle

Usually we cannot predict the specific form of probability distribution and only know there is a proper probability distribution which can fit the training samples. The expected risk depends on probability distributionF(x,y); it is hard to minimize the risk functional directly by constructing predictive function of minimized expected risk with several samples. Therefore, Empirical Risk Minimization (ERM) principle is proposed to minimize risk functional in the traditional machine learning method,(2)Rempα=1n∑i=1nyi-fxi,α2.ERM has been widely used in the least square method for regression problem, maximum likelihood method for probability density estimation, and neural network learning [21]. But ERM is based on the large number theorem, completely dependent on the sum of squared error of the samples. With less samples, small empirical risk cannot guarantee minimizing expected risk. There may happen oversmoothing for the RBF and neural network, which is based on ERM principle [22]: little training error for sample points but large testing error for other points.


### 2.2. Structural Risk Minimization Principle

Generalization ability means the capacity to predict or estimate unknown phenomenon by machine learning. ERM principle cannot guarantee minimizing expected risk. Therefore, SRM principle is proposed based on the statistical learning theory. It provides the relationship between empirical risk and real risk, which is known as the bound of the generalization ability. SRM principle divides the real risk of machine learning into empirical risk and confidence interval [4]:(3)Rα≤Rempα+Φhn.h is the Vapnik-Chervonenkis (VC) dimension, which is the most important theoretical basis of the statistical learning theory. It defines the capacity of the function set and reflects the generalization ability of the machine learning. It is the best descriptive indicators for the capacity of function set learning until now.From (3), we know ERM principle is unconscionable with limited samples. When h/n becomes larger, empirical risk decreases, but real risk does not always become lower. When the dimension of samples is fixed, reduced h (VC) may decrease confidence interval, so the empirical risk comes close to real risk. In this case, less empirical risk stands for less expected risk. Increasing the capacity of samples n also decreases confidence interval to reduce real risk, but it is difficult to get plenty of samples because of the limitation of computational cost.Therefore, decreasing VC dimension is the most suitable way to reduce real risk. For a given observation set, SRM principle chooses the proper function(y-f(x,αnk))2 from the subset Sk with minimized risk to minimize the empirical risk. So SRM principle is a compromise method to balance the fitting precision and the complexity of fitting function.


### 2.3. SRM Index Based on Smoothness Measure

Smoothness measure is used to evaluate the generalization ability. In the projection space formed by kernel function, smoothness measure has its more direct and natural definition. It can be described as the norm of Hilbert space of the kernel function, which is defined as follows.Every functionf(x) in Hilbert space of kernel function K can be expressed as(4)fx=∑i=1nαikx,xi.Then the norm off(x) in Hilbert space is(5)fR2=∑i=1nαi2λi.λi is the eigenvalue of kernel K. The generalization ability can be measured by evaluating W2. The optimal regression problem becomes(6)minW2s.t.Wϕ+b=Y,where W2 is the smoothness measure of sample points.


### 2.4. Mercer Theorem

The criteria to determine whether a function is kernel function is as follows.I is the domain of the independent variables u and u′. There is a nonnegative real function K(u,u′); it is continuous in I×I. The sufficient and necessary condition for K(u,u′) to be expressed as a series(7)Ku,u′=∑iλiψiuψiu′is that all functions satisfy ∫g2(u)du<∞,g≠0 and also satisfy the following condition:(8)∬ku,u′gugu′dudu′>0,where the series is convergence in I×I. λi is the eigenvalue (λi>0) of K(u,u′). ψi(u) is the eigenvector of K(u,u′). A function satisfying the above definition is known as Mercer kernel and can be expressed as the inner product in Hilbert space,(9)Ku,u′=k·,u,k·,u′=ϕu,ϕu′,where , is the operator of inner product. ϕ:u→k(·,u) is a nonlinear transformation of kernel function.


## 2.1. Empirical Risk Minimization Principle

Usually we cannot predict the specific form of probability distribution and only know there is a proper probability distribution which can fit the training samples. The expected risk depends on probability distributionF(x,y); it is hard to minimize the risk functional directly by constructing predictive function of minimized expected risk with several samples. Therefore, Empirical Risk Minimization (ERM) principle is proposed to minimize risk functional in the traditional machine learning method,(2)Rempα=1n∑i=1nyi-fxi,α2.ERM has been widely used in the least square method for regression problem, maximum likelihood method for probability density estimation, and neural network learning [21]. But ERM is based on the large number theorem, completely dependent on the sum of squared error of the samples. With less samples, small empirical risk cannot guarantee minimizing expected risk. There may happen oversmoothing for the RBF and neural network, which is based on ERM principle [22]: little training error for sample points but large testing error for other points.


## 2.2. Structural Risk Minimization Principle

Generalization ability means the capacity to predict or estimate unknown phenomenon by machine learning. ERM principle cannot guarantee minimizing expected risk. Therefore, SRM principle is proposed based on the statistical learning theory. It provides the relationship between empirical risk and real risk, which is known as the bound of the generalization ability. SRM principle divides the real risk of machine learning into empirical risk and confidence interval [4]:(3)Rα≤Rempα+Φhn.h is the Vapnik-Chervonenkis (VC) dimension, which is the most important theoretical basis of the statistical learning theory. It defines the capacity of the function set and reflects the generalization ability of the machine learning. It is the best descriptive indicators for the capacity of function set learning until now.From (3), we know ERM principle is unconscionable with limited samples. When h/n becomes larger, empirical risk decreases, but real risk does not always become lower. When the dimension of samples is fixed, reduced h (VC) may decrease confidence interval, so the empirical risk comes close to real risk. In this case, less empirical risk stands for less expected risk. Increasing the capacity of samples n also decreases confidence interval to reduce real risk, but it is difficult to get plenty of samples because of the limitation of computational cost.Therefore, decreasing VC dimension is the most suitable way to reduce real risk. For a given observation set, SRM principle chooses the proper function(y-f(x,αnk))2 from the subset Sk with minimized risk to minimize the empirical risk. So SRM principle is a compromise method to balance the fitting precision and the complexity of fitting function.


## 2.3. SRM Index Based on Smoothness Measure

Smoothness measure is used to evaluate the generalization ability. In the projection space formed by kernel function, smoothness measure has its more direct and natural definition. It can be described as the norm of Hilbert space of the kernel function, which is defined as follows.Every functionf(x) in Hilbert space of kernel function K can be expressed as(4)fx=∑i=1nαikx,xi.Then the norm off(x) in Hilbert space is(5)fR2=∑i=1nαi2λi.λi is the eigenvalue of kernel K. The generalization ability can be measured by evaluating W2. The optimal regression problem becomes(6)minW2s.t.Wϕ+b=Y,where W2 is the smoothness measure of sample points.


## 2.4. Mercer Theorem

The criteria to determine whether a function is kernel function is as follows.I is the domain of the independent variables u and u′. There is a nonnegative real function K(u,u′); it is continuous in I×I. The sufficient and necessary condition for K(u,u′) to be expressed as a series(7)Ku,u′=∑iλiψiuψiu′is that all functions satisfy ∫g2(u)du<∞,g≠0 and also satisfy the following condition:(8)∬ku,u′gugu′dudu′>0,where the series is convergence in I×I. λi is the eigenvalue (λi>0) of K(u,u′). ψi(u) is the eigenvector of K(u,u′). A function satisfying the above definition is known as Mercer kernel and can be expressed as the inner product in Hilbert space,(9)Ku,u′=k·,u,k·,u′=ϕu,ϕu′,where , is the operator of inner product. ϕ:u→k(·,u) is a nonlinear transformation of kernel function.


## 3. Kriging Based on SRM Principle

### 3.1. Kriging Model

Kriging model includes two parts:(10)yx=fTxβ+zx,where fT(x)β is a polynomial regression model and z(x) is a deviation model. The covariance matrix of z(x) is(11)CovZxi,Zxj=σ2Kθ,xi,xj,where  K is the correlative coefficient matrix. K(θ,xi,xj) is the space correlation function of any two sample points xi and xj in sample points set. It plays a decisive role to fitting precision. θ is the parameters of K. dk=xi(k)-xj(k) and the common correlation functions include the following [23].Exponential function is(12)Kθk,dk=exp⁡-θkdk.Gaussian function is(13)Kθk,dk=exp-θkdk2.Linear function is(14)Kθk,dk=max⁡0,1-θkdk.Spherical function is(15)Kθk,dk=1-1.5ξk+0.5ξk3,ξk=min⁡1,θkdk.Cube function is(16)Kθk,dk=1-3ξk2+2ξk3,ξk=min⁡1,θkdk.Applying the linear combination of the responsey^ of known samples to predict the response y(x) of any given samples,(17)y^=CTY.The interpolated coefficients of Kriging were obtained by the condition of unbiasedness and the principle of minimum variation. The mean value of error must be zero with minimized varianceσ2 to guarantee the unbiasedness of the fitting process. The following is the corresponding Lagrangian minimization problem:(18)LC,λ=σ21+CTKC-2CTr-λTFTC-f.The parameter of correlative coefficient matrixK can be obtained by minimizing the Maximum Likelihood Estimation (MLE) [23].


### 3.2. Kriging Process

The output of Kriging model can be expressed as(19)y^=fTβ∗+rTK-1Y-Fβ∗,(20)rT=Kx,x1,Kx,x2,…,Kx,xn.From Section2.4, if K(x,xi) satisfies Mercer theorem, it can be expressed as an inner product in Hilbert space(21)Kx,xi=ϕx,ϕxi.Equation (19) is replaced with(22)y^=fTβ∗+ϕx·∑i=1nϕxi·K-1Y-Fβ∗.Equation (22) is simplified as(23)y^=Wϕx+b,where(24)W=∑i=1nϕxi·K-1Y-Fβ∗,b=fTβ∗.The transformed equation (23) has the same form as the kernel function of SVR. It constructs a constant nonlinear mapping ϕ:Rn→Z from input space Rn to feature space Z. Input variable xi has been mapped into feature space Z, and a linear model is generated for linear learning in feature space. The correlative coefficient matrix K(θ,xi,xj) of Kriging has the same mean to the kernel matrix of SVR; it has also been called kernel matrix in this paper. Then W2 can be used to evaluate the smoothness of the fitting function; it has been called smoothness measure. From the above derivation we knew that SRM principle also can be used to minimize the VC dimension of Kriging model to obtain minimized expected risk [4].Kernel matrixK must be positive definite symmetric matrix or conditioned positive definite symmetric matrix. The kernel matrix formed by Gauss basis function satisfies Mercer theorem. There are also many other functions which can be found in [24].


### 3.3. Anisotropic Kernel Function

The basis function of Kriging can be expressed as(25)φc,r=fcj,ri,j,i,j=1,2,…,n.The parameterc of basis function means the contribution rate of every sample point to Kriging model. For a problem with n samples, the size of parameter c also is n. As the size of samples increases, the scale of the kernel parameter optimization problem also becomes larger. These may remarkably increase Kriging modeling time. Usually the parameter cj is considered as the same value for all samples; the basis function reduces to an isotropic model. This simplification reduces the design space of Kriging model and weakens fitting effect. The following anisotropic kernel function with different contribution by each dimension is used to balance the fitting effect and computational cost: (26)φθ,r=frθti,j,t=1,2,…,m,i,j=1,2,…,n,rθti,j=∑t=1mθtxit-xjt2,where m is the dimension of sample x. θt is the contribution of the tth dimension. This improved kernel function has the advantage of constant scale of kernel parameter optimization problem for a given problem with any size of samples.


### 3.4. SRM-Kriging

The kernel parameter optimization problem of Kriging model with smoothness measure is(27)minRθ=W2,where θ is the kernel parameter. W2 is the smoothness measure as objective function:(28)W2=WTW=K-1Y-Fβ∗T∑i=1nϕxiT∑i=1nϕxi·K-1Y-Fβ∗=Y-Fβ∗T·K-1·Y-Fβ∗,where β∗ is the generalized least squares solution of the polynomial problem Fβ=Y. The derivative of W2 for θ is(29)∂W22θ∂θ=-Y-Fβ∗T·K-1∂K∂θK-1·Y-Fβ∗.Letα=K-1(Y-Fβ∗); the derivative can be(30)∂W22θ∂θ=-αT∂K∂θα.The derivative for eachθt is(31)∂Ki,j∂θt=dKi,jdr·∂r∂θt=dkxit,xjtdr·xit-xjt22r,j≠i,0,j=i.Then we get(32)∂W22θt∂θt=-αT·dkxit,xjtdr·xit-xjt22ri=1,…,nj=1,…,n·α,j≠i,0,j=i,where (i,j) is the matrix composed by every derivative value at (i,j). dk/dr is the derivative of basis function for the distance of samples. Based on this derivative information, the efficient Sequential Quadratic Programming (SQP) method is used to optimize the kernel parameter optimization problem of Kriging model.


## 3.1. Kriging Model

Kriging model includes two parts:(10)yx=fTxβ+zx,where fT(x)β is a polynomial regression model and z(x) is a deviation model. The covariance matrix of z(x) is(11)CovZxi,Zxj=σ2Kθ,xi,xj,where  K is the correlative coefficient matrix. K(θ,xi,xj) is the space correlation function of any two sample points xi and xj in sample points set. It plays a decisive role to fitting precision. θ is the parameters of K. dk=xi(k)-xj(k) and the common correlation functions include the following [23].Exponential function is(12)Kθk,dk=exp⁡-θkdk.Gaussian function is(13)Kθk,dk=exp-θkdk2.Linear function is(14)Kθk,dk=max⁡0,1-θkdk.Spherical function is(15)Kθk,dk=1-1.5ξk+0.5ξk3,ξk=min⁡1,θkdk.Cube function is(16)Kθk,dk=1-3ξk2+2ξk3,ξk=min⁡1,θkdk.Applying the linear combination of the responsey^ of known samples to predict the response y(x) of any given samples,(17)y^=CTY.The interpolated coefficients of Kriging were obtained by the condition of unbiasedness and the principle of minimum variation. The mean value of error must be zero with minimized varianceσ2 to guarantee the unbiasedness of the fitting process. The following is the corresponding Lagrangian minimization problem:(18)LC,λ=σ21+CTKC-2CTr-λTFTC-f.The parameter of correlative coefficient matrixK can be obtained by minimizing the Maximum Likelihood Estimation (MLE) [23].


## 3.2. Kriging Process

The output of Kriging model can be expressed as(19)y^=fTβ∗+rTK-1Y-Fβ∗,(20)rT=Kx,x1,Kx,x2,…,Kx,xn.From Section2.4, if K(x,xi) satisfies Mercer theorem, it can be expressed as an inner product in Hilbert space(21)Kx,xi=ϕx,ϕxi.Equation (19) is replaced with(22)y^=fTβ∗+ϕx·∑i=1nϕxi·K-1Y-Fβ∗.Equation (22) is simplified as(23)y^=Wϕx+b,where(24)W=∑i=1nϕxi·K-1Y-Fβ∗,b=fTβ∗.The transformed equation (23) has the same form as the kernel function of SVR. It constructs a constant nonlinear mapping ϕ:Rn→Z from input space Rn to feature space Z. Input variable xi has been mapped into feature space Z, and a linear model is generated for linear learning in feature space. The correlative coefficient matrix K(θ,xi,xj) of Kriging has the same mean to the kernel matrix of SVR; it has also been called kernel matrix in this paper. Then W2 can be used to evaluate the smoothness of the fitting function; it has been called smoothness measure. From the above derivation we knew that SRM principle also can be used to minimize the VC dimension of Kriging model to obtain minimized expected risk [4].Kernel matrixK must be positive definite symmetric matrix or conditioned positive definite symmetric matrix. The kernel matrix formed by Gauss basis function satisfies Mercer theorem. There are also many other functions which can be found in [24].


## 3.3. Anisotropic Kernel Function

The basis function of Kriging can be expressed as(25)φc,r=fcj,ri,j,i,j=1,2,…,n.The parameterc of basis function means the contribution rate of every sample point to Kriging model. For a problem with n samples, the size of parameter c also is n. As the size of samples increases, the scale of the kernel parameter optimization problem also becomes larger. These may remarkably increase Kriging modeling time. Usually the parameter cj is considered as the same value for all samples; the basis function reduces to an isotropic model. This simplification reduces the design space of Kriging model and weakens fitting effect. The following anisotropic kernel function with different contribution by each dimension is used to balance the fitting effect and computational cost: (26)φθ,r=frθti,j,t=1,2,…,m,i,j=1,2,…,n,rθti,j=∑t=1mθtxit-xjt2,where m is the dimension of sample x. θt is the contribution of the tth dimension. This improved kernel function has the advantage of constant scale of kernel parameter optimization problem for a given problem with any size of samples.


## 3.4. SRM-Kriging

The kernel parameter optimization problem of Kriging model with smoothness measure is(27)minRθ=W2,where θ is the kernel parameter. W2 is the smoothness measure as objective function:(28)W2=WTW=K-1Y-Fβ∗T∑i=1nϕxiT∑i=1nϕxi·K-1Y-Fβ∗=Y-Fβ∗T·K-1·Y-Fβ∗,where β∗ is the generalized least squares solution of the polynomial problem Fβ=Y. The derivative of W2 for θ is(29)∂W22θ∂θ=-Y-Fβ∗T·K-1∂K∂θK-1·Y-Fβ∗.Letα=K-1(Y-Fβ∗); the derivative can be(30)∂W22θ∂θ=-αT∂K∂θα.The derivative for eachθt is(31)∂Ki,j∂θt=dKi,jdr·∂r∂θt=dkxit,xjtdr·xit-xjt22r,j≠i,0,j=i.Then we get(32)∂W22θt∂θt=-αT·dkxit,xjtdr·xit-xjt22ri=1,…,nj=1,…,n·α,j≠i,0,j=i,where (i,j) is the matrix composed by every derivative value at (i,j). dk/dr is the derivative of basis function for the distance of samples. Based on this derivative information, the efficient Sequential Quadratic Programming (SQP) method is used to optimize the kernel parameter optimization problem of Kriging model.


## 4. Test Cases

Nine standard test functions with different dimension are used to validate the effectiveness of the proposed method. The improved Kriging model also has been evaluated by sample points with uniform distribution and nonuniform distribution.


### 4.1. Standard Test Functions

There are nine standard test functions with different dimension.Fun1 is(33)fx=11+3x2.Fun2 is(34)fx=sin⁡xx.Fun3 is(35)fx1,x2=30+xsin⁡x4+e-y2.Fun4 is(36)fx1,x2=2x12-1.05x14+x166-x1x2-x22.Fun5 is(37)fx=x12+x22+x1x2-14x1-16x2+x3-102+4x4-52+x5-32+2x6-12+5x72+7x8-112+2x9-102+x10-72+45.Fun6 is(38)fx=∑i=110xici+ln⁡xix1+⋯+x10,x∈1,5c=-6.089,-17.164,-34.054,-5.914,-24.721,-14.986,-24.100,-10.708,-26.662,-22.179.Fun7 is(39)fx=∑i=116∑j=116ai,jxi2+xi+1xj2+xj+1.Fun8 is(40)fx=∑i=130xi2.Fun9 is(41)fx=1100∑i=130100xi2-xi+12+xi-12.These functions are from one dimension to thirty dimensions with minor different nonlinearity, which can be used to evaluate the performance of the surrogate model under different scales of variables.These test functions are divided into three scales according to their dimension: small scale, middle scale, and large scale. The sampling information of the above functions is in Table1.Table 1
Sampling information of the test functions.
Test function
Dimension
Bound
Samples size
Test sample size
Scale
Fun1
1
[
-
6,6
]
25
600
Small
Fun2
1
[
-
12,12
]
25
600
Small
Fun3
2
[
-
3,3
]
50
1 000
Small
Fun4
2
[
-
5,5
]
50
1 000
Small
Fun5
10
[
-
10,11
]
300
2 000
Middle
Fun6
10
[
1,5
]
300
2 000
Middle
Fun7
16
[
-
1,1
]
400
3 000
Middle
Fun8
30
[
0,10
]
600
3 000
Large
Fun9
30
[
-
5,5
]
600
3 000
LargeKriging with SRM (SRM-Kriging), Kriging with MLE (MLE-Kriging), and Kriging with constant kernel parameter (CON-Kriging) are three methods tested using the above test cases. Gauss function is chosen as the kernel function for all the three methods. Multiple correlation coefficientR2 and root-mean-square error (RMSE) are used to evaluate the fitting precision.


### 4.2. Samples with Uniform Distribution

The optimal Latin hypercube sampling method is used to generate uniform distributed samples. Figure1 and Table 2 are the comparison of the generalization ability between different test functions with uniform distribution sampling. It is observed that, as the dimension of variables increases, the fitting precision of CON-Kriging decreases rapidly, especially for the large-scale cases (Fun7~Fun9); MLE-Kriging has better fitting precision than CON-Kriging via using optimized kernel parameter, but still with poor fitting precision for large-scale cases. SRM-Kriging has the best fitting precision than CON-Kriging and MLE-Kriging; even for the large-scale cases, it also has the smallest RMSE.Table 2
Comparison of the generalization ability between different test functions with uniform distribution sampling.
Test functions
R
2
RMSE
CON-Kriging
SRM-Kriging
MLE-Kriging
CON-Kriging
SRM-Kriging
MLE-Kriging
Fun1
Failure
0.999 39
0.672 59
0.372 85
0.005 82
0.134 61
Fun2
1.000 00
1.000 00
1.000 00
0.000 00
0.000 00
0.000 00
Fun3
0.988 30
0.991 93
0.999 20
1.197 99
0.994 75
0.313 37
Fun4
0.950 40
0.980 54
0.979 91
103.012 78
64.513 05
65.463 16
Fun5
0.894 21
0.993 22
0.986 45
326.976 85
82.749 71
117.016 48
Fun6
0.999 93
0.999 99
0.999 98
0.694 53
0.263 94
0.341 66
Fun7
0.723 31
0.969 00
0.899 10
11.309 11
3.783 29
6.824 92
Fun8
Failure
0.931 03
0.105 14
168.556 75
42.903 68
154.449 27
Fun9
Failure
0.755 16
0.179 30
1 010.540 9
499.434 93
913.770 52Figure 1
Comparison ofR2 between different scale of test functions with uniform distribution sampling.Figure2 shows the fitting results of the Fun1 and Fun2 under uniform distribution sampling. From the fitting result of Fun1 we knew that CON-Kriging and MLE-Kriging have poor fitting precision, especially for CON-Kriging; it had oscillated badly. This illustrated kernel function optimization is very important for improving fitting precision of surrogate model. Compared with CON-Kriging and MLE-Kriging, SRM-Kriging is almost closed to original function. It has better generalization ability than others.Figure 2
Fitting results of the Fun1 and Fun2 with uniform distribution sampling.
(a)
Fitting result of Fun1 with uniform distribution
(b)
Fitting result of Fun2 with uniform distribution


### 4.3. Samples with Nonuniform Distribution

In a practical application, the samples are always not satisfied uniform distribution standard. This will do some impact to surrogate model building. A typical example is the sequential surrogate model [25]; as adding points goes on, the distribution of samples changes continuously; an aggregation distribution of samples has appeared. The performance of surrogate model with nonuniform distribution samples is also important. At present there are a few researches for surrogate model with nonuniform distribution samples [26]. In this section, normal distribution is used to imitate and generate nonuniform distribution samples. The above three surrogate models are tested using these nonuniform distribution points. The mean and standard deviation of the normal distribution of test functions are listed in Table 3.Table 3
Mean and standard deviation of the normal distribution of test functions.
Test functions
Dimension
Bound
Mean
Standard deviation
Fun1
1
[
-
6,6
]
0.0
3.0
Fun2
1
[
-
12,12
]
0.0
6.0
Fun3
2
[
-
3,3
]
0.0
1.5
Fun4
2
[
-
5,5
]
0.0
2.5
Fun5
10
[
-
10,11
]
0.0
5.0
Fun6
10
[
1,5
]
3.0
1.0
Fun7
16
[
-
1,1
]
0.0
0.25
Fun8
30
[
-
10,10
]
0.0
5.0
Fun9
30
[
-
5,5
]
0.0
2.530 times of random normal distribution sampling are generated to test these three methods and the average value ofR2 and RMSE is used to make a comparison. Figure 3 and Table 4 are the comparison of the generalization ability between different test functions with nonuniform distribution sampling.Table 4
Comparison of the generalization ability between different test functions with nonuniform distribution sampling.
Test functions
R
2
RMSE
CON-Kriging
SRM-Kriging
MLE-Kriging
CON-Kriging
SRM-Kriging
MLE-Kriging
Fun1
Failure
0.955 64
failure
985.97 356
0.049 79
0.564 43
Fun2
1.000 00
0.999 99
0.999 99
0.000 15
0.000 83
0.000 79
Fun3
0.886 67
0.957 91
0.948 37
3.753 33
2.293 57
2.347 32
Fun4
0.474 05
0.983 03
0.999 97
336.163 41
60.180 79
2.715 26
Fun5
0.882 14
0.996 40
0.997 44
354.549 56
61.200 31
61.109 76
Fun6
0.999 90
0.999 98
0.999 99
0.840 81
0.378 94
0.318 47
Fun7
0.424 08
0.747 45
0.677 50
21.787 80
11.903 82
13.896 88
Fun8
Failure
0.338 04
0.009 51
457.409 39
190.136 04
394.725 25
Fun9
0.004 02
0.135 85
0.054 74
2 385.597 16
1 506.804 29
1 889.119 78Figure 3
Comparison ofR2 between different scale of test functions with nonuniform distribution sampling.It is observed that, for the test cases with nonuniform distribution sampling, the fitting precision of all methods decreased remarkably, especially for large-scale problem. CON-Kriging has very poor performance, even for small-scale problems. MLE-Kriging is better than CON-Kriging. SRM-Kriging still has the best fitting precision than the others. Although all three methods are failed for large-scale problems, but SRM-Kriging still has the smallest RMSE.Figure4 shows the fitting results of the Fun1 and Fun2 with nonuniform distribution sampling. From the fitting result we knew that all fitting curves for Fun1 and Fun2 are distorted. For Fun1, the fitting curves generated by CON-Kriging and MLE-Kriging even cannot fit the basic trends; SRM-Kriging obviously is better than the others and is in line with the basic trends except the left region with little samples. It has been proved that generalization ability is more suitable for evaluating comprehensive performance of surrogate model. For Fun2, three methods are all distorted at the right region with less samples, but the fitting performance of SRM-Kriging and MLE-Kriging is much better than CON-Kriging, which proves the importance of kernel parameter optimization.Figure 4
The fitting results of the Fun1 and Fun2 with nonuniform distribution sampling.
(a)
The fitting result of Fun1 with nonuniform distribution
(b)
The fitting result of Fun2 with nonuniform distribution


## 4.1. Standard Test Functions

There are nine standard test functions with different dimension.Fun1 is(33)fx=11+3x2.Fun2 is(34)fx=sin⁡xx.Fun3 is(35)fx1,x2=30+xsin⁡x4+e-y2.Fun4 is(36)fx1,x2=2x12-1.05x14+x166-x1x2-x22.Fun5 is(37)fx=x12+x22+x1x2-14x1-16x2+x3-102+4x4-52+x5-32+2x6-12+5x72+7x8-112+2x9-102+x10-72+45.Fun6 is(38)fx=∑i=110xici+ln⁡xix1+⋯+x10,x∈1,5c=-6.089,-17.164,-34.054,-5.914,-24.721,-14.986,-24.100,-10.708,-26.662,-22.179.Fun7 is(39)fx=∑i=116∑j=116ai,jxi2+xi+1xj2+xj+1.Fun8 is(40)fx=∑i=130xi2.Fun9 is(41)fx=1100∑i=130100xi2-xi+12+xi-12.These functions are from one dimension to thirty dimensions with minor different nonlinearity, which can be used to evaluate the performance of the surrogate model under different scales of variables.These test functions are divided into three scales according to their dimension: small scale, middle scale, and large scale. The sampling information of the above functions is in Table1.Table 1
Sampling information of the test functions.
Test function
Dimension
Bound
Samples size
Test sample size
Scale
Fun1
1
[
-
6,6
]
25
600
Small
Fun2
1
[
-
12,12
]
25
600
Small
Fun3
2
[
-
3,3
]
50
1 000
Small
Fun4
2
[
-
5,5
]
50
1 000
Small
Fun5
10
[
-
10,11
]
300
2 000
Middle
Fun6
10
[
1,5
]
300
2 000
Middle
Fun7
16
[
-
1,1
]
400
3 000
Middle
Fun8
30
[
0,10
]
600
3 000
Large
Fun9
30
[
-
5,5
]
600
3 000
LargeKriging with SRM (SRM-Kriging), Kriging with MLE (MLE-Kriging), and Kriging with constant kernel parameter (CON-Kriging) are three methods tested using the above test cases. Gauss function is chosen as the kernel function for all the three methods. Multiple correlation coefficientR2 and root-mean-square error (RMSE) are used to evaluate the fitting precision.


## 4.2. Samples with Uniform Distribution

The optimal Latin hypercube sampling method is used to generate uniform distributed samples. Figure1 and Table 2 are the comparison of the generalization ability between different test functions with uniform distribution sampling. It is observed that, as the dimension of variables increases, the fitting precision of CON-Kriging decreases rapidly, especially for the large-scale cases (Fun7~Fun9); MLE-Kriging has better fitting precision than CON-Kriging via using optimized kernel parameter, but still with poor fitting precision for large-scale cases. SRM-Kriging has the best fitting precision than CON-Kriging and MLE-Kriging; even for the large-scale cases, it also has the smallest RMSE.Table 2
Comparison of the generalization ability between different test functions with uniform distribution sampling.
Test functions
R
2
RMSE
CON-Kriging
SRM-Kriging
MLE-Kriging
CON-Kriging
SRM-Kriging
MLE-Kriging
Fun1
Failure
0.999 39
0.672 59
0.372 85
0.005 82
0.134 61
Fun2
1.000 00
1.000 00
1.000 00
0.000 00
0.000 00
0.000 00
Fun3
0.988 30
0.991 93
0.999 20
1.197 99
0.994 75
0.313 37
Fun4
0.950 40
0.980 54
0.979 91
103.012 78
64.513 05
65.463 16
Fun5
0.894 21
0.993 22
0.986 45
326.976 85
82.749 71
117.016 48
Fun6
0.999 93
0.999 99
0.999 98
0.694 53
0.263 94
0.341 66
Fun7
0.723 31
0.969 00
0.899 10
11.309 11
3.783 29
6.824 92
Fun8
Failure
0.931 03
0.105 14
168.556 75
42.903 68
154.449 27
Fun9
Failure
0.755 16
0.179 30
1 010.540 9
499.434 93
913.770 52Figure 1
Comparison ofR2 between different scale of test functions with uniform distribution sampling.Figure2 shows the fitting results of the Fun1 and Fun2 under uniform distribution sampling. From the fitting result of Fun1 we knew that CON-Kriging and MLE-Kriging have poor fitting precision, especially for CON-Kriging; it had oscillated badly. This illustrated kernel function optimization is very important for improving fitting precision of surrogate model. Compared with CON-Kriging and MLE-Kriging, SRM-Kriging is almost closed to original function. It has better generalization ability than others.Figure 2
Fitting results of the Fun1 and Fun2 with uniform distribution sampling.
(a)
Fitting result of Fun1 with uniform distribution
(b)
Fitting result of Fun2 with uniform distribution


## 4.3. Samples with Nonuniform Distribution

In a practical application, the samples are always not satisfied uniform distribution standard. This will do some impact to surrogate model building. A typical example is the sequential surrogate model [25]; as adding points goes on, the distribution of samples changes continuously; an aggregation distribution of samples has appeared. The performance of surrogate model with nonuniform distribution samples is also important. At present there are a few researches for surrogate model with nonuniform distribution samples [26]. In this section, normal distribution is used to imitate and generate nonuniform distribution samples. The above three surrogate models are tested using these nonuniform distribution points. The mean and standard deviation of the normal distribution of test functions are listed in Table 3.Table 3
Mean and standard deviation of the normal distribution of test functions.
Test functions
Dimension
Bound
Mean
Standard deviation
Fun1
1
[
-
6,6
]
0.0
3.0
Fun2
1
[
-
12,12
]
0.0
6.0
Fun3
2
[
-
3,3
]
0.0
1.5
Fun4
2
[
-
5,5
]
0.0
2.5
Fun5
10
[
-
10,11
]
0.0
5.0
Fun6
10
[
1,5
]
3.0
1.0
Fun7
16
[
-
1,1
]
0.0
0.25
Fun8
30
[
-
10,10
]
0.0
5.0
Fun9
30
[
-
5,5
]
0.0
2.530 times of random normal distribution sampling are generated to test these three methods and the average value ofR2 and RMSE is used to make a comparison. Figure 3 and Table 4 are the comparison of the generalization ability between different test functions with nonuniform distribution sampling.Table 4
Comparison of the generalization ability between different test functions with nonuniform distribution sampling.
Test functions
R
2
RMSE
CON-Kriging
SRM-Kriging
MLE-Kriging
CON-Kriging
SRM-Kriging
MLE-Kriging
Fun1
Failure
0.955 64
failure
985.97 356
0.049 79
0.564 43
Fun2
1.000 00
0.999 99
0.999 99
0.000 15
0.000 83
0.000 79
Fun3
0.886 67
0.957 91
0.948 37
3.753 33
2.293 57
2.347 32
Fun4
0.474 05
0.983 03
0.999 97
336.163 41
60.180 79
2.715 26
Fun5
0.882 14
0.996 40
0.997 44
354.549 56
61.200 31
61.109 76
Fun6
0.999 90
0.999 98
0.999 99
0.840 81
0.378 94
0.318 47
Fun7
0.424 08
0.747 45
0.677 50
21.787 80
11.903 82
13.896 88
Fun8
Failure
0.338 04
0.009 51
457.409 39
190.136 04
394.725 25
Fun9
0.004 02
0.135 85
0.054 74
2 385.597 16
1 506.804 29
1 889.119 78Figure 3
Comparison ofR2 between different scale of test functions with nonuniform distribution sampling.It is observed that, for the test cases with nonuniform distribution sampling, the fitting precision of all methods decreased remarkably, especially for large-scale problem. CON-Kriging has very poor performance, even for small-scale problems. MLE-Kriging is better than CON-Kriging. SRM-Kriging still has the best fitting precision than the others. Although all three methods are failed for large-scale problems, but SRM-Kriging still has the smallest RMSE.Figure4 shows the fitting results of the Fun1 and Fun2 with nonuniform distribution sampling. From the fitting result we knew that all fitting curves for Fun1 and Fun2 are distorted. For Fun1, the fitting curves generated by CON-Kriging and MLE-Kriging even cannot fit the basic trends; SRM-Kriging obviously is better than the others and is in line with the basic trends except the left region with little samples. It has been proved that generalization ability is more suitable for evaluating comprehensive performance of surrogate model. For Fun2, three methods are all distorted at the right region with less samples, but the fitting performance of SRM-Kriging and MLE-Kriging is much better than CON-Kriging, which proves the importance of kernel parameter optimization.Figure 4
The fitting results of the Fun1 and Fun2 with nonuniform distribution sampling.
(a)
The fitting result of Fun1 with nonuniform distribution
(b)
The fitting result of Fun2 with nonuniform distribution


## 5. Conclusion

More and more attention is paid to the improvement of fitting precision as the widely spread and applied surrogate model. This paper studies the assessment criteria of surrogate model and discusses the importance of the generalization ability. Based on these, the kernel function optimization method is proposed to improve the fitting precision. Some standard benchmarks are tested to verify the effectiveness of the improved method based on sample points with uniform distribution and nonuniform distribution. In conclusion, we have the following.(1)
SRM-Kriging method based on anisotropic kernel function optimization is proposed to replace the coefficient of each samples with the coefficient of the component of samples. The computational cost is reduced.(2)
A comparison is carried out among CON-Kriging, SRM-Kriging, and MLE-Kriging. Results show that kernel function optimization is very important to improve the fitting precision of surrogate model. SRM principle provides a more effective evaluation for the generalization ability. SRM-Kriging has better performance than the others, especially for problems with nonuniform distribution sampling.(3)
From the Kriging process we know that Kriging can be regarded as the special case of the SVR with zero empirical risk. By setting the insensitive loss function of SVR to zero, the SVR optimization problem with regularization parameters and kernel parameters is transformed to a simple optimization problem with independent kernel parameters and a certain problem to evaluate coefficients. It reduces the design dimension and simplifies the implement process.The distribution of samples may become more and more complicated. This paper also studies the fitting precision of Kriging model with uniform sampling and nonuniform sampling. The results show that Kriging with SRM principle has better fitting performance, but still with low precision for high dimension problems. Further research may perform full analysis of surrogate model with different nonuniform distribution sampling to find more suitable selection mechanism of the kernel function and efficient kernel parameter optimization method.



---

*Source: 3021950-2017-09-11.xml*