## 5 Using Advanced Supervised ML Regression Models for a Better Fit  
In an attempt to enhance the accuracy of the predictive model and, possibly, reduce the heteroscedasticity present in the models from the previous section, we utilized a comprehensive range of supervised ML regression algorithms. These included various types of regression trees (fine, medium, and coarse) [34], support vector machines with diverse kernels such as quadratic, cubic, and Gaussian kernels of varying coarseness [35], Gaussian process (GP) regression models with different kernels [36], kernel approximation regression models, ensembles of trees [37], and neural networks [38]. We obtained the best results using the GP regression models followed closely by the neural network approach whereas other ML algorithms did not prove to be suitable for our interpolation problem. Note that we performed a random search procedure to optimize the selection of suitable hyperparameters for each tested ML regression model. Therefore, we decided to provide only the best fit obtained using the GP regression in this chapter to avoid redundancy. The mentioned GP regression is a well-known non-parametric Bayesian approach to regression that utilizes stochastic Gaussian processes and is generally considered an excellent choice for multidimensional interpolation problems. It completely eliminates the usage of the least-squares method in the regression analysis, which could possibly eliminate the previously observed heteroscedasticity of predicted results. In this paper, the GP regression model will be implemented using the scikit-learn library [39]. Basically, the GP regression model applied to any input data as an approximation model can be briefly described as a combination of a prior mean function and a covariance or kernel function:  
\[Y(x)\sim\mathcal{GP}(m(x),k(x,x^{\prime})) \tag{4}\]  
Figure 8: Residuals vs. predicted values obtained using Equation (3): (a) Training set, (b) Test set. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Metric** & **Training Set** & **Test Set** \\ \hline RMSE & 1.380918 & 1.435027 \\ MAE & 1.043982 & 1.101439 \\ \(R^{2}\) & 0.998735 & 0.998557 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Regression quality metrics for the second regression function given by Equation (3). where \(m(x)\) represents the mean function, \(k(x,x^{\prime})\) represents the covariance (kernel) function, and \((x,x^{\prime})\) represents the input data points. In our analysis, we assumed that the prior mean is the training data's mean. The kernel function \(k\) is practically a function that accepts as input two points in the input space and, as an output, produces how similar these points are, based on some notion of distance between them. A whole array of kernel functions were tested in this paper, including the radial basis function (RBF) kernel, Matern kernel, rational quadratic kernel, and Exp-Sine-Squared kernel, but the best results were obtained using the non-isotropic Matern kernel function. The Matern covariance (kernel) function [40] in general represents a popular choice in GP regression analysis due to its flexibility in controlling the smoothness of the function. Unlike the RBF kernel, it does not over-smooth the predicted values. Moreover, the non-isotropic version of it can additionally handle different length scales for different dimensions. The general form of the Matern kernel function is represented by the following general term:  
\[k(x,x^{\prime})=\sigma^{2}\cdot\frac{2^{1-\
u}}{\Gamma(\
u)}\cdot\left(\frac{ \sqrt{2\cdot\
u}}{l}\cdot|x-x^{\prime}|\right)^{\
u}\cdot K_{\
u}\left(\frac{ \sqrt{2\cdot\
u}}{l}\cdot|x-x^{\prime}|\right) \tag{5}\]  
where \(\sigma\) is the kernel variance, \(l\) is the length-scale parameter, \(|x-x^{\prime}|\) is the Euclidean distance between input points, \(K_{\
u}\) is a modified Bessel function of the second kind or order \(\
u\), and \(\Gamma\) is the Gamma function. Modification of the \(\
u\) parameter significantly changes the kernel function.