## 3 Development of the Machine-Learning-Driven Surrogate Models
### ML-Driven Surrogate Models  
The predictor function \(\hat{f}(x)\) can be fitted with different ML models. This paper investigates four popular ML regression algorithms that do not impose assumptions on the form of the modeled function, namely neural networks (NN), Gaussian process regressor (GPR), random forest (RF), and gradient boosting trees (GBT). _Neural Network._ The feedforward NN ([33], Chapter 16.5) is composed of stacked layers of neurons in which the weighted sum of the inputs goes through a nonlinear activation function at each neuron. The outputs of each layer are the inputs of the subsequent layer in a feedforward process. The NN error is usually measured by the Euclidean distance between the predictor output \(\hat{f}(x_{i})\) and the target value \(y_{i}\), and the NN parameters, weights and biases are fitted with gradient descent via backpropagation to minimize the prediction error. _Gaussian Process Regressor._ GPR [34] is a probabilistic model that assumes a Gaussian process prior. Therefore, for any finite set of inputs \(\mathbf{x}\):  
\[f(\mathbf{x})\sim\mathcal{N}(\mathbf{\mu},\mathbf{K}), \tag{33}\]  
where \(\mu_{i}(x_{i})\) is the mean function, and \(K_{ij}(x_{i},x_{j})\) is the covariance function, also known as kernel. Given the training dataset \(\{\mathbf{x},\mathbf{y}\}\) and considering that new input-output pairs \(\{x_{*},y_{*}\}\) belong to the same joint Gaussian distribution:  
\[ \begin{bmatrix}\mathbf{y}\\ y_{*}\end{bmatrix} \sim\mathcal{N}\bigg{(} \begin{bmatrix}\mathbf{\mu}\\ \mu_{*}\end{bmatrix} _{\prime} \begin{bmatrix}\mathbf{K}&K_{*}\\ K_{*}^{\top}&K_{**}\end{bmatrix} \bigg{)}, \tag{34}\]  
one can condition the prior distribution on the observations \(p(\mathbf{y}_{*}|\mathbf{y})\) and infer new output values \(y_{*}\), sampling them from the joint posterior distribution. Usually, the mean function is defined as zero, and a Gaussian noise is considered in the observations.The optimization of the kernel hyperparameters is commonly performed to maximize the marginal likelihood of the distribution. _Random Forest._ Decision trees perform recursive partitions of the input space using heuristic methods to select the split feature \(s\) and threshold \(t\) that minimize the impurity criterion \(G(s,t)\). For regression problems, the impurity criterion is commonly based on the mean squared error (MSE) of each subspace created by the partition. RF is an ensemble model made of several decision trees to reduce the variance of single tree predictions, which is achieved by averaging the output of each decision tree in the RF [35]. Each decision tree of the RF is trained with a different subset of the training dataset using the bagging technique to ensure that each predictor is unique and contributes to generalization improvement. _Gradient Boosting Trees._ GBT is an ML model that uses the boosting technique to create an ensemble of decision trees ([33], Chapter 16.4). In GBT, decision trees are trained sequentially, where each decision tree trains on the mispredicted data of the previous one,thus focusing mainly on reducing the predictor bias. The prediction error is differentiable and is minimized using a gradient descent optimizer. After the surrogate is trained, it can predict the system response in new points of the design space and be applied to optimization and uncertainty propagation of the system parameters. Furthermore, global and local sensitivity analyses can be used to increase the black-box interpretability and can either be obtained as a by-product of ML methods or be straightforwardly implemented [36; 37]. The vast literature on sensitivity analyses applied in ML includes methods for NN [38; 39], GPR [40], XGB [41], RF [35] and model-agnostic approaches [37]. In this paper, the mean decrease in impurity (MDI) approach [35] was applied to verify the RF-based surrogate physical consistency.