## 2 Materials and Methods
### Machine Learning Regression
#### 2.3.8 Extreme Gradient Boosting Regression (XGBR)  
XGBR is also an ensemble machine learning algorithm based on the principle of gradient boosting. In the gradient boosting technique, multiple weakly predictive models, which are usually decision trees, are combined to create a stronger ensemble model [87]. The algorithm efficiently handles both classification and regression tasks by generating and working with boosted trees in parallel [75]. The core concept of the XGBR is its objective function which is defined based on the conditions of gradient boosting. For a given dataset D = {(x_i, y_i}) comprising of 'n' samples and'm' features, 'z' additive functions f\({}_{2}\)(x_i), such that \(f_{z}\)\(\epsilon\)\(F\) space of regression trees, are utilized to approximate the predictions made by the ensemble model, as shown in Equation (19).  
\[\dot{y}_{i}=\sum_{z=1}^{Z}f_{z}(x_{i}) \tag{19}\]  
Later, to optimize the ensemble tree and reduce errors, the objective function (Equation (20)) of the XGBR is minimized, where '\(l\)' is the convex loss function utilized to quantify the disparity between actual and predicted values of the target variable and \(\Omega(f_{t})\) is the regularization term. This process is repeated for 't' iterations.  
\[L^{t}=\sum_{i=1}^{n}l\Big{(}y_{i},\dot{y}_{i}^{(t-1)}+f_{t}(x_{i})\Big{)}+ \Omega(f_{t}) \tag{20}\]  
In the regularization term, \(\Omega\) serves as the penalty factor to account for the complexity of the regression tree approach and it is calculated using Equation (21), with the regularization shrinkage parameter '\(\lambda\)' and pruning penalty constant '\(\gamma\)' to control the complexity of the model and prevent overfitting.  
\[\Omega(f_{k})=\gamma T+\frac{1}{2}\lambda\|w\|^{2} \tag{21}\]  
In each of the successive weak tree learners, the prediction error of the prior model is corrected with the goal of finding an optimal output value (\(w\)) at the leaf node and can be expressed as the ratio of the first and the second derivative of the loss function known as the gradient and the hessian, respectively. The loss function is then solved using second-order Taylor's approximation. At every split, the optimal output value is determined by equating the loss function to zero and then the XGBR evaluates split gain (G) to determine the improvement. Split gain is calculated using similarity scores (S) at the root and the leaf nodes using G = S\({}_{\text{LL}}\) + S\({}_{\text{RL}}\) - S\({}_{\text{R}}\), where 'S\({}_{\text{LL}}\)', 'S\({}_{\text{RL}}\)', and 'S\({}_{\text{R}}\)' are the similarity scores computed using Equation (22) for the left leaf node, right leaf node, and root node, respectively [74,88].  
\[S=\frac{1}{2}\frac{\sum_{i=1}^{n}g_{i}^{2}}{\lambda+\sum_{i=1}^{n}h_{i}^{2}} \tag{22}\]  
The procedure is iteratively repeated until reaching the terminal node, where the prediction 'Ti' associated with this specific terminal node is multiplied by the learning rate '\(\epsilon\)'. The resulting value is then added to the previous prediction, effectively enhancing and boosting the predictive power of the model at each successive step, as shown in Equation (23). The fundamental structure of an XGBR model is shown in Figure 10.  
\[\hat{y}=p_{i}+\epsilon T_{i}+\dots\epsilon T_{n}, \tag{23}\]