## 3 Materials and Methods
### Model Training
#### 3.2.2 XGBoost  
The XGBoost algorithm was created to find better ways to enhance decision trees. Initially, it was designed as a self-contained program that generated prediction models from input data. However, XGBoost's integration with standard interface systems enabled it to evolve into a more robust package that utilizes computational resources to produce accurate predictions in less time [48]. Boosting is an assembly method that combines different weak models into a robust model to minimize training errors. Usually, weak models tend to overfit, so having several additive models allows for a better model. In boosting, a random sample of data is selected, and a model is applied and trained sequentially, where each model tries to compensate for the weak points of the previous one [49], as demonstrated in Figure 6.  
Figure 5: Structural overview of LSTM. XGBoost is a variation of the gradient boosting algorithm that adds predictors to an array in sequence to correct errors from previous predictors. This method combines gradient descent and boosting. XGBoost uses this method with minor modifications that improve the regularization target function. Considering a database \(D\) with \(m\) features and \(n\) number of examples \(D=\{(x_{i},y_{i})\}(D=n_{i}\in\mathbb{R}^{m},y_{i}\in\mathbb{R}^{n})\), XGBoost can be used to create an ensemble tree model that uses \(K\) additive functions to predict the output \(\hat{y}_{i}\), as expressed in Equation (11) [50]:  
\[\hat{y}_{i}=\phi(x_{i})=\sum_{k=1}^{K}f_{k}(x_{i}),\quad f_{k}\in\mathcal{F}, \tag{11}\]  
where \(\mathcal{F}=\{f(x)=w_{q(x)}\}(q:\mathbb{R}^{m}\to T,w\in\mathbb{R}^{T})\) represents the Classification and Regression Trees (CARTs). In this context, \(q\) refers to the structure of a tree that maps an instance to a specific leaf index. \(T\) represents the total number of leaves on each tree. Meanwhile, \(f_{k}\) pertains to a distinct tree structure with leaf weights \(w\). \(K\) denotes the number of trees utilized in the model. Equation (11) can be solved by finding the best set of functions by minimizing the regularization function and cost, as expressed in Equation (12):  
\[ \begin{split}\mathcal{L}(\phi)&=\sum_{i}l(\hat{y}_{ i},y_{i})+\sum_{k}\Omega(f_{k})\\ \text{where}&\Omega(f)=\gamma T+\frac{1}{2}\lambda \parallel w\parallel^{2}\end{split}  \tag{12}\]  
In Equation (12), \(l\) stands for the cost function, which calculates the difference between the predicted value \(\hat{y}_{i}\) and the target value \(y_{i}\). \(\Omega\) penalizes the model's complexity, which helps avoid overfitting by smoothing out the final learned weights. The objective function will return to traditional gradient tree boosting if the regularization term is zero. The tree ensemble model described in Equation (11) contains functions treated as parameters that cannot be optimized using traditional methods in a Euclidean space. To continue training after adding a new function \(f\) to the model, another function (tree) is added at the \(t-th\) iteration, as shown in Equation (13):  
\[\mathcal{L}^{(t)}=\sum_{i=1}^{n}l(y_{i},\hat{y}_{i}^{(t-1)}+  
Defining \(I_{j}=\{i|q(x_{i})=j\}\) as an leaf instance of \(j\), Equation (14) can be rewritten as Equation (15):  
\[\mathcal{L}^{(t)}=\sum_{j=1}^{T}[(\sum_{i\in I_{j}}g_{i})w_{j}+\frac{1}{2}(\sum_ {i\in I_{j}}h_{i}+\lambda)w_{j}^{2}]+\gamma T \tag{15}\]  
For a fixed structure of \(q(x)\), the optimal value of the weight \(w_{j}^{*}\) of leaf \(j\) is obtained with Equation (16):  
\[\mathcal{L}^{(t)}(q)=-\frac{1}{2}\sum_{j=1}^{T}\frac{(\sum_{i\in I_{j}}g_{i})^ {2}}{\sum_{i\in I_{j}}h_{i}+\lambda}+\gamma T \tag{16}\]  
Enumerating all possible tree structures \(q\) is often complicated. Therefore, a greedy algorithm begins with a single leaf and gradually adds branches to the tree. When there is a split, the sets IL and IR represent the instances of the left and right nodes, respectively. I = IL IR represents the instances of the entire tree.