## 3 Materials and Methods
### Gradient Boosting  
As illustrated in Figure 3, the gradient boosting algorithm is an ensemble learning technique that incrementally improves performance by combining multiple weak learners, typically shallow decision trees. This algorithm performs excellently in both regression and classification tasks and is particularly effective for handling complex nonlinear data. \begin{table}
\begin{tabular}{c c} \hline \hline
**Parameter** & **Value** \\ \hline Batch size & 128 \\ Weight decay & L2 0.001 \\ Epochs & 300 \\ Learning rate & 0.0005 \\ Optimizer & Adam \\ Workers & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: RNN training parameters. The fundamental idea of gradient boosting is to iteratively improve the model by adding new trees that minimize the residual errors at each stage. Initially, a base model \(F_{0}(x)\) is established, and optimized using the value \(\gamma\) that minimizes the loss function \(L(y_{i},\gamma)\) for the given dataset:  
\[F_{0}(x)=arg\sum_{i=1}L(y_{i},\gamma) \tag{5}\]  
where \(F_{0}(x)\) generates the initial predictions, while the loss function \(L\) measures the error between the predicted and actual values. In the first stage, the model learns from the data based on this initial prediction. In each subsequent stage, a new tree is trained using the residuals, which represent the difference between the previous model's predictions and the actual values. The residuals correct the existing model's prediction errors, allowing the newly added tree to refine the overall model performance. These residuals are computed as the gradient of the loss function, mathematically expressed as follows:  
\[r_{im}=-\frac{\partial L(y_{i},F(x_{i}))}{\partial F(x_{i})} \tag{6}\]  
where \(r_{im}\) represents the residual for the \(i\) data point and corresponds to the gradient of the loss function at the m iteration. Using these residuals, a new tree \(T_{m}(x)\) is trained, which corrects the model in the direction that minimizes the loss. The model updates its predictions by combining the previous prediction with the newly learned tree, scaled by the learning rate \(\eta\). This update process is mathematically expressed as follows:  
\[F_{m}(x)=F_{m-1}(x)+\eta T_{m}(x) \tag{7}\]  
This process progressively reduces residuals as more trees are added, enhancing the model's predictive performance. The learning rate \(\eta\) serves as a hyperparameter that controls the contribution of each newly added tree, playing a crucial role in regulating model complexity and preventing overfitting. In regression tasks, the final predicted value \(\hat{y}\) is obtained by sequentially summing the predictions from all trees, which can be expressed as follows:  
\[\hat{y}=\sum_{i=1}^{N}a_{i}T_{i}(x) \tag{8}\]  
where \(N\) represents the total number of trees and \(a_{i}\) denotes the weight of each tree. Each tree is designed to correct the learned residuals, improving the model's accuracy. For  
Figure 3: Gradient boosting model structure. classification tasks, the model computes the prediction probabilities for each class and selects the class with the highest probability as the final predicted value. Gradient boosting provides flexibility by allowing various hyperparameters to be tuned for optimal performance based on the dataset's characteristics. Key hyperparameters include the number of trees, learning rate, maximum tree depth, and minimum number of samples per node. Adjusting these parameters helps prevent overfitting and improves generalization performance. In particular, a lower learning rate allows the model to train more trees, reducing the contribution of each individual tree while preventing excessive model complexity. Thanks to these characteristics, gradient boosting is recognized as a powerful model capable of effectively capturing important patterns and interactions between variables in high-dimensional datasets with numerous features.