## 2 Materials and Methodology
### Methodology
#### 2.2.4 XGBoost Method  
XGBoost was developed by [PERSON] and [PERSON] [38]. It represented an advanced and scalable implementation of gradient-boosting machines. The fundamental concept is that each new model is designed to fit the prediction residual of the previous model. The predicted result is obtained by aggregating the results of each model, as shown in the following equation:  
\[\hat{y}_{i}=\sum_{k=1}^{K}f_{k}(x_{i}), \tag{4}\]  
where \(i=1,2,\ldots,n\). \(n\) is the number of samples and \(f_{k}\) is the \(k\)-th regression tree function.  
The objective function is composed of a loss function \(L(y_{i},\hat{y}_{i})\) and a regularization item \(\Omega(f_{i})\), as shown in Equation (5):  
\[f_{obj}=\sum_{i=1}^{n}L(y_{i},\hat{y}_{i})+\sum_{k=1}^{K}\Omega(f_{k}), \tag{5}\]  
As mentioned above, in time \(t\), the objective function can be expressed as in Equation (6):  
\[f_{obj}^{(t)}=\sum_{i=1}^{n}L(y_{i},\hat{y}_{i}^{(t-1)}+f_{t}(x_{i}))+\Omega(f _{t})+C, \tag{6}\]  
It seems complicated except for the case of the loss function; XGBoost takes the Taylor expansion to approximate this, as shown in Equation (7):  
\[f_{obj}^{(t)}\approx\sum_{i=1}^{n}\left[L(y_{i},\hat{y}_{i}^{(t-1)})+g_{i}f_{ t}(x_{i})+\frac{1}{2}h_{i}f_{t}{}^{2}(x_{i})\right]+\Omega(f_{t})+C, \tag{7}\]  
where the \(g_{i}\) and \(h_{i}\) are as follows:  
\[g_{i}=\frac{\partial L\bigg{(}y,f_{t-1}\bigg{(}\overrightarrow{\mathbf{X}} \bigg{)}\bigg{)}}{\partial f_{t-1}\bigg{(}\overrightarrow{\mathbf{X}}\bigg{)} },h_{i}=\frac{\partial^{2}L\bigg{(}y,f_{t-1}\bigg{(}\overrightarrow{\mathbf{X }}\bigg{)}\bigg{)}}{\partial f^{2}{}_{t-1}\bigg{(}\overrightarrow{\mathbf{X}} \bigg{)}} \tag{8}\]  
The prediction result in \(t\) time can be expressed as:  
\[f_{t}(x)=w_{q(x)},w\in R^{T},q:R^{d}\rightarrow\{1,2,\ldots,T\} \tag{9}\]  
where \(q\) is the structure of the tree and \(w\) is the leaf weight of the tree.  
Regularization item \(\Omega(f_{i})\) represents the complexity of the model to avoid model overfitting, as shown in Equation (10):  
\[\Omega(f_{t})=\gamma T+\frac{1}{2}\lambda\parallel w\parallel^{2}, \tag{10}\]  
where \(T\) are several leaf nodes, \(w\) is the sum of the leaf node scores in trees, and \(\gamma,\lambda\) are adjusted parameters.  
Thus, the objective function can be expressed as follows:  
\[f_{obj}^{(t)}=\sum_{j=1}^{T}[(\sum_{i\in I_{j}}g_{i})w_{j}+\frac{1}{2}(\sum_{i \in I_{j}}h_{i}+\lambda)w_{j}^{2}]+\gamma T, \tag{11}\]  
where \(I_{j}\) represents the set of leaf samples in the \(j\)-th tree.  
Then, we define the \(G_{j}=\sum_{i\in I_{j}}g_{i}\), \(H_{j}=\sum_{i\in I_{j}}h_{i}\) objective function that can be simplified as follows:  
\[f_{obj}=-\frac{1}{2}\sum_{j=1}^{T}\frac{G_{j}}{H_{I}+\lambda}+\gamma T \tag{12}\]  
Multiple parameters in XGBoost can be fine-tuned to improve the performance of the model. In this paper, we selected three key parameters in XGBoost for optimization: learning rate, the maximum tree depth of each regression tree, and the number of regression trees. The learning rate is an important parameter in XGBoost that controls the contribution of each weak learner to the final prediction. Specifically, the learning rate determines the scaling factor applied to the prediction of each weak learner, with smaller values leading to a more conservative model that may require more weak learners to achieve high accuracy and larger values leading to a more aggressive model that may be prone to overfitting and instability. The rest of the two parameters are similar to those in the RF model.