## 2 Materials and Methods
### Identification Methods
#### 2.3.3 Feature Classification
Finally, the probabilities \(p_{kM}(\mathbf{x}_{i})\) mapped by the final CARTs can be used to classify the test samples. ```
1 Initialize the prediction values \(Fu(\mathbf{x})\) = 0, for \(k\) = 1,..., \(K\), \(I\) = 1,..., \(n\). 2 for\(m\) = 1 to \(M\)do
3 Calculate the probability of classifying \(\mathbf{x}\) into the \(k\)-th class: \(p_{k}(x)=\exp_{k}^{(m-1)}(x)/\sum_{k=1}^{K}\exp_{k}(x)\), \(k\) = 1,..., \(K\). 4 for\(k\) = 1 to \(K\)
5 Predict the residual errors of \(\mathbf{x}\) for the \(k\)-th class: \(\hat{y}_{ik}=y_{ik}-p_{k}(\mathbf{x}_{i}),i=1,...,n\)
6 Construct a CART regressor \(T^{(m)}_{k}\) based on the residuals and training samples. 7 Calculate the prediction value of the leaf nodes: \(\hat{y}_{jkm}=\frac{K-1}{K}\cdot\frac{\sum_{\mathbf{x}_{i}\in p_{jkm}}\hat{y}_{ik} \left(\mathds{1}(-\hat{y}_{ik})\right)}{\sum_{\mathbf{x}_{i}\in p_{jkm}}\hat{y}_{ik} \left(\mathds{1}(-\hat{y}_{ik})\right)}\), where \(R_{jkm}\) is the set of nodes in the \(j\)-th leaf node of \(T^{(m)}_{k}\). 8 Update the prediction value of \(T_{k}\) for \(\mathbf{x}_{i}\), \(i\) = 1, 2,..., \(n\): \(F^{(m)}_{k}(\mathbf{x}_{i})=F^{(m-1)}_{k}(\mathbf{x}_{i})+\eta\sum_{j=1}^{J}\gamma_{ jkm}\mathds{1}(\mathbf{x}_{i}\in Bjkm)\), where \(J\) denotes the leaf node number in \(T^{(m)}_{k}\), \(\eta\) is the learning rate parameter. 9 end
```  
**Algorithm 3** GBDT [17]. The pseudo-code of GBDT [17] is shown in Algorithm 3. For \(K\)-class (K > 2) classification problems, GBDT sequentially constructs \(K\times M\) CART regressors, where \(M\) is the number of iterations. In each iteration, an individual CART regressor is fitted for each class in expectation of minimizing a greedy loss function, which is equal to the negative gradients of the multiclass log loss [17]. As shown in line 3 of Algorithm 3, using the Sigmoid function, the continuous prediction value \(F_{k}(\mathbf{x}_{i})\) of the constructed CARTs are mapped to probabilities that \(\mathbf{x}_{i}\) belongs to the \(k\)-th class \(p_{k}(\mathbf{x}_{i})\). During the construction of CART regressors, the mean squared error with the improvement score by [PERSON] [17] was used as the splitting criteria to induce the trees and leaf nodes. Finally, the probabilities \(p_{kM}(\mathbf{x}_{i})\) mapped by the final CARTs can be used to classify the test samples. ```
1 Initialize the prediction values \(Fu(\mathbf{x})\) = 0, for \(k\) = 1,..., \(K\), \(I\) = 1,..., \(n\). 2 for\(m\) = 1 to \(M\)do
3 Calculate the probability of classifying \(\mathbf{x}\) into the \(k\)-th class: \(p_{k}(\mathbf{x})=\exp_{k}^{(m-1)}(x)/\sum_{k=1}^{K}\exp_{k}(x)\), \(k\) = 1,..., \(K\). 4 for\(k\) = 1 to \(K\)
5 Predict the residual errors of \(\mathbf{x}\) for the \(k\)-th class: \(\hat{y}_{ik}=y_{ik}-p_{k}(\mathbf{x}_{i}),i=1,...,n\)
6 Construct a CART regressor \(T^{(m)}_{k}\) based on the residuals and training samples. 7 Calculate the prediction value of the leaf nodes: \(\hat{y}_{jkm}=\frac{K-1}{K}\cdot\frac{\sum_{\mathbf{x}_{i}\in p_{jkm}}\hat{y}_{ik} \left(\mathds{1}(-\hat{y}_{ik})\right)}{\sum_{\mathbf{x}_{i}\in p_{jkm}}\hat{y}_{ik} \left(\mathds{1}(-\hat{y}_{ik})\right)}\), where \(R_{jkm}\) is the set of nodes in the \(j\)-th leaf node of \(TAn additional regularization term in the loss function used for tree building. The objective function to be minimized in the \(m\)-th model training iteration of XGBoost is as follows:

\[\mathcal{L}^{(m)}=\sum_{i=1}^{n}l\left(y_{i},\hat{y}_{i}^{(m-1)}+f_{m}(x_{i}) \right)+\Omega(f_{m}), \tag{1}\]

where \(l\) is the loss function, \(\hat{y}_{i}^{(m-1)}\) is the prediction for the \(i\)-th sample as the \(m\)-th iteration, and \(\Omega(f_{m})\) is the regularization item that is proportional to the model complexity. 2. Usage of second-order gradient statistics. In order to minimize the objective function, GBoost additionally uses its second-order Taylor expansion approximation, whereas GBDT only calculates its negative first-order gradient. The objective function in XGBoost can be approximated as follows.