## 3 Model Selection
### Classification Prediction
A larger value of \(C\) allows the optimization to choose a smaller-margin hyperplane if that hyperplane does a better job of getting all the training points classified correctly. The hinge loss function \(\max(0,1-y_{i}(W^{T}X_{i}))\) is piecewise linear and is used to compute the loss of a prediction. It penalizes predictions that are on the wrong side of the margin. (3) Gradient Boosting Tree (GBT) [28]: Also known as gradient boosting machine, GBT is a machine learning method that combines decision trees to create a strong predictive model. It iteratively builds a series of decision trees by minimizing the loss function, with each tree attempting to correct the errors made by the previous one. The calculation process of GBT can be summarized in five steps: initializing a decision tree, iterating over the training data, calculating the gradient, building a new decision tree, and updating the model. The process begins by initializing a decision tree as the base model. In each iteration, GBT calculates the gradient of the loss function with respect to the predictions made by the current model. Then, GBT calculates the gradient based on the difference between the actual target values and the predictions made by the current ensemble of trees. Next, based on the gradient, GBT constructs a new decision tree that aims to reduce the errors made by the ensemble of trees constructed so far. Finally, after generating the new decision tree, GBT combines it with the previous models to update the ensemble. The new model is created by adding the predictions of the newly generated tree to the predictions of the existing ensemble, with the objective of minimizing the overall loss function. The calculation formula is as follows:  
\[\hat{y}_{GPT}=\sum\limits_{t=1}^{T}\alpha_{t}\cdot\hat{y}_{t} \tag{9}\]  
where \(\hat{y}_{GPT}\) is the prediction result of the gradient lift tree, \(T\) is the number of trees, \(\alpha_{t}\) is the weight of t tree, and \(\hat{y}_{t}\) is the prediction result of t tree. In GBT, the weights \(\alpha_{t}\) are usually fixed or calculated through optimization algorithms such as gradient descent, aiming at reducing the loss function of the overall model, and are not directly based on the error rate of each weak learner. (4) K-Nearest Neighbors (KNN) [29]: The optimization problem in KNN lies in determining the optimal value of K, which represents the number of neighbors considered for classification or regression. The choice of K can significantly impact the performance of the KNN algorithm. The optimization can be expressed as follows: first, for each unknown sample, the distance in the feature space is calculated; then, the nearest K neighborsare found; last, the category of neighbors (for classification problems) is voted on. The calculation formula is as follows:  
\[\hat{y}_{KNN}=\text{argmax}\sum_{i=1}^{K}I(y_{i}=y) \tag{10}\]  
where \(\hat{y}_{KNN}\) is the prediction result of KNN, \(K\) is the number of neighbors, and \(y_{i}\) is the category of the \(i\) neighbor. (5) AdaBoost [31]: Also known as adaptive boosting, AdaBoost is an iterative algorithm that constructs a series of weak learners by progressively emphasizing the misclassified samples. These weak learners are then combined to create a more powerful or stronger learner. The calculation process is as follows: a weak learner is initialized; the training dataset is iterated for each iteration; for each sample, the prediction error of the current learner is calculated; the weight of the sample is updated according to the error; and the new weak learner is trained with the updated weight. The calculation formula is follows:  
\[\hat{y}_{Ada}=\sum_{t=1}^{T}\alpha_{t}\cdot\hat{y}_{t} \tag{11}\]  
where \(\hat{y}_{Ada}\) is the prediction result of AdaBoost, \(T\) is the number of learners, \(\alpha_{t}\) is the weight of the t learner, and \(\hat{y}_{t}\) is the prediction result of the t learner.