## 3 Model Selection
### Classification Prediction  
In this section, we address the issue of predicting student achievement using categorical grades instead of continuous values. This is done to account for potential variations in the number of students in different grade categories, which could result in large errors if regression models are directly used for predictions. Furthermore, the use of discrete grade categories makes the model more interpretable and easier to handle. To tackle this categorical prediction task, we conducted experiments using five different algorithms: random forest, support vector machine, gradient boosting tree, K-nearest neighbors, and AdaBoost. These algorithms were applied to predict grades in four distinct courses: Data Structures, Compilation Principles, Algorithm Analysis and Design, and Operating Systems. Let us now provide a brief introduction to these algorithms. (1) Random Forest [25]: Random forest is an ensemble learning method that utilizes decision trees to improve the model's generalization capability and reduce the risk of overfitting. Each decision tree is trained independently, and the final prediction is obtained by averaging or taking the majority vote of the predictions from all the decision trees. The calculation process of random forest is as follows: Each decision tree randomly selects a subset of samples from the training data through a process called \"bootstrap aggregating\" or \"bagging\". For each selected sample, a random subset of k features (where k is an integer less than the total number of features) is chosen. The decision tree is built based on the selected features and undergoes a process of dividing the data into smaller subsets. The above steps are repeated for all the decision trees until each tree is trained. The calculation formula is follows:  
\[\hat{y}_{RF}=\frac{1}{T}\sum_{t=1}^{T}\hat{y}_{t} \tag{7}\]  
where \(\hat{y}_{RF}\) is the prediction of the random forest, \(T\) is the number of trees, and \(\hat{y}_{t}\) is the prediction of the t tree. (2) Support Vector Machine (SVM) [26]: SVM is a supervised learning algorithm used for classification. It works by mapping the input data into a high-dimensional feature space and then finding the optimal hyperplane that separates the data points belonging to different classes. The objective of the SVM is to minimize classification errors on the training set while maximizing the margin between the hyperplane and the support vectors. The process of SVM classification can be outlined in three steps: mapping the data, finding the hyperplane, and classifying based on distance. The SVM uses a kernel function to map the input data from the original space to a higher-dimensional feature space. Then, in the high-dimensional space, the SVM identifies the hyperplane that best separates the data points from different classes. Once the hyperplane is determined, the SVM calculates the distance between each support vector (data point that has the most influence on the location of the hyperplane) and the hyperplane. The sign of this distance determines the classification of a data point. Data points on one side of the hyperplane are assigned to one class, while those on the opposite side are assigned to the other class. The optimization problem can be expressed as follows:  
\[\min\frac{1}{2}W^{T}W+C\sum\limits_{i=1}^{N}\max(0,1-y_{i}(W^{T}X_{i})) \tag{8}\]  
The vector \(W\) represents the weights of the model. The term \(\frac{1}{2}W^{T}W\) is a regularization term that penalizes the complexity of the model; it effectively tries to keep the weights small to avoid overfitting. The sum \(\sum\limits_{i=1}^{N}\max(0,1-y_{i}(W^{T}X_{i}))\) is the sum of the hinge loss for each of the \(N\) samples in the dataset. Each \(X_{i}\) is a feature vector representing the \(i\)-th sample, and each \(y_{i}\) is the corresponding true label, which is usually +1 or \(-1\) in binary classification tasks. \(C\) is a hyperparameter that controls the trade-off between maximizing the margin and minimizing the classification error.