# A Machine Learning Framework for Student Retention Policy Development: A Case Study
## 4 Experimental Results and Discussion
### Dropout Prediction Using Machine Learning Algorithms  
In order to answer RQ2, we studied the predictive performance of eight commonly used ML classifiers. Generally, the performance of machine learning algorithms may be improved through feature engineering and parameter tuning. For each of the machine learning classifiers, parameter tuning was performed using the heuristic parameter tuning method grid search. The proposed framework was employed to test the performance of the ML classifiers using optimal parameter settings. The performance of machine learning classifiers with default parameters, trained using all features, is summarized in Table 4. With the default settings, the highest Accuracy and F1-score were achieved by the CatBoost classifier with an Accuracy of 83% and an F-score of 82%. The RF classifier achieved the second-highest results, with an F1-score of 81% and an Accuracy of 82%. MLP followed suit with an F1-score of 80% and the remaining machine learning models, namely NB, LR, KNN, and SVC, exhibited lower performance, with F1-scores of 49%, 37%, 63%, and 38%, respectively. Following parameter tuning, all ML algorithms were retrained using the optimal parameter settings. The highest improvement was seen for CARTs where the F1-score was improved by 6%. The results are shown in Table 5. The performance of both CatBoost and Random Forest classifiers improved by 1%. The MLP was the third highest performing classifier with an F1-score of 80% and an Accuracy of 81%. \begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline
**Models** & **Accuracy** & **Precision** & **Recall** & **Dropout F1** & **Grad F1** & **Macro F1** & **Weighted F1** \\ \hline
**Support Vector Classifier (SVC)** & 0.60 & 0.50 & 0.50 & 0.0 & 0.75 & 0.38 & 0.45 \\ \hline
**K-Nearest Neighbors (KNN)** & 0.66 & 0.64 & 0.63 & 0.54 & 0.73 & 0.63 & 0.65 \\ \hline
**Logistic Regression (LR)** & 0.59 & 0.30 & 0.50 & 0.0 & 0.75 & 0.37 & 0.44 \\ \hline
**Gaussian Naive Bayes (NB)** & 0.61 & 0.58 & 0.53 & 0.23 & 0.74 & 0.49 & 0.54 \\ \hline
**Artificial Neural Network (MLP)** & 0.81 & 0.81 & 0.79 & 0.74 & 0.85 & 0.80 & 0.81 \\ \hline
**Random Forest (RF)** & 0.82 & 0.81 & 0.80 & 0.76 & 0.85 & 0.81 & 0.82 \\ \hline
**Categorical Boosting (CatBoost)** & **0.83** & **0.83** & **0.81** & **0.78** & **0.86** & **0.82** & **0.83** \\ \hline
**Classification and Regression Trees (CART)** & 0.75 & 0.74 & 0.75 & 0.70 & 0.79 & 0.74 & 0.75 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance measures of ML models with default parameters set using all features. As Figure 11 shows, the parameter tuning process improved the Macro F1-score of the SVC significantly from 38% to 74%, highlighting the importance of optimizing hyperparameters. Similarly, the CARTs model also shows improvements after tuning, with the Accuracy rate increasing from 74% to 80%. The LR model shows a modest improvement of 1%, resulting in a Macro F1-score of 38%. This indicates that while some models can achieve a high Macro F1-score, their effectiveness depends heavily on parameter selection. Among the models evaluated, the best classification performances were achieved, following the tuning process, by the optimized CatBoost, RF, and MLP algorithms with Macro F1-scores of 82%, 81%, and 80%, respectively. This suggests that these intrinsic ensemble methods, which aggregate predictions from multiple models, are especially effective for this type of classification problem. It is also worth noting that these three models showed strong performance even with their default parameters, highlighting their ability to manage the dataset's complexities. Figure 11: Comparison of Macro F1-score rates for machine learning models with default and optimized parameters.