# A Study on Dropout Prediction for University Students Using Machine Learning
## 4 Experimental Results
### Influence of Oversampling  
As mentioned earlier, the source data used for dropout prediction are highly skewed, where the dropout records account for only 5% of the total data. Such data imbalance can have a significant impact on the prediction performance because it can lead to an overfitting problem where the training can be biased toward a class with a higher distribution. For example, the ratio of P and N classes in our source data is 5:95. In this case, the training can be biased toward the N class, which leads to the result that the prediction performance for the N class increases while the performance for the P class may significantly decrease. One of the popular methods to resolve the overfitting problem is oversampling. In this technique, artificial records are generated and added to the minor class, e.g., the P class, until the ratio between the classes becomes even. Among the various oversampling techniques, SMOTE (Synthetic Minority Oversampling Technique) [20] is the most commonly used. Data augmentation in SMOTE is performed based on K-NN. Figure 4 illustrates how artificial records are generated in SMOTE. It considers only the records in the P class when generating new records. From the P class data, a reference vector is first selected. Then, K neighboring vectors close to the reference vector are chosen using K-NN. These vectors are connected to the reference vector with lines, as shown in Figure 4. Finally, new vectors are randomly extracted from the connection lines and added to the minor class until the ratio between the classes becomes even. Table 10 shows the prediction performance when SMOTE is applied to the proposed prediction models. SMOTE was applied only to the training data, not to the test data. From the results shown in Table 10, we can see that the F1-scores of all models except the RF model decreased after applying SMOTE. Compared with the results in Table 8, the average  
Figure 4: Data augmentation in SMOTE. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Measure**} & \multicolumn{3}{c}{**Existing Models**} & \multicolumn{3}{c}{**Proposed Model**} \\ \cline{2-6}  & **Ref. No.** & **Algorithm** & **Score** & **Algorithm** & **Score** \\ \hline F1-score & [8] & CatBoost+XGBoost & 0.808 & LightGBM & 0.840 \\ Precision & [9] & Ridge Regression & 0.739 & LightGBM & 0.867 \\ F1-score & [29] & RF & 0.810 & LightGBM & 0.840 \\ F1-score & [30] & SVM & 0.804 & LightGBM & 0.840 \\ Accuracy & [32] & DNN & 0.768 & LightGBM & 0.955 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Performance comparison of the proposed LightGBM model and the existing models discussed in [8; 9; 29; 30; 32]. F1-score of the models dropped from 0.796 to 0.762. This is due to the fact that the average value of precision has dropped significantly from 0.848 to 0.677. On the other hand, the average value of recall improved from 0.752 to 0.882. As an exceptional case, the F1-score of the RF model slightly increased from 0.827 to 0.830. It was not possible to measure the performance of the LightGBM model since an error occurred due to a conflict between the LightGBM and SMOTE packages. We also checked the influence of ADASYN and Borderline-SMOTE on the performance of the proposed prediction models. Tables 11 and 12 show the performance after ADASYN and Borderline-SMOTE were applied, respectively. ADASYN showed the lowest performance, with an average F1-score of 0.735, among the three oversampling techniques discussed in this paper. The performance of Borderline-SMOTE was also worse than that of SMOTE, with an average F1-score of 0.742. Figure 5 visualizes the change in F1-scores of the proposed prediction models before and after applying SMOTE, ADASYN, and Borderline-SMOTE. The average F1-scores of all models decreased after applying the oversampling techniques, except the RF model; the performance improvement of the RF model was also not so significant. Among the oversampling techniques, SMOTE showed the best performance, whereas ADASYN performed the worst.