# Forecasting Students Dropout: A UTAD University Study
## 4 Results and Discussion  
Four different models (CatBoost; Random Forest; XGBoost; ANN) were built. Since all models, except ANN, are tree-based, they do not benefit from feature scaling. Therefore, the ANN model was implemented into a pipeline (Figure 7), which applies additional data pre-processing before it gets fitted into the model. First, the features were scaled between  
Figure 4: Age importance box plot (red: success, magenta: dropout). Figure 5: Number of fails per curricular unit (feature red: used in the model, feature blue: discarded). Figure 6: Data division strategy. Adapted from: [[https://scikit-learn.org/stable/modules/cross_validation.html](https://scikit-learn.org/stable/modules/cross_validation.html)]([https://scikit-learn.org/stable/modules/cross_validation.html](https://scikit-learn.org/stable/modules/cross_validation.html)) (accessed on 17 January 2022). zero and one. The data was then balanced using the ROS technique (Section 3.5), as it provided more reliable results then unbalanced data. Every model was then submitted to the stratified 10-fold cross-validation test, described before. During this test, the tuning of each model's hyperparameters was executed, as these have a significant impact on the model's prediction performance. The hyperparameters used for each model are illustrated in Table 3.  
Every model was then submitted to the stratified 10-fold cross-validation test. This technique allowed to estimate the performance of each model on unseen data. The results are displayed in Table 4. As noticeable, RF provided the best overall metrics [27], only losing in recall to XGBoost. One important metric is AUROC, which defines how accurately each model discriminates between classes. It is one of the most commonly adopted metrics to measure the model's performance in classification problems. The obtained ROC curves, for RF, on the previous test, are displayed in Figure 8. With a mean value of 0.91 and a standard deviation of 0.05, this model can very precisely and consistently distinguish a dropout student from a non-dropout. \begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **Hyperparameters** \\ \hline
**CatBoost** & iterations = 50 \\  & learning\_rate = 0.9 \\  & l2\_leaf\_reg = 11 \\  & class\_weights = [1, 1.67] \\  & eval\_metric = AUC \\  & max\_depth = 1 \\ \hline
**Random Forest** & class\_weight = [1, 1.67] \\  & max\_depth = 7 \\  & min\_samples\_leaf = 2 \\  & min\_samples\_split = 10 \\ \hline
**XGBoost** & learning\_rate = 0.15 \\  & scale\_pos\_weight = 1.67 \\  & colsample\_bytree = 0.7 \\  & n\_estimators = 100 \\  & min\_child\_weight = 7 \\  & max\_depth = 5 \\  & gamma = 0.4 \\ \hline
**ANN** & activation = logistic \\  & alpha = 0.001 \\  & early\_stopping = True \\  & hidden\_layer\_sizes = 12 \\  & learning\_rate\_init = 0.6 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Model’s hyperparameters. Figure 7: ANN pipeline architecture. RF also presented an F1-Score of 0.81 with a standard deviation of 0.22, being the best across all models. This metric represents the harmonic mean between recall and precision, being a good measure of the model's performance. The final validation data (unseen data) was then fitted into each model, as a last sanity test for their generalization capability. The results are displayed in Table 5. In this test, XGBoost showed the best results. However, it is expected that the RF will present a more stable performance, due to the slightly more reliable results obtained in the cross-validation. Most of the articles reviewed make the prediction considering several characteristics external to the academic context. These models are good when there is data to feed them. At UTAD, the recorded features were scarce, and it was necessary to use models with existing data. In this context, models considering only academic marks and the age of students were used to predict school dropout, achieving good results (F1-score of 0.87). Some of the