# A Machine Learning Framework for Student Retention Policy Development: A Case Study
## 4 Experimental Results and Discussion
### Using Prediction Results to Enhance Student Retention Policies
#### 4.3.1 Visualization of Results by the HEI  
The proposed framework employs clustering followed by a decision tree analysis as a visualization tool. In previous research, clustering and decision trees have proven to be effective tools for identifying patterns in student behavior and linking these patterns to academic performance outcomes [44; 45; 46; 47]. For instance, [PERSON], [PERSON] [44] and [PERSON] [47] employed decision trees to provide a visualization for clusters formed for the educational data used. In our study, we employ clustering and decision tree analysis as visualization tools in a similar way: first predicting students at risk, then categorizing them into clusters, and finally applying decision tree analysis to enable tailored retention strategies for each cluster. One of the main aims of this work is to provide a framework that can be used by HEIs to profile at-risk students and devise tailored intervention strategies or policies to retain these students. In Figure 1, this stage is indicated as the \"Analysis of At-Risk Students and Policy Generation\" framework.  
An HEI may devise several different policies to retain students at risk of dropping out. The clustering algorithm is used to partition all at-risk students into classes where similar students are grouped together and thus a policy may be devised to target each group. However, clusters are not easy to interpret, and additional tools must be used to visualize them. When the data are multidimensional, 2D cluster plots might be insufficient for visualization. Therefore, decision trees that are categorized as intrinsically explainable ML models may be used as a visualization tool by policymakers to understand and visualize the clusters formed. In the current case study, the predictions of the best classifier, namely CatBoost, are fed into the clustering algorithm producing three clusters. The number of clusters corresponds to the number of different policies or strategies an HEI is planning to implement. The data can be analyzed to identify the optimal number of clusters using methods such as the Elbow method or Silhouette analysis, but that may result in a higher  
Figure 12: Comparison of Macro F1-score rates for machine learning models with optimized parameters for all features and five features.  
number of clusters or policies, which may be impractical for HEIs. Figure 13 presents the first three levels of the decision tree built using the data on at-risk students with cluster labels. The decision tree is pruned at level three to allow better visualization of the clustering logic. This choice of limiting the tree depth results in a more human-interpretable tree at the expense of Accuracy as can be seen in the impurity values of the leaf nodes of the tree.  
Inspection of the decision tree of Figure 13 shows that academic performance in terms of the number of courses repeated is an important determinant for clusters. Additionally, the number of courses passed at the end of semester 2 and the number of courses failed at the second semester are used to partition the date at the second level of the decision tree. This is in line with the expectation that academic performance is an important determinant of the risk of dropping out. Note that the data used for clustering and building the decision tree can be augmented with additional data from the HEI data resources such as attendance and homework marks. Employing additional information at this level may enable a more accurate analysis of the needs of students.