# A Machine Learning Framework for Student Retention Policy Development: A Case Study
## 3 Methodology
### Proposed Framework
#### 3.2.3 Evaluation Measures  
The evaluation of classifier performance in this study is conducted using the metrics of Recall, Precision, and F1-score. These evaluation metrics are based on the prediction performance indicators True Positive (TP), True Negative (TN), False Positive (FP), and False Negatives (FN), which are defined as follows:  
\[\text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FP}+\text{TN}+ \text{FN}} \tag{1}\]  
\[\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}} \tag{2}\]  
\[\text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}} \tag{3}\]  
\[\text{F1}-\text{Score}=2\times\frac{\text{Recall}\times\text{Precision}}{ \text{Recall}+\text{Precision}} \tag{4}\] \begin{table}
\begin{tabular}{p{113.8 pt} p{113.8 pt} p{113.8 pt}} \hline \hline
**ML Algorithm** & **Pros** & **Cons** \\ \hline \multirow{3}{*}{**Random Forest (RF)**} & Highly accurate, robust to overfitting, and works well with diverse datasets [36]. & Expensive and difficult to interpret, especially with many trees [37]. \\ \hline \multirow{3}{*}{**Categorical Boosting (CatBoost)**} & Excels at handling categorical data efficiently and reduces overfitting, making it ideal for tabular datasets with many categorical features [22]. & Expensive and slower compared to simpler models, especially for large datasets [38]. \\ \hline \multirow{3}{*}{**Classification and Regression Trees (CART)**} & Easy to interpret and useful for decision-making in domains requiring transparency [39]. & Prone to overfitting unless pruning or ensemble methods are applied [40]. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: _Cont._\[\text{Weightted\,F1}-\text{Score}\;=\;\frac{\sum_{\text{i}=1}^{\text{N}}\text{Support }_{\text{i}}\;\times\;\text{F1}-\text{Score}_{\text{i}}}{\sum_{\text{i}=1}^{ \text{N}}\text{Support}_{\text{i}}} \tag{5}\]  
\[\text{Macro\,F1}-\text{Score}\;=\;\frac{1}{\text{N}}\sum_{\text{i}=1}^{\text{N} }\text{F1}-\text{Score} \tag{6}\]  
The value TP represents the total number of positive entries in the dataset that are predicted correctly as positive by the classifier, whereas FP represents the total number of negative entries in the dataset that are incorrectly predicted as positive by the classifier. Similarly, TN represents the total number of negative values the classifier predicts correctly as negative, and FN represents the total number of positive entries that the classifier predicts incorrectly as negative. As demonstrated in Equation (1), Accuracy provides the ratio of correctly identified instances to the total number of instances in the dataset. The Precision shown in Equation (2) represents the ratio of correctly predicted positive instances to the total number of positive predictions. On the other hand, Recall, which is also known as sensitivity, represents the ratio of the correctly predicted positive samples to the total number of positive samples in the dataset as presented in Equation (3). The F1-score given in Equation (4) is the harmonic mean of Precision and Recall and is the preferred metric in cases where the dataset is imbalanced. When aggregating the performance of multiple classes, the weighted F1-score shown in Equation (5) averages the F1-score of the classes by considering their distribution in the dataset. Thus, it assigns more weight to the majority class that has more instances. In Equation (5), _Support_, is the number of true instances in class \(i\). On the other hand, the Macro F1-score given in Equation (6) computes the average of F1-scores calculated for each class independently, treating each class equally regardless of the number of instances it contains. Using the F1-score, Macro F1-score, and weighted-average F1-score provides a comprehensive evaluation of classifier performance. The F1-score is useful for assessing the balance between Precision and Recall for individual classes. The Macro F1-score gives equal importance to all classes, which is valuable when dealing with imbalanced datasets.