# Modeling E-Behaviour, Personality and Academic Performance with Machine Learning
## 2 Methodology
### Evaluation Metrics for Student Risk Classification  
The Results Summary in Table 10 was used as an evaluation template for the classification problems in Sections 3.1, 3.6 and 3.7. The best-case scenario (where the e-behaviour model obtained a 100% accuracy) occurred where all students with an At-risk Outcome label were Flagged, and all students with a Safe Outcome label received an Ignored Safety Score. Given an Outcome, At-risk, precision measured the proportion of students who were correctly Flagged as At-risk, \(a\), against the total number of Flagged students. Precision was calculated as \(\frac{a}{a+c}\), where \(c\) represented the number of students who should have been Ignored as Safe. Recall measured the proportion of students who were correctly Flagged as At-risk, \(a\), against all At-risk observations, \(\frac{a}{a+b}\). The same calculation generalised to the Safe Outcome. It is essential to know the most important metrics to measure when evaluating a classifier's performance. Consider the Results Summary in Table 10. A perfectly accurate model resulted in a \(b\) and \(c\) equal to 0. The precision and recall scores would be 1 for both the At-risk and Safe Outcomes. None of our models achieved perfect accuracy--they conducted trade-offs regarding precision and recall. For a model whose objective was to classify all students who were at risk of failing, higher precision-recall scores for the At-risk Outcome werepreferred over higher precision-recall scores for the Safe Outcome. Furthermore, maximising the recall of the At-risk Outcome, \(\frac{a}{a+b}\) (where the classifier recalled all students who were at risk) was preferred to maximising either the precision of the At-risk Outcome or the precision-recall scores of Safe students. Recall-maximisation would likely cause a low precision for the At-risk Outcome class (a high \(c\)). In such a case, however, no student who was At-risk would have been incorrectly Ignored. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{4}{c}{**Safety Score**} \\  & \multicolumn{2}{c}{**(Prediction)**} \\ \hline  & \multicolumn{2}{c}{**Flagged**} & **Ignored** & **Total** \\ \hline Outcome & **At-risk** & \(a\) & \(b\) & \(a+b\) \\ \cline{2-5} (True Label) & Safe & \(c\) & \(d\) & \(c+d\) \\ \hline  & Total & \(a+c\) & \(b+d\) & \(N=a+b+c+d\) \\ \hline  & **Outcome** & **Precision** & **Recall** \\ \hline  & At-risk & \(\frac{a}{a+c}\) & \(\frac{a}{a+b}\) & \\ \hline  & Safe & \(\frac{d}{d+b}\) & \(\frac{d}{d+c}\) & \\ \hline \hline \end{tabular}
\end{table} 
Table 10: Format of the Summary of Results. The Overall Accuracy of a Model  
While precision and recall are important metrics to measure a binary classifier's performance, they represent four different views of accuracy that must be analysed separately (precision and recall for At-risk and Safe Outcomes). When evaluating a model or accuracy, it is useful to obtain a single metric. A widely-used _accuracy_ measure calculates the ratio between the correctly classified number of observations and the total number of observations. This _accuracy_ is a good measure for balanced data, not for imbalanced data. For instance, if a test dataset contains 100 observations with an At-risk:Safe split of 10:90, a classifier can obtain an accuracy of 90% by classifying all students as Safe. An accuracy measure that combines the harmonic mean of precision and recall of either class is the _f-1 score_, whose effectiveness is surveyed by [PERSON] and [PERSON] [54]. The f-1 score produces two metrics (one for each Outcome) and does not concisely summarise the model's accuracy. By contrast, _Cohen's Kappa_, \(\kappa\)[55] is a metric that captures accuracy with a single value. The formula,  
\[\kappa=(p_{o}-p_{e})/(1-p_{e}), \tag{29}\]  
measures the _agreement_ between the _predicted_ Safety Score and the _true_ Outcome. [PERSON] and [PERSON] [56] suggest using the scale in Table 11 to interpret the significance of \(\kappa\) values.