# An Early Warning System to Detect At-Risk Students in Online Higher Education
## 6 Case Studies: Computer Fundamentals and Databases
### Performance of the Warning Level Classification  
Regarding the accuracy of the warning level classification, the results for each assessment activity are shown in Table 5 for both courses. On the one hand, the table shows the number of students assigned to each classification level (_No._) and the final performance of the students for each warning level (_Fail, Pass_). The set of messages to be sent according to the warning level classification (_WL_) has been previously discussed in Section 5 (see Figure 7). The final performance for each case gives insights about the correct assignation of the students. As it can be observed, in the case of CF and for the AA1, students were mostly assigned to the yellow warning level. This was due to the low-quality of the submodel, and it is consistent with the results previously discussed (see Table 5). In subsequent assessment activities, all the colors were used. It is worth noting that the accuracy improved, and students assigned to the yellow color were those students that passed the assessment activity, but with a grade lower than the grade suggested by the prediction. The green color was correctly assigned for 74.5% of the students, and the red and black (non-submitted) to 93.5% and 100%, respectively on AA2. Similar results were observed in the AA3 and the AA4. Some students assigned to red color in the AA2 moved to black in the upcoming assessment  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline  & **TP** & **FP** & **TN** & **FN** & **ACC(\%)** & **TNR(\%)** & **TPR(\%)** & **F\({}_{1,5}\)(\%)** & **Algorithm** & **Semester** \\ \hline
**P\({}_{\text{AA1}}\)** & 164 & 42 & 92 & 34 & 77.11 & 68.66 & 82.83 & 81.81 & Decision Tree & 20171 \\
**P\({}_{\text{AA2}}\)** & 171 & 10 & 121 & 44 & 84.39 & 92.37 & 79.53 & 83.60 & Decision Tree & 20182 \\
**P\({  
activities because they did not submit previous activities, causing course dropout. The yellow color only served as medium risk level at first assessment activities, whereas students were progressively moved to the respective correct risk level on the final ones. Concerning the DB course, and on the contrary of CF, some students were assigned to the green color in AA1. This is consistent with the fact that the grade obtained in AA1 was a good indicator for predicting course success. A percentage of 78.2 of the students were correctly informed about their chances of success, and this ratio increased significantly in the upcoming assessment activities up to 89.8 and 93.6% in AA2 and AA3, respectively. This is consistent with the fact that AA2, AA3, and AA5 were (at least two of them) mandatory activities, and students prioritize the submission of AA2 and AA3 in order to accomplish this requirement as soon as possible in the course timeline. In fact, most of the students assigned to the green color delivered AA2, AA3, and AA5. Similar to CF, yellow color warned students that their performance was under the prediction in all the assessment activities. Students assigned to the red color appear in AA2 (first potentially mandatory assessment activity), and 73.3% were correctly identified at-risk, and this ratio increased until 80.0% in AA3. Clearer than in the case of CF, students assigned to red color in AA2 moved to black color in the subsequent assessment activities. We also checked whether there is statistical significance on the final mark distribution comparing the semester where the case studies were run concerning the previous semester (20182, i.e., 2019 spring semester). The objective is to see whether the EWS has impacted the performance of the courses. We used the unpaired two-sample Wilcoxon test due to the non-normal distribution of the final mark [61]. Here, we assume as the null hypothesis that the marks are worse or equal than in the previous semester. Note that, the dropout students are not taken into account.