# An Early Warning System to Detect At-Risk Students in Online Higher Education
## 3 Methods and Context
### Generation and Evaluation of the Predictive Model  
The GAR model is built by using Python and the machine learning Scikit-Learn library [52], while the statistical analysis has been performed by the analytical tool R [53]. The GAR model for a given course is trained based on the historical data of previous semesters from the 2016 fall semester (i.e., 2016), and the test is performed on the data of the last historical available semester from the UOC data mart (that is hosted in Amazon S3). For example, for semester 20171, the models are trained with data from semesters 20161 and 20162 to test it on the semester 20171. During the validation test, four algorithms are tested: NB, DT, KNN, SVM. In order to evaluate the classification algorithms, the GAR model is built for each course of the institution based on the number of assessment activities (see Example 1 in Section 3.3). Each course has a different number of assessment activities and some courses have a final exam. In such cases, the submodel for the final exam is not generated since the final grade can be straightforwardly computed from all grades of the assessment activities and the final exam, and no prediction is needed. Then, the training and test are performed for each course and submodel. In order to analyze the results globally for the whole institution, the value of each metric described in Section 3.3 is ordered and uniformly distributed among the semester timeline based on the submission date of the respective assessment activity associated with the submodel. For instance, for a course with four assessment activities, the prediction of the first assessment activity is set to the position at 20%, the second at 40%, the third at 60%, and the fourth at 80% of the semester timeline. Note that, this distribution changes on courses with a different number of activities. This distribution helps to identify at each stage of the timeline the average quality of the GAR model based on the submitted assessment activities at that time for the whole institution. Finally, in order to evaluate the different classification algorithms, the LOESS regression [54] is used. Although a linear regression would show a pretty linear perception of the increment of the accuracy, this would not represent the correct relationship between metrics and timeline. The LOESS regression shows a better approximation due to the large number of values. The NB was selected in [3] as the best classification algorithm to be used in the institution based on the performance observed on the four metrics. Precisely, the selection was mainly done by the results of the TPR and F\({}_{1,5}\) since these metrics tend to indicate the algorithm that mostly detects at-risk students. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **20171** & **20172** & **20181** & **20182** \\ \hline Number of courses & 889 & 830 & 968 & 979 \\ Min-Max activities / course & 3–14 & 3–11 & 3–11 & 3–11 \\ Semesters for training (from–to) & 20161–20162 & 20161–20171 & 20161–20172 & 20161–20181 \\ Registries training set & 260153 & 362368 & 474957 & 585936 \\ Semester for testing & 20171 & 20172 & 20181 & 20182 \\ Registries test set & 102215 & 112589 & 110979 & 138746 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Description of the dataset. Currently, we have more data in the data mart, and further analysis can be done on how it is evolving the GAR model among the semesters. Specifically, three more semesters can be analyzed (i.e., until 2019 spring semester). We can answer the research questions RQ1 and RQ2 by (1) analyzing how the NB performance evolves during these three semesters; (2) proposing a method to obtain the best classification algorithm and training set for each assessment activity and course; and, finally, (3) determining a method to deduce a threshold to consider the GAR model of a course as a high- or low-quality model. During the semester, trained models are used by means of an operation that is run on the daily information available at the UOC data mart.