# An Early Warning System to Detect At-Risk Students in Online Higher Education
## 4 Algorithms and Training Dataset Selection for the GAR Model
### Quality Threshold Identification  
Some predictive models aimed at early identify at-risk students defined the best point of intervention during the semester. This point is deduced from the experimental results [23] or is fixed before the evaluation [59]. Our EWS performs interventions from the first assessment activity. Thus,  
Figure 3: LOESS regression plots of the GAR model using Naive Bayes Classifier, selecting best algorithm (Best Alg.) and selecting best algorithm and training set (Best A. TS.) for (**a**) Accuracy, (**b**) TNR, (**c**) TPR, and (**d**) F-score F\({}_{1,5}\) metrics for 20182 semester. the concept of intervention point has nonsense. In our system, the submodels (i.e., the training model applied on each assessment activity) are classified based on high- or low-quality models based on the TPR and TNR since the intervention measures are adapted based on the quality of the model. For instance, an intervention action based on a likelihood to fail the course cannot be applied as-it-is when the submodel has a low-quality TPR. Similarly, a prraise message based on a likelihood to pass cannot be applied when the submodel has a low-quality TNR. In those cases, the interventions are adapted when the likelihood to fail or pass cannot be assured (see Section 5 for further details). This classification is performed based on a quality threshold that can be defined globally for the institution or individually for each course based on the teacher's experience. In this section, we seek the quality threshold globally applicable to demonstrate that the GAR model can be valid with a unique threshold for the whole institution and avoid particular cases. Also, we want to analyze whether the method presented in Algorithm 2 to obtain the best algorithm classifier and training set can improve this threshold with respect to [3] where the threshold selection was performed by observation without any data analysis. The quality threshold identification can be reduced to an optimization problem since the objective is to maximize the threshold while the number of submodels considered high-quality do not worsen significantly. However, this problem is unfeasible to be solved because there is no optimal solution. Thus, we define a function to be maximized in order to approximate the problem  
\[f(x)=x+\frac{\sum_{i\in T}\omega_{i}\sum_{j\in S_{i}}\ j\ \Big{|}\ TPR(j)>x}{\sum_{i\in T} \omega_{i}} \tag{3}\]  
where \(x\) is the threshold value, \(T\in\{0\%,\ 5\%,\ 10\%,\ \ldots,\ 95\%,\ 100\%\}\) positions of the semester timeline, \(w_{i}\) is a weight that can be assigned to the position \(i\) of the semester timeline, and \(S_{i}\) is the set of all the submodels that the submission deadline is in the position \(i\). In summary, the function seeks to sum the threshold value with the weighted average of the number of submodels that their TPR is above the threshold for each position of the semester timeline. We defined a weighted average due to it might being interesting to give more relevancy to some positions of the semester. Note that, we assume in our optimization problem all positions with the same weight (i.e., \(w_{i}=1\)). Moreover, the TPR is used since we are interested to particularly maximize the threshold over submodels for detecting at-risk students. After solving the optimization problem for the 2019 spring semester with an R-based script, the result is illustrated in Figure 4a where _f(x)_ summands are plotted in the axis. As we can observe, both summands generate a curve of Pareto Points with the optimum on the threshold 75% (i.e., the function _f(x)_ maximized on the threshold 75%). Thus, we define high-quality submodels as all submodels with a TPR higher than 75%. In order to further compare with [3], we computed the percentage of explored courses where the TPR and TNR are higher than thresholds 70% and 75% (see Figure 4b). Note that, we removed the multiple plots for the TNR because there was only an average increment of 1-3% over each threshold.