# A Predictive Analytics Infrastructure to Support a Trustworthy Early Warning System
## 5 Results and Discussion
### Technical Robustness and Safety  
The infrastructure described in this paper is especially relevant with regard to this requirement. The infrastructure has been designed to secure data and to increase reproducibility. Data are secured in the persistence data tier encrypted by password, and Docker technology makes the system reproducible on any server. As a research project, the main drawback of the system is the use of one server to support the complete infrastructure. The system is currently deployed on a single server, and any failure on the hardware or software level may negatively impact the availability. However, the system can be easily deployed with Docker technology to a cloud infrastructure and benefits from its advantages such as high reliability and fault tolerance. The accuracy is a crucial factor in the system because learners are classified on risk levels and receive corresponding recommendations based on such risk. Thus, stakeholders will mistrust the system if learners are classified erroneously. To prove the accuracy of the PGAR model, we tested the model in all the institution courses. The objective is to show the average accuracy on each point of the semester in the whole institution. The models were trained with data from the data mart from the 2016 Fall to the 2018 Fall semesters and tested on the 2019 Spring semester. Using the query language for the ETL process, we gathered the learners' profile information, the grades from the assessment activities, and the pass/fail binary information about passing the course. Nine hundred and seventy-nine courses were analyzed, wherein the number of activities ranged from 3 to 11. In the end, 585,936 registries were used for training, and 138,746 were used for testing. The PGAR model was built as described in Section 4.2.1. The accuracy is checked by the metrics defined in Equation (1). To distribute the metric results through the semester, we checked the accuracy for each submodel and course, and it was uniformly distributed among the semester timeline on the basis of the submission date of the respective assessment activity. For instance, for a course with four activities, the metrics values for each activity were distributed to the positions 20%, 40%, 60%, and 80% of the timeline. This distribution aims to identify the average quality on the basis of the activities submitted at each point of the timeline. The average quality is evaluated on the basis of a LOESS regression [85] since this regression shows a better approximation than the linear one when there are many scattered values. The results are plotted in Figure 9, where the LOESS regression is shown, and the scattered plots are the metrics values for all the trained submodels. Table 3 summarizes the LOESS regression for each metric across the semester. As we can observe, the average accuracy (\(ACC\)) for the whole institution was higher than 75%. Specifically, the accuracy ranged from 77.56% to 95.25% from the beginning to the end of the semester, respectively. However, detecting at-risk and non-at-risk learners is quite different because learners at UOC tend to pass the courses and, therefore, classes are imbalanced. Detecting non-at-risklearners (_TNR_) is easier. The accuracy of detecting non-at-risk learners ranged from 84.79% at the beginning of the semester to 96.91% at the end. Detecting at-risk learners started with a lower accuracy at 55.71%, but it reached 91.22% at the end of the course. As we can observe, there were many submodels with low accuracy in the first activities. Although the system tried to find the best classification algorithm for each submodel, the accuracy was highly impacted by the data mart. As aforesaid, data from the data mart were not curated, and inconsistencies could lead to imprecise models. Although the loader service has a module to find such discrepancies, some submodels cannot capture the learners' behavior for the corresponding activity.