# Probabilistic Models for Competence Assessment in Education
## 3 Results
### Experiments
#### 3.2.1 Experiments on Bayesian Networks
Figure 8: Posterior predictive sampling from \(PG_{\text{1}}\)-bias. Studying the Error Evolution  
To compare the performance of both models based on Bayesian networks, we studied the evolution of the models' root mean squared error (\(RMSE\), Equation (20)) as a function of the number of observed true grades, keeping the number of students in a class, \(S\), constant. \[RMSE=\sqrt{\frac{1}{N}\sum_{n=1}^{N}(y_{n}-\hat{y_{n}})^{2}} \tag{20}\]  
where \(\hat{y}\) is the true value, \(y\) is the prediction, and \(N\) stands for the number of data points. Grades ranged from 0 to 10, both included. We examined the error evolution in both models as we introduced more teachers' grades as ground truths. We assessed two different criteria to determine which new grade to introduce in each step of the loop:  
1. Random choice (baseline): The next observed ground truth is chosen randomly. 2. Total \(RMSE\) decreasing policy: At each iteration, we picked and observed the true grade (i.e., the teacher's grade) of that student whose assessment was introducing the highest root mean squared error. As a result, we obtain four curves for each model:  
* The red line shows the evolution of the estimations' \(RMSE\) as we introduce new ground truths following a random policy. * The yellow, discontinuous line shows the evolution of the estimations' \(RMSE\) without considering the known ground truths to correct for overly optimistic low error values. Additionally, in this case, a random policy for ground truth injection is followed. * The blue line shows the evolution of the estimations' \(RMSE\) as we introduce new ground truths following an \(RMSE\) decreasing policy. * The violet, discontinuous line shows the same information as the yellow line for the case of the \(RMSE\) decreasing policy. Figure 10 shows the resulting four lines in the case of \(PG_{1}\)-bias. Looking at the pair of continuous lines, we can see how \(RMSE\) falls as the number of known ground truths increases in all cases. As expected, computing the \(RMSE\) of all the grades (ground truths included) produces more optimistic errors. Moreover, it seems that the \(RMSE\) of the unknown grades (discontinuous lines) decreases much more slowly than for the continuous counterpart, that is, adding new ground truths does not reflect quickly on better quality of the predictions. According to Figure 6, introducing a ground truth \(s_{u}\) has an impact on the bias \(b_{v}\) of those peers \(v\) who have made an assessment of \(u\): \(z_{u}^{v}\). There is also a flow of probabilistic influence towards the grader's reliability \(\tau\), which in the case of \(PG1\)-bias is common to all students. The informativeness of the prior determines the relative relevance that observed data and domain expertise have on the posterior distribution parameters. Figure 9: Posterior predictive sampling from \(PG\)-bivariate. Hence, we believe that the choice of an informative prior over \(b_{v}\) is preventing the model from showing an erratic behavior as new data is incorporated, but it also makes it more insensitive to the observations. In general, the random policy yielded worse results for the discontinuous and continuous pairs of lines (red line showing higher error than the blue line, and yellow line showing higher error than the violet line). We can see in Figure 11 the corresponding curves for \(PG\)-bivariate. In this case, the value of the \(RMSE\) was higher, which indicates that our predictions are, in general, worse than those of \(PG_{1}\)-bias. We see again that the random policy yields worse results than the \(RMSE\)-decreasing policy (red line above blue line, and yellow line above violet line). As in the previous case, the slopes of the continuous lines are sharper than those of the discontinuous lines, which implies that the probabilistic influence from the observed teacher grades towards the predictions does not flow easily.