# Probabilistic Models for Competence Assessment in Education
## 3 Results
### Experiments
#### 3.2.1 Experiments on Bayesian Networks  
Posterior predictive sampling (Equation (19)) is a useful method to compare predictions of new data based on past observations with the real data points. This comparison gives a sense of the estimations' quality and may help during the tuning of the hyperparameters. \[\pi_{S}(y|\hat{y})=\int\pi_{S}(\theta|\hat{y})\pi_{S}(y|\theta)\text{d}\theta \tag{19}\]  
We sampled from the posterior predictive distribution of the two tested \(BN\) models, namely \(PG_{1}\)-bias and \(PG\)-bivariate, comparing the histograms of predictions with the histogram of samples. Sampling from the posterior predictive distribution helped us calibrate the models' hyperparameters for the prior distribution. In general, weakly informative and very weakly informative priors were proposed. However, we introduced some domain knowledge by centering the distributions of grades around positive numbers and restricting the variance parameters to only positive values, to name a couple of examples. Recalling the prior distribution equation for model \(PG_{\text{1}}\)-bias (Equation (10)), the chosen values in our implementation for \((\alpha_{0},\beta_{0},\gamma_{0},\eta_{0},\mu_{0})\) were \(\alpha_{0}=0.01\) (very weak prior), \(\beta_{0}=0.01\) (very weak prior), \(\gamma_{0}=10.0\) (weak prior), \(\eta_{0}=1.0\), and \(\mu_{0}=5.0\), whereas in the case of \(PG\)-bivariate, the chosen hyperparameters were \(x_{0}=0.0\) (weak prior), \(\gamma=0.25\) (weak prior), \(\bar{\mu}=5.0\), \(\bar{\sigma}=10.0\) (weak prior), and \(\eta=1.0\) (weak prior). Figure 8 represents a histogram of predicted peer grades \(z\)'s from the posterior predictive distribution in model \(PG_{\text{1}}\)-bias for a class of 50 students (without ground truths). We observe the experimental histogram of observed peer grades in blue, \(\bar{z}\). The histogram of predictions (\(z\)) represents the distribution of data not part of the training set. This kind of prediction was repeatedly executed (20 times) to ensure that the results were consistent. Peer grades ranged from 0 to 10, following Equation (13). We calculated the percentage overlapping between the histogram of samples and the histogram of predictions using the vector of elementwise minima [14]. The overlapping range was between 0% (no overlap) and 100% (identical distributions). On average, we observed an overlapping of 82%. The range of values obtained for the overlapping in all the performed experiments was between 76% and 84%. This value implies that the model captures the data generation system's dynamics, which results in samples \(z\) that follow a very similar distribution to that from which \(\bar{z}\) were sampled. A similar figure to Figure 8 is obtained when we perform a posterior predictive sampling of the model presented in this paper, \(PG\)-bivariate. The overlapping, in this case, fell within the range [73-81%], with an average value of 77% in 20 different runs (see Figure 9). Although in this case, the model does not offer a causal explanation of the parameters \(\mu_{v}\) and \(\sigma_{v}\) from the normal distribution governing \(z^{v}\) (Equation (14)), the dynamics of the data generation system are captured. Intuitively, one would have expected a notably worse performance than \(PG_{\text{1}}\)-bias for several reasons. First of all, \(PG_{\text{1}}\)-bias models each use peer grade as a random variable that follows its normal distribution. On the other hand, the definition of the location parameter as the sum of the real grade and the graders' bias allows a direct ascension of the flow of probabilistic influence towards \(s_{v}\). In the case of \(PG\)-bivariate, we adopt the perspective of a passive observer, representing the set of grades emitted by a referee as a random variable following a normal probability density function. Notice that in this case, no mention of who is evaluated by whom is being made. Hence, the location parameter \(\mu_{v}\) cannot be decoupled as in the former case to further ease the probabilistic influence flow.