# Diag-Skills: A Diagnosis System Using Belief Functions and Semantic Models in ITS
## 6 Discussion and Conclusions  
As presented in the previous section, Diag-Skills showed a good prediction accuracy in the two studies. We noticed a better precision when the predictions were based on both top-down and bottom-up propagation (85% precision in the first experiment and 92% precision in the second experiment). Furthermore, we investigated the prediction when there were no effective evaluations to rely on by comparing Diag-Skills' predictions with the instructor's predictions. We added to these the cases of the first experiment where Diag-Skills failed to predict the correct state. The results showed that in this case, the module would at least make the same predictions as the instructor, as the Kappa measure showed an almost perfect agreement. This result suggests that Diag-Skills is capable of replicating human diagnosis with a high level of agreement. Moreover, when analyzing the incorrect predictions, we noticed that the predicted state was always better than the effective one and never the opposite. This proves, to some extent, that Diag-Skills manages to capture the slips and the cases when there is an improvement in learning, but fails to predict the states when there is a major flaw in the skill. This should be investigated by further evaluations. Nevertheless, we noticed that in the second experiment, the mass distributions of incorrect predictions were all marked by some degree of conflict. Indeed, the conflict's belief \(m(c)\) was always greater than 0.2, indicating the presence of conflict between beliefs coming from different propagations. However, this amount of conflict was not great enough to consider that this state is conflictual. We believe, even if this amount of conflict is not great enough, that this information should be provided to the orchestration engine in order to warn it of a possible unexpected flaw in the skill. These results suggest that Diag-Skills' approach is a promising method for the diagnosis of learners' skills. In particular, we showed that is possible to represent all kinds of data imperfection using a single formal framework and that is possible to highlight inconsistencies when they are relevant using this same framework. This has direct implications on the orchestration process in ITS as this information facilitates the identification of the skills that require additional evaluations. At a second level, it also has implications on the evaluation process in ITS, as this could indicate that the assessor (a computer system or an instructor) may have made inconsistent assessments and that the evaluation process should be revised. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Diag-Skills Predictions** & _Acquired_ & _Not Acquired_ & _Probably Acquired_ & _Probably Not Acquired_ \\ \hline _acquired_ & 16 & 0 & 0 & 0 \\ _not acquired_ & 0 & 6 & 2 & 0 \\ _probably acquired_ & 0 & 0 & 12 & 0 \\ _probably not acquired_ & 0 & 0 & 2 & 12 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Confusion matrix that summarizes the comparison between Diag-Skills’ predictions and the effective states of the skill “Create interactive component’s states”. Aside from the contributions, we would like to highlight some limitations. First, at this stage, the diagnosis relies on evaluation scores to determine the skills' states. It also depends to some extent on the number of previous evaluations. However, it does not explicitly take into consideration the amount of time between two successive evaluations nor the time between the course and the evaluation. Future work should address how this parameter can be integrated into the revision process. Also, in the translation function, we set the threshold beyond which a skill's state is considered to be conflictual to 0.75 (similarly to other states). However, the experiments showed that in reality, the states were conflictual even when the mass related to the conflict was much smaller than 0.75.