## 5 Error Analysis, Explainability and Meta-Evaluation
### Meta-Evaluation  
This meta evaluation aims at assessing the differences and similarities between the clinical pathways chosen initially by the nurses (and used as ground truth to build and evaluate the models), the ML models and a clinical expert. For this study a total of 200 examples were chosen randomly, 10 for each of the 10 most frequent and the 10 least frequent clinical pathways. These examples were presented to a clinical expert and to the 2017-2018 SVM and CNN models. For some examples (totalling 14), and looking only to the \"Contact Reason\" information, the clinical expert was not able to decide the correct pathway, so these examples were not considered for the analysis. It is important to stress out that by having access only to the \"Contact Reason\" text, the clinical expert was not able to consider any other information the nurse had when attending the call. It is also important to have in mind that, although being specifically trained for choosing the most suitable clinical pathway, the choice, seen as ground truth by the ML algorithms, is made by different nurses (for the 3 years it totals 1888 different nurses) and with a diversity of clinical practice. Table 6 resumes, per clinical pathway, the agreement between each pair of decisions (clinical expert, ground truth, SVM and CNN). Observing the total number of agreements one can conclude that, while the agreement of the ML models is the highest (166), the clinical expert agrees the less with the pathway chosen by the SNS24 working nurses (58). Moreover, the number of agreements between both ML models and the nurses or the clinical expert is similar (74 and 70 with the nurses; 104 and 109 with the clinical expert, for SVM and CNN, respectively) but they are higher with the clinical expert by 30 cases or more. Looking at the agreements with the clinical expert (columns e-g, e-s, e-c), a conclusion that stands out is that the agreement with the ML models (columns e-s, e-c) is much higher \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline
**Clinical Pathway** & **Total** & **e-g** & **e-s** & **e-c** & **g-s** & **g-c** & **s-c** \\ \hline Cough & 10 & 2 & 4 & 5 & 5 & 5 & 9 \\ Nausea and vomiting problem & 9 & 2 & 4 & 3 & 3 & 3 & 8 \\ Abdominal pain & 10 & 5 & 5 & 6 & 6 & 7 & 9 \\ Oropharynx problem & 9 & 4 & 6 & 7 & 3 & 4 & 9 \\ Rash & 8 & 1 & 2 & 3 & 4 & 4 & 7 \\ Flu syndrome & 10 & 3 & 7 & 8 & 4 & 3 & 8 \\ Diarrhea & 9 & 1 & 2 & 3 & 4 & 4 & 9 \\ Urinary problem & 10 & 7 & 8 & 7 & 9 & 10 & 9 \\ Body temperature change problem & 10 & 3 & 7 & 7 & 5 & 3 & 7 \\ Chest pain & 9 & 5 & 7 & 8 & 7 & 6 & 9 \\ \hline Sub-total (10 most frequent) & 94 & 33 & 52 & 57 & 50 & 49 & 84 \\ \hline Fainting or lipothymia & 10 & 3 & 6 & 5 & 5 & 4 & 7 \\ Foreign body problem & 10 & 8 & 6 & 5 & 8 & 7 & 8 \\ Emergency & 7 & 3 & 2 & 2 & 2 & 0 & 6 \\ Breastfeeding problem & 10 & 0 & 6 & 7 & 3 & 2 & 9 \\ Asthma or wheezing problem & 9 & 0 & 6 & 6 & 0 & 0 & 8 \\ Elbow problem & 10 & 4 & 2 & 3 & 2 & 3 & 9 \\ Solar exposure problem & 8 & 6 & 5 & 6 & 4 & 5 & 8 \\ Crisis adaptation problem & 8 & 0 & 6 & 6 & 0 & 0 & 9 \\ Measles problem & 10 & 0 & 6 & 6 & 0 & 0 & 10 \\ Heat problem & 10 & 1 & 7 & 6 & 0 & 0 & 8 \\ \hline Sub-total (10 least frequent) & 92 & 25 & 52 & 52 & 24 & 21 & 82 \\ \hline TOTAL & 186 & 58 & 104 & 109 & 74 & 70 & 166 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Agreement between experienced analyst (e), ground truth (g) and SVM (s) and CNN (c) models. when compared with the agreement with the SNS24 working nurses (column e-g); this is true for the least and for the most frequent pathways, with the number of agreements more than doubling for the least frequent pathways (52 for ML models vs. 25 for the nurses). Moreover, the agreement with the ML models is similar for the most and least frequent pathways (between 52 and 57). On the other hand, the agreement with the nurses is higher for the most frequent pathways when compared to the least frequent ones (33 vs. 25).