## 5 Error Analysis, Explainability and Meta-Evaluation
### Error Distribution by Class  
Some disadvantageous facets of the problem at hands are: the unbalanced nature of the dataset, the considerable number of clinical pathways with pathways sharing common symptoms as referred by the healthcare experts (for instance, \"Abdominal pain\", \"Nausea and vomiting problems\" and \"Diarrhea\", or \"Crying child problem\" and \"Body temperature change problem\") and the existence of very short text sequences describing the contact reason of the call. Upon these conditions, we consider that the results achieved are very satisfactory. Table 4 shows the individual precision, recall and F1 values for the 10 most and least common pathways along with minimum, maximum, average and standard deviation values obtained for the SVM and CNN 2017-2018 models (Table A2, in Appendix A, presents the results for all pathways sorted, in descending order, by the pathway support). As can be observed, there is a huge performance difference between classes: F1 ranges between 0.000 and 0.931 for CNN and 0.013 and 0.919 for SVM (precision and recall have similar ranges). As expected, the average performance is higher for 10 most frequent pathways when compared with the 10 least frequent ones, with F1 equal or above 0.792 and equal or below 0.395, respectively. Moreover, standard deviation is much higher for the least common pathways; for example, and for the SVM model, the least common pathways \begin{table}
\begin{tabular}{r c c c c c c} \hline \hline  & \multicolumn{3}{c}{**SVM**} & \multicolumn{3}{c}{**CNN**} \\ \hline
**Training Set** & **Prec** & **Rec** & **F1** & **Prec** & **Rec** & **F1** \\ \hline
3 months & 0.754 & 0.743 & 0.745 & 0.657 & 0.632 & 0.642 \\
2017 & 0.761 & 0.755 & 0.755 & 0.776 & 0.772 & 0.772 \\
2018 & 0.768 & 0.766 & 0.765 & 0.782 & 0.780 & 0.779 \\
2017–2018 & 0.771 & 0.768 & 0.768 & 0.785 & 0.783 & 0.782 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Test set precision, recall and F1 (weighted average) of SVM and CNN models with different training sets. have a value of 0.260 while most common pathways present a value of 0.082 (similar trends are valid for the CNN model and for precision and recall). Nonetheless, there are pathways, with similar support values, that seem to be much more difficult to classify than others: for example, while \"Emergency\" presents an F1 value of 0.140, \"Foreign body problem\" has an F1 of 0.679 (with support of 3201 and 3220, respectively); the case is similar, but with a smaller difference, between \"Flu syndrome\" and \"Diarrhea\", with the F1 values of 0.609 and 0.810 and support of 41827 and 37202, respectively. Figure 10 shows the F1 values vs. the support of the class over the test set for the 2017-2018 SVM and CNN models. It can be observed that, as expected, the class prediction performance tends to increase with a higher number of examples. In a few cases, this tendency is interrupted (see Table 10, in Appendix A for the pathways names): \"Geriatric problems\" presents a sharp decrease when compared to pathways with similar support such as \"Diabetes problem\" or \"Finger problem\", with F1 values of 0.460, 0.711, and 0.697 for the SVM model, respectively; the same occurs with \"Nonspecific problem\" when compared to \"Blood pressure problem\" or \"Anxiety problem\" (F1 values of 0.386, 0.780, and 0.735 for the SVM model, respectively) or \"Flu syndrome\" when compared to \"Diarrhea\" or \"Urinary problem\" (F1 values of 0.609, 0.810, and 0.885 for the SVM model, respectively). On the other hand, \"Solar exposure problem\" and \"Elbow problem\" present higher performance than pathways with few similar cases. This analysis