## 4 Results
### Evaluation of Prediction Models  
As previously mentioned models were built to assess the impact of larger datasets and older models in the model performance. Table 2 presents the balanced accuracy of the models when considering the 1, 3 and 5 most probable clinical pathways (top-1, top-3, top-5) along with the average time to predict 1000 examples (over 10 runs). The 3 months models were built and assessed in previous works [2; 3].  
Observing the top-1 accuracies, they range from 63.1% and 78.3% for CNN model with 3 months and 2 years data, respectively. It is curious to observe that the lowest and highest performance were obtained with a deep neural network architecture, which is in accordance with previous studies: deep architectures are able to achieve higher performances than other algorithms when a large amount of training data exists, overfitting easily when there is not enough data. We can also observe that SVM models with different sizes of data are more stable than CNN ones: the difference in accuracy between the 3 months and the 2017-2018 SVM and CNN data models is 2.5% and 15.1%, respectively.  
It is also possible to conclude that adding more data consistently improves the models and that models built with more recent data are also better. For example, using the 2 previous years increments the accuracy by 0.2% when compared with a model built with only the previous year (2017-2018 vs. 2018); using a model built with data from the previous year increments the accuracy by 0.8% (CNN) to 1.1% (SVM) when compared to a model built with the data from 2 years apart (2018 vs. 2017).  
When looking at the model performance considering the 3 most probable pathways (top-3), SVM is substantially better than CNN for the 3 months model (15.2% higher) presenting very similar accuracies for the other models (with SVM surpassing CNN in 2017  
\begin{table}
\begin{tabular}{r c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{**Accuracy**} & \multicolumn{3}{c}{**Pred Time (ms)**} \\ \hline  & & **SVM** & & & **CNN** & & **SVM** & **CNN** \\ \hline
**Training Set** & **Top 1** & **Top 3** & **Top 5** & **Top 1** & **Top 3** & **Top 5** & & \\ \hline
3 months & 0.743 & 0.927 & 0.959 & 0.631 & 0.775 & 0.804 & 14.82 & 12.31 \\
2017 & 0.755 & 0.932 & 0.963 & 0.772 & 0.931 & 0.961 & 54.32 & 12.41 \\
2018 & 0.766 & 0.938 & 0.968 & 0.780 & 0.937 & 0.964 & 54.57 & 12.50 \\
2017–2018 & 0.768 & 0.940 & 0.969 & 0.782 & 0.940 & 0.967 & 92.48 & 14.19 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Test set balanced accuracy (top 1, top 3 and top 5 most probable clinical pathways) and average prediction time (for 1000 examples) of SVM and CNN models with different training sets.  
Figure 9: Distribution of referrals for the 10 most common pathways.  
and 2018 models by 0.1% and having an equal accuracy for 2017-2018 model). This trend is also true considering the 5 most probable pathways: SVM is 15.5%, 0.2%, 0.4% and 0.2% higher than CNN for the 3 months, 2017, 2018 and 2017-2018 models.  
Finally, observing the prediction time, and as expected, SVM takes more time to predict when models are built with more data, with CNN maintaining an almost fixed time independently of the size of the training set.  
Table 3 presents the weighted precision, recall (values are repeated here as they are usually presented along with precision and F1) and F1 values over the test set (2019 data) of the 8 trained models (2 algorithms: SVM and CNN; 3 datasets: 3 months, 2017 data, 2018 data and 2017-2018 data).  
Similarly to balanced accuracy (recall), the minimum and maximum precision and F1 values were obtained for the CNN model with 3 months and 2 years data, respectively; precision range between 65.7% and 78.5%, and F1 between 64.2% and 78.2%. We can also observe, when comparing CNN and SVM models, the same trend for these measures (as observed in accuracy): the performance difference between the 3 months and 2 years models varies between 1.7% (precision) and 2.3% (F1) for SVM while, for CNN, it varies between 12.8% (precision) and 14% (F1).