# Clinical Trial Classification of SNS24 Calls with Neural Networks
## 8 Conclusions and Future Work  
SNS24 is a telephone and digital public service which provides clinical services. SNS24 plays a significant role in identifying users' clinical situations according to their symptoms. There are a group of possible clinical algorithms defined at present, and choosing the appropriate clinical algorithm is important in each telephone triage. We carried out experiments on a dataset containing 269,654 call records belonging to 51 classes. Each record is labeled with a clinical pathway chosen by the nurse that answered the call. Three neural network architectures were trained to produce classification models. CNN, RNN, and transformers-based approaches achieve accuracies of 76.67%, 78.03%, and 78.15%, respectively. The best performance obtained, with an accuracy of 80.57%, was reached using a fine-tuned transformers-based model. The fine-tuning strategy used showed up to be efficient to improve the model performance. Also, the experiments revealed that, when setting the appropriate learning rate, the Adam optimizer achieved better performance compared to SGD. These results suggest that using deep learning is an effective and promising approach to aid the clinical triage of phone call services. We carried out further analysis using the fine-tuned neural network models. We used the trained model to analyze similar clinical symptoms. These results provide valuable \begin{table}
\begin{tabular}{c c c} \hline \hline
**Agreement** & **Id** & **Count** \\ \hline all experts & 1, 6, 9, 11, 13, 15 & 6 \\ triage, NN & 7 & 1 \\ triage, SVM & 12 & 1 \\ SVM, NN & 2, 3, 5, 8, 10, 14, 16, 19 & 8 \\ none & 4, 17, 18, 20 & 4 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Agreements between SNS24 triage, neural network, and SVM experts for 20 examples. The _Id_ column indicates the texts that the experts agreed with, and the referring text can be found with the _Id_ in Table 16. The _Count_ column calculates the total number of agreements among the experts. analysis for the SNS24 Health Calls center to do deep analysis among similar symptoms. We also compared the predictions from the neural network model, the support vector machines model from our previous work, and the original labels provided by SNS24. As future work, and since the neural network models performed well on the SNS24 clinical trial classification task, we intend to fine-tune these models with other hyperparameters besides the ones explored in this work. As a continuation of the work, we plan to build models using a much larger SNS24 dataset that includes three full years of data. To support the choices made by ML models and better understand why one clinical pathway is chosen over others, we intend to apply explainable artificial intelligence (XAI) techniques. Pursuing the line of work from Section 7 we intend to use a new subset of examples and extend the presented comparisons to the new models being created with the three-years dataset, as well as adding the insight from a few experienced nurses from the health-line. Conceptualization and methodology, T.G. and P.Q. ; software, validation and formal analysis, H.Y. ; investigation, R.V. ([PERSON]); resources, C.S.P., J.O., M.C.F., J.M., A.R.P., N.F. and C.G. ; data curation, R.V. ([PERSON]); writing, H.Y. and T.G. ; visualization, H.Y. ; supervision, T.G. ; project administration, P.Q. and T.G. ; funding acquisition, P.Q. All authors have read and agreed to the published version of the manuscript. This research work was funded by FCT (Fundacao para a Ciencia e a Tecnologia), I.P, within the project SNS24.Scout.IA: Aplicacao de Metodologias de Inteligencia Artificial e Processamento de Linguagem Natural no Servico de Triagem, Aconsehamento e Encaminhamento do SNS24 (ref. DSAIPA/AI/0040/2019). Not applicable. Authors would like to thank [PERSON] for helping building the experimental environments. The authors declare no conflict of interest. The following abbreviations are used in this manuscript: