# Clinical Trial Classification of SNS24 Calls with Neural Networks
## 5 Results and Discussion  
We consider three deep neural network architectures in our experiments. We use these neural networks combined with different representations for words and texts. These built models and their respective testing accuracies results are presented in Table 5.  
Among them, the transformers-based model using BERT-base language model achieves a result of 78.15%. The CNN-based approach achieved an accuracy of 76.56%, and the RNN-based approach achieved an accuracy of 75.88%. The transformers-based approach is shown to be better than the one using the CNN-based or RNN-based approach. We then fine-tune the transformers-based models. A better model with a higher accuracy score is obtained after we fine-tune the hyperparameters on learning rate, optimizer, number of epochs, and batch size step by step. The best fine-tuning model achieves an accuracy of 80.57% on the testing set. We also compare the performance between the Adam and SGD optimizers using their validation accuracy distributions over the 150 epochs. In Figure 8, we present the results with a box plot using the data that both optimizers achieve with their best learning rate, where Adam has a learning rate of \(1\times 10^{-6}\) and SGD has a rate of \(1\times 10^{-4}\) The outliers that are smaller than the minimum for both optimizers are removed in the figure for a clear presentation. This does not affect the comparison between the distributions. We choose a learning rate of \(1\times 10^{-6}\) for Adam and \(1\times 10^{-4}\) for SGD because their learning curves show clear downward trends in losses and upward trends in accuracies. Also, they are able to achieve comparative performances compared to other learning rates (see Figures A1 and A2). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Optimizer** & **Learning Rate** & **Epochs** & **Batch Size** & **Testing Accuracy** \\ \hline SGD & 0.1 & 10 & 128 & 12.12\% \\ SGD & 0.01 & 10 & 128 & 73.76\% \\ SGD & \(1\times 10^{-3}\) & 10 & 128 & 79.72\% \\ SGD & \(1\times 10^{-4}\) & 10 & 128 & 75.01\% \\ SGD & \(1\times 10^{-5}\) & 10 & 128 & 38.89\% \\ SGD & \(1\times 10^{-6}\) & 10 & 128 & 15.01\% \\ SGD & \(1\times 10^{-7}\) & 10 & 128 & 6.26\% \\ \hline Adam & \(1\times 10^{-6}\) & 150 & 128 & 79.96\% \\ SGD & \(1\times 10^{-4}\) & 150 & 128 & 79.76\% \\ \hline Adam & \(1\times 10^{-6}\) & 50 & 16 & 80.57\% \\ [PERSON] & \(1\times 10^{-6}\) & 50 & 32 & 80.19\% \\ [PERSON] & \(1\times 10^{-6}\) & 50 & 64 & 80.20\% \\ [PERSON] & \(1\times 10^{-6}\) & 50 & 128 & 80.25\% \\ [PERSON] & \(1\times 10^{-6}\) & 50 & 256 & 80.08\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: _Cont._ \begin{table}
\begin{tabular}{c c c} \hline \hline
**Deep Learning Architecture** &
\begin{tabular}{c} **Fine-Tuned** \\ **Hyperparameters** \\ \end{tabular} & **Testing Accuracy** \\ \hline CNN-based architecture & - & 76.56\% \\ \hline RNN-based architecture & - & 75.88\% \\ \hline Transformers-based architecture & - & 78.15\% \\  & learning rate & 79.67\% (Adam, lr = \(1\times 10^{-5}\)) \\  & learning rate & 79.72\% (SGD, lr = \(1\times 10^{-3}\)) \\  & epochs & 79.96\% (Adam, epochs = 150) \\  & epochs & 79.76\% (SGD, epochs = 150) \\  & batch size & 80.57\% \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Results comparison among the three deep learning architectures. We observe that the model with Adam optimizer achieves better top accuracy on the validation dataset than the model with SGD. Besides, when comparing Figures 6 and 7, Adam has been found to attain the top accuracy performance with a smaller number of epochs necessary (epoch = 41), while SGD requires more to converge (epoch = 47).