# Clinical Trial Classification of SNS24 Calls with Neural Networks
## 4 Experiments
### Fine-Tuning of the Transformers-Based Models
#### 4.3.3 Fine-Tuning of Learning Rate  
During this phase, we explore the effect of different learning rate settings. In particular, we focus on the variants of the learning rate with each optimizer.  
The setting of the learning rate plays a decisive role in the convergence of a neural network model. A too-small learning rate will make a training algorithm converge slowly, while a too-large learning rate will make the training algorithm diverge [38]. A traditional default value for the learning rate is 0.1, and we use this as a starting point on our task problem [38; 39]. For each optimizer (Adam and SGD), we try learning rates within the set of {0.1, 0.01, \(1\times 10^{-3}\), \(1\times 10^{-4}\), \(1\times 10^{-5}\), \(1\times 10^{-6}\), \(1\times 10^{-7}\)}. We maintain the same training procedure as the preliminary models described in Section 4.2.3 except for changing the learning rate. We use the same number of epochs (epochs = 10) as a start. The rest of the hyperparameters remain the same (see Section 4.2.3).  
When varying the learning rate settings with optimizer Adam (Figure 11) and SGD (Figure 12), the training and validation curves of the train and validation dataset are depicted. The results on accuracy as well as loss values are reported.  
Observing the curves learned by using Adam with different learning rates, we find that when setting the learning rate as \(1\times 10^{-3}\) or bigger, the training models fails to converge (refer Appendix B Figure 11). Especially when the learning rate is set with an aggressive value of 0.1, we observe that the loss increases rather than decreases. When the value is set as \(1\times 10^{-4}\) or \(1\times 10^{-5}\), the two models archive similar performance, and the models converge too early on the validation dataset. We also observe that the loss on the validation dataset increases with more epochs. When setting a rather small value of \(1\times 10^{-7}\), we observe that the loss decreases too slow compared to the other learning rates (the loss is around 2.3 after 10 epochs). When setting the learning rate as \(1\times 10^{-6}\), we observe that the results clearly show a downward trend in loss and upward in accuracy over the epochs. This sign shows the model is learning the problem and has learned the predictive skill.  
So, when choosing Adam as the optimizer, the appropriate setting of learning rate is \(1\times 10^{-6}\). The loss curves of the training and validation datasets are decreasing, and they are still showing downward trends by the pre-defined epochs. This observation suggests that the prediction performance could be potentially improved with more epochs (training with more epochs than 10). The best accuracy on the test dataset achieves a score of 79.67%, obtained with the learning rate of \(1\times 10^{-5}\) and Adam optimizer.  
When varying the learning rate setting with optimizer SGD, we can analyze the results similarly as with the Adam optimizer (refer Appendix B Figure 12). We observe that the model fails to converge when the learning rate is set as 0.1 or 0.01; the model converges too early with a learning rate of \(1\times 10^{-3}\); the loss decreases very slow when the learning rate is \(1\times 10^{-5}\) or smaller. The best accuracy on the test dataset achieves a score of 79.72%, obtained with the learning rate of \(1\times 10^{-3}\) and SGD optimizer. But the curve shows no obvious downward trend on \(1\times 10^{-3}\) after 10 epochs. So, with SGD as the optimizer, the appropriate learning rate setting is \(1\times 10^{-4}\).