## 5 Classification
### Hyperparameter Selection  
The split into a training set and a validation set was 70% and 30%, respectively. The initial weights for all layers were drawn from the Gaussian distribution with mean 0 and standard deviation 0.01. Initial biases were set to zero. The L2 value was set to 0.0005 and mini-batch size was set to 128. The values of all the previously mentioned hyperparameters were fixed and we used manual tuning only for choosing the combination of the number of epochs and the learning rate drop period (LRDP). Two models with different values of the two parameters were trained on the original data set (i.e., no data augmentation applied). One model was trained on the augmented data set with \(s=1100\) and \(s=350\), respectively. Several models with various values of the two parameters were trained on the augmented data set with \(s=200\) and \(s=50\), respectively. The results of training these models are presented in Table 2, in which performance is presented as true positive rate (TPR, i.e., sensitivity). The initial values of the two parameters applied to training on each data set are selected empirically. As a result of running these tests, the best model in terms of performance is the model trained on the augmented data set with \(s=50\) (i.e., 2,439,744 training examples), the number of epochs = 8, and the LRDP = 3. Initial learning rate was set to 0.01 and when the same value was applied to the number of epochs and the LRDP the learning rate was kept constantly at its initial value. The learning rate decay schedule (LRDS) was applied when the values of the number of epochs and the LRDP were different of each other. In the LPDS method, the learning rate is dropped by a factor of 0.1 (i.e., the updated learning rate will be the current learning rate \(\times\) 0.1) when a given number of epochs is reached. This given number of epochs is the effective value of the LRDP. Motivation for using the LPDS method is as training proceeds with shorter leaps on the loss function surface from some point on, the optimal value for the weights (i.e., in terms of performance as a classifier) can be found more accurately. If only the short leaps would be applied, the number of epochs should be very large, thus resulting in significant increase of training time. The challenge is to find the points from where on the learning rate should be reduced. We approached this problem in two ways. We fixed the LRDP value and altered the number of epochs. Initially, the problem was to find a suitable starting value for the LRDP. It was intuitively clear that the LRDP value should increase as the number of epochs increases. A small value of the LRDP combined with a high value of the number of epochs w  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Number of Training Examples** & **Number of Epochs** & **LRDP** & **TPR Training** & **TPR Generalization** \\ \hline
9312 & 30 & 30 & 0.7175 & 0.6995 \\
9312 & 60 & 60 & 0.7362 & 0.7052 \\
121,056 & 25 & 10 & 0.8687 & 0.8662 \\
363,168 & 18 & 7 & 0.9137 & 0.9187 \\
623,904  
underfitting. We also fixed the number of epochs and altered the LRDP value instead. The same initial value problem concerns this approach as well. However, the size of the respective data set should give some guidance for choosing the initial values. Moreover, as the number of training examples increases, the number of epochs should decrease in order to avoid overfitting. We applied the dropout technique for improving the performance of our CNN [34; 36]. We trained models with fixed hyperparameter values with and without the dropout technique. If overfitting occurs, the results in terms of classification performance should be better as the dropout technique is applied compared to those models for which it is not applied. These tests indicate that some overfitting occurs when the models were trained on the augmented data sets but not necessarily on the original data sets.