## 6 Results and Discussion
### Experimental Results
#### 6.2.2 Classification Results  
Our experimental results are shown in two folds. First, bootstrapping is adopted, and a limited set of results are reported to show the performance of HyCAD across several testing data partitions and investigate its stability relative to Norm-VGG16 and RESNET-50 and to clarify the impact of hand-crafted features' fusion with deep learning architectures. In the second fold of experiments, elaborate results are reported on the same training and test percentages used by [PERSON] et al. [4] and [PERSON] et al. [20]. The results of the best run are displayed to be able to compare to [PERSON] et al. [4] and [PERSON] et al. [20]. 1. Bootstrapping Experiments Fold  
Ten bootstrapping experiments are conducted to evaluate the performance of the models at different boot strapped test partitions. In each experiment, the 109,312 images are shuffled and a random selection of 108,312 images is used as training dataset and the rest of the 1000 images are used as testing dataset. Each new partition tested on our implemented architectures (HyCAD, Norm-VGG16 integrating kernel regularizer and ResNet-50 integrating kernel regularizer) taking into consideration that the testing dataset is partitioned equally between four classes (CNV, DME, DRUSEN and NORMAL). Table 3 presents the mean values and the standard deviation of overall accuracy and urgent group sensitivity for 10 different bootstrapped partitions. The depicted results reveal the improvement achieved through integrating the handcrafted features with Norm-VGG16 integrating kernel regularizer. The proposed HyCAD architecture attains the highest mean accuracy and mean sensitivity for the critical Urgent referrals compared to Norm-VGG16 integrating kernel regularizer and ResNet-50. In addition, it attains the lowest standard deviation in comparison to the pure CNN architectures. Compared to Norm-VGG16 integrating kernel regularizer, HyCAD achieves a substantial increase of 2.9% and 4.9% in terms of mean accuracy and mean sensitivity of Urgent referrals, respectively. In addition, the HyCAD model has a lower standard deviation relative to the pure deep learning architecture Norm-VGG16 integrating kernel regularizer. Such findings emphasize the positive impact of fusing hand-crafted features with learned features from deep learning architecture. The Training and Validation losses and accuracies for the two deep learning architecture (Norm-VGG16 kernel regularized and ResNet-50 kernel regularized) for each epoch are shown in Figure 13. The Norm-VGG16 architecture best model achieves training accuracy of 94.86% and training loss of 15.75%. In addition, it achieves a testing accuracy of 97.3% and testing loss of 8.34%. ResNet-50 achieves a training accuracy of 96.95% and a training loss of 9.02%. In addition, it achieves a testing accuracy of 97% and a testing loss of 8.76%. From the shown learning curves, it is manifest that the performance stabilizes around the 15 th epoch, which can lead to reducing the training time considerably. The confusion matrices of the best models of the presented experimental scenarios on the test set are presented in Figure 14. ResNet-50 (Figure 14C) achieves the lowest testing accuracy of 97%. The model's sensitivity for Urgent referral group is 98.6%, while its specificity is 95.4%. The confusion matrix output by applying the best Norm-VGG16 model on the test set is presented in Figure 14B. The confusion matrix illustrates that 973 out of 1000 testing images were correctly classified. The model's sensitivity for the Urgent referral group is 98%, while its specificity is 96.6%. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Mean Accuracy \(\pm\) Std** & **Mean UR Sensitivity \(\pm\) Std** \\ \hline The proposed HyCAD architecture & 97.2 \(\pm\) 1.2 & 98.1 \(\pm\) 1.1 \\ \hline The proposed Norm-VGG16 & 94.3 \(\pm\) 2.2 & 93.2 \(\pm\) 4.6 \\ (kernel regularized) architecture & 96.2 \(\pm\) 1.4 & 97.1 \(\pm\) 1.5 \\ \hline \hline \end{tabular}
\end{table}