## 4 Results of System Performance
### Results of Approach to Merging Features of CNN with Hand-Crafted
#### 4.6.4 Confusion Matrix  
The confusion matrix is the gold standard for evaluating all systems. In this paper, the performance of ANN for diagnosing mixed features between deep learning and hand-crafted features was evaluated. This technique is highly efficient in representing WSIs of  
Figure 11: Error histogram display for diagnosing abnormal cell images of the cervix by ANN based on the features of (**a**). VGG-16, FCH, GLCM and LBP (**b**). GoogLeNet, FCH, GLCM and LBP.  
Figure 12: Display gradient for diagnosing abnormal cell images of the cervix by ANN based on the features of (**a**). VGG-16, FCH, GLCM and LBP (**b**). GoogLeNet, FCH, GLCM and LBP.  
-cervical abnormal cells, fusing VGG-16, FCH, GLCM and LBP features into feature vectors, and fusing GoogLeNet, FCH, GLCM and LBP features into feature vectors. These vectors were fed to an ANN classifier to classify all features of cervical cancer images into five classes (five types of cervical cancer). ANN produced the confusion matrix as a quaternary matrix, in which the main diagonal represents the set of correctly categorized images. The rest of the matrix cells represent the incorrectly categorized images, as in Figure 13. ANN reached an accuracy of 99.4% when using the hybrid features of VGG-16 and hand-crafted. In contrast, the same network when using GoogLeNet and hand-crafted features achieved 99.3% accuracy.  
Regarding samples that the ANN based on the features of VGG-16, FCH, GLCM and LBP failed to diagnose correctly: For the cervix_dyk class, the system failed to diagnose 7 images correctly. For class cervix_koc, the system failed to diagnose 10 images correctly. For class cervix_mep, the system failed to diagnose 4 images correctly. For the class cervix_pab, the system failed to diagnose 5 images correctly. For the cervix_sfi class, the system failed to diagnose 4 images correctly.  
Regarding samples that the ANN based on the features of GoogLeNet, FCH, GLCM and LBP failed to diagnose correctly: For the cervix_dyk class, the system failed to diagnose 6 images correctly. For class cervix_koc, the system failed to diagnose 5 images correctly. For class cervix_mep, the system failed to diagnose 8 images correctly. For the class cervix_pab, the system failed to diagnose 7 images correctly. For the cervix_sfi class, the system failed to diagnose 7 images correctly.  
These approaches are novel and have shown promising results in diagnosing abnormal cell images in the non-cancerous cervical cancer data set. Table 5 summarizes the performance of the ANN classifier with mixed features extracted using CNN and fused with the hand-crafted features. ANN classifier when using features of VGG-16 and hand-crafted yielded an accuracy of 99.4%, specificity of 100%, sensitivity of 99.35%, AUC of 99.89% and precision of 99.42%. In contrast, the same classifier (ANN) with GoogLeNet and hand-crafted features reached an accuracy of 99.3%, specificity of 100%, sensitivity of 99.12%, AUC of 99.56% and precision of 99.23%.  
Figure 13: Confusion matrix for diagnosing abnormal cell images of the cervix by ANN based on the features of (**a**). VGG-16, FCH, GLCM and LBP and (**b**). GoogLeNet, FCH, GLCM and LBP.