## 4 Empirical Study
### Experimental Discussion  
To realize the recognition quality of the well-known CNNs, in this paper, a comprehensive evaluation was proposed, as detailed above. In the following, an overall empirical discussion is further given, for detailed analysis. * The medical data in real applications is not easy to collect. To make the experiment more reliable, the experimental data were gathered from Kaohsiung Chang Gung Memorial Hospital, instead of crawling the Web. Therefore, due to the problem of insufficient data, overfitting occurred and heavily impacted the recognition quality. To cope with this problem, two operations were adopted: data augmentation and transfer learning. Here, an issue to clarify is the performance comparisons among learning by original data, learning by data augmentation, learning by transfer learning, and learning by fusing data augmentation and transfer learning. This comparison can be explained by Figure 11, summarizing Tables 5 and 8 and Figure 10. From this comparison, we can determine that, whatever the model, without the fusion of transfer learning and data augmentation, the best accuracy cannot be achieved. * In addition to the four observations mentioned in Section 4.4, another point to show is the performance comparison in terms of the AUC. Figure 12 reveals the AUC comparisons of models with accuracies larger than 0.9, namely RN50, RN101, DN121, DN201, INv3, and ENB0. Basically, it can be divided into two observation spaces, by the line where the false positive rate (FPR) equals 0.1. Before FPR 0.1, DN121 is better than the others. To the contrary, the performances of RN101 and DN201 are relatively close but higher than those of the other models, after FPR 0.1. Overall, RN101, DN201, and INv3 are the highly considerable models, in terms of the AUC. * Besides the AUC, a further comparison to show the interdiscrimination ability of TP, TN, FP, and FN for each model is the hybrid validation, which indicates the average of the F1-score, AUC, and kappa metrics. Figure 13 depicts the validation result, showing that the top three models were DN201, DN169, and RN101, although their values were close. This result is consistent with those given above. However, by considering the overall performances in Table 8, the best two models for recognizing scaphoid fractures are DN201 and RN101. * In addition to the effectiveness evaluation, another important issue is the efficiency. Considering this issue, Figure 14 shows the execution time of compared models. The comparison result revealed that DenseNet, InceptionNet, and EfficientNet had higher costs than the other models. By integrating the results of Figures 13 and 14, three further viewpoints are given here. First, from the effectiveness point of view, DN201, DN169, and RN101 are three candidate models. Second, from the efficiency point of view, VGG16, RN50, and RN152 are three candidate models. Third, from a balanced viewpoint, the top three models are VGG16, RN50, and RN152 because the referred balances between test performance and training cost are higher than that of the other \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**ModelMetric** & **Sensitivity** & **Specificity** & **Precision** & **F1-Score** & **AUC** & **Kappa** & **Accuracy** \\ \hline VGG16 & 0.861 & 0.806 & 0.816 & 0.838 & 0.860 & 0.667 & 0.833 \\ VGG19 & 0.833 & 0.722 & 0.750 & 0.789 & 0.870 & 0.556 & 0.778 \\ RN50 & 0.889 & 0.833 & 0.842 & 0.865 & 0.910 & 0.722 & 0.861 \\ RN101 & 0.889 & * 0.889 & * 0.889 & 0.889 & * 0.950 & 0.778 & 0.889 \\ RN152 & 0.806 & 0.806 & 0.806 & 0.806 & 0.880 & 0.611 & 0.806 \\ DN121 & 0.917 & 0.833 & 0.846 & 0.880 & 0.930 & 0.750 & 0.875 \\ DN169 & 0.917 & 0.861 & 0.868 & 0.892 & 0.890 & 0.778 & 0.889 \\ DN201 & * 0.944 & 0.861 & 0.872 & * 0.907 & 0.910 & * 0.806 & * 0.903 \\ INv3 & 0.889 & * 0.889 & * 0.889 & 0.889 & 0.930 & 0.778 & 0.889 \\ ENB0 & * 0.944 & 0.778 & 0.810 & 0.872 & 0.920 & 0.722 & 0.861 \\ \hline \hline \end{tabular} Note that * indicates the best performance in each metric (attribute).  
\end{table}