# Classification of Shoulder X-ray Images with Deep Learning Ensemble Models
## 4 Experiments
### Classification Results
#### 4.4.3 Classification Results of Our Ensemble Models  
ResNeXt50 with Spinal FC, DenseNet169 with Standard FC, and DenseNet201 with Spinal FC were used in EL1:  
* ResNeXt50 with Spinal FC was selected because the AUC score achieved by detecting fracture images is the second highest, with 0.8783, after DenseNet169 with Standard FC. * DenseNet169 with Standard FC was selected because its classification had the highest test accuracy, [PERSON]'s kappa score, and AUC score among all models used in this study. * DenseNet201 with Spinal FC was selected because the test accuracy and [PERSON]'s kappa score were the second highest after DenseNet169 with Standard FC among all models and the highest among models with Spinal FC. The classification results of the EL1 model are provided in the following figures and tables. In Table 14, ResNeXt50b represents the ResNeXt50 model with Spinal FC, DenseNet169 represents the DenseNet169 model with Standard FC, and DenseNet201b represents the DenseNet201 model with Spinal FC. The table above shows that, with the EL1 model, the test accuracy is 0.8455 and [PERSON]'s kappa score is 0.6907. Therefore, the highest classification result obtained with DenseNet169 with Standard FC given in the previous section was exceeded. The confusion matrix specified in Figure 27 was a result of the classification performed using the EL1 model. The following results are concluded as per the figure: TN: 249, FP: 36, FN: 51, and TP: 227. The ROC curve representing the change in the TP rate and FP rate of the EL1 model entails an AUC score of 0.8862 for Class 0 (normal) and Class 1 (abnormal, fracture) in Figure 28. This AUC score is also higher than the AUC achieved with ResNet169 with Spinal FC in the previous section. \begin{table}
\begin{tabular}{l l l l l l} \hline \hline
**Models** & **Test Acc.** & **Pre.** & **Recall** & **F1-Score** & **Cohen’s Kappa** \\ \hline ResNeXt50b & 0.8241 & 0.825 & 0.825 & 0.825 & 0.648 \\ DenseNet169a & 0.8419 & 0.845 & **0.84** & 0.84 & 0.6834 \\ DenseNet201b & 0.8294 & 0.83 & 0.83 & 0.83 & 0.6588 \\ EL1 & **0.8455** & **0.8631** & 0.8165 & **0.8455** & **0.6907** \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Classification results of EL1. Three different single classification models and a sub-ensemble model were used in the EL2 model: ResNet34 with Spinal FC, DenseNet201, DenseNet169 with Standard FC, and sub-ensemble models. The TP and TN values in the confusion matrix scores and recall scores from the 26 different classification models used in the first stage were taken into consideration when choosing the three single models used in the EL2 model and developing the sub-ensemble model. The results of the classification carried out with the EL2 model are provided in Table 15, where ResNet34b represents the ResNet34 model with Spinal FC, DenseNet169a represents the DenseNet169 model with Standard FC, DenseNet201b represents the DenseNet201 model with Spinal FC, and SubEnsemble represents the developed sub-ensemble model. Figure 28: ROC curves and AUC results of EL1. Figure 27: Confusion matrix results of EL1. The table shows that [PERSON]'s kappa is 0.6942, and the test accuracy score is 0.8472. Therefore, the EL2 model outperforms the 26 CNN-based models and the EL1 model. Figure 29 presents the confusion matrix scores achieved as a result of the classification carried out with the EL2 model: TN: 248, FP: 36, FN: 49 and TP: 229. For comparison, the TP in the EL1 model was 227. The ROC curves and AUC scores achieved as a result of the classification performed with the EL2 model are provided in Figure 30. The AUC scores are 0.8698 for Class 0 (normal, negative) and 0.8695 for Class 1 (abnormal, positive, fracture). Although these AUC scores appear to be slightly decreased compared to the EL1 model, since the test accuracy, [PERSON]'s kappa scores, and the number of detected shoulder bone fracture images (TP) in the EL2 model are higher than the EL1 model, the EL2 model overall provides better results.