# Ensemble Deep Learning for the Detection of COVID-19 in Unbalanced Chest X-ray Dataset
## 4 Experimental Setting, Results and Discussions
### Experimental Results and Discussions
The classification results of Approach 2, of CNNs with image augmentation, are listed in Table 6. Image augmentation significantly improved the prediction performance of every CNN. DenseNet201, InceptionResNetV2, MobileNetV2 and NasNetMobile achieved 98.57%, 98.5%, 98.43% and 98.2% accuracy, respectively. Table 7 tabulates the classification results of each CNN with undersampling: Approach 3. This improved most CNNs' performance, except InceptionResNetV2, compared to Approach 0. The accuracies of MobileNetV2 and InceptionResNetV2 were slightly decreased. Compared to Approach 1 and Approach 2, Approach 3 slightly decreased the classification accuracy. Many CXRs from normal and COVID-19 classes were discarded for undersampling; therefore, some CXRs with valuable information may have been lost, thereby causing noticeable information loss and underfitting the model. The classification results of Approach 4 are given in Table 8. Oversampling greatly improved the prediction accuracy of each CNN compared to Approach 0. Among all CNNs, InceptionResNetV2 and XceptionNet achieved the highest accuracies, of 98.67% and 98.63%, respectively, and outperformed Approach 1, Approach 2, and Approach 3. The performance of MobileNetV2, DenseNet201, and ResNet50 were on par with Approach 1 and Approach 2 and generated more sensitive results than Approach 3. The classification results of Approach 5, hybrid sampling, are listed in Table 9, and show significant improvements over Approach_0, attaining comparable results with Approach 2. However, it was less sensitive than Approach 1, Approach 2 and Approach 4 and produced \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Unbalanced Data**} & \multirow{2}{*}{**Methods**} & \multicolumn{3}{c}{**Data**} & \multicolumn{3}{c}{**Hyperparameters of CNNs**} \\ \cline{3-8}  & & **COVID-19** & **Normal** & **Pneumonia** & \(\alpha\) & \(\beta\_1\) & \(\beta\_2\) \\ \hline Approach 0 & Normal loss & 2893 & 8154 & 1076 & 0.001 & 0.9 & 0.999 \\ Approach 1 & Weighted loss & 2893 & 8154 & 1076 & 0.001 & 0.9 & 0.999 \\ Approach 2 & Image augmentation & 5393 & 4363 & 8154 & 0.001 & 0.9 & 0.999 \\ Approach 3 & Undersampling & 1013 & 1223 & 1076 & 0.001 & 0.9 & 0.999 \\ Approach 4 & Oversampling & 8100 & 8154 & 8070 & 0.001 & 0.9 & 0.999 \\ Approach 5 & Hybrid sampling & 2893 & 2854 & 2690 & 0.001 & 0.9 & 0.999 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The data associated with each approach for handling unbalanced class, and hyperparameters for fine-tuning CNNs. lower accuracy. From these empirical results, we found that CNNs with the five proposed approaches improved the overall performance of COVID-19 detection compared to Approach 0. The approaches which involved increasing the number of CXRs performed better than other approaches. Image augmentation and oversampling approaches outperformed undersampling and hybrid sampling approaches that involved the elimination of CXRs. Approach 1 (using weighted loss), which did not transform or alter the number of samples in the dataset, yielded an accuracy on par with Approach 2 and Approach 4 (adding more images), and outperformed Approach 3 and Approach 5 (elimination of images). In the literature, ensemble learning frequently outperforms single models. Inspired by those results, here, we combined the top five models from each approach as the ensemble classifier using majority soft voting and hard voting. The performance of ensemble classifiers is evaluated in Table 10. Ensemble learning improved the overall prediction and those with soft voting attained better performance compared to hard voting. The ensemble classifier of the top five models in Approach 2 (augmentation) achieved the highest accuracy, 99.23%, followed by those of Approach 4 with 99.17% and those of Approach 1 with an accuracy of 99.03%. To graphically evaluate the ensemble classifiers, we plotted receiver operating characteristic (ROC) curves in Figure 16. ROC plots the true positive rate against the true negative rate to evaluate each class.