# Ensemble Deep Learning for the Detection of COVID-19 in Unbalanced Chest X-ray Dataset
## 4 Experimental Setting, Results and Discussions
### Performance Metrics  
Multi classification of CXRs can be regarded as a set of many binary classification problems--one for each class. Thus, the evaluation metrics for multiclass classification are computed in the context of the binary classification in which the classes are simply either positive or negative. The term \"positive\" and \"negative\" will be changed depending on the true label of a particular class of the image. In our case, for COVID-19 class, COVID-19 CXRs are considered as \"positive\" whereas pneumonia and normal CXRs are regarded as \"negative\". The same concept straightforwardly applies to normal class and pneumonia class as well. When considering the \"COVID-19\" class, a true positive occurs when an actual \"COVID-19\" CXR is predicted as \"COVID-19\" class. Any other prediction of COVID-19 CXR as \"pneumonia\" or \"normal\" will be considered a false negative. This applies to every single class. For a given prediction, there are multiple classes that are considered true negatives. For instance, if we are considering \"COVID-19\" class, a true \"COVID-19\" CXR must be predicted as \"COVID-19\" to be a true positive. However, if we are considering the class of \"COVID-19\", then the original \"pneumonia\" could be labeled as \"pneumonia\", or \"normal\" as long as it is not predicted as \"COVID-19\" and it will still be a true negative for the class \"COVID-19\".) For COVID-19 class, false positive occurs when pneumonia or normal CXRs are wrongly predicted as COVID-19.  
This concept will apply to all classes. We can define TruePositive, TrueNegative, FalsePositive and TrueNegative as follows:  
* TruePositive denotes the number of correctly classified CXRs in which the true label is positive and prediction is positive for the particular class.
* TrueNegative refers to the number of CXRs in which the true label is negative and prediction is negative for the particular class.
* FalsePositive represents the number of CXRs which are negative but predicted as the positive for the particular class.
* FalseNegative (FN) denotes the number of CXRs which are positive but predicted as the negative for the particular class.  
From the above categories, we can derive the evaluation metrics to judge the performance of the machine learning or deep learning classifiers. In this study, the classification performance of each model was evaluated using five measures: sensitivity, specificity, accuracy, f-measure, and AUC. They are formulated as follows:  
\[Sensitivity,\ Recall=\frac{TruePositive}{TruePositive+FalseNegative}\times 100\% \tag{10}\]  
\[Specificity=\frac{TrueNegative}{TrueNegative+FalsePositive}\times 100\% \tag{11}\]  
\[Precision=\frac{TruePositive}{TruePositive+FalsePositive}\times 100\% \tag{12}\]  
\[Accuracy=\frac{TruePositive+TrueNegative}{TruePositive+FalsePositive+ TrueNegative+FalseNegative}\times 100\% \tag{13}\]  
\[F-measure=\left(2\times\frac{Precision\times Recall}{Precision+Recall}\right) \times 100\% \tag{14}\]  
AUC of ROC is computed from the ROC curve which plots sensitivity against 1-specificity to judge the quality of the classifier. The larger AUC-ROC indicates the better prediction quality. AUC of ROC is usually computed for binary classification. To extend the ROC curves and AUC of ROC for multi classification, the outputs are binarized per class. This means that we plotted ROC for each class and computed AUC of ROC per class. The mean AUC is used as the final AUC score of each model.