# Ensemble Deep Learning for the Detection of COVID-19 in Unbalanced Chest X-ray Dataset
## 3 Methodology
### Classification of COVID-19, Normal and Pneumonia Using Deep Convolutional Neural Networks
#### 3.3.1 Approach 1: Deep Learning with Weighted Loss on an Unbalanced Dataset  
Pretrained CNNs were trained on the normal cross-entropy loss to backpropagate the errors, minimizing the loss. However, using a normal cross-entropy loss with our highly unbalanced dataset will incentivize the models to prioritize the majority class, because it contributes more to the loss. Simply put, the majority class will dominate the loss. Therefore, training the deep learning models with a uniformly balanced dataset is preferred so that the positives and negatives of each training class would have an equal contribution to the loss. The first approach to balance the class distribution without altering or transforming the original dataset is to modify the loss of the CNN models. Here, we substituted the cross-entropy loss with the weighted loss to handle the unbalanced dataset. Figure 7 presents a classification of normal, pneumonia, and COVID-19 using CNN models with the weighted cross-entropy loss. The weighted loss will be replaced as the loss of every 11 pretrained CNNs. In this approach, we applied weighting the loss on the one-class learning strategy for each class in which the samples of the particular class are considered as positive and all other samples as negatives. Suppose we would utilize the cross-entropy loss for prediction of each class: the loss of deep CNNs on the \(i\)th training sample can be formulated as Equation (2). \[L_{cross\_entropy(x_{i})}=-(y_{i}\log(f(x_{i}))+(1-y_{i})\log(1-f(x_{i}))) \tag{2}\]  
where \(x_{i}\) and \(y_{i}\) are the input features extracted from the pretrained CNNs and the label, and \(f(x_{i})\) is the prediction output of CNN, i.e., the predicted probability value of a certain  
Figure 6: The unbalanced class distribution of the dataset. Figure 7: Transfer learning of CNNs with the weighted loss on the unbalanced dataset. class. For every training image, either \(y_{i}=0\) or \(else(1-y_{i})=0\). Therefore, only one among these terms has the contribution to the loss and the other term is multiplied by zero and becomes zero [43]. The overall cross-entropy loss for the whole training dataset \(T\) of size \(N\) can be formatted as in Equation (3). \[L_{cross\_entropy}(T)=-\frac{1}{N}\Bigg{(}\sum_{positive\ samples}\log(f(x_{i}))+\sum_{ negative\ samples}\log(f(x_{i})(1-f(x_{i}))) \tag{3}\]  
From the above equation, it can be seen that, if the training set is largely unbalanced with a small number of positive class, the negative class will dominate the loss. The contribution of each class (i.e., positive or negative) can be computed as a summation of the contribution over all the training cases for each class, in Equations (4) and (5) [43]. As shown in Figure 7(a), the contribution of each class varies significantly and is unbalanced, especially in COVID-19 and pneumonia classes. \[freq_{positive}=\frac{number\ of\ positive\ samples}{N(all\ samples\ in\ each\ class)} \tag{4}\]  
\[freq_{negative}=\frac{number\ of\ negative\ samples}{N(all\ samples\ in\ each\ class)} \tag{5}\]  
To equalize the contribution of loss for each class, we balanced the contribution of positive and negative labels using weighted loss, which was computed by multiplying each example from each class by a class-specific weight factor, \(w_{positive}\) and \(w_{negative}\). To ensure that the overall loss contribution of each class was equal, we performed Equation (6). The weight of each class could be computed using Equations (7) and (8). The weighted cross-entropy for each training sample is defined as Equation (9). Figure 7(b) plots the equally contributed frequency of each class after loss-weighting. It shows that when using the weighted loss, the positive and negative labels within each class would have the same aggregate contribution to the loss function [43].