# Self-Training with Entropy-Based Mixup for Low-Resource Chest X-ray Classification
## 3 Proposed Learning Model
### Training the Teacher Model Using Entropy-Based Mixup  
_Step 1_ is to train the teacher model with a small number of labeled data (\(X_{L}\)) and the augmented data (\(X_{A}\)). In the experiment with the labeled data only (Section 4.2 Effects of Entropy-Based Mixup), the average AUROC of the teacher model was 0.6409, while the AUROC of the hernia class was the lowest at 0.5498. This is mainly due to the class imbalance, with only 27 image samples (0.15% of total training data) of the hernia class. To solve the problem, the number of samples is increased by data augmentation with the Mixup algorithm.  
New samples are generated by performing weighted linear interpolation of images and labels of two randomly extracted data as follows.  
\[\hat{x}=\lambda x_{i}+(1-\lambda)x_{j}, \tag{1}\]  
\[\hat{y}=\lambda y_{i}+(1-\lambda)y_{j}, \tag{2}\]  
where \(\lambda\in[0,1]\).  
In above equations, \((x_{i},y_{i}),\ (x_{j},y_{j})\) denote the two images and labels selected for Mixup, and \(\lambda\) is the ratio of mixing the two samples, sampled from the beta distribution. \((\hat{x},\hat{y})\) is the image and label created by synthesizing with Mixup.  
For effective sample synthesis, we propose the entropy-based selection method, which can alleviate the overfitting problem and increase the number of samples. Given a classification model \(M\), the entropy represents the uncertainty of the model's predictions. We calculate the entropy on the basis of the probability distribution generated by the teacher classifier as follows:  
\[Entropy(y)=-\sum_{i}y_{k}\times\log(y_{k}), \tag{3}\]  
Figure 1: Learning model that combines entropy-based Mixup and self-training.  
where \(y_{k}\) represents the prediction probability of class \(k\). If the entropy is very low, no additional training samples are needed because they are close to the ground-truth class, and, if the entropy is high, additional training samples should be generated because they are far from the ground-truth class. Therefore, we select the sample with the highest entropy and the sample with the lowest entropy to perform Mixup. Figure 2 shows the entropy-based sample selection process. From labeled data, we compute the entropy of each data by the teacher model, sort them by the entropy, and then continuously select data samples of highest and lowest entropy for Mixup. Figure 3 shows the example result of Mixup. Through this process of data augmentation, the model can alleviate the problem of overfitting that can occur due to the imbalanced distribution of the training data.  
There are two ways of oversampling: (a) equalizing the total number of samples for each class to the number of samples of largest class _(Mixup to same size)_; (b) increasing the number of samples for every class by a certain percentage _(Mixup to same ratio)_. Figure 4 shows the two different strategies of oversampling. Through experiments, we observed that the Mixup to same ratio produced a better performance; hence, we adopted it for other experiments.  
Figure 3: Sample synthesis for Mixup of high-entropy sample with low-entropy sample.  
Figure 2: Sample selection for entropy-based Mixup. The best performance is achieved by selecting sample data with highest entropy and sample data with lowest entropy. The used samples are removed, and the remaining data are mixed in the same way.