# Investigating the Performance of FixMatch for COVID-19 Detection in Chest X-rays
## 5 Exploratory Imbalance Analysis
### Results and Discussion  
#### 5.2.1 Overall Results  
The top-1 accuracies on the test data were computed for each of the tests, where the proportion of labeled samples was varied from 80% to 5% across the four levels of data imbalance. The results can be seen in Figure 4.  
A number of interesting trends can be observed from this set of results. Firstly, as discussed previously, given uniform balance in the data, an accuracy of 78% can be obtained with just 80% labeled data. This is closely comparable to the 81% reported by [PERSON] et al. (2018), using their ensemble approach on the same dataset. A second interesting observation is that, even with 5% labeled data, the accuracy for the uniform imbalance is as high as 72%, which is only a 6% deterioration as compared with 80% labeled data and a 9% deterioration compared with fully supervised techniques. The performance with a uniform data imbalance is generally consistent with decreasing amounts of labeled data, reporting only a 1% drop until as low as 10% of the data used being labeled. This could provide great utility in the case of future outbreaks of contagious diseases, where, at the early stages, there is generally a shortage of labeled data, and medical practitioners who label the data are otherwise occupied in containing the outbreak. In order to further analyze these results, the overall macro-average FI-scores across the tests are computed as seen in Figure 5.  
Figure 4: Overall top-1 accuracy of FixMatch for varied labeled samples and 4 levels of imbalance.  
A greater amount of variation can be seen in the case of the F1-scores, where a greater deterioration can be seen even in the case of the uniform class. As can be seen from both result metrics, an expected drop in performance is seen when the algorithm is trained with imbalanced data. This is rather concerning, as, in the case of a newly discovered disease, samples from the disease would be much fewer than those of other similar diseases, and this would bias the model toward the majority when, indeed, the disease of interest might be the minority. Based on the results, it can be noted that the distribution containing equal numbers of majority and minority classes seemed to display the poorest performance with lower proportions of labeled data. This could particularly be due to there being well-defined majority classes which the model would tend to be biased toward, along with an equal number of minority classes which the model fails to generalize to. Similarly, the distribution with lesser majority has a long right tail leading to it that has a large number of minority classes, as is evidenced from the results. The distribution with many majority classes, while imbalanced, has a less strongly defined tail, meaning the minority classes are represented less poorly than those in the previous distributions. This is reflected in the results, as well, when looking at the accuracies and F1-scores, which reveal a general trend of the distribution, with many majority classes having performance closest to the results with the uniform balance. In general, a noted trend is seen where larger amounts of labeled data are needed by the model to learn accurately in cases where an imbalance in data is present. Indeed, the case of 80% labeled data shows the least variability in results even with a high imbalance in data, as the accuracy varies from 78% to 73%, and the F1-score varies from 0.77 to 0.65. This is in contrast to the extreme case of only 5% of the data being labeled, where the accuracy varies from 72% to 66%, and the F1-score, in particular, has a large range, as it varies from 0.68 to 0.50.