# Investigating the Performance of FixMatch for COVID-19 Detection in Chest X-rays
## 5 Exploratory Imbalance Analysis
### Results and Discussion
#### 5.2.3 Statistical Analysis of Confusion Matrices  
In order to analyze the consistency in the types of errors made by the models across the various test cases, the Adjusted Rand Index (ARI) and Adjusted Mutual Information (AMU) scores across the confusion matrices of the models were computed to determine how closely their errors matched. The formulas for these metrics are shown in Equations (10) and (11), respectively. \[ARI=\frac{\Sigma_{ij}\left( \begin{array}{c}n_{ij}\\ 2\end{array} \right)-\frac{\Sigma_{i}\left( \begin{array}{c}a_{i}\\ 2\end{array} \right)\Sigma_{j}\left( \begin{array}{c}b_{j}\\ 2\end{array} \right)}{\left( \begin{array}{c}n\\ 2\end{array} \right)}} \tag{10}\]  
where \(n_{ij}\) is the value of an element in row \(i\) and column \(j\) in the contingency table, \(a_{i}\) is the sum of the elements in row \(i\), \(b_{j}\) is the sum of the elements in column \(j\) and \(n\) is the total count in the table. \[AMU(U,V)=\frac{MI(U,V)-E(MI(U,V))}{avg(H(U),\ H(V))-E(MI(U,V))} \tag{11}\]  
where \(MI(U,V)\) is the mutual information between \(U\) and \(V\), \(E(MI(U,V))\) is the expected mutual information between \(U\) and \(V\), and \(H(U)\) is the entropy associated with \(U\). The ARI and AMU were computed for various cases in order to gain a better understanding of the obtained results. In order to fully understand the variability in the predictions of the model when trained on the imbalanced data, the ARI and AMU between the confusion matrices generated when trained on uniform distribution, and each of the imbalance distributions was computed. Tables 13-15 show the obtained ARI and AMU results alongside the differences in the macro-average F1-scores for the corresponding tests in order to enable a complete analysis. As can be seen in this case, the equal majority-minority distribution is the most different from the uniform distribution, and this is evidenced by the low ARI score for higher amounts of labeled data. However, as the proportion of labeled data reduces, the correlation increases despite the difference in F1-scores increasing. This is indicative that, as the labeled data decreases, the model fails to adequately generalize for both distributions and, therefore, makes the same kinds of mistakes in the confusion matrices. An interesting observation in the class containing many majority classes is that, while the ARI scores are generally constant, meaning that the correlation between the confusion matrices remains approximately the same, we can also see that the difference in F1-score remains more or less constant despite the decreasing level of labeled data beyond around 40% of the labeled samples. This suggests that, for the case of this distribution, the lowering of the proportion of labeled data has a similar effect on the model regardless of the distribution. In the case of the few majority class distribution, it can be seen that the ARI scores indicate a significant level of correlation, with the confusion matrices generated through the uniform distribution. This could possibly be due to the few majority class distribution being the closest to the uniform distribution and, therefore, providing similar results. Having looked at the obtained results and the correlation between the confusion matrices, this exploratory study might suggest that the FixMatch algorithm is, indeed, somewhat resistant to imbalanced data, as indicated by the ARI results. In recent times, works such as CReST [74], DARP [75], BiS [76], DASO [77] and ABC [78] have been \begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{4}{c}{**Uniform Distribution vs. Equal Majority–Minority Distribution**} \\ \hline
**\% Labeled Data** & **ARI** & **AMU** & **Difference in F1-Scores** \\ \hline
80 & 0.14948 & 0.27771 & 0.12 \\
60 & 0.28827 & 0.39391 & 0 \\
40 & 0.44089 & 0.56003 & 0.06 \\
20 & 0.41037 & 0.49083 & 0.04 \\
10 & 0.42611 & 0.45006 & 0.10 \\
5 & 0.36895 & 0.50927 & 0.18 \\ \hline \hline \end{tabular}
\end{table}