## 3 Automated Diagnosis of COVID-19
### Image-Level Diagnosis: Classification-Based Models
#### 3.1.5 Evaluation  
Researchers evaluated their proposed models with several metrics in experiments. The most used metrics are accuracy and the Area Under Curve (AUC). Accuracy is the ratio of correctly classified samples. AUC is the area under the ROC curve, which is the graph of the function between the true positive rate and false positive rate. The average accuracy and AUC of diagnosing models based on X-ray scanning are 94.76% and 96.94, and the average accuracy and AUC of CT scanning-based models are 90.13% and 94.76. Theoretically, 3D CT scanning contains more information than 2D X-ray scanning, and CT scanning can also avoid the occlusion of ribs compared with X-ray scanning. However, X-ray scanning-based models achieve better performance. We consider the reason for this to be that the large size of X-ray training sets helps these models, whereas CT scanning is relatively more difficult to collect. The average training set size of X-ray-based models is 4185, and the average size of CT-based models is only 1417.  
Although the performance of existing models is relatively high (average accuracy of 93.59% and average AUC of 95.75), the size of the test set is worth noting; in some models, the test set only has a few COVID-19 samples. The average COVID-19/total ratio of test sets is 0.274:1, which is highly imbalanced (the ratio of training sets is 0.3:1, which is also imbalanced). In Table 4, we also color the table cells according to the number of samples in the training and testing dataset (green and red are, respectively, corresponding to higher and lower than the average, and the saturation is correlated with the difference from the average). Some researchers reproduced the experiment with different datasets, but achieved a significantly lower performance compared to the originally reported performance [192]. The reason for this might be due to the model overfitting and the lack of appropriate  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline  & & [44,48,52,74, & & [55,70,75, & & [41,57,66,69, 76, 76,79,84, 76, 77,98,153, 154,186] & [87,92,116,118,122, 124,133,155, 174,176] & [85,79,217, 139,141, 139,144, 139,144, 135, 174,176] & [1,190] & [151,162,191] & [94,110,134, 135,142,149, 177,178] \\ \hline COVID-19 & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) \\ Normal & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & \(\surd\) & - & - \\ VP & \(\surd\) & - & \(\surd\) & - & - & \(\surd\) & - \\ BP & - & \(\surd\) & \(\surd\) & - & - & - & - \\ CAP & - & - & - & \(\surd\) & - & - & \(\surd\) & - \\ \hline NCP & - & - & - & - & \(\surd\) & - & - & \(\surd\) \\ Total & 5 & 5 & 16 & 4 & 12 & 2 & 3 & 23 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Classes involved in multi-class classification models. VP: viral pneumonia; BP: bacterial pneumonia; CAP: community-acquired pneumonia; NCP: non-COVID-19 pneumonia.  
control of patients and ground truth labels. Moreover, models in Table 4 are evaluated in different datasets, and most of them are private datasets of combined datasets. We think a proper benchmark test set is vital for further research of this area. Experiments on the same benchmark test set can also benefit the hospitals' selection of diagnosing models. In addition, how to combine the accuracy, AUC and other evaluation criteria, such as the precision, recall and time complexity, in order to choose the best model for practical applications is still an open question [193].