## 5 Discussion
### Biased Model Performance Evaluation  
We think that the major problem of COVID-19 diagnosis model-related research is the lack of benchmarks. There is not a well-recognized benchmark testing set (e.g., something like ImageNet [213] for image classification, MS-COCO [214] for object detection and Cityscapes [215] for semantic segmentation) that can be used to compare COVID-19 diagnosis models. As a result, different papers used different datasets, making their results not fully comparable. In addition, this problem leads to the appearance of many papers that benchmark existing deep models on different datasets repeatedly. Most importantly, this creates difficulties for researchers in assessing the effectiveness of numerous newly proposed methods.  
* _Recommendation: Well-recognized institutions should establish benchmarks (using baseline models and a high-quality dataset and releasing reproducible codes) as soon as possible._  
In addition to the problem of lacking a benchmark, each individual performance evaluation report is not fully reliable. Many papers used no validation set (which means that the authors directly tune the hyperparameters on the test set), leading to the problem of data leakage. Moreover, in many studies, the testing set is too small. As in Figure 9, we show the distributions of sample numbers (COVID-19 samples vs. total samples) in the testing set of COVID-19 diagnosis models. For papers that did not explicitly declare the size of the training and test set, we calculate them with the corresponding train-test split ratio. Unfortunately, most studies used a very small testing set: the number of COVID-19 samples of more than 50% studies is smaller than 50. There are even several methods that used fewer than 10 COVID-19 samples to evaluate their methods [111; 116; 153]. We think this is one of the core reasons for the production of over-optimistic reports.  
* _Recommendation: The testing set should not be used for validation. In addition, the testing set should be sufficiently large; otherwise, it cannot give an accurate estimation of the model performance._  
Figure 8: Histograms of the reported performance of automated diagnosis models for COVID-19. We calculate the proportion of the reported model accuracy and AUC metrics that falls within each interval. The model with very high claimed performance (e.g., >95%) accounts for a high proportion.  
In spite of the scale of COVID-19 diagnosis datasets, their quality is also problematic to some extend. For data acquisition, several datasets are a combination of different datasets, which would lead to potential data leakage and the production of an over-optimistic estimation of the model performance. Some datasets are collected from the figures in the PDF file of published papers, whereas some datasets convert the original DICOM file format into PNG or JPG format. Both of them would lead to a decrease in image quality [9]. Moreover, in many datasets, different classes of samples are collected from different sources, which would lead to a high risk of model bias [11].  
* _Recommendation: When making a new dataset public, researchers should guarantee its quality and provide as much detailed information as possible._