## 1 Introduction  
The SARS-CoV-2 (COVID-19) pandemic began in the spring of 2020. In the battle between humans and the novel coronavirus, medical professionals and scientists contributed a large amount. At the same time, a popular reaction from other research communities, including the AI community, is \"how can we help?\". In practice, the diagnosis of COVID-19 can be based on two methods: the Reverse Transcription Polymerase Chain Reaction (RT-PCR) test and radiography imaging (X-ray or CT scan). Current AI technologies can hardly help with RT-PCR tests, but they can assist the radiography imaging-based diagnosis of COVID-19 from a computer vision perspective. In the virus-stricken area, radiologists have a heavy burden on analyzing massive scanning images, so automated diagnosis models have great potential value in supporting medical decisions. The AI community responded very quickly to this point. The first automated diagnosis model (from [PERSON] et al. [1]) was posted on MedRXiv on 17 February 2020, when the total number of confirmed cases was only around 64,000 (current global confirmed cases: more than 400,000,000 (data retrieved from [[https://covid19.who.int](https://covid19.who.int)]([https://covid19.who.int](https://covid19.who.int)), accessed on 11 February 2022)), nearly a month before the World Health Organization (WHO) announced COVID-19 as a global pandemic (12 March 2020). As shown in Figure 1, following the work of [PERSON] et al. [1], a large number of researchers dived into this field, making the automated diagnosis of COVID-19 become a noticeable research hotspot in the AI community. Meanwhile, an unprecedented amount of papers have been presented. As of this writing, there are more than 1320 manuscripts on arXiv (Google Scholar search: COVID-19 [\"CT\" OR \"X-ray\"] [\"machine learning\" OR \"deep learning\" OR \"artificial intelligence\"] site:arxiv.org) and 535 manuscripts on medRxiv (Google Scholar search: COVID-19 [\"CT\" OR \"X-ray\"] [\"machine learning\" OR \"deep learning\" OR \"artificial intelligence\"] site:medrxiv.org). As shown in Table 1, at least 13 special issues on this topic have been held by various journals. Moreover, there are more than 20 review or survey papers [2] that have been presented to the best of our knowledge. The research on the automated diagnosis of COVID-19 is highly interdisciplinary. A good study should be qualified with both clinical standards and AI standards. From the clinical perspective, the model should avoid making mistakes such as improper data collection, data prepossessing and data augmentations. From the AI perspective, the models should have rigorous experiment design, enough robustness and a good generalization ability. However, unfortunately, while there are thousands of studies that appeared in these two years, very few of them are qualified from both sides. On 27 March 2020, a group from Europe presented an early review of the diagnosis models [3;4]. Among hundreds of covered studies, they found that \"all models were rated at high or unclear risk of bias\" according to the result obtained from PROBAST [5] (a prediction model risk of bias assessment tool). On 13 November 2020, [PERSON] [6] described AI for COVID-19 imaging as \"A Hammer in Search of a Nail\". He called for moving beyond studies that repeatedly show that AI can detect COVID-19. Reliable diagnosis models that can meet real-world clinical requirements are more urgently needed. However, this need was not met by subsequent studies. On 15 March 2021, a group from the UK [7] published an article pointing out that \"our review finds that none of the models identified are of potential clinical use due to methodological flaws and/or underlying biases\". Later, a review paper [8] noted that \"The vast majority of manuscripts were found to be deficient regarding potential use in clinical practice\". The problems of the research on the automated diagnosis of COVID-19 are not limited to an insufficient application value.