## 2 Methods
### Study Design for Multi-Level CNN Model Training and Testing  
The training protocol was conceived to have five rounds, and in each round, testing was conducted at four levels on images differing in quality or similarity (as detailed inTable 1). In each test, a three-class classification (normal, COVID-19, and pneumonia) was performed. By training one model through several rounds with augmented datasets and testing its performance on datasets with decreasing similarities, it was aimed to (1) select the optimal CNN model, (2) test the model's ability for interpolation and extrapolation, and (3) test the model's ability to learn from new data. Round 1, referred to as the \"baseline training\", incorporated 11,538 radiographic images (baseline, or \"Base\" for short in Table 1). It was aimed at validating the CNN models (i.e., Level 1) as well as evaluating whether the models can classify new samples that are either similar (Level 2), outliers (Level 3), or have higher levels of dissimilarity (Level 4). In doing so, a preliminary evaluation of the model's interpolation and extrapolation capacity was obtained, based on which further training and testing could be designed. To evaluate the effects of re-training on model performance, five rounds of training were performed in this study with incrementally extended data. The selection of added images was determined by the testing results in Round 1. In Round 2, 25% of the outlier (Level 3) images were added in the hope that the CNN model could learn the inherent features of those outliers and thus improve the classification accuracy of these outlier images. In Round 3, 50% of the outlier images were added. In comparison to Round 2, this round aimed to quantitatively evaluate the performance improvement as a function of added images. Round 2 and Round 3 training exemplified the processes of model development and optimization within one medical center based on its own data. In contrast, Rounds 4 and 5 training represented scenarios where the automated diagnostic system was used in other medical centers, and when generating sub-optimal performances, it was further improved by including new images to re-train the system. Similarly, the two-round training (Rounds 4 and 5) was designed to quantify performance improvements vs. added images. Round 4 included 25% of dataset 3 training images in addition to the Base and 50% outlier, while Round 5 included 50% of dataset 3 training images (Figure 1b, lowest row). To evaluate the network classification performance, various indices were quantified, including accuracy, sensitivity, specificity, precision, AUC (area under the curve), and ROC (receiver operating characteristic) curve. For three-class classification (i.e., 1, 2, 3), the performance matrices were extended from their binary counterparts. Based on a three-class confusion matrix (blue dashed rectangle) in Figure 1c, the accuracy is the ratio of correct predictions over all predictions (Figure 1c, left bottom). The category-wise matrices (specificity, sensitivity, and precision) were calculated using the equations listed to the right of Figure 1c. For the Receiver Operating Characteristic (ROC) curve, the One-vs-Rest approach (OvR) was used, e.g., COVID vs. (normal plus pneumonia). In this study, all category-wise matrices were presented for COVID unless stated otherwise. The CNN model training/testing was conducted using an AMD Ryzen 3960X 24-Core workstation with 3.79 GHz processors, 256 GB RAM, and 24 GB GPU (PNY NVIDIA