## 2 Methods
### Selection of CNN Models  
The selection of CNN models depended on the training purpose and the model's specificities. The purpose of this study was to develop a CNN-based automated diagnostic system with high performance for COVID-19 and pneumonia patients based on chest X-ray images. Relevant criteria included the following: high classification accuracy, high sensitivity and specificity, robust performance across centers, the ability to learn continuously, and the ability to identify discriminative features among categories. Features learned are often abstract, beyond human comprehension or interpolation. However, sometimes they can capture some inherent differences that are consistent with our perceptions, which can be useful in understanding either the etiology or manifestations of a specific disease. Ideally, only defining features are learned from the training dataset; subsequent addition of highly similar images will not bring new features to the CNN model and thus can only marginally improve the CNN model. Only new images with new features introduced to the model will see extra features to refine the classification of the test dataset. Four convolutional neural network (CNN) models were selected for this study: AlexNet, ResNet-50, MobileNet, and VGG-19. AlexNet and ResNet-50 were selected because they were the 2012 and 2015 winners of the ImageNet competition, respectively [32; 33]. AlexNet was groundbreaking in its use of GPUs for training deep neural networks, while ResNet-50 introduced residual connections between different layers to improve gradient flow and enable the training of even deeper neural networks [34; 35]. MobileNet was chosen for its simpler architecture and smaller computational requirements [36; 37; 38]. It will be desirable to run a computer-aided diagnostic (CAD) system on a personal computer or even a smartphone, provided it can achieve sufficient diagnostic accuracy. VGG-19 was selected for its simplicity, which uses only 3 \(\times\) 3 convolutional layers stacked on top of each other in increasing depths. One drawback of VGG-19 is its requirement for more memory than the other three models, primarily due to its fully connected layers. To determine whether the architecture of a pre-trained network is suitable for a test dataset, the following steps were taken. First, the network was trained and tested on the same dataset, providing an initial evaluation of the model's performance on one testing dataset. In this step, various metrics are evaluated, including accuracy, AUC, specificity, sensitivity, precision, and ROC, to identify areas for improvement (Figure 1c). Second, the pre-trained network was fine-tuned based on the identified areas for improvement by adjusting the hyperparameters or replacing some of its layers to improve its performance. After fine-tuning, the pre-trained network was re-evaluated on the test dataset until its performance reached a threshold classification accuracy. The fine-tuned network model was further tested on additional datasets from the same medical center that the model had never seen before (for instance, newly acquired images). This step evaluates the pre-trained model's interpolation capacity. In the following step, the model was tested on datasets from other medical centers that belong to the same category (e.g., COVID-19 images, but the model has never seen them before). This step evaluates the pre-trained model's extrapolation capacity. Even though the X-ray images acquired at any medical center share inherent similarities among COVID-19 patients, differences among COVID-19 images can exist across medical centers, and the performance of the pre-trained model can be affected by such differences. In the next step, the model was re-trained by adding new images to the training dataset and re-tested. In this step, two metrics are evaluated: (1) the improvement in performance, and (2) the minimal number of images needed to reach a satisfactory classification accuracy.