## 3 Results
### Model Selection  
ROC (receiver operating characteristic) curves are displayed for the training phase of Rounds 1, 2, and 4. The ROC curves at Levels 1 and 2 are near perfect and thus are not presented. At Level 3 (upper panel, Figure 5), for the Outliers testing dataset, re-training with extended data moved all curves toward the top-left corner of the graph, indicating improved discriminatory power for all models considered (Figure 5a-c). Among them, VGG-19 exhibited the lowest improvement in all training phases. The other three models showed insignificant discrepancies in Round 4 (Figure 5c); however, in Round 2, ResNet-50 showed the best performance (learnability), with a much lower threshold of false positive rate for a 95% detection rate than AlexNet and MobileNet. At Level 4 (lower panel, Figure 5), for external testing images, the ROC curves were already skewed towards the upper-left corner. Model performances constantly increased from Round 1 to Round 4, and ResNet-50 remained the optimal one among the four. The ROC curves of ResNet-50 are compared across Rounds 1-5 in Figure 6 at Level 3 and Level 4. Note that in Round 2 and Round 3, the training dataset was extended by adding 25% and 50% outlier images (Level 3). As a result, the detection rate on the Level 3 dataset increased drastically from Round 1 to Round 2, as well as from Round 2 to Round 3, but at a lower rate (Figure 6a). In Rounds 4 and 5, adding external source (Level 4) images did not improve ResNet-50's detection rate on the Level 3 dataset, but rather decreased it slightly (Figure 6a). At Level 4 (i.e., dataset from external centers), very high detection  
Figure 4: Performance comparison among different models at Level 3 among five rounds of training: (**a**) accuracy and (**b**) precision. rates were achieved by adding external but relevant training images, as evidenced by the ROC curves that were highly skewed towards the upper-left corner. As a result, the ResNet-50 model trained in Round 5 was more accurate and had wider applicability than the ResNet-50 model trained in Round 1. To evaluate the multi-round training effects on category-wise performance (i.e., normal, COVID-19, and pneumonia), the misclassification rate at Level 4 by ResNet-50 was further broken down into individual groups in Round 1 and Round 5, as shown in Figure 7a. Note that in Round 1, the Level-4 ResNet-50 accuracy was 87.4% (Table 2), which resulted from individual accuracies of 78.8% for normal, 83.0% for COVID-19, and 94.4% for pneumonia. For each category, the misclassification rates for one of the other two categories were also calculated. For instance, a 22.2% misclassification rate (100-78.8%) of normal cases included 2.1% of cases inaccurately predicted as COVID-19 (brown color) and 20.1% as pneumonia (grey color), as shown in Figure 7a. Similarly, a 17.7% misclassification rate of COVID-19 all came from their incorrect prediction as normal, while a 5.6% misclassification rate of pneumonia included 5.3% incorrectly predicted as normal and the remaining 0.3% as COVID-19 (Figure 7a). Figure 5: ROC (receiver operating characteristic) curves for COVID detection at Level 3 (Outliers) and Level 4 (External center) in (**a**) Round 1, (**b**) Round 2, and (**c**) Round 4. Figure 6: ROC curves of ResNet-50 in Rounds 1–5 training when tested on datasets of (**a**) Level 3 (outliers) and (**b**) Level 4 (external center). In Round 5, the misclassifications of normal as COVID-19 and pneumonia were greatly reduced, improving the model's specificity (right vs. left panels, Figure 7a). A similarly significant reduction was also observed in the misclassification of COVID-19 as normal, thus improving the model's sensitivity. However, the misclassification of COVID-19 as pneumonia increased (i.e., from 0% to 1.9%). The misclassification of pneumonia as normal also increased slightly (i.e., from 5.1% to 6.2%, Figure 7a), which would partially reduce the model's sensitivity.