## 4 Discussion  
In this study, we wanted to investigate the viability of training a model suitable for pneumothorax detection in an emergency scenario, by using public chest X-ray datasets for training (creating a multi-source dataset) and evaluation (in order to assess the generalizability and robustness in unseen data). We selected the EfficientNet-B5 model and trained it using a multi-source training strategy with 322,569 pooled chest X-rays from six public datasets, showing an overall AUROC of 0.961 and PR-AUC of 0.825 in the external test set (which include five unseen public datasets not used for training or validation), where the pneumothorax prevalence was 7.6% (4081 out of 53,426 samples). Those results show robust discrimination on heterogeneous unseen data, which was the main objective of our work, in order to assess the potential usage in a clinical scenario. Nevertheless, threshold tuning remains an important task that can substantially modify the clinical application of trained models. In our case, we chose to lower the threshold to 0.2, to sacrifice specificity for safety, so we minimize the number of missed pneumothoraces at the cost of more false alarms. Our results in the validation set showed that, at a recall of >80%, we obtained a ratio of 3:1 for false positives versus true positives. If we selected a threshold to increase the recall to >90%, we would have obtained a ratio larger than 6:1, which would have decreased the PPV too much and given too many false alarms to make the model clinically useful. The choice of this threshold yields a recall of 0.862 in the external test set, which still gives 13.7% false negatives, probably still too high in a real-world scenario given the potential severity of the disease. In a real scenario, missed pneumothoraces are generally small pneumothoraces or in anteroposterior radiographic projections, which are more difficult to detect; but, in triage systems, we should minimize false negatives as low as technically feasible (approaching 0%) without a high number of false positives. Although the PPV in the external test set is not too high (0.569), given the low prevalence of the disease, it gives a ratio of false positives versus true positives of ~0.8:1, which corresponds to nearly a false alarm for every real pneumothorax, something that is acceptable in a clinical scenario given the triage setting and the severity of the condition. The results are similar across individual test datasets except for the BRAX dataset, which show worse results compared to the rest of the datasets. An error analysis (shown in \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Target Metric** & **Target Value** & **Achieved Metric** & **Achieved Value** & **Threshold** \\ \hline Recall & 0.80 & Precision & 0.264 & 0.232 \\ \hline Recall & 0.85 & Precision & 0.213 & 0.160 \\ \hline Recall & 0.90 & Precision & 0.167 & 0.101 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Calculated precision values at different fixed recall values. We estimated a minimum of 80% of recall for minimizing the number of missed pneumothoraces; thus, we calculated precision values at different recall values over 0.8. Figure 11) suggests images from pediatric patients and lateral images in the false positives (suggesting potential labelling issues), and images without pneumothorax in the false negatives, which suggest potential labelling issues or treated pneumothoraces; all those issues should be investigated further in order to improve the evaluation of models trained or evaluated using this dataset. The Indiana dataset also showed decreased metrics compared to other test datasets, but an error analysis does not show clear problems in false negatives (missed pneumothoraces).