## 4 Discussion
We also performed an analysis of the contribution of the individual datasets by training single-source models and comparing their performance with the multi-source dataset, showing that the multi-source model outperforms the single-source counterparts' global metrics (AUROC, PR-AUC, and F1-score) and almost in all specific metrics. For instance, specificity is higher in models trained using the VinDR-CXR dataset, but that is probably due to the low prevalence of pneumothorax in that dataset, producing very few false positives but a very low recall. The MIMIC-CXR-trained model has a higher PPV probably due to their data distribution (a moderate pneumothorax prevalence, and high number of anteroposterior views). The model trained using the CheXpert dataset has the largest recall (0.908) probably due to the largest prevalence of pneumothorax (8.7%), which, in addition to the class-balance oversampling, effectively lowers its internal decision boundary at the cost of extra false positives. But, even if specific single-source models outperform the multi-source model in a specific metric, it always comes at the cost of worse overall performance. Training using multi-source heterogeneous data improves generalizability and model robustness and mitigates the 'domain shift' where models trained with one dataset significantly decrease their performance when tested against external datasets that resemble real-world performance [71]. This effect is something that has been extensively discussed before for the pneumothorax scenario and is something that heavily affects some of the used datasets. For example, the NIH dataset suffered from incorrect labelling coming from NLP techniques used in radiology report extraction, and also from confounder biases coming from the correlation between treated pneumothoraces (labelled with 'pneumothorax') and chest tubes (that did not contain any label), creating spurious correlations [31; 72]. Using expert labelling and chest tube information can help mitigate these issues, which also happens (to a lesser extent) in other datasets [73; 74; 75; 76]. With the objective of mitigating this domain shift, we incorporated auxiliary tasks to the model, by also predicting no disease and support devices, in order to prevent shortcut learning and enforce the model to learn real pneumothorax features. Figure 11: Error analysis of false positives from the BRAX dataset, showing several images that would not have to be used as input, such as pediatric images (three on the first row) or images from lateral views (three in the second row), suggesting potential limitations in the evaluation of the performance in this dataset. The CheXNet [77] was the first paper which attracted wide interest from the global scientific community in chest X-ray classification (although many of the papers that were published in that period used only the NIH dataset [78; 79], which, as we discussed, had problems that needed to be mitigated). The release of many public datasets in 2019 (including CheXpert, MIMIC-CXR, and PadChest), which featured more and better labels, and more radiologic projections, and included radiology reports, had enabled richer scientific research that identified and resolved obstacles to developing models suited for real-world scenarios. Recent studies using multicenter datasets have achieved pneumothorax AUROCs in the 0.94-0.98 range [65; 80; 81; 82; 83]. Our off-the-shelf EfficientNet-B5 model matches the upper end of that spectrum (AUROC 0.961 and PR-AUC 0.825) while being trained on a broad, carefully curated pool of six public datasets and validated on five additional public datasets (with >53,000 radiographs). These results demonstrate that data harmonization, label cleaning, and domain-aware adjustments are more critical for performance and generalizability than specific and elaborate network architectures. Despite these findings, comparing our metrics to those of other studies requires caution for several reasons.