## 3 Results
### Threshold Optimization  
A small split (5%) of the data from the training datasets was used as a held-out (validation) set, in order to tune the model hyperparameters; amongst them, we used the validation set to choose the optimal threshold value for the predictions.  
Binary and multilabel models use sigmoid activation functions in the last layer, which means it outputs a continuous probability score in the range of [0, 1] for each label. ROC and PR curves are able to show the overall performance at multiple classification thresholds, offering a global interpretation of the model ability to linearly separate two given classes in a binary classification, which is reflected in the value of their metrics in the area under the curve (AUC).  
In a real-world scenario, a fixed threshold needs to be set to consider a prediction as positive or negative. A 0.5 value for the classification threshold is considered the standard, although the actual value can be modified to be more suitable for a given task.  
In the context of disease triage, and, even more, in the specific case of pneumothorax (where minimizing the number of false negatives is a priority, given the severity and potential mortality of the disease), lowering the classification threshold is generally beneficial, although some careful assessment needs to be made to avoid increasing the false positives too much (which would increase the untrustworthiness of the model and would make it unsuitable and unusable in a real-world scenario).  
Table 12 shows the different metrics for the validation set at different threshold values, as well as the number of true positives, false positives, false negatives, and true negatives.  
Figure 8: Confusion matrices for all combinations of training datasets in vertical axis (individual single-source and also multi-source as all) and all test datasets in the horizontal axis (including micro-average as Overall).  
Figure 9 shows the F1 values at different classification thresholds, showing the value where the F1-score is maximized (0.641). That threshold maximizes F1 but does not lead to a useful model due to its inability to linearly separate the classes, as shown in Figure 10, which shows the confusion matrices at different specific threshold scores.  
Our aim was to minimize the number of false negatives (missed pneumothoraces) by pushing the recall as high as possible while maintaining the precision (lowering the number of false positives to a reasonable number). We estimated 0.8 as the minimum recall for  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Threshold** & **PPV (Precision)** & **Sensitivity (Recall)** & **Specificity** & **NPV** & **F1-Score** & **TP** & **FP** & **FN** & **TN** & **Total** \\ \hline
0.0 & 0.0506 & 1.0000 & 0.0000 & 0.0000 & 0.0963 & 1393 &  
achieving our target and, thus, evaluated the different precision values with this objective. Table 13 shows the calculated precision values at fixed recall values over 0.8, showing that precision was not as high as would be advisable.  
In a real-world triaging scenario and considering the severity of the disease, we decided that, in order to maintain the recall and minimize the number of false negatives, we would use a threshold of 0.2 in our work. That threshold would mean, considering the estimated prevalence of the disease, around three false positives per true positive (3:1 ratio). That ratio, although higher than desirable, is low enough to still be clinically useful at the benefit of reducing the number of missed pneumothoraces.