# Normalizing Flows for Out-of-Distribution Detection: Application to Coronary Artery Segmentation
## 3 Results and Discussion
### Evaluation on Expert Annotations  
The results in the previous section indicate that the herein proposed model is superior to the Glow-like baseline. Hence, we first ran the proposed model (trained on the augmented trainset) on the secondary dataset described in Section 2.1. We employed two relative thresholds of 60% and 90% of the mean log-probability value observed on the primary dataset's test split, when no perturbation was applied. All lesions which had _at least one_ cross-Section log-probability value under the 60% threshold were selected as candidates with possibly wrong mask annotations, yielding a total of 31 of these lesions. To construct the bin of candidates with possibly correct mask annotations, we randomly sampled 31 lesions from those for which all cross-Section log-probability values were above the 90% threshold. Thus, we did not consider lesions which had any intermediate values of log-probability (i.e., between the 60% and 90% thresholds) without also having at least one low log-probability cross-section, to avoid the effect of model uncertainty for data which it considers to be near the separation manifold between correctly annotated lesions and faulty ones. We then further excluded lesions which had a reference diameter (computed as average of healthy proximal and distal diameters) lower than 1.5 mm (the typical threshold employed in CCTA based studies assessing CAD anatomically and functionally). As a result, a test set containing 56 lesions from 35 patients was employed for the evaluation. Figure 12: AuRoC performance for the baseline and proposed architecture when tested against mask morphing in various directions. Training done on augmented dataset using the hinge loss in Equation (3). The selected test set was manually and independently annotated by three expert readers at lesion-level: each lesion was marked as being either \"correctly\" or \"incorrectly annotated\", based on the following instructions: a lesion should be marked as \"incorrectly annotated\" if _at least one_ cross-sectional contour would require editing; otherwise, if _no_ cross-sectional contour requires editing, then the lesion should be marked as \"correctly annotated\". The annotator instructions were devised so as to match the procedure employed to construct this separate test set, with the goal of being able to directly compare the labels from the NF model to the ones provided by the annotators. The Glow-like baseline NF model was also applied on the 56 lesions and the same relative thresholds and criteria were employed to classify each lesion. Evaluating the two NF models against the human annotations was framed as a binary classification problem. Table 5 summarizes relevant metrics (accuracy, sensitivity, specificity, PPV and NPV) for the proposed and baseline models. Annotation consensus was obtained through a majority vote between the three annotators. The mean inter-user metric values were obtained by averaging all 6 possible metric values pertaining to pairs of annotators, e.g., Annotator_1 (as GT) versus Annotator_3 (as Prediction), Annotator_3 (as GT) versus Annotator_1 (as Prediction), etc. When compared against annotation consensus, the proposed model has higher performance than the baseline on all considered metrics. Of special interest is the sensitivity metric, which measures the percentage of NF-flagged lesions as being incorrect from the set of lesions considered incorrect by the human annotators' consensus. The higher this metric value, the more capable is a NF model in detecting faulty segmentation masks. We observe that the proposed model has sensitivity of 76.0%, close to the inter-user value of 79.0%, while the baseline model only achieves 48%. Overall, according to the majority vote of the expert readers, 25 lesions were annotated as requiring editing, out of which 17 had unanimous annotations and 8 had non-unanimous annotations.