## 5 Discussion  
The user study reported in the previous sections confirmed the simulations presented in Section 1: teams of domain experts could mislabel cases, even when they were large enough to reach statistically significant consensus, and this was achieved by an overwhelming majority of raters, as it happened in approximately one case out of 20 in our study. This suggested that any metric, used for assessing the reliability of a given multi-rater ground truth, that did not take into account the accuracy of the involved raters was likely to provide an over-estimated and over-optimistic assessment. On the other hand, as regards confidence, we could make two observations. First, as can be seen from Figure 8b, the raters' confidence had only a very weak correlation with the raters' error: indeed, as can be seen from Figure 8a, the two highest confidence levels accounted for over 50% of all the annotation errors. This suggested that confidence and accuracy indeed represented two orthogonal concepts. Second, we could observe that the values of \(\alpha\), \(k\), and \(\sigma\) were relatively close (\(\alpha=k=0.65\), \(\sigma=0.62\)): the small difference between the \(\alpha\) and kappa measures could be explained by the fact that the class proportions in the dataset were almost perfectly evenly balanced, as both metrics considered class balance in order to model chance effects; in contrast, the small difference between the degree of concordance \(\sigma\) and the above-mentioned measures could be seen as a confirmation that leveraging the self-reported confidence in the definition of \(\varrho\) (through \(\sigma\)) was not only meaningful, as the confidence levels were (intuitively) inversely correlated with random guesses by the raters, but it also provided a practical and flexible model of chance effects that did not necessarily depend on the  
Figure 8: Results in graphical format: (**a**) The distribution of the labels in terms of right/wrong annotation, with respect to the confidence levels. For each confidence level, the percentages in the left column indicate the confidence level proportion with respect to all the cases; the top and bottom percentages in the right column indicate the proportions of right (resp. wrong) labels in the corresponding confidence level. (**b**) Relationship between the raters’ confidence and the probability of error; the regression line suggests a mild inverse proportion. whole dataset distribution (as is the case for other common reliability measures), but yielded equivalent results. Interestingly, making a naive estimate on the value of \(\sigma\), based on the average confidence, we obtained (by multiplication with \(P_{o}=0.82\)) an expected \(\sigma\) equal to 0.61, which was almost equivalent to the observed one. This suggested that the proposed \(\sigma\) could also be interpreted (provided that the distribution of confidence had small variance around its average) as a chance-corrected modification of the agreement \(P_{o}\), in which the chance-correction was given by the average confidence among the raters. On the other hand, we detected a large difference between the weighted reliability score \(\varrho\) and the reliability measures based only on an estimate of the chance-adjusted agreement (i.e., \(\alpha\) and \(\kappa\)). This difference, as previously mentioned, could be explained by the fact that our score also took the competence of the raters into account and thus provided a more robust and comprehensive account of ground truth reliability. As we argued previously, this behavior is desirable as it allows assigning more weight to agreements that are likely to be true compared to agreements on possibly wrong labels.