## 3 Method  
As stated in Section 1, we consider the raters' reliability as a composite construct comprising the inter-rater agreement, the raters' confidence (i.e., the self-perceived ability to make a convincing decision for a specific case, or to avoid guessing), and their competence (i.e., the ability to make the right decision, in general).  
In what follows, we will consider a rating an association between a case and a label (equivalently: the relationship between an instance and its target class): that is, a single labeling. Intuitively (see Figure 6, an agreement occurs when two raters assign the same label to the same case (or object, instance, phenomenon, etc.). We do not know whether a rating is right (i.e., including the true, but unknown label), but we assume that the more agreements (which, as mentioned above, are detected by the simple identity of responses given by a pair of raters) a rating gets, the stronger the evidence of it being right, and hence its reliability. In the \"race\" to become \"truth\" (or better yet, part of the ground truth), if we adopt majority voting (which is a simple, but effective method), the rating with the higher number of agreements wins.  
Figure 5: The general procedure proposed to get an estimate of the actual reliability of a classification model.  
However, we should not consider all agreements equal. In the literature, it is common to consider that some agreements are due to chance. Usually, the effect of chance is discounted on the basis of a model that is derived from the whole distribution of ratings in the dataset [7]. We consider this approach a limitation of these models, in that metrics that are adjusted in this way cannot be applied to single cases, but only to whole datasets, and the bigger these are, the better. In order to overcome this common limitation, we developed an alternative metric that first defines a single-agreement reliability, then a case reliability (averaging over all of the agreements), and finally, a dataset reliability (averaging over all of the cases included in the dataset).  
To define an agreement reliability, our method leverages the confidence of the raters involved, when we ask them to estimate the confidence of their ratings. If this confidence is high, we assume the raters have not guessed. If both raters (who agree with each others) are highly confident, we assume their agreement is genuine, that is not due to chance.  
However, we must also take into consideration the competence of the raters, as a way to discern between agreements made on a correct label and agreements that are wrong.  
To this aim, we consider the rater's skill. This can be estimated in many ways, both qualitatively and quantitatively. In this latter case, a proxy dimension of the raters' competence is their average accuracy. The average accuracy of each single rater can then be estimated on the basis of a specific evaluation test or, more conveniently, on the basis of the majority voting of the other raters involved.  
With reference to Figure 6, our method to quantify the reliability can be described as going backwards from an observed rating about which two raters agree, i.e., an agreement: first, we evaluate whether, based on the raters' confidence for the given rating, the observed agreement is genuine or only due to chance; second, we consider whether the agreed-upon rating is right or not, based on the raters' competence with respect to the discriminative task at hand.  
Figure 6: Graphical presentation of the general idea of the weighted reliability score, \(\varrho\). This reliability score is multi-dimensional as it encompasses both the confidence of the raters and their competence. In this graphical example, rater \(r_{i}\) is much less competent than rater \(r_{j}\), and hence, the probability that her/his rating is correct is lower.