## 3 Method
### Derivation of the Weighted Reliability Score  
We will assume the following setting: each rater in a group of \(k\) is asked to provide a label (from a given label set \(C=\{1,...,n\}\), which we assume to be categorical) to each case \(x\) in a dataset \(S\). We denote rater \(r_{i}\)'s labeling of case \(x\) as \(r_{i}(x)\). Furthermore, each rater \(r_{i}\) is also asked to report his/her confidence \(c_{i}(x)\) in each label he/she gives. As previously stated, we consider reliability to be a composite construct, defined by the raters' confidence and competence. As anticipated above, our method can be seen as going backwards from an observed agreement; thus, we first need to ask whether an observed agreement was genuine or due to chance. To this aim, we assume a two-step decision procedure, which can be seen as an instantiation of the standard decision-theoretic setting for decision under risk [29]: first, the decision maker flips a biased coin to choose between random choice (i.e., the random selection of a possible alternative according to an a priori probability distribution) with probability \(1-c_{i}(x)\) and peaked choice (i.e., the selection of an alternative according to an informed, case-dependent, distribution) with probability \(c_{i}(x)\). Here, \(c_{i}(x)\) is the confidence expressed by the rater in her/his decision \(r_{i}(x)\), which we equate to the unconditional probability of the fact that the observed label \(r_{i}(x)\) has been chosen genuinely and not simply by chance (i.e., \(c_{i}(x)=P(r_{i}(x)\text{ genuinely})\)). Then:  
* If rater \(r_{i}\) has \"chosen\" random choice, then we assume her/his label is selected according to a distribution \(\mathbf{p}_{r_{i}}(x)=\langle p(1),...,p(n)\rangle\) where \(p(i)=\frac{1}{n}\) (assuming a uniform distribution of the alternatives) or \(p(i)=e(i)\) where \(e(i)\) is an empirical prior estimate of the real prevalence of \(i\) (this can be derived from a ground truth labeling, if available, or considering all the labelings given by the multiple raters);
* If rater \(r_{i}\) has \"chosen\" peaked choice, then we assume her/his label is selected according to a distribution \(\mathbf{d}_{r_{i}}=\langle d(1),...,d(n)\rangle\) where \(\exists i.d(i)=1\). The conditional estimate:  
\[P(r_{i}(x)\text{ genuinely}|r_{i}(x))\]  
can be obtained as:  
\[\hat{c}_{i}(x)=\frac{c_{i}(x)}{c_{i}(x)+(1-c_{i}(x))p(r_{i}(x))} \tag{1}\]  
Thus, given two raters \(r_{i},r_{j}\) who have been observed to agree on case \(x\) (i.e., \(r_{i}(x)=r_{j}(x)\)), we want to quantify the probability that the observed agreement is not due to chance; this means that both raters selected the peaked distribution, that is a Genuine Agreement (GA) occurred. Assuming that the two ratings were given independently, then the pairwise GA is defined as:  
\[GA_{x}(r_{i},r_{j})= \begin{cases}0&r_{i}(x)\
eq r_{j}(x)\\ \hat{c}_{i}(x)\hat{c}_{j}(x)&\text{otherwise}\end{cases}  \tag{2}\]  
We can now define the degree of concordance \(\sigma\), as a measure of the proportion of genuine agreement (in either a single case or in a whole set of cases): this metric is simply defined as the average of GA both case-wise and sample-wise. First, we define the case-wise degree of concordance:  
\[\sigma(x,R)= \begin{pmatrix}m\\ 2\end{pmatrix} ^{-1}\sum_{r_{i}\
eq r_{j}\in R}GA_{x}(r_{i},r_{j}) \tag{3}\]where the factor \((\frac{m}{2})^{-1}=\frac{2(m-2)!}{m! }\) is the number of pairs of raters in \(R\). We then define the degree of concordance as the average of the case-wise \(\sigma\) over the whole sample \(S\), that is:  
\[\sigma(S,R)=\frac{1}{|S|}\sum_{x\in S}\sigma(x,R) \tag{4}\]  
Thus far, we only considered the agreement dimension of our definition of reliability; by following our backward reasoning (see Figure 6), we then discount the obtained degree of concordance with the raters' competence in the task at hand. To this aim, we need an estimate of the accuracy \(a\hat{c}c_{i}\) of each rater \(i\) on each case \(x\).