## 3 Method
### Paradoxes of Reliability Measures  
Given the large number of proposed reliability measures, there has been an interest in proving one metric \"better\" than the others, in some respect. This has typically resulted in focusing on so-called paradoxes, i.e., statements asserting that a given class of reliability measures violates a set of intuitive properties. While we do not necessarily subscribe to viewing each and every violation as a paradox (e.g., see [31; 32; 25; 33] for an account of this meta-perspective on so-called paradoxes of reliability), in what follows, we will show that the increased flexibility of the \(\sigma\) (resp. \(\rho\)) can be used to overcome (or simply shed light on) these properties. In particular, we will focus on two paradoxes that have been widely studied and discussed in the relevant literature [21; 22; 25; 34] in regard to the most common and frequently adopted reliability measures, that is kappa and \(\alpha\). **Paradox 1**. : _High agreement, but low reliability: Let \(C=\{0,1\}\) be the set of possible classes. Suppose that all raters agree on assigning \(0\) to all cases but one, and on assigning \(1\) to the remaining case. Then, the observed agreement is perfect, but the value of kappa (or \(\alpha\)) is \(\leq 0\). If all the raters would have agreed on assigning the same class to all labels, then the value of kappa (or \(\alpha\)) would be undefined._  
**Paradox 2**. : _In the case of two raters, when considering the distributions of labels for the raters, unbalanced distributions produce higher values of kappa (or \(\alpha\)) than more balanced ones even when the observed agreement is the same._  
We argue that both of these paradoxes arise from the interdependence in the definition of kappa and \(\alpha\) between the model of the chance effects and the distribution of the labels among the raters. Thus, the paradoxical properties mentioned above are an intrinsic feature of any reliability score that does not separately models the self-agreement of the raters and their chance effects. Moreover, in regard to Paradox 2, we doubt this should be considered a negative feature: a sound reliability measure should exhibit this behavior whenever a model of the chance effects of the raters is not available: in the cases described by this paradox, if one of the two classes is much more represented in the dataset (and this mirrors the real population of interest), then the likelihood that the raters agreed just by chance is high, since they could have agreed just by selecting the most probable answer. In the extreme case where all the raters assigned all the cases to the same label, then the reliability of the dataset would be undefined as there is no way to understand if the observed agreement was genuine or due to chance. However, this observation does not apply to the weighted reliability score \(\varrho\) (nor to \(\sigma\)) due to the increasing flexibility of the chance model we adopted in its definition. In particular, the value of \(\sigma\) depends not only on the distribution of the classes in the dataset (which influences only the weighting factor \(\mathbf{p}_{r_{i}}\) in the definition of \(\hat{c_{i}}\)), but also on the confidence values \(c_{i}\). Thus, formally:  
**Theorem 1**. : _Let \(D\) be a dataset, and let \(r_{i}(x)\) be the labeling given by rater \(i\) on case \(x\), with \(C\) the set of class labels. Let \(\forall i,x,j,y.r_{i}(x)=r_{j}(y)=c\in C\). Then, \(\sigma>0\), and further, \(\sigma\) depends only on \(c_{i}(x)\) and \(\mathbf{p}_{r_{i}}(x)\)._  
Proof. : From the definition of \(\sigma\) in Equation (4), it is evident that \(\sigma\geq 0\) and \(\sigma=0\) iff \(\forall x\in D,i\in\{1,...,k\}.c_{i}(x)=0\). Obviously, also, the definition of \(\sigma\) only depends on the stated factors. **Definition 1**. : _We say that two probability distributions \(p_{1},p_{2}\) are marginal symmetric if there exists a permutation \(\pi\) s.t.