# Weighted Kappas for3×3 Tables

Authors: Matthijs J. Warrens
Journal: Journal of Probability and Statistics
 (2013)

---

## Abstract

Weighted kappa is a widely used statistic for summarizing inter-rater agreement on a categorical scale. For rating scales with three categories, there are seven versions of weighted kappa. It is shown analytically how these weighted kappas are related. Several conditional equalities and inequalities between the weighted kappas are derived. The analytical analysis indicates that the weighted kappas are measuring the same thing but to a different extent. One cannot, therefore, use the same magnitude guidelines for all weighted kappas.


---

## Body

## 1. Introduction

In biomedical, behavioral, and engineering research, it is frequently required that a group of objects is rated on a categorical scale by two observers. Examples are the following: clinicians that classify the extent of disease in patients; pathologists that rate the severity of lesions from scans; and experts that classify production faults. Analysis of the agreement between the two observers can be used to assess the reliability of the rating system. High agreement would indicate consensus in the diagnosis and interchangeability of the observers. Various authors have proposed statistical methodology for analyzing agreement. For example, for modeling patterns of agreement, the loglinear models proposed in Tanner and Young [1] and Agresti [2, 3] can be used. However, in practice researchers are frequently only interested in a single number that quantifies the degree of agreement between the raters [4, 5]. Various statistics have been proposed in the literature [6, 7], but the most popular statistic for summarizing rater agreement is the weighted kappa introduced by Cohen [8].Weighted kappa allows the use of weighting schemes to describe the closeness of agreement between categories. Each weighting scheme defines a different version or special case of weighted kappa. Different weighting schemes have been proposed for the various scale types. In this paper, we only consider scales of three categories. This is the smallest number of categories for which we can distinguish three types of categorical scales, namely, nominal scales, continuous-ordinal scales, and dichotomous-ordinal scales [9]. A dichotomous-ordinal scale contains a point of “absence” and two points of “presence”, for example, no disability, moderate disability, or severe disability. A continuous-ordinal scale does not have a point of “absence”. The scale can be described by three categories of “presence”, for example, low, moderate, or high. Identity weights are used when the categories are nominal [10]. In this case, weighted kappa becomes the unweighted kappa introduced by Cohen [11], also known as Cohen’s kappa. Linear weights [12, 13] or quadratic weights [14, 15] can be used when the categories are continuous ordinal. The modified linear weights introduced in Cicchetti [9] are suitable if the categories are dichotomous ordinal.Although weighted kappa has been used in thousands of research applications [16], it has also been criticized by various authors [17–19]. Most of the criticism has focused on a particular version of weighted kappa, namely, Cohen’s kappa for nominal categories. Weighted kappa and unweighted kappa correct for rater agreement due to chance alone using the marginal distributions. For example, in the context of latent class models, de Mast [18] and de Mast and van Wieringen [6] argued that the premise that chance measurements have the distribution defined by the marginal distributions cannot be defended. It is, therefore, difficult to interpret the value of Cohen’s kappa, and it makes the question of how large or how small the value should be arbitrary. Using signal detection theory, Uebersax [19] showed that different agreement studies with different marginal distributions can produce the same value of Cohen’s kappa. Again, this makes the value difficult to interpret. Alternative statistics for summarizing inter-rater agreement are discussed in, for example, de Mast [18] and Perreault and Leigh [20].Although the choice for a specific version of weighted kappa usually depends on the type of categorical scale at hand, it frequently occurs that weighted kappas corresponding to different weighting schemes are applied to the same data. For example, Cohen’s kappa for nominal scales [11] is also frequently applied when the categories are continuous ordinal. When different weighted kappas are applied to the same data, they usually produce different values [5, 21]. For understanding the behavior of weighted kappa and its dependence on the weighting scheme, it is useful to compare the different versions of weighted kappa analytically [21]. For example, if the agreement table is tridiagonal, then the value of the quadratically weighted kappa exceeds the value of the linearly weighted kappa, which, in turn, is higher than the value of unweighted kappa [22, 23]. An agreement table is tridiagonal if it has nonzero elements only on the main diagonal and on the two diagonals directly adjacent to the main diagonal. These analytic results explain orderings of the weighted kappas that are observed in practice.In this paper, we consider scales that consist of three categories and compare the values of seven special cases of weighted kappa. There are several reasons why the case of three categories is an interesting topic of investigation. First of all, various scales that are used in practice consist of three categories only. Examples can be found in Anderson et al. [24] and Martin et al. [25]. Furthermore, the case of three categories is the smallest case where symmetrically weighted kappas in general have different values, since all weighted kappas with symmetric weighting schemes coincide with two categories. Finally, as it turns out, with three categories we may derive several strong analytic results, which do not generalize to the case of four or more categories. The seven weighted kappas belong to two parameter families. For each parameter family, it is shown that there are only two possible orderings of its members. Hence, despite the fact that the paper is limited to weighted kappas for three categories, we present various interesting and useful results that deepen our understanding of the application of weighted kappa.The paper is organized as follows. In Section2 we introduce notation and define four versions of weighted kappa. In Section 3, we introduce the three category reliabilities of a 3×3 agreement table as special cases of weighted kappa. The two parameter families are defined in Section 4. In Section 5, we present several results on inequalities between the seven weighted kappas. In Section 6, we consider the case that all special cases of weighted kappa coincide. Section 7 contains a discussion.


## 2. Weighted Kappas

Suppose that two raters, each, independently classify the same set of objects (individuals, observations) into the same set of three categories that are defined in advance. For a population ofn objects, let πij for i,j∈{1,2,3} denote the proportion classified into category i by the first observer and into category j by the second observer. Table 1 presents an abstract version of a 3×3 population agreement table of proportions. The marginal totals π1+,π2+,π3+ and π+1,π+2,π+3 indicate how often raters 1 and 2 used the categories 1, 2, and 3. Four examples of 3×3 agreement tables from the literature with frequencies are presented in Table 2. The marginal totals of the tables are in bold. For each table, the last column of Table 2 contains the corresponding estimates of seven weighted kappas. Between brackets behind each point estimate is the associated 95% confidence interval. Definitions of the weighted kappas are presented below.Table 1
Notation for a3×3 agreement table with proportions.
Rater 2
1
2
3
Total
1
π
11
π
12
π
13
π
1
+
Rater 1
2
π
21
π
22
π
23
π
2
+
3
π
31
π
32
π
33
π
3
+
Total
π
+
1
π
+
2
π
+
3
1Table 2
Four examples of3×3 agreement tables from the literature with corresponding values of weighted kappas.
Category labels
3
×
3 table
Kappas
Estimates
95
% CI
Psychotic
106
10
4
120
κ
^
=
.
429
(
.
323
–
.
534
)
Neurotic
22
28
10
60
κ
^
ℓ
=
.
492
(
.
393
–
.
592
)
Personality disorder
2
12
6
20
κ
^
q
=
.
567
(
.
458
–
.
676
)
130
50
20
200
κ
^
c
=
.
536
(
.
434
–
.
637
)
Spitzer et al. [26]
κ
^
1
=
.
596
(
.
481
–
.
710
)
Personality types
κ
^
2
=
.
325
(
.
182
–
.
468
)
κ
^
3
=
.
222
(
.
024
–
.
420
)
No atopy
136
12
1
149
κ
^
=
.
730
(
.
645
–
.
815
)
Atopy, no neurodermatitis
8
59
4
71
κ
^
ℓ
=
.
737
(
.
652
–
.
822
)
Neurodermatitis
2
4
6
12
κ
^
q
=
.
748
(
.
651
–
.
845
)
146
75
11
232
κ
^
c
=
.
759
(
.
678
–
.
840
)
Simonoff [27]
κ
^
1
=
.
786
(
.
703
–
.
869
)
Stability of atopic disease
κ
^
2
=
.
720
(
.
624
–
.
817
)
κ
^
3
=
.
497
(
.
240
–
.
754
)
Negative
1360
63
8
1431
κ
^
=
.
675
(
.
632
–
.
719
)
Low positive
61
66
13
140
κ
^
ℓ
=
.
761
(
.
725
–
.
798
)
High positive
10
16
137
163
κ
^
q
=
.
830
(
.
798
–
.
862
)
1431
145
158
1734
κ
^
c
=
.
744
(
.
705
–
.
782
)
Castle et al. [28]
κ
^
1
=
.
716
(
.
672
–
.
760
)
Results of hybrid  capture  testing
κ
^
2
=
.
415
(
.
339
–
.
491
)
κ
^
3
=
.
839
(
.
794
–
.
884
)
Good recovery
36
4
1
41
κ
^
=
.
689
(
.
549
–
.
828
)
Moderate disability
5
20
4
29
κ
^
ℓ
=
.
735
(
.
610
–
.
861
)
Severe disability
0
1
9
10
κ
^
q
=
.
788
(
.
667
–
.
910
)
41
25
14
80
κ
^
c
=
.
741
(
.
614
–
.
868
)
Anderson et al. [24]
κ
^
1
=
.
750
(
.
605
–
.
895
)
Glasgow  outcome  scale  scores
κ
^
2
=
.
610
(
.
427
–
.
793
)
κ
^
3
=
.
707
(
.
489
–
.
925
)Recall that weighted kappa allows the use of weighting schemes to describe the closeness of agreement between categories. For each cell probabilityπij, we may specify a weight. A weighting scheme is called symmetric if for all i,j cell probabilities πij and πji are assigned the same weight. The weighting schemes can be formulated from either a similarity or a dissimilarity perspective. Definitions of weighted kappa in terms of similarity scaling can be found in Warrens [13, 22]. For notational convenience, we will define the weights in terms of dissimilarity scaling here. For the elements on the agreement diagonal, there is no disagreement. The diagonal elements are, therefore, assigned zero weight [8, page 215]. The other six weights are non-negative real numbers wi for i∈{1,2,…,6}. The inequality wi>0 indicates that there is some disagreement between the assignments by the raters. Categories that are more similar are assigned smaller weights. For example, ordinal scale categories that are one unit apart in the natural ordering are assigned smaller weights than categories that are more units apart.Table3 presents one general and seven specific weighting schemes from the literature. The identity weighting scheme for nominal categories was introduced in Cohen [11]. The top table in Table 2 is an example of a nominal scale. The quadratic weighting scheme for continuous-ordinal categories was introduced in Cohen [8]. The quadratically weighted kappa is the most popular version of weighted kappa [4, 5, 15]. The linear weighting scheme for continuous-ordinal categories was introduced in Cicchetti and Allison [29] and Cicchetti [30]. The second table in Table 2 is an example of a continuous-ordinal scale. The dichotomous-ordinal weighting scheme was introduced in Cicchetti [9]. The two bottom tables in Table 2 are examples of dichotomous-ordinal scales. All weighting schemes in Table 3, except the general symmetric and the quadratic, are special cases of the weighting scheme with additive weights introduced in Warrens [31].Table 3
Eight weighting schemes for3×3 tables.
Name
Source
Scale type
Symbol
Scheme
0
w
3
w
2
General symmetric
[8]
κ
w
w
3
0
w
1
w
2
w
1
0
0
1
1
Identity
[11]
Nominal
κ
1
0
1
1
1
0
0
1
2
Linear
[29, 30]
Continuous-ordinal
κ
ℓ
1
0
1
2
1
0
0
1
4
Quadratic
[8]
Continuous-ordinal
κ
q
1
0
1
4
1
0
0
2
3
[9, 31]
Dichotomous-ordinal
κ
c
2
0
1
3
1
0
0
1
1
Reliability category 1
Dichotomous
κ
1
1
0
0
1
0
0
0
1
0
Reliability category 2
Dichotomous
κ
2
1
0
1
0
1
0
0
0
1
Reliability category 3
Dichotomous
κ
3
0
0
1
1
1
0In this paper, we only consider weighted kappas with symmetric weighting schemes. For notational convenience, we define the following six coefficients:(1)a1=π23+π32,b1=π2+π+3+π3+π+2,a2=π13+π31,b2=π1+π+3+π3+π+1,a3=π12+π21,b3=π1+π+2+π2+π+1.
To avoid pathological cases, we assume that b1,b2,b3>0. The coefficients a1, a2, and a3 reflect raw disagreement between the raters, whereas b1, b2, and b3 reflect chance-expected disagreement. The general formula of weighted kappa for 3×3 tables with symmetric weights will be denoted by κw. In terms of the coefficients a1,a2,a3 and b1,b2,b3, this weighted kappa is defined as
(2)κw=1-w1a1+w2a2+w3a3w1b1+w2b2+w3b3.
The value of κw lies between 1 and -∞. The numerator w1a1+w2a2+w3a3 of the fraction in (2) reflects raw weighted disagreement. It is a weighted sum of the cell probabilities πij that are not on the main diagonal of the 3×3 table, and it quantifies the disagreement between the raters. The denominator w1b1+w2b2+w3b3 of the fraction in (2) reflects weighted disagreement under chance. It is a weighted sum of the products πi+π+j for i≠j. High values of w1a1+w2a2+w3a3 correspond to high disagreement. If w1a1+w2a2+w3a3=0, then we have κw=1, and there is perfect agreement between the observers. Furthermore, we have κw=0 if the raw weighted disagreement is equal to the weighted disagreement under chance.Special cases ofκw are obtained by using the specific weighting schemes in Table 3 in the general formula (2). Unweighted kappa, linearly weighted kappa, quadratically weighted kappa, and Cicchetti’s weighted kappa are, respectively, defined as
(3)κ=1-a1+a2+a3b1+b2+b3,κℓ=1-a1+2a2+a3b1+2b2+b3,κq=1-a1+4a2+a3b1+4b2+b3,κc=1-a1+3a2+2a3b1+3b2+2b3.
Assuming a multinominal sampling model with the total numbers of objects n fixed, the maximum likelihood estimate of the cell probability πij for i,j∈{1,2,3} is given by π^ij=nij/n, where nij is the observed frequency. Note that the a1,a2,a3 and b1,b2,b3 are functions of the cell probabilities πij. The maximum likelihood estimate κ^w of κw in (2) is obtained by replacing the cell probabilities πij by π^ij [32]. The last column of Table 2 contains the estimates of the weighted kappas for each of the four 3×3 tables. For example, for the top table of Table 2, we have κ^=.429, κ^ℓ=.492, κ^q=.567, and κ^c=.536. Between brackets behind the kappa estimates are the 95% confidence intervals. These were obtained using the asymptotic variance of weighted kappa derived in Fleiss et al. [33].


## 3. Category Reliabilities

With a categorical scale, it is sometimes desirable to combine some of the categories [34], for example, when two categories are easily confused, and then calculate weighted kappa for the collapsed table. If we combine two of the three categories, the 3×3 table collapses into a 2×2 table. For a 2×2 table, all weighted kappas with symmetric weighting schemes coincide. Since we have three categories, there are three possible ways to combine two categories. The three κ-values of the collapsed 2×2 tables are given by
(4)κ1=1-a2+a3b2+b3,κ2=1-a1+a3b1+b3,κ3=1-a1+a2b1+b2.
These three kappas are obtained by using the three bottom weighting schemes in Table 3 in the general formula (2). The last column of Table 2 contains the estimates of these weighted kappas for each of the four 3×3 tables.Weighted kappaκi for i∈{1,2,3} corresponds to the 2×2 table that is obtained by combining the two categories other than category i. The 2×2 table reflects how often the two raters agreed on the category i and on the category “all others”. Weighted kappa κi for i∈{1,2,3}, hence, summarizes the agreement or reliability between the raters on the single category i, and it is, therefore, also called the category reliability of i [10]. It quantifies how good category i can be distinguished from the other two categories. For example, for the second table of Table 2, we have κ^1=.786, κ^2=.720, and κ^3=.497. The substantially lower value of κ3 indicates that the third category is not well distinguished from the other two categories.Unweighted kappaκ and linearly weighted kappa κℓ are weighted averages of the category reliabilities. Unweighted kappa is a weighted average of κ1, κ2, and κ3, where the weights are the denominators of the category reliabilities [10]:
(5)(b2+b3)κ1+(b1+b3)κ2+(b1+b2)κ3(b2+b3)+(b1+b3)+(b1+b2)=κ.
Since κ is a weighted average of the category reliabilities, the κ-value always lies between the values of κ1, κ2, and κ3. This property can be verified for all four tables of Table 2. Therefore, when combining two categories, the κ-value can go either up or down, depending on which two categories are combined [34]. The value of κ is a good summary statistic of the category reliabilities if the values of κ1, κ2, and κ3 are (approximately) identical. Table 2 shows that this is not the case in general. With an ordinal scale, it only makes sense to combine categories that are adjacent in the ordering. We should, therefore, ignore κ2 with ordered categories, since this statistic corresponds to the 2×2 table that is obtained by merging the two categories that are furthest apart. Furthermore, note that for the two bottom 3×3 tables of Table 2 the first category is the “absence” category. If the scale is dichotomous ordinal and category 1 is the “absence” category, then κ1 is the κ-value of the 2×2 table that corresponds to “absence” versus “presence” of the characteristic.The statisticκℓ is a weighted average of κ1 and κ3, where the weights are the denominators of the category reliabilities [13, 35]:
(6)(b2+b3)κ1+(b1+b2)κ3(b2+b3)+(b1+b2)=κℓ.
Since κℓ is a weighted average of the category reliabilities κ1 and κ3, the κℓ-value always lies between the values of κ1 and κ3. This property can be verified for all four tables of Table 2. Unlike κq, statistic κℓ can be considered an extension of κ to ordinal scales that preserves the “weighted average” property [13, 35]. The value of κℓ is a good summary statistic of κ1 and κ3 if the two weighted kappas are (approximately) identical. This is the case for the two bottom tables of Table 2.The statisticκc is also a weighted average of κ1 and κ3, where the weights are 2(b2+b3) and (b1+b2):
(7)2(b2+b3)κ1+(b1+b2)κ32(b2+b3)+(b1+b2)=κc.
A proof can be found in Warrens [31].


## 4.   Families of Weighted Kappas

In this section, we show that the seven weighted kappas introduced in Sections2 and 3 are special cases of two families. Let r≥0 be a real number. Inspection of the formulas κ2, κ, κℓ, and κq shows that they only differ on how the coefficients a2 and b2 are weighted. The first family is, therefore, given by
(8)λr=1-a1+ra2+a3b1+rb2+b3.
For r=0,1,2,4, we have, respectively, the special cases κ2, κ, κℓ, and κq.Recall thatκℓ and κc are weighted averages of the category reliabilities κ1 and κ3. This motivates the following definition. Let s∈[0,1]. Then the second family is defined as
(9)μs=(1-s)(b2+b3)κ1+s(b1+b2)κ3(1-s)(b2+b3)+s(b1+b2).
The family μs consists of the weighted averages of κ1 and κ3 where the weights are multiples of (b2+b3) and (b1+b2). For s=0,1/3,1/2,1, we have, respectively, the special cases κ1, κc, κℓ, and κ3. Note that κℓ belongs to both λr and μs.The following proposition presents a formula for the family in (9) that will be used in Theorem 6 below.Proposition 1.
The family in (9) is equivalent to
(10)μs=1-sa1+a2+(1-s)a3sb1+b2+(1-s)b3.Proof.
Sinceκ1 and κ3 are equal to, respectively,
(11)κ1=b2+b3-(a2+a3)b2+b3,κ3=b1+b2-(a1+a2)b1+b2,
we can write (9) as
(12)μs=(1-s)(b2+b3-a2-a3)+s(b1+b2-a1-a2)(1-s)(b2+b3)+s(b1+b2)=1-(1-s)(a2+a3)+s(a1+a2)(1-s)(b2+b3)+s(b1+b2),
which is identical to the expression in (10).


## 5. Inequalities

In this section, we present inequalities between the seven weighted kappas. We will use the following lemma repeatedly.Lemma 2.
Letu,v≥0 and r,w,z>0. Then one has the following:
(13)(i)uw<vz⟺uw<ru+vrw+z;(ii)uw=vz⟺uw=ru+vrw+z;(iii)uw>vz⟺uw>ru+vrw+z.Proof.
Sincew and z are positive numbers, we have u/w<v/z, or uz<vw. Adding ruw to both sides, we obtain u(rw+z)<w(ru+v), or u/w<(ru+v)/(rw+z).Theorem3 classifies the orderings of the special cases of the family λr in (8).Theorem 3.
Forr<r′ one has the following:
(14)(i)λr<λr′⟺a1+a3b1+b3>a2b2;(ii)λr=λr′⟺a1+a3b1+b3=a2b2;(iii)λr>λr′⟺a1+a3b1+b3<a2b2.Proof.
The inequalityλr<λr′ is equivalent to
(15)a1+ra2+a3b1+rb2+b3>a1+r′a2+a3b1+r′b2+b3.
Since r<r′, it follows from Lemma 2 that inequality (15) is equivalent to
(16)a1+ra2+a3b1+rb2+b3>(r′-r)a2(r′-r)b2=a2b2.
Applying Lemma 2 for a second time, we find that inequality (16) is equivalent to
(17)a1+a3b1+b3>a2b2.
This completes the proof.Theorem3 shows that, in practice, we only observe one of two orderings of κ2, κ, κℓ, and κq. In most cases, we have κ2<κ<κℓ<κq. For example, in Table 2 all 3×3 tables exhibit this ordering. For all these 3×3 tables, it holds that (a1+a3)/(b1+b3)>a2/b2. Furthermore, if the 3×3 table would be tridiagonal [22, 23], we would have a2=0, and the inequality (a1+a3)/(b1+b3)>a2/b2 would also hold. The other possibility is that we have κ2>κ>κℓ>κq. The only example from the literature where we found this ordering is the 3×3 table presented in Cohen [11]. The table in Cohen satisfies the condition in (iii) of Theorem 3. We conclude that, with ordinal scales, we almost always have the ordering κ2<κ<κℓ<κq. The equality condition in Theorem 3 is discussed in Section 6.Theorem4 classifies the orderings of the special cases of the family μs in (9).Theorem 4.
Fors<s′, one has the following:
(18)(i)μs<μs′⟺a1+a2b1+b2>a2+a3b2+b3;(ii)μs=μs′⟺a1+a2b1+b2=a2+a3b2+b3;(iii)μs>μs′⟺a1+a2b1+b2<a2+a3b2+b3.Proof.
The special cases ofμs are weighted averages of κ1 and κ3. For s<s′, we have μs<μs′ if and only if κ1<κ3; that is, a statistic that gives more weight to κ3 will be higher if the κ3-value exceeds the κ1-value. Furthermore, we have κ1<κ3⇔(19)a1+a2b1+b2>a2+a3b2+b3.
This completes the proof.Theorem4 shows that, in practice, we only observe one of two orderings of κ3, κℓ, κc, and κ1. We either have the ordering κ3<κℓ<κc<κ1, which is the case in the first, second, and fourth 3×3 tables of Table 2, or we have κ3>κℓ>κc>κ1, which is the case in the third 3×3 table in Table 2.Proposition5 follows from Theorems 3 and 4 and the fact that κ is a weighted average of κ1, κ2, and κ3 [10].Proposition 5.
Consider the following:(20)(i)κ<κ3<κ1⟺κ2<κ<κ3<κℓ<κc<κ1;(ii)κ<κ1<κ3⟺κ2<κ<κ1<κc<κℓ<κ3,κq;(iii)κ3<κ1<κ⟺κ3,κq<κℓ<κc<κ1<κ<κ2;(iv)κ1<κ3<κ⟺κ1<κc<κℓ<κ3<κ<κ2.Proposition5 shows that we have an almost complete picture of how the seven weighted kappas are ordered just by comparing the values of κ, κ1, and κ3. The double inequality κ<κ3<κ1 holds for the fourth 3×3 table of  Table 2, whereas the inequality κ<κ1<κ3 holds for the third 3×3 table of Table 2. Both tables have a dichotomous-ordinal scale. Recall that κc corresponds to a weighting scheme specifically formulated for dichotomous-ordinal scales. It turns out that the κc-value can be both lower and higher than the κℓ-value with dichotomous-ordinal scales. Which statistic is higher depends on the data. Furthermore, κ tends to be smaller than κ1 and κ3. The condition κ<κ1,κ3 can be interpreted as an increase in the κ-value if we combine the middle category of the 3-category scale with one of the outer categories. This way of merging categories makes sense if the categories are ordered.


## 6. Equalities

Apart from the equality conditions in (ii) of Theorems3 and 4, we only considered inequalities between the weighted kappas in the previous section. Unless there is perfect agreement, the values of the weighted kappas are usually different. Table 4 contains three hypothetical agreement tables that we have constructed to illustrate that the three equality conditions in Theorems 3, 4, and 6 (below) are not identical. For the top table in Table 4, we have (a1+a3)/(b1+b3)=a2/b2, which is equivalent to the equality κ2=κ=κℓ=κq (Theorem 3). Although all weighted kappas of the family λr coincide, the kappas not belonging to this family produce different values. For the middle table in Table 4 we have (a1+a2)/(b1+b2)=(a2+a3)/(b2+b3), which is equivalent to the equality κ3=κℓ=κc=κ1 (Theorem 4). Although all weighted kappas of the family μs coincide, the kappas that do not belong to this family produce different values.Table 4
Three hypothetical3×3 agreement tables with corresponding values of weighted kappas.
Categories
3
×
3 table
Kappas
1
4
1
0
5
κ
^
=
.
617
κ
^
1
=
.
475
2
1
2
0
3
κ
^
ℓ
=
.
617
κ
^
2
=
.
617
3
3
0
12
15
κ
^
q
=
.
617
κ
^
3
=
.
736
8
3
12
23
κ
^
c
=
.
572
1
6
0
1
7
κ
^
=
.
581
κ
^
1
=
.
635
2
3
6
0
9
κ
^
ℓ
=
.
635
κ
^
2
=
.
479
3
0
3
6
9
κ
^
q
=
.
668
κ
^
3
=
.
635
9
9
7
25
κ
^
c
=
.
635
1
11
1
0
12
κ
^
=
.
603
κ
^
1
=
.
603
2
2
5
0
7
κ
^
ℓ
=
.
603
κ
^
2
=
.
603
3
2
1
3
6
κ
^
q
=
.
603
κ
^
3
=
.
603
15
7
3
25
κ
^
c
=
.
603For the bottom table in Table4, we have the stronger condition a1/b1=a2/b2=a3/b3. Theorem 6 (below) shows that this condition is equivalent to the case that all weighted kappas, that is, all special cases of (2), coincide.Theorem 6.
The following conditions are equivalent:(21)(i)a1b1=a2b2=a3b3=c≥0;(ii)κw=1-c;(iii)λr=λt=μsforr≠t,s≠12;(iv)λr=μs=μtforr≠2,s≠t.Proof.
In words, (ii) means that all special cases of (2) are identical. Therefore, (ii)⇒(iii),(iv). We first show that (i)⇒(ii). It then suffices to show that (iii),(iv)⇒(i).
If (i) holds, we have(22)a2b2=c1a1c1b1,a3b3=c2a1c2b1
for certain c1,c2>0. Hence,
(23)κw=1-w1a1+w2c1a1+w3c2a1w1b1+w2c1b1+w3c2b1=1-a1(w1+w2c1+w3c2)b1(w1+w2c1+w3c2)=1-a1b1=1-c.
Thus, all special cases of weighted kappa in (2) coincide if (i) is valid.
Next, we show that(iii),(iv)⇒(i). Consider condition (iii) first. If two special cases of λr are identical, it follows from Theorem 3 that all of them are identical. Hence, we have κ2=μs for a certain s∈[0,1] with s≠1/2. Using formula (10), we have κ2=μs⇔(24)a1+a3b1+b3=sa1+a2+(1-s)a3sb1+b2+(1-s)b3.
Combining (24) with a2/b2=(a1+a3)/(b1+b3) (Theorem 3), we obtain
(25)a2b2=a1+a3b1+b3=sa1+a2+(1-s)a3sb1+b2+(1-s)b3.
Applying Lemma 2 to the outer ratios of (25), we obtain
(26)a2b2=a1+a3b1+b3=sa1+(1-s)a3sb1+(1-s)b3.
First, suppose that  s<1/2. Applying Lemma 2 to the right-hand side equality of (26), we obtain
(27)a2b2=a1+a3b1+b3=(1-2s)a3(1-2s)b3=a3b3,
or a2/b2=a3/b3. Applying Lemma 2 to the second and fourth term of the triple equality (27), we obtain a1/b1=a3/b3. Thus, we have a1/b1=a2/b2=a3/b3, which completes the proof for s<1/2. Next, suppose that s>1/2. Applying Lemma 2 to the right-hand side equality of (26), we obtain
(28)a2b2=a1+a3b1+b3=(2s-1)a1(2s-1)b1=a1b1,
or a1/b1=a2/b2. Applying Lemma 2 to the second and fourth terms of the triple equality (28), we obtain a1/b1=a3/b3. Thus, we also have a1/b1=a2/b2=a3/b3 for s>1/2, which completes the proof for condition (iii).
Next, consider condition (iv). If two special cases ofμs are identical, it follows from Theorem 4 that all of them are identical. Hence, we have κ1=κ3=λr for a certain r≥0 and r≠2. We have κ3=κ1=λr⇔(29)a1+a2b1+b2=a2+a3b2+b3=a1+ra2+a3b1+rb2+b3.
First, suppose that r>2. Applying Lemma 2 to the outer ratios of (29), we obtain
(30)a1+a2b1+b2=a2+a3b2+b3=(r-1)a2+a3(r-1)b2+b3.
Applying Lemma 2 to the right-hand side equality of (30) gives
(31)a1+a2b1+b2=a2+a3b2+b3=(r-2)a2(r-2)b2=a2b2.
Applying Lemma 2 to the outer ratios of (31), we obtain a1/b1=a2/b2, while applying Lemma 2 to the second and fourth terms of the triple equality (31), we obtain a3/b3=a2/b2. Thus, we have a1/b1=a2/b2=a3/b3.
Finally, ifr<2, then consider the equality κ1=κ3=κℓ=λr⇔(32)a2+a3b2+b3=a1+a2b1+b2=a1+2a2+a3b1+2b2+b3=(2/r)a1+2a2+(2/r)a3(2/r)b1+2b2+(2/r)b3.
Since 2/r>1, applying Lemma 2 to the right-hand side equality of (32) gives
(33)a1+a2b1+b2=a2+a3b2+b3=a1+2a2+a3b1+2b2+b3=(2/r-1)a1+(2/r-1)a3(2/r-1)b1+(2/r-1)b3=a1+a3b1+b3.
However,
(34)a1+a2b1+b2=a2+a3b2+b3=a1+a3b1+b3
is equivalent to κ1=κ2=κ3. Since κ is a weighted average of κ1, κ2, and κ3, we must have κ=κ2. But then condition (iii) holds, and we have already shown that (iii)⇒(i). This completes the proof for condition (iv).Theorem6 shows that all weighted kappas for 3×3 tables are identical if we have the double inequality a1/b1=a2/b2=a3/b3. If this condition holds, the equalities (a1+a3)/(b1+b3)=a2/b2 and (a1+a2)/(b1+b2)=(a2+a3)/(b2+b3) also hold. Theorem 6 also shows that if any two special cases of the family λr are equal to a member of the family μs other than κℓ, then all weighted kappas coincide. Furthermore, if any two special cases of the family μs are identical to a member of the family λr other than κℓ, then all weighted kappas must be identical.


## 7. Discussion

Since it frequently happens that different versions of the weighted kappa are applied to the same contingency data, regardless of the scale type of the categories, it is useful to compare the various versions analytically. For rating scales with three categories, we may define seven special cases of weighted kappa. The seven weighted kappas belong to two different parameter families. Only the weighted kappa with linear weights belongs to both families. For both families, it was shown that there are only two possible orderings of its members (Theorems3 and 4). We conclude that with ordinal scales consisting of three categories, quadratically weighted kappa usually produces higher values than linearly weighted kappa, which in turn has higher values than unweighted kappa.Since there are only a few possible orderings of the weighted kappas, it appears that the kappas are measuring the same thing, but to a different extent. Various authors have presented magnitude values for evaluating the values of kappa statistics [36–38]. For example, an estimated value of 0.80 generally indicates good or excellent agreement. There is general consensus in the literature that uncritical use of these guidelines leads to questionable decisions in practice. If the weighted kappas are measuring the same thing, but some kappas produce substantially higher values than others, then the same guidelines cannot be applied to all weighted kappas. However, using the same guidelines for different kappas appears to be common practice. If one wants to work with magnitude guidelines, then it seems reasonable to use stricter criteria for the quadratically weighted kappa than for unweighted kappa, since the former statistic generally produces higher values.The quadratically and linearly weighted kappas were formulated for continuous-ordinal scale data. However, in practice, many scales are dichotomous ordinal (see, e.g., Anderson et al. [24] and Martin et al. [25]). In this case, the application of the weighted kappa proposed by Cicchetti [9] or the additively weighted kappa introduced in Warrens [31] is perhaps more appropriate. Unfortunately, Cicchetti’s weighted kappa has been largely ignored in the application of kappa statistics. In most applications, the quadratically weighted kappa is used [4, 5]. The observation that the quadratically weighted kappa tends to produce the highest value for many data may partly explain this popularity. As pointed out by one of the reviewers, to determine whether Cicchetti’s weighted kappa has real advantages, the various weighted kappas need to be compared on the quality and efficiency of prediction. This is a possible topic for future work.



---

*Source: 325831-2013-06-16.xml*