# Measurement of Interobserver Disagreement: Correction of Cohen’s Kappa for Negative Values

Authors: Tarald O. Kvålseth
Journal: Journal of Probability and Statistics
 (2015)

---

## Abstract

As measures of interobserver agreement for both nominal and ordinal categories, Cohen’s kappa coefficients appear to be the most widely used with simple and meaningful interpretations. However, for negative coefficient values when (the probability of) observed disagreement exceeds chance-expected disagreement, no fixed lower bounds exist for the kappa coefficients and their interpretations are no longer meaningful and may be entirely misleading. In this paper, alternative measures of disagreement (or negative agreement) are proposed as simple corrections or modifications of Cohen’s kappa coefficients. The new coefficients have a fixed lower bound of −1 that can be attained irrespective of the marginal distributions. A coefficient is formulated for the case when the classification categories are nominal and a weighted coefficient is proposed for ordinal categories. Besides coefficients for the overall disagreement across categories, disagreement coefficients for individual categories are presented. Statistical inference procedures are developed and numerical examples are provided.


---

## Body

## 1. Introduction

When two (or more) observers are independently classifying observations or items (objects) into the same set ofk mutually exclusive and exhaustive categories, it may be of interest to have a summary description of the extent to which the observers agreed in their classifications. The total probability (proportion) of agreement is one such obvious summary measure. However, since some agreement is to be expected purely by chance, Cohen [1] introduced thekappa coefficient of agreement as one that corrects for the chance-expected agreement. Cohen’s kappa has since become widely used in a variety of situations and discussed extensively in various textbooks (e.g., [2–5]) and a wide variety of journal publications (e.g., [6–10]).In order to define the kappa coefficient in terms of probabilities (proportions), letp
i
j be the probability that a random observation is assigned to category i by Observer 1 and to category j by Observer 2 for i
=
1
,
…
,
k and j
=
1
,
…
,
k. Furthermore, let p
i
+ denote the probability that a randomly chosen observation is assigned to category i by Observer 1 and p
+
j the probability that a randomly chosen observation is assigned to category j by Observer 2 (i
,
j
=
1
,
…
,
k). If these probabilities are represented in terms of a two-way contingency table with rows i
=
1
,
…
,
k and columns j
=
1
,
…
,
k, then p
i
j becomes the probability in cell (i
,
j) and {
p
i
+
} becomes the marginal row distribution and {
p
+
j
} becomes the marginal column distribution. With the row categories and the column categories being the same, ∑
i
=
1
k
p
i
i is the total probability of agreement between the two observers. Cohen [1] used the overall statistical independence as the condition for chance agreement and defined K as (1)
K
=
P
A
O
-
P
A
C
1
-
P
A
C
,
P
A
O
=
∑
i
=
1
k
p
i
i
,
P
A
C
=
∑
i
=
1
k
p
i
+
p
+
iwith P
A
O and P
A
C being the observed agreement probability and the chance-expected agreement probability, respectively. In terms of the observed and chance-expected disagreement probabilities P
D
O and P
D
C, K can alternatively be expressed as (2)
K
=
1
-
P
D
O
P
D
C
,
P
D
O
=
∑
i
=
1
,
i
≠
j
k
∑
j
=
1
,
i
≠
j
k
p
i
j
,
P
D
C
=
∑
i
=
1
,
i
≠
j
k
∑
j
=
1
,
i
≠
j
k
p
i
+
p
+
j
.It is clear from (1)-(2) that K
=
1 if the interobserver agreement is perfect, that is, if P
A
O
=
1
(
P
D
O
=
0
), K
=
0 if P
A
O
=
P
A
C
(
P
D
O
=
P
D
C
), and K
<
0 if P
A
O
<
P
A
C
(
P
D
O
>
P
D
C
). The case of negative K-values will be discussed further in the next section.In addition to measuring the overall agreement between two observers, it may be of interest to assess their level of agreement for specific categories. Spitzer et al. [11] first proposed such a measure by collapsing the original k
×
k table into a 2 × 2 table, one such 2 × 2 table for each category i
=
1
,
…
,
k, and then computing K in (1)-(2) for each such 2 × 2 table (see also [2, Chapter 18]). As a simpler procedure yielding the same numerical results, Kvålseth [12] proposed the following form of kappa for the specific category i (i
=
1
,
…
,
k):(3)
K
i
=
p
i
i
-
p
i
+
p
+
i
p
-
i
-
p
i
+
p
+
i
,
p
-
i
=
p
i
+
+
p
+
i
2
(4)
=
1
-
∑
D
i
∑
D
i
p
i
j
∑
D
i
∑
D
i
p
i
+
p
+
j
,where ∑
D
i
∑
D
i denotes the summation over all disagreement cells for category i. With, say, k
=
3, D
2 consists of cells (
2
,
1
), (
2
,
3
), (
1
,
2
), and (
3
,
2
). For complete agreement with respect to category i, K
i
=
1 when p
i
i
=
p
i
+
=
p
+
i, K
i
=
0 for the independence p
i
i
=
p
i
+
p
+
i, and K
i
<
0 when observed disagreement exceeds chance disagreement.To account for the potential fact that some disagreements may be more serious than others, as when thek categories have a natural order, Cohen [13] and Cicchetti and Allison [14] independently introduced theweighted kappa
K
w, which can be expressed as (5)
K
w
=
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
-
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
1
-
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
(6)
=
1
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
j
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j
,where each weight w
i
j
∈
[
0,1
], with w
i
i
=
0 and v
i
j
=
1
-
w
i
j for all i and j and with the following logical weight choices (e.g., [2, page 609]):(7)
w
i
j
=
i
-
j
k
-
1
o
r
w
i
j
=
i
-
j
k
-
1
2
;
i
,
j
=
1
,
…
,
k
.For a specific category i, Kvålseth [15] proposed the following measure as an extension of (4):(8)
K
w
i
=
1
-
∑
D
i
∑
D
i
w
i
j
p
i
j
∑
D
i
∑
D
i
w
i
j
p
i
+
p
+
jwith D
i denoting the set of all disagreement cells for category i. The values of these weighted measures equal 1 for perfect agreement and 0 if observed agreement equals chance agreement, with negative values if observed agreement is less than chance agreement.The kappa coefficients in (1)–(8) may be appropriate measures of agreement when their values are nonnegative, but not when their values are negative as discussed in the next section. From a theoretical point of view at least, it is certainly troublesome that their negative values lack appropriate meaning and validity. This paper presents simple corrections or modifications of the kappa coefficients in (1)–(8) such that the negative values of the corrected coefficients provide appropriate representation of the extent to which the observers disagree. Statistical inference procedures for the new coefficients or measures are developed. Numerical examples are also given.


## 2. Comments on Kappa

One of the most appealing properties of kappa, and undoubtedly a reason for its popularity, is its simplicity and transparency. All the kappa coefficients in (1)–(8) have intuitively appealing and meaningful interpretations. In the case of K in (1)-(2), for example, it seems most meaningful to interpret any K-value in terms of (2) as the proportional difference between P
D
C and P
D
O, that is, the relative extent to which the observed disagreement probability P
D
O is less than the disagreement probability P
D
C attributable to chance. By comparison, the norming used in (1) is not unique, with any number of different potential denominators d such that (
P
A
O
-
P
A
C
)
/
d
≤
1 [16].Complete statistical independence, that is,p
i
j
=
p
i
+
p
+
j for i
,
j
=
1
,
…
,
k, is a sufficient, but not a necessary, condition for the kappa coefficients in (1)–(8) to take on value 0. In fact, for K
=
0 in (1) and k
>
2, it is not necessary that p
i
i
=
p
i
+
p
+
i for i
=
1
,
…
,
k. It is possible that K
=
0 even if p
i
j
≠
p
i
+
p
+
j for all i and j when k
>
2. As a simple example, consider(9)
p
11
=
1
6
,
p
12
=
1
6
,
p
13
=
0
,
p
21
=
0
,
p
22
=
1
12
,
p
23
=
3
12
=
1
4
,
p
31
=
1
6
,
p
32
=
1
12
,
p
33
=
1
12
,where all marginal probabilities equal 1/3. And p
i
j
≠
p
i
+
p
+
j for all i and j, but K
=
0. In this case, from (3), K
1
=
1
/
4, K
2
=
K
3
=
-
1
/
8, and, from (6) and (7), K
w
=
0.06 for w
i
j
=
i
-
j
/
(
k
-
1
).Note that the two expressions forK in (1)-(2) are weighted arithmetic means of the expressions for K
i in (3)-(4). Thus, from (1) and (3), for instance, it is seen that(10)
K
=
∑
i
=
1
k
u
i
K
i
,
u
i
=
p
-
i
-
p
i
+
p
+
i
1
-
∑
i
=
1
k
p
i
+
p
+
i
,
i
=
1
,
…
,
k
.Similarly, for the weighted measures in (6) and (8),(11)
K
w
=
∑
i
=
1
k
u
i
K
w
i
,
u
i
=
∑
D
i
∑
D
i
w
i
j
p
i
+
p
+
j
∑
i
=
1
k
∑
D
i
∑
D
i
w
i
j
p
i
+
p
+
j
,
i
=
1
,
…
,
k
.In order to show that the interobserver agreement for a specific categoryi can be determined directly from (3)-(4), without the need to collapse the original k
×
k table as suggested by Spitzer et al. [11], consider that the original k
×
k table with probability components p
i
i, p
i
+, and p
+
i for category i is collapsed into the following 2 × 2 table:(12)
p
11
2
=
p
i
i
,
p
12
2
=
p
i
+
-
p
i
i
,
p
21
2
=
p
+
i
-
p
i
i
,
p
22
2
=
1
-
p
i
+
-
p
+
i
+
p
i
i
.When (12) is substituted into (1), K
i in (3) results immediately. However, no such corresponding procedure applies to K
w in (6) and K
w
i in (8). Note that, for k
=
2, K
1
=
K
2
=
K and K
w
1
=
K
w
2
=
K
w.In spite of its wide appeal, kappa is not without some criticism or controversy, especially related to its dependence on the marginal distributions{
p
i
+
} and {
p
+
j
} (see, e.g., [4, pages 168–173]). The chance agreement (disagreement) for all the kappa coefficients in (1)–(8) is based on the marginal distributions. If those distributions are highly uneven (nonuniform) and nearly symmetric, the values of the kappa coefficients may become unreasonably small due to the relatively large chance agreements.A clear limitation of the kappa coefficients relates to situations when the values of those coefficients become negative and lack meaningful interpretations. This limitation has generally been ignored in published studies, partly perhaps because such studies using kappa have typically involved positive kappa values. Negative kappa values could, however, lead to incorrect interpretations, results, and conclusions. Also, if, for instance,K
>
0 in (1)-(2), it is possible that some K
i
<
0 in (3)-(4).For the overall kappa in (1)-(2), when P
A
O
<
P
A
C so that K
<
0, K has no reasonable meaning in terms of (1), but -
K does in terms of (2); that is, -
K is the relative extent to which P
D
O exceeds P
D
C. The same argument applies to K
i in (3)-(4). However, two serious limitations of all the kappa coefficients are that, for negative values, (a) the coefficients have no fixed lower bounds, making it impossible to appropriately assess the size or magnitude of coefficient values, and (b) the coefficients take on negative values that do not appear reasonable as discussed below.The minimum values-
P
A
C
/
(
1
-
P
A
C
) of K in (1) and -
p
i
+
p
+
i
/
(
p
-
i
-
p
i
+
p
+
i
) of K
i in (3) depend exclusively on the marginal distributions {
p
i
+
} and {
p
+
j
}. Values such as K
=
-
0.4 or K
i
=
-
0.2 are uninformative since they cannot be related to any fixed lower bounds on K or K
i such as -
1, irrespective of the marginal distributions. There is no basis for making any interpretation or statement such as K
=
-
0.5 indicating a “moderate,” “low,” or “high” level of disagreement between the two observers.There is also some confusion in the literature about the minimum value ofK, with some stating that the minimum value is -
∞ or -
P
A
O
/
(
1
-
P
A
O
) [5, page 4] and others stating that it is -
1 when p
i
+
=
p
+
j
=
1
/
k for all i and j [17, page 113]. Such statements are clearly incorrect. In fact, the minimum value K
=
-
P
A
C
/
(
1
-
P
A
C
) equals -
1 if, and only if, P
A
C
=
0.5. Similarly, the minimum value of K
i in (3) equals -
1 only when the harmonic mean 2
p
i
+
p
+
i
/
(
p
i
+
+
p
+
i
) of p
i
+ and p
+
i equals 0.5.What is needed are chance-corrected measures of disagreement, both weighted and unweighted, which have fixed lower bounds of-
1 and which are attainable irrespective of the marginal distributions. This requirement has also been clearly emphasized by others [18]. Such measures will be introduced in the next section as simple corrections or modifications of the existing kappa coefficients.


## 3. Proposed Kappa Coefficients of Disagreement

### 3.1. Overall Coefficients

WhenP
A
O
<
P
A
C and hence P
D
O
>
P
D
C, it seems most logical and intuitive to define negative overall kappa as(13)
K
-
=
-
P
D
O
-
P
D
C
1
-
P
D
C
=
-
1
-
P
A
O
P
A
C
,where P
A
O
,
P
A
C
,
P
D
O, and P
D
C are the probabilities defined in (1)-(2). Consequently,(14)
K
c
o
r
r
e
c
t
e
d
=
K
in  (
1
)-(
2
)
,
i
f
P
A
O
≥
P
A
C
,
K
-
i
n
13
,
i
f
P
A
O
≤
P
A
C
,where, of course, K
=
K
-
=
0 for P
A
O
=
P
A
C. Except for the minus sign, K
- in (13) follows from K in (1)-(2) by simply substituting disagreement probabilities for the corresponding agreement probabilities.The properties ofK
- can be summarized as follows:(P1)
K
- is well defined if at least two cells of the contingency table contain nonzero probabilities.
(P2)
K
-
∈
[
0
,
-
1
], with K
-
=
0, if observed agreement (disagreement) equals chance agreement (disagreement) (i.e., P
A
O
=
P
A
C or P
D
O
=
P
D
C) and K
-
=
-
1 if, and only if, P
A
O
=
0
(
P
D
O
=
1
).
(P3)
K
- can take on value -
1 for any marginal distributions {
p
i
+
} and {
p
+
j
}.
(P4)
K
- has a meaningful interpretation as the relative extent to which the observed agreement probability is less than that expected by chance alone.
(P5)
K
- takes on values that appear reasonable throughout its 0 to -
1 range.While Properties (P1)–(P4) are immediately apparent from the definition in (13), Property (P5) needs an explanation. This can most simply be done for the k
=
2 category case and without undue loss of generality since, for any data set with k
>
2, there exists an equivalent 2 × 2 table with the same K
--value. Therefore, one may consider a 2 × 2 table such as the one in Table 1 with the marginal probabilities r and 1
-
r (0
≤
r
≤
1). The first two entries in each cell correspond to the cases when K
-
=
-
1 and 0, respectively, while the third entry equals the weighted arithmetic mean of the other two entries with weights λ and 1
-
λ
(
0
≤
λ
≤
1
).Table 1
A 2 × 2 contingency table with marginal probabilitiesr and 1
-
r and with the entries in each cell as follows: first entry corresponds to P
A
D
=
0, second entry corresponds to P
A
O
=
P
A
C, and third entry is the weighted mean of the other two entries where 0
≤
λ
≤
1.
Observer 1
Observer 2
Total
Category 1
Category 2
Category 1
0
r
r
r
(
1
-
r
)
r
2
(
1
-
λ
)
r
(
1
-
r
)
λ
r
+
(
1
-
λ
)
r
2
Category 2
1
-
r
0
1
-
r
(
1
-
r
)
2
r
(
1
-
r
)
λ
(
1
-
r
)
+
(
1
-
λ
)
(
1
-
r
)
2
(
1
-
λ
)
r
(
1
-
r
)
Total
1
-
r
r
1.00In order for the values ofK
- to be considered reasonable throughout the [
-
1
,
0
]-interval, the only logical condition would clearly seem to be that the value of K
- for the weighted mean cell probabilities should equal the weighted mean value of K
- for the other cell probabilities with the same weights λ and 1
-
λ; that is, (15)
K
-
m
e
a
n
p
r
o
b
a
b
i
l
i
t
i
e
s
=
λ
K
-
=
-
1
+
1
-
λ
K
-
=
0
=
-
λ
.By substituting the expressions for the mean cell probabilities from Table 1 into K
- in (13), it is seen that K
- does meet the condition in (15), irrespective of the marginal probabilities r and 1
-
r. This assumes, of course, as with Cohen’s K, that chance agreement (disagreement) based on the marginal probabilities is reasonable.By contrast, substituting the mean probabilities from Table1 into K in (1)-(2) gives(16)
K
m
e
a
n
p
r
o
b
a
b
i
l
i
t
i
e
s
=
-
2
r
1
-
r
1
-
2
r
1
-
r
λshowing the strong dependence of K on the marginal probabilities. The parenthetical term in (16) equals 1 if r
=
0.5 and approaches 0 as the marginal distributions become highly uneven or nonuniform (i.e., as r approaches 0 or 1).WhenK
w
<
0 in (5)-(6) and hence ∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
<
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j and ∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
j
>
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j and with the sets of weights {
v
i
j
} and {
w
i
j
} as defined in (7), the following weighted negative kappa is proposed:(17)
K
w
-
=
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
j
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j
1
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j
=
-
1
-
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
jand hence(18)
K
w
c
o
r
r
e
c
t
e
d
=
K
w
in  (
5
)-(
6
)
,
i
f
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
≥
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
,
K
w
-
i
n
17
,
i
f
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
≤
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
.Except for the minus sign, K
w
- in (17) follows from (5)-(6) by simply substituting {
w
i
j
} for {
v
i
j
} in (5) and {
v
i
j
} for {
w
i
j
} in (6).K
w
- is well defined if at least two cells of the k
×
k table contain nonzero probabilities. It is also apparent from (17) that K
w
- takes on values between 0 and -
1, inclusive, with K
w
-
=
0 if p
i
j
=
p
i
+
p
+
j for all i and j (as a sufficient but not necessary condition). Also K
w
-
=
-
1 if, and only if, p
i
j = 0 for all i and j except for i
=
1 and j
=
k and i
=
k and j
=
1, that is, when the only nonzero probabilities occur in the corner cells (1
,
k) and (k
,
1) and the weights are of the type of form as in (7). These properties of K
w
- all appear to be reasonable.By contrast, ifp
1
k
≠
0, p
k
1
≠
0, and all other p
i
j = 0, K
w in (5)-(6) becomes K
w
=
-
2
p
1
k
p
k
1
/
(
1
-
2
p
1
k
p
k
1
), which equals -
1 only if p
1
k
=
p
k
1
=
0.5. Otherwise, the value of K
w increases as p
1
k and p
k
1 become increasingly different, approaching 0 as p
1
k
-
p
k
1 approaches 1. Such behavior of K
w
<
0 makes any reasonable interpretation of negative K
w-values impossible and meaningless.


### 3.2. Specific Category Coefficients

Just as theK and K
w coefficients are inappropriate for negative values, so is the category-specific coefficient K
i in (3)-(4) as pointed out in Section 2. Therefore, for K
i
<
0, another coefficient is needed that satisfies the reasonable requirements that its value equals 0 when p
i
i
=
p
i
+
p
+
i and equals -
1 when p
i
i
=
0. The following proposition seems most reasonable:(19)
K
i
c
o
r
r
e
c
t
e
d
=
K
i
in  (
3
)-(
4
)
,
i
f
p
i
i
≥
p
i
+
p
+
i
,
K
i
-
=
-
1
-
p
i
i
p
i
+
p
+
i
,
i
f
p
i
i
≤
p
i
+
p
+
i
.
K
i
- is well defined unless either p
i
+
=
0 or p
+
i
=
0 (and hence p
i
i
=
0). K
i
-
=
-
1 if, and only if, p
i
i
=
0 (and p
i
+
≠
0 and p
+
i
≠
0). For the weighted mean cell probabilities in Table 1, K
1
-
=
K
2
-
=
-
λ. Also, analogous to (10),(20)
K
-
=
∑
i
=
1
k
u
i
K
i
-
,
u
i
=
p
i
+
p
+
i
∑
i
=
1
k
p
i
+
p
+
iwith K
- defined in (13).In terms of weightsv
i
j
=
1
-
w
i
j, with the types of w
i
j as in (7), the proposed specific-category weighted kappa coefficient may be defined as (21)
K
w
i
c
o
r
r
e
c
t
e
d
=
K
w
i
i
n
8
,
i
f
K
w
i
≥
0
,
K
w
i
-
=
-
1
-
∑
h
=
1
k
v
h
i
p
h
i
+
∑
j
=
1
k
v
i
j
p
i
j
∑
h
=
1
k
v
h
i
p
h
+
p
+
i
+
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
,
i
f
K
w
i
≤
0
,where, as always, the first subscript refers to the table row and the second subscript to the table column. Note that the component for cell (
i
,
i
) appears twice in K
w
i
-. Note also that, analogous to (20), K
w
- in (17) is the weighted arithmetic mean of K
w
1
-
,
…
,
K
w
k
- in (21) with weights based on the denominator in (21) for i
=
1
,
…
,
k. It is apparent from (21) that, for the weights in (7) and with v
i
j
=
1
-
w
i
j, K
w
i
-
=
-
1 if p
i
+
=
p
+
i
=
0 for all i, but also K
w
1
-
=
K
w
k
-
=
-
1 if p
1
k and p
k
1 are the only nonzero probabilities in the table.


## 3.1. Overall Coefficients

WhenP
A
O
<
P
A
C and hence P
D
O
>
P
D
C, it seems most logical and intuitive to define negative overall kappa as(13)
K
-
=
-
P
D
O
-
P
D
C
1
-
P
D
C
=
-
1
-
P
A
O
P
A
C
,where P
A
O
,
P
A
C
,
P
D
O, and P
D
C are the probabilities defined in (1)-(2). Consequently,(14)
K
c
o
r
r
e
c
t
e
d
=
K
in  (
1
)-(
2
)
,
i
f
P
A
O
≥
P
A
C
,
K
-
i
n
13
,
i
f
P
A
O
≤
P
A
C
,where, of course, K
=
K
-
=
0 for P
A
O
=
P
A
C. Except for the minus sign, K
- in (13) follows from K in (1)-(2) by simply substituting disagreement probabilities for the corresponding agreement probabilities.The properties ofK
- can be summarized as follows:(P1)
K
- is well defined if at least two cells of the contingency table contain nonzero probabilities.
(P2)
K
-
∈
[
0
,
-
1
], with K
-
=
0, if observed agreement (disagreement) equals chance agreement (disagreement) (i.e., P
A
O
=
P
A
C or P
D
O
=
P
D
C) and K
-
=
-
1 if, and only if, P
A
O
=
0
(
P
D
O
=
1
).
(P3)
K
- can take on value -
1 for any marginal distributions {
p
i
+
} and {
p
+
j
}.
(P4)
K
- has a meaningful interpretation as the relative extent to which the observed agreement probability is less than that expected by chance alone.
(P5)
K
- takes on values that appear reasonable throughout its 0 to -
1 range.While Properties (P1)–(P4) are immediately apparent from the definition in (13), Property (P5) needs an explanation. This can most simply be done for the k
=
2 category case and without undue loss of generality since, for any data set with k
>
2, there exists an equivalent 2 × 2 table with the same K
--value. Therefore, one may consider a 2 × 2 table such as the one in Table 1 with the marginal probabilities r and 1
-
r (0
≤
r
≤
1). The first two entries in each cell correspond to the cases when K
-
=
-
1 and 0, respectively, while the third entry equals the weighted arithmetic mean of the other two entries with weights λ and 1
-
λ
(
0
≤
λ
≤
1
).Table 1
A 2 × 2 contingency table with marginal probabilitiesr and 1
-
r and with the entries in each cell as follows: first entry corresponds to P
A
D
=
0, second entry corresponds to P
A
O
=
P
A
C, and third entry is the weighted mean of the other two entries where 0
≤
λ
≤
1.
Observer 1
Observer 2
Total
Category 1
Category 2
Category 1
0
r
r
r
(
1
-
r
)
r
2
(
1
-
λ
)
r
(
1
-
r
)
λ
r
+
(
1
-
λ
)
r
2
Category 2
1
-
r
0
1
-
r
(
1
-
r
)
2
r
(
1
-
r
)
λ
(
1
-
r
)
+
(
1
-
λ
)
(
1
-
r
)
2
(
1
-
λ
)
r
(
1
-
r
)
Total
1
-
r
r
1.00In order for the values ofK
- to be considered reasonable throughout the [
-
1
,
0
]-interval, the only logical condition would clearly seem to be that the value of K
- for the weighted mean cell probabilities should equal the weighted mean value of K
- for the other cell probabilities with the same weights λ and 1
-
λ; that is, (15)
K
-
m
e
a
n
p
r
o
b
a
b
i
l
i
t
i
e
s
=
λ
K
-
=
-
1
+
1
-
λ
K
-
=
0
=
-
λ
.By substituting the expressions for the mean cell probabilities from Table 1 into K
- in (13), it is seen that K
- does meet the condition in (15), irrespective of the marginal probabilities r and 1
-
r. This assumes, of course, as with Cohen’s K, that chance agreement (disagreement) based on the marginal probabilities is reasonable.By contrast, substituting the mean probabilities from Table1 into K in (1)-(2) gives(16)
K
m
e
a
n
p
r
o
b
a
b
i
l
i
t
i
e
s
=
-
2
r
1
-
r
1
-
2
r
1
-
r
λshowing the strong dependence of K on the marginal probabilities. The parenthetical term in (16) equals 1 if r
=
0.5 and approaches 0 as the marginal distributions become highly uneven or nonuniform (i.e., as r approaches 0 or 1).WhenK
w
<
0 in (5)-(6) and hence ∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
<
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j and ∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
j
>
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j and with the sets of weights {
v
i
j
} and {
w
i
j
} as defined in (7), the following weighted negative kappa is proposed:(17)
K
w
-
=
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
j
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j
1
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j
=
-
1
-
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
jand hence(18)
K
w
c
o
r
r
e
c
t
e
d
=
K
w
in  (
5
)-(
6
)
,
i
f
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
≥
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
,
K
w
-
i
n
17
,
i
f
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
j
≤
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
.Except for the minus sign, K
w
- in (17) follows from (5)-(6) by simply substituting {
w
i
j
} for {
v
i
j
} in (5) and {
v
i
j
} for {
w
i
j
} in (6).K
w
- is well defined if at least two cells of the k
×
k table contain nonzero probabilities. It is also apparent from (17) that K
w
- takes on values between 0 and -
1, inclusive, with K
w
-
=
0 if p
i
j
=
p
i
+
p
+
j for all i and j (as a sufficient but not necessary condition). Also K
w
-
=
-
1 if, and only if, p
i
j = 0 for all i and j except for i
=
1 and j
=
k and i
=
k and j
=
1, that is, when the only nonzero probabilities occur in the corner cells (1
,
k) and (k
,
1) and the weights are of the type of form as in (7). These properties of K
w
- all appear to be reasonable.By contrast, ifp
1
k
≠
0, p
k
1
≠
0, and all other p
i
j = 0, K
w in (5)-(6) becomes K
w
=
-
2
p
1
k
p
k
1
/
(
1
-
2
p
1
k
p
k
1
), which equals -
1 only if p
1
k
=
p
k
1
=
0.5. Otherwise, the value of K
w increases as p
1
k and p
k
1 become increasingly different, approaching 0 as p
1
k
-
p
k
1 approaches 1. Such behavior of K
w
<
0 makes any reasonable interpretation of negative K
w-values impossible and meaningless.


## 3.2. Specific Category Coefficients

Just as theK and K
w coefficients are inappropriate for negative values, so is the category-specific coefficient K
i in (3)-(4) as pointed out in Section 2. Therefore, for K
i
<
0, another coefficient is needed that satisfies the reasonable requirements that its value equals 0 when p
i
i
=
p
i
+
p
+
i and equals -
1 when p
i
i
=
0. The following proposition seems most reasonable:(19)
K
i
c
o
r
r
e
c
t
e
d
=
K
i
in  (
3
)-(
4
)
,
i
f
p
i
i
≥
p
i
+
p
+
i
,
K
i
-
=
-
1
-
p
i
i
p
i
+
p
+
i
,
i
f
p
i
i
≤
p
i
+
p
+
i
.
K
i
- is well defined unless either p
i
+
=
0 or p
+
i
=
0 (and hence p
i
i
=
0). K
i
-
=
-
1 if, and only if, p
i
i
=
0 (and p
i
+
≠
0 and p
+
i
≠
0). For the weighted mean cell probabilities in Table 1, K
1
-
=
K
2
-
=
-
λ. Also, analogous to (10),(20)
K
-
=
∑
i
=
1
k
u
i
K
i
-
,
u
i
=
p
i
+
p
+
i
∑
i
=
1
k
p
i
+
p
+
iwith K
- defined in (13).In terms of weightsv
i
j
=
1
-
w
i
j, with the types of w
i
j as in (7), the proposed specific-category weighted kappa coefficient may be defined as (21)
K
w
i
c
o
r
r
e
c
t
e
d
=
K
w
i
i
n
8
,
i
f
K
w
i
≥
0
,
K
w
i
-
=
-
1
-
∑
h
=
1
k
v
h
i
p
h
i
+
∑
j
=
1
k
v
i
j
p
i
j
∑
h
=
1
k
v
h
i
p
h
+
p
+
i
+
∑
j
=
1
k
v
i
j
p
i
+
p
+
j
,
i
f
K
w
i
≤
0
,where, as always, the first subscript refers to the table row and the second subscript to the table column. Note that the component for cell (
i
,
i
) appears twice in K
w
i
-. Note also that, analogous to (20), K
w
- in (17) is the weighted arithmetic mean of K
w
1
-
,
…
,
K
w
k
- in (21) with weights based on the denominator in (21) for i
=
1
,
…
,
k. It is apparent from (21) that, for the weights in (7) and with v
i
j
=
1
-
w
i
j, K
w
i
-
=
-
1 if p
i
+
=
p
+
i
=
0 for all i, but also K
w
1
-
=
K
w
k
-
=
-
1 if p
1
k and p
k
1 are the only nonzero probabilities in the table.


## 4. Statistical Inferences

Consider now that the coefficients (measures) discussed above are all sample estimates (and estimators) based on the sample probabilitiesp
i
j
=
n
i
j
/
N (i
,
j
=
1
,
…
,
k) with frequencies (counts) n
i
j and sample size N
=
∑
i
=
1
k
∑
j
=
1
k
n
i
j. p
i
j’s are maximum likelihood estimates (and estimators) of the unknown population probabilities π
i
j
(
i
,
j
=
1
,
…
,
k
) on which the corresponding population coefficients are based such as the population coefficient K
w
π
i
j
=
1
-
∑
i
=
1
k
∑
j
=
1
k
w
i
j
π
i
j
/
∑
i
=
1
k
∑
j
=
1
k
w
i
j
π
i
+
π
+
j corresponding to the sample coefficient K
w in (6). It may then be of interest to make statistical inferences about the population coefficients corresponding to the sample coefficients discussed above.Such statistical inferences would probably be most meaningful in terms of the construction of confidence intervals for the overall kappa coefficients in (14) and (18). The inference procedure needs necessarily to be approximated for reasonably large sample size N and be based on thedelta method (e.g., [19, Chapter 14]) or resampling methods such as thebootstrap and thejackknife (e.g., [20, 21]). The delta method is chosen in this paper. By developing the procedure based on the K
w expression in (6), the procedures for K
w
- in (17), K
- in (13), and K in (1) follow as special cases by the appropriate selection of the set of weights w
i
j. Fleiss et al. [2] gave the estimated large sample variance of K
w based on the expression in (5) without presenting any intermediate steps. Instead, the expression in (6) will be used here as being more convenient and some of the important intermediate steps will be presented.Then, lettingK
w in (6) denote both the sample estimate and estimator of the corresponding population coefficient K
w
π
i
j (based on population probabilities π
i
j
,
π
i
+, and π
+
j for i
=
1
,
…
,
k and j
=
1
,
…
,
k), it follows from the delta method that, under multinomial sampling (when the k categories and the sample size N area priori fixed), the estimator K
w is approximately normally distributed with mean K
w
π
i
j and estimated variance Var
^
⁡
(
K
w
) if N is reasonably large.In order to derive the estimated variance ofK
w, express K
w
π
i
j as K
w
π
i
j
=
1
-
a
/
b
=
1
-
d and let a
i
j
′
,
b
i
j
′, and d
i
j
′ denote the partial derivatives of these quantities with respect to π
i
j, with π
i
j then being replaced with the estimated probabilities p
i
j for all i and j. Then, (22)
Var
^
⁡
K
w
=
Var
^
⁡
d
=
N
-
1
∑
i
=
1
k
∑
j
=
1
k
p
i
j
d
i
j
′
2
-
∑
i
=
1
k
∑
j
=
1
k
p
i
j
d
i
j
′
2
,where(23)
d
i
j
′
=
b
-
1
w
i
j
-
a
b
-
2
b
i
j
′for all i and j. It is found that(24)
b
i
j
′
=
∑
j
=
1
k
w
i
j
p
+
j
+
∑
i
=
1
k
w
i
j
p
i
+
=
w
-
i
+
+
w
-
+
jso that, from (23)-(24),(25)
d
i
j
′
=
b
-
1
w
i
j
-
a
b
-
2
w
-
i
+
+
w
-
+
j
=
b
-
1
w
i
j
-
1
-
K
w
w
-
i
+
+
w
-
+
jfrom which one gets(26)
∑
i
=
1
k
∑
j
=
1
k
p
i
j
d
i
j
′
=
-
a
b
-
1
=
-
1
-
K
w
.When (25) and (26) are substituted into (22), one obtains(27)
Var
^
⁡
K
w
=
N
-
1
b
-
2
∑
i
=
1
k
∑
j
=
1
k
p
i
j
w
i
j
-
1
-
K
w
w
-
i
+
+
w
-
+
j
2
-
1
-
K
w
2
,where b
=
∑
i
=
1
k
∑
j
=
1
k
w
i
j
p
i
+
p
+
j and w
-
i
+ and w
-
+
j are defined in (24). This variance formula, which gives the same numerical results as the formula given by Fleiss et al. [22], can then be used for interval estimation.By comparingK
w
- in (17) with the expression for K
w in (6), it follows from (27) that the estimated variance of K
w
- is given by (28)
Var
^
⁡
K
w
-
=
N
-
1
b
-
2
∑
i
=
1
k
∑
j
=
1
k
p
i
j
v
i
j
-
1
-
K
w
-
v
-
i
+
+
v
-
+
j
2
-
1
-
K
w
-
2
,where b
=
∑
i
=
1
k
∑
j
=
1
k
v
i
j
p
i
+
p
+
j is the denominator in (17) and (29)
v
-
i
+
=
∑
j
=
1
k
v
i
j
p
+
j
,
v
-
+
j
=
∑
i
=
1
k
v
i
j
p
i
+
.By settingw
i
j
=
1 for all i
≠
j and w
i
j
=
0 for all i
=
j, K
w in (6) reduces to K in (2) and, furthermore, w
-
i
+
=
1
-
p
+
i and w
-
+
j
=
1
-
p
j
+ in (24) so that, from (27), it is found that (30)
Var
^
⁡
K
=
N
-
1
P
D
C
-
2
P
D
O
1
-
P
D
O
+
1
-
K
2
∑
i
=
1
k
∑
j
=
1
k
p
i
j
2
-
p
+
i
-
p
j
+
2
-
2
1
-
K
∑
i
=
1
,
i
≠
j
k
∑
j
=
1
,
i
≠
j
k
p
i
j
2
-
p
+
i
-
p
j
+
,where P
D
O and P
D
C are defined in (2). Similarly, by setting v
i
j
=
1 for all i
=
j and v
i
j
=
0 for all i
≠
j, K
w
- in (17) reduces to K
- in (13) and, furthermore, v
-
i
+
=
p
+
i and v
-
+
j
=
p
j
+ in (29) so that, from (28), the following result is obtained:(31)
Var
^
⁡
K
-
=
N
-
1
P
A
C
-
2
P
A
O
1
-
P
A
O
+
P
A
O
P
A
C
2
∑
i
=
1
k
∑
j
=
1
k
p
i
j
p
+
i
+
p
j
+
2
-
2
P
A
O
P
A
C
∑
i
=
1
k
p
i
i
p
+
i
+
p
i
+
,where P
A
O and P
A
C are defined in (1). The expression in (30) is somewhat different from that given by Fleiss et al. [22], but they are found to give exactly the same numerical results.If it should be of interest to test the null hypothesis that the population equivalent to one of the new coefficients is equal to zero, then the same procedure as proposed by Fleiss et al. [22] for the case of Cohen’s K and K
w would involve replacing p
i
j with p
i
+
p
+
j for all i and j in the variance expressions in (28) or (30). However, a simpler method would be to use the chi-square goodness-of-fit statistics χ
2 or G
2 to test for independence (noting again that independence is a sufficient but not necessary condition for the coefficients to equal zero).


## 5. Numerical Examples

### 5.1. Example 1:K
-

Instead of one pair of observers assigning each ofN items (observations) to one of k categories, consider the statistically equivalent situation in which each of N pairs of observers assigns one item to one of k categories. For example, among N
=
100 randomly selected couples, each spouse answers a question with k
=
3 choice categories C
1
,
C
2, and C
3. The (fictitious) data are given in Table 2.Table 2
Results fromN = 100 couples answering a multiple-choice question with three choice categories (fictitious data).
Wives
Husbands
Total
C
1
C
2
C
3
C
1
0.04
0.35
0.21
0.60
C
2
0.22
0.02
0.01
0.25
C
3
0.08
0.01
0.06
0.15
Total
0.34
0.38
0.28
1.00WithP
A
O
=
0.12 and P
A
C
=
0.3410 in Table 2, it follows from (13) that K
-
=
-
0.65, indicating a substantial disagreement between husbands and wives. By contrast, the corresponding value of Cohen’s K in (1) is found to be K
=
-
0.34, which could have been interpreted as indicating a much lower level of disagreement. However, since K has no fixed lower bound as discussed above, any interpretation or conclusion based on K
=
-
0.34 would be invalid and misleading.The next question may then be, how do the disagreements on the individual categories contribute to the overall disagreement ofK
-
=
-
0.65? The answer from Table 2 and (19) is found to be K
1
-
=
-
0.80, K
2
-
=
-
0.79, and K
3
=
0.10. Therefore, the substantial overall disagreement K
-
=
-
0.65 is attributable to the high disagreement on each of the categories C
1 and C
2, whereas category C
3 involves a very low level of agreement. By comparison, the negative values from (3) or (4) would have been substantially different, with K
1
=
-
0.62 and K
2
=
-
0.34.In order to construct a confidence interval for the population equivalentK
-
(
{
π
i
j
}
) of K
- based on the data in Table 2, it is found from (31) that Var
^
⁡
(
K
-
)
=
0.0115. Then, since the estimator K
-, with the sample estimate of -
0.65, is approximately normally distributed with mean K
-
(
{
π
i
j
}
) and estimated variance of 0.0115, an approximate 95% confidence interval for K
-
(
{
π
i
j
}
) becomes -
0.65
±
1.96
0.0115 or [
-
0.86
,
-
0.44
].


### 5.2. Example 2:K
w
-

Consider now that the categories in Table2 are ordinal so that the weighted kappa coefficients would be appropriate. Then, with the weights w
i
j
=
|
i
-
j
|
/
(
k
-
1
) in (7) and v
i
j
=
1
-
w
i
j for all i and j, it is found from Table 2 that ∑
i
=
1
3
∑
j
=
1
3
v
i
j
p
i
j
=
0.4150 and ∑
i
=
1
3
∑
j
=
1
3
v
i
j
p
i
+
p
+
j
=
0.5610 so that, from (17)-(18), K
w
-
=
-
0.2602
=
-
0.26. This weighted disagreement value differs considerably from the above K
-
=
-
0.65 value when the three categories are considered to be nominal.In terms of the disagreements for the individual categories, it is found from (21) and Table 2 that K
w
1
-
=
-
0.35, K
w
2
-
=
-
0.18, and K
w
3
-
=
-
0.12. Again, these results differ considerably from those in the nominal case considered in Example 1. Note that the arithmetic mean -
0.22 of K
w
i
-’s does not differ greatly from the overall K
w
-
=
-
0.26.An interval estimate for the population measureK
w
-
(
{
π
i
j
}
) can be derived from (28) by first computing v
-
i
+ and v
-
+
j for each of i and j from (29) and Table 2, giving v
-
i
+
=
0.5300,0.6900, and 0.4700 for i
=
1
,
2, and 3, respectively, and v
-
+
j
=
0.7250,0.6250, and 0.2750 for j
=
1
,
2, and 3, respectively. Then, from (28), with b
=
0.5610 and K
w
-
=
-
0.2602, it is found that Var
^
⁡
(
K
w
-
)
=
0.0028. Consequently, a 95% confidence interval for K
w
-
(
{
π
i
j
}
) is given by -
0.2602
±
1.96
0.0028 or [
-
0.36
,
-
0.16
].


### 5.3. Logistic Transformation

Instead of making statistical inferences about the kappa coefficients directly, as done above, it is likely advantageous to do so indirectly via the logistic transformation. Therefore, in the case ofK
- in (13), consider the following logistic transformation of 1
+
K
- and its inverse:(32)
L
=
log
⁡
1
+
K
-
-
K
-
,
K
-
=
-
1
1
+
e
L
.Since the derivative d
L
/
d
K
-
=
-
1
/
K
-
(
1
+
K
-
), the estimated variance of L becomes(33)
Var
^
⁡
L
=
1
K
-
1
+
K
-
2
Var
^
⁡
K
-
,where Var
^
⁡
(
K
-
) is given in (31). An approximate confidence interval for the population equivalent of L can then be constructed based on (33), with the corresponding confidence interval for K
-
(
{
π
i
j
}
) resulting from the inverse transform in (32).In the case ofK
w
- in (17), K
- in (32)-(33) is simply replaced with K
w
-. For K and K
w in (1) and (5)-(6), the transformation becomes log
⁡
K
/
(
1
-
K
) and log
⁡
K
w
/
(
1
-
K
w
). With such transformations, the lower end of a confidence interval for K
- or K
w
- cannot be less than -
1 and the upper end of a confidence interval K or K
w cannot exceed 1. Most importantly, the normal distribution approximation is likely to be improved with the above logistic transforms. Unless the sample size N is very large, the distributions of the kappa coefficients are likely to be skewed, especially when a coefficient is near -
1 or 1. For instance, when, say, the population coefficient K
-
π
i
j
=
-
0.9, the estimator K
- cannot be much smaller than K
-
(
{
π
i
j
}
), but it could be much larger with nonnegligible probability. The logistic transformation to the (
-
∞
,
∞
)-interval tends to correct for such skewness and provide for a more rapid convergence to normality.In Example1, with K
-
=
-
0.6481 and Var
^
⁡
(
K
-
)
=
0.0115, it follows from (32)-(33) that L
=
-
0.6107 and Var
^
⁡
(
L
)
=
0.2211. An approximate 95% confidence interval for the population equivalent of L is then given by -
0.6107
±
1.96
0.2211 or [
-
1.5323
,
0.3109
]. Then, from the inverse transform in (32), it follows that an approximate 95% confidence interval for K
-
(
{
π
i
j
}
) has the limits -
1
/
(
1
+
e
-
1.5323
)
=
-
0.82 and -
1
/
(
1
+
e
0.3109
)
=
-
0.42, that is, the interval [
-
0.82
,
-
0.42
]. This confidence interval is slightly shorter than the interval [
-
0.86
,
-
0.44
] determined above. Similarly, with K
w
- substituted for K
- in (32)-(33) and with K
w
-
=
-
0.2602 and Var
^
⁡
(
K
-
)
=
0.0028 from Example 2, it is found that an approximate 95% confidence interval for the population coefficient K
w
-
(
{
π
i
j
}
) is [
-
0.38
,
-
0.17
]. This interval differs little from the interval [
-
0.36
,
-
0.16
] determined above when applying the inference procedure directly to K
w
-.


## 5.1. Example 1:K
-

Instead of one pair of observers assigning each ofN items (observations) to one of k categories, consider the statistically equivalent situation in which each of N pairs of observers assigns one item to one of k categories. For example, among N
=
100 randomly selected couples, each spouse answers a question with k
=
3 choice categories C
1
,
C
2, and C
3. The (fictitious) data are given in Table 2.Table 2
Results fromN = 100 couples answering a multiple-choice question with three choice categories (fictitious data).
Wives
Husbands
Total
C
1
C
2
C
3
C
1
0.04
0.35
0.21
0.60
C
2
0.22
0.02
0.01
0.25
C
3
0.08
0.01
0.06
0.15
Total
0.34
0.38
0.28
1.00WithP
A
O
=
0.12 and P
A
C
=
0.3410 in Table 2, it follows from (13) that K
-
=
-
0.65, indicating a substantial disagreement between husbands and wives. By contrast, the corresponding value of Cohen’s K in (1) is found to be K
=
-
0.34, which could have been interpreted as indicating a much lower level of disagreement. However, since K has no fixed lower bound as discussed above, any interpretation or conclusion based on K
=
-
0.34 would be invalid and misleading.The next question may then be, how do the disagreements on the individual categories contribute to the overall disagreement ofK
-
=
-
0.65? The answer from Table 2 and (19) is found to be K
1
-
=
-
0.80, K
2
-
=
-
0.79, and K
3
=
0.10. Therefore, the substantial overall disagreement K
-
=
-
0.65 is attributable to the high disagreement on each of the categories C
1 and C
2, whereas category C
3 involves a very low level of agreement. By comparison, the negative values from (3) or (4) would have been substantially different, with K
1
=
-
0.62 and K
2
=
-
0.34.In order to construct a confidence interval for the population equivalentK
-
(
{
π
i
j
}
) of K
- based on the data in Table 2, it is found from (31) that Var
^
⁡
(
K
-
)
=
0.0115. Then, since the estimator K
-, with the sample estimate of -
0.65, is approximately normally distributed with mean K
-
(
{
π
i
j
}
) and estimated variance of 0.0115, an approximate 95% confidence interval for K
-
(
{
π
i
j
}
) becomes -
0.65
±
1.96
0.0115 or [
-
0.86
,
-
0.44
].


## 5.2. Example 2:K
w
-

Consider now that the categories in Table2 are ordinal so that the weighted kappa coefficients would be appropriate. Then, with the weights w
i
j
=
|
i
-
j
|
/
(
k
-
1
) in (7) and v
i
j
=
1
-
w
i
j for all i and j, it is found from Table 2 that ∑
i
=
1
3
∑
j
=
1
3
v
i
j
p
i
j
=
0.4150 and ∑
i
=
1
3
∑
j
=
1
3
v
i
j
p
i
+
p
+
j
=
0.5610 so that, from (17)-(18), K
w
-
=
-
0.2602
=
-
0.26. This weighted disagreement value differs considerably from the above K
-
=
-
0.65 value when the three categories are considered to be nominal.In terms of the disagreements for the individual categories, it is found from (21) and Table 2 that K
w
1
-
=
-
0.35, K
w
2
-
=
-
0.18, and K
w
3
-
=
-
0.12. Again, these results differ considerably from those in the nominal case considered in Example 1. Note that the arithmetic mean -
0.22 of K
w
i
-’s does not differ greatly from the overall K
w
-
=
-
0.26.An interval estimate for the population measureK
w
-
(
{
π
i
j
}
) can be derived from (28) by first computing v
-
i
+ and v
-
+
j for each of i and j from (29) and Table 2, giving v
-
i
+
=
0.5300,0.6900, and 0.4700 for i
=
1
,
2, and 3, respectively, and v
-
+
j
=
0.7250,0.6250, and 0.2750 for j
=
1
,
2, and 3, respectively. Then, from (28), with b
=
0.5610 and K
w
-
=
-
0.2602, it is found that Var
^
⁡
(
K
w
-
)
=
0.0028. Consequently, a 95% confidence interval for K
w
-
(
{
π
i
j
}
) is given by -
0.2602
±
1.96
0.0028 or [
-
0.36
,
-
0.16
].


## 5.3. Logistic Transformation

Instead of making statistical inferences about the kappa coefficients directly, as done above, it is likely advantageous to do so indirectly via the logistic transformation. Therefore, in the case ofK
- in (13), consider the following logistic transformation of 1
+
K
- and its inverse:(32)
L
=
log
⁡
1
+
K
-
-
K
-
,
K
-
=
-
1
1
+
e
L
.Since the derivative d
L
/
d
K
-
=
-
1
/
K
-
(
1
+
K
-
), the estimated variance of L becomes(33)
Var
^
⁡
L
=
1
K
-
1
+
K
-
2
Var
^
⁡
K
-
,where Var
^
⁡
(
K
-
) is given in (31). An approximate confidence interval for the population equivalent of L can then be constructed based on (33), with the corresponding confidence interval for K
-
(
{
π
i
j
}
) resulting from the inverse transform in (32).In the case ofK
w
- in (17), K
- in (32)-(33) is simply replaced with K
w
-. For K and K
w in (1) and (5)-(6), the transformation becomes log
⁡
K
/
(
1
-
K
) and log
⁡
K
w
/
(
1
-
K
w
). With such transformations, the lower end of a confidence interval for K
- or K
w
- cannot be less than -
1 and the upper end of a confidence interval K or K
w cannot exceed 1. Most importantly, the normal distribution approximation is likely to be improved with the above logistic transforms. Unless the sample size N is very large, the distributions of the kappa coefficients are likely to be skewed, especially when a coefficient is near -
1 or 1. For instance, when, say, the population coefficient K
-
π
i
j
=
-
0.9, the estimator K
- cannot be much smaller than K
-
(
{
π
i
j
}
), but it could be much larger with nonnegligible probability. The logistic transformation to the (
-
∞
,
∞
)-interval tends to correct for such skewness and provide for a more rapid convergence to normality.In Example1, with K
-
=
-
0.6481 and Var
^
⁡
(
K
-
)
=
0.0115, it follows from (32)-(33) that L
=
-
0.6107 and Var
^
⁡
(
L
)
=
0.2211. An approximate 95% confidence interval for the population equivalent of L is then given by -
0.6107
±
1.96
0.2211 or [
-
1.5323
,
0.3109
]. Then, from the inverse transform in (32), it follows that an approximate 95% confidence interval for K
-
(
{
π
i
j
}
) has the limits -
1
/
(
1
+
e
-
1.5323
)
=
-
0.82 and -
1
/
(
1
+
e
0.3109
)
=
-
0.42, that is, the interval [
-
0.82
,
-
0.42
]. This confidence interval is slightly shorter than the interval [
-
0.86
,
-
0.44
] determined above. Similarly, with K
w
- substituted for K
- in (32)-(33) and with K
w
-
=
-
0.2602 and Var
^
⁡
(
K
-
)
=
0.0028 from Example 2, it is found that an approximate 95% confidence interval for the population coefficient K
w
-
(
{
π
i
j
}
) is [
-
0.38
,
-
0.17
]. This interval differs little from the interval [
-
0.36
,
-
0.16
] determined above when applying the inference procedure directly to K
w
-.


## 6. Conclusion

If Cohen’s kappa is accepted as an appropriate measure of interobserver agreement, as many do judging by its widespread use, then the corrections proposed here for negative kappa values should be equally acceptable. Of course, since the chance-expected disagreement (or agreement) terms in the new coefficients also depend exclusively on the marginal distributions, the criticism by some that Cohen’s coefficients depend too much on the marginal distributions would similarly apply to the new coefficients. Such concern is particularly important in cases of highly uneven (nonuniform or “skewed”) marginal distributions. If, however, those distributions are fairly even (uniform), Cohen’s kappa and hence the measures proposed in this paper for interobserver disagreement (negative agreement) would seem to be reasonably acceptable agreement-disagreement measures.



---

*Source: 751803-2015-09-30.xml*