# Fixed-Effects Modeling of Cohen's Weighted Kappa for Bivariate Multinomial Data: A Perspective

Authors: Jingyun Yang; Vernon M. Chinchilli
Journal: Journal of Probability and Statistics
 (2011)

---

## Abstract

Cohen's kappa and weighted kappa statistics are the conventional methods used frequently in measuring agreement for categorical responses. In this paper, through the perspective of a generalized inverse, we propose an alternative general framework of the fixed-effects modeling of Cohen's weighted kappa, proposed by Yang and Chinchilli (2011). Properties of the proposed method are provided. Small sample performance is investigated through bootstrap simulation studies, which demonstrate good performance of the proposed method. When there are only two categories, the proposed method reduces to Cohen's kappa.


---

## Body

## 1. Introduction

Measurement of agreement is used widely in diverse areas of scientific research to assess the reproducibility of a new assay, instrument, or method, the acceptability of a new or generic process and methodology, as well as in method comparison. Examples include the agreement when two or more methods or raters simultaneously assess a response [1, 2] or when one rater makes the same assessment at two times [3], the agreement of a newly developed method with a gold standard method [4], and the agreement of observed values with predicted values [5].Traditionally, kappa [6] is used for measurement of agreement for categorical responses and weighted kappa [7] for ordinal responses. The concordance correlation coefficient [8, 9] is often used when the responses are continuous [10, 11]. In this paper, we focus on the measurement of agreement for categorical responses, that is, when the responses are either nominal or ordinal.Suppose there is a bivariate response(X,Y), where each of X and Y yields a categorical response. For convenience, the categories are denoted as 0,1,2,…,k. Suppose the bivariate distribution is defined as in Table 1.Table 1
The bivariate probability table.
X∖Y012⋯i⋯j⋯kPX(x)0p00p01p02⋯p0i⋯p0j⋯p0kp0·1p10p11p12⋯p1i⋯p1j⋯p1kp1·2p20p21p22⋯p2i⋯p2j⋯p2kp2·⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮ipi0pi1pi2⋯pii⋯pij⋯pikpi·⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮jpj0pj1pj2⋯pji⋯pjj⋯pjkpj·⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮⋮kpk0pk1pk2⋯pki⋯pkj⋯pkkpk·PY(y)p·0p·1p·2⋯p·i⋯p·j⋯p·k1Note: there arek+1 categories in the response. PX(x) and PY(y) denote the marginal probabilities.Cohen [6] proposed a coefficient of agreement, called the kappa coefficient, for nominal scales of responses, which is defined by(1.1)κ=po-pc1-pc,
where po is the observed proportion of agreement and pc is the proportion of agreement expected by chance.To deal with ordinal responses, Cohen [7] proposed a weighted version of the kappa statistic:(1.2)κw=1-qo′qc′=1-∑vijpoij∑vijpcij,
with (1.3)qo′=∑vijpoijvmax⁡,qc′=∑vijpcijvmax⁡,
where vij is the nonnegative weight assigned to the disagreement for the cell ij, vmax is the weight assigned to the maximum disagreement, poij is the proportion of joint judgement in the ij cell, and pcij is the proportion expected by chance in the same cell.Despite the fact that they may fail to work well under certain situations [12, 13], Cohen’s kappa and weighted kappa coefficients have been widely used in various areas as a measure of agreement for categorical responses, partly due to their ease in calculation; see, for example, Justice et al. [14], Kerosuo and Ørstavik [15], Landers et al. [16], and Suebnukarn et al. [17]. In the meanwhile, researchers have been exploring alternative methods for measuring agreement for categorical data; see Agresti [18], Barlow [19], Broemeling [20], Carrasco and Jover [21], Dou et al. [22], Fanshawe et al. [23], Graham and Jackson [24], King and Chinchilli [25], Kraemer [26], Landis and Koch [27], Laurent [28], Lin et al. [29], and Svensson [30], to name only a few. It is worth noting that, for ordinal or binary data, the concordance correlation coefficient (CCC) [8] based on the squared function of distance reduces to Cohen’s weighted kappa when the Fleiss-Cohen weight function is used and the CCC based on the absolute value function of distance to the power δ=1 is equivalent to Cohen’s weighted kappa if Cicchetti-Allison weight function is used [25]. In addition, variance of the CCC is identical to the one given by Fleiss et al. [31] if the Fleiss-Cohen weight function and the GEE approach are used [29].Yang and Chinchilli developed a fixed-effects modeling of kappa and weighted kappa [32, 33]. Their simulation studies and illustrative examples show good performance of the proposed methods. In this paper, we propose another version of fixed-effects modeling of Cohen’s kappa and weighted kappa through a perspective of a generalized inverse. We want to stress that this paper is not to demonstrate the advantage of the proposed method in constructing an agreement coefficient. Rather, it provides an alternative form of a general framework and direction under which new and better matrix functions could be explored.In Section2 we provide the derivation and properties of the proposed method. In Section 3, we compare the small sample performance of the proposed method with that of three other agreement coefficients through bootstrap simulation studies. Examples are given in Section 4 followed by a conclusion.


## 2. The Generalized Inverse Version of Fixed-Effects Modeling of Kappa and Weighted Kappa

### 2.1. Derivation

Using similar notation as in Yang and Chinchilli [32], we let X and Y be categorical variables, with the categories being designated as 0,1,2,…,k, as displayed in Table 1. Let pij=Pr(X=i,Y=j), i,j=0,1,2,…,k, denote the bivariate probability, and let pi·=Pr(X=i) and p·j=Pr(Y=j) denote the marginal probabilities. Let I(·) denote the indicator function, and define the (k+1)×1 vectors as (2.1)ZX=[I(X=0)I(X=1)I(X=2)⋮I(X=k)],ZY=[I(Y=0)I(Y=1)I(Y=2)⋮I(Y=k)].Define the(k+1)×(k+1) matrices PD and PI as(2.2)PD=E[(ZX-ZY)(ZX-ZY)T],PI=EInd[(ZX-ZY)(ZX-ZY)T],
where EInd[·] denotes expectation under the assumption that X and Y are independent. It easily can be verified that PD and PI are nonnegative definite matrices, denoted by PD≥0 and PI≥0.Yang and Chinchilli [32] show that(2.3)PI-PD=[2(p00-p0⋅p⋅0)(p01-p0⋅p⋅1)+(p10-p1⋅p⋅0)⋯(p0k-p0⋅p⋅k)+(pk0-pk⋅p⋅0)(p01-p0⋅p⋅1)+(p10-p1⋅p⋅0)2(p11-p1⋅p⋅1)⋯(p1k-p1⋅p⋅k)+(pk1-pk⋅p⋅1)⋮⋮⋱⋮(p0k-p0⋅p⋅k)+(pk0-pk⋅p⋅0)(p1k-p1⋅p⋅k)+(pk1-pk⋅p⋅1)⋯2(pkk-pk⋅p⋅k)].PI and PD are (k+1)×(k+1) matrices of rank k because PI·1=0 and PD·1=0, where 1 is a (k+1)×1 vector of unit values. Thus, PI·1=0·1, indicating that 0 is an eigenvalue for PI with corresponding (standardized) eigenvector e0=1/k+1. The same is true for PD as well. Let EI=[e0e1⋯ek] denote the (k+1)×(k+1) matrix of eigenvectors for PI. By definition, EI is orthogonal, so EI(EI)T=(EI)TEI=I, which yields that 1Te1=0,…,1Tek=0. Because PI is not of full rank, its inverse does not exist, but it does have a Moore-Penrose generalized inverse [34, 35], denoted by (PI)+. If ΛI=Diag(0,λ1,…,λk) denotes the (k+1)×(k+1) diagonal matrix of eigenvalues for PI, then the Moore-Penrose generalized inverse of ΛI is (ΛI)+=Diag(0,(λ1)-1,…,(λk)-1) and the Moore-Penrose generalized inverse of PI is (PI)+=EI(ΛI)+(EI)T.It easily can be verified that-PI≤PI-PD≤PI, which implies that(2.4)-((PI)+)1/2PI((PI)+)1/2≤((PI)+)1/2PI((PI)+)1/2-((PI)+)1/2PD((PI)+)1/2≤((PI)+)1/2PI((PI)+)1/2,
where ((PI)+)1/2=EI((ΛI)+)1/2(EI)T.Since((PI)+)1/2PI((PI)+)1/2=I-1·1T/(k+1), (2.4) can be rewritten as (2.5)-{I-1⋅1T/(k+1)}≤{I-1⋅1T/(k+1)}-((PI)+)1/2PD((PI)+)1/2≤{I-1⋅1T/(k+1)}.Supposeg(A) is a function of the symmetric matrix A and g(·) satisfies the following two definitions [32].Definition 2.1.
Supposeg(·) is a ℝ(k+1)×(k+1)↦ℝ function. g(·) is said to be nondecreasing if for all nonnegative definite matrices A(k+1)×(k+1), B(k+1)×(k+1), where A≤B, one has g(A)≤g(B).Definition 2.2.
Supposeg(·) is a ℝ(k+1)×(k+1)↦ℝ function. g(·) is said to be a scale-equivariant function [36] if g(CA)=cg(A) for all matrices and constant c.Then a class of agreement coefficients is given by(2.6)κg*=1-g{(PI+)1/2PD(PI+)1/2}g{(I-(1/(k+1))1⋅1T)}.Ifg(A)=tr(A), then(2.7)κtr⁡*=1-1ktr⁡(PD(PI)+).If we setg(A)=tr(W*A), then(2.8)κtr⁡*(W)=1-tr⁡(W*PD*(PI)+)tr⁡(W)-(1/(k+1))∑i=0k∑j=0kWij,
where W is a nonnegative definite (k+1)×(k+1) symmetric matrix of agreement weights with wij=1 for i=j=0,1,2,…,k and 0≤wij<1 for i≠j=0,1,2,…,k. Two frequently used weighting schemes are the Cicchetti-Allison weights [37](2.9)wij=1-|i-j|k,fori,j=0,1,2,…,k,
and the Fleiss-Cohen weights [38](2.10)wij=1-(i-j)2k2,fori,j=0,1,2,…,k.We can also useg(A)=λA1, the largest eigenvalue of A as another function for g(·), which leads to a new agreement coefficient κle*(W).Therefore, including the coefficients that we propose in this paper, there are four novel coefficients to assess agreement for categorical data:(1)κtr(W), (2)κle(W), (3)κtr*(W), and (4)κle*(W). In practice, all these indices can be estimated using their sample counterparts. In this paper, a “∧” is used to denote the estimate of an index.


### 2.2. Properties

In general, ifg(·) satisfies the properties in the two aforementioned definitions, then κg*(W) has the following properties:(1)
-1≤κg*(W)≤1,(2)
κg*(W)=1 if X=Y with probability one, that is, pij=0 for all i≠j,(3)
κg*(W)=-1 if and only if pii=0 for each i=0,1,2,…,k and pij=pji=0.5 for one choice of i≠j, excluding the set of degenerate cases,(4)
κg*(W)=0 if X and Y are independent.Proof.
Without loss of generality, we setW=I. The proofs for other weight matrices W are slightly more complex.(1)
From the previous proof, we know that(2.11)0≤(PI+)1/2PD(PI+)1/2≤2(I-1k+11⋅1T).
Since g(·) is a nondecreasing and scale-equivariant function, we have
(2.12)0≤g{(PI+)1/2PD(PI+)1/2}≤2g{(I-1k+11⋅1T)}.
It is then straightforward to see that -1≤κg*≤1.(2)
IfX=Y with probability one, that is, pij=0 for all i≠j, then PD=0 [32]. It is then straightforward to see that κg*=0.(3)
Excluding the set of degenerate cases, ifpii=0 for each i=0,1,2,…,k and pij=pji=0.5 for one choice of i≠j, then PI-PD=-PI⇒PI=(1/2)PD [32]. Hence, (PI)+=((1/2)PD)+=2(PD)+ and ((PI)+)1/2=2((PD)+)1/2. Therefore, (PI+)1/2PD(PI+)1/2=2(PD+)1/2PD(PD+)1/2 and (PI+)1/2PI(PI+)1/2=(PD+)1/2PD(PD+)1/2. Hence, κg*=-1.(4)
IfX and Y are independent, then PI=PD [32]. It is straightforward to see that κg*=0.Lemma 2.3.
Whenk=1, that is, when there are only two categories, the four coefficients are equivalent and they all reduce to Cohen’s kappa.For proof of the lemma, please see the appendix.


## 2.1. Derivation

Using similar notation as in Yang and Chinchilli [32], we let X and Y be categorical variables, with the categories being designated as 0,1,2,…,k, as displayed in Table 1. Let pij=Pr(X=i,Y=j), i,j=0,1,2,…,k, denote the bivariate probability, and let pi·=Pr(X=i) and p·j=Pr(Y=j) denote the marginal probabilities. Let I(·) denote the indicator function, and define the (k+1)×1 vectors as (2.1)ZX=[I(X=0)I(X=1)I(X=2)⋮I(X=k)],ZY=[I(Y=0)I(Y=1)I(Y=2)⋮I(Y=k)].Define the(k+1)×(k+1) matrices PD and PI as(2.2)PD=E[(ZX-ZY)(ZX-ZY)T],PI=EInd[(ZX-ZY)(ZX-ZY)T],
where EInd[·] denotes expectation under the assumption that X and Y are independent. It easily can be verified that PD and PI are nonnegative definite matrices, denoted by PD≥0 and PI≥0.Yang and Chinchilli [32] show that(2.3)PI-PD=[2(p00-p0⋅p⋅0)(p01-p0⋅p⋅1)+(p10-p1⋅p⋅0)⋯(p0k-p0⋅p⋅k)+(pk0-pk⋅p⋅0)(p01-p0⋅p⋅1)+(p10-p1⋅p⋅0)2(p11-p1⋅p⋅1)⋯(p1k-p1⋅p⋅k)+(pk1-pk⋅p⋅1)⋮⋮⋱⋮(p0k-p0⋅p⋅k)+(pk0-pk⋅p⋅0)(p1k-p1⋅p⋅k)+(pk1-pk⋅p⋅1)⋯2(pkk-pk⋅p⋅k)].PI and PD are (k+1)×(k+1) matrices of rank k because PI·1=0 and PD·1=0, where 1 is a (k+1)×1 vector of unit values. Thus, PI·1=0·1, indicating that 0 is an eigenvalue for PI with corresponding (standardized) eigenvector e0=1/k+1. The same is true for PD as well. Let EI=[e0e1⋯ek] denote the (k+1)×(k+1) matrix of eigenvectors for PI. By definition, EI is orthogonal, so EI(EI)T=(EI)TEI=I, which yields that 1Te1=0,…,1Tek=0. Because PI is not of full rank, its inverse does not exist, but it does have a Moore-Penrose generalized inverse [34, 35], denoted by (PI)+. If ΛI=Diag(0,λ1,…,λk) denotes the (k+1)×(k+1) diagonal matrix of eigenvalues for PI, then the Moore-Penrose generalized inverse of ΛI is (ΛI)+=Diag(0,(λ1)-1,…,(λk)-1) and the Moore-Penrose generalized inverse of PI is (PI)+=EI(ΛI)+(EI)T.It easily can be verified that-PI≤PI-PD≤PI, which implies that(2.4)-((PI)+)1/2PI((PI)+)1/2≤((PI)+)1/2PI((PI)+)1/2-((PI)+)1/2PD((PI)+)1/2≤((PI)+)1/2PI((PI)+)1/2,
where ((PI)+)1/2=EI((ΛI)+)1/2(EI)T.Since((PI)+)1/2PI((PI)+)1/2=I-1·1T/(k+1), (2.4) can be rewritten as (2.5)-{I-1⋅1T/(k+1)}≤{I-1⋅1T/(k+1)}-((PI)+)1/2PD((PI)+)1/2≤{I-1⋅1T/(k+1)}.Supposeg(A) is a function of the symmetric matrix A and g(·) satisfies the following two definitions [32].Definition 2.1.
Supposeg(·) is a ℝ(k+1)×(k+1)↦ℝ function. g(·) is said to be nondecreasing if for all nonnegative definite matrices A(k+1)×(k+1), B(k+1)×(k+1), where A≤B, one has g(A)≤g(B).Definition 2.2.
Supposeg(·) is a ℝ(k+1)×(k+1)↦ℝ function. g(·) is said to be a scale-equivariant function [36] if g(CA)=cg(A) for all matrices and constant c.Then a class of agreement coefficients is given by(2.6)κg*=1-g{(PI+)1/2PD(PI+)1/2}g{(I-(1/(k+1))1⋅1T)}.Ifg(A)=tr(A), then(2.7)κtr⁡*=1-1ktr⁡(PD(PI)+).If we setg(A)=tr(W*A), then(2.8)κtr⁡*(W)=1-tr⁡(W*PD*(PI)+)tr⁡(W)-(1/(k+1))∑i=0k∑j=0kWij,
where W is a nonnegative definite (k+1)×(k+1) symmetric matrix of agreement weights with wij=1 for i=j=0,1,2,…,k and 0≤wij<1 for i≠j=0,1,2,…,k. Two frequently used weighting schemes are the Cicchetti-Allison weights [37](2.9)wij=1-|i-j|k,fori,j=0,1,2,…,k,
and the Fleiss-Cohen weights [38](2.10)wij=1-(i-j)2k2,fori,j=0,1,2,…,k.We can also useg(A)=λA1, the largest eigenvalue of A as another function for g(·), which leads to a new agreement coefficient κle*(W).Therefore, including the coefficients that we propose in this paper, there are four novel coefficients to assess agreement for categorical data:(1)κtr(W), (2)κle(W), (3)κtr*(W), and (4)κle*(W). In practice, all these indices can be estimated using their sample counterparts. In this paper, a “∧” is used to denote the estimate of an index.


## 2.2. Properties

In general, ifg(·) satisfies the properties in the two aforementioned definitions, then κg*(W) has the following properties:(1)
-1≤κg*(W)≤1,(2)
κg*(W)=1 if X=Y with probability one, that is, pij=0 for all i≠j,(3)
κg*(W)=-1 if and only if pii=0 for each i=0,1,2,…,k and pij=pji=0.5 for one choice of i≠j, excluding the set of degenerate cases,(4)
κg*(W)=0 if X and Y are independent.Proof.
Without loss of generality, we setW=I. The proofs for other weight matrices W are slightly more complex.(1)
From the previous proof, we know that(2.11)0≤(PI+)1/2PD(PI+)1/2≤2(I-1k+11⋅1T).
Since g(·) is a nondecreasing and scale-equivariant function, we have
(2.12)0≤g{(PI+)1/2PD(PI+)1/2}≤2g{(I-1k+11⋅1T)}.
It is then straightforward to see that -1≤κg*≤1.(2)
IfX=Y with probability one, that is, pij=0 for all i≠j, then PD=0 [32]. It is then straightforward to see that κg*=0.(3)
Excluding the set of degenerate cases, ifpii=0 for each i=0,1,2,…,k and pij=pji=0.5 for one choice of i≠j, then PI-PD=-PI⇒PI=(1/2)PD [32]. Hence, (PI)+=((1/2)PD)+=2(PD)+ and ((PI)+)1/2=2((PD)+)1/2. Therefore, (PI+)1/2PD(PI+)1/2=2(PD+)1/2PD(PD+)1/2 and (PI+)1/2PI(PI+)1/2=(PD+)1/2PD(PD+)1/2. Hence, κg*=-1.(4)
IfX and Y are independent, then PI=PD [32]. It is straightforward to see that κg*=0.Lemma 2.3.
Whenk=1, that is, when there are only two categories, the four coefficients are equivalent and they all reduce to Cohen’s kappa.For proof of the lemma, please see the appendix.


## 3. Small Sample Performance

### 3.1. Simulation Design

In this section, we compare the performance of Cohen’s weighted kappa (κtr(W)), κle(W), κtr*(W), and κle*(W). We used the bootstrap to estimate the sample variances of the last three coefficients because the asymptotic distributions of their estimates are too complex.We used Matlab (version 7.10.0.499, The MathWorks Inc., Natick, Mass) to generate samples following multinominal distributions. Twelve different distributions were employed, representing different levels of agreement (from poor to excellent) and different numbers of categories in the responses (from 3 to 5). For details about the distributions and the design of the simulations, see Yang and Chinchilli [32]. We calculated the true coefficient values for κ, κle(W), κtr*(W), and κle*(W) as well as their estimate, bias, mean square error, the empirical variance, and the bootstrap variance. The weights we used are Cicchetti-Allison weights.We did 5000 simulations for each sample size in each distribution and 1000 bootstrap replicates for each simulation. The observed variances of the proposed method and the mean bootstrap variances are compared. We follow Yang and Chinchilli [32] in the interpretation of the results. That is, agreement measures greater than 0.75 or so may be taken to represent excellent agreement beyond chance, values between 0.40 and 0.75 may be taken to represent fair to good agreement beyond chance and values below 0.40 or so may be taken to represent poor agreement.


### 3.2. Simulation Results and Discussion

Table2 gives the simulation results for the four coefficients with the weighting scheme being Cicchetti-Allison weights (the simulation results are split into two tables because it is unwieldy to fit in one page).Table 2
Simulation results.
(a)Casenkκ1κ̂1b1κ2κ̂2b2κ3κ̂3b3κ4κ̂4b418020.4950.490−0.0050.4670.458−0.0090.4930.487−0.0070.4880.482−0.007116020.4950.4950.0000.4670.466−0.0020.4930.492−0.0010.4880.488−0.0012802−0.128−0.1270.001−0.030−0.036−0.006−0.122−0.1210.002−0.097−0.0960.00121602−0.128−0.1260.001−0.030−0.031−0.002−0.122−0.1200.002−0.097−0.0950.00238020.8510.850−0.0010.8410.834−0.0070.8480.845−0.0020.8450.843−0.003316020.8510.8510.0000.8410.838−0.0030.8480.847−0.0010.8450.844−0.00148020.5380.535−0.0040.5560.554−0.0020.5830.580−0.0040.6060.602−0.004416020.5380.537−0.0020.5560.5560.0000.5830.581−0.0020.6060.604−0.00258030.0560.053−0.003−0.016−0.019−0.0030.1410.138−0.0030.1710.167−0.004516030.0560.053−0.003−0.016−0.020−0.0040.1410.138−0.0030.1710.168−0.00368030.3980.395−0.0030.3050.299−0.0060.3920.387−0.0050.3780.371−0.007616030.3980.397−0.0010.3050.303−0.0020.3920.390−0.0020.3780.374−0.0037803−0.174−0.1720.002−0.254−0.2530.001−0.166−0.1650.001−0.190−0.1880.00271603−0.174−0.1740.000−0.254−0.2540.000−0.166−0.1660.000−0.190−0.1890.00088030.1480.1500.002−0.058−0.0530.0050.0180.0210.004−0.013−0.0090.004816030.1480.1480.000−0.058−0.0560.0020.0180.0190.001−0.013−0.0110.0029803−0.319−0.3130.007−0.411−0.4040.007−0.296−0.2900.006−0.322−0.3160.00691603−0.319−0.3160.003−0.411−0.4080.003−0.296−0.2930.003−0.322−0.3190.003108040.5640.561−0.0030.5430.532−0.0100.5150.505−0.0090.5220.511−0.0111016040.5640.562−0.0010.5430.538−0.0050.5150.510−0.0050.5220.516−0.00511804−0.075−0.0750.000−0.087−0.089−0.002−0.075−0.076−0.001−0.087−0.0870.000111604−0.075−0.076−0.001−0.087−0.090−0.003−0.075−0.076−0.001−0.087−0.088−0.001128040.7910.7900.0000.7690.767−0.0020.7030.699−0.0040.7010.684−0.0171216040.7910.7920.0010.7690.7700.0000.7030.702−0.0010.7010.694−0.007Note:n is the sample size, k+1 is equal to the number of categories in the response. κ1 is the true coefficient value for Cohen’s weighted kappa, κ2 stands for κle(W), κ3 stands for κtr*(W), and κ4 stands for κle*(W). κ̂1, κ̂2, κ̂3, and κ̂4 are their estimates, respectively. b1, b2, b3, and b4 are the biases, respectively.(b)Casenkκb2κb3κb4ms1ms2ms3ms4v2ev2bv3ev3bv4ev4b18020.4500.4820.4770.0070.0110.0080.0090.0110.0110.0080.0080.0090.009116020.4620.4900.4850.0040.0060.0040.0040.0060.0060.0040.0040.0040.0042802−0.041−0.119−0.0940.0040.0070.0040.0050.0070.0070.0040.0040.0050.00421602−0.034−0.119−0.0940.0020.0040.0020.0020.0040.0030.0020.0020.0020.00238020.8270.8430.8390.0030.0050.0040.0040.0050.0050.0040.0040.0040.004316020.8350.8450.8430.0020.0020.0020.0020.0020.0020.0020.0020.0020.00248020.5470.5760.5980.0050.0070.0040.0040.0070.0070.0040.0040.0040.004416020.5540.5790.6020.0020.0040.0020.0020.0040.0040.0020.0020.0020.0025803−0.0220.1350.1630.0080.0160.0070.0080.0160.0150.0070.0070.0080.00851603−0.0220.1370.1660.0040.0080.0040.0040.0080.0080.0040.0030.0040.00468030.2950.3820.3650.0080.0150.0080.0100.0150.0140.0080.0070.0100.009616030.3010.3870.3710.0040.0070.0040.0050.0070.0070.0040.0040.0050.0057803−0.252−0.163−0.1860.0040.0080.0030.0040.0080.0080.0030.0030.0040.00471603−0.254−0.166−0.1880.0020.0040.0020.0020.0040.0040.0020.0020.0020.0028803−0.0490.025−0.0060.0060.0100.0030.0040.0100.0100.0030.0030.0040.00481603−0.0540.021−0.0090.0030.0050.0010.0020.0050.0050.0010.0010.0020.0029803−0.399−0.284−0.3110.0030.0080.0040.0060.0080.0080.0040.0050.0060.00791603−0.406−0.291−0.3170.0020.0040.0020.0030.0040.0040.0020.0020.0030.003108040.5220.4960.5000.0060.0090.0080.0080.0090.0090.0080.0080.0080.0081016040.5330.5050.5110.0030.0040.0040.0040.0040.0050.0040.0040.0040.00411804−0.091−0.075−0.0870.0030.0070.0030.0040.0070.0070.0030.0030.0040.004111604−0.091−0.076−0.0880.0010.0040.0010.0020.0040.0040.0010.0010.0020.002128040.7630.6960.6710.0040.0070.0050.0070.0070.0070.0050.0050.0070.0081216040.7680.7000.6860.0020.0040.0020.0030.0040.0030.0020.0020.0030.003Note:n is the sample size, k+1 is equal to the number of categories in the response. κb2, κb3, and κb4 are the mean bootstrap estimates for κle(W), κtr*(W), and κle*(W), respectively. ms1, ms2, ms3, and ms4 are the MSE for the estimates of the four coefficients, respectively. v2e and v2b are the empirical variance and the bootstrap variance for κle(W), respectively. v3e and v3b are the empirical variance and the bootstrap variance for κtr*(W), respectively. v4e and v4b are the empirical variance and the bootstrap variance for κle*(W), respectively.In almost all the cases, the four agreement coefficients give the same classification in terms of the magnitude of the degrees of agreement. An interesting case is case 6 where the degree of agreement is around the boundary of poor and medium agreement. Cohen’s weighted kappa andκtr*(W) indicate close to medium agreement while κle(W) and κle*(W) indicate poor agreement, according to our rule of interpretation of the results. Biases of all the four coefficients are small, and, as sample sizes increase, they become almost negligible. The bootstrap turns out to be a good alternative as an estimation of the sample variances, which in many cases gives the estimates of the variances equivalent to the observed values.It also should be noted that the true values of coefficients within Table2(a) can vary, especially for cases 5–9 (k=3) and cases 10–12 (k=4). This suggests that, as table size (k+1)×(k+1) increases, the coefficients could lead to different interpretations.


## 3.1. Simulation Design

In this section, we compare the performance of Cohen’s weighted kappa (κtr(W)), κle(W), κtr*(W), and κle*(W). We used the bootstrap to estimate the sample variances of the last three coefficients because the asymptotic distributions of their estimates are too complex.We used Matlab (version 7.10.0.499, The MathWorks Inc., Natick, Mass) to generate samples following multinominal distributions. Twelve different distributions were employed, representing different levels of agreement (from poor to excellent) and different numbers of categories in the responses (from 3 to 5). For details about the distributions and the design of the simulations, see Yang and Chinchilli [32]. We calculated the true coefficient values for κ, κle(W), κtr*(W), and κle*(W) as well as their estimate, bias, mean square error, the empirical variance, and the bootstrap variance. The weights we used are Cicchetti-Allison weights.We did 5000 simulations for each sample size in each distribution and 1000 bootstrap replicates for each simulation. The observed variances of the proposed method and the mean bootstrap variances are compared. We follow Yang and Chinchilli [32] in the interpretation of the results. That is, agreement measures greater than 0.75 or so may be taken to represent excellent agreement beyond chance, values between 0.40 and 0.75 may be taken to represent fair to good agreement beyond chance and values below 0.40 or so may be taken to represent poor agreement.


## 3.2. Simulation Results and Discussion

Table2 gives the simulation results for the four coefficients with the weighting scheme being Cicchetti-Allison weights (the simulation results are split into two tables because it is unwieldy to fit in one page).Table 2
Simulation results.
(a)Casenkκ1κ̂1b1κ2κ̂2b2κ3κ̂3b3κ4κ̂4b418020.4950.490−0.0050.4670.458−0.0090.4930.487−0.0070.4880.482−0.007116020.4950.4950.0000.4670.466−0.0020.4930.492−0.0010.4880.488−0.0012802−0.128−0.1270.001−0.030−0.036−0.006−0.122−0.1210.002−0.097−0.0960.00121602−0.128−0.1260.001−0.030−0.031−0.002−0.122−0.1200.002−0.097−0.0950.00238020.8510.850−0.0010.8410.834−0.0070.8480.845−0.0020.8450.843−0.003316020.8510.8510.0000.8410.838−0.0030.8480.847−0.0010.8450.844−0.00148020.5380.535−0.0040.5560.554−0.0020.5830.580−0.0040.6060.602−0.004416020.5380.537−0.0020.5560.5560.0000.5830.581−0.0020.6060.604−0.00258030.0560.053−0.003−0.016−0.019−0.0030.1410.138−0.0030.1710.167−0.004516030.0560.053−0.003−0.016−0.020−0.0040.1410.138−0.0030.1710.168−0.00368030.3980.395−0.0030.3050.299−0.0060.3920.387−0.0050.3780.371−0.007616030.3980.397−0.0010.3050.303−0.0020.3920.390−0.0020.3780.374−0.0037803−0.174−0.1720.002−0.254−0.2530.001−0.166−0.1650.001−0.190−0.1880.00271603−0.174−0.1740.000−0.254−0.2540.000−0.166−0.1660.000−0.190−0.1890.00088030.1480.1500.002−0.058−0.0530.0050.0180.0210.004−0.013−0.0090.004816030.1480.1480.000−0.058−0.0560.0020.0180.0190.001−0.013−0.0110.0029803−0.319−0.3130.007−0.411−0.4040.007−0.296−0.2900.006−0.322−0.3160.00691603−0.319−0.3160.003−0.411−0.4080.003−0.296−0.2930.003−0.322−0.3190.003108040.5640.561−0.0030.5430.532−0.0100.5150.505−0.0090.5220.511−0.0111016040.5640.562−0.0010.5430.538−0.0050.5150.510−0.0050.5220.516−0.00511804−0.075−0.0750.000−0.087−0.089−0.002−0.075−0.076−0.001−0.087−0.0870.000111604−0.075−0.076−0.001−0.087−0.090−0.003−0.075−0.076−0.001−0.087−0.088−0.001128040.7910.7900.0000.7690.767−0.0020.7030.699−0.0040.7010.684−0.0171216040.7910.7920.0010.7690.7700.0000.7030.702−0.0010.7010.694−0.007Note:n is the sample size, k+1 is equal to the number of categories in the response. κ1 is the true coefficient value for Cohen’s weighted kappa, κ2 stands for κle(W), κ3 stands for κtr*(W), and κ4 stands for κle*(W). κ̂1, κ̂2, κ̂3, and κ̂4 are their estimates, respectively. b1, b2, b3, and b4 are the biases, respectively.(b)Casenkκb2κb3κb4ms1ms2ms3ms4v2ev2bv3ev3bv4ev4b18020.4500.4820.4770.0070.0110.0080.0090.0110.0110.0080.0080.0090.009116020.4620.4900.4850.0040.0060.0040.0040.0060.0060.0040.0040.0040.0042802−0.041−0.119−0.0940.0040.0070.0040.0050.0070.0070.0040.0040.0050.00421602−0.034−0.119−0.0940.0020.0040.0020.0020.0040.0030.0020.0020.0020.00238020.8270.8430.8390.0030.0050.0040.0040.0050.0050.0040.0040.0040.004316020.8350.8450.8430.0020.0020.0020.0020.0020.0020.0020.0020.0020.00248020.5470.5760.5980.0050.0070.0040.0040.0070.0070.0040.0040.0040.004416020.5540.5790.6020.0020.0040.0020.0020.0040.0040.0020.0020.0020.0025803−0.0220.1350.1630.0080.0160.0070.0080.0160.0150.0070.0070.0080.00851603−0.0220.1370.1660.0040.0080.0040.0040.0080.0080.0040.0030.0040.00468030.2950.3820.3650.0080.0150.0080.0100.0150.0140.0080.0070.0100.009616030.3010.3870.3710.0040.0070.0040.0050.0070.0070.0040.0040.0050.0057803−0.252−0.163−0.1860.0040.0080.0030.0040.0080.0080.0030.0030.0040.00471603−0.254−0.166−0.1880.0020.0040.0020.0020.0040.0040.0020.0020.0020.0028803−0.0490.025−0.0060.0060.0100.0030.0040.0100.0100.0030.0030.0040.00481603−0.0540.021−0.0090.0030.0050.0010.0020.0050.0050.0010.0010.0020.0029803−0.399−0.284−0.3110.0030.0080.0040.0060.0080.0080.0040.0050.0060.00791603−0.406−0.291−0.3170.0020.0040.0020.0030.0040.0040.0020.0020.0030.003108040.5220.4960.5000.0060.0090.0080.0080.0090.0090.0080.0080.0080.0081016040.5330.5050.5110.0030.0040.0040.0040.0040.0050.0040.0040.0040.00411804−0.091−0.075−0.0870.0030.0070.0030.0040.0070.0070.0030.0030.0040.004111604−0.091−0.076−0.0880.0010.0040.0010.0020.0040.0040.0010.0010.0020.002128040.7630.6960.6710.0040.0070.0050.0070.0070.0070.0050.0050.0070.0081216040.7680.7000.6860.0020.0040.0020.0030.0040.0030.0020.0020.0030.003Note:n is the sample size, k+1 is equal to the number of categories in the response. κb2, κb3, and κb4 are the mean bootstrap estimates for κle(W), κtr*(W), and κle*(W), respectively. ms1, ms2, ms3, and ms4 are the MSE for the estimates of the four coefficients, respectively. v2e and v2b are the empirical variance and the bootstrap variance for κle(W), respectively. v3e and v3b are the empirical variance and the bootstrap variance for κtr*(W), respectively. v4e and v4b are the empirical variance and the bootstrap variance for κle*(W), respectively.In almost all the cases, the four agreement coefficients give the same classification in terms of the magnitude of the degrees of agreement. An interesting case is case 6 where the degree of agreement is around the boundary of poor and medium agreement. Cohen’s weighted kappa andκtr*(W) indicate close to medium agreement while κle(W) and κle*(W) indicate poor agreement, according to our rule of interpretation of the results. Biases of all the four coefficients are small, and, as sample sizes increase, they become almost negligible. The bootstrap turns out to be a good alternative as an estimation of the sample variances, which in many cases gives the estimates of the variances equivalent to the observed values.It also should be noted that the true values of coefficients within Table2(a) can vary, especially for cases 5–9 (k=3) and cases 10–12 (k=4). This suggests that, as table size (k+1)×(k+1) increases, the coefficients could lead to different interpretations.


## 4. Examples

In this section, we use two empirical examples to illustrate the application and compare the result of the four agreement coefficients. The first example is from Brenner et al. [39] who examined the agreement of cause-of-death data for breast cancer patients between Ontario Cancer Registry (OCR) and classification results from Mount Sinai Hospital (MSH). OCR, established in 1964, collects information about all new cases of cancer in the province of Ontario while MSH has a systematic and rigorous monitoring and follow-up of confirmed node-negative breast cancer patients. Due to its relative complete and accurate data, MSH also provides specialist-determined cause of death. 1648 patients entered the analysis through linking the OCR data to the MSH study patients via the OCR standard procedure. The purpose of the study is to examine the degree of agreement between OCR, which is often used in cancer studies, and MSH, which was taken as the reference standard. A missed cancer-related death is considered of greater importance. The data are summarized in Table 3.Table 3
Data for example 1.
Ontario Cancer RegistryMSH studyNo death informationCompeting cause of deathBreast cancerTotalNo death information1331661343Competing cause of death191297155Breast cancer521124150Total13551561371648Note: OCR and MSH studies classify cause of death for 1648 breast cancer patients as having no cancer-related information, competing cause of death, and breast cancer death.The second example is from Jiménez-Navarro et al. [40] who studied the concordance of oral glucose tolerance test (OGTT), proposed by the World Health Organization, in the diagnosis of diabetes mellitus (DM) which may possibly lead to future epidemic of coronary disease. OGTT classifies a diagnosis result as DM, glucose intolerance, or normal. Eighty-eight patients admitted with an acute coronary syndrome who had no previous diagnosis of DM underwent percutaneous coronary revascularization and received OGTT the day after revascularization and one month after that. Researchers are interested in the reproducibility of OGTT performed at these two different time points. The data are summarized in Table 4.Table 4
Data for example 2.
OGTT at revascularizationOGTT at one month laterNormalGlucose intoleranceDMTotalNormal172322Glucose intolerance2210436DM1011930Total49231688Note: OGTT classifies 88 patients admitted with an acute coronary syndrome with no previous DM as having DM, being glucose intolerance or normal at the day after revascularization and one month later.We calculated all the four agreement coefficients as well as their corresponding 95% confidence intervals (CI). The 95% CI for Cohen’s weighted kappa was based on the large sample variance derived by Fleiss et al. [31]. 5000 bootstrap replicates were used in the calculation of the CI for the other three coefficients. Table 5 gives the results for the above two examples. In all the calculations, the Cicchetti-Allison weights were used (see (2.9)).Table 5
Results for the two examples.
κW95% CIκle95% CIκtr*(W)95% CIκle*(W)95% CIExample  10.900(0.843, 0.957)0.924(0.899, 0.944)0.872(0.840, 0.902)0.881(0.848, 0.910)Example  20.203(0.079, 0.327)0.257(0.078, 0.417)0.202(0.052, 0.352)0.218(0.058, 0.374)Note: The 95% CI was forκW calculated using the large sample variance derived by Fleiss et al. [31]. The CIs for the other three coefficients were calculated using bootstrap method with 5000 replicates.As can be seen from Table5, there seems to be no substantial difference among all the four coefficients using the aforementioned interpretation rule. For example 1, all coefficients indicate that there is very strong agreement between OCR and MSH studies in classifying the cause of death for the 1648 cancer patients. The second example, however, indicates that OGTT, when taken at two different time points with a span of one month, has very poor reproducibility in the diagnosis of DM.


## 5. Conclusion

We developed a fixed-effects modeling of Cohen’s kappa and weighted kappa for bivariate multinomial data through the perspective of generalized inverse, which provides an alternative form of framework to the one proposed by Yang and Chinchilli [33]. The proposed method also allows the application of different matrix functions, as long as they satisfy certain conditions. When there are only two categories in the response, the coefficient reduces to Cohen’s kappa. The proposed method is new and promising in that it allows the use of other matrix functions, such as the largest eigenvalue of a matrix, that might also yield reasonable, and possibly better, result. Exploration of such matrix functions is our future research topic.



---

*Source: 603856-2011-12-10.xml*