# Power Weighted Versions of Bennett, Alpert, and Goldstein’sS

Authors: Matthijs J. Warrens
Journal: Journal of Mathematics
 (2014)

---

## Abstract

A weighted version of Bennett, Alpert, and Goldstein’sS, denoted by S
r, is studied. It is shown that the special cases of S
r are often ordered in the same way. It is also shown that many special cases of S
r tend to produce values close to unity, especially when the number of categories of the rating scale is large. It is argued that the application of S
r as an agreement coefficient is not without difficulties.


---

## Body

## 1. Introduction

In behavioral and biomedical science it is frequently required to measure the intensity of a behavior or a disease. Examples are the degree of arousal of a speech-anxious participant while giving a presentation, the severity of lesions from scans, or the severity of sedation during opioid administration for pain management. The intensity of these phenomena is usually classified by a single observer using a rating scale with ordered categories, for example, mild, moderate, or severe. To avoid that the observer did not fully understand what he or she was asked to interpret, the categories must be clearly defined. To measure the reliability of the rating scale researchers typically ask two observers to rate independently the same set of subjects. Analysis of the agreement between the observers can then be used to asses the reliability of the scale. High agreement between the ratings of the observers usually indicates consensus in the diagnosis and interchangeability of the classifications of the observers.For assessing agreement on an ordinal scale various statistical methodologies have been developed. For example, the loglinear models presented in Tanner and Young [1] and Agresti [2, 3] can be used for analyzing the patterns of agreement and potential sources of disagreement. Applications of these models can be found in Becker [4] and Graham and Jackson [5]. However, it turns out that researchers are usually only interested in a coefficient that (roughly) summarizes the agreement in a single number. The most commonly used coefficient for summarizing agreement on an ordinal scale is weighted kappa proposed in Cohen [6] ([5, 7]). Cohen [8] proposed coefficient kappa as an index of agreement when the rating scale has nominal (unordered) categories [9]. The coefficient corrects for agreement due to chance. Weighted kappa extends Cohen’s original kappa to rating scales with ordered categories. In the latter case there is usually more disagreement between the observers on adjacent categories than on categories that are further apart. With weighted kappa it is possible to describe the closeness between categories using weights. Both kappa and weighted kappa are standard tools in assessing agreement and have been used in thousands of applications [10, 11]. The most commonly used version of weighted kappa is quadratic kappa [5, 7].Various authors have identified difficulties with the interpretation of kappa for nominal categories [7, 12–17]. Cohen’s kappa is a function of the marginal totals, the base rates of the categories, which indicate how often the categories were used by the observers [18–20]. Cohen’s kappa tends to produce much lower values for skewed marginal distributions. Furthermore, kappas from samples with different base rates are not comparable [13, 16]. de Mast and van Wieringen [16] and de Mast [17] studied kappa and kappa-type coefficients in the context of a latent class model. These authors argued that the problematic behavior of kappa is explained from the fact that it is a coefficient of predictive association, instead of a pure coefficient of agreement. Other authors have identified difficulties with the interpretation of quadratic kappa for ordered categories as well. Quadratic kappa behaves as a measure of association, instead of an agreement coefficient [5]. The value of quadratic kappa also tends to increase as the number of categories increases [21]. Furthermore, quadratic kappa cannot discriminate between tables with very different levels of exact agreement [22].A commonly proposed alternative for Cohen’s kappa for nominal categories is coefficientS, originally proposed in Bennett et al. [23] ([24–26]). Since coefficient S is a linear transformation of the raw agreement and not a function of the marginal totals, it does not exhibit the interpretation difficulties of the kappa coefficients [10, 27]. Furthermore, under the latent class model discussed in de Mast and van Wieringen [16] and de Mast [17], coefficient S is the only agreement coefficient that can be given some justification. Coefficient S is equivalent to coefficient C in Janson and Vegelius [28], coefficient RE in Janes [29], and kappa
n in Brennan and Prediger [12]. In the case of two categories coefficient S is equivalent to coefficients discussed in, among others, Holley and Guilford [30], Maxwell [31], and Krippendorff [32].Recently, Gwet [33] proposed a weighted version of coefficient S for rating scales with ordinal categories. In this paper this coefficient will be denoted by S
w. The generalization proposed in [33] is analogous to the generalization of kappa [8] to weighted kappa [6]. The weighting schemes that can be used with S
w are identical to the weighting schemes of weighted kappa. The most commonly used weighting schemes for weighted kappa are the linear weights [34–36] and the quadratic weights [22, 37, 38]. In this paper we study how S
w behaves as an agreement coefficient for rating scales with ordinal categories. More precisely, we study a special case of S
w which will be denoted by S
r. Special cases of S
r are coefficient S and the coefficients that are obtained if we use the linear and quadratic weighting schemes. We present several properties of S
r that indicate that the application of S
w as an agreement coefficient is not without problems.The paper is organized as follows. In Section2 we introduce notation and define coefficients S
w and S
r. In Section 3 it is shown that there is a simple ordering of the special cases of S
r if a certain mild condition holds. Since this requirement is often met in real life, the special cases of S
r are usually ordered in the same way. In Section 4 we present properties of S
r for tridiagonal agreement tables. It is shown that many special cases of S
r tend to produce values close to unity, especially when the number of categories of the rating scale is large. Section 5 contains a discussion.


## 2. Weighted Coefficients

In this section we introduce notation and define the coefficientsS
w and S
r. Gwet [33, page 56] defines S
w in terms of similarity scaling. However, for notational convenience, we will define S
w in terms of dissimilarity scaling here. If the weights are dissimilarities, pairs of categories that are further apart are usually assigned higher weights.Suppose two fixed observers independently rate the same set ofn subjects using the same set of c
≥
2 ordered categories that are defined in advance. For a population of subjects, let π
i
j denote the proportion classified in category i by the first observer and in category j by the second observer for 1
≤
i
,
j
≤
c. Furthermore, let n
i
j denote the contingency table of observed frequencies. The probabilities π
i
j can be estimated from the n
i
j. Assuming a multinominal sampling model with the total number of subjects n fixed, the maximum likelihood estimate of π
i
j is given by π
^
i
j
=
n
i
j
/
n [39, 40].Since the labels of the rows and columns ofn
i
j are identical, the contingency table is usually called an agreement table. Table 1 is an example of an agreement table. The data of Table 1 come from Holmquist et al. [41]. Seven pathologists, labeled A to G, classified each of 118 slides in terms of carcinoma in situ of the uterine cervix, based on the most involved lesion, using ordered categories, (1) negative, (2) atypical squamous hyperplasia, (3) carcinoma in situ, (4) squamous carcinoma with early stromal invasion, and (5) invasive carcinoma. The data can also be found in Landis and Koch [42]. Table 1 is the cross classification of the ratings of pathologists A and D.Table 1
Frequencies of pairwise classifications of carcinoma by pathologists A and D [41].
Pathologist A
Pathologist D
Row
1
2
3
4
5
Total
(1) Negative
25
1
0
0
0
26
(2) Atypical squamous hyperplasia
11
15
0
0
0
26
(3) Carcinoma in situ
1
22
13
2
0
38
(4) Squamous carcinoma
1
8
9
4
0
22
(5) Invasive carcinoma
0
2
1
2
1
6
Column totals
38
48
23
8
1
118Letw
i
j for i
,
j
∈
1,2
,
…
,
c be nonnegative real numbers with w
i
i
=
0. The numbers w
i
j are used as weights, one for each cell of the table π
i
j. If we formulate Gwet’s approach in terms of dissimilarity scaling, then Gwet [33] presented the coefficient
(1)
S
w
≔
1
-
∑
i
=
1
c
∑
j
=
1
c
π
i
j
w
i
j
1
/
c
2
∑
i
=
1
c
∑
j
=
1
c
w
i
j
.
Coefficient S
w is well defined if we require that at least one w
i
j is nonzero. With n fixed, the maximum likelihood estimate of (1) under a multinomial sampling model is given by
(2)
S
^
w
≔
1
-
c
2
n
∑
i
=
1
c
∑
j
=
1
c
n
i
j
w
i
j
∑
i
=
1
c
∑
j
=
1
c
w
i
j
.
In this paper we are interested in a particular weighting scheme. Let r
≥
0 be a nonnegative real number and consider the weight function
(3)
w
i
j
r
≔
0
for
i
=
j
;
i
-
j
r
for
i
≠
j
.
Using weight function (3) in (1) we obtain the weighted coefficient
(4)
S
r
=
1
-
c
2
∑
i
=
1
c
∑
j
=
1
c
π
i
j
i
-
j
r
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
.
Various well-known weighting schemes are special cases of weighting scheme (3). For r
=
0 we have the identity weights
(5)
w
i
j
0
=
1
i
≠
j
=
0
for
i
=
j
;
1
for
i
≠
j
.
For c
=
4 categories weighting scheme (5) is given by
(6)
0
1
1
1
1
0
1
1
1
1
0
1
1
1
1
0
.
If we use r
=
0 in (4) we obtain
(7)
S
0
=
1
-
1
-
∑
i
=
1
c
π
i
i
1
/
c
2
c
2
-
c
=
∑
i
=
1
c
π
i
i
-
1
/
c
1
-
1
/
c
.
Coefficient (7) is Bennett et al.’s [23] S, an agreement coefficient proposed for rating scales with nominal categories [12, 28, 29]. Coefficient S
0 is thus a special case of coefficient (4). The value of S
0 is 1 if there is perfect agreement between the observers and 0 when ∑
i
π
i
i
=
1
/
c. For Table 1 we have the estimate S
^
0
=
0.364.If we user
=
1 in (3) we obtain the linear weights w
i
j
(
1
)
=
|
i
-
j
| [34–36]. For c
=
4 categories the linear weighting scheme is given by
(8)
0
1
2
3
1
0
1
2
2
1
0
1
3
2
1
0
.
The linear coefficient is denoted by S
1. For Table 1 we have the estimate S
^
1
=
0.597.If we user
=
2 in (3) we obtain the quadratic weights w
i
j
(
2
)
=
(
i
-
j
)
2 [22, 37, 38]. For c
=
4 categories the quadratic weighting scheme is given by
(9)
0
1
4
9
1
0
1
4
4
1
0
1
9
4
1
0
.
The quadratic coefficient is denoted by S
2. For Table 1 we have the estimate S
^
2
=
0.758.Finally, if we user
=
0.5 in (3) we obtain the radical weights w
i
j
(
0.5
)
=
|
i
-
j
| [33, page 63, 64]. For c
=
4 categories the radical weighting scheme is given by
(10)
0
1
2
3
1
0
1
2
2
1
0
1
3
2
1
0
.
The radical coefficient is denoted by S
0.5. For Table 1 we have the estimate S
^
0.5
=
0.487. Hence, for Table 1 we have the ordering S
^
0
<
S
^
0.5
<
S
^
1
<
S
^
2.Finally, forc
=
2 categories coefficient S
r becomes
(11)
S
0
=
π
11
+
π
22
-
1
/
2
1
-
1
/
2
=
2
π
11
+
π
22
-
1
.
Since all special cases of S
r coincide for c
=
2 categories, there are no examples of 2
×
2 tables in this paper.


## 3. Conditional Inequalities

If we apply coefficientsS
0, S
0.5, S
1, and S
2 to the same rating data we consistently find the triple inequality S
^
0
<
S
^
0.5
<
S
^
1
<
S
^
2. For example, consider the data entries in Table 2. Table 2 presents various statistics of 20 agreement tables from the literature. The first column of Table 2 specifies the source of the agreement table and the second column shows whether the table has size 3
×
3, 4
×
4, or 5
×
5. Columns 3 to 6 of Table 2 contain the values of the estimates S
^
0, S
^
0.5, S
^
1, and S
^
2. For all entries except the first we have the triple inequality S
^
0
<
S
^
0.5
<
S
^
1
<
S
^
2.Table 2
Various statistics for 20 agreement tables from the literature.
Source
Number of categories
Coefficients
Statistics condition (15)
S
^
0
S
^
0.5
S
^
1
S
^
2
a
^
1
a
^
2
a
^
3
a
^
4
Cohen [8]
3
.550
.550
.550
.550
.050
.050
Martin et al. [49]
3
.958
.963
.968
.979
.007
0
Simonoff [50, p. 307]
3
.800
.817
.835
.871
.030
.006
Anderson et al. [51]
3
.419
.455
.494
.569
.081
.031
Anderson et al. [51]
3
.719
.746
.775
.831
.044
.006
Anderson et al. [51]
3
.250
.307
.367
.484
.109
.031
Agresti [39, p. 368]
4
.514
.603
.688
.824
.056
.006
0
Agresti [39, p. 378]
4
.304
.411
.513
.675
.075
.014
.007
Simonoff [50, p. 288]
4
.683
.748
.810
.905
.040
0
0
Simonoff [50, p. 303]
4
.647
.704
.758
.840
.040
.004
.006
Simonoff [50, p. 303]
4
.124
.253
.379
.591
.090
.028
.002
Hand et al. [52]
4
.239
.326
.409
.549
.072
.029
.013
Hand et al. [52]
4
.450
.525
.597
.711
.056
.014
.009
Hand et al. [52]
4
.652
.724
.791
.896
.043
0
0
Ne
´methy et al. [53]
4
.787
.831
.872
.936
.027
0
0
Simonoff [50, p. 272]
5
.919
.939
.956
.979
.007
.001
.000
0
Seddon et al. [54]
5
.803
.854
.898
.956
.019
.001
0
0
Bohannon and Smith [55]
5
.833
.878
.917
.967
.017
0
0
0
Ne
´methy et al. [53]
5
.738
.805
.863
.940
.025
.002
0
0
Maria and Victorino [56]
5
.800
.854
.900
.960
.020
0
0
0As a second example we consider the data on diagnosis of carcinoma from Holmquist et al. [41]. Seven pathologists labeled A to G classified each of 118 slides in terms of carcinoma in situ of the uterine cervix, based on the most involved lesion, using five ordered categories. Table 1 is the cross classification of the ratings of pathologists A and D. Table 3 presents various statistics of the 21 pairwise agreement tables for the seven pathologists. Columns 2 to 5 of Table 3 contain the values of the estimates S
^
0, S
^
0.5, S
^
1, and S
^
2. For all 21 tables we have the triple inequality S
^
0
<
S
^
0.5
<
S
^
1
<
S
^
2. The quantities in the last four columns of Tables 2 and 3 are defined and discussed.Table 3
Various statistics for the 21 pairwise agreement tables between seven pathologists [41].
Pathologists
Coefficients
Statistics condition (15)
S
^
0
S
^
0.5
S
^
1
S
^
2
a
^
1
a
^
2
a
^
3
a
^
4
(A, B)
.544
.648
.740
.871
.039
.008
0
0
(A, C)
.417
.549
.661
.805
.053
.003
.004
.004
(A, D)
.364
.487
.597
.758
.050
.014
.006
0
(A, E)
.428
.567
.688
.854
.052
.007
0
0
(A, F)
.216
.338
.449
.614
.055
.021
.013
.004
(A, G)
.513
.628
.730
.871
.043
.007
0
0
(B, C)
.428
.556
.666
.816
.050
.008
0
.004
(B, D)
.322
.470
.603
.794
.056
.016
0
0
(B, E)
.576
.691
.788
.915
.042
0
0
0
(B, F)
.258
.368
.476
.655
.047
.032
.006
0
(B, G)
.714
.782
.841
.924
.025
.004
0
0
(C, D)
.492
.604
.703
.843
.043
.008
.002
0
(C, E)
.386
.525
.645
.807
.054
.008
0
.004
(C, F)
.375
.506
.619
.771
.053
.008
.004
.004
(C, G)
.566
.671
.762
.883
.040
.003
.002
0
(D, E)
.258
.412
.550
.752
.059
.018
.002
0
(D, F)
.428
.563
.682
.847
.051
.008
0
0
(D, G)
.481
.621
.740
.896
.052
0
0
0
(E, F)
.153
.289
.412
.595
.060
.023
.013
.004
(E, G)
.544
.658
.756
.890
.042
.004
0
0
(F, G)
.375
.486
.592
.761
.043
.025
0
0Tables2 and 3 illustrate that the ordering S
^
0
<
S
^
0.5
<
S
^
1
<
S
^
2 is often found with real life data. This suggests that S
r is usually increasing in r. The triple inequality does not hold in general, but it holds if a certain condition is valid. This sufficient condition is defined below. Recall that π
i
j is the agreement table with proportions. Define the quantities
(12)
a
j
=
1
2
c
-
j
∑
i
=
1
c
-
j
π
i
,
i
+
j
+
π
i
+
j
,
i
,
for
1
≤
j
≤
c
-
1
.
For fixed j, the quantity a
j in (12) is the sum of all elements of π
i
j that are j steps removed from the main diagonal, divided by 2
(
c
-
j
). Since there are precisely 2
(
c
-
j
) elements that are j steps removed from the main diagonal, a
j is the average disagreement of the elements that are j steps removed from the main diagonal. Since the elements of π
i
j that are j steps removed from the main diagonal correspond to pairs of categories that are j steps apart, a
1 is the average disagreement between the observers on adjacent categories, a
2 is the average disagreement on all categories that are two steps apart, and so on.With ordered categories it is natural to assume that(13)
a
j
≥
a
j
+
1
,
for
1
≤
j
≤
c
-
2
.
Condition (13) states that the average disagreement between the observers on categories that are closer in the ordering is higher than on categories that are further apart in the ordering. Since condition (13) depends on the unobserved probabilities π
i
j it cannot be verified directly. If we replace the probabilities π
i
j by the n
i
j
/
n we obtain the estimates
(14)
a
^
j
=
1
2
n
c
-
j
∑
i
=
1
c
-
j
n
i
,
i
+
j
+
n
i
+
j
,
i
,
0000000000000
for
1
≤
j
≤
c
-
1
.
To check whether inequality (13) is reasonable for real life data we may check if the inequality
(15)
a
^
j
≥
a
^
j
+
1
,
for
1
≤
j
≤
c
-
2
,
holds. It turns out that condition (15) holds for many real life agreement tables with ordered categories. This is to be expected if the rating scale has been thoughtfully constructed, since in this case one expects that the disagreement between the observers on categories that are closer in the ordering is higher than on categories that are further apart in the ordering. For example, consider the data in Table 1. We have
(16)
a
^
1
=
0.050
,
a
^
2
=
0.014
,
a
^
3
=
0.006
,
a
^
4
=
0
,
or a
^
1
>
a
^
2
>
a
^
3
>
a
^
4. Hence, condition (15) holds for Table 1. Furthermore, the last four columns of Tables 2 and 3 contain the estimates a
^
1, a
^
2, a
^
3, and a
^
4 for the various agreement tables. For all entries of Table 2 condition (15) holds. Moreover, for all entries except the first, the inequalities are strict. For the first entry of Table 2 [8] we have
(17)
a
^
1
=
a
^
2
=
0.05
.
If the a
^
j are all equal, all special cases of S
^
r coincide. It should be noted that the data in Cohen [8] are artificial. Condition (15) also holds for most entries of Table 3. The three exceptions are the entries corresponding to the pairs (A,C), (B,C), and (C,E).Theorem2 below shows that S
r is increasing in r if condition (13) holds. Thus, if (13) holds there is a simple relationship between the special cases of coefficient (4). In particular, if (13) holds we have the triple inequality
(18)
S
0
≤
S
0.5
≤
S
1
≤
S
2
.
Lemma 1 is used in the proof of Theorem 2.Lemma 1.
Leta
i for 1
≤
i
≤
m be nonnegative real numbers and let b
i and d
i for 1
≤
i
≤
m be positive real numbers. If
(19)
a
i
≥
a
j
,
f
o
r
i
<
j
,
(20)
b
i
d
i
>
b
j
d
j
,
f
o
r
i
<
j
,
then
(21)
∑
i
=
1
m
a
i
b
i
∑
i
=
1
m
b
i
≥
∑
i
=
1
m
a
i
d
i
∑
i
=
1
m
d
i
.
Furthermore, inequality (21) is strict if two a
i are distinct.Proof.
We start with the first part of the assertion. From (20) it follows that b
i
d
j
>
b
j
d
i for i
<
j. Since b
i
d
j
-
b
j
d
i
>
0 for i
<
j it follows from (19) that
(22)
b
i
d
j
-
b
j
d
i
a
i
≥
b
i
d
j
-
b
j
d
i
a
j
,
for
i
<
j
.
Summing (22) over all i and j with 1
≤
i
<
j
≤
m we obtain
(23)
∑
i
<
j
a
i
b
i
d
j
+
∑
i
<
j
a
j
b
j
d
i
≥
∑
i
<
j
a
i
b
j
d
i
+
∑
i
<
j
a
j
b
i
d
j
.
Adding ∑
i
=
1
m
‍
a
i
b
i
d
i to both sides of (23) we obtain
(24)
∑
i
=
1
m
a
i
b
i
∑
j
=
1
m
d
j
≥
∑
i
=
1
m
a
i
d
i
∑
j
=
1
m
b
j
.
Since ∑
j
=
1
m
‍
b
j and ∑
j
=
1
m
‍
d
j are positive, inequality (24) is equivalent to (21). Finally, note that if two a
i are distinct, then (22) and hence (24) are strict.Theorem 2.
Letp
,
q
>
0 be real numbers with p
<
q. If condition (13) holds, then S
p
≤
S
q, with equality, if and only if all a
j in (13) are equal.Proof.
Using (4) we have S
p
≤
S
q if and only if
(25)
∑
i
=
1
c
∑
j
=
1
c
π
i
j
i
-
j
p
∑
i
=
1
c
∑
j
=
1
c
i
-
j
p
≥
∑
i
=
1
c
∑
j
=
1
c
π
i
j
i
-
j
q
∑
i
=
1
c
∑
j
=
1
c
i
-
j
q
.
Since |
i
-
j
|
=
0 for i
=
j we have the identities
(26)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
π
i
j
=
∑
j
=
1
c
-
1
j
r
∑
i
=
1
c
-
j
π
i
,
i
+
j
+
π
i
+
j
,
i
=
2
∑
j
=
1
c
-
1
j
r
c
-
j
a
j
,
(27)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
=
2
∑
j
=
1
c
-
1
j
r
c
-
j
.
Using (26) and (27), inequality (25) can be written as
(28)
∑
j
=
1
c
-
1
j
p
c
-
j
a
j
∑
j
=
1
c
-
1
j
p
c
-
j
≥
∑
j
=
1
c
-
1
j
q
c
-
j
a
j
∑
j
=
1
c
-
1
j
q
c
-
j
.
Let b
j
=
j
p
(
c
-
j
) and d
j
=
j
q
(
c
-
j
) for 1
≤
j
≤
c
-
1. Because 0
≤
p
<
q, the quantity b
j
/
d
j
=
j
p
-
q is strictly decreasing in j. Since (19) and (20) hold, validity of inequality (28) and thus validity of the inequality S
p
≤
S
q follow from application of Lemma 1.


## 4. Tridiagonal Agreement Tables

In practice it frequently occurs that an agreement table with ordered categories is (approximately) tridiagonal. A tridiagonal table is a square matrix that has nonzero elements only on the main diagonal, the first diagonal below this, and the first diagonal above the main diagonal. If the agreement table is tridiagonal there is only disagreement between the observers on adjacent categories. In this section we present results that hold if the agreement table is tridiagonal. In this case we havea
2
=
a
3
=
⋯
=
0 and it follows that condition (13) holds. It may be that the results also hold if condition (13) is valid. Note that Theorem 2 is always valid for tridiagonal agreement tables.The tridiagonal tables in Table2 have a
^
2
=
0 if c
=
3, a
^
2
=
a
^
3
=
0 if c
=
4, and a
^
2
=
a
^
3
=
a
^
4
=
0 if c
=
5. For 6 of the 20 entries in Table 2 the agreement table is tridiagonal. In Table 3 the tridiagonal tables have a
^
2
=
a
^
3
=
a
^
4
=
0. The agreement tables of the pairs (B,E) and (D,G) are tridiagonal. Many other agreement tables corresponding to the entries of Tables 2 and 3 are approximately tridiagonal: only a few disagreements are not on the diagonals directly below and above the main diagonal.


### 4.1. Upper Bound Unity

Theorem2 and Tables 2 and 3 show that S
r is usually increasing in r. This implies that Bennett et al.’s S is usually a lower bound of the other special cases of S
r. Furthermore, it suggests that S
r goes to unity as r increases, regardless of the data at hand. Theorem 3 formalizes this observation for tridiagonal agreement tables.Theorem 3.
Ifc is fixed and π
i
j is tridiagonal, then S
r
→
1 as r
→
∞.Proof.
Ifπ
i
j is tridiagonal, (4) becomes
(29)
S
r
=
1
-
c
2
∑
i
=
1
c
-
1
π
i
,
i
+
1
+
π
i
+
1
,
i
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
.
Since the elements of π
i
j sum to unity, we have the inequality
(30)
c
2
∑
i
=
1
c
-
1
π
i
,
i
+
1
+
π
i
+
1
,
i
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
≤
c
2
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
.
The right-hand side of (30) does not depend on the data. Since the denominator ∑
i
=
1
c
‍
∑
j
=
1
c
‍
|
i
-
j
|
r is increasing in r, we can, for fixed c, make the right-hand side of (30) arbitrary small. Hence, S
r
→
1 as r
→
∞.


### 4.2. An Inequality for Differences

Since the parameterr in (4) is a nonnegative real number there are uncountably infinite many special cases of S
r. Theorems 2 and 3 together with Tables 2 and 3 show that all these special cases usually lie between S
0 and 1. Tables 2 and 3 also show that the positive differences S
1
-
S
0 and S
2
-
S
1 are quite substantial. This suggests that most elements of the sequence (
S
0
,
S
1
,
S
2
,
…
) will lie close to 1 and that consecutive differences S
1
-
S
0
,
S
2
-
S
1
,
S
3
-
S
2
,
… become smaller and smaller.In this section we present a particular result for the positive differencesS
1
-
S
0 and S
2
-
S
1. Theorem 5 below shows that S
2
-
S
1 never exceeds S
1
-
S
0. We first derive explicit formulas of ∑
i
=
1
c
‍
∑
j
=
1
c
‍
|
i
-
j
|
r for r
=
0,1
,
2 in Lemma 4.Lemma 4.
It holds that(31)
∑
i
=
1
c
∑
j
=
1
c
1
i
≠
j
=
c
c
-
1
;
(32)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
=
c
c
2
-
1
3
;
(33)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
2
=
c
2
c
2
-
1
6
.Proof.
We only present proofs of identities (32) and (33). We will use the following identities of sums of powers of integers (see, e.g., [43]):
(34)
∑
k
=
1
m
k
=
m
m
+
1
2
;
(35)
∑
k
=
1
m
k
2
=
m
m
+
1
2
m
+
1
6
;
(36)
∑
k
=
1
m
k
3
=
m
2
m
+
1
2
4
.
Using identities (27), (34), and (35) we have
(37)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
=
2
∑
j
=
1
c
-
1
c
-
j
j
=
2
c
∑
j
=
1
c
-
1
j
-
2
∑
j
=
1
c
-
1
j
2
=
c
2
c
-
1
-
c
c
-
1
2
c
-
1
3
=
c
c
2
-
1
3
.
Furthermore, using identities (27), (35), and (36) we have
(38)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
2
=
2
∑
j
=
1
c
-
1
c
-
j
j
2
=
2
c
∑
j
=
1
c
-
1
j
2
-
2
∑
j
=
1
c
-
1
j
3
=
c
2
c
-
1
2
c
-
1
3
-
c
2
c
-
1
2
2
=
c
2
c
2
-
1
6
.Theorem 5.
Ifπ
i
j is tridiagonal, then S
1
-
S
0
≥
S
2
-
S
1, with equality, if and only if c
=
2,3.Proof.
The formula ofS
r for the case that π
i
j is tridiagonal is given in (29). The inequality S
1
-
S
0
≥
S
2
-
S
1 or 2
S
1
≥
S
0
+
S
2 is equivalent to
(39)
2
∑
i
=
1
c
∑
j
=
1
c
i
-
j
≤
1
∑
i
=
1
c
∑
j
=
1
c
1
i
≠
j
+
1
∑
i
=
1
c
∑
j
=
1
c
i
-
j
2
.
Using identities (31), (32), and (33), inequality (39) becomes
(40)
6
c
c
2
-
1
≤
1
c
c
-
1
+
6
c
2
c
2
-
1
=
c
c
+
1
+
6
c
2
c
2
-
1
,
or, equivalently, c
2
-
5
c
+
6
=
(
c
-
2
)
(
c
-
3
)
≥
0.If the agreement table is tridiagonal, Theorem5 shows that S
1
-
S
0
=
S
2
-
S
1 if and only if c
=
2 or c
=
3. Recall that for c
=
2 categories all special cases of coefficient S
r coincide. Thus, for c
=
2, we have S
1
-
S
0
=
S
2
-
S
1
=
0. Furthermore, an illustration for c
=
3 categories is the second entry of Table 2. For this entry we have S
^
0
=
0.95775, S
^
1
=
0.96831, S
^
2
=
0.97887, and S
1
-
S
0
=
S
2
-
S
1
=
0.01056.


### 4.3. Dependence on Number of Categories

A criticism against the use of Bennett et al.’sS is that the coefficient tends to produce higher values for agreement tables with more categories [26]. More precisely, if the raw agreement ∑
i
=
1
c
‍
π
i
i is constant in (7) we have S
0
→
∑
i
=
1
c
‍
π
i
i as c
→
∞. Thus, if the rating scale has many categories, we have S
0
≈
∑
i
=
1
c
‍
π
i
i, and S
0 is not a chance-corrected coefficient.WhileS
0 has the raw agreement ∑
i
=
1
c
‍
π
i
i as an upper bound, it appears that the values of many other special cases of S
r tend to go to unity as the number of categories of the rating scale increases. For example, suppose that π
i
j is tridiagonal and that ∑
i
=
1
c
‍
π
i
i
=
2
/
3. Using (29) and (35) the formula of S
1 is given by
(41)
S
1
=
1
-
c
c
2
-
1
.
We have S
1
→
1 as c
→
∞. Since S
1 is usually a lower bound for all special cases of S
r with r
>
1 (Theorem 2), it follows that all coefficients with r
≥
1 go to unity as the number of categories increases. Dependence on the number of categories is considered an undesirable property of S
r.


## 4.1. Upper Bound Unity

Theorem2 and Tables 2 and 3 show that S
r is usually increasing in r. This implies that Bennett et al.’s S is usually a lower bound of the other special cases of S
r. Furthermore, it suggests that S
r goes to unity as r increases, regardless of the data at hand. Theorem 3 formalizes this observation for tridiagonal agreement tables.Theorem 3.
Ifc is fixed and π
i
j is tridiagonal, then S
r
→
1 as r
→
∞.Proof.
Ifπ
i
j is tridiagonal, (4) becomes
(29)
S
r
=
1
-
c
2
∑
i
=
1
c
-
1
π
i
,
i
+
1
+
π
i
+
1
,
i
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
.
Since the elements of π
i
j sum to unity, we have the inequality
(30)
c
2
∑
i
=
1
c
-
1
π
i
,
i
+
1
+
π
i
+
1
,
i
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
≤
c
2
∑
i
=
1
c
∑
j
=
1
c
i
-
j
r
.
The right-hand side of (30) does not depend on the data. Since the denominator ∑
i
=
1
c
‍
∑
j
=
1
c
‍
|
i
-
j
|
r is increasing in r, we can, for fixed c, make the right-hand side of (30) arbitrary small. Hence, S
r
→
1 as r
→
∞.


## 4.2. An Inequality for Differences

Since the parameterr in (4) is a nonnegative real number there are uncountably infinite many special cases of S
r. Theorems 2 and 3 together with Tables 2 and 3 show that all these special cases usually lie between S
0 and 1. Tables 2 and 3 also show that the positive differences S
1
-
S
0 and S
2
-
S
1 are quite substantial. This suggests that most elements of the sequence (
S
0
,
S
1
,
S
2
,
…
) will lie close to 1 and that consecutive differences S
1
-
S
0
,
S
2
-
S
1
,
S
3
-
S
2
,
… become smaller and smaller.In this section we present a particular result for the positive differencesS
1
-
S
0 and S
2
-
S
1. Theorem 5 below shows that S
2
-
S
1 never exceeds S
1
-
S
0. We first derive explicit formulas of ∑
i
=
1
c
‍
∑
j
=
1
c
‍
|
i
-
j
|
r for r
=
0,1
,
2 in Lemma 4.Lemma 4.
It holds that(31)
∑
i
=
1
c
∑
j
=
1
c
1
i
≠
j
=
c
c
-
1
;
(32)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
=
c
c
2
-
1
3
;
(33)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
2
=
c
2
c
2
-
1
6
.Proof.
We only present proofs of identities (32) and (33). We will use the following identities of sums of powers of integers (see, e.g., [43]):
(34)
∑
k
=
1
m
k
=
m
m
+
1
2
;
(35)
∑
k
=
1
m
k
2
=
m
m
+
1
2
m
+
1
6
;
(36)
∑
k
=
1
m
k
3
=
m
2
m
+
1
2
4
.
Using identities (27), (34), and (35) we have
(37)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
=
2
∑
j
=
1
c
-
1
c
-
j
j
=
2
c
∑
j
=
1
c
-
1
j
-
2
∑
j
=
1
c
-
1
j
2
=
c
2
c
-
1
-
c
c
-
1
2
c
-
1
3
=
c
c
2
-
1
3
.
Furthermore, using identities (27), (35), and (36) we have
(38)
∑
i
=
1
c
∑
j
=
1
c
i
-
j
2
=
2
∑
j
=
1
c
-
1
c
-
j
j
2
=
2
c
∑
j
=
1
c
-
1
j
2
-
2
∑
j
=
1
c
-
1
j
3
=
c
2
c
-
1
2
c
-
1
3
-
c
2
c
-
1
2
2
=
c
2
c
2
-
1
6
.Theorem 5.
Ifπ
i
j is tridiagonal, then S
1
-
S
0
≥
S
2
-
S
1, with equality, if and only if c
=
2,3.Proof.
The formula ofS
r for the case that π
i
j is tridiagonal is given in (29). The inequality S
1
-
S
0
≥
S
2
-
S
1 or 2
S
1
≥
S
0
+
S
2 is equivalent to
(39)
2
∑
i
=
1
c
∑
j
=
1
c
i
-
j
≤
1
∑
i
=
1
c
∑
j
=
1
c
1
i
≠
j
+
1
∑
i
=
1
c
∑
j
=
1
c
i
-
j
2
.
Using identities (31), (32), and (33), inequality (39) becomes
(40)
6
c
c
2
-
1
≤
1
c
c
-
1
+
6
c
2
c
2
-
1
=
c
c
+
1
+
6
c
2
c
2
-
1
,
or, equivalently, c
2
-
5
c
+
6
=
(
c
-
2
)
(
c
-
3
)
≥
0.If the agreement table is tridiagonal, Theorem5 shows that S
1
-
S
0
=
S
2
-
S
1 if and only if c
=
2 or c
=
3. Recall that for c
=
2 categories all special cases of coefficient S
r coincide. Thus, for c
=
2, we have S
1
-
S
0
=
S
2
-
S
1
=
0. Furthermore, an illustration for c
=
3 categories is the second entry of Table 2. For this entry we have S
^
0
=
0.95775, S
^
1
=
0.96831, S
^
2
=
0.97887, and S
1
-
S
0
=
S
2
-
S
1
=
0.01056.


## 4.3. Dependence on Number of Categories

A criticism against the use of Bennett et al.’sS is that the coefficient tends to produce higher values for agreement tables with more categories [26]. More precisely, if the raw agreement ∑
i
=
1
c
‍
π
i
i is constant in (7) we have S
0
→
∑
i
=
1
c
‍
π
i
i as c
→
∞. Thus, if the rating scale has many categories, we have S
0
≈
∑
i
=
1
c
‍
π
i
i, and S
0 is not a chance-corrected coefficient.WhileS
0 has the raw agreement ∑
i
=
1
c
‍
π
i
i as an upper bound, it appears that the values of many other special cases of S
r tend to go to unity as the number of categories of the rating scale increases. For example, suppose that π
i
j is tridiagonal and that ∑
i
=
1
c
‍
π
i
i
=
2
/
3. Using (29) and (35) the formula of S
1 is given by
(41)
S
1
=
1
-
c
c
2
-
1
.
We have S
1
→
1 as c
→
∞. Since S
1 is usually a lower bound for all special cases of S
r with r
>
1 (Theorem 2), it follows that all coefficients with r
≥
1 go to unity as the number of categories increases. Dependence on the number of categories is considered an undesirable property of S
r.


## 5. Discussion

Bennett et al.’s [23] S is an agreement coefficient for ratings scales with nominal categories that has been discovered and rediscovered by many authors [12, 28, 29]. Recently, a weighted version of S was proposed by Gwet [33] for rating scales with ordinal categories. In this paper we presented various properties of a special case of this weighted version, denoted by S
r, where r is a nonnegative real number. Bennett et al.’s [23] S corresponds to S
0, while S
1 and S
2 are the versions of S
r that are obtained by using, respectively, the linear and quadratic weighting schemes.It was first studied how the different versions ofS
r are related. Theorem 2 shows that S
r is increasing in r if the average disagreement between the observers on adjacent categories is greater than the average disagreement on categories that are 2 steps apart and if the latter is greater than the average disagreement on categories that are 3 steps apart and so on. Hence, in this case, there is a simple relationship between the values of the special cases of S
r. It turns out that Theorem 2 is quite a strong result. First of all, the result involves all special cases of S
r, and there are uncountably infinite many versions of S
r. Secondly, the sufficient condition holds for many data tables reported in this paper (see Tables 2 and 3). Since S
r is usually increasing in r, its special cases are essentially measuring the same thing.For the application of Cohen’s kappa and weighted kappa, various authors have presented target values for evaluating the values of the kappa coefficients [44–47]. There is general consensus in the literature that uncritical application of such magnitude guidelines leads to practically questionable decisions. Warrens [48] argued that, since quadratic kappa produces values that are substantially higher than the values produced by Cohen’s kappa, the same guidelines cannot be used for both coefficients. A similar argument applies here. Tables 2 and 3 show that coefficients S
0, S
1, and S
2 produce quite different values. Thus, despite the fact that the coefficients measure the same thing, they do this to a different extent. If one accepts the use of magnitude guidelines, different criteria need to be developed for the different special cases of S
r.Finally, a number of results were presented that illustrate that many special cases ofS
r tend to produce values close to unity, regardless of the data at hand. This is especially the case when r is high (r
≥
2) and the number of categories of the rating scale is large (5 or more). These results were only proved for agreement tables that are tridiagonal, but the high estimates in Tables 2 and 3 suggest that the results hold under more general conditions. The dependence of S
r on the number of categories implies that different criteria need to be formulated depending on the number of categories. Developing different criteria for different coefficients and different number of categories seems an impossible task. Hence, coefficient S
r is useless as a general agreement coefficient. It is advised to limit the application of weighted versions of Bennett et al. S to one or two coefficients, for example, S
0 for rating scales with nominal categories and S
1 for rating scales with ordinal categories.



---

*Source: 231909-2014-12-03.xml*