When to use Scott’s π or Krippendorff's α, if ever?

Scott’s π (1955) and Krippendorff's α (1980) have been among the most-often used or recommended general indicators of reliability. This article presents paradoxes showing that neither can be a general indicator. We show that π or α should be used only when (a) coders enforce a predetermined quota as the first priority and (b) coders maximize chance coding as the second priority. Because the two conditions rarely hold, π or α should rarely be used. When to Use Scott’s π or Krippendorff's α, If Ever? Page 4 of 38 WHEN TO USE SCOTT’S π OR KRIPPENDORFF’S α, IF EVER? Inter-coder or test-retest reliability is widely used to assess measurement quality, in such disciplines as communication, medical studies, and marketing research. Various methods have been proposed to calculate reliability, including Yule’s Y (1912), Bennett et al’s S (1954), Scott’s π (1955), Osgood’s index (1959), Cohen’s κ (1960), Holsti’s coefficient (1969), Maxwell’s RE (1977), Krippendorff's α (1980), Perreault & Leigh’s Ir (1989), Feinstein & Cicchetti’s ψ (1990b), Potter & Levine-Donnerstein’s redefined π (1999), and Gwet’s γ (2006). Each has been recommended and used as the general indicator of the same concept, reliability. Yet they offer different formulas and produce different, sometimes drastically different, results. This paper focuses on two of them, Scott's π and Krippendorff's α. I. Reliability vs. Reliabilities Across various disciplines of social sciences and medical studies, Cohen’s κ (1960), which I have discussed in a separate paper, is the most often used indicator of reliability. Scott’s π (1955) is the second most popular, especially among communication researchers. In Social Sciences Index (SSCI), between 1994 and 2010, Scott (1955) was cited 261 times and Krippendorff (1980) 243 times. During the same period, in Communication and Mass Media Complete (CMMC), “Scott’s Pi” had 597 citations, which rose from 11 in 1994 to 61 in 2009; “Krippendorff’s Alpha” had 145 citations, which also rose from 2 in 1994 to 26 in 2009. At the time of our search, the full year statistics were not yet available for 2010. If Krippendorff’s α has not been the most frequently used, it may be among the most strongly recommended. Respected communication methodologists such as Hayes & Krippendorff (2007) and Krippendorff (2004b) argued that α should be the only general indicator to use. When to Use Scott’s π or Krippendorff's α, If Ever? Page 5 of 38 The assumption is that the various indicators measure the same concept of reliability. Yet the indicators disagree with each other, as they produce different—sometimes drastically different—results, even when the underlying between-coder agreement is the same. As reliability means agreement (Riffe, Lacy, & Fico, 1998), these indicators of reliability themselves are not reliable, if they indeed measure the same concept. Under the assumption of “various indicators for one reliability,” methodologists debated which indicator is the best, whether to use any or several of them in a study, and how to fix or cope when some indicators, especially Cohen’s κ, behave unexpectedly (e.g., Brennan & Prediger, 1981; Zwick , 1988; Lombard, Snyder-Duch & Bracken, 2002; Krippendorff, 2004b). This paper ventures with a different assumption. Given that the indicators produce different results, there might be several reliability concepts, each having its own indicator. Only one can be the general indicator, while the others specialized. Under the assumption of “various indicators for various reliabilities,” we should analyze each indicator to determine whether it is general or special and, if special, to outline its boundaries. As I have discussed Cohen’s κ elsewhere, this paper will focus on Scott's π and Krippendorff’s α. I will first briefly review the formulas of π and α. I will then introduce 18 paradoxes to show that neither measures general reliability. Assuming that they might measure reliability in special circumstances, I will analyze the mathematics of π and α to pin point these circumstances. I will use the simplest case—binary scales with two coders. The conclusions also apply to nominal scales of more categories with more coders. II. Scott’s π and Krippendorff's α Agreement rate (ar) has been the simplest indicator of reliability recommended (Budd & Thorp 1963; Holsti 1969).