Stochastic Models of Quality Control on Test Misgrading.

In the last decade, essay items have been incorporated in major educational assessments, such as the National Assessment of Educational Progress (NAEP) and the Third International Mathematics and Science Study (TIMSS) (Allen, Carlson, & Zelenak, 1999; Martin & Kelly, 1996). Meanwhile, classroom teachers are urged to use essay questions to complement multiple-choice items. Various responses generated from essay items demand a large amount of manpower in test grading. While no graders intend to make mistakes, accidental errors are likely to occur during the human operations (Wang, 1993). The purpose of this study is to examine the chance of test misgrading using appropriate models in statistics. The estimation of inadvertent grading errors can serve as a basis for quality control in educational and psychological measurements. Literature Review Statistical models have been sought to enhance quality control in various projects. In industrial statistics, quality control measures are adopted mainly to ensure the total number of inferior incidents below a threshold k. Bissell (1970) reviewed, Incident counts form an important class of data, arising particularly in manufacturing processes and accident studies.... It is often assumed that such events follow the Poisson Law. The assumptions of constant mean level and independence are often violated in practice. (p. 215) In educational and psychological measurements, test misgrading can be treated as a specific type of incidents. In a classroom setting, Lyman (1998) noted that "Every teacher recognizes that grades are somewhat arbitrary and subjective" (p. 107). In a large-scale assessment, it is even more difficult to assume the same level of average performance among various graders. Accordingly, the assumption of a constant mean performance level is often violated in small- and large-scale assessments, which makes the Poisson model unsuitable for most real-life applications (Rasch, 1980; Wang, 1993). Whenever the assumption of Poisson distribution does not hold, statisticians tend to adopt alternative models to strengthen the quality control process. In particular, Johnson and Kotz (1969) pointed out, "The negative binomial distribution is very often a first choice as alternative when it is felt that a Poisson distribution might be inadequate" (p. 125). Edward and Gurland (1961) compared a class of distributions applicable to accidents, and reported that "the negative binomial gives an appreciably better fit than the Poisson distribution" (p. 504). Nonetheless, the negative binomial model has yet to be adopted in education to analyze test misgrading (Rasch, 1980; Wang, 1993). In contrast, researchers in other fields have applied the negative binomial distribution on a wide range of topics. Barnwal and Paul (1988) reviewed applications of negative binomial models on count data, and noted that "Count data which follow the negative binomial distribution arise in numerous areas of biostatistics (Anscombe, 1949; Bliss & Fisher, 1953; Bliss & Owen, 1958; McCaughran & Arnold, 1976)" (p. 215). In military industries, much earlier applications have been made by Greenwood and Yule (1920). They reported that the negative binomial distribution gave a better fit than did the Poisson distribution to accidents in munitions factories in England during the First World War. Besides the count data, Ross and Preece (1985) added that "The negative binomial distribution is often appropriate for data for aggregated organism; it can arise from various different models (Anscombe, 1950, p. 360; Bliss, 1953, p. 185ff; Boswell & Patil, 1970; Freeman, 1980)" (p. 323). The various applications have resulted in different presentations of the negative binomial distribution. Consequently, as was noted by Barnwal and Paul (1988), "Different authors have expressed the negative binomial distribution in different forms" (p. …