Evaluating the Quality of Analytic Ratings with Mokken Scaling

AbstractGreatly influenced by the work of Rasch (1960/1980), Mokken (1971) presented a nonparametric scaling procedure that is based on the theory of invariant measurement, but draws upon less strict requirements related to the scale of measurement. Because they are theoretically and empirically related to Rasch models, Mokken's nonparametric models have been recognized as a useful exploratory tool for examining data in terms of the basic requirements for invariant measurement before the application of a parametric model. In particular, recent research has explored the use of polytomous versions of Mokken's (1971) nonparametric scaling models as a technique for evaluating the quality of holistic ratings (Wind & Engelhard, in press) and rating scales (Wind, 2014) for performance assessments in terms of the requirements for invariant measurement. The current study continues the extension of Mokken scaling to performance assessments by exploring the degree to which Mokken-based rating quality indices can be used to explore the quality of ratings assigned within domains on an analytic rubric. Using an illustrative analysis, this study demonstrates the use of a generalized rating design to explore the quality of analytic ratings within the framework of Mokken scaling. Findings from the illustrative analysis suggest that a generalized rating design can be used to examine the quality of analytic ratings in terms of the requirements for invariant measurement.Key words: Mokken scaling; analytic rubric; performance assessment; nonparametric item response theory(ProQuest: ... denotes formula omitted.)Two major scoring systems are widely employed in large-scale rater-mediated performance assessments: holistic and analytic scoring. Holistic scoring systems involve the use of rubrics whose performance-level descriptors (PLDs) focus on the evaluation of a student's performance as a whole. Although multiple components of a response may be considered, a single score is assigned to reflect the overall quality of the performance. In contrast, analytic scoring systems involve the use of rubrics with PLDs specified for multiple domains on which separate scores are assigned to reflect specific components of a student's performance. For example, an analytic scoring rubric in a large-scale writing assessment might include domains related to style, organization, and conventions. As noted in the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014), the use of analytic scoring rubrics "provides valuable information for diagnostic purposes and lends itself to evaluating strengths and weaknesses of test takers. However, validation will be required for diagnostic interpretations for particular uses of the separate scores" (p. 79). Across both holistic and analytic scoring systems, the diagnostic value of rater-assigned scores for informing educational decisions depends on the degree to which scoring systems demonstrate useful psychometric properties, including validity, reliability, and fairness.Current operational methods for monitoring ratings typically include indices of rating quality that are classified within three major categories: 1) rater agreement, 2) rater errors and systematic biases, and 3) rater accuracy (Johnson, Penny, & Gordon, 2009; Murphy & Cleveland, 1991). Indices of rater agreement describe the degree to which raters assign matching (or in some cases, adjacent) scores to the same performance; rater errors and systematic biases are used to describe specific patterns or trends in rating behavior that are believed to contribute to the assignment of scores different from those warranted by a student's performance; rater accuracy is often defined in practice as a match between operational ratings and those established as "true" or "known" ratings by individuals or committees of expert raters (Johnson, Penny, & Gordon, 2009; Murphy & Cleveland, 1991). When indices of rating quality based on these three categories are applied, high levels of agreement, low levels of error and systematic bias, and high levels of accuracy are assumed to reflect high-quality ratings. …