NAEP Validity Studies: Improving the Information Value of Performance Items in Large Scale Assessments. Working Paper No. 2003-08
Author: Pearson, P. David
Date: 2003-04-00
Type: Information Analyses
Subjects: Measurement,National Competency Tests,Test Items,Performance,Responses,Item Response Theory,Factor Analysis,Correlation,Test Interpretation,Research Problems,Educational Research,Multiple Choice Tests,Scoring Rubrics,Cognitive Processes,Difficulty Level,Scoring,Test Bias,Reading Tests

The purpose of this essay is to explore both what is known and what needs to be learned about the information value of performance items &quot;when they are used in large scale assessments.&quot; Within the context of the National Assessment of Educational Progress (NAEP), there is substantial motivation for answering these questions. Over the past decade, in order to adequately portray the breadth and depth of important curriculum standards, NAEP designers have invested substantial time and energy in creating extended constructed-response items and enormous financial resources in scoring these items. While these items are popular with curriculum experts within various content areas (Linn, Glaser, &amp; Bohrnstedt, 1997), it is not clear whether they possess the marginal utility required to justify their cost; that is, they may not provide new information above and beyond that which is provided by a more standard mix of multiple-choice and short items with known measurement characteristics and much more economical scoring protocols. Even worse, there is some reason to believe, based on research in non-NAEP settings, that extended constructed-response items may provide negative returns in terms of the overall goal of accurate measurement of performance on some broad domain such as reading, mathematics, or science (see Forsyth, Hambleton, Linn, Mislevy, and Yen, 1996). To investigate this claim, the authors traversed the &quot;information value&quot; terrain along as many paths as they could find--combing the measurement literature to determine the various ways in which scholars have conceptualized and operationalized the &quot;information value&quot; construct, reviewing research conducted within those approaches, consulting essays that emphasize the importance of strong conceptual grounding in content frameworks, studying the broadest available construal of the construct of validity (e.g., Messick, 1989), and, finally, and most unsatisifyingly, attempting to determine, at a conceptual and philosophical level, what the assessment community means when they talk about the information provided by assessments. This essay is organized into four sections. First, the authors consider the construct of information value in its broadest philosophical sense and then describe the classical ways of operationalizing it. Second, they review the available literature within each of these operational traditions (IRT, factor analysis, correlational studies). Third, they consider some alternative versions of information value, based more on cognitive, conceptual, and pragmatic considerations. Finally, they outline a series of studies that they believe ought to be supported by the National Center for Educational Statistics (NCES) in order to answer the question of how NAEP can be modified to increase its &quot;information value.&quot;