A Demonstration of Using Person-Fit Statistics in Standard Setting.

The use of person-fit methods to determine the extent to which a panelist's ratings fit the item response theory (IRT) models used in the National Assessment of Educational Progress (NAEP) is demonstrated. Person-fit methods are statistical methods that allow the identification of nonfitting response vectors. To determine whether panelists' ratings fit the IRT models used in the NAEP, the 1(z) statistic (F. Drasgow, M. Levine, and E. Williams, 1985) was used. Rating data from the 1994 NAEP achievement level setting process were obtained for grade 12 geography, for which 29 panelists (primarily teachers) set levels. A response vector was created for each panelist for each achievement level using each of three p-value criteria and simulated item score string estimation (ISSE) values were created. The 1(z) statistic was calculated for each of the 27 response vectors associated with each of the 29 panelists. Means and standard deviations of the 1(z) distributions were computed for each cell of the experimental design and are presented in table form. Typically, they indicated that the simulated ISSE ratings or response vectors underfit the model. The results of this study provide preliminary information about the use of person-fit statistics in standard setting and are the basis for additional planned studied. (Contains 3 tables, 3 figures, and 15 references.) (SLD) ******************************************************************************** * Reproductions supplied by EDRS are the best that can be made * * from the original document. * ******************************************************************************** U, Co to PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY LIICK TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) 1 A Demonstration of Using Person-Fit Statistics in Standard Setting' Luz Bay Michael L. Nering ACT U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (EIC) ErIC:is document has been R reproduced as received from the person or organization originating it. Minor changes have been made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy In setting achievement levels' for the National Assessment of Educational Progress (NAEP), there has always been an assumption that panelists' ratings fit the NAEP IRT models? As stated in ACT, 1993, the NAEP Achievement Levels-Setting (ALS) process assumes that "The item response theory (IRT) analysis and scaling of the NAEP is sufficiently valid and precise so that the IRT parameter estimates can be used to map the panelists' judgments onto the NAEP scale. (p. 2)" This assumption has serious implications yet it has not been investigated by previous researchers. What if the Panelists' Ratings Do Not Fit the Model For the 1994 and 1996 ALS processes, achievement levels cutpoints were set by asking panelists to estimate the probability that a student performing at the borderline of each achievement level will answer each dichotomous item correctly. This item rating method is commonly referred to as the modified-Angoff method (##ref). A second method was used for polytomous items where panelists provided the average score of students performing at the borderline of each achievement level (basic, proficient, and advanced). This method is referred to as the mean estimation (ME) method (ACT, 1994). Suppose a panelist's rating for a dichotomous item fits a linear model. This scenario is depicted in the graph in Figure 1 where the S-shaped curve is the characteristic curve for the item and the line represents the linear model that governs the panelist's rating. Suppose B is where the panelists meant to set the cutpoint, thus giving the item a r rating. That rating will be translated to the score scale as B thus setting a cutpoint that is lower than where the panelist had intended. Person-fit statistics are typically used to determine the extent to which the ability estimate for an examinee represents the underlying latent trait of interest. If an examinee has a fit index that is relatively large, then we assume that his/her response does not follow the underlying test model. Likewise, if a panelist's ratings do not follow our underlying test model, then we assume that his/her ratings do not reflect cutscores that realistic determine what is basic, proficient and advanced. 'Paper presented in M.L. Nering (Moderator), Innovations in Person-fit Research. Related papers session at the meeting of the National Council of Measurment in Education, April 14-16, 1998, San Diego, CA. The research reported here was supported by contract #ZA97001001 from the National Assessment Governing Board to ACT. The Use of Person Fit to Model Panelist Fit For the 1998 NAEP in Civics and Writing, ACT proposed a new item rating method to set achievement levels cutpoints. In this method panelists are required to indicate whether a student performing at the borderline of each achievement level -Basic, Proficient or Advanced, is likely to respond to each dichotomous item correctly (i.e., Yes="1" and No="0"). Additionally, for each polytomous item scored from 0 to n, panelists are required to indicate the likely score of a student performing at the borderline of each achievement level. The ratings provided by each panelist for a mix of polytomous and dichotomous items create an item score string for a student performing at the borderline of each achievement level; thus, the name item score string estimation (ISSE) method. The cutpoints can then be computed using any method that is similar to methods used for computing scale scores for students, although the interest is on group statistics (i.e., cutpoints set by a group of judges). 3 Because each cutpoint represents the scale score of students performing at the borderline of each achievement level, a judge's ISSE ratings is a response vector of a student performing at the borderline. Since each cutpoint is computed using the same procedures that are used to compute student scores, person-fit methods may be used to determine whether a judge's ratings are congruent with the underlying IRT models. Thus, a statistic such as lz(Drasgow, Levine, & Williams, 1985) can be determined for each judge, where: 1 1 -E(1 ° ) [var(10)11 (1) The lo term in Equation 1 is defined by Levine and Rubin (1979) as simply the unstandardized person-fit statistic: " 10 = In nP;(0)uVi(0). n-u) (2) The expected value [E(1o)] variance [var(lo)] in Equation 1 can be respectively defined as: