A Comparison of Scoring Options for Omitted and Not-Reached Items through the Recovery of IRT Parameters When Utilizing the Rasch Model and Joint Maximum Likelihood Estimation.

This paper utilized the Rasch model and Joint Maximum Likelihood Estimation to study different scoring options for omitted and not-reached items. Three scoring treatments were studied. The first method treated omitted and not-reached items as “ignorable/blank”. The second treatment, scored omits as incorrect with “0” and left not-reached as blank and the third treatment scored both omitted and not-reached items as incorrect with “0”. These scoring treatments were studied across two levels of missing data: “. 81% omit rate/10% not-reached” and “1.62% omit rate/20% not-reached” and two levels of not reaching the end of the test: independent of ability and dependent on ability. In terms of Bias, RMSD, and the number of outlier differences between estimated and “true” parameters, performance was best when omitted and not-reached items were ignored or left blank. Performance was poorest when omitted and not-reached items were scored as incorrect with “0”. A Comparison of Scoring Options for Omitted and Not-Reached Items Through the Recovery of IRT Parameters When Utilizing the Rasch Model and Joint Maximum Likelihood Estimation Background Several decisions are required before test items and examinee abilities can be calibrated. Choices regarding the most appropriate Item Response Theory (IRT) model, the IRT software and estimation method to be utilized and the most appropriate scoring treatment for omitted and not-reached items are some of the most important decisions to be made. The primary focus of this paper is the study and comparison of scoring options for omitted and not-reached items when the Rasch model and Joint Maximum Likelihood Estimation (JMLE) are utilized. By definition, an omitted item is one which an examinee chooses not to answer because he or she feels that they lack sufficient knowledge to answer the item correctly. Instead of guessing at a response, the examinee chooses to leave the item unanswered. In contrast not-reached items are those that an examinee does not respond to because they are unable to finish the test in the allotted time. Through the utilization of models incorporating IRT, investigators have studied the optimal method for scoring missing data not only in terms of the scoring option chosen but also in terms of the estimation method used. Joint Maximum Likelihood Estimation (JMLE) and Marginal Maximum Likelihood Estimation (MMLE) are two common estimation methods used within IRT. With JMLE, item and ability parameters are estimated simultaneously with an item’s difficulty estimate being dependent upon only those examinees who responded to that particular item and an examinee’s ability estimate being dependent upon only the set of items that were responded to by that examinee. In contrast, MMLE treats the ability distribution as known and uses this distribution in the estimation of the item parameters. Once the item parameters are estimated, the examinee abilities are then directly estimated. Several studies (Lord 1974, Mislevy & Wu 1988, DeAyala, Plake & Impara 2001) have investigated the use of MMLE in the treatment of missing data and have recommended that omitted items should not be ignored or left missing in their scoring treatment. These studies also note that scoring omitted items as incorrect (0) is problematic since even the lowest ability examinee has some probability of getting an omitted item correct. The general recommendation is to score omitted items as fractionally correct. With respect to not-reached items, Lord (1980) and Mislevy (1988) suggested that these items as opposed to omitted items could be ignored in their scoring treatment as long the items are answered in sequential order and the test is "nearly nonspeeded". In studies utilizing the Rasch Model and JMLE, Ludlow & Leary (1999) found that treating omitted items as “ignorable/blank” inflated the directly estimated abilities for examinees who omitted items and deflated the item difficulties for items with omitted data. Likewise, scoring not-reached items as incorrect with a 0, deflated the directly estimated abilities for examinees who did not reach the end of the test and inflated the item difficulties for items positioned at the end of the test. A two stage process was recommended that first incorporated an item calibration with omitted items being treated as incorrect and not-reached items scored as blank/missing. This would be followed by a person calibration with omitted and not-reached items scored as incorrect with a zero. Shin (2009) studied the impact of different scoring methods on IRT-based true score equating. The study recommended that omitted and not-reached items should be ignored and left blank and should not be scored as incorrect. The study also found that the benefits of treating omitted and not-reached items as “ignorable/blank” increased as the sample size increased. Demars (2003) utilized the one-parameter model (1PL) and both JMLE and MMLE to investigate the potential link between ability differences and not-reached rates and found that when not-reached items were left blank/missing and not-reached rates were independent of ability, both JMLE and MMLE produced unbiased item parameter estimates. However, when not-reached rates were linked to examinee ability, JMLE produced unbiased item parameter estimates while MMLE produced difficulty estimates that were consistently lower than the true item parameter estimates. DeMars concluded that although JMLE theory allows for ignoring missing data, MMLE does so only if notreached rates are random or independent of examinee ability. In contrast to the work described above, Koretz, Lewis, Skewes-Cox, and Burstein (1993) utilized data from the 1990 NAEP mathematics assessment to study the relationship between missing data and item as well as population group characteristics. They found that differences in omit and not-reached rates were partially attributable to ability/proficiency differences. This study implements the Rasch model as operationalized through WINSTEPS 3.65 which utilizes JMLE (also called UCON). With the Rasch Model the likelihood function is conditioned on the raw score as a sufficient statistic for the ability parameter. This is reflected in a subtle difference in the manner in which abilities are estimated relative to IRT models in general. With the Rasch model, ability estimates are adjusted until the difference between an examinee’s expected score and his or her observed raw score is sufficiently small. With IRT models, ability estimates are continually adjusted until the difference between an examinee’s computed item probabilities and his or her item response vector is negligibly small. In both cases, ability estimation is discontinued once a logit convergence criterion has been met. Three different scoring methods were investigated. First, omitted and not-reached items were ignored or left blank, second, they were scored as incorrect with item scores of zero and third, omitted items were scored as incorrect and not-reached items were left blank. These three scoring methods were evaluated under two conditions. The first being when not-reached rates were independent of ability. In this case whether an examinee reached the end of the test or not was simply a matter of random selection. The second condition was when not-reached rates were dependent on ability with lower ability examinees having a greater likelihood of not reaching the end of the test. Lastly, these three scoring methods with not-reached being independent and dependent on ability were evaluated across two levels of missing data (“.81% omit rate\10% not-reached” and “1.62% omit rate\20% not-reached”). Method The basic simulation and calibration process involved the creation and calibration of 24 data sets. This simulation and calibration process which is described below was executed 10 times for a total of 240 data sets calibrated across 240 runs. Data were first simulated for 500 examinees across 40 items using WINGEN2. Two data sets were simulated. The first data set was simulated utilizing the 1PL with the mean and standard deviation (SD) for items and abilities set to 0 and 1 respectively and the second data set utilized the two-parameter model (2PL) with the mean and SD for items and abilities set to 0 and 1 accompanied by an item discrimination parameter modeled to have a mean of 1 with an SD of . 25. In both cases the items were sequenced from easiest to hardest before response data were simulated. These “complete” data sets were then calibrated with the resulting item and ability parameter estimates being treated as the “true” parameter estimates. Omitted and not-reached data were then programmatically embedded to create additional data sets to include two levels of “missing” data. The first level (“.81% omit rate\10% not-reached”) incorporated an omit rate of . 81% that was modeled after a paper-pencil format version of a Math and Reading/ELA test that was administered by a large school district in the southwest United States at grades 3-12 during the fall of 2010. The 10% not-reached rate was modeled after a paper-pencil format version of the Math and Reading components of a commonly used standardized test that is administered at grades 3-8. For the (“.81% omit rate\10% not-reached”) level, each item was omitted an average of 4 times (0.81% of all examinees) with 10% of the examinees not reaching the end of test. For the second level both the omit and not-reached rates were doubled so that a more extreme “missing” condition could be studied (1.62% omit rate\20% notreached”). In this second level, each item was omitted an average of 8 times (1.62% of all examinees) with 20% of the examinees not reaching the end of the test (Appendix tables A.1 and A.3). Items were selected for omission randomly and were programmatically embedded according to the notion that omits could also be guessed at. Hence item data that had originally been scored as 1 was selected