Running Head : Comparison of Three Weighting Procedures A Comparison of Three Weighting Procedures for High-and Low-Stakes Examinations with Mixed Item Formats in Different Subject Areas

The interchangeability of scores yielded by three weighting procedures applied to low-stakes achievement tests and to high-stakes examinations containing both selected response (SR) items and constructed response (CR) items in Language Arts and Mathematics was examined. The three scoring procedures included an unweighted procedure in which scores from the set of SR items and the set of CR items/tasks were added; a weighted procedure in which the CR items were weighted so that the CR and SR items contributed equally; and pattern scoring in which each item was individually weighted. While the different weighting procedures yielded similar score distributions for all four tests at the group level, they were sufficiently dissimilar at the student level to warrant using then interchangeably. Pattern scoring provided the smallest standard errors, particularly at the lower end of the ability distribution. Whereas test stakes was not a factor, subject area may be a factor. Further, difference between the three score distributions suggest that care must be taken in choosing one weighting procedure over the others in a criterion-referenced situation, especially when a cut-score is set in the tail of the score distribution. Comparison of Three Weighting Procedures 3 A Comparison of Three Procedures for Weighting Selected and Constructed Items on Highand Low-Stakes Examinations in Different Subject Areas Most large-scale assessments today contain both selected response (SR) and constructed response (CR) items so as to obtain a more comprehensive assessment of the knowledge and skills possessed by students. An issue that arises with the inclusion of the two item types is how they should be represented or weighted in the total score (Haertel, 2006; Kolen, 2006). For example, a CR item, especially one that calls for an extended response, requires a longer response time than a single SR item. Consequently, given fixed administration times and the need to cover a broad domain, the number of CR items is less than the number of SR items in large-scale assessment instruments. Therefore, the number of score points derived from the CR items is generally less than the number of score points derived from the SR items. Without correcting for the unequal number of score points from each item type, the content and cognitive processes assessed by CR items may be de-emphasized in favor of the content and processes assessed by SR items. This is undesirable given the content and cognitive processes assessed by CR items are important attributes of the domain about which inferences are to be drawn. Weighting procedures can be used to address this apparent imbalance. Sometimes the weights are explicitly determined by policy. For example, a policy may require that the scores obtained from the SR and CR items be equal in value. To achieve the desired weighting, it may be necessary to explicitly weight the SR and CR items differently. Other times the policy is such that the items are weighted implicitly as in pattern scoring under a particular IRT model. For example, the weights that result when the twoor three-parameter IRT model is used for the SR items and Murkaki’s (1992) generalized Comparison of Three Weighting Procedures 4 partial credit model is used for the CR items are optimal in the sense of maximizing test information and minimizing the standard error of measurement, regardless of item format. Prior to the introduction of IRT models, judgmental procedures were employed to determine the weights to be applied. For example, after correcting for unequal standard deviations, the SR and CR components could be differentially weighted so that each contributed the same or a pre-determined and different number of points to the total score. Alternatively, if an external criterion was available, regression procedures could be used to obtain implicit weights to be applied to the subtest scores derived from the SR items and the subtest scores derived from the CR items that maximized the squared multiple correlation coefficient (Gulliksen, 1950). While the implicit weights employed in IRT scoring are often acceptable, there may be occasions where explicit weights are required within an IRT framework. Weighting of the CR response component may be required so that the contribution of the constructed responses to the total score matches what is required. One option is to multiply the portion of the Test Characteristic Curve contributed by the item or subset of items to be weighted. For example, if time constraints prevent the administration of two extended response items, then the expected score for the single extended response item can be multiplied by two and then added to the expected item scores for the remaining items to obtain the expected total score. Another alternative, employed by the Education Quality and Accountability Office to score the Ontario Secondary School Literacy Test, involves adding together the scores of the trained raters who score each of the two long writing items. The initial score range was changed from 0 to 8 to 0 to 16, and these Comparison of Three Weighting Procedures 5 “item” scores were input into the IRT analysis (EQAO, 2005). Thus, there are different options for explicitly weighting items in an IRT framework. Schaeffer, Henderson-Montero, Julian, and Bene (2002) were among the first to examine the effect on test scores of different weighting systems within an IRT framework. They examined three different ways of combining SR and CR scores yielded by low-stakes Grade 9 Biology and English field tests that contained both item types: unit weighting procedure in which a student’s total score was equal to the number of points earned from the SR items plus the number of points earned from the CR items, where the number of points was set by the test developer; a differential weighting in which the SR items and CR items contributed the same number of points toward the total score; and implicit weighting in which the SR and CR items were implicitly weighted according to the IRT models used. Items within each content area were calibrated using the 3PL model for SR items and the generalized (Muraki, 1992) partial credit model for the CR items so that the items were placed on the same scale simultaneously. The computer program PARDUX, which uses marginal maximum likelihood procedures implemented with the EM algorithm (Bock &Aitken, 1981), was used for this calibration. The item parameters were placed on the score scale using WINFLUX (Burket, 1999). The scaling parameters were a multiplier of 50 and an additive constant of 500. The scale scores were rounded to the nearest integer value and the lowest (LOSS) and highest (HOSS) possible scale scores were set at 300 and 700, respectively. Whereas the variances of the distributions, correlation patterns, and percentages of students in each of four hypothetical proficiency categories (English only) for the total group and the gender and ethnic subgroups considered were comparable, the scale score Comparison of Three Weighting Procedures 6 means for Biology were consistently lower then the scale score means for English across the three weighting procedures (495.1, 496.6, and 493.1 vs. 500.9, 501.2, and 503.7 for the total group). Implicit weighting provided the smallest standard errors, particularly at the lower end of the scale score distributions. Sykes and Hou (2003) also addressed the issue of different weighting procedures using a low-stakes Grade 8 field test for writing that contained both SR and CR items. In particular, they were interested in the influence of CR items and the way they are handled in a writing test. In addition to the three weighting procedures used by Schaeffer et al. (2002), they considered four additional procedures: a weighted score that deliberately increased the weighting of the CR items by a factor of two; a summed score that involved the sum of two raters’ scores on the CR items; a “long form” in which 18 SR items and eight CR items were added to the examination; and an “all SR item long form” in which 20 additional SR items were added and the CR items were removed. Item calibration was completed using PARDUX and WINFLUX as described above for Schaeffer et al. (2002). Sykes and Hou (2003) found that the test characteristic curves (TCC) for the different weighting procedures were, with the exception of the all multiple-choice form, quite similar. The all multiple-choice form TCC was above the other TCCs for scale scores below 375 and above 525, and was steeper within the middle of the ability scale due to increased item discrimination in the middle of the midrange of scores. Implicit weighting provided the smallest standard errors across the ability range for all of the forms containing CR items, particularly for scale scores less than 400. Differences at the 10 and 90 percentiles between the operational form and each of the other forms varied, Comparison of Three Weighting Procedures 7 respectively, between -13.0 and -34.0 and 13.5 and 32.0 The all multiple-choice form was found to have the highest test reliability (α̂ = 0.90), with the two summed CR forms and the CRx2 form having slightly lower and equal reliabilities (α̂ = 0.84). Low-Stakes Tests versus High-Stakes Examinations Large-scale assessments are often described as low-stakes and high-stakes. For example, the Primary, Junior and Grade 9 assessments in Ontario are considered to be low-stakes and the Ontario Secondary School Literacy Test is considered to be hightakes. Whereas student level data are provided at the student level for all of these assessments, the student level data does not count toward a school mark at the Primary and Junior levels and between 0 and 30% at the Grade 9 level, with the decision as to how much made by individual teachers. In contrast, students must pass the OSSLT or take a special literacy course after failing the OSSLT at least once in order to satisfy graduati