Effects of Scale Transformation and Test Termination Rule on the Precision of Ability Estimates in CAT. ACT Research Report Series.

Error indices (bias, standard error of estimation, and root mean square error) obtained on different scales of measurement under different test termination rules in a CAT context were examined. Four ability estimation methods (M L£, WLE, EAP, and MAP), three measurement scales (0 , number correct score, and ACT score scale), and three test termination rules (fixed length, fixed standard error, and target information) were studied. The findings indicate that the amount and direction of bias, standard error of estimation, and root mean square error obtained under different ability estimation methods is influenced both by measurement scale and by test termination rule in a CAT environment. WLE performed the best among the four ability estimation methods on the ACT score scale with a target information termination rule. Effects of Scale Transformation and Test Termination Rule on the Precision of Ability Estimates in CAT Computerized adaptive testing (CAT) is designed to construct a unique test for each examinee, so that the test targets the exam inee’s estimated ability level. Theoretically, CAT has many advantages over paper-and-pencil (P&P) tests. One often mentioned advantage is its measurement efficiency or capability to deliver shorter tests. CAT also provides examinees with the benefits of test on demand and immediate test scoring and reporting. With the recent development in computer technology and psychometric knowledge, the popularity of CAT is increasing. Several high-stake testing programs have implemented CAT versions of P&P tests (Eignor, Way, Stocking, & Steffen, 1993; Sands, Waters, & McBride, 1997). and some others are moving towards using CAT as an alternative test-delivery method (Miller & Davey, 1999). Although CAT has many advantages, there are issues that need to be considered in the application of CAT. For example, the effects of scale transformations and test termination rules on the precision of ability estimation methods have not yet been fully investigated. M ost computerized adaptive tests (CATs) use item selection and scoring algorithms that depend on item response theory (IRT). However, it may be difficult for the general public with limited psychometric knowledge to understand the meaning of the 0 scale. Thus, the measurement scales of CATs are often transformed from 0 to more familiar scales, such as number correct (NC) score or reported score scales (Stocking, 1996). The test termination rule is another important factor that has to be considered when CATs are implemented. The choice of a stopping rule is affected by several factors, for example, the comparability between P&P tests and CATs, cost (e.g., cost of computer sitting time), measurement efficiency, and among others. Because most CATs use IRT as the testing model, much of the previous research on CAT has been based on the IRT 0 scale. For example, studies that compared different ability estimation methods in terms of error indices, such as'b ias'standard error of estimation (SE), and root mean squared error (RMSE) of those methods, most often made the comparison on the 0 scale (e.g., Bock & Mislevv, 1982; Crichton, 1981; W ang & Vispoel, 1998; Wang, Hanson, & Lau, 1999; Warm, 1989; Weiss & MacBride, 1984). This tendency is quite natural because much’of the basic engine of CAT, such as the item selection algorithm, is based on IRT parameters that are directly related to the 0 scale. However, when CATs move to actual implementation, the final reported score scale is rarely a linear transformation of the 0 scale. For example, the GRE CAT uses the same score scale as the P&P version of the test, which is a nonlinear transformation of the estimated 0 to a NC score and then to a reported score (e.g., Eignor & Schaeffer, 1995; Eignor et al., 1993). One question arising is whether the previous research results regarding the properties of different ability estimation methods or other CAT components (e.g., test termination rule) are scale specific. A recent study indicated that the error indices obtained from the maximum likelihood estimation (MLE) method may be drastically different on the NC scale than on the 0 scale (Yi, 1998). Lord (1980, Chapter 6) provided a theoretical derivation of the effects of scale transformation on the shape of information function. He indicated that “the units in terms of which information is measured depend on the units used to measure ability. ” (p. 87). In a general IRT setting, SE is directly related to information function, thus Lord’s derivation provides an indication on how scale transformation can affect SE. However, it is not clear how scale transformation will affect the precision of different ability estimation methods in a CAT environment. Additionally, the effects of a nonlinear transformation of scale on bias have not been examined. Therefore, a major purpose of the current study was to systematically investigate the effects of nonlinear scale transformation on the properties of ability estimation methods in a CAT context. 2