bluuebunny/crossref_metadata_2025 huggingface dataset:
Multilevel Factor Analysis of Class and Student Achievement Components
Authors: Muthén Bengt O.
Published: September 2005

This article analyzes mathematics achievement data from the Second International Mathematics Study (SIMS; Crosswhite, Dossey, Swafford, McKnight, & Cooney, 1985) in which U.S. students are measured at the beginning and end of eighth grade. The aim of the article is to address some substantive analysis questions in the SIMS data and show the potential of multilevel factor analysis methodology. Issues related to between‐ and within‐class decomposition of achievement variance and the change of this decomposition over the course of the eighth grade are studied. As a starting point, random effects ANOVA is considered for each achievement score. Each score contains a large amount of measurement error. The effects of unreliability on variance decomposition are shown with the help of a multilevel factor analysis model. Unreliability has severely distorting effects on this type of ANOVA while multilevel factor analysis gives results corresponding to what would be obtained with perfectly reliable scores.
Equivalent Estimates of Borderline Group Performance in Standard Setting
Authors: Norcini John J. , Shea Judy A.
Published: September 2005

The purpose of this study was to determine if a linear procedure, typically applied to an entire examination when equating scores and reseating judges' standards, could be used with individual item data gathered through Angoffs standard‐setting method (1971). Specifically, experts estimates of borderline group performance on one form of a test were transformed to be on the same scale as experts' estimates of borderline group performance on another form of the test. The transformations were based on examinees' responses to the items and on judges' estimates of borderline group performance. The transformed values were compared to the actual estimates provided by a group of judges. The equated and reseated values were reasonably close to those actually assigned by the experts. Bias in the estimates was also relatively small. In general, the reseating procedure was more accurate than the equating procedure, especially when the examinee sample size for equating was small.
Measuring and Validating Cognitive Modifiability as an Ability: A Study in the Spatial Domain
Authors: Embretson Susan E.
Published: September 2005

Measuring cognitive modifiability from the responsiveness of an individual's performance to intervention has long been viewed (e.g., Dearborne, 1921) as an alternative to traditional (static) ability measurement. Currently, dynamic testing, in which cues or instruction are presented with ability test items, is a popular method for assessing cognitive modifiability. Despite the long‐standing interest, however, little data exists to support the validity of cognitive modifiability measures in any ability domain. Several special methodological difficulties have limited validity studies, including psychometric problems in measuring modifiability (i.e., as change), lack of appropriate validation criteria, and difficulty in linking modifiability to cognitive theory. In this article, relatively new developments for solving the validation problems are applied to measuring and validating spatial modifiability. Criterion‐related validity for predicting learning in an applied knowledge domain, as well as construct validity, is supported.
Estimating Population Characteristics From Sparse Matrix Samples of Item Responses
Authors: Mislevy Robert J. , Beaton Albert E. , Kaplan Bruce , Sheehan Kathleen M.
Published: September 2005

The multiple‐matrix item sampling designs that provide information about population characteristics most efficiently administer too few responses to students to estimate their proficiencies individually. Marginal estimation procedures, which estimate population characteristics directly from item responses, must be employed to realize the benefits of such a sampling design. Numerical approximations of the appropriate marginal estimation procedures for a broad variety of analyses can be obtained by constructing, from the results of a comprehensive extensive marginal solution, files of plausible values of student proficiencies. This article develops the concepts behind plausible values in a simplified setting, sketches their use in the National Assessment of Educational Progress (NAEP), and illustrates the approach with data from the Scholastic Aptitude Test (SA T).
A Comparison of the Performance of Simulated Hierarchical and Linear Testlets
Authors: Wainer Howard , Kaplan Bruce , Lewis Charles
Published: September 2005

A series of computer simulations were run to measure the relationship between testlet validity and the factors of item pool size and testlet length for both adaptive and linearly constructed testlets. We confirmed the generality of earlier empirical findings (Wainer, Lewis, Kaplan, & Braswell, 1991) that making a testlet adaptive yields only modest increases in aggregate validity because of the peakedness of the typical proficiency distribution.
Conditional Standard Errors of Measurement for Scale Scores
Authors: Kolen Michael J. , Hanson Bradley A. , Brennan Robert L.
Published: September 2005

Standard errors of measurement of scale scores by score level (conditional standard errors of measurement) can be valuable to users of test results. In addition, the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1985) recommends that conditional standard errors be reported by test developers. Although a variety of procedures are available for estimating conditional standard errors of measurement for raw scores, few procedures exist for estimating conditional standard errors of measurement for scale scores from a single test administration. In this article, a procedure is described for estimating the reliability and conditional standard errors of measurement of scale scores. This method is illustrated using a strong true score model. Practical applications of this methodology are given. These applications include a procedure for constructing score scales that equalize standard errors of measurement along the score scale. Also included are examples of the effects of various nonlinear raw‐to‐scale score transformations on scale score reliability and conditional standard errors of measurement. These illustrations examine the effects on scale score reliability and conditional standard errors of measurement of (a) the different types of raw‐to‐scale score transformations (e.g., normalizing scores), (b) the number of scale score points used, and (c) the transformation used to equate alternate forms of a test. All the illustrations use data from the ACT Assessment testing program.
Resolution of a Regression Paradox in Pretest‐Posttest Designs
Authors: Samsa Gregory P.
Published: September 2005

Regression to the mean (RTM) of individuals is the tendency for extreme individuals to become less extreme on remeasurement; RTM of group means describes this same tendency among group means. Under the classical test model, in pretest‐posttest designs where subjects are selected on the basis of extreme values at the pretest, RTM of group means will always occur for the attribute used to select extreme subjects. For other attributes, however, RTM of group means requires a positive correlation between that attribute's measurement error and the measurement error of the attribute used in the selection. Thus, while all attributes will evidence RTM of individuals, extreme groups do not always regress to the mean. RTM depends most fundamentally on the magnitude of the pretest measurement error.
A Comparison of Self‐Adapted and Computerized Adaptive Tests
Authors: Wise Steven L. , Plake Barbara S. , Johnson Phillip L. , Roos Linda L.
Published: September 2005

According to item response theory (IRT), examinee ability estimation is independent of the particular set of test items administered from a calibrated pool. Although the most popular application of this feature of IRT is computerized adaptive (CA) testing, a recently proposed alternative is self‐adapted (SA) testing, in which examinees choose the difficulty level of each of their test items. This study compared examinee performance under SA and CA tests, finding that examinees taking the SA test (a) obtained significantly higher ability scores and (b) reported significantly lower posttest state anxiety. The results of this study suggest that SA testing is a desirable format for computer‐based testing.
An Empirical Investigation of Thurstone and IRT Methods of Scaling Achievement Tests
Authors: Becker Douglas F. , Forsyth Robert A.
Published: September 2005

The purpose of this study was to investigate the nature and characteristics of the measurement scales developed using Thurstone and item response theory (IRT) methods of scaling achievement tests for the same set of data. Expanded standard score scales were created using Thurstone, one‐parameter IRT, and three‐parameter IRT models, and descriptive information on achievement growth and variability was obtained for examinees in Grades 9 through 12 in the subject areas of vocabulary, reading, and mathematics. The results indicated increasing variability in all three test areas for all three scaling methods as grade level increased. In addition, greater average growth across grades was observed at the 90th percentile as compared to the 10th percentile, even with the IRT‐based scales.
Scaling Performance Assessments: Strategies for Managing Local Item Dependence
Authors: Yen Wendy M.
Published: September 2005

Performance assessments appear on a priori grounds to be likely to produce far more local item dependence (LID) than that produced in the use of traditional multiple‐choice tests. This article (a) defines local item independence, (b) presents a compendium of causes of LID, (c) discusses some of LID's practical measurement implications, (d) details some empirical results for both performance assessments and multiple‐choice tests, and (e) suggests some strategies for managing LID in order to avoid negative measurement consequences.
Sampling Variability of Performance Assessments
Authors: Shavelson Richard J. , Baxter Gail P. , Gao Xiaohong
Published: September 2005

In this article, performance assessments are cast within a sampling framework. More specifically, a performance assessment is viewed as a sample of student performance drawn from a complex universe defined by a combination of all possible tasks, occasions, raters, and measurement methods. Using generalizability theory, we present evidence bearing on the generalizability and convergent validity of performance assessments sampled from a range of measurement facets and measurement methods. Results at both the individual and school level indicate that task‐sampling variability is the major source ofmeasurment error. Large numbers of tasks are needed to get a reliable measure of mathematics and science achievement at the elementary level. With respect to convergent validity, results suggest that methods do not converge. Students' performance scores, then, are dependent on both the task and method sampled.
Least Squares Models to Correct for Rater Effects in Performance Assessment
Authors: Raymond Mark R. , Viswesvaran Chockalingam
Published: September 2005

This study describes three least squares models to control for rater effects in performance evaluation: ordinary least squares (OLS); weighted least squares (WLS); and ordinary least squares, subsequent to applying a logistic transformation to observed ratings (LOG‐OLS). The models were applied to ratings obtained from four administrations of an oral examination required for certification in a medical specialty. For any single administration, there were 40 raters and approximately 115 candidates, and each candidate was rated by four raters. The results indicated that raters exhibited significant amounts of leniency error and that application of the least squares models would change the pass‐fail status of approximately 7% to 9% of the candidates. Ratings adjusted by the models demonstrated higher reliability and correlated slightly higher than observed ratings with the scores on a written examination.
Simultaneous DIF Amplification and Cancellation: Shealy‐Stout's Test for DIF
Authors: Nandakumar Ratna
Published: September 2005

The present study investigates the phenomena of simultaneous DIF amplification and cancellation and SIBTEST's role in detecting such. A variety of simulated test data were generated for this purpose. In addition, real test data from various sources were analyzed. The results from both simulated and real test data, as Sheafy and Stout's theory (1993a, 1993b) suggests, show that the SIBTEST is effective in assessing DIF amplification and cancellation (partially or fully) at the test score level. Finally, methodological and substantive implications of DIF amplification and cancellation are discussed.
Reconsidering Standards and Criteria
Authors: Cizek Gregory J.
Published: September 2005

An early debate about the nature of setting standards on educational achievement tests centered on the extent to which resulting standards were arbitrary. Subsequent research in the area has advanced solutions to many practical standard setting problems, but the more fundamental issue regarding the empirical grounding of judgmental standard setting procedures has remained unresolved and largely unaddressed. This article reviews some of the salient elements of the debate about the nature of standard setting on educational assessments and suggests that the dispute can never be satisfactorily resolved within the current paradigm. A reconcep‐tualization of the nature of standard setting is proposed, and suggestions for future research are provided.
Teachers' Grading Practices: Meaning and Values
Authors: Brookhart Susan M.
Published: September 2005

Classroom teachers do not always follow recommended grading practices. Why not? It is possible to conceptualize this question as a validity issue and ask whether teachers' concerns over the many uses of grades outweigh concerns about the interpretation of grades. The purpose of this study was to investigate the meaning classroom teachers associate with grades, the value judgments they make when considering grades, and whether the meaning or values associated with grades differed by whether teachers had measurement instruction. A sample of 84 teachers, 40 with and 44 without measurement instruction, responded to classroom grading scenarios in two ways–with multiple‐choice responses indicating what they would do and with written responses to the question, “Why did you make this choice?” A coding scheme based on Messick's (1989a, 1989b) progressive matrix of facets of validity was used for quantitative and qualitative analyses of written responses. The meaning of grades is closely related to the idea of student work; grades are pay students earn for activities they perform. The relationship of this notion to classroom management should be investigated. Teachers do make value judgments when assigning grades and are especially concerned about being fair. Teachers also are concerned about the consequences of grade use, especially for developing student self‐esteem and good attitudes toward future school work. Measurement instruction made very little difference, although it did reduce the amount of self‐referenced grading reported.
Factors Which Influence Precision of School‐Level IRT Ability Estimates
Authors: Tate Richard L. , King FJ
Published: September 2005

The precision of the group‐level IRT model applied to school ability estimation is described, assuming use of Bayesian estimation with precision represented by the standard deviation of the posterior distribution. Similarities and differences between the school‐level model and the familiar individual‐level IRT model are considered. School size and between‐school variability, two factors not relevant at the student level, are dominant determinants of school‐level precision. Under the multiple‐matrix sampling design required for the school‐level IRT, the number of items associated with a scale does not influence the precision at the school level. Also, the effects of school ability and item quality on school‐level precision are often relatively weak. It was found that the use of Bayesian estimation could result in a systematic distortion of the true ranking of schools based on ability because of an estimation bias which is a function of school size.
The Relationship of Essay and Multiple‐Choice Scores With Grades in College Courses
Authors: Bridgeman Brent , Lewis Charles
Published: September 2005

Essay and multiple‐choice scores from Advanced Placement (AP) examinations in American History, European History, English Language and Composition, and Biology were matched with freshman grades in a sample of 32 colleges. Multiple‐choice scores from the American History and Biology examinations were more highly correlated with freshman grade point averages than were essay scores from the same examinations, but essay scores were essentially equivalent to multiple‐choice scores in correlations with course grades in history, English, and biology. In history courses, men and women received comparable grades and had nearly equal scores on the AP essays, but the multiple‐choice scores of men were nearly one half of a standard deviation higher than the scores of women.
A Distribution‐Free Approach for Comparing Growth of Knowledge
Authors: Tan E. S. , Imbos Tj. , Does R. J. M. M.
Published: September 2005

The longitudinal testing of student achievement requires the solution of several new problem areas. In this article, several small groups of medical students at the University of Limburg Medical School in Maastricht, The Netherlands, are compared with respect to their performances. The results indicate, that, despite the possession of more knowledge at entrance, students with a low rate of growth of knowledge in the first year demonstrate a lower level of knowledge after the second academic year and continue to do so throughout the academic program when compared to students who show a higher rate of growth of knowledge in the first year. The analysis has been carried out using a distribution‐free version of a longitudinal IRT‐model suggested by Albers, Does, Imbos, and Janssen (1989). Furthermore, growth of knowledge has been described by means of a general regression model. Statistical inferences are possible using a randomization design extended to the situation where the observations are time‐dependent proportions of correct answers.
The Effects of Score Group Width on the Mantel‐Haenszel Procedure
Authors: Clauser Brian , Mazor Kathleen M. , Hambleton Ronald K.
Published: September 2005

Previous research examining the effects of reducing the number of score groups used in the matching criterion of the Mantel‐Haenszel procedure, when screening for DIF, has produced ambiguous results. The goal of this study was to resolve the ambiguity by examining the problem with a simulated data set. The main results from this study call into question the preliminary recommendations of several other researchers that four or more score groups are sufficient and produce stable results. Although considerable stability and very little Type I error was noted with equal ability distribution comparisons, with unequal ability distributions, the Type I error rate was substantially inflated. These results argue against the appropriateness of implementing the procedure by collapsing score groups. The current data suggest that more than modest reductions in the number of score groups cannot be recommended when the ability distributions of the reference and focal groups differ
Are Tests Comprising Both Multiple‐Choice and Free‐Response Items Necessarily Less Unidimensional Than Multiple‐Choice Tests?An Analysis of Two Tests
Authors: Thissen David , Wainer Howard , Wang Xiang‐Bo
Published: September 2005

We consider the relationship between the multiple‐choice and free‐response sections on the Computer Science and Chemistry tests of the College Board's Advanced Placement program. Restricted factor analysis shows that the free‐response sections measure the same underlying proficiency as the multiple‐choice sections for the most part. However, there is also a significant, if relatively small, amount of local dependence among the free‐response items that produces a small degree of multidimensionauty for each test
Conceptualization and Measurement of Cognitive Holding Power
Authors: Stevenson John C. , Evans Glen T.
Published: September 2005

The concept of cognitive holding power is synthesized from theories of settings and of cognitive structures and is conceptualized as a characteristic of a learning setting that presses students into different kinds of cognitive activity. Settings which press students into using first‐ or second‐order cognitive procedures are regarded as having first‐ or second‐order cognitive holding power. The development of an instrument to measure these two dimensions of cognitive holding power is outlined. The independence of the dimensions, their reliabilities and validity, and factor structures are examined. Each dimension was found to have high reliability across vocational education and high school settings, and each was correlated as predicted with other classroom variables. The potential contribution of this research to understanding the relationship between different approaches to the teaching of problem solving and the ability to undertake problem‐solving transfer tasks is outlined.
An Empirical Study of Computerized Adaptive Test Administration Conditions
Authors: Lunz Mary E. , Bergstrom Betty A.
Published: September 2005

This empirical study was designed to determine the impact of computerized adap‐ tive test (CAT) administration formats on student performance. Students in medical technology programs took a paper‐and‐pencil and an individualized, computerized adaptive test. Students were randomly assigned to adaptive test administration for‐ mats to ascertain the effect on student performance of altering: (a) the difficulty of the first item, (b) the targeted level of test difficulty, (c) minimum test length, and (d) the opportunity to control the test. Computerized adaptive test data were analyzed with ANCO VA. The paper‐and.pencil test was used as a covariate to equalize abil‐ ity variance among cells. The only significant main effect was for opportunity to control the test. There were no significant interactions among test administration formats. This study provides evidence concerning adjusting traditional computer‐ ized adaptive testing to more familiar testing modalities.
Performance Versus Objective Testing and Gender: An Exploratory Study of an Advanced Placement History Examination
Authors: Breland Hunter M. , Danos Despina O. , Kahn Helen D. , Kubota Melvin Y. , Bonner Marilyn W.
Published: September 2005

To explore a phenomenon of gender differences in Advanced Placement examinations, random samples of free‐response test booklets were taken from the 1986 examination in U.S. History. These examinations were chosen because they consistently show significant gender differences in objective scores but no gender differences in free‐response scores. A rescoring of the free responses was conducted that focused on their historical content. This rescoring was conducted by readers other than those who conducted the original scoring and involved tallies of specific historical points made, supporting evidence given, and factual errors. Ratings were also made of handwriting quality, neatness, and English composition quality of the free responses. Analyses conducted indicate that free‐response tasks of the type examined may have inherent characteristics that reward English composition abilities, and that some females may compensate for inferior historical knowledge with superior English composition abilities.
Score Gains, Attitudes, and Behavior Changes Due to Answer‐Changing Instruction
Authors: Prinsell Catherine P. , Ramsey Philip H. , Ramsey Patricia P.
Published: September 2005

Six undergraduate and three graduate classes were given multiple‐choice tests with subsequent evaluation of answer changes. The 300 students were tested twice, once before and once after instruction on answer changing. After each test, students were asked to complete two forms. The forms evaluated attitude toward answer changing, reasons for changing, and confidence in final answers. Students showed a significant increase in favorability toward answer changing after instruction. No significant change was found in number of answers changed. Psychology students were found to change significantly more items than were business students. Mean gain score did not change significantly after instruction. It was concluded that although instruction does lead to a change in attitude in answer changing, the number of changes and overall gain due to answer changing do not change. It was also determined that students continue to make significant gains even when their confidence in the final answer is less than 50 on a 100‐point scale.
Assessment Approaches and Cognitive Styles
Authors: Lu Chin‐hsieh , Suen Hoi K.
Published: September 2005

The outcomes on multiple‐choice tests and performance‐based assessments for field‐independent and field‐dependent students were examined. A substantial interaction between cognitive style and assessment approach was found. Results suggested that performance‐based assessment tended to favor field‐independent subjects. Dependent on the purpose and intended use of assessment, this finding may raise concerns for validity based on either fairness or curriculum relevance.
Answering the New SAT Reading Comprehension Questions Without the Passages
Authors: Powers Donald E. , Leung Susan Wilson
Published: September 2005

It has been reasonably well established that test takers can, to varying degrees, answer some reading comprehension questions without reading the passages on which the questions are based, even for carefully constructed measures like the Scholastic Aptitude Test (SAT). The aim of this study was to determine what test‐taking strategies examinees use, and which are related to test performance, when reading passages are not available. The research focused on reading comprehension questions similar to those that will be used in the revised SAT, to be introduced in 1994. The most often cited strategies involved choosing answers on the basis of consistency with other questions and reconstructing the main theme of a missing passage from all of the questions and answers in a set. These strategies were more likely to result in successful performance on individual test items than were any of many other possible (and less constructrelevant) strategies.
Using Logistic Regression and the Mantel‐Haenszel With Multiple Ability Estimates to Detect Differential Item Functioning
Authors: Mazor Kathleen M. , Kanjee Anil , Clauser Brian E.
Published: September 2005

Logistic regression has recently been advanced as a viable procedure for detecting differential item functioning (DIF). One of the advantages of this procedure is the considerable flexibility it offers in the specification of the regression equation. This article describes incorporating two ability estimates into a single regression analysis, with the result that substantially fewer items exhibit DIF. A comparable analysis is conducted using the Mantel‐Haenszel with similar results. It is argued that by simultaneously conditioning on two relevant ability estimates, more accurate matching of examinees in the reference and focal groups is obtained, and thus multidimensional item impact is not mistakenly identified as DIF.
Assessing Differential Item Functioning in Direct Writing Assessments: Problems and an Example
Authors: Welch Catherine J. , Miller Timothy R.
Published: September 2005

The recent emphasis on various types of performance assessments raises questions concerning the differential effects of such assessments on population subgroups. Procedures for detecting differential item functioning (DIF) in data from performance assessments are available but may be hindered by problems that stem from this mode of assessment. Foremost among these are problems related to finding an appropriate matching variable. These problems are discussed and results are presented for three methods for DIF detection in polytomous items using data from a direct writing assessment. The purpose of the study is to examine the effects of using different combinations of internal and external matching variables. The procedures included a generalized Mantel‐Haenszel statistic, a technique based on meta‐analysis methodology, and logistic discriminant function analysis. In general, the results did not support the use of an external matching criterion and indicated that continued problems may be expected in attempts to assess DIF in performance assessments.
Effect of Rasch Calibration on Ability and DIF Estimation in Computer‐Adaptive Tests
Authors: Zwick Rebecca , Thayer Dorothy T. , Wingersky Marilyn
Published: September 2005

In a previous simulation study of methods for assessing differential item functioning (DIF) in computer‐adaptive tests (Zwick, Thayer, & Wingersky, 1993, 1994), modified versions of the Mantel‐Haenszel and standardization methods were found to perform well. In that study, data were generated using the 3‐parameter logistic (3PL) model and this same model was assumed in obtaining item parameter estimates. In the current study, the 3PL data were used but the Rasch model was assumed in obtaining the item parameter estimates, which determined the information table used for item selection. Although the obtained DIF statistics were highly correlated with the generating DIF values, they tended to be smaller in magnitude than in the 3PL analysis, resulting in a lower probability of DIF detection. This reduced sensitivity appeared to be related to a degradation in the accuracy of matching. Expected true scores from the Rasch‐based computer‐adaptive test tended to be biased downward, particularly for lower‐ability examinees
The Conventional Wisdom About Group Mean Scores
Authors: Brennan Robert L.
Published: September 2005

Not infrequently, investigators assume that reliability for groups is greater than reliability for persons, and/or that error variance for groups is less than error variance for persons. Using generalizability theory, it is shown that this conventional wisdom is not necessarily true. Examples are provided from the course evaluation literature and the performance testing literature
Cognitive Design Principles and the Successful Performer: A Study on Spatial Ability
Authors: Embretson Susan E.
Published: September 2005

An important trend in educational measurement is the use of principles of cognitive psychology to design achievement and ability test items. Many studies show that manipulating the stimulus features of items influences the processes, strategies, and knowledge structures that are involved in solution. However, little is known about how cognitive design influences individual differences. That is, does applying cognitive design principles change the background skills and abilities that are associated with successful performance? This study compared the correlates of two spatial ability tests that used the same item type but different test design principles (cognitive design versus psychometric design). The results indicated differences in factorial complexity in the two tests; specifically, the impact of verbal abilities was substantially reduced by applying the cognitive design principles.
Using Trilinear Plots for NAEP State Data
Authors: Wainer Howard
Published: September 2005

Understanding the distribution of achievement levels of students’ performance in the National Assessment of Educational Progress (NAEP) is aided through the use of the trilinear chart. In this article, this chart is described and its use illustrated with data from the 1992 state NAEP mathematics assessment. It is shown that one can see readily the trends in performance for different demographic groups for all of the 44 participating jurisdictions simultaneously. It is suggested that this graphical form may be useful in other contexts, as well.
Generalizability and Validity of a Mathematics Performance Assessment
Authors: Lane Suzanne , Liu Mei , Ankenmann Robert D. , Stone Clement A.
Published: September 2005

The QUASAR Cognitive Assessment Instrument (QCAI) is designed to measure program outcomes and growth in mathematics. It consists of a relatively large set of open‐ended tasks that assess mathematical problem solving, reasoning, and communication at the middle‐school grade levels. This study provides some evidence for the generalizability and validity of the assessment. The results from the generalizability studies indicate that the error due to raters is minimal, whereas there is considerable differential student performance across tasks. The dependability of grade level scores for absolute decision making is encouraging; when the number of students is equal to 350, the coefficients are between .80 and .97 depending on the form and grade level. As expected, there tended to be a higher relationship between the QCAI scores and both the problem solving and conceptual subtest scores from a mathematics achievement multiple‐choice test than between the QCAI scores and the mathematics computation subtest scores.
Simulation Studies of the Effects of Small Sample Size and Studied Item Parameters on SIBTEST and Mantel‐Haenszel Type I Error Performance
Authors: Roussos Louis A. , Stout William F.
Published: September 2005

Two simulation studies investigated Type I error performance of two statistical procedures for detecting differential item functioning (DIF): SIBTEST and Mantel‐Haenszel (MH). Because MH and SIBTEST are based on asymptotic distributions requiring “large” numbers of examinees, the first study examined Type 1 error for small sample sizes. No significant Type I error inflation occurred for either procedure. Because MH has the potential for Type I error inflation for non‐Rasch models, the second study used a markedly non‐Rasch test and systematically varied the shape and location of the studied item. When differences in distribution across examinee group of the measured ability were present, both procedures displayed inflated Type 1 error for certain items; MH displayed the greater inflation. Also, both procedures displayed statistically biased estimation of the zero DIF for certain items, though SIBTEST displayed much less than MH. When no latent distributional differences were present, both procedures performed satisfactorily under all conditions.
Applying the Mantel‐Haenszel Procedure to Complex Samples of Items
Authors: Allen Nancy L. , Donoghue John R.
Published: September 2005

This Monte Carlo study examined the effect of complex sampling of items on the measurement of differential item functioning (DIF) using the Mantel‐Haenszel procedure. Data were generated using a 3‐parameter logistic item response theory model according to the balanced incomplete block (BIB) design used in the National Assessment of Educational Progress (NAEP). The length of each block of items and the number of DIF items in the matching variable were varied, as was the difficulty, discrimination, and presence of DIF in the studied item. Block, booklet, pooled booklet, and extra‐information analyses were compared to a complete data analysis using the transformed log‐odds on the delta scale. The pooled booklet approach is recommended for use when items are selected for examinees according to a BIB design. This study has implications for DIF analyses of other complex samples of items, such as computer administered testing or another complex assessment design.
Decision Theory Models for Validating Course Placement Tests
Authors: Sawyer Richard
Published: September 2005

Most American postsecondary institutions have course placement systems for their first‐year students. Placement systems typically consist of an assessment component (to estimate students’ probability of success in standard first‐year courses) and an instructional component (in which academically underprepared students are taught the skills and knowledge they need to succeed in the standard courses). Validity issues related to these functions are discussed in the context of decision theory, and methods are proposed for determining appropriate cutoff scores on placement tests.
Detecting DIF for Polytomously Scored Items: An Adaptation of the SIBTEST Procedure
Authors: Chang Hua‐Hua , Mazzeo John , Roussos Louis
Published: September 2005

Shealy and Stout (1993) proposed a DIF detection procedure called SIBTEST and demonstrated its utility with both simulated and real data sets'. Current versions of SIBTEST can be used only for dichotomous items. In this article, an extension to handle polytomous items is developed. Two simulation studies are presented which compare the modified SIBTEST procedure with the Mantel and standardized mean difference (SMD) procedures. The first study compares the procedures under conditions in which the Mantel and SMD procedures have been shown to perform well (Zwick, Donoghue, & Grima, 1993). Results of Study I suggest that SIBTEST performed reasonably well, but that the Mantel and SMD procedures performed slightly better. The second study uses data simulated under conditions in which observed‐score DIF methods for dichotomous items have not performed well. The results of Study 2 indicate that under these conditions the modified SIBTEST procedure provides better control of impact‐induced Type I error inflation than the other procedures.
Test Theory Reconceived
Authors: Mislevy Robert J.
Published: September 2005

Educational test theory consists of statistical and methodological tools to support inference about examinees’ knowledge, skills, and accomplishments. Its evolution has been shaped by the nature of users’ inferences, which have been framed almost exclusively in terms of trait and behavioral psychology, and focused on students’ tendency to act in prespecified ways in prespecified domains of tasks. Progress in the methodology of test theory enabled users to extend the range of inference and ground interpretations more solidly within these psychological paradigms. Developments in cognitive and developmental psychology have broadened the range of inferences we wish to make about students’ learning to encompass conjectures about the nature and acquisition of their knowledge. The same underlying principles of inference that led to standard test theory can support inference in this broader universe of discourse. Familiar models and methods‐sometimes extended, sometimes reinterpreted, sometimes applied to problems wholly different from those for which they were first devised‐can play a useful role to this end.
Improving the Matching for DIF Analysis by Conditioning on Both Test Score and an Educational Background Variable
Authors: Clauser Brian E. , Nungester Ronald J. , Swaminathan Hariharan
Published: September 2005

When tests are designed to measure dimensionally complex material, DIF analysis with matching based on the total test score may be inappropriate. Previous research has demonstrated that matching can be improved by using multiple internal or both internal and external measures to more completely account for the latent ability space. The present article extends this line of research by examining the potential to improve matching by conditioning simultaneously on test score and a categorical variable representing the educational background of the examinees. The responses of male and female examinees from a test of medical competence were analyzed using a logistic regression procedure. Results show a substantial reduction in the number of items identified as displaying significant DIF when conditioning is based on total test score and a variable representing educational background as opposed to total test score only.
Computerized Cognitive Diagnostic Adaptive Testing: Effect on Remedial Instruction as Empirical Validation
Authors: Tatsuoka Kikumi K. , Tatsuoka Maurice M.
Published: September 2005

The purpose of this study is to show the usefulness of cognitive diagnoses for remedial instruction. Cognitive diagnoses were done by an adaptive testing system using the rule‐space methodology, which was developed by K. K. Tatsuoka and her associates (K. K. Tatsuoka, 1983, 1990; K. K. Tatsuoka & M. M. atsuoka, 1987; M. M. Tatsuoka & K. K. Tatsuoka, 1989). The results of the study strongly indicate that knowing students'knowledge states prior to remediation is very effective and that the rule‐space method can effectively diagnose students' knowledge states and can point out ways for remediating their errors quickly with minimum effort. It is also found that the design of instructional units for remediation can be effectively guided by the rule‐space model, because the determination of all possible knowledge states in a domain of interest, given an incidence matrix, is based on a partially ordered tree structure of knowledge states, which is equivalent to item‐score patterns determined logically from the incidence matrix.
Using Performance Standards to Link Statewide Achievement Results to NAEP
Authors: Waltman Kristie K.
Published: September 2005

The purpose of this study was to investigate the comparability in score meaning of the performance regions on the ITBS and NAEP mathematics score scales that resulted from using performance standards to establish two separate links: socially moderated and statistically moderated. A socially moderated link was established by using the same achievement level descriptions in an ITBS standard‐setting study that were used in a NAEP standard‐setting study. A statistically moderated link was accomplished by using an equipercentile procedure. The primary findings were that (a) social moderation yielded cutscores on the ITBS scales that resulted in larger percentages of Iowa public fourth‐grade students being classified within the basic, proficient, and advanced achievement regions than those reported by NAEP; (b) the equipercentile link yielded percentages on the ITBS scale that were similar to those reported by NAEP for “type of community” subgroups; and (c) for students taking both assessments, the corresponding achievement regions on the NAEP and ITBS scales produced low to moderate percents of agreement in student classification.
On Information of Percentile Ranks
Authors: de Gruijter Dato N. M.
Published: September 2005

Recently May and Nicewander concluded that percentile ranks are inferior to raw scores as indicators of latent ability. Here it is argued that their arguments are faulty.
Modeling Item Response Times With a Two‐State Mixture Model: A New Method of Measuring Speededness
Authors: Schnipke Deborah L. , Scrams David J.
Published: September 2005

Speededness refers to the extent to which time limits affect examinees'test performance, and it is often measured by calculating the proportion of examinees who do not reach a certain percentage of test items. However, when tests are number‐right scored (i.e., no points are subtracted for incorrect responses), examinees are likely to rapidly guess on items rather than leave them blank. Therefore, this traditional measure of speededness probably underestimates the true amount of speededness on such tests. A more accurate assessment of speededness should also reflect the tendency of examinees to rapidly guess on items as time expires. This rapid‐guessing component of speededness can be estimated by modeling response times with a two‐state mixture model, as demonstrated with data from a computer‐ administered reasoning test. Taking into account the combined effect of unreached items and rapid guessing provides a more complete measure of speededness than has previously been available.
Development and Demonstration of Multidimensional IRT‐Based Internal Measures of Differential Functioning of ltems and Tests
Authors: Oshima T. C. , Raju Nambury S. , Flowers Claudia P.
Published: September 2005

This article defines and demonstrates a framework for studying differential item functioning (DIF) and differential test functioning (DTF) for tests that are intended to be multidimensional The procedure introduced here is an extension of unidimensional differential functioning of items and tests (DFIT) recently developed by Raju, van der Linden, & Fleer (1995). To demonstrate the usefulness of these new indexes in a multidimensional IRT setting, two‐dimensional data were simulated with known item parameters and known DIF and DTE The DIF and DTF indexes were recovered reasonably well under various distributional differences of Os after multidimensional linking was applied to put the two sets of item parameters on a common scale. Further studies are suggested in the area of DIF/DTF for intentionally multidimensional tests.
Standard Setting: An Alternative Approach
Authors: Impara James C. , Plake Barbara S.
Published: September 2005

Since 1971 there have been a number of studies in which a cut score has been set using a method proposed by Angoff (1971). In this method, each member of a panel of judges estimates for each test question the proportion correct for a specific target group of examinees. Prior and contemporary research suggests that this is a difficult task for judges. Angoff also proposed that judges simply indicate whether or not an examinee from the target group will be able to answer each question correctly (the yes/no method). We report on the results of two studies that compare a yes/no estimation with a proportion correct estimation. The two studies demonstrate that both methods produce essentially equal cut scores and that judges find the yes/no method more comfortable to use than the estimated proportion correct method.
Using New Proximity Measures With Hierarchical Cluster Analysis to Detect Multidimensionality
Authors: Roussos Louis A. , Stout William F. , Marden John I.
Published: September 2005

A new approach for partitioning test items into dimensionally distinct item clusters is introduced. The core of the approach is a new item‐pair conditional‐covariance‐based proximity measure that can be used with hierarchical cluster analysis. An extensive simulation study designed to test the limits of the approach indicates that when approximate simple structure holds, the procedure can correctly partition the test into dimensionally homogeneous item clusters even for very high correlations between the latent dimensions. In particular, the procedure can correctly classify (on average) over 90% of the items for correlations as high as .9. The cooperative role that the procedure can play when used in conjunction with other dimensionality assessment procedures is discussed.
Simulating the Use of Disclosed Items in Computerized Adaptive Testing
Authors: Stocking Martha L. , Ward William C. , Potenza Maria T.
Published: September 2005

Regular use of questions previously made available to the public (i.e., disclosed items) may provide one way to meet the requirement for large numbers of questions in a continuous testing environment, that is, an environment in which testing is offered at test taker convenience throughout the year rather than on a few prespecified test dates. First it must be shown that such use has effects on test scores small enough to be acceptable. In this study simulations are used to explore the use of disclosed items under a worst‐case scenario which assumes that disclosed items are always answered correctly. Some item pool and test designs were identified in which the use of disclosed items produces effects on test scores that may be viewed as negligible.