bluuebunny/crossref_metadata_2025 huggingface dataset:
Relationships Among Multiple–Choice and Open–Ended Analytical Questions
Authors: Bridgeman Brent , Rock Donald A.
Published: September 2005

Exploratory and confirmatory factor analyses were used to explore relationships among existing item types and three new computer–administered item types for the analytical scale of the Graduate Record Examination General Test. One new item type was an open–ended version of the current multiple–choice analytical reasoning item type. The other new item types had no counterparts on the existing test. The computer tests were administered at four sites to a sample of students who had previously taken the GRE General Test. Scores from the regular GRE and the special computer administration were matched for a sample of 349 students. Factor analyses suggested that the new item types with no counterparts in the existing GRE were reliably assessing unique constructs but the open–ended analytical reasoning items were not measuring anything beyond what is measured by the current multiple–choice version of these items.
Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model
Authors: Engelhard George
Published: September 2005

This study describes several categories of rater errors (rater severity, halo effect, central tendency, and restriction of range). Criteria are presented for evaluating the quality of ratings based on a many‐faceted Rasch measurement (FACETS) model for analyzing judgments. A random sample of 264 compositions rated by 15 raters and a validity committee from the 1990 administration of the Eighth Grade Writing Test in Georgia is used to illustrate the model. The data suggest that there are significant differences in rater severity. Evidence of a halo effect is found for two raters who appear to be rating the compositions holistically rather than analytically. Approximately 80% of the ratings are in the two middle categories of the rating scale, indicating that the error of central tendency is present. Restriction of range is evident when the unadjusted raw score distribution is examined, although this rater error is less evident when adjusted estimates of writing competence are used
A Comparison of Equal Percentile and Partial Credit Equatings for Performance‐Based Assessments Composed of Free‐Response Items
Authors: Huynh Huynh , Ferrara Steven
Published: September 2005

This study compares the equal percentile (EP) and partial credit (PC) equatings for raw scores derived from performance‐based assessments composed of free‐response (open‐ended) items clustered around long reading selections or multistep mathematics problems. Data are from the Maryland School Performance Assessment Program. The results suggest that Masters (1982; Wright & Masters, 1982) partial credit model may be useful for equating examinations composed of moderately easy (or not too difficult)items sharing a first principal component with at least 25% of the total variance. This conclusion appears to hold even in the presence of some level of response dependency for the items within each cluster. Although visible discrepancies were found between PC and EP equated scores in the skewed tail of the score distributions, the direction of these discrepancies is unpredictable. Therefore, it cannot be concluded from the study that the two methods give equivalent results when the distributions are markedly skewed.
A Note on Interpretation of Formulas for the Reliability of Differences
Authors: Zimmerman Donald W.
Published: September 2005

It is widely recognized that the reliability of a difference score depends on the reliabilities of the constituent scores and their intercorrelation. Authors often use a well‐known identity to express the reliability of a difference as a function of the reliabilities of the components, assuming that the intercorrelation remains constant. This approach is misleading, because the familiar formula is a composite function in which the correlation between components is a function of reliability. An alternative formula, containing the correlation between true scores instead of the correlation between observed scores, provides more useful information and yields values that are not quite as anomalous as the ones usually obtained
How Well Can We Compare Scores on Test Forms That Are Constructed by Examinees Choice?
Authors: Wainer Howard , Wang Xiang‐Bo , Thissen David
Published: September 2005

When an exam consists, in whole or in part, of constructed‐response items, it is a common practice to allow the examinee to choose a subset of the questions to answer. This procedure is usually adopted so that the limited number of items that can be completed in the allotted time does not unfairly affect the examinee. This results in the de facto administration of several different test forms, where the exact structure of any particular form is determined by the examinee. However, when different forms are administered, a canon of good testing practice requires that those forms be equated to adjust for differences in their difficulty. When the items are chosen by the examinee, traditional equating procedures do not strictly apply due to the nonignorable nature of the missing responses. In this article, we examine the comparability of scores on such tests within an IRT framework. We illustrate the approach with data from the College Board's Advanced Placement Test in Chemistry
The Effect of Speededness on Parameter Estimation in Item Response Theory
Authors: Oshima T. C.
Published: September 2005

There is a paucity of research in item response theory (IRT) examining the consequences of violating the implicit assumption of nonspeededness. In this study, test data were simulated systematically under various speeded conditions. The three factors considered in relation to speededness were proportion of test not reached (5%, 10%, and 15%), response to not reached (blank vs. random response), and item ordering (random vs. easy to hard). The effects of these factors on parameter estimation were then examined by comparing the item and ability parameter estimates with the known true parameters. Results indicated that the ability estimation was least affected by speededness in terms of the correlation between true and estimated ability parameters. On the other hand, substantial effects of speededness were observed among item parameter estimates. Recommendations for minimizing the effects of speededness are discussed
They Think Less of My Handwritten Essay If Others Word Process Theirs?
Authors: Powers Donald E. , Fowles Mary E. , Farnum Marisa , Ramsey Paul
Published: September 2005

A study was undertaken to determine the effects on essay scores of intermingling handwritten and word‐processed versions of student essays. A sample of examinees, each of whom had produced both a handwritten and a word‐processed essay, was drawn from a larger sample of students who had participated in a pilot study of a new academic skills assessment battery. Students’original handwritten essays were converted to word‐processed versions, and their original word‐processed essays were converted to handwritten versions. Analyses revealed higher average scores for essays scored in the handwritten mode than for essays scored as word processed, regardless of the mode in which essays were originally produced. Several hypotheses were advanced to explain the discrepancies between scores on handwritten and word‐processed essays. The training of essay readers was subsequently modified on the basis of these hypotheses, and the experiment was repeated using the modified training with a new set of readers.
On the Relative Value of Multiple‐Choice, Constructed Response, and Examinee‐Selected Items on Two Achievement Tests
Authors: Lukhele Robert , Thissen David , Wainer Howard
Published: September 2005

Using analyses based on fitting item response models to data from the College Board's Advanced Placement exams in chemistry and United States history, we found that the constructed response portion of the tests yielded little information over and above that provided by the multiple‐choice sections. These tests also allow examinees to select subsets of the constructed response items; we found that scoring on the basis of the selections themselves provided almost as much information as did scoring on the basis of the answers
An Empirical Examination of the IRT Information of Polytomously Scored Reading Items Under the Generalized Partial Credit Model
Authors: Donoghue John R.
Published: September 2005

Using Muraki's (1992) generalized partial credit IRT model, polytomous items (responses to which can be scored as ordered categories) from the 1991 field test of the NAEP Reading Assessment were calibrated simultaneously with multiple‐choice and short open‐ended items. Expected information of each type of item was computed. On average, four‐category polytomous items yielded 2.1 to 3.1 times as much IRT information as dichotomous items. These results provide limited support for the ad hoc rule of weighting k‐category polytomous items the same as k ‐ 1 dichotomous items for computing total scores. Polytomous items provided the most information about examinees of moderately high proficiency; the information function peaked at 1.0 to 1.5, and the population distribution mean was 0. When scored dichotomously, information in polytomous items sharply decreased, but they still provided more expected information than did the other response formats. For reference, a derivation of the information function for the generalized partial credit model is included.
Content Specificity of Expert Judgments in a Standard‐Setting Study
Authors: Plake Barbara S. , Impara James C. , Potenza Maria T.
Published: September 2005

This study investigated the comparability of Angoff‐based item ratings on a general education test battery made by judges from within‐content specialties and across content domains. Judges were from English, mathematics, science, and social studies specialties in teacher education programs in a midwestem state. Cutscores established from the judges’ratings of out‐of‐content items differed little from the cutscores set using the ratings made by the content specialists. Further, out‐of‐content ratings by judges were not more influenced by performance data than were the ratings provided by judges rating items within their content specialty. The degree to ‐which these results generalize to other content specialties needs to be investigated.
The Development and Validation of a Formula for Measuring Single‐Sentence Test Item Readability
Authors: Homan Susan , Hewitt Margaret , Linder Jean
Published: September 2005

This study describes the development and validation of the Homan‐Hewitt Readability Formula. This formula estimates the readability level of single‐sentence test items. Its initial development is based on the assumption that differences in readability level will affect item difficulty. The validation of the formula is achieved by (a) estimating the readability levels of sets of test items predicted to be written at 2nd‐ through 8th‐grade levels; (b) administering the tests to 782 students in grades 2 through 5; (3) using the class means as the unit of analyses and subjecting the data to a two‐factor repeated measures ANOVA. Significant differences were found on class mean performance scores across the levels of readability. These results indicated that a relationship exists between students’reading grade levels and their responses to test items written at higher readability levels.
The Validity of Scores From Alternative Methods of Assessing Spelling Achievement
Authors: Frisbie David A. , Cantor Nancy K.
Published: September 2005

The purpose of this study was to gather evidence to examine the validity of alternative methods for assessing the spelling achievements of students in Grades 2–7. Students in Grades 2, 3, 5, and 7 took both a dictation spelling test and an objective spelling test. Scores from the Iowa Tests of Basic Skills were available for nearly all students. Comparisons among item formats centered on difficulty, content coverage and efficiency, reliability, relationship with dictation scores, the influence of context, and relationship to overall achievement. Overall, no single objective format stood out above the others, but some demonstrated superiority to the dictation format on several dimensions.
The Relation Between Information‐Processing Variables and Test‐Retest Stability for Questionnaire Items
Authors: Otter Martha E. , Mellenbergh Gideon J. , Glopper Kees de
Published: September 2005

Recently developed cognitive theories explain why some questionnaire items can be answered in a reliable and valid manner and others cannot. Those theories distinguish two components: (a) the interpretation or understanding of a question and (b) the role of memory. The present study investigates the ability of these two components to forecast the test‐retest association coefficients of 207 pilot questionnaire items used in an international study of reading literacy in which 2 populations were involved: Grade 5 and Grade 2 of primary and secondary education, respectively. The analysis of the data showed that for both populations, both components forecast the relative sizes of the test‐retest correlation coefficients. The results strongly suggest that if one wishes to use questionnaire items in research about relationships, then the items should be as unambiguous as possible. Moreover, the information needed to formulate an answer must be easily accessible in the respondent's memory.
A Comparison of Item Selection Routines in Linear and Adaptive Tests
Authors: Schnipke Deborah L. , Green Bert F.
Published: September 2005

Two item selection algorithms were compared in simulated linear and adaptive tests of cognitive ability. One algorithm selected items that maximally differentiated between examinees. The other used item response theory (IRT) to select items having maximum information for each examinee. Normally distributed populations of 1,000 cases were simulated, using test lengths of 4, 5, 6, and 7 items. Overall, adaptive tests based on maximum information provided the most information over the widest range of ability values and, in general, differentiated among examinees slightly better than the other tests. Although the maximum differentiation technique may be adequate in some circumstances, adaptive tests based on maximum information are clearly superior.
The Psychometric Characteristics of Choice Items
Authors: Fitzpatrick Anne R. , Yen Wendy M.
Published: September 2005

This study investigated the psychometric characteristics of constructed‐response (CR) items referring to choice and non‐choice passages administered to students in Grades 3, 5, and 8. The items were scaled using item response theory (IRT) methodology. The results indicated no consistent differences in the difficulty and discrimination of the items referring to the two types of passages. On the average, students' scale scores on the choice and non‐choice passages were comparable. Finally, the choice passages differed in terms of overall popularity and in their attractiveness to different gender and ethnic groups
Detection of Differential Item Functioning in Multiple Groups
Authors: Kim Seock‐Ho , Cohen Allan S. , Park Tae‐Hak
Published: September 2005

Detection of differential item functioning (DIF) is most often done between two groups of examinees under item response theory. It is sometimes important, however, to determine whether DIF is present in more than two groups. In this article we present a method for detection of DIF in multiple groups. The method is closely related to Lard's chi‐square for comparing vectors of item parameters estimated in two groups. An example using real data is provided.
Measurement Model for Linking Individual Learning to Processes and Knowledge: Application to Mathematical Reasoning
Authors: Embretson Susan E.
Published: September 2005

Contemporary instructional theories increasingly emphasize the importance of linking an individual's learning to changes in cognitive processes and knowledge structures. In this article, an extension of the multidimensional Rasch model for learning and change (MRMLC) is presented so as to permit theories of processes and knowledge structures to be incorporated into the item response model. Like the MRMLC, this extension (MRMLC+) resolves some basic problems in measuring individual change and permits adaptive testing so that precise estimates of learning may be obtained. Additionally, MRMLC+ permits individual learning to be linked to substantive changes in processing and knowledge. An application to a study on the impact of short‐term instruction on mathematical problem solving shows the potential of MRMLC + for interpretations. In this study, a theoretically plausible model of knowledge structures (Mayer, Larkin, & Kadane, 1984) provides the basis of individual learning interpretations.
Estimation of the Reliability of Differences Under Revised Reliabilities of Component Scores
Authors: Feldt Leonard S.
Published: September 2005

Projecting the changes in the reliability of a difference score (d =× ‐ Y) as a consequence of changes in the reliabilities of Xand Y does not represent a straightforward application of the Spearman‐Brown formula. Formulas are developed for estimating the changes in the reliability of X‐Yunder two possible assumptions: (a) ×and Y have equal variances both before and after their reliabilities are altered, and (b) ×and Y have unequal variances before and after×and Y are modified. The second of these situations, which includes the first as a special case, is probably the more common.
Exact Versus Asymptotic Mantel‐Haenszel DIF Statistics: A Comparison of Performance Under Small‐Sample Conditions
Authors: Parshall Cynthia G. , Miller Timothy R.
Published: September 2005

This study evaluated exact testing (Agresti, 1992) as a method for conducting Mantel‐Haenszel DIF analyses (Holland & Thayer, 1988) with relatively small samples. Sample‐size restrictions limit the standard asymptotic Mantel‐Haenszel for many practical applications; however, new developments in computing technology have made exact testing procedures feasible. The highly discrete distributions that are likely to occur in small‐sample DIF analyses could yield very different results for asymptotic versus exact methods. It is therefore important to determine under controlled conditions the extent to which the exact approach is effective in correctly identifying DIF. A series of computer simulations were conducted in which 3 levels of induced bias (IRT b‐parameter differences between groups of .25, .50, and .75) and 4 sample sizes (reference group= 500,focal group = 25, 50, 100, and 200) were investigated. Power comparisons at .01 and .05 alpha levels were carried out between the exact testing procedure and the conventional Mantel‐Haenszel
Effects of Calculator Use on Scores on a Test of Mathematical Reasoning
Authors: Bridgeman Brent , Harvey Anne , Braswell James
Published: September 2005

A sample of college‐bound juniors from 275 high schools took a test consisting of 70 math questions from the SAT. A random half of the sample was allowed to use calculators on the test. Both genders and three ethnic groups (White, African American, and Asian American) benefitted about equally from being allowed to use calculators; Latinos benefitted slightly more than the other groups. Students who routinely used calculators on classroom mathematics tests were relatively advantaged on the calculator test. Test speededness was about the same whether or not students used calculators. Calculator effects on individual items ranged from positive through neutral to negative and could either increase or decrease the validity of an item as a measure of mathematical reasoning skills. Calculator effects could be either present or absent in both difficult and easy items
Componential IRT Models for Polytomous Items
Authors: Hoskens Machteld , Boeck Paul De
Published: September 2005

Componential IRT models for polytomous items are of particular interest in two contexts: Componential research and test development. We assume that there are basic components, such as processes and knowledge structures, involved in solving cognitive tasks. In Componential research, the subtask paradigm may be used to isolate such components in subtasks. In test development, items may be composed such that their response alternatives correspond with specific combinations of such components. In both cases the data may be modeled as polytomous items. With Bock's (1972) nominal model as a general framework, transformation matrices can be used to constrain the parameters of the response categories so as to reflect the Componential design of the response categories. In this way, both main effects and interaction effects of components can be studied. An application to a spelling task demonstrates this approach
Probability‐Based Inference in a Domain of Proportional Reasoning Tasks
Authors: Béland Anne , Mislevy Robert J.
Published: September 2005

Educators and psychologists are increasingly interested in modeling the processes and knowledge structures by which people learn and solve problems. Meaningful progress has been mode in developing cognitive models in several domains and in devising observational settings that provide clues about subjects' cognition from this perspective. Less attention has been paid to drawing inferences or making decisions with such data, even though observations provide only imperfect information about cognition, and complex interrelationships can exist among observations and theoretical variables. This article discusses the use o f probability‐based reasoning to explicate hypothesized and empirical relationships and to structure inference in this context. Ideas are illustrated with an example concerning proportional reasoning.
Evaluating Rater Accuracy in Performance Assessments
Authors: Engelhard George
Published: September 2005

A new method for evaluating rater accuracy within the context of performance assessments is described. Accuracy is defined as the match between ratings obtained from operational raters and those obtained from an expert panel on a set of benchmark, exemplar, or anchor performances. An extended Rasch measurement model called the FACETS model is presented for examining rater accuracy. The FACETS model is illustrated with 373 benchmark papers rated by 20 operational raters and an expert panel. The data are from the 1993field test of the High School Graduation Writing Test in Georgia. The data suggest that there are statistically significant differences in rater accuracy; the data also suggest that it is easier to be accurate on some benchmark papers than on others. A small example is presented to illustrate how the accuracy ordering of raters may not be invariant over different subsets of benchmarks used to evaluate accuracy.
Confidence Intervals for the Proportion of Mastery in Criterion‐Referenced Measurement
Authors: Feldt Leonard S.
Published: September 2005

A relatively simple method is developed to obtain confidence intervals for a student's proportion of domain mastery in criterion‐referenced or mastery measurement situations. The method uses the binomial distribution as a model for the student's scores under hypothetically repeated assessments. Though the use of this model is not a new idea, the method of implementation has not been proposed previously. The technique makes use of widely available F tables and hence does not require elaborate computer equipment or proprietary computer programs.
Clarification to “Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model”
Authors: Engelhard George
Published: September 2005

In the article “Examining Rater Errors in the Assessment of Written Composition With a Many‐Faceted Rasch Model” (JEM, Volume 31, Number 2, Summer 1994), the data presented in Figure 3 may be misleading. The “four clear spikes” (p. 106) that appear in Figure 3 were highlighted by the automatic scaling procedure used by the computer program that generated this histogram; as is well known, the use of different scaling units would yield histograms with different shapes (Moore & McCabe, 1993). For example, when the same data are presented as a bar chart (see Figure 1 below) rather than as a histogram, the four spikes are not evident. As graphical procedures become more readily available to measurement researchers, additional research and discussion are needed regarding standards for evaluating data displays that do not simply reproduce the actual data values.
Conditional Standard Errors of Measurement for Scale Scores Using IRT
Authors: Kolen Michael J. , Zeng Lingjia , Hanson Bradley A.
Published: September 2005

An IRT method for estimating conditional standard errors of measurement of scale scores is presented, where scale scores are nonlinear transformations of number‐correct scores. The standard errors account for measurement error that is introduced due to rounding scale scores to integers. Procedures for estimating the average conditional standard error of measurement for scale scores and reliability of scale scores are also described. An illustration of the use of the methodology is presented, and the results from the IRT method are compared to the results from a previously developed method that is based on strong true‐score theory.
Estimation of Measurement Error Variance at Specific Score Levels
Authors: Feldt Leonard S. , Quails Audrey L.
Published: September 2005

An improved method is derived for estimating conditional measurement error variances, that is, error variances specific to individual examinees or specific to each point on the raw score scale of the test. The method involves partitioning the test into short parallel parts, computing for each examinee the unbiased estimate of the variance of part‐test scores, and multiplying this variance by a constant dictated by classical test theory. Empirical data are used to corroborate the principal theoretical deductions.
Using an Approximate Chi‐Square Statistic to Test the Number of Dimensions Underlying the Responses to a Set of Items
Authors: Gessaroli Marc E. , De Champlain André F.
Published: September 2005

An approximate χ2 statistic based on McDonald's (1967) nonlinear factor analytic representation of item response theory was proposed and investigated with simulated data. The results were compared with Stout's T statistic (Nandakumar & Stout, 1993; Stout, 1987). Unidimensional and two‐dimensional item response data were simulated under varying levels of sample size, test length, test reliability, and dimension dominance. The approximate χ2 statistic had good control over Type I errors when unidimensional data were generated and displayed very good power in identifying the two‐dimensional data. The performance of the approximate χ2 was at least as good as Stout's T statistic in all conditions and was better than Stout's T statistic with smaller sample sizes and shorter tests. Further implications regarding the potential use of nonlinear factor analysis and the approximate χ2 in addressing current measurement issues are discussed.
The Effect of Multidimensionality on IRT True‐Score Equating for Subgroups of Examinees
Authors: De Champlain André F.
Published: September 2005

The purpose of this study was to assess the dimensionality of two forms of a large‐scale standardized test separately for 3 ethnic groups of examinees and to investigate whether differences in their latent trait composites have any impact on unidimensional item response theory true‐score equating functions. Specifically, separate equating functions for African American and Hispanic examinees were compared to those of a Caucasian group as well as the total test taker population. On both forms, a 2‐dimensional model adequately accounted for the item responses of Caucasian and African American examinees, whereas a more complex model was required for the Hispanic subgroup. The differences between equating functions for the 3 ethnic groups and the total test taker population were small and tended to be located at the low end of the score scale.
A Comparison of Alternative Matching Strategies for DIF Detection in Tests That Are Multidimensional
Authors: Clauser Brian E. , Nungester Ronald J. , Mazor Kathleen , Ripkey Douglas
Published: September 2005

Most currently accepted approaches for identifying differentially functioning test items compare performance across groups after first matching examinees on the ability of interest. The typical basis for this matching is the total test score. Previous research indicates that when the test is not approximately unidimensional, matching using the total test score may result in an inflated Type I error rate. This study compares the results of differential item functioning (DIF) analysis with matching based on the total test score, matching based on subtest scores, or multivariate matching using multiple subtest scores. Analysis of both actual and simulated data indicate that for the dimensionally complex test examined in this study, using the total test score as the matching criterion is inappropriate. The results suggest that matching on multiple subtest scores simultaneously may be superior to using either the total test score or individual relevant subtest scores.
Gender Differences in College Mathematics Grades and SAT‐M Scores: A Reanalysis of Wainer and Steinberg
Authors: Bridgeman Brent , Lewis Charles
Published: September 2005

Wainer and Steinberg (1992) showed that within broad categories of first‐year college mathematics courses (e.g., calculus) men had substantially higher average scores on the mathematics section of the SAT (SAT‐M) than women who earned the same letter grade. However, three aspects of their analyses may lead to unwarranted conclusions. First, they focused primarily on differences in SAT‐M scores given course grades when the more important question for admissions officers is the difference in course grades given scores on the predictor. Second, they failed to account for differences among calculus courses (e.g., calculus for engineers versus calculus for liberal arts students). Most importantly, Wainer and Steinberg focused on the use of SAT‐M as a single predictor. A reanalysis presented here indicated that a more appropriate composite indicator made up of both SAT‐M and high school grade point average demonstrated minuscule gender differences for both calculus and precalculus courses.
Scaling Performance Assessments: A Comparison of One‐Parameter and Two‐Parameter Partial Credit Models
Authors: Fitzpatrick Anne R. , Link Valerie B. , Yen Wendy M. , Burket George R. , Ito Kyoko , Sykes Robert C.
Published: September 2005

In one study, parameters were estimated for constructed‐response (CR) items in 8 tests from 4 operational testing programs using the l‐parameter and 2‐ parameter partial credit (IPPC and 2PPC) models. Where multiple‐choice (MC) items were present, these models were combined with the 1‐parameter and 3‐parameter logistic (IPL and 3PL) models, respectively. We found that item fit was better when the 2PPC model was used alone or with the 3PL model. Also, the slopes of the CR and MC items were found to differ substantially. In a second study, item parameter estimates produced using the IPL‐IPPC and 3PL‐2PPC model combinations were evaluated for fit to simulated data generated using true parameters known to fit one model combination or ttle other. The results suggested that the more flexible 3PL‐2PPC model combination would produce better item fit than the IPL‐1PPC combination.
Effects of Applying Different Time Limits to a Proposed GRE Writing Test
Authors: Powers Donald E. , Fowles Mary E.
Published: September 2005

In order to determine the role of time limits on both test performance and test validity, we asked approximately 300 volunteers–prospective graduate students–to each write two essays–one in a 40‐minute time period and the other in 60 minutes. Analyses revealed that, on average, test performance was significantly better when examinees were given 60 minutes instead of 40. However, there was no interaction between test‐taking style (fast vs. slow) and time limits. ‘That is’, examinees who described themselves as slow writers/test takers did not benefit any more (or any less) from generous time limits than did their quicker counterparts. In addition, there was no detectable effect of different time limits on the meaning of essay scores, as suggested by their relationship to several nontest indicators of writing ability.
Item‐Bundle DIF Hypothesis Testing: Identifying Suspect Bundles and Assessing Their Differential Functioning
Authors: Douglas Jeffrey A. , Roussos Louis A. , Stout William
Published: September 2005

This article proposes two multidimensional IRT model‐based methods of selecting item bundles (clusters of not necessarily adjacent items chosen according to some organizational principle) suspected of displaying DIF amplification. The approach embodied in these two methods is inspired by Shealy and Stout's (1993a, 1993b) multidimensional model for DIF. Each bundle selected by these methods constitutes a DIF amplification hypothesis. When SIBTEST (Shealy & Stout, 1993b) confirms DIF amplification in selected bundles, differential bundle functioning (DBF) is said to occur. Three real data examples illustrate the two methods for suspect bundle selection. The effectiveness of the methods is argued on statistical grounds. A distinction between benign and adverse DIF is made. The decision whether flagged DIF items or DBF bundles display benign or adverse DIF/DBF must depend in part on nonstatistical construct validity arguments. Conducting DBF analyses using these methods should help in the identification of the causes of DIF/DBF.
Developing and Scoring an Innovative Computerized Writing Assessment
Authors: Davey Tim , Godwin Janet , Mittelholtz David
Published: September 2005

We describe the development and administration of a recently introduced computer‐based test of writing skills. This test asks the examinee to edit a writing passage presented on a computer screen. To do this, the examinee moves a cursor to a suspect section of the passage and chooses from a list of alternative ways o f rewriting that section. Any or all parts o f the passage can be changed, as often as the examinee likes. An able examinee identifies and fixes errors in grammar, organization, and style, whereas a less able examinee may leave errors untouched, replace an error with another error, or even introduce errors where none existed previously. All these response alternatives contrive to present both obvious and subtle scoring difficulties. These difficulties were attacked through the combined use of option weighting and the sequential probability ratio test, the result o f which is to classify examinees into several discrete ability groups. Item calibration was enabled by augmenting sparse pretest samples through data meiosis, in which response vectors were randomly recombined to produce offspring that retained much of the character of their parents. These procedures are described, and operational examples are offered.
A Computer‐Based Task for Measuring the Representational Component of Quantitative Proficiency
Authors: Bennett Randy Elliot , Sebrechts Marc M.
Published: September 2005

In this study, we created a computer‐delivered problem‐solving task based on the cognitive research literature and investigated its validity for graduate admissions assessment. The task asked examinees to sort mathematical word problem stems according to prototypes. Data analyses focused on the meaning of sorting scores and examinee perceptions of the task. Results showed that those who sorted well tended to have higher GRE General Test scores and college grades than did examinees who sorted less proficiently. Examinees generally preferred this task to multiple‐choice items like those found on the General Test's Quantitative section and felt the task was a fairer measure of their ability to succeed in graduate school. Adaptations of the task might be used in admissions tests, as well as for instructional assessments to help lower‐ scoring examinees localize and remediate problem‐solving difficulties.
Variance Estimation for Differential Test Functioning Based on Mantel‐Haenszel Statistics
Authors: Camilli Gregory , Penfield Douglas A.
Published: September 2005

This article concerns the simultaneous assessment of DIF for a collection of test items. Rather than an average or sum in which positive and negative DIF may cancel, we propose an index that measures the variance of DIF on a test as an indicator of the degree to which different items show DIF in different directions. It is computed from standard Mantel‐Haenszel statistics (the logodds ratio and its variance error) and may be conceptually classified as a variance component or variance effect size. Evaluated by simulation under three item response models (IPL, 2PL, and 3PL), the index is shown to be an accurate estimate of the DTF generating parameter in the case of the 1PL and 2PL models with groups of equal ability. For groups of unequal ability, the index is accurate under the I PL but not the 2PL condition; however, a weighted version of the index provides improved estimates. For the 3PL condition, the DTF generating parameter is underestimated. This latter result is due in part to a mismatch in the scales of the log‐odds ratio and IRT difficulty.
Evaluating an Automatically Scorable, Open‐Ended Response Type for Measuring Mathematical Reasoning in Computer‐Adaptive Tests
Authors: Bennett Randy Elliot , Steffen Manfred , Singley Mark Kevin , Morley Mary , Jacquemin Daniel
Published: September 2005

The first generation of computer‐based tests depends largely on multiple‐choice items and constructed‐response questions that can be scored through literal matches with a key. This study evaluated scoring accuracy and item functioning for an open‐ended response type where correct answers, posed as mathematical expressions, can take many different surface forms. Items were administered to 1,864 participants in field trials of a new admissions test for quantitatively oriented graduate programs. Results showed automatic scoring to approximate the accuracy of multiple‐choice scanning, with all processing errors stemming from examinees improperly entering responses. In addition, the items functioned similarly in difficulty, item‐total relations, and male‐female performance differences to other response types being considered for the measure.
Information and Reliability for Percentile Ranks and Other Monotonic Transformations of the Number‐Correct Score: Reply to De Gruijter
Authors: May Kim O. , Nicewander W. Alan
Published: September 2005

De Gruijter (1997) has claimed that May and Nicewander (1994) erred on two points that indicated that both reliability and information in the number‐correct score could be reduced considerably by converting to percentile ranks. The points raised by De Gruijter are very well taken, and we are pleased to have an opportunity to respond.
Measurement Characteristics of the lEA Reading Literacy Scales for 9‐ and 10‐Year‐Olds at Country and Individual Levels
Authors: Gustafsson Jan‐Eric
Published: September 2005

This article presents an approach for studying measurement characteristics of instruments designed for comparative studies of educational achievement. It presents a reanalysis of data from 22 countries and 73,818 9‐ and 10‐year‐old students from the IEA reading literacy study. Using 2‐level confirmatory factor analysis procedures derived by Muthén (1990, 1994), it is demonstrated how the overall variance in performance on items from 3 domains—‐Documents, Narrative Prose, and Expository Text—may be decomposed into multiple latent sources of variance at country and individual levels. For the Documents items, a 1‐factor model was fitted at both levels. For Narrative Prose and Expository Text items, 2 latent variables (a general dimension of reading performance and a factor capturing high versus low performance on passages towards the end of the tests) were fitted at both country and individual levels. For the Documents dimension, estimates of country scores were close to those of the original analysis; however, for performance in the Narrative Prose and Expository Text domains, differences between the original analysis and the reanalysis were noted for many countries.
Choice Among Essay Topics: Impact on Performance and Validity
Authors: Bridgeman Blent , Morgan Rick , Wang Ming‐mei
Published: September 2005

This study assessed the ability of history students to choose the essay topic on which they can get the highest score. A second, equally important question was whether the score on the chosen topic was more highly related to other indicators of proficiency in history than the score on the unchosen topic. Overall, for both U.S. and European history, scores were about one third of a standard deviation higher for the preferred topic than for the other topic. For U.S. history, about 32% of the students made the wrong choice; that is, 32% got a higher score on the other topic than on the preferred topic. In European history, 29% made the wrong choice. In the U.S. history sample, the preferred essay correlated .40 with an external criterion score, compared to .34 for the other essay; in the European history sample, the preferred essay correlated .52 with the external criterion, compared to .44 for the other topic.
A Tree‐Based Approach to Proficiency Scaling and Diagnostic Assessment
Authors: Sheehan Kathleen M.
Published: September 2005

A new procedure for generating instructionally relevant diagnostic feedback is proposed. The approach involves first constructing a strong model of student proficiency and then testing whether individual students’ observed item response vectors are consistent with that model. Diagnoses are specified in terms of the combinations of skills needed to score at increasingly higher levels on a test's reported score scale. The approach is applied to the problem of developing diagnostic feedback for the SAT I Verbal Reasoning test. Using a variation of Wright's (1977) person‐fit statistic, it is shown that the estimated proficiency mode accounts for 91% of the “explainable” variation in students’ observed item response vectors.
The Effect of Coaching on the Predictive Validity of Scholastic Aptitude Tests
Authors: Allalouf Avi , Ben‐Shakhar Gershon
Published: September 2005

The present study was designed to examine whether coaching affects the predictive validity and fairness of scholastic aptitude tests. Two randomly allocated groups, coached and uncoached, were compared, and the results revealed that although coaching enhanced scores on the Israeli Psychometric Entrance Test by about 25% of a standard deviation, it did not affect predictive validity and did not create a prediction bias. These results refute claims that coaching reduces predictive validity and creates a bias against the uncoached examinees in predicting the criterion. The results are consistent with the idea that score improvement due to coaching does not result strictly from learning specific skills that are irrelevant to the criterion.
A Comparison of Developmental Scales Based on Thurstone Methods and Item Response Theory
Authors: Williams Valerie S. L. , Pommerich Mary , Thissen David
Published: September 2005

A developmental scale for the North Carolina End‐of‐Grade Mathematics Tests was created using a subset of identical test forms administered to adjacent grade levels. Thurstone scaling and item response theory (IRT) techniques were employed to analyze the changes in grade distributions across these linked forms.Three variations of Thurstone scaling were examined, one based on Thurstone's 1925 procedure and two based on Thurstone's 1938 procedure. The IRT scaling was implemented using both BiMain and Multilog.All methods indicated that average mathematics performance improved from Grade 3 to Grade 8, with similar results for the two IRT analyses and one version of Thurstone's 1938 method.The standard deviations of the IRT scales did not show a consistent pattern across grades, whereas those produced by Thurstone's 1925 procedure generally decreased; one version of the 1938 method exhibited slightly increasing variation with increasing grade level, while the other version displayed inconsistent trends.
Simultaneous Assembly of Multiple Test Forms
Authors: van der Linden Wim J. , Adema Jos J.
Published: September 2005

An algorithm for the assembly of multiple test forms is proposed in which the multiple‐form problem is reduced to a series of computationally less intensive two‐form problems. At each step, one form is assembled to its true specifications; the other form is a dummy assembled only to maintain a balance between the quality of the current form and the remaining forms. It is shown how the method can be implemented using the technique of O‐1 linear programming. Two empirical examples using a former item pool from the LSAT are given—one in which a set of parallel forms is assembled and another in which the targets for the information functions of the forms are shifted systematically.
Some Practical Examples of Computer‐Adaptive Sequential Testing
Authors: Luecht Richard M. , Nungester Ronald J.
Published: September 2005

Computerized testing has created new challenges for the production and administration of test forms. Many testing organizations engaged in or considering computerized testing may find themselves changing from well‐established procedures for handcrafiing a small number of paper‐and‐pencil test forms to procedures for mass producing many computerized test forms. This paper describes an integratedapproach to test development and administration called computer‐adaptive sequential testing, or CAST. CAST is a structured approach to test construction which incorporates both adaptive testing methods with automated test assembly to allow test developers to maintain a greater degree of control over the production, quality assurance, and administration of different types of computerized tests. CAST retains much of the efficiency of traditional computer adaptive testing (CAT) and can be modified for computer mastery testing (CMT) applications. The CAST framework is described in detail and several applications are demonstrated using a medical licensure example.
Projecting to the NAEP Scale: Results from the North Carolina End‐of‐Grade Testing Program
Authors: Williams Valerie S. L. , Rosa Kathleen Rees , McLeod Lori D. , Thissen David , Sanford Eleanor E.
Published: September 2005

Data from the North Carolina End‐of‐Grade test of eighth‐grade mathematics are used to estimate the achievement results on the scale of the National Assessment of Educational Progress (NAEP) Trial State Assessment. Linear regression models are used to develop projection equations to predict state NAEP results in the future, and the results of such predictions are compared with those obtained in the 1996 administration of NAEP Standard errors of the parameter estimates are obtained using a bootstrap resampling technique.
Components of Rater Error in a Complex Performance Assessment
Authors: Clauser Brian E. , Clyman Stephen G. , Swanson David B.
Published: September 2005

Numerous studies have examined performance assessment data using generaliz‐ability theory. Typically, these studies have treated raters as randomly sampled from a population, with each rater judging a given performance on a single occasion. This paper presents two studies that focus on aspects of the rating process that are not explicitly accounted for in this typical design. The first study makes explicit the “committee” facet, acknowledging that raters often work within groups. The second study makes explicit the “rating‐occasion” facet by having each rater judge each performance on two separate occasions. The results of the first study highlight the importance of clearly specifying the relevant facets of the universe of interest. Failing to include the committee facet led to an overly optimistic estimate of the precision of the measurement procedure. By contrast, failing to include the rating‐occasion facet, in the second study, had minimal impact on the estimated error variance.