bluuebunny/crossref_metadata_2025 huggingface dataset:
Sociocultural Differences in Examinees' Attitudes Toward Scholastic Ability Exams
Authors: Zeidner Moshe
Published: September 2005

The major aim of the present study was threefold: (a) to compare the test attitudes and perceptions o f examinees of varying sociocultural group membership toward verbal and nonverbal standardized ability tests; (b) to determine the degree of covariation between test attitudes and test scores; and (c) to delineate the properties and potential applications of a test attitude or feedback inventory specifically designed to assess examinees’ perceptions of key situational variables in the test context. The feedback inventory was administered to a sample of 259 seventh grade students in Israel immediately following standardized group scholastic ability testing procedures. On the whole, few meaningful group differences in test attitudes were observed by social class, ethnicity, or sex. However, a nonverbal test was generally rated more favorably than a verbal test, among varying sociocultural and sex subgroups. Considered together, test attitude scales share a meaningful proportion o f variance with the test score on both verbal and nonverbal tests. However, in view o f the negligible ethnic and social class differences in test attitudes and the nonsignificant interaction between test attitudes and background variables, the data provide little support for the situational bias claim
The Influence of Labels and Positions in Rating Scales
Authors: Klockars Alan J. , Yamagishi Midori
Published: September 2005

On occasion, rating scales are used in which the verbal labels are compressed to saturate one end of the response continuum. In these scales, differences arise between the normal meaning of the label and its scalar position. This paper investigated the influence of the label and the position in defining the meaning of the labeled position. Three forms of labeled rating scale were constructed where the labels FAIR and GOOD were systematically moved. Subjects judged the desirability of premarked responses on the scales. In addition, two forms were used to determine the desirability of the unlabeled scale position and the desirability of the label independent of a rating scale. When label and position differed in meaning, the labeled position was rated as a compromise between the scale value of the label and the position
Comparing State SAT Scores: Problems, Biases, and Corrections
Authors: Gohmann Stephan F.
Published: September 2005

Comparing SAT scores among states using regression analysis leads to biased results because states differ in the proportion of students taking the exam. When the proportion of students taking the exam is included in the regression equation, the results can be biased because of misspecifieation bias. A method intended to correct for selection bias is presented, and empirical results suggest that sample selection bias is present in SAT score regressions. Regression equations and state rankings are compared between the selection‐corrected equation and equations for which the selection problem is not addressed. The proposed method is one of many available as possible solutions to the selection problem. Alternative methods may produce different results
Differential Item Functioning Resulting From The Use of Different Solution Strategies
Authors: Tatsuoka Kikumi K. , Linn Robert L. , Tatsuoka Maurice M. , Yamamoto Kentaro
Published: September 2005

The present study investigates the degree to which item “bias” techniques can lead to interpretable results when groups are defined in terms of specified differences in the cognitive processes involved in students' problem‐solving strategies. A large group of junior high school students who took a test on subtraction of fractions was divided into two subgroups judged by the rule‐space model to be using different problem‐solving strategies. It was confirmed by use of Mantel‐Haenszel (MH) statistics that these subgroups showed different performances on items with different underlying cognitive tasks. We note that, in our case, we are far from faulting items that show differential item functioning (D1F) between two groups defined in terms of different solution strategies. Indeed, they are “desirable” items, as explained in the discussion section
Gender Bias in the Prediction of College Course Performance
Authors: MeCornack Robert L. , McLeod Mary M.
Published: September 2005

Is the relationship of college grades to the traditional predictors of aptitude test scores and high school grades different for men and women? The usual gender bias of underpredicting the grade point averages of women may result from gender‐related course selection effects. This study controlled course selection effects by predicting single course grades rather than a composite grade from several courses. In most of the large introductory courses studied, no gender bias was found that would hold up on cross‐validation in a subsequent semester. Usually, it was counterproductive to adjust grade predictions according to gender. Grade point average was predicted more accurately than single course grades
The Raven Progressive Matrices: A Review of National Norming Studies and Ethnic and Socioeconomic Variation Within the United States
Authors: Raven John
Published: September 2005

In this paper, some recent results relating to the stability o f scores on the Raven Progressive Matrices Test for different subgroups within and between the United Kingdom, the United States, and other Western societies are summarised. Subsequent sections deal with variation over time. A possible explanation for the variation in norms over time and between ethnic groups within the United States is offered.
Combining Data on Criticality and Frequency in Developing Test Plans for Licensure and Certification Examinations
Authors: Kane Michael T. , Kingshury Carole , Colton Dean , Estes Carmen
Published: September 2005

Job analysis is a critical component in evaluating the validity of many high‐stakes testing programs, particularly those used for licensure or certification. The ratings of criticality and frequency of various activities that are derived from such job analyses can be combined in a number of ways. This paper develops a multiplicative model as a natural and effective way to combine ratings o f frequency and criticality in order to obtain estimates of the relative importance of different activities for practice. An example of the model's use is presented. The multiplicative model incorporates adjustments to ensure that the effective weights of frequency and criticality are appropriate.
Differential Item Functioning on the SAT‐M Braille Edition
Authors: Bennett Randy Elliot , Rock Donald A. , Novatkoski Inge
Published: September 2005

This study attempted to pinpoint the causes of differential item difficulty for blind students taking the braille edition of the Scholastic Aptitude Test's Mathematical section (SAT‐M). The study method involved reviewing the literature to identify factors that might cause differential item functioning for these examinees, forming item categories based on these factors, identifying categories that functioned differentially, and assessing the functioning o f the items comprising deviant categories to determine if the differential effect was pervasive. Results showed an association between selected item categories and differential functioning, particularly for items that included figures in the stimulus, items for which spatial estimation was helpful in eliminating at least two of the options, and items that presented figures that were small or medium in size. The precise meaning of this association was unclear, however, because some items from the suspected categories functioned normally, factors other than the hypothesized ones might have caused the observed aberrant item behavior, and the differential difficulty might reflect real population differences in relevant content knowledge
The Relationship of Content Characteristics of GRE Analytical Reasoning Items to Their Difficulties and Discriminations
Authors: Chalifour Clark L. , Powers Donald E.
Published: September 2005

In actual test development practice, the number o f test items that must be developed and pretested is typically greater, and sometimes much greater, than the number that is eventually judged suitable for use in operational test forms. This has proven to be especially true for one item type–analytical reasoning‐that currently forms the bulk of the analytical ability measure of the GRE General Test. This study involved coding the content characteristics of some 1,400 GRE analytical reasoning items. These characteristics were correlated with indices of item difficulty and discrimination. Several item characteristics were predictive of the difficulty of analytical reasoning items. Generally, these same variables also predicted item discrimination, but to a lesser degree. The results suggest several content characteristics that could be considered in extending the current specifications for analytical reasoning items. The use of these item features may also contribute to greater efficiency in developing such items. Finally, the influence of these various characteristics also provides a better understanding of the construct validity of the analytical reasoning item type.
A Method for Studying Differential Distractor Functioning
Authors: Green Bert F. , Crone Carolyn R. , Folk Valerie Greaud
Published: September 2005

A method of analyzing test item responses is advocated to examine differential item functioning through distractor choices of those who answer an item incorrectly. The analysis, called Differential Distractor Functioning, uses log‐linear models of a three‐way contingency table to examine whether there is an interaction of population subgroup and option choice when ability is held constant. The analysis is explained and is exemplified in an analysis of the Verbal portion of a recent Scholastic Aptitude Test.
Multiple‐Choice Models: The Distractors Are Also Part of the Item
Authors: Thissen David , Steinberg Lynne , Fitzpatrick Anne R.
Published: September 2005

This paper describes an item response model for multiple‐choice items and illustrates its application in item analysis. The model provides parametric and graphical summaries of the performance of each alternative associated with a multiple‐choice item; the summaries describe each alternative's relationship to the proficiency being measured. The interpretation of the parameters of the multiple‐choice model and the use of the model in item analysis are illustrated using data obtained from a pilot test of mathematics achievement items. The use of such item analysis for the detection of flawed items, for item design and development, and for test construction is discussed.
Computer‐Assisted Writing of Achievement Test Items: Toward a Future Technology
Authors: Millman Jason , Westman Ronald S.
Published: September 2005

Five appproaches to writing test items with computer assistance are descibed. The authors propose a scheme for modeling knowledge by a set of structures and outline a system for implementing it.
The Future of Item Analysis
Authors: Wainer Howard
Published: September 2005

This paper reviews the role of the item in test construction and suggests some new methods of item analysis. A look at dynamic, graphical item analysis is provided that utilizes the advantages of modern high speed, highly interactive computing. Several illustrations are provided
Trace Lines for Testlets: A Use of Multiple‐Categorical‐Response Models
Authors: Thissen David , Steinberg Lynne , Mooney Jo Ann
Published: September 2005

It is not always convenient or appropriate to construct tests in which individual items are fungible. There are situations in which small clusters of items (testlets) are the units that are assembled to create a test. Using data from a test of reading comprehension constructed of four passages with several questions following each passage, we show that local independence fails at the level of the individual questions. The questions following each passage, however, constitute a testlet. We discuss the application to testlet scoring of some multiple‐category models originally developed for individual items, In the example examined, the concurrent validity of the testlet scoring equaled or exceeded that of individual‐item‐level scoring
Effect of the Medium of Item Presentation on Examinee Performance and Item Characteristics
Authors: Spray Judith A. , Ackerman Terry A. , Reckase Mark D. , Carlson James E.
Published: September 2005

Studies that have investigated differences in examinee performance on items administered in paper‐and‐pencil form or on a computer screen have produced equivocal results. Certain item administration procedures were hypothesized to be among the most important variables causing differences in item performance and ultimately in test scores obtained from these different administration media. A study where these item administration procedures were made as identical as possible for each presentation medium is described. In addition, a methodology is presented for studying the difficulty and discrimination of items under each presentation medium as a post hoc procedure.
Assessing Error in Behavioral Data: Problems of Sequencing
Authors: Rowley Glenn L.
Published: September 2005

Both classical test theory and generalizability theory focus on measurement error as a group property. Thus, common estimates o f errors o f measurement are developed for all members o f a group. But with behavioral data, unlike specifically test data, it is sometimes possible to estimate error separately for each individual. This enables one to ask questions about the relationships between error o f measurement and other characteristics o f the individual. Consequently, it also makes possible the use of regression techniques to call upon group data to improve the estimates o f individual measurement error. The focus on the individual also lays bare the possibility o f sequencing effects, and it is shown that, even in the absence o f trend, autocorrelation can cause standard procedures to grossly underestimate the magnitude o f measurement error. Classroom observation data are examined for autocorrelation, and recommendations are made about the scheduling o f data collection so as to minimize its effects.
Effects of Item Wording on Sex Bias
Authors: McLarty Joyce R. , Noble A. Candace , Huntley Renee M.
Published: September 2005

This study examined the effects of gender‐related item‐wording changes on the performance of male and female examinees. Mathematics word problems and English language items were created in neuter, male, and female versions. Items were administered to randomly equivalent samples of about 300 high school juniors and seniors. Loglinear analysis was used to assess the impact of item gender and its interaction with examinee sex on the difficulty and discrimination of each item in each context. No items were found to have sex bias in either context. Mathematics items did not have different difficulty or discrimination in the three gender versions. Neither mathematics nor English items had different discrimination levels in the three gender‐related versions. Some English items, however, were found to have different difficulty levels in the three gender‐related versions. These difficulty differences were not systematic.” none of the three gender versions appeared consistently more or less difficult than the others.
Reply to Andrich and Henning
Authors: Divgi D. R.
Published: September 2005

This paper discusses Andrich's (1989) criticisms of three points made by Divgi (1986). Some important details omitted in Divgi's earlier paper concerning the inconsistency of the unconditional estimator of Rasch difficulties are provided. In the cases of specific objectivity and person‐fit statistics, Andrich criticizes positions Divgi never took or meant to imply. The paper also answers some criticisms by Henning (1989).
On the Stability of Students’Rules of Operation for Solving Arithmetic Problems
Authors: Tatsuoka Kikumi K. , Birenbaum Menucha , Arnold Jerry
Published: September 2005

The purpose o f this study was to examine the consistency with which students applied procedural rules for solving signed‐number operations across identical items presented in different orders. A test with 64 open‐ended items was administered to 161 eighth graders. The test consisted o f two 32‐item subtests containing identical items. The items in each subtest were in random order. Students’responses to each subtest were compared with respect to the identified underlying rules o f operation used to solve each problem set. The results indicated that inconsistent rule application was common among students who had not mastered signed‐number arithmetic operations. In contrast, mastery level students, those who use the right rules, show a stable pattern o f rule application in signed‐number arithmetic. These results are discussed in light of the hypothesis testing approach to the learning process.
Toward a Psychometrics for Testlets
Authors: Wainer Howard , Lewis Charles
Published: September 2005

In 1987 the testlet was introduced as one way of dealing with a variety of problems that might occur with the algorithmic construction of tests. In the short time since its introduction, its range of plausible usages has been broadened considerably through the work of other researchers. In this paper we examine three different applications of the testlet concept and describe the psychometric models which seem most suitable for each application. In each case, the testlet concept gracefully solves a problem that would have been awkward with other, more traditional approaches.
Estimation of Classification Consistency When the Probability of a Correct Response Varies
Authors: Spray Judith A. , Welch Catherine J.
Published: September 2005

The purpose of this study was to examine the effect that large, within‐examinee item difficulty variability had on estimates of the proportion of consistent classification of examinees into mastery categories over two test administrations. The classification consistency estimate was based on a single test administration from an estimation procedure suggested by Subkoviak (1976). Analyses of simulated data revealed that the use of a single estimate for an examinee's probability of a correct response, even when that probability varied greatly within a test for an examinee, did not affect the estimation of the proportion of consistent classifications.
Differential Item Functioning for Minority Examinees on the SAT
Authors: Sehmitt Alieia P. , Dorans Nell J.
Published: September 2005

The standardization approach to assessing differential item functioning (DIF), including standardized distractor analysis, is described. The results of studies conducted on Asian Americans, Hispanics (Mexican Americans and Puerto Ricans), and Blacks on the Scholastic Aptitude Test (SAT) are described and then synthesized across studies. Where the groups were limited to include only examinees who spoke English as their best language, very few items across forms and ethnic groups exhibited large DIF. Major findings include evidence of differential speededness (where minority examinees did not complete SAT‐Verbal sections at the same rate as White students with comparable SAT‐Verbal scores) for Blacks and Hispanics and, when the item content is of special interest, advantages for the relevant ethnic group. In addition, homographs tend to disadvantage all three ethnic groups, but the effect of vertical relationships in analogy items are not as consistent. Although these findings are important in understanding DIF, they do not seem to account for all differences. Other variables related to DIF still need to be identified. Furthermore, these findings are seen as tentative until corroborated by studies using controlled data collection designs.
Using Differential Item Functioning Procedures to Explore Sources of Item Difficulty and Group Performance Characteristics
Authors: Scheuneman Janice Dowd , Gerritz Kalle
Published: September 2005

Statistics used to detect differential item functioning can also reflect differential strengths and weaknesses in the performance characteristics of population subgroups. In turn, item features associated with the differential performance patterns are likely to reflect some facet of the item task and hence its difficulty, that might previously have been overlooked. In this study, several item features were identified and coded for a large number of reading comprehension items from the two admissions testing programs. Item features included subject matter content, various properties of item structure, cognitive demand indicators, and semantic content (propositional analysis). Differential item functioning was evaluated for males and females and for White and Black examinees. Results showed a number of significant relationships between item features and indicators of differential item functioning—many of which were consistent across testing programs. Implications of the results for related areas of research are discussed.
A Simulation and Comparison of Flexilevel and Bayesian Computerized Adaptive Testing
Authors: De Ayala R. J. , Dodd Barbara G. , Koch William R.
Published: September 2005

Computerized adaptive testing (CAT) is a testing procedure that adapts an examination to an examinee's ability by administering only items of appropriate difficulty for the examinee. In this study, the authors compared Lord's flexilevel testing procedure (flexilevel CAT) with an item response theory‐based CAT using Bayesian estimation of ability (Bayesian CAT). Three flexilevel CATs, which differed in test length (36, 18, and 11 items), and three Bayesian CATs were simulated; the Bayesian CATs differed from one another in the standard error of estimate (SEE) used for terminating the test (0.25, 0.10, and 0.05). Results showed that the flexilevel 36‐ and 18‐item CATs produced ability estimates that may be considered as accurate as those of the Bayesian CAT with SEE = 0.10 and comparable to the Bayesian CAT with SEE = 0.05. The authors discuss the implications for classroom testing and for item response theory‐based CAT.
The Construction of Customized Two‐Stage Tests
Authors: Adema Jos J.
Published: September 2005

In this paper mixed integer linear programming models for customizing two‐stage tests are given. Model constraints are imposed with respect to test composition, administration time, inter‐item dependencies, and other practical considerations. It is not difficult to modify the models to make them useful in constructing multistage tests.
Item Difficulty of Four Verbal Item Types and an Index of Differential Item Functioning for Black and White Examinees
Authors: Freedle Roy , Kostin Irene
Published: September 2005

In this study, the authors explored the importance of item difficulty (equated delta) as a predictor of differential item functioning (DIF) of Black versus matched White examinees for four verbal item types (analogies, antonyms, sentence completions, reading comprehension) using 13 GRE‐disclosed forms (988 verbal items) and 11 SAT‐disclosed forms (935 verbal items). The average correlation across test forms for each item type (and often the correlation for each individual test form as well) revealed a significant relationship between item difficulty and DIF value for both GRE and SAT. The most important finding indicates that for hard items, Black examinees perform differentially better than matched ability White examinees for each of the four item types and for both the GRE and SAT tests! The results further suggest that the amount of verbal context is an important determinant of the magnitude of the relationship between item difficulty and differential performance of Black versus matched White examinees. Several hypotheses accounting for this result were explored.
Instructionally Sensitive Psychometrics: Application of a New IRT‐Based Detection Technique to Mathematics Achievement Test Items
Authors: Muthén Bengt O. , Kao Chih‐Fen , Burstein Leigh
Published: September 2005

Achievement modeling is carried out in groups of students characterized by heterogeneous instructional background. Extensions of item response theory models incorporate variables reflecting different amounts of opportunity‐to‐leam (OTL). The effects of these OTL variables are studied with respect to their influence on both the latent trait and the item performance directly. Such direct effects may reflect instructionally sensitive items. U.S. eighth‐grade mathematics data from the Second International Mathematics Study are analyzed. Here, the same test is taken by students enrolled in typical instruction and students enrolled in elementary algebra classes. It is shown that the new analysis provides a more detailed way to examine the influence of instruction on responses to test items than does conventional item response theory.
Gender Bias in Predicting College Academic Performance: A New Approach Using Item Response Theory
Authors: Young John W.
Published: September 2005

The possibility of differential prediction of college academic performance for men and women is, at present, an issue of concern both to measurement specialists and to the general public. In prior studies, the use of a single equation to predict grade point average (GPA) from preadmissions measures has generally led to the systematic underprediction of women's college grades. In the following study, Item Response Theory (IRT) was used to develop a form of adjusted cumulative GPA, called the IRT‐based GPA. Significant underprediction for women occurred using a version of the cumulative GPA as the outcome measure. In contrast, the use of the IRT‐based GPA indicated no significant underprediction for men or women. A single regression equation worked best in predicting both men's and women's IRT‐based GPA. In addition, the IRT‐based GPA was substantially more predictable from preadmissions measures than the cumulative GPA.
Influence of Prior Distributions on Detection of DIF
Authors: Cohen Allan S. , Kim Seock‐Ho , Subkoviak Michael J.
Published: September 2005

Detection of differential item functioning (DIF) on items intentionally constructed to favor one group over another was investigated on item parameter estimates obtained from two item response theory‐based computer programs, LOGIST and BILOG. Signed‐ and unsigned‐area measures based on joint maximum likelihood estimation, marginal maximum likelihood estimation, and two marginal maximum a posteriori estimation procedures were compared with each other to determine whether detection of DIF could be improved using prior distributions. Results indicated that item parameter estimates obtained using either prior condition were less deviant than when priors were not used. Differences in detection of DIF appeared to be related to item parameter estimation condition and to some extent to sample size.
Longitudinal Analyses of Academic Achievement
Authors: Williamson Gary L. , Appelbaum Mark , Epanchin Alex
Published: September 2005

Mathematical models of individual growth are the basis for analyzing academic achievement over time. This study demonstrates that much can be learned about academic growth from the analysis of individual growth curves. In addition, we illustrate the aggregation of individual responses to provide descriptions of institutional growth. Although more efficient statistical methods are available, the simple approach taken here serves to demonstrate the logic and approach to analyzing longitudinal data.
Current Validity of 1975 and 1985 SATs: Implications for Validity Trends Since the Mid‐1970s
Authors: Stricker Lawrence J.
Published: September 2005

The aim of this study was to appraise whether different forms of the SA T used since the mid‐1970s varied in their correlations with academic performance criteria in the same cohort of examinees. A 1975 form and a 1985 form were administered to two random samples of high school juniors, and self‐reported grade point average and high school rank were obtained. The SAT‐Verbal and SAT‐Mathematical scores generally had similar correlations with the grade criteria in the two samples. The principal conclusion is that the 1975 form of the SAT does not have greater validity than the 1985 form in assessing academic performance, at least at the high school level. This outcome offers no support for the hypothesis that the decline in the SAT's ability to predict college grades since the mid‐1970s, observed in recent research, is attributable to changes in the test.
A Comparison of Item Sampling Plans in the Application of Multiple Matrix Sampling
Authors: Gressard Risa P. , Loyd Brenda H.
Published: September 2005

This study used a Monte Carlo approach to investigate the effect of item sampling by item stratification on parameter estimation when applying multiple matrix sampling to achievement data. From the results of this study it was concluded that the item sampling method and sampling plan which is a practical compromise in terms of precision and sample size is one based on item stratification by item discrimination and a sampling plan with a moderate number of subtests. This sampling condition provides reasonable precision of the mean and variance estimates but requires only a moderately sized sample.
A Comparison of Multiple‐Choice and Constructed Figural Response Items
Authors: Martinez Michael E.
Published: September 2005

In contrast to multiple‐choice test questions, figural response items call for constructed responses and rely upon figural material, such as illustrations and graphs, as the response medium. Figural response questions in various science domains were created and administered to a sample of 4th‐, 8th‐, and 12th‐grade students. Item and test statistics from parallel sets of figural response and multiple‐choice questions were compared. Figural response items were generally more difficult, especially for questions that were difficult (p < .5) in their constructed‐response forms. Figural response questions were also slightly more discriminating and reliable than their multiple‐choice counterparts, but they had higher omit rates. This article addresses the relevance of guessing to figural response items and the diagnostic value of the item type. Plans for future research on figural response items are discussed.
Reasons for Changing Answers: An Evaluation Using Personal Interviews
Authors: Schwarz Shirley P. , McMorris Robert F. , DeMers Lawrence P.
Published: September 2005

Researchers investigating answer changing have consistently found the preponderance of changes on objective items to be from wrong to right, but little is understood about the mechanisms involved in this phenomenon. In this study, personal interviews were combined with instruction in answer‐changing research to investigate further the processes involved in answer changing. Students changed answers and gained from changing, with those in the upper two thirds of the classes gaining the most. Each test‐taking strategy produced a mean gain, but particular strategies were not significantly correlated with percentage of gain or percentage of change. Most students reported changing answers for thoughtful reasons such as rereading, rethinking, or remembering more information; very few changes were due to clerical errors. For each reason, most changes were wrong‐to‐right. We conclude that reconsideration of test items is probably underestimated in answer‐changing studies. The role of memory should be considered in why people change and in how successful they judge their changing to have been.
Performance Modeling That Integrates Latent Trait and Class Theory
Authors: Gitomer Drew H. , Yamamoto Kentaro
Published: September 2005

The qualitative characterization of individual performance that is central to modem psychological theory is not adequately modeled by traditional psychometric theory that assumes, among other things, unidimensionality. In the present study, data are presented that are more adequately modeled by HYBRID, a model that incorporates both latent trait and latent class components. The latent classes were defined by a cognitive analysis of the understanding that individuals have for a circumscribed domain. In addition to providing a better statistical fit, the analysis also improves the amount of diagnostic information available for a given individual.
A Comparison of Angoff's Design I and Design II for Vertical Equating Using Traditional and IRT Methodology
Authors: Harris Deborah J.
Published: September 2005

Practical considerations in conducting an equating study often require a trade‐off between testing time and sample size. A counterbalanced design (Angoff's Design II) is often selected because, as each examinee is administered both test forms and therefore the errors are correlated, sample sizes can be dramatically reduced over those required by a spiraling design (Angoff's Design I), where each examinee is administered only one test form. However, the counterbalanced design may be subject to fatigue, practice, or context effects. This article investigated these two data collection designs (for a given sample size) with equipercentile and IRT equating methodology in the vertical equating of two mathematics achievement tests. Both designs and both methodologies were judged to adequately meet an equivalent expected score criterion; Design II was found to exhibit more stability over different samples.
Use of Restricted Item Response Models for Examining Item Difficulty Ordering and Slope Uniformity
Authors: Lane Suzanne
Published: September 2005

This article demonstrates the utility of restricted item response models for examining item difficulty ordering and slope uniformity for an item set that reflects varying cognitive processes. Twelve sets of paired algebra word problems were developed to systematically reflect various types of cognitive processes required for successful performance. This resulted in a total of 24 items. They reflected distance‐rate–time (DRT), interest, and area problems. Hypotheses concerning difficulty ordering and slope uniformity for the items were tested by constraining item difficulty and discrimination parameters in hierarchical item response models. The first set of model comparisons tested the equality of the discrimination and difficulty parameters for each set of paired items. The second set of model comparisons examined slope uniformity within the complex DRT problems. The third set of model comparisons examined whether the familiarity of the story context affected item difficulty for two types of complex DRT problems. The last set of model comparisons tested the hypothesized difficulty ordering of the items.
Building Algebra Testlets: A Comparison of Hierarchical and Linear Structures
Authors: Wainer Howard , Lewis Charles , Kaplan Bruce , Braswell James
Published: September 2005

Earlier (Wainer & Lewis, 1990), we reported the initial development of a testlet‐based algebra test. In this account, we provide the details of this excursion into the use of testlets. A pretest of two 15–item algebra tests was carried out in which examinees' performance on a 4‐item subset of each test (a 4–item testlet) was used to predict performance on the entire test. Two models for constructing the testlets were considered: hierarchical (adaptive) and linear (fixed format). These models are compared with each other. It was found on cross–validation that, although an adaptive testlet is superior to a fixed format testlet, this superiority is modest, whereas the potential cost of that superiority is considerable. It was concluded that in circumstances similar to those we report a fixed format testlet that uses the best items in a pool can do almost as well as the optimal adaptive testlet of equal length from that same pool.
Evaluation of Procedure‐Based Scoring for Hands‐On Science Assessment
Authors: Baxter Gail P. , Shavelson Richard J. , Goldman Susan R. , Pine Jerry
Published: September 2005

This article evaluates a procedure‐based scoring system for a performance assessment (an observed paper towels investigation) and a notebook surrogate completed by fifth‐grade students varying in hands‐on science experience. Results suggested interrater reliability of scores for observed performance and notebooks was adequate (>.80) with the reliability of the former higher. In contrast, interrater agreement on procedures was higher for observed hands‐on performance (.92) than for notebooks (.66). Moreover, for the notebooks, the reliability of scores and agreement on procedures varied by student experience, but this was not so for observed performance. Both the observed‐performance and notebook measures correlated less with traditional ability than did a multiple‐choice science achievement test. The correlation between the two performance assessments and the multiple‐choice test was only moderate (mean = .46), suggesting that different aspects of science achievement have been measured. Finally, the correlation between the observed‐performance scores and the notebook scores was .83, suggesting that notebooks may provide a reasonable, albeit less reliable, surrogate for the observed hands‐on performance of students.
Effects of Linking Methods on Detection of DIF
Authors: Kim Seock‐Ho , Cohen Allan S.
Published: September 2005

Studies of differential item functioning under item response theory require that item parameter estimates be placed on the same metric before comparisons can be made. The present study compared the effects of three methods for linking metrics: a weighted mean and sigma method (WMS); the test characteristic curve method (TCC); and the minimum chi‐square method (MCS), on detection of differential item functioning. Both iterative and noniterative linking procedures were compared for each method. Results indicated that detection of differentially functioning items following linking via the test characteristic curve method gave the most accurate results when the sample size was small. When the sample size was large, results for the three linking methods were essentially the same. Iterative linking provided an improvement in detection of differentially functioning items over noniterative linking particularly with the .05 alpha level. The weighted mean and sigma method showed greater improvement with iterative linking than either the test characteristic curve or minimum chi‐square method.
The Design of the National Assessment of Educational Progress
Authors: Johnson Eugene G.
Published: September 2005

The key features of the design of the National Assessment of Educational Progress (NAEP) are discussed with particular emphasis on the design to be used for the 1992 assessment. An overview of the design and its philosophy are given with a description of the multicomponent solution to the twin requirements of reliably measuring trends in achievement while responding to changing educational priorities and advances in measurement technology. The student sample designs for the National Assessment and the Trial State Assessment are described. The focused‐balanced incomplete block (focused‐BIB) spiraling method of item sampling is discussed and compared with simpler matrix sampling designs. The impact of the NAEP design on the analysis of assessment data is discussed.
Developing the NAEP Content‐Area Frameworks and Innovative Assessment Methods in the 1992 Assessments of Mathematics, Reading, and Writing
Authors: Mullis Ina V. S.
Published: September 2005

This article provides an overview of the consensus processes for the development of the frameworks underlying the NAEP assessments, with emphasis on those for the 1990 and 1992 assessments of mathematics, the 1992 assessment of reading, and the 1994 assessment of science. In addition, innovative assessment techniques included in the 1992 assessments of mathematics, reading, and writing are described, including use of mathematics tools, oral interviews, and portfolio assessment.
The Stability of IRT b Values
Authors: Sykes Robert C. , Fitzpatrick Anne R.
Published: September 2005

This study investigated possible explanations for an observed change in Rasch item parameters (b values) obtained from consecutive administrations of a professional licensure examination. Considered in this investigation were variables related to item position, item type, item content, and elapsed time between administrations of the item. An analysis of covariance methodology was used to assess the relations between these variables and change in item b values, with the elapsed time index serving to control for differences that could be attributed to average or pool changes in b values over time. A series of analysis of covariance models were fitted to the data in an attempt to identify item characteristics that were significantly related to the change in b values after the time elapsed between item administrations had been controlled. The findings indicated that the change in item b values was not related either to item position or to item type. A small, positive relationship between this change and elapsed time indicated that the pool b values were increasing over time. A test of simple effects suggested the presence of greater change for one of the content categories analyzed. These findings are interpreted, and suggestions for future research are provided.
The Consistency of Detecting Item Bias Across Different Test Administrations: Implications of Another Failure
Authors: Skaggs Gary , Lissitz Robert W.
Published: September 2005

Several item bias detection methods were applied to the analysis of bias among males and females for items from a curriculum‐based mathematics test. The focus of this analysis was the consistency of the methods across different test administrations of the same items. The results indicated that, of the methods studied, the Mantel–Haenszel (M–H) and IRT‐based sum‐of‐squares methods were the most consistent. However, the degree of reliability and agreement for these methods was modest at best. As with most prior research, no reasonable explanation could be found for the most consistently flagged items. A likely reason for this lies in the confusion of visible genetic group characteristics with their instructional backgrounds. A multidimensional perspective of item bias is proposed for future research that will take such confounding into account.
A Comparison of Quantitative Questions in Open‐Ended and Multiple‐Choice Formats
Authors: Bridgeman Brent
Published: September 2005

Open–ended counterparts to a set of items from the quantitative section of the Graduate Record Examination (GRE–Q) were developed. Examinees responded to these items by gridding a numerical answer on a machine‐readable answer sheet or by typing on a computer. The test section with the special answer sheets was administered at the end of a regular GRE administration. Test forms were spiraled so that random groups received either the grid‐in questions or the same questions in a multiple‐choice format. In a separate data collection effort, 364 paid volunteers who had recently taken the GRE used a computer keyboard to enter answers to the same set of questions. Despite substantial format differences noted for individual items, total scores for the multiple‐choice and open‐ended tests demonstrated remarkably similar correlational patterns. There were no significant interactions of test format with either gender or ethnicity.
Measurement Problems
Authors: Wainer Howard
Published: September 2005

History teaches the continuity of science; the developments of tomorrow have their genesis in the problems of today. Thus any attempt to look forward is well begun with an examination of unsettled questions. Since a clearer idea of where we are going smoothes the path into the unknown future, a periodic review of such questions is prudent. The present day, lying near the juncture of the centuries, is well suited for such a review. This article reports 16 unsolved problems in educational measurement and points toward what seem to be promising avenues of solution.
Small‐Sample Equating With Log‐Linear Smoothing
Authors: Livingston Samuel A.
Published: September 2005

This study investigated the extent to which log‐linear smoothing could improve the accuracy of common‐item equating by the chained equipercentile method in small samples of examinees. Examinee response data from a 100‐item test were used to create two overlapping forms of 58 items each, with 24 items in common. The criterion equating was a direct equipercentile equating of the two forms in the full population of 93,283 examinees. Anchor equatings were performed in samples of 25, 50, 100, and 200 examinees, with 50 pairs of samples at each size level. Four equatings were performed with each pair of samples: one based on unsmoothed distributions and three based on varying degrees of smoothing. Smoothing reduced, by at least half, the sample size required for a given degree of accuracy. Smoothing that preserved only two moments of the marginal distributions resulted in equatings that failed to capture the curvilinearity in the population equating.
On the Stability of Performance Assessments
Authors: Ruiz‐Primo Maria Araceli , Baxter Gail P. , Shavelson Richard J.
Published: September 2005

This study examined the stability of scores on two types of performance assessments, an observed hands‐on investigation and a notebook surrogate. Twenty‐nine sixth‐grade students in a hands‐on inquiry‐based science curriculum completed three investigations on two occasions separated by 5 months. Results indicated that: (a) the generalizability across occasions for relative decisions was, on average, moderate for the observed investigations (.52) and the notebooks (.50); (b) the generalizability for absolute decisions was only slightly lower; (c) the major source of measurement error was the person by occasion (residual) interaction; and (d) the procedures students used to carry out the investigations tended to change from one occasion to the other.
How to Equate Tests With Little or No Data
Authors: Mislevy Robert J. , Sheehan Kathleen M. , Wingersky Marilyn
Published: September 2005

Standard procedures for equating tests, including those based on item response theory (IRT), require item responses from large numbers of examinees. Such data may not be forthcoming for reasons theoretical, political, or practical. Information about items' operating characteristics may be available from other sources, however, such as content and format specifications, expert opinion, or psychological theories about the skills and strategies required to solve them. This article shows how, in the IRT framework, collateral information about items can be exploited to augment or even replace examinee responses when linking or equating new tests to established scales. The procedures are illustrated with data from the Pre‐Professional Skills Test (PPST).
Assessment of Differential Item Functioning for Performance Tasks
Authors: Zwick Rebecca , Donoghue John R. , Grima Angela
Published: September 2005

Although the belief has been expressed that performance assessments are intrinsically more fair than multiple‐choice measures, some forms of performance assessment may in fact be more likely than conventional tests to tap construct‐irrelevant factors. The assessment of differential item functioning (DIF) can be helpful in investigating the effect on subpopulations of the introduction of performance tasks. In this study, two extensions of the Mantel‐Haenszel (MH; 1959) procedure that may be useful in assessing DIP in performance measures were explored. The test of conditional association proposed by Mantel (1963) seems promising as a test of DIF for pofytomous items when the primary interest is in the between‐group difference in item means, conditional on some measure of ability. The generalized statistic proposed by Mantel and Haenszel may be more useful than Mantel's test when the entire response distributions of the groups are of interest. Simulation results showed that, for both inferential procedures, the studied item should be included in the matching variable, as in the dichotomous case. Descriptive statistics that index the magnitude of DIP were also investigated.