bluuebunny/crossref_metadata_2025 huggingface dataset:
Male‐Female Differences in SAT‐Verbal Ability Among Students of High SAT‐Mathematical Ability
Authors: Dorans Neil J. , Livingston Samuel A.
Published: September 2005

It has been suggested that females who score extremely high on the mathematical portions of the Scholastic Aptitude Test (SAT) do so because they have very high verbal skills, whereas some males score extremely high on the mathematical portion despite their relatively low verbal skills. This hypothesis was investigated with data from two SAT administrations by comparing the conditional distributions of SAT‐Verbal (SAT‐V), given SAT‐Mathematical (SAT‐M), for males and females. Evidence for and against the hypothesis was observed. As implied by the hypothesis, the males had lower conditional SAT‐V mean scores. However, in contradiction to the hypothesis, the males did not have greater conditional variances of SAT‐V given SAT‐M.
Limitations of the Score‐Difference Method in Detecting Cheating in Recognition Test Situations
Authors: Roberts Dennis M.
Published: September 2005

An examination is made of a score‐difference model for the detection of cheating based on the difference between two scores for an examinee: one based on the appropriate scoring key and another based on an alternative, inappropriate key. The normal approximation to the binomial distribution model is used to predict characteristics of chance test score distributions and difference score distributions. Results show that students' difference scores can vary widely by chance alone, even for short tests. It is argued that the score‐difference method could falsely accuse many students as cheaters. It is suggested that the score‐difference method is seriously flawed and has little to recommend it.
Gender‐Based Differential Item Performance in Mathematics Achievement Items
Authors: Doolittle Allen E. , Cleary T. Anne
Published: September 2005

A procedure for the detection of differential item performance (DIP) is used to investigate the relationships between characteristics of mathematics achievement items and gender differences in performance. Eight randomly equivalent samples of high school seniors were each given a unique form of the ACT Assessment Mathematics Usage Test (ACTM). Students without requisite mathematics courses were deleted from the samples to reduce the confounding effects of differences in instruction at the high school level. Signed measures of DIP were obtained for each item in the eight ACTM forms. These DIP estimates were then analyzed in a 6 × 8 (item category by form) experimental design. A significant item category effect was found indicating a relationship between item characteristics and gender‐based DIP. Predictions, based on previous research about the categories of items that would contribute to gender‐based DIP, were supported: Geometry and mathematics reasoning items were relatively more difficult for female examinees and the more algorithmic, computation‐oriented items were relatively easier.
Item Clusters and Computerized Adaptive Testing: A Case for Testlets
Authors: Wainer Howard , Kiely Gerard L.
Published: September 2005

It is observed that many sorts of difficulties may preclude the uneventful construction of tests by a computerized algorithm, such as those currently in favor in Computerized Adaptive Testing (CAT). In this essay we discuss a number of these problems, as well as some possible avenues of solution. We conclude with the development of the “testlet,” a bundle of items that can be arranged either hierarchically or linearly, thus maintaining the efficiency of an adaptive test while keeping the quality control of test construction that is possible currently only with careful expert scrutiny. Performance on the separate testlets is aggregated to yield ability estimates.
Confidence Intervals for True Scores Under an Answer‐Until‐Correct Scoring Procedure
Authors: Wilcox Rand R.
Published: September 2005

Under an answer‐until‐correct scoring procedure, many measurement problems can be solved when certain cognitive models of examinee behavior can be assumed (Wilcox, 1983). Point estimates of true score under these models are available, but the problem of obtaining a confidence interval has never been addressed. Two simple methods for obtaining a confidence interval are suggested that give good results when the sample size is reasonably large, say, greater than or equal to 20, and when true score is not too close to zero or one. A third procedure is suggested that can also be used to get slightly better results where again the sample size is assumed to be reasonably large and true score is not too close to zero or one. For small sample sizes or situations where true score is close to zero or one, a fourth procedure is described that always gives conservative results.
Assessing the Dimensionality of NAEP Reading Data
Authors: Zwick Rebecca
Published: September 2005

The reading data from the 1983–84 National Assessment of Educational Progress survey were scaled using a unidimensional item response theory model. To determine whether the responses to the reading items were consistent with unidimensionality, the full‐information factor analysis method developed by Bock and associates (1985) and Rosenbaum's (1984) test of unidimensionality, conditional (local) independence, and monotonicity were applied. Full‐information factor analysis involves the assumption of a particular item response function; the number of latent variables required to obtain a reasonable fit to the data is then determined. The Rosenbaum method provides a test of the more general hypothesis that the data can be represented by a model characterized by unidimensionality, conditional independence, and monotonicity. Results of both methods indicated that the reading items could be regarded as measures of a single dimension. Simulation studies were conducted to investigate the impact of balanced incomplete block (BIB) spiraling, used in NAEP to assign items to students, on methods of dimensionality assessment. In general, conclusions about dimensionality were the same for BIB‐spiraled data as for complete data.
Race‐Based Differential Prediction in Air Force Technical Training Programs
Authors: Houston Walter M. , Noviek Melvin R.
Published: September 2005

Bayesian Johnson‐Neyman methodology was used to investigate differential prediction by race in United States Air Force technical training programs. Meaningful Johnson‐Neyman regions of differences were found in eight of nine comparisons. In all nine training courses the regressions for blacks were flatter than for whites and the race‐differentiated regression lines intersected within the range of predictor test scores. In six cases the cut‐score for course qualification was within the Johnson‐Neyman region, and in every case the bias was negative for blacks. It is noted that if the cut score had been set substantially higher the bias would have been positive for blacks in all cases. This analysis may explain why earlier studies that averaged bias across the predictor distribution yielded mixed results. It is hypothesized that the consistent results obtained here are a consequence of the lower predictability found in the black subpopulation.
On the Use of IRT Models With Judgmental Standard Setting Procedures
Authors: Kane Michael T.
Published: September 2005

In judgmental standard setting procedures (e.g., the Angoff procedure), expert raters establish minimum pass levels (MPLs) for test items, and these MPLs are then combined to generate a passing score for the test. As suggested by Van der Linden (1982), item response theory (IRT) models may be useful in analyzing the results of judgmental standard setting studies. This paper examines three issues relevant to the use of lRT models in analyzing the results of such studies. First, a statistic for examining the fit of MPLs, based on judges' ratings, to an IRT model is suggested. Second, three methods for setting the passing score on a test based on item MPLs are analyzed; these analyses, based on theoretical models rather than empirical comparisons among the three methods, suggest that the traditional approach (i.e., setting the passing score on the test equal to the sum of the item MPLs) does not provide the best results. Third, a simple procedure, based on generalizability theory, for examining the sources of error in estimates of the passing score is discussed.
Sensitivity of Item Difficulties to Curricular Validity
Authors: Mehrens William A. , Phillips S. E.
Published: September 2005

The issue of the match of what is taught to what is tested recently has received increased emphasis. Many recent studies have examined the extent of the mismatch between a local set of objectives/instruction and the content of a nationally standardized test. Phillips and Mehrens (1985, 1987; see also Mehrens & Phillips, 1986) have conducted a series of studies that show that, within a district, different textbook series and informal curricula generally have no significant impact on test total, objective, or item scores. The present study explored further the curricular validity differences across textbooks and the relationship between item statistics and measures of curricular validity. In particular, this study sought to determine whether item p‐values appear to be related to measures of curricular validity based on three mathematics textbook series used at a given grade and in a previous grade. The results of this study indicated that the textbooks differed somewhat in content coverage when using a 180‐cell matrix classification. However, these differences were not great, especially when the textbooks in both grades 5 and 6 were considered. Further, all three series covered almost all of the 53 cells in the matrix covered by the Stanford Achievement Test, and the differences that did exist in textbook content coverage had no observable relationship to differences in item p‐values. In addition, the mean difficulty level of the Stanford Achievement Test items classified by cell were similar for students using the different textbooks, despite the differences among textbooks in location, presentation, and organization of content.
Item Discrimination: When More Is Worse
Authors: Masters Geofferey N.
Published: September 2005

High item discrimination can be a symptom o f a special kind of measurement disturbance introduced by an item that gives persons o f high ability a special advantage over and above their higher abilities. This type o f disturbance, which can be interpreted as a form o f item “bias,” can be encouraged by methods that routinely interpret highly discriminating items as the “best” items on a test and may be compounded by procedures that weight items by their discrimination. The type of measurement disturbance described and illustrated in this paper occurs when an item is sensitive to individual differences on a second, undesired dimension that is positively correlated with the variable intended to be measured. Possible secondary influences o f this type include opportunity to learn, opportunity to answer, and test wiseness
A Comparative Study of the Effects of Recency of Instruction on the Stability of IRT and Conventional Item Parameter Estimates
Authors: Cook Linda L. , Eignor Daniel R. , Taft Hessy L.
Published: September 2005

A potential concern for individuals interested in using item response theory (IRT) with achievement test data is that such tests have been specifically designed to measure content areas related to course curriculum and students taking the tests at different points in their coursework may not constitute samples from the same population. In this study, data were obtained from three administrations of two forms of a Biology achievement test. Data from the newer of the two forms were collected at a spring administration, made up of high school sophomores just completing the Biology course, and at a fall administration, made up mostly of seniors who completed their instruction in the course from 6–18 months prior to the test administration. Data from the older form, already on scale, were collected at only a fall administration, where the sample was comparable to the newer form fall sample. IRT and conventional item difficulty parameter estimates for the common items across the two forms were compared for each of the two form/sample combinations. In addition, conventional and IRT score equatings were performed between the new and old forms for each o f the form sample combinations. Widely disparate results were obtained between the equatings based on the two form/sample combinations. Conclusions are drawn about the use o f both classical test theory and IRT in situations such as that studied, and implications o f the results for achievement test validity are also discussed
The Effect of Various Factors on Standard Setting
Authors: Norcini John J. , Shea Judy A. , Kanya D. Theresa
Published: September 2005

This paper reports two studies of standard setting using Angoff's method. Results of the first study suggest that specialization within broad content areas does not affect an expert's estimates of the performance of the borderline group. This is reassuring because the knowledge base of many professions is so large that no individual can be considered an expert in all aspects of it. Results of the second study support the recommendation that performance data be provided during the standard‐setting process. They are frequently used by experts, but will not have an impact on the standard unless the distribution of item difficulties is skewed markedly. It also increases the correspondence between p‐values and estimates of borderline group performance, thereby reducing errors in pass/fail decisions. Overall, the results support recommendations often made in standard‐setting literature, but they need to be replicated with other groups of experts
The Projected Impact of the Correction for Guessing on Individual Scores
Authors: Albanese Mark A.
Published: September 2005

This article presents estimates of the effects of the use of formula scoring on an individual examinee's score. The results of this analysis suggest that under plausible assumptions, using test characteristics derived from several studies, some examinees would increase their scores by one half standard deviation or more if they were to answer items omitted under formula directions
Building a Unidimensional Test Using Multidimensional Items
Authors: Reckase Mark D. , Ackerman Terry A. , Carlson James E.
Published: September 2005

This paper demonstrates, both theoretically and empirically, using both simulated and real test data, that sets of items can be selected that meet the unidimensionality assumption of most item response theory models even though they require more than one ability for a correct response. Sets of items that measure the same composite of abilities as defined by multidimensional item response theory are shown to meet the unidimensionality assumption. A method for identifying such item sets is also presented
Equating Minimum‐Competency Tests: Comparisons of Methods
Authors: Hills John R. , Subhiyah Raja G. , Hirsch Thomas M.
Published: September 2005

The 1986 scores from Florida's Statewide Student Assessment Test, Part II (SSAT‐II), a minimum‐competency test required for high school graduation in Florida, were placed on the scale of the 1984 scores from that test using five different equating procedures. For the highest scoring 84 % of the students, four of the five methods yielded results within 1.5 raw‐score points of each other. They would be essentially equally satisfactory in this situation, in which the tests were made parallel item by item in difficulty and content and the groups of examinees were population cohorts separated by only 2 years. Also, the results from six different lengths of anchor items were compared. Anchors of 25, 20, 15, or 10 randomly selected items provided equatings as effective as 30 items using the concurrent IRT equating method, but an anchor of 5 randomly selected items did not
A Comparison of Six Methods for Combining Multiple IRT Item Parameter Estimates
Authors: McKinley Robert L.
Published: September 2005

Six procedures for combining sets of IRT item parameter estimates obtained from different samples were evaluated using real and simulated response data. In the simulated data analyses, true item and person parameters were used to generate response data for three different‐sized samples. Each sample was calibrated separately to obtain three sets of item parameter estimates for each item. The six procedures for combining multiple estimates were each applied, and the results were evaluated by comparing the true and estimated item characteristic curves. For the real data, the two best methods from the simulation data analyses were applied to three different‐sized samples and the resulting estimated item characteristic curves were compared to the curves obtained when the three samples were combined and calibrated simultaneously. The results support the use of covariance matrix‐weighted averaging and a procedure that involves sample‐size‐weighted averaging of estimated item characteristic curves at the center of the ability distribution
Differential Use of Item Information by Judges Using Angoff and Nedeisky Procedures
Authors: Smith Robert L. , Smith Jeffrey K.
Published: September 2005

Competency examinations in a variety of domains require setting a minimum standard of performance. This study examines the issue of whether judges using the two most popular methods for setting cut scores (Angoff and Nedelsky methods) use different sources of information when making their judgments. Thirty‐one judges were assigned randomly to the two methods to set cut scores for a high school graduation test in reading comprehension. These ratings were then related to characteristics of the items as well as to empirically obtained p values. Results indicate that judges using the Angoff method use a wider variety of information and yield estimates closer to the actual p values. The characteristics of items used in the study were effective predictors of judges’ ratings, but were far less effective in predicting p values
Item Pool Maintenance in the Presence of Item Parameter Drift
Authors: Bock R. Darrell , Murakl Eiji , Pfeiffenberger Will
Published: September 2005

Differential linear drift of item location parameters over a 10 ‐year period is demonstrated in data from the College Board Physics Achievement Test. The relative direction of drift is associated with the content of the items and reflects changing emphasis in the physics curricula of American secondary schools. No evidence of drift of discriminating power parameters was found. Statistical procedures for detecting, estimating, and accounting for item parameter drift in item pools for long‐term testing programs are proposed
The Generalizability of Content Validity Ratings
Authors: Crocker Linda , Llabre Maria , Miller M. David
Published: September 2005

The problem of assessing the content validity (or relevance) of standardized achievement tests is considered within the framework of generalizability theory. Four illustrative designs are described that may be used to assess test‐item fit to a curriculum. For each design, appropriate variance components are identified for making relative and absolute item (or test) selection decisions. Special consideration is given to use of these procedures for determining the number of raters and/or schools needed in a content‐validation decisionmaking study. Application of these procedures is illustrated using data from an international assessment of mathematics achievement
Effects of Improving the Reliability of the GPA on Prediction Generally and on Comparative Predictions for Gender and Race Particularly
Authors: Elliott Rogers , Strenta A. Christopher
Published: September 2005

The reliability of a method of adjusting grade point averages for differences in departmental grading standards was examined, as were the effects of such adjustments on the predictive validity of high school grades, SAT scores, and achievement test scores. The index of differential grading standards for all on‐time graduates of the Dartmouth College class of 1986 was quite reliable, and its use in adjusting grade averages increased predictive validity, reduced its erosion over years, reduced the apparent underprediction of women, and improved predictions for blacks. Differential group enrollment in courses in the science division seems to account for much of the effect of adjustment on grades. Improvement in the reliability of the criterial grade averages also was shown to have similar effects on gender and race prediction in another data set
School Effectiveness Indices Revisited: Cross‐Year Stability
Authors: Mandeville Garrett K.
Published: September 2005

School effectiveness indices (SEIs) based on residuals from regressing test performance onto prior test performance and a socioeconomic status (SES) measure were obtained for 2 consecutive years for 431 elementary schools. The resulting SEIs were found to be reasonably stable year to year, the correlations ranging from. 34 to .66, depending on grade level (1–4) and subject (reading and mathematics). To aid in the identification of the factors that affect the stability of school achievement, correlations of the SEIs across subjects and grade levels were obtained also. It was determined that SEIs reflecting the performance of students at the same grade level were relatively stable, whether the same or different students were involved. However, SEIs reflecting the performance of students at different grade levels were very unstable. This suggests that grade‐within‐school effects dominate whatever global school effects operate in elementary schools. Implications for effective schools research, the design of school recognition/reward programs, and research and measurement specialists in general are discussed
A Method of Estimating Rater Reliability
Authors: van den Bergh Huub , Eiting Mindert H.
Published: September 2005

A method for assessing rater reliability by means of a design of overlapping rater teams is presented. The products to be rated are split randomly into m disjoint subsamples, m equaling the number of raters. Each rater rates at least two subsamples according to a prefixed design. The covariances or correlations of the ratings can be analyzed with LISREL models, resulting in estimates of the rater reliabilities. Models in which the rater reliabilities are congeneric, tauequivalent, or parallel can be tested. We address problems concerning the identification and the degrees of freedom of the models and present two examples based on essay ratings.
Statistical Reasoning in Psychometric Models and Educational Measurement
Authors: Andrich David
Published: September 2005

This paper explores three points raised by Divgi (1986) in order to explicate the important distinction between deterministic and statistical reasoning in the application of statistical models to educational measurement. The three issues concern the relationship between data and esimation equations, the distinction between parameters and parameter estimates, and the power of tests of fit of responses across the ability continuum.
Does the Rasch Model Really Work for Multiple‐Choice Items? Take Another Look: A Response to Divgi
Authors: I‐leaning Grant
Published: September 2005

Divgi's (1986) study concludes, largely on the basis of a proposed test o f fit that is designed to be more powerful than Wright and Panchapakesan's (I 969) test, that the Rasch model is never appropriate for use with multiple‐choice type test items. This paper is an attempt to refute the conclusions of Divgi's study.
Cognitive and Developmental Effects on Item Performance in Intelligence and Achievement Tests for Young Children
Authors: Willson Victor L.
Published: September 2005

Performance on items in intelligence and achievement tests can be represented in terms of child development and information processes. Research is reviewed on item performance that supports developmental and information processing effects, particularly in children. Some suggestions for item development in intelligence and achievement tests are presented.
Characteristics of SAT Quantitative Items Showing Improvement After Coaching Among Black Students From Low‐Income Families: An Exploratory Study
Authors: Johnson Sylvia T. , Wallace Michael B.
Published: September 2005

The present exploratory investigation is an extension of a 1985 study that evaluated a Scholastic Aptitude Test preparation program involving black urban students. The present study was conducted to determine if there are identifiable characteristics of quantitative items associated with a susceptibility to coaching. Items showing p‐value improvements of. 10 after coaching were compared with items not showing such improvements; comparisons were made in terms of the content, type. format, level of cognitive requirement, and position of the items. Items showing improvement were found across content areas, formats, and type. Findings are discussed in relation to the usefulness and improvement of well‐designed supplemental instructional programs that have the potential to affect criterion performance in mathematics
A Longitudinal Hierarchical Linear Model for Estimating School Effects and Their Stability
Authors: Willms J. Douglas , Raudenbush Stephen W.
Published: September 2005

This paper presents a general longitudinal model for estimating school effects and their stability. Previous research on the stability of school performance over successive years has produced inconsistent findings. We argue that the findings have been inconsistent for at least two reasons: researchers have estimated different types of school effects, and they have not distinguished between instability due to true changes in school performance and instability due to measurement and sampling error. We describe two different types of school effects, each relevant to a different policy audience, and we present a longitudinal model that is capable of separating true changes in school effects from sampling and measurement error. The model also provides a means for estimating the effects of school policies and practices while controlling statistically for the effects of factors exogenous to the schooling system. This paper provides an example of the approach based on data describing two cohorts of students from one Education Authority in Scotland. It concludes with a discussion of the limitations of the model and implications for those collecting indicators of school performance for planning and evaluation purposes.
Using Restricted Latent Class Models to Map the Skill Structure of Achievement Items
Authors: Haertel Edward H.
Published: September 2005

This paper presents a new method for using certain restricted latent class models, referred to as binary skills models, to determine the skills required by a set o f test items. The method is applied to reading achievement data from a nationally representative sample o f fourth‐grade students and offers useful perspectives on test structure and examinee ability, distinct from those provided by other methods o f analysis. Models fitted to small, overlapping sets o f items are integrated into a common skill map, and the nature o f each skill is then inferred from the characteristics o f the items for which it is required. The reading comprehension items examined conform closely to a unidimensional scale with six discrete skill levels that range from an inability to comprehend or match isolated words in a reading passage to the abilities required to integrate passage content with general knowledge and to recognize the main ideas o f the most difficult passages on the test.
Does Guessing Really Help?
Authors: Angoff William H.
Published: September 2005

This study examines the claim that attempting, or guessing at, more items yields improved formula scores. Two samples of students who had taken a form of the SA T‐ Verbal consisting of three parallel half‐hour sections, were used to form the following scores on each of the three sections: the number of attempts, a guessing index, the formula score, and (indirectly) an approximation to an ability score. Correlations were obtained separately for the two samples between the attempts, and the guessing index, on one section, the formula score on a second section, and ability as measured by the third section. The partial correlations obtained hovered near zero, suggesting, contrary to conventional opinion, that, on average, attempting more items and guessing are not helpful in yielding higher formula scores, and that, therefore, formula scoring is not generally disadvantageous to the student who is less willing to guess and attempt an item that he or she is not sure of. On closer examination, however, it became clear that the advantages of guessing depend, at least in part, on the ability of the examinee. Although the relationship is generally quite weak, it is apparently the case that more able examinees do tend to profit somewhat from guessing, and would therefore be disadvantaged by their reluctance to guess. On the other hand, less able examinees may lower their scores i f they guess.
Comparison of Item Preequating and Random Groups Equating Using IRT and Equipercentile Methods
Authors: Kolen Michael J. , Harris Deborah J.
Published: September 2005

An item‐preequating design and a random groups design were used to equate forms of the American College Testing (ACT) Assessment Mathematics Test. Equipercentile and 3‐parameter logistic model item‐response theory (IRT) procedures were used for both designs. Both pretest methods produced inadequate equating results, and the IRT item preequating method resulted in more equating error than had no equating been conducted. Although neither of the item preequating methods performed well, the results from the equipercentile preequating method were more consistent with those from the random groups method than were the results from the IRT item pretest method. Item context and position effects were likely responsible, at least in part, for the inadequate results for item preequating. Such effects need to be either controlled or modeled, and the design further researched before the item preequating design can be recommended for operational use.
Effect of Eliciting Verbal Reports of Thinking on Critical Thinking Test Performance
Authors: Norris Stephen P.
Published: September 2005

Verbal reports of examinees' thinking on multiple‐choice critical thinking test items can provide useful validation data only if the verbal reporting does not change the course of examinees' thinking and performance. Using a completely randomized factorial design, 343 senior high school students were divided into five groups. In four of the groups, different procedures were used to elicit students' thinking as they worked through Part A of a critical thinking test of observation appraisal (Norris & King, 1983). In the control group, students took the same test in paper‐and‐pencil format. There were no significant differences in test performance among the five groups nor in the quality of thinking among the four groups from whom verbal reports of thinking were elicited. These results are evidence that verbal reports of thinking can meet one of the necessary conditions of useful validation data—namely, that collecting the data does not alter examinees' thinking and performance. Some analyses found significant interviewer main effects and sex‐by‐interviewer and elicitation‐level‐by‐interviewer‐by‐sex‐by‐grade interaction effects. Analysis of these interactions suggested that the role of the interviewer might limit the generality of the technique.
Equivalent Pass/Fail Decisions
Authors: Norcini John J.
Published: September 2005

In competency testing, it is sometimes difficult to properly equate scores of different forms of a test and thereby assure equivalent cutting scores. Under such circumstances, it is possible to set standards separately for each test form and then scale the judgments of the standard setters to achieve equivalent pass/fail decisions. Data from standard setters and examinees for a medical certifying examination were reanalyzed. Cutting score equivalents were derived by applying a linear procedure to the standard‐setting results. These were compared against criteria along with the cutting score equivalents derived from typical examination equating procedures. Results indicated that the cutting score equivalents produced by the experts were closer to the criteria than standards derived from examinee performance, especially when the number of examinees used in equating was small. The root mean square error estimate was about 1 item on a 189‐item test.
Influence of Type of Judge, Normative Information, and Discussion on Standards Recommended for the National Teacher Examinations
Authors: Busch John Christian , Jaeger Richard M.
Published: September 2005

There are few empirical investigations of the consequences of using widely recommended data collection procedures in conjunction with a specific standardsetting method such as the Angoff (1971) procedure. Such recommendations include the use of several types of judges, the provision of normative information on examinees' test performance, and the opportunity to discuss and reconsider initial recommendations in an iterative standard‐setting procedure. This study of 236 expert judges investigated the effects of using these recommended procedures on (a) average recommended test standards, (b) the variability of recommended test standards, and (c) the reliability of recommended standards for seven subtests of the National Teacher Examinations Communication Skills and General Knowledge Tests. Small, but sometimes statistically significant, changes in mean recommended test standards were observed when judges were allowed to reconsider their initial recommendations following review of normative information and discussion. Means for public school judges changed more than did those for college or university judges. In addition, there was a significant reduction in the within‐group variability of standards recommended for several subtests. Methods for estimating the reliability of recommended test standards proposed by Kane and Wilson (1984) were applied, and their hypothesis of positive covariation between empirical item difficulties and mean recommended standards was confirmed. The data collection procedures examined in this study resulted in substantial increases in the reliability of recommended test standards.
Method of Measurement and Gender Differences in Scholastic Achievement
Authors: Bolger Niall , Kellaghan Thomas
Published: September 2005

Gender differences in scholastic achievement as a function of method of measurement were examined by comparing the performance of 15‐year‐old boys (N = 739) and girls (N = 758) in Irish schools on multiple‐choice tests and free‐response tests (requiring short written answers) of mathematics, Irish, and English achievement. Males performed significantly better than females on multiple‐choice tests compared to their performance on free‐response examinations. An expectation that the gender difference would be larger for the languages and smaller for mathematics because of the superior verbal skills attributed to females was not fulfilled.
Adjusting the Cumulative GPA Using Item Response Theory
Authors: Young John W.
Published: September 2005

In college admissions, the predictive validity of preadmissions measures such as standardized test scores and high school grades is of wide interest. These measures are most often validated against the criterion of the first‐year grade point average (GPA). However, neither the first‐year GPA nor the four‐year cumulative GPA is an adequate indicator of academic performance through four years of college. In this study, Item Response Theory (IRT) is used to develop a more reliable measure of performance, called an IRT‐based GPA, which is used to estimate the validity of traditional preadmissions information. The data are preadmissions information and course grades for the Class of 1986 at Stanford University (N = 1564). Principal factor analysis is used as a precursor to determine the dimensionality of the course data and to partition courses into approximately unidimensional subsets, each of which is scaled independently. Results show a substantial increase in predictability when the IRT‐based GPA is used instead of the usual GPA.
Conditional Standard Error of Measurement in Prediction
Authors: Woodruff David
Published: September 2005

Previous methods for estimating the conditional standard error of measurement (CSEM) at specific score or ability levels are critically discussed, and a brief summary of prior empirical results is given. A new method is developed that avoids theoretical problems inherent in some prior methods, is easy to implement, and estimates not only a quantity analogous to the CSEM at each score but also the conditional standard error of prediction (CSEP) at each score and the conditional true score standard deviation (CTSSD) at each score, The new method differs from previous methods in that previous methods have concentrated on attempting to estimate error variance conditional on a fixed value of true score, whereas the new method considers the variance of observed scores conditional on a fixed value of an observed parallel measurement and decomposes these conditional observed score variances into true and error parts. The new method and several older methods are applied to a variety of tests, and representative results are graphically displayed. The CSEM‐Iike estimates produced by the new method are called conditional standard error of measurement in prediction (CSEMP) estimates and are similar to those produced by older methods, but the CSEP estimates produced by the new method offer an alternative interpretation of the accuracy of a test at different scores. Finally, evidence is presented that shows that previous methods can produce dissimilar results and that the shape of the score distribution may influence the way in which the CSEM varies across the score scale.
Integrating Cognitive and Psychometric Models to Measure Document Literacy
Authors: Sheehan Kathleen , Mislevy Robert J.
Published: September 2005

The Survey of Young Adult Literacy conducted in 1985 by the National Assessment of Educational Progress included 63 items that elicited skills in acquiring and using information from written documents. These items were analyzed using two different models: (1) a qualitative cognitive model, which characterized items in terms of the processing tasks they required, and (2) an item response theory (IRT) model, which characterized items difficulties and respondents' proficiencies simply by tendencies toward correct response. This paper demonstrates how a generalization of Fischer and Seheibleehner's Linear Logistic Test Model can be used to integrate information from the cognitive analysis into the IRT analysis, providing a foundation for subsequent item construction, test development, and diagnosis of individuals skill deficiencies.
Multidimensionality and IRT‐Based Item lnvariance Indexes: The Effect of Between‐Group Variation in Trait Correlation
Authors: Oshima Takako C. , Miller M. David
Published: September 2005

Using a bidimensional two‐parameter logistic model, the authors generated data for two groups on a 40‐item test. The item parameters were the same for the two groups, but the correlation between the two traits varied between groups. The difference in the trait correlation was directly related to the number of items judged not to be invariant using traditional unidimensional IRT‐based unsigned item invariance indexes; the higher trait correlation leads to higher discrimination parameter estimates when a unidimensional IRT model is fit to the multidimensional data. In the most extreme case, when rθ1 θ2= Ofor one group and r θ1 θ2= 1.0 for the other group, 33 out of 40 items were identified as not invariant. When using signed indexes, the effect was much smaller. The authors, therefore, suggest a cautious use of IRT‐based item invariance indexes when data are potentially multidimensional and groups may vary in the strength of the correlations among traits.
A Note on the Bias of UCON Item Parameter Estimation in the Rasch Model
Authors: de Gruijter Dato N. M.
Published: September 2005

Divgi (1986) demonstrated that the bias of UCON item parameter estimates is not removed by the factor (n − 1)/n. Andrich (1989) argued in this journal that the demonstration was faulty. In this note a complete proof of Divgfs conclusion is presented.
The Use of Loglinear Models for Assessing Differential Item Functioning Across Manifest and Latent Examinee Groups
Authors: Kelderman Henk , Macready George B.
Published: September 2005

Loglinear latent class models are used to detect differential item functioning (DIF). These models are formulated in such a manner that the attribute to be assessed may be continuous, as in a Rasch model, or categorical, as in Latent Class Mastery models. Further, an item may exhibit DIF with respect to a manifest grouping variable, a latent grouping variable, or both. Likelihood‐ratio tests for assessing the presence of various types of DIF are described, and these methods are illustrated through the analysis of a “real world” data set.
An Investigation of Classification Consistency Indexes Estimated Under Alternative Strong True Score Models
Authors: Hanson Bradley A. , Brennan Robert L.
Published: September 2005

Using several data sets, the authors examine the relative performance of the beta binomial model and two other more general strong true score models in estimating several indexes of classification consistency. It is shown that the beta binomial model can provide inadequate fits to raw score distributions compared to more general models. This lack of fit is reflected in differences in decision consistency indexes computed using the beta binomial model and the other models. It is recommended that the adequacy of a model in fitting the data be assessed before the model is used to estimate decision consistency indexes. When the beta binomial model does not fit the data, the more general models discussed here may provide an adequate fit and, in such cases, would be more appropriate for computing decision consistency indexes.
Detecting Differential Item Functioning Using Logistic Regression Procedures
Authors: Swaminathan Hariharan , Rogers H. Jane
Published: September 2005

A logistic regression model for characterizing differential item functioning (DIF) between two groups is presented. A distinction is drawn between uniform and nonuniform DIF in terms of the parameters of the model. A statistic for testing the hypothesis of no DIF is developed. Through simulation studies, it is shown that the logistic regression procedure is more powerful than the Mantel‐Haenszel procedure for detecting nonuniform DIF and as powerful in detecting uniform DIF.
Equivalence of Free‐Response and Multiple‐Choice Items
Authors: Bennett Randy Elliot , Rock Donald A. , Wang Minhwei
Published: September 2005

This study examined the relationship of multiple‐choice and free‐response items contained on the College Board's Advanced Placement Computer Science (APCS) examination. Confirmatory factor analysis was used to test the fit of a two‐factor model where each item format marked its own factor. Results showed a single‐factor solution to provide the most parsimonious fit in each of two random‐half samples. This finding might be accounted for by several mechanisms, including overlap in the specific processes assessed by the multiple‐choice and free‐response items and the limited opportunity for skill differentiation afforded by the year‐long APCS course.
Traditional Dimensionality Versus Essential Dimensionality
Authors: Nandakumar Ratna
Published: September 2005

This article addresses testing the hypothesis of one versus more than one dominant (essential) dimension in the possible presence of minor dimensions. The method used is Stout's statistical test of essential unidimensionality, which is based on the theory of essential unidimensionality. Differences between the traditional definition of dimensionality provided by item response theory, which counts all dimensions present, and essential dimensionality, which counts only dominant dimensions, are discussed. As Monte Carlo studies demonstrate, Stout's test of essential unidimensionality tends to indicate essential unidimensionality in the presence of one dominant dimension and one or more minor dimensions that have a relatively small influence on item scores. As the influence of the minor dimensions increases, Stout's test is more likely to reject the hypothesis of essential unidimensionality. To assist in interpreting these studies, a rough index of the deviation from essential unidimensionality is proposed.
Differential Testlet Functioning: Definitions and Detection
Authors: Wainer Howard , Sireci Stephen G. , Thissen David
Published: September 2005

It is sometimes sensible to think of the fundamental unit of test construction as being larger than an individual item. This unit, dubbed the testlet, must pass muster in the same way that items do. One criterion of a good item is the absence of DIF–the item must function in the same way in all important subpopulations of examinees. In this article, we define what we mean by testlet DIF and provide a statistical methodology to detect it. This methodology parallels the IRT‐based likelihood ratio procedures explored previously by Thissen, Steinberg, and Wainer (1988, in press). We illustrate this methodology with analyses of data from a testlet‐based experimental version of the Scholastic Aptitude Test (SAT).
Comparison of Methods for Combining the Minimum Passing Levels for Individual Items into a Passing Score for a Test
Authors: Plake Barbara S. , Kane Michael T.
Published: September 2005

The purpose of this study was to compare several methods for determining a passing score on an examination from the individual raters' estimates of minimal pass levels for the items. The methods investigated differ in the weighting that the estimates for each item receive in the aggregation process. An IRT‐based simulation method was used to model a variety of error components of minimum pass levels. The results indicate little difference in estimated passing scores across the three methods. Less error was present when the ability level of the minimally competent candidates matched the expected difficulty level of the test. No meaningful improvement in passing score estimation was achieved for a 50‐item test as opposed to a 25‐item test; however, the RMSE values for estimates with 10 raters were smaller than those for 5 raters. The results suggest that the simplest method for aggregating minimum pass levels across the items in a test–adding them up–is the preferred method.
Smoothing Methods for Estimating Test Score Distributions
Authors: Kolen Michael J.
Published: September 2005

Frequency distributions of test scores may appear irregular and, as estimates of a population distribution, contain a substantial amount of sampling error. Techniques for smoothing score distributions are available that have the capacity to improve estimation. In this article, estimation/smoothing methods that are flexible enough to fit a wide variety of test score distributions are reviewed. The methods are a kernel method, a strong true–score model–based method, and a method that uses polynomial log–linear models. The use of these methods is then reviewed, and applications of the methods are presented that include describing and comparing test score distributions, estimating norms, and estimating equipercentile equivalents in test score equating. Suggestions for further research are also provided.
The Performance of the Mantel‐Haenszel Procedure Across Samples and Matching Criteria
Authors: Ryan Katherine E.
Published: September 2005

This study examined the reliability of the Mantel‐Haenszel indexes across different samples of test takers as well as across sample sizes and investigated whether these indexes are robust to item context effects. Mathematics data from the Second International Mathematics Study (SIMS; 1985) for U.S. eighth‐grade students were analyzed. The results suggest that the MH D‐DIF is robust to item context effects. However, larger sample sizes than those used in this investigation (N = 141‐167 for the focal group) may be necessary to obtain stable estimates from the Mantel‐Haenszel procedure.