bluuebunny/crossref_metadata_2025 huggingface dataset:
RELIABILITY OF CRITERION‐REFERENCED TESTS: A DECISION‐THEORETIC FORMULATION
Authors: SWAMINATHAN H. , HAMBLETON RONALD K. , ALGINA JAMES
Published: September 2005

It has been suggested that the primary purpose for criterion‐referenced testing in objective‐based instructional programs is to classify examinees into mastery states or categories on the objectives included in the test. We have proposed that the reliability of the criterion‐referenced test scores be defined in terms of the consistency of the decision‐making process across repeated administrations of the test. Specifically, reliability is defined as a measure of agreement over and above that which can be expected by chance between the decisions made about examinee mastery states in repeated test administrations for each objective measured by the criterion‐referenced test.
DETERMINING THE CHARACTERISTICS OF THE IDEAL PROFESSOR: AN ALTERNATIVE APPROACH
Authors: SUBKOVIAK MICHAEL J. , LEVIN JOEL R.
Published: September 2005

In a recently conducted study it was found that an effective college teacher could be characterized chiefly in terms of “research,”“teaching,” and “service to the university.” The present experiment corroborated these findings using a relatively unknown approach to data collection, analysis, and interpretation. A free‐response method of data collection, in conjunction with nonmetric multidimensional scaling, produced results highly similar to those of the previous study. In addition, the solution permitted a straight‐forward means of assessing the performance of individual faculty members.
THE INFLUENCE OF SOCIOECONOMIC STATUS ON THE PREDICTABILITY OF COLLEGE PERFORMANCE
Authors: WRIGHT ROBERT J. , BEAN ANDREW G.
Published: September 2005

Although numerous studies have examined the validity of scholastic aptitude measures as predictors of college performance for black and white students, few studies have investigated the validity of these predictors for different socioeconomic levels within a racially homogeneous population. To study the influence of socioeconomic status (SES) on the predictability of college performance, a sample of 1,631 white freshmen attending a large urban university was divided into homogeneous subgroups on each of three measures: (a) family income, (b) father's occupation, and (c) mother's education. Verbal and Quantitative Scholastic Aptitude Test scores and high school class rank were used to predict freshman grade‐point average within each subgroup. For all three socioeconomic measures, lower cross‐validated multiple correlations were associated with lower levels of SES. Possible explanations for these results are discussed and implications for research are presented.
“GAME” VERSUS “TEST” INSTRUCTIONS FOR THE WISC1
Authors: BRIDGEMAN BRENT , STRANG HAROLD R. , BUTTRAM JOAN
Published: September 2005

“Game” or “test” instructions on either verbal or nonverbal WISC scales were given to 160 third‐ and sixth‐grade children. Ss in one condition were told they were going to take several tests, while Ss in the other condition were told they were going to play several games. Significant differences in performance due to task definition were found only on verbal tasks at the sixth‐grade level with test instructions yielding superior performance. Results at the third‐grade level failed to replicate previous results which suggested game instructions produce superior performance on nonverbal tasks.
EVALUATING COMPENSATORY EDUCATION: OVER WHAT PERIOD OF TIME SHOULD ACHIEVEMENT BE MEASURED?
Authors: DAVID JANE L. , PELAVIN SOL H.
Published: September 2005

An assumption fundamental to compensatory education is that greater achievement can change the academic future of disadvantaged students, which may in turn enhance their “life chances.” Therefore, one of the goals of compensatory education is to increase the achievement of disadvantaged students. To change students' futures, this increase in achievement should be evident subsequent to participation in a compensatory‐education program. At a minimum, an increase in achievement should persist over the summer following a school‐year program. Evaluations of compensatory education in general, however, and of Title I of the Elementary and Secondary Education Act (ESEA) in particular, have not included measures of sustained achievement. Instead, judgments of program success have been based on students' achievement during the school year: that is, on a spring posttest score adjusted in some way for the preceding fall pretest score.
A STUDY OF HYPOTHESES BASIC TO THE USE OF RIGHTS AND FORMULA SCORES
Authors: ANGOFF WILLIAM H. , SCHRADER WILLIAM B.
Published: September 2005

The hypothesis that some students, when tested under formula directions, omit items about which they have useful partial knowledge implies that such directions are not as fair as rights directions, especially to those students who are less inclined to guess. This hypothesis may be called the differential effects hypothesis. An alternative hypothesis states that examinees would perform no better than chance expectation on items that they would omit under formula directions but would answer under rights directions. This may be called the invariance hypothesis. Experimental data on this question were obtained by conducting special test administrations of College Board SAT‐verbal and Chemistry tests and by including experimental tests in a Graduate Management Admission Test administration. The data provide a basis for evaluating the two hypotheses and for assessing the effects of directions on the reliability and parallelism of scores for sophisticated examinees taking professionally developed tests. Results support the invariance hypothesis rather than the differential effects hypothesis.
CONSTRUCTING AN ITEM BANK USING PARTIAL CREDIT SCORING
Authors: MASTERS GEOFFEREY N.
Published: September 2005

A method for banking test items scored in several ordered response categories is described. Each item is seen as an ordered sequence of steps, and test forms are equated using the estimated difficulties of the steps in their shared items. Procedures for analyzing the internal consistency of individual links and for analyzing the coherence of an entire linking structure are described. The methodology is used to link six forms of a mathematics problem solving test.
EMPIRICAL COMPARISON OF SELECTED ITEM BIAS DETECTION PROCEDURES WITH BIAS MANIPULATION
Authors: SUBKOVIAK MICHAEL J. , MACK JOANNE S. , IRONSON GAIL H. , CRAIG ROBERT D.
Published: September 2005

Biased test items were intentionally imbedded within a set of test items, and the resulting instrument was administered to large samples of blacks and whites. Three popular item bias detection procedures were then applied to the data: (1) the three‐parameter item characteristic curve procedure, (2) the chi‐square method, and (3) the transformed item difficulty approach. The three‐parameter item characteristic curve procedure proved most effective at detecting the intentionally biased test items; and the chi‐square method was viewed as the best alternative. The transformed item difficulty approach has certain limitations yet represents a practical alternative if sample size, lack of computer facilities, or the like preclude the use of the other two procedures.
OBTAINING MAXIMUM LIKELIHOOD TRAIT ESTIMATES FROM NUMBER‐CORRECT SCORES FOR THE THREE‐PARAMETER LOGISTIC MODEL
Authors: YEN WENDY M.
Published: September 2005

A procedure is presented for obtaining maximum likelihood trait estimates from number‐correct (NC) scores for the three‐parameter logistic model. The procedure produces an NC score to trait estimate conversion table, which can be used when the hand scoring of tests is desired or when item response pattern (IP) scoring is unacceptable for other (e.g., political) reasons. Simulated data are produced for four 20‐item and four 40‐item tests of varying difficulties. These data indicate that the NC scoring procedure produces trait estimates that are tau‐equivalent to the IP trait estimates (i.e., they are expected to have the same mean for all groups of examinees), but the NC trait estimates have higher standard errors of measurement than IP trait estimates. Data for six real achievement tests verify that the NC trait estimates are quite similar to the IP trait estimates but have higher empirical standard errors than IP trait estimates, particularly for low‐scoring examinees. Analyses in the estimated true score metric confirm the conclusions made in the trait metric.
A METHOD FOR REACHING A COMPROMISE BETWEEN ABSOLUTE AND RELATIVE STANDARDS IN EXAMINATIONS
Authors: BEUK CEES H.
Published: September 2005

Examining committees often need to reach a compromise between absolute and relative standards. Unfortunately, the way in which the compromise is achieved is usually unclear. This paper proposes a systematic method for reaching a compromise. In this method, the estimated passing score (level of minimum knowledge) is assumed to be related to the expected pass rate (percentage of successful candidates) through a simple linear function. The examination results define a function relating the percentage of candidates who would be successful given a specified passing score to the passing score. The intersection of both functions gives the required compromise.
SELF DESCRIPTION QUESTIONNAIRE III: THE CONSTRUCT VALIDITY OF MULTIDIMENSIONAL SELF‐CONCEPT RATINGS BY LATE ADOLESCENTS
Authors: MARSH HERBERT W. , O'NEILL ROSALIE
Published: September 2005

This research examines the validity of self‐concept interpretations of scores from a new instrument for use with university‐aged respondents. The Self Description Questionnaire III (SDQ III) was designed to measure 13 factors of self‐concept, and these dimensions were identified with conventional and confirmatory factor analyses. In two different studies, the reliabilities of the 13 factors were high (median alpha = 0.89) and correlations among the factors were low (median r = 0.09). Correlations among a wide variety of validity criteria and the multiple dimensions of self‐concept measured by the SDQ III formed a logical and theoretically consistent pattern of relationships. Academic achievement measures in language and mathematics were substantially correlated with self‐concepts in the same areas but not with other self‐concept factors. Ratings by significant others for all 13 SDQ HI scales were substantially correlated with the measures of corresponding self‐concepts, but were not substantially correlated with the measures of noncorresponding self‐concepts. These findings offer strong support for the construct validity of both self‐concept and interpretations based upon the SDQ III.
COMPUTERIZED DIAGNOSTIC TESTING
Authors: McARTHUR DAVID L. , CHOPPIN BRUCE H.
Published: September 2005

Efforts to use computers for diagnostic testing in education require appropriate psychometric and technological strategies and, equally important, good theoretical foundations as to what constitutes a diagnosis and how it is to be discerned in an examinee's performance. Although at present this combination is infrequent in education, for some years it has been in evidence in computer‐managed diagnosis in medicine. For comparable success in the field of education, it is argued that artificial intelligence algorithms may provide a suitable avenue for developing diagnostic testing further.
TECHNOLOGY AND TESTING: STATE OF THE ART AND TRENDS FOR THE FUTURE
Authors: BAKER FRANK B.
Published: September 2005

Until recently, computers and computer related technology have been used in testing primarily to provide large scale programs with cost effective scoring and reporting. However, the “microcomputer revolution” is beginning to have a significant impact on how technology is being employed in testing. The present paper reviews a number of existing applications where microcomputers have been successfully employed in the context of testing. Then, future trends in both technology and instruction are examined for possible impact upon testing.
RUMORS REGARDING THE DEATH OF THE EQUIPERCENTILE ASSUMPTION MAY HAVE BEEN GREATLY EXAGGERATED
Authors: TALLMADGE G. KASTEN
Published: September 2005

Powers, Slaughter, and Helmick (1983) recently analyzed selection, pretest, and posttest scores collected from large numbers of students in two cohorts. These analyses led them to conclude that the equipercentile assumption underlying norm‐referenced evaluation methodology is “inappropriate.” Re‐examination of the data suggests that there is strong support for the validity of the equipercentile assumption in the selection and pretest scores they present. The observed “gains” from pre‐ to posttests are better attributed to stakeholder bias, posttests that match the curriculum content too closely, or a combination of these two factors than to inappropriateness of the equipercentile assumption. Annual testing where the posttest from one year also serves as the pretest for the next is suggested as a promising solution to both of the cited threats to the internal validity of norm‐referenced evaluations.
ESTIMATING THE RELIABILITY, VALIDITY, AND INVALIDITY OF ESSAY RATINGS
Authors: BLOK H.
Published: September 2005

In an essay rating study multiple ratings may be obtained by having different raters judge essays or by having the same rater(s) repeat the judging of essays. An important question in the analysis of essay ratings is whether multiple ratings, however obtained, may be assumed to represent the same true scores. When different raters judge the same essays only once, it is impossible to answer this question. In this study 16 raters judged 105 essays on two occasions; hence, it was possible to test assumptions about true scores within the framework of linear structural equation models. It emerged that the ratings of a given rater on the two occasions represented the same true scores. However, the ratings of different raters did not represent the same true scores. The estimated intercorrelations of the true scores of different raters ranged from .415 to .910. Parameters of the best fitting model were used to compute coefficients of reliability, validity, and invalidity. The implications of these coefficients are discussed.
CHOICE OF VARIABLES AND GENDER DIFFERENTIATED PREDICTION WITHIN SELECTED ACADEMIC PROGRAMS
Authors: GAMACHE LEANN M. , NOVICK MELVIN R.
Published: September 2005

The existence of differential prediction of 2‐year grade point average is reported for gender groups within programs of study at a large state university. Johnson‐Neyman analyses indicate regions on the predictor score scales where differential prediction has practical impact within each program. A significant portion of students is generally affected. It is shown that careful selection of a subset of variables reduces differential prediction while maintaining predictive potency.
VALIDITY OF APPROXIMATION TECHNIQUES FOR DETECTING ITEM BIAS
Authors: SHEPARD LORRIE A. , CAMILLI GREGORY , WILLIAMS DAVID M.
Published: September 2005

The purpose of this research was to recommend an item bias procedure when the number of minority examinees is too small to use preferred three‐parameter IRT methods. The chi‐square, Angoff delta‐plot, andpseudo‐IRT indices were compared with both real and simulated data. For the real test data a criterion of known bias had been established by cross‐validated IRT‐3 results. The findings from the Math Test and the simulated test were consistent. The pseudo‐IRT approach was best (measured by both correlations and percent agreement) in delecting criterion bias. The chi‐square was close in accuracy to the pseudo‐IRT index. The Angoff delta‐plot method was found to be inadequate on both heuristic and empirical grounds. In extreme cases it even identified items as biased against whites that were simulated to be biased against blacks. However, a modified Angoff index, where p‐value differences were regressed on item point biserials (and the residualized values used as the index), was nearly as good as the chi‐square in identifying known bias. A final caution was offered regarding the use of item bias techniques. The statistical flags should never be used mechanically to discard items; rather they should be used to inspect items for possible differences in meaning.
EFFECTS OF COACHING ON GRE APTITUDE TEST SCORES
Authors: POWERS DONALD E.
Published: September 2005

Test preparation activities were determined for a large representative sample of Graduate Record Examination (GRE) Aptitude Test takers. About 3% of these examinees had attended formal coaching programs for one or more sections of the test.

After adjusting for differences in the background characteristics of coached and uncoached students, effects on test scores were related to the length and the type of programs offered. The effects on GRE verbal ability scores were not significantly related to the amount of coaching examinees received, and quantitative coaching effects increased slightly but not significantly with additional coaching. Effects on analytical ability scores, on the other hand, were related significantly to the length of coaching programs, through improved performance on two analytical item types, which have since been deleted from the test.

Overall, the data suggest that, when compared with the two highly susceptible item types that have been removed from the GRE Aptitude Test, the test item types in the current version of the test (now called the GRE General Test) appear to show relatively little susceptibility to formal coaching experiences of the kinds considered here.
A COMMENT ON McCAULEY AND COLBERG'S CONCEPTION OF CROSS‐CULTURAL TRANSPORTABILITY OF TESTS
Authors: VAN DE VIJVER FONS J. R. , POORTINGA YPE H.
Published: September 2005

Recently McCauley and Colberg described a theory of transportability, an approach to devising test items for cross‐cultural comparative research that are free from unwanted cultural effects. They also provided data to demonstrate the feasibility of their approach. In this comment it is argued that the transportability notion resembles earlier cross‐cultural work and does not add new insights into the vexed problem of cross‐cultural comparison. It is also stated that McCauley and Colberg's statistical checks do not preclude the possibility of bias.
THE CONCURRENT VALIDITY OF STANDARDIZED ACHIEVEMENT TESTS BY CONTENT AREA USING TEACHERS' RATINGS AS CRITERIA
Authors: HOPKINS KENNETH D. , GEORGE CATHERINE A. , WILLIAMS DAVID D.
Published: September 2005

To assess the concurrent validity of standardized achievement tests using teachers' ratings (and rankings) of pupils' academic achievement as criteria, 42 teachers evaluated each of their students (n = 1,032) in each of five major curricular areas prior to the administration of a battery of standardized achievement tests. The teachers were directed to rate each student's proficiency disregarding attendance, attitude, deportment, and so on. Within‐class correlation coefficients were computed to eliminate rater leniency bias. The standardized achievement tests were found to have substantial concurrent validity in reading, math, language arts, science, and social studies. The normalized teacher ranks yielded significantly higher validity coefficients than did the ratings, although the magnitude of the difference was small. The concurrent validity coefficients for language arts, reading, and math were significantly higher than those in science and social studies.
THE IMPORTANCE OF CONTENT REPRESENTATION FOR COMMON‐ITEM EQUATING WITH NONRANDOM GROUPS
Authors: KLEIN LAWRENCE W. , JARJOURA DAVID
Published: September 2005

The equating accuracy of content‐representative anchors versus nonrepresen‐tative but substantially longer anchors is compared. Content representation was defined as a match between anchors and total test of the percentage of items in each of several content areas. Through a chain of equatings it was found that content representation in anchors was critical for the testing program studied. The results are explained in terms of differences in mean profiles (by content area) of the nonrandom groups used for equating.
THE IMPACT OF ITEM PHRASING ON THE VALIDITY OF ATTITUDE SCALES FOR ELEMENTARY SCHOOL CHILDREN
Authors: BENSON JERI , HOCEVAR DENNIS
Published: September 2005

The purpose of the study was to examine the effect of item phrasing on the validity of a Likert‐type attitude scale. Three content similar scales were composed of 15 items, either all positive, all negative, or a mixture of positive and negative items. Five hundred twenty‐two students in grades 4–6 responded to one of the three forms. Results from the all positive and negative forms indicated that item means, variances, and factor structures differed significantly. Inspection of item means suggested that it was difficult for the students to indicate agreement by disagreeing with a negative statement. Analyses of the mixed phrasing form indicated factors based upon item phrasing, not item content. Taken together, the results suggest that the technique of balancing item phrasing when used with elementary students appears to affect adversely the validity of attitude measurement.
THE EFFECTS OF VIOLATIONS OF UNIDIMENSIONALITY ON THE ESTIMATION OF ITEM AND ABILITY PARAMETERS AND ON ITEM RESPONSE THEORY EQUATING OF THE GRE VERBAL SCALE
Authors: DORANS NEIL J. , KINGSTON NEAL M.
Published: September 2005

One of the major assumptions of item response theory (IRT)models is that performance on a set of items is unidimensional, that is, the probability of successful performance by examinees on a set of items can be modeled by a mathematical model that has only one ability parameter. In practice, this strong assumption is likely to be violated. An important pragmatic question to consider is: What are the consequences of these violations? In this research, evidence is provided of violations of unidimensionality on the verbal scale of the GRE Aptitude Test, and the impact of these violations on IRT equating is examined. Previous factor analytic research on the GRE Aptitude Test suggested that two verbal dimensions, discrete verbal (analogies, antonyms, and sentence completions)and reading comprehension, existed. Consequently, the present research involved two separate calibrations (homogeneous) of discrete verbal items and reading comprehension items as well as a single calibration (heterogeneous) of all verbal item types. Thus, each verbal item was calibrated twice and each examinee obtained three ability estimates: reading comprehension, discrete verbal, and all verbal. The comparability of ability estimates based on homogeneous calibrations (reading comprehension or discrete verbal) to each other and to the all‐verbal ability estimates was examined. The effects of homogeneity of item calibration pool on estimates of item discrimination were also examined. Then the comparability of IRT equatings based on homogeneous and heterogeneous calibrations was assessed. The effects of calibration homogeneity on ability parameter estimates and discrimination parameter estimates are consistent with the existence of two highly correlated verbal dimensions. IRT equating results indicate that although violations of unidimensionality may have an impact on equating, the effect may not be substantial.
DETECTING EXPERIMENTALLY INDUCED ITEM BIAS USING THE ITERATIVE LOGIT METHOD
Authors: KOK FRANK G. , MELLENBERGH GIDEON J. , VAN DER FLIER HENK
Published: September 2005

A test for mental arithmetic was constructed, consisting of items written in Dutch (the subjects’ native language), Spanish, and Roman numerals. A group of 286 subjects received some information on Spanish numerals. The group was randomly split into a Spanish Group and a Roman Group. The Spanish Group received further instruction on Spanish numerals, while the Roman Group got instruction on Roman numerals. Checks on the experimental manipulations showed that the Spanish Group had better knowledge of Spanish numerals than the Roman Group, whereas the Roman Group had better knowledge of Roman numerals. From the total test two subtests were constructed: a 30‐item Dutch/Spanish subtest (15 items in Dutch and 15 in Spanish), and a 25‐item Dutch/Roman subtest (15 items in Dutch and 10 in Roman). The Dutch items were unbiased between the Spanish and Roman groups, whereas the Spanish items of the Dutch/Spanish subtest were biased against the Roman Group, and the Roman items of the Dutch/Roman subtest were biased against the Spanish Group. The iterative logit method was applied to the two subtests. The method showed very good results in detecting biased items.
ESSAY TEST SCORING: INTERACTION OF RELEVANT VARIABLES
Authors: CHASE CLINTON I.
Published: September 2005

In studies of essay tests, a single independent variable, such as penmanship, is often observed and conclusions are made about the relevance of that variable in scoring tests. But in this study it is hypothesized that the readers of an essay respond to a variable in terms of its context with other variables, that is, readers may be more tolerant of poor penmanship for males than females, especially if expectations for the writer are involved. In this investigation sex, race, reader expectation, and quality of handwriting were crossed to study their interaction effects. A single student essay was contrived. Each of 80 readers was given the essay plus a student's record card, which identified the student as a high or low achiever (expectancy), black or white, male or female. The essay was written in either good or poor handwriting. A reader read only one essay under one combination of conditions. The results showed complex interactions of expectations, writing, and sex within race.
A PROBLEM IN CALCULATING GROUP SCORES ON NORM‐REFERENCED TESTS
Authors: BAGLIN ROGER F.
Published: September 2005

Norm‐referenced standardized achievement tests are designed, and commonly used, for obtaining group scores. Various methods are used to calculate and express group scores in terms of common derived scores, such as percent ile ranks. Publishers' scaled scores are ordinarily used in these procedures, with the result that the group scores can possess anomalous characteristics. The group scores can vary widely, depending on not only the measure of central tendency but also the type of derived score employed. A reason for this situation is hypothesized to be the use of inappropriate statistical procedures to develop publishers' scaled scores. Practitioners need to be aware of this problem and to document their procedures when calculating and reporting group scores. Test publishers are urged to avoid the use of scaling procedures that are seen as responsible for this problem.
FIVE PITFALLS ENCOUNTERED WHILE TRYING TO COMPARE STATES ON THEIR SAT SCORES
Authors: WAINER HOWARD
Published: September 2005

Recent research (Page & Feifs, 1985; Powell & Steelman, 1984; Steelman & Powell, 1985) attempts to draw inferences about the relative standing of the states on the basis of mean SAT scores. These papers point out that this cannot be done without statistically adjusting for various differences among the states. In this paper I identify five serious errors that, when made, call into question the validity of such inferences. In addition, I describe some plausible ways to avoid the errors.
ESTIMATING THE RELIABILITY OF MULTIPLE TRUE‐FALSE TESTS
Authors: FRISBIE DAVID A. , DRUVA CYNTHIA A.
Published: September 2005

This study was designed to examine the level of dependence within multiple true‐false (MTF) test item clusters by computing sets of item intercorrelations with data from a test composed of both MTF and multiple choice (MC) items. It was posited that internal analysis reliability estimates for MTF tests would be spurious due to elevated MTF within‐cluster intercorrelations. Results showed that, on the average, MTF within‐cluster dependence was no greater than that found between MTF items from different clusters, between MC items, or between MC and MTF items. But item for item, there was greater dependence between items within the same cluster than between items of different clusters.
THE EFFECTS OF THE DELETION OF MISFITTING PERSONS ON VERTICAL EQUATING VIA THE RASCH MODEL
Authors: PHILLIPS S. E.
Published: September 2005

The purpose of the study was to compare Rasch model equatings of multilevel achievement test data before and after the deletion of misfitting persons. The Rasch equatings were also compared with an equating obtained using the equipercentile method. No basis could be found in the results for choosing between the two Rasch equatings. The deletion of misfitting persons produced minor improvements in Rasch model fit to the data. Both Rasch equatings produced results that differed from the results of the equipercentile equating. The Rasch data also indicated that the misfitting persons deleted in the second Rasch equating tended to be from the lower portion of the achievement distribution, suggesting that they may have been guessing.
A COMPARISON OF EXAMINEE SAMPLING AND MULTIPLE MATRIX SAMPLING IN TEST DEVELOPMENT
Authors: GARG RASHMI , BOSS MARVIN W. , CARLSON JAMES E.
Published: September 2005

For the purpose of obtaining data to use in test development, multiple matrix sampling (MMS) plans were compared to examinee sampling plans. Data were simulated for examinees, sampled from a population with a normal distribution of ability, responding to items selected from an item universe. Three item universes were considered: one that would produce a normal distribution of test scores, one a moderately platykurtic distribution, and one a very platykurtic distribution. When comparing sampling plans, total numbers of observations were held constant. No differences were found among plans in estimating item difficulty. Examinee sampling produced better estimates of item discrimination, test reliability, and test validity. As total number of observations increased, estimates improved considerably, especially for those MMS plans with larger subtest sizes. Larger numbers of observations were needed for tests designed to produce a normal distribution of test scores. With an adequate number of observations, MMS is seen as an alternative to examinee sampling in test development.
USING DEMOGRAPHIC SUBGROUP AND DUMMY VARIABLE EQUATIONS TO PREDICT COLLEGE FRESHMAN GRADE AVERAGE
Authors: SAWYER RICHARD
Published: September 2005

The purpose of this study was to determine whether adjustments for the differential prediction observed among sex, racial/ethnic, or age subgroups in one freshman class at a college could be used to improve prediction accuracy for these subgroups in future freshman classes. For older students, dummy variable and separate subgroup prediction equations were found, on cross‐validation, to be more accurate than the total group equations. For sex subgroups, dummy variable and separate subgroup equations were only moderately effective in improving prediction accuracy. For racial/ethnic subgroups, they were more often than not less accurate, on cross‐validation, than total group equations. Among all three kinds of demographic subgroupings, shifts over time in colleges' mean grade averages were found to be a more important source of prediction bias than differential prediction.
TIME ALLOCATION AND PATTERNS OF ITEM RESPONSE
Authors: MILLER M. DAVID
Published: September 2005

A measure of student patterns of item response is usually interpreted as a measure of carelessness or guessing. However, when aggregated to the class level, it was shown that the index identifies classes that have a poor match between test content and instructional coverage. In particular, Sato's (1980) caution index identified classes whose pattern of time allocation across content areas was atypical. In addition, it was found that the mean caution index for a class can best be interpreted knowing the within‐class standard deviation of the index.
DIAGNOSING STUDENTS' ERRORS FROM THEIR RESPONSE SELECTIONS IN LANGUAGE ARTS
Authors: WEBB NOREEN M. , HERMAN JOAN L. , CABELLO BEVERLY
Published: September 2005

This set of studies examined the consistency of student response patterns on a test of language arts, as a first step toward designing a computerized adaptive test to diagnose errors. A diagnostic domain‐referenced language arts test was designed so that the choice of response would immediately point to a specific misconception in pronoun usage. This direct correspondence between error and diagnosis was designed to facilitate classroom instruction and remediation. Analysis of students' response choices on matched items and analysis of students' rationales for selecting their responses showed that student behavior was not always consistent and could be used to diagnose some errors but not others.
MATHEMATICS AND SCIENCE TEST SCORES AS RELATED TO COURSES TAKEN IN HIGH SCHOOL AND OTHER FACTORS
Authors: JONES LYLE V. , DAVENPORT ERNEST C. , BRYSON ALOHA , BEKHUIS TANJA , ZWlCK REBECCA
Published: September 2005

For the 1980 sophomore cohort from High School and Beyond (National Opinion Research Center, 1983a, 1983b), a strong relation was found between senior‐year mathematics achievement test score and the number of high school mathematics courses taken at the level of Algebra I and above. Unexpectedly, the relation was slightly stronger for courses the students reported taking than for courses as recorded on the student's high school transcript. The strength of relation was reduced but remained substantial after a covariance adjustment for student socioeconomic status (SES), sophomore‐year verbal achievement test score, and sophomore‐year mathematics test score. In contrast, the relation between senior‐year science test score and the number of high school science courses taken was far weaker than for mathematics, both before and after covariance adjustment.
FACTOR ANALYSIS OF THE K–ABC FOR GROUPS OF ANGLO AND MEXICAN AMERICAN CHILDREN
Authors: VALENCIA RICHARD R. , RANKIN RICHARD J.
Published: September 2005

Factor analyses of the Kaufman Assessment Battery for Children (K–ABC) were performed on separate groups of Anglo (n = 100) and Mexican American (n = 100) fifth‐grade children to determine the comparability of underlying structures and to examine the existence of possible bias in construct validity of the K–ABC for each group. The subjects were selected from two contiguous school districts and stratified by grade, socioeconomic status (SES), and language status. A principal factor analysis produced two factors in the Mental Processing area and three factors in the total battery (Mental Processing plus Achievement areas) for the two groups. Examination of the coefficients of congruence indicated factorial similarity across four of five factors. Furthermore, the factors that emerged for the Anglo and Mexican American groups corresponded to the scales comprising the K–ABC. The results of the investigation provide some evidence for the construct validity of the K–ABC for the two ethnic groups. Further test bias research on the K–ABC is recommended.
THE CORRECTION FOR GUESSING: A FURTHER ANALYSIS OF ANGOFF AND SCHRADER
Authors: ALBANESE MARK A.
Published: September 2005

This study reexamines results reported by Angoff and Schrader (1981, 1984) regarding formula directions and rights directions for standardized tests. In that study, it was concluded that the two scoring directions were essentially equivalent. In this study, methodological concerns are discussed and additional data analyses undertaken. Among the methodological concerns discussed are the potential problems in using volunteers for the College Board phase of the study and the likelihood of treatment contamination in the Graduate Management Admissions Test (GMAT) phase. Estimates of success rates of the rights directions group on items omitted and not reached by the formula group, were beyond chance levels by 0.6 % to 13.0 % for the College Board tests and at or below chance levels for the GMAT. Alternative interpretations of the data and suggestions for additional research are proposed.
TEACHER EDUCATION AND TEACHER‐PERCEIVED NEEDS IN EDUCATIONAL MEASUREMENT AND EVALUATION
Authors: GULLICKSON ARLEN R.
Published: September 2005

Professors and teachers were compared relative to their perspectives on preservice educational measurement courses. Twenty‐four professors from different colleges in seven states and 360 teachers from elementary and secondary schools in one midwestern state responded via mailed questionnaire. Professors reported the emphasis given to each of eight topics in preservice educational measurement courses, and teachers reported the emphasis they believed should be given to each topic. In five of the eight content areas, the relative emphases given by professors differed from that recommended by teachers. Major differences emerged in nontest evaluation, statistical analysis, and formative and summative evaluation. Implications of these results are discussed.
THE SUBSET SELECTION TECHNIQUE FOR MULTIPLE‐CHOICE TESTS: AN EMPIRICAL INQUIRY
Authors: JARADAT DERAR , SAWAGED SARI
Published: September 2005

The impact of the Subset Selection Technique (SST) for administering and scoring multiple‐choice items on certain properties of a test was compared with that of the two other commonly used methods, the Number Right (NR) and the Correction for Guessing Formula (CFG). Under SST, examinees are instructed to select any number of response alternatives, the objective being to include the correct answer in the chosen set. The effects of each scoring method on the psychometric properties of a test and on the performance of examinees with different achievement levels and/or risk‐taking propensities were investigated. Results indicated that SST outperformed the other two methods, producing not only higher reliability and validity coefficients for the test, but doing so without favoring high risk takers. The superiority of SST may be attributed to two interrelated factors: the efficiency of the technique in controlling for guessing and the encouragement provided examinees to use their partial knowledge in responding.
The Hierarchical Structure of Self‐Concept and the Application of Hierarchical Confirmatory Factor Analysis
Authors: Marsh Herbert W.
Published: September 2005

This investigation (a) tested the ability of an a priori hierarchical structure of self‐concept derived from the Shavelson model to explain responses to the Self Description Questionnaire (SDQ) III, and (b) demonstrated the application and problems with the use of hierarchical confirmatory factor analysis (HCFA). A first‐order factor analysis clearly identified all 13 facets of self‐concept that the SDQ 111 is designed to measure. A series of HCFA models supported the separation of the 13 SDQ III facets of self‐concept into academic and nonacademic components, and the academic facets into math/academic and verbal/academic components. However, support for separation of nonacademic facets into the physical, social, and moral second‐order factors was not strong. Third‐order HCFA models resulted in a clearly defined hierarchical general self‐concept that was substantially related to Esteem and to physical, social, and emotional components of self‐concept, but not to the academic and moral components. However, the hierarchy was so weak that first‐order facets could not be accurately inferred from the higher order facets.
An Experimental, Exploratory Study of Causes of Bias in Test Items
Authors: Seheuneman Janiee Dowd
Published: September 2005

This study evaluated 16 hypotheses, subsumed under 7 more general hypotheses, concerning possible sources of bias in test items for black and white examinees on the Graduate Record Examination General Test (GRE). Items were developed in pairs that were varied according to a particular hypothesis, with each item from a pair administered in different forms of an experimental portion of the GRE. Data were analyzed using log linear methods. Ten of the 16 hypotheses showed interactions between group membership and the item version indicating a differential effect of the item manipulation on the performance of black and white examinees. The complexity of some of the interactions found, however, suggested that uncontrolled factors were also differentially affecting performance.
A Domain‐Referenced Approach to Diagnostic Testing Using Generalizability Theory
Authors: Webb Noreen M. , Herman Joan L. , Cabello Beverly
Published: September 2005

This paper describes a four‐step approach to constructing diagnostic test profiles that provide precise but practical information on students' instructional needs. The approach is based on the specification and analysis of a domain and uses generalizability theory to determine which skills within the domain need to be assessed to diagnose gaps in students' skills and to estimate score profiles. A 64‐item test of pronoun use was constructed to represent 32 categories of usage defined by different combinations of five factors in the domain. Generalizability analyses were conducted to determine the optimal number of categories to be included in students' profiles and the number of items needed for each category, and to produce univariate and multivariate estimates of students' universe scores. Multivariate profiles of universe scores were the most accurate and differed substantially from observed score and univariate universe score profiles.
Attitudes, Behaviors, and Reasons for Changing Responses Following Answer‐Changing Instruction
Authors: McMorris Robert F. , DeMers Lawrence P. , Schwarz Shirley P.
Published: September 2005

Contrary to the adage warning against changing test answers, mean gain from changing has been an invariant research finding. Consistency of this gain was tested for students instructed about the research results, and composition of the gain was analyzed by examining the students' reasons for changing. Students in six graduate measurement classes instructed about the answer‐changing literature responded to three exams and a questionnaire. Mean gain remained positive and consistent with gain for previously studied uninstructed groups; amount of change was also stable. “Rethinking the item and conceptualizing a better answer” was the most frequent reason given for changing. “Rereading the item and better understanding the question” was the second most cited reason, followed by “rereading/rethinking” combined, and “making a clerical error.” For each frequently used reason, wrong‐to‐right (WR) changes were in the majority. Implications for research and practice are discussed.
Effects of “On‐Line” Test Feedback on the Seriousness of Subsequent Errors
Authors: Birenbaum Menucha , Tatsuoka Kikumi K.
Published: September 2005

The present study examined the effect of three modes of feedback on the seriousness of error types committed on a post‐test. The measure of seriousness of error types, or wrong rules, indicates to what extent the wrong rules deviate from the right rule. An on‐line, free‐response, six‐item pre‐test covering addition of signed numbers was administered to 263 eighth graders. Upon completion of the pre‐test, the students were randomly assigned to receive one of the following kinds of feedback to responses to a second six‐item test: (a) information about whether or not the response was correct, (b) the correct answer, or (c) the correct rule for solving the problem. The effect of the feedback mode on the seriousness of errors committed on a third six‐item test, the post‐test, was found to be differential and dependent upon the seriousness of errors committed on the pre‐test.
The Stability of School Effectiveness Indices Across Grade Levels and Subject Areas
Authors: Mandeville Garrett K. , Anderson Lorin W.
Published: September 2005

School effectiveness indices (SEIs) based on regressing achievement test performance onto earlier test performance and a socioeconomic status (SES) measure were obtained for eight subject–grade level combinations for a large sample of elementary schools. School means based on longitudinally matched student scores comprised the data set used in the analysis. The resulting SEIs were found to be somewhat unstable across subject areas (reading and mathematics) and very unstable across grade levels (1 through 4). Grade‐to‐grade correlations of the SEIs measuring mathematics performance, although small, tended to be statistically significant, whereas those measuring reading performance were generally nonsignificant. Thus, school effects may be more readily discernible in some subject areas than in others. Implications for research on effective schools and for the design of school recognition programs based on student test performance are discussed.
Who Benefits Most From Preparing for a “Coachable” Admissions Test?
Authors: Powers Donald E.
Published: September 2005

A previous study of the initial, preoperational version of the Graduate Record Examinations (GRE) analytical ability measure (Powers & Swinton, 1984) revealed practically and statistically significant effects of test familiarization on analytical test scores. (Two susceptible item types were subsequently removed from the test.) Data from this study were reanalyzed for evidence of differential effects for subgroups of examinees classified by age, ethnicity, degree aspiration, English language dominance, and performance on other sections of the GRE General Test. The results suggested little, if any, difference among subgroups of examinees with respect to their response to the particular kind of test preparation considered in the study. Within the limits of the data, no particular subgroup appeared to benefit significantly more or significantly less than any other subgroup.
An Application of Item Response Theory in the Comparison of Four Conventional Item Discrimination Indices for Criterion‐Referenced Tests
Authors: Shannon Gregory A. , Cliver Barbara A.
Published: September 2005

Several recent papers have argued for the usefulness of item response theory (IRT) methods of assessing item discrimination power for criterion‐referenced tests (CRTs). Conventional methods continue to be used more widely, however, for reasons that include some practical constraints associated with the use of IRT methods. To provide users with information that may help them to decide on which conventional indices to employ in evaluating CRT items, Spearman rank‐order correlations were computed between IRT‐derived item information functions (llFs) and four conventional discrimination indices: the phi‐coefficient, the B‐index, phi/phi max, and the agreement statistic. The rank‐order correlations between the phi‐coefficient and the llFs were very high, with a median of .96. The remaining conventional indices, with the exception of phi‐over‐phi‐max, also correlated well with the IIF. Theoretical explanations for these relationships are offered.
Language and Cultural Characteristics That Explain Differential Item Functioning for Hispanic Examinees on the Scholastic Aptitude Test
Authors: Sehmitt Alieia P.
Published: September 2005

The standardization methodology was used to help identify item characteristics that might explain differential item functioning among Hispanics on the Scholastic Aptitude Test. Results indicated that true cognates or words with a common root in English and Spanish and content of special interest for Hispanics seemed to help Hispanics performance. Limited occurrence of false cognates (words that appear to be cognates but have different meanings in both languages) and of homographs (words that are spelled alike but have different meanings in English) restricted their evaluation. Nevertheless, examination of items with false cognates or homographs gave some evidence indicating that their occurrence might make items unexpectedly more difficult for Hispanic examinees