bluuebunny/crossref_metadata_2025 huggingface dataset:
MODELS OF EVALUATION AND THEIR RELATION TO STUDENT CHARACTERISTICS
Authors: BIGGS J. B. , BRAUN P. H.
Published: September 2005

The union and disjunction models for combining individual test marks to yield final grade distributions were outlined. It was expected that these models would differentially favor students according to their characteristic methods of learning and studying. The models were examined empirically in two educational psychology classes; five performance assessments were available, as well as eleven dimensions of study behavior and academic attitudes. While there was a substantial correlation between the distributions derived from the two models, they were found significantly to favor different student characteristics. The union model, relative to the disjunction model, favored students who were dependent, who interrelated different aspects of their course work, who scheduled their work and who rote‐learned material. These differences were interpreted in terms of the educational assumptions underlying the two models. Some implications of the study for grading practice were suggested.
MODERATING THE PREDICTION OF GRADES IN FRESHMAN ENGINEERING1
Authors: WOOD DONALD A. , LANGEVIN MICHAEL J.
Published: September 2005

This study concerned the identification of an optimal moderator within a test battery for enhancing prediction of first semester engineering grades for 522 college freshmen. Moderated multiple regression disclosed high school rank (HSR) as the best moderator of SAT‐M, the best predictor, in both a validation and hold‐out group. SAT‐M overall validity of .39 was increased to .53 and an overall R of .47 was raised to .61 when only those highest in HSR were considered.
DISCRIMINATION AMONG INDIVIDUALS V. DISCRIMINATION AMONG GROUPS
Authors: LEWY ARIEH
Published: September 2005

Achievement tests are used for discrimination both among individuals and among classes. Item selection procedures which are recommended for constructing tests for individual differentiation may not be adequate for tests for discrimination among classes. It is suggested that the item intraclass correlation should be used as an index for item selection in tests designed for the measurement of class performance. An example is presented which shows that using the intraclass correlation for item selection yields different results than using item‐test correlation coefficients.
DISCRIMINANT ANALYSIS OF STUDY STRATEGIES CONNECTED WITH COLLEGE GRADE SUCCESS IN DIFFERENT MAJOR FIELDS
Authors: GOLDMAN ROY D. , WARREN REBECCA
Published: September 2005

A study techniques questionnaire was administered to 538 students enrolled in upper division college classes. Multivariate analysis of variance was employed to compare the centroids of questionnaire responses of students in four major fields who had above and below average grades. There were significant differences in group centroids for the comparison between major fields and between grade levels. There was no significant interaction. The major differences between students with above and below average grades appeared to be reflected in “clerical diligence” and cognitive “activity.” Successful students were not only more diligent in their study habits, but also more likely to transform actively scholastic information. For the comparison of major field groups, two discriminant functions were significant. One function seemed to reflect mathematical‐logical thinking while the other reflected “applied‐subjective” thinking. The major field groups formed a science‐nonscience continuum on the first function but not on the second.
AGREEMENT AMONG NCME MEMBERS ON SELECTED ISSUES IN EDUCATIONAL MEASUREMENT
Authors: HARRIS WILBUR S.
Published: September 2005

This study surveyed 145 NCME members to determine the extent of their agreement on 40 controversial issues in educational measurement. Two types of agreement were obtained for each issue. First, the degree of agreement with the issues as presented. Second, the degree of agreement among the group of respondents. The percent of respondents choosing each of three responses to each issue, and a mean, variance, and “consensus” score for each issue were computed. The mean scores indicated the degree to which the respondents agreed with the issues. They varied from + .91 to ‐ .71, when + 1 would indicate maximum agreement with an issue, and ‐ 1 maximum disagreement. The “consensus” scores varied from .90 to .15, and had a mean of .40 where 1 would indicate maximum agreement among the group, and 0 maximum disagreement.
A NOTE ON THE POWER OF STATISTICAL TESTS IN THE JOURNAL OF EDUCATIONAL MEASUREMENT
Authors: BREWER JAMES K. , OWEN PATRICIA W.
Published: September 2005

The power of statistical tests recently appearing in the JEM was determined using the power calculation guidelines proposed by Cohen (1969). All the articles containing tests of significance were surveyed. The results indicated that power was generally below .50 for small effect sizes and above .50 for medium and large effect sizes. A suggestion for reporting statistical results to include power of the tests was made.
A FACET‐FACTORIAL APPROACH TO THE CONSTRUCTION OF RATING SCALES TO MEASURE COMPLEX BEHAVIORS
Authors: ABELES HAROLD F.
Published: September 2005

The purpose of this study was to examine a technique for the development of performance rating scales to measure achievement in courses whose objectives require complex behaviors not easily measurable with paper and pencil achievement tests. A facet‐factorial approach to rating scale construction was employed (i.e. the behavior was conceptualized as multidimensional and items for the scales were selected by employing factor analytical techniques) to construct scales to measure clarinet music performance. The three major results of the study were: 1) a thirty‐item rating scale based on a six factor structure of clarinet music performance; 2) high inter‐judge reliability estimates for both the total score (above .90) and the scale scores (above .60); and, 3) criterion‐related validity coefficients greater than .80. Results of the investigation suggest that the facet‐factorial approach can be an effective technique for the construction of rating scales to measure complex behavior such as music performance.
NARRATIVE FORMAT TESTING REPORTS AND TRADITIONAL TESTING REPORTS: A COMPARATIVE STUDY
Authors: MATHEWS WALTER M.
Published: September 2005

This article reports a comparative study of teacher acceptance of two kinds of testing reports that were generated for Form A of the Iowa Tests of Basic Skills at the fourth‐grade level. Locally produced reports similar to the traditional ones provided by the test publisher were evaluated against experimental reports that were computer generated in a narrative format. On a random basis, 52 teachers from 16 schools received either the traditional or experimental testing reports for their class, and were asked to evaluate the reports on six scales: clear, useful, meaningful, valuable, sufficient, and accurate, for each of three purposes: information for the teacher on individual students, class‐summary information, and information for use at a parent‐teacher meeting. On 15 of the 18 comparisons between the traditional and experimental reports, the experimental reports were rated significantly higher.
EFFECTS OF EMPIRICAL OPTION WEIGHTING ON RELIABILITY AND VALIDITY OF AN ACADEMIC APTITUDE TEST1
Authors: REILLY RICHARD R. , JACKSON REX
Published: September 2005

Item options of shortened forms of the GRE Verbal and Quantitative tests were empirically weighted by two variants of a method originally attributed to Guttman (1941). When compared with formula scores, it was found that tests scored with the empirical weights were more reliable but less valid when correlated with undergraduate GPA. A factor analysis revealed large increases in variance accounted for by the first factor. It was suggested that the weighting procedures used tended to capitalize on omitting behavior which, although a highly reliable tendency, may be invalid.
HIDDEN OPPORTUNITIES IN THE PREDICTION OF COLLEGE GRADES FOR DIFFERENT SUBGROUPS
Authors: GOLDMAN ROY D.
Published: September 2005

The purpose of this paper was to examine the problems and opportunities of academic prediction for different ethnic groups. Several recent studies of academic prediction for blacks and whites were reviewed in regard to: 1) the situation in which the data were obtained; 2) the prediction technique employed, and 3) the data distribution likely to give rise to the obtained prediction indices. It was suggested that a total‐group regression equation which “benefits” a minority group by overpredicting mean grade may actually be very disadvantageous if accompanied by a large error of estimate. The damage can be produced by precluding selection of the most qualified minority group members and thus lowering the groups’ performance. Differential process theory was proposed as a potential source of explanations for differential prediction. It was proposed that alternative strategic approaches to scholastic tasks might alter the covariance of predictor tests with grades. Finally, it was suggested that, under certain circumstances, the patterns of standardized regression weights in the prediction of grades, might suggest group difference in problem‐solving strategies.
A NOTE ON ALLOCATING ITEMS TO SUBTESTS IN MULTIPLE MATRIX SAMPLING AND APPROXIMATING STANDARD ERRORS OF ESTIMATE WITH THE JACKKNIFE
Authors: SHOEMAKER DAVID M.
Published: September 2005

Investigated empirically through post mortem item‐examinee sampling were the relative merits of two alternative procedures for allocating items to subtests in multiple matrix sampling and the feasibility of using the jackknife in approximating standard errors of estimate. The results indicate clearly that a partially balanced incomplete block design is preferable to random sampling in allocating items to subtests. The jackknife was found to better approximate standard errors of estimate in the latter item allocation procedure than in the former.
VALIDITY OF THE DISCRIMINATION INDEX AS A MEASURE OF ITEM QUALITY1
Authors: PYRCZAK FRED
Published: September 2005

Numerous writers have suggested that the discrimination index may be helpful in identifying faulty test items. The purpose of this study was to investigate systematically the validity of the index for this purpose. To attain this objective, two forms of an arithmetic‐reasoning test were written. In each form, the items were designed to vary in quality with respect to nine item‐writing principles, and on the basis of the responses of 364 examinees, a discrimination index was computed for each item. Next, the items were rated independently for quality by three judges who used a check list of the nine item‐writing principles. The average of their ratings for each item was used as the criterion for determining the validity of the indices. The results indicate that the discrimination index is a moderately valid measure of item quality. The implications of this finding are discussed.
BIAS IN SELECTION
Authors: COLE NANCY S.
Published: September 2005

Possible bias in selection procedures used for employment and college admissions is of crucial social and educational importance. However, there are many different definitions of what constitutes bias with each definition based on different values and with different implications for how selection should be accomplished. A number of these definitions of bias and their implications are examined, and a new conditional probability model of fairness based on equal opportunity for potentially successful applicants is presented. This conditional probability model is proposed as an intuitively appealing and socially desirable model for use in many selection situations in employment and college admissions.
DIFFERENTIAL WEIGHTING BY JUDGED DEGREE1OF CORRECTNESS
Authors: PATNAIK DURGADAS , TRAUB ROSS E.
Published: September 2005

Two conventional scores and a weighted score on a group test of general intelligence were compared for reliability and predictive validity. One conventional score consisted of the number of correct answers an examinee gave in responding to 69 multiple‐choice questions; the other was the formula score obtained by subtracting from the number of correct answers a fraction of the number of wrong answers. A weighted score was obtained by assigning weights to all the response alternatives of all the questions and adding the weights associated with the responses, both correct and incorrect, made by the examinee. The weights were derived from degree‐of‐correctness judgments of the set of response alternatives to each question. Reliability was estimated using a split‐half procedure; predictive validity was estimated from the correlation between test scores and mean school achievement. Both conventional scores were found to be significantly less reliable but significantly more valid than the weighted scores. (The formula scores were neither significantly less reliable nor significantly more valid than number‐correct scores.)
THE EFFECT OF RACE OF EXAMINER ON THE MENTAL TEST SCORES OF WHITE AND BLACK PUPILS1
Authors: JENSEN ARTHUR R.
Published: September 2005

An entire elementary school system with 60% white and 40% black pupils was given several abiity tests group‐administered by 12 white and eight black examiners (Es). The tests measured verbal and nonverbal IQ, perceptual‐motor cognitive development, “speed and persistence” under neutral and motivating instructions, listening‐attention, and short‐term rote memory for numbers. With the exception of the “speed and persistence” test, on which white Es yielded significantly and consistently higher mean scores than black Es for both white and black pupils across grades one to six, the results for the various cognitive ability tests showed that the race of the E did not produce large or consistent effects in the testing of white and black pupils.
ADAPTATION OF AN INTELLIGENCE TEST FROM ENGLISH TO FRENCH1
Authors: BHUSHAN VIDYA
Published: September 2005

This study aimed to discover the usefulness of adapting the Otis Quick‐Scoring Mental Ability Tests for the French culture. Beta Test (Forms EM and FM) were translated into French and administered to 635 students of grades four to eight. To correct invalid items or distractors, a revision followed the statistical analysis. Revised tests were then administered to 1,686 students of grades five to nine. From this and related studies, it is concluded that an intelligence test developed for a particular culture may be invalid when simply translated for another.
VARIANCE STABILIZING TRANSFORMATION OF THE STEPPED‐UP RELIABILITY COEFFICIENT1
Authors: LORD FREDERIC M.
Published: September 2005

The stepped‐up reliability coefficient does not have the same standard error as an ordinary correlation coefficient. Fisher's z‐transformation should not be applied to it. Appropriate procedures are suggested.
EFFECTS OF DIFFERENT SAMPLES ON ITEM AND TEST CHARACTERISTICS OF CRITERION‐REFERENCED TESTS
Authors: HALADYNA THOMAS MICHAEL
Published: September 2005

Although many have rejected classical test construction and analysis procedures for criterion‐referenced tests, the present study was concerned with the possibility that classical procedures are both applicable and appropriate when samples of both mastery and nonmastery examinees are employed. A rationale for using these samples was presented, and empirical evidence was gathered which supported the practice of combining samples to increase the variance of test scores and thereby permit the proper estimate of reliability and item validities.
A STUDY OF THE VALIDITY OF SELF‐REPORTED MEASURES OF ACADEMIC GROWTH
Authors: POHLMANN JOHN T. , BEGGS DONALD L.
Published: September 2005

This study was undertaken to examine the relationship between self‐reported and pre‐post measures of academic growth. Self‐reported and pre‐post measures were obtained in three areas, simple cognitive, complex cognitive, and attitudinal. The subjects were 162 graduate students enrolled in six different graduate courses. Partial correlations relating self‐reported measures of growth to posttest performance on measures of achievement (simple and complex cognitive)and attitude, controlling for pretest performance indicated that self‐reported measures of academic growth were primarily related to growth in attitudes toward the subject matter of a course.
THE NATURE OF OBJECTIVITY WITH THE RASCH MODEL
Authors: WHITELY SUSAN E. , DAWIS RENÁ V.
Published: September 2005

Although it has been claimed that the Rasch model leads to a higher degree of objectivity in measurement than has been previously possible, this model has had little impact on test development. Population‐invariant item and ability calibrations, together with the statistical equivalency of any two item subsets, are supposedly possible if the item pool has been calibrated by the Rasch model. Initial research has been encouraging, but the implications of underlying assumptions and operational computations in the Rasch model for trait theory have not been clear from previous work. The current paper presents an analysis of the conditions under which the claims of objectivity will be substantiated, with special emphasis on the nature of equivalent forms. It is concluded that the real advantages of the Rasch model will not be apparent until the technology of trait measurement becomes more sophisticated.
SCRAMBLING CONTENT IN ACHIEVEMENT TESTING: AN APPLICATION OF MULTIPLE MATRIX SAMPLING IN EXPERIMENTAL DESIGN
Authors: SIROTNIK KEN , WELLINGTON ROGER
Published: September 2005

This study was designed to research the question of scrambling item content in the construction of achievement tests, so that very general implications could be drawn for both examinee and item populations. To achieve this generality, the methodology of multiple matrix sampling was combined with a simple two group experimental design: a random group of 8th graders responded to mathematics, science, social studies, reading, and language arts achievement items organized in a scrambled (random) test format, while another random group responded to the same items organized in a fixed (segregated by subject matter) test format. The results indicated that scrambling cognitive test items has minimal or no effect on mean examinee test performance or on any of the other parameters included in the analysis.
QUICK ESTIMATES OF THE RELATIVE EFFICIENCY OF TWO TESTS AS A FUNCTION OF ABILITY LEVEL
Authors: LORD FREDERIC M.
Published: September 2005

When comparing two tests that measure the same trait, an overall comparison is not enough. Separate comparisons should be made at different levels of the trait. A simple, practical, approximate formula is given for doing this. The adequacy of the approximation is illustrated using data comparing seven nationally known sixth‐grade reading tests.
A DEFENSE OF “RATIONAL” PERSONNEL SELECTION, AND TWO NEW METHODS
Authors: DARLINGTON RICHARD B.
Published: September 2005

This paper presents two new techniques for use in personnel selection. One is a simple graphical technique which non‐technical personnel can use to help in deciding how many minority applicants to admit to an institution. The other is a large‐scale computer technique which an institution can use to increase personnel diversity on many variables (including culture) simultaneously. Both techniques appear to have major advantages over the particular technique this author advocated previously (1971). However, the new techniques are based on the same general view the author has expressed earlier (1971, 1973): that personnel selection questions concerning culture must be handled rationally—that is, by human judgment—rather than by mechanical formulas. Early sections of the paper summarize and defend that view.
SELECTION BIAS: MULTIPLE MEANINGS
Authors: LINN ROBERT L.
Published: September 2005

The common approach to studies of predictive bias is analyzed within the context of a conceptual model in which predictors and criterion measures are viewed as fallible indicators of idealized qualifications. Selection on the basis of idealized qualifications depends on acceptance of a meritocratic principle, a principle which is itself subject to debate. Even with acceptance of this principle, however, it is shown that there will generally be a wide range of uncertainty as the result of the fallible nature of the indicators. Only when boundary conditions defined by Birnbaum (1979) are exceeded will the results be unambiguous. An example of such a case is presented. It is argued, however, that such instances are probably somewhat rare.
DETECTION OF A SKILL DICHOTOMY USING STANDARDIZED ACHIEVEMENT TEST ITEMS
Authors: HAERTEL EDWARD
Published: September 2005

Multiple‐choice reading comprehension items from a conventional, norm‐referenced reading comprehension test are successfully analyzed using a simple latent class model. A classification rule for assigning respondents to “mastery” or “nonmastery” states is presented which simplifies the scoring procedure of Macready and Dayton (1977). A procedure is also derived for estimating the “true,” or “disattenuated,” latent cross‐classification of masters versus nonmasters for two tests, and illustrated using two sets of items from the same content domain. Results support the use of latent class, state mastery models with more heterogeneous item pools than has been advocated by previous authors.
A COMPARISON OF THREE METHODS FOR ESTABLISHING MINIMUM STANDARDS ON THE NATIONAL TEACHER EXAMINATIONS
Authors: CROSS LAWRENCE H. , IMPARA JAMES C. , FRARY ROBERT B. , JAEGER RICHARD M.
Published: September 2005

Minimum standards were established for the National Teacher Examinations (NTE) area examinations in mathematics and in elementary education by independent panels of teacher educators who had been instructed in the use of either the Angoff, Nedelsky, or Jaeger procedures. Of these three procedures, only the Jaeger method requires that normative data be provided to the judges when evaluating the items. However, it was of interest to study the effect such information would have upon the standards obtained using the other two methods. Therefore, the design incorporated three sequential review sessions with the level of normative information different for each. A three‐factor ANOVA revealed significant main effects for methods and sessions but not for subject area. None of the interactions was significant. The anticipated failure rates, the psychometric characteristics of the ratings, and other factors suggest that the Angoff procedure, as modified during the second session of this study, yields the most defensible standards for the NTE area examinations.
AN ITERATIVE ITEM BIAS DETECTION METHOD
Authors: VAN DER FLIER HENK , MELLENBERGH GIDEON J. , ADÈR HERMAN J. , WIJN MARINA
Published: September 2005

Two strategies for assessing item bias are discussed: methods that compare (transformed) item difficulties unconditional on ability level and methods that compare the probabilities of correct response conditional on ability level. In the present study, the logit model was used to compare the probabilities of correct response to an item by members of two groups, these probabilities being conditional on the observed score. Here the observed score serves as an indicator of ability level. The logit model was iteratively applied: In the Tth iteration, the T items with the highest value of the bias statistic are excluded from the test, and the observed score indicator of ability for the (T + 1)th iteration is computed from the remaining items. This method was applied to simulated data. The results suggest that the iterative logit method is a substantial improvement on the noniterative one, and that the iterative method is very efficient in detecting biased and unbiased items.
THE USE OF A FACTOR‐ANALYTIC MODEL FOR ASSESSING THE VALIDITY OF GROUP COMPARISONS
Authors: HANNA GILA
Published: September 2005

This paper assesses the validity of a comparison of mean test scores for two groups of students, and of a longitudinal comparison of means within each group. Using LISREL, confirmatory factor analyses were carried out (a) to test the hypotheses of similar factor patterns, equal units of measurement, and equal accuracy of measurement between the two groups, and (b) to estimate the correlation between the latent traits measured by two successive test administrations in each group. The results indicate (a) that a comparison of the group means may be invalid because, although the factor pattern was the same for both groups, the factors were not measured in the same units, and (b) that longitudinal comparisons within each group are seriously complicated by evidence of structural change.
STANDARD ERRORS OF MEASUREMENT AT DIFFERENT ABILITY LEVELS
Authors: LORD FREDERIC M.
Published: September 2005

Four methods are outlined for estimating or approximating from a single test administration the standard error of measurement of number‐right test score at specified ability levels or cutting scores. The methods are illustrated and compared on one set of real test data.
THE USE OF MODEL ESSAYS TO REDUCE CONTEXT EFFECTS IN ESSAY SCORING
Authors: HUGHES DAVID C. , KEELING BRIAN
Published: September 2005

Several studies have shown the existence of context effects in the scoring of essays. That is, essays receive higher marks when preceded by poor quality scripts than when preceded by good quality scripts. The present study investigated the effectiveness of providing scorers with model essays to reduce the influence of context. Context effects persisted despite the use of model essays during scoring.
TECHNICAL GUIDELINES FOR ASSESSING COMPUTERIZED ADAPTIVE TESTS
Authors: GREEN BERT F. , BOCK R. DARRELL , HUMPHREYS LLOYD G. , LINN ROBERT L. , RECKASE MARK D.
Published: September 2005

Guidelines are proposed for evaluating a computerized adaptive test. Topics include dimensionality, measurement error, validity, estimation of item parameters, item pool characteristics and human factors. Equating CAT and conventional tests is considered and matters of equity are addressed.
APPLICATION OF COMPUTERIZED ADAPTIVE TESTING TO EDUCATIONAL PROBLEMS
Authors: WEISS DAVID J. , KINGSBURY G. GAGE
Published: September 2005

Three applications of computerized adaptive testing (CAT) to help solve problems encountered in educational settings are described and discussed. Each of these applications makes use of item response theory to select test questions from an item pool to estimate a student's achievement level and its precision. These estimates may then be used in conjunction with certain testing strategies to facilitate certain educational decisions. The three applications considered are (a) adaptive mastery testing for determining whether or not a student has mastered a particular content area, (b) adaptive grading for assigning grades to students, and (c) adaptive self‐referenced testing for estimating change in a student's achievement level. Differences between currently used classroom procedures and these CAT procedures are discussed. For the adaptive mastery testing procedure, evidence from a series of studies comparing conventional and adaptive testing procedures is presented showing that the adaptive procedure results in more accurate mastery classifications than do conventional mastery tests, while using fewer test questions.
A METHODOLOGICAL STUDY OF THE TORRANCE TESTS OF CREATIVITY
Authors: LISSITZ ROBERT W. , WILLHOFT JOSEPH L.
Published: September 2005

The sensitivity of creative thinking as measured by the Torrance Tests with regard to an experimenter‐induced response set was investigated. The 198 subjects were divided into four groups. Before performing the Unusual Uses Activity (Verbal Form A), each group was given a unique set of instructions. Group I received standard instructions and acted as a control. Other treatments were varied in terms of the types of responses that were encouraged. Group II was encouraged to be “practical and reasonable”; Group III was encouraged to list “as many ideas” as possible; and Group IV was encouraged to include all “unusual, weird, or illogical” ideas. Discriminant analysis yeilded two significant functions, suggesting that univariate analysis of the Torrance scales of Fluency. Flexibility, and Originality can be misleading. A multivariate Dunnett test showed that Torrance Tests were highly sensitive to brief experimenter manipulation. A very modest induced attitude shift resulted in a shift of up to approximately a standard deviation in the creativity scores.
EVALUATING AND PREDICTING SURVEY EFFICIENCY USING GENERALIZABILITY THEORY
Authors: JOHNSON SANDRA , BELL JOHN F.
Published: September 2005

A long‐term science performance monitoring program began in England, Wales, and Northern Ireland in 1980 with the first in an initial series of annual national sample surveys of the science performances of II‐, 13‐ and 15‐year‐old pupils. The assessment framework underlying this program is process‐oriented, consisting of a number of subcategories of science activity, some of which are assessed in practical mode. Pupils are randomly selected for testing according to a complex sampling scheme. Questions are also selected randomly to represent the various subcategories.

From the start of this program, it was intended to appeal to generalizability theory for a suitable estimation paradigm, and in this paper some preliminary applications of G‐theory are described. The results of these applications would suggest that computerized question‐banking, domain‐sampling of questions, and G‐theory together provide a useful new technology for this kind of performance monitoring exercise. The issue of interpretability might still remain a problem, however, unless the question domains can be clearly defined, and can be reflected in the question pools with consistency over time.
LEMS OF ANALYST AND OBSERVER AGREEMENT IN NATURALISTIC NARRATIVE DATA
Authors: SCOTT M.M , HATFIELD JAMES G.
Published: September 2005

The use of naturalistic narrative data is increasing but without increasing methodological rigor. Assumptions underlying such methods prohibit an investigator from interrupting the natural habitat by use of such standard laboratory procedures as equal observation intervals for all subjects, equal behavior units, or other artificial equalization of the behavioral data stream. Data analytic techniques commonly used in standard laboratory research present problems in the analysis of naturalistic data of this type inasmuch as most assume equal observation intervals, equal unit intervals, or some other fundamental character of equality. These problems are particularly acute with respect to estimates of analyst and observer agreement. This paper addresses these problems and discusses the advantages and disadvantages of several possible solutions.
COMPROMISE MODELS FOR ESTABLISHING EXAMINATION STANDARDS
Authors: DE GRUIJTER DATO N. M.
Published: September 2005

Cutoff scores based on absolute standards can be unacceptable in terms of the number of failures they produce. Cutoff scores based on relative standards, that is, cutoff scores set to achieve a fixed percentage of failures, can be unacceptable because an acceptable performance level for passed examinees cannot be guaranteed. In some situations one can improve upon an absolute standard using a compromise model, which draws on the information in the observed score distribution for a test to adjust the standard. Three compromise models are described and compared in this article.
THE ECOLOGY OF CLASSROOM ASSESSMENT
Authors: STIGGINS RICHARD J. , BRIDGEFORD NANCY J.
Published: September 2005

Research on classroom assessment has tended to focus on standardized tests and has paid minimal attention to teacher‐developed assessments. As a result, we have a narrow understanding of the classroom assessment environment. This study was designed to broaden that understanding by exploring the nature and quality of teacher‐developed assessment. Teachers from a range of grades, subjects, and school districts described their patterns of test use, concerns about assessment, and use of performance assessment by completing an extensive questionnaire. When responses are summarized across teachers, the results suggest that the foundation and structure of classroom assessment consists primarily of teacher‐developed assessments, with performance assessment serving as one of the key assessment tools. Teachers are concerned about assessment and know that improvement may be needed, but they will need help to bring about necessary changes. Action plans are suggested for enhancing the quality of teacher‐developed tests.
FACE VALIDITY REVISITED
Authors: NEVO BARUCH
Published: September 2005

The literature of the concept of face validity is surveyed. A definition of face validity involving four facets is proposed. Empirical data suggesting that face validity is a characteristic of tests that can be validly and reliably measured are presented. Implications for research and for practical use are discussed.
SAT SCORES AND AMERICAN STATES: SEEKING FOR USEFUL MEANING
Authors: PAGE ELLIS B. , FEIFS HELMUTS
Published: September 2005

Despite difficulties, states frequently seek to compare their test performance with that of other states, or with the published national norms. All states have some students taking the Scholastic Aptitude Test (SAT) exams for college admissions, but because the numbers vary widely, when the state means are made public by the College Boards, there is debate about their interpretation. Here the 50 state SAT means are the criterion in a multivariate model, together with the state variables of percent taking the SATs (very influential), percent of minority in the state, average income, unemployment rate, and per pupil expenditure. Regression is analyzed (without expenditure) to examine the state residuals compared with the “expected” scores. And briefly noted is the difficulty of using alternative tests for such comparisons, as these are currently normed, selected, and used.
DIFFERENCES IN INSTRUCTIONAL SENSITIVITY BETWEEN ITEM FORMATS AND BETWEEN ACHIEVEMENT TEST ITEMS
Authors: HANSON RALPH A. , MCMORRIS ROBERT F. , BAILEY JERRY D.
Published: September 2005

Building achievement tests which are sensitive to the instructional effects of school programs concerns both practitioners and researchers in education. To produce such tests, empirical procedures to guide item selection are needed. In this paper, an operational framework and a set of empirical procedures for this task are presented. Within this framework, item sensitivity is linked to instructional implementation. A simple components of variance model has been used to provide actual estimates of instructional sensitivity. These procedures are illustrated using data from a comparative study of alternative item formats for a criterion‐referenced test. Even when items were closely matched to instructional content specifications, important differences in instructional sensitivity emerged. These differences were found between the same items presented in different formats as well as between different items presented within the same format. Implications of these results for developing criterion‐referenced achievement tests are discussed.
MULTIPLE PROCESSING STRATEGIES AND THE CONSTRUCT VALIDITY OF VERBAL REASONING TESTS
Authors: EMBRETSON SUSAN , SCHNEIDER LISA M. , ROTH DAVID L.
Published: September 2005

This study examines the influence of processing strategies, and the associated metacomponents that determine when to apply them, on the construct validity of a verbal reasoning test. Three strategies for solving verbal analogy items were examined: a rule‐oriented strategy, an association strategy, and a partial rule strategy. Construct validity was studied in two separate stages: construct representation and nomothetic span. For construct representation, evidence was obtained that all three strategies, and their related metacomponents, are associated with performance on analogy items. For nomothetic span, the current study found that all three strategies contribute to individual differences in verbal reasoning and to the predictive validity of the test. The results of this study also point to the utility of metacomponents as constructs for describing and understanding test performance. Implications of the results for test development and theories of aptitude are elaborated.
DOES RESEARCH SUPPORT THE USE OF “PURPOSE QUESTIONS” ON READING COMPREHENSION TESTS?
Authors: ROWE DEBORAH W.
Published: September 2005

The inclusion of “purpose questions” before passages in reading comprehension tests is evaluated in light of recent research related to schema theory and adjunct questions. Schema‐theoretic studies of prepassage cues to the theme of a reading selection suggest that purpose questions may serve a schema activating function. Recent studies of higher level (nonliteral) prepassage questions also support this view. Conclusions commonly drawn from studies using literal level prepassage questions cannot be applied directly to the use of purpose questions on the Metropolitan Achievement Tests. While research supports the use of purpose questions, gaps and weaknesses in the available data exist. Suggestions for further study are provided.
CAN A TEST BE TOO RELIABLE?
Authors: WAINER HOWARD
Published: September 2005

It is shown that summary statistics that are commonly used to measure test quality (reliability, mean rbis, and mean proportion correct) can be seriously misleading. This is demonstrated and explained.
DETECTING IMPACTS OF CURRICULAR DIFFERENCES IN ACHIEVEMENT TEST DATA
Authors: MEHRENS WILLIAM A. , PHILLIPS S. E.
Published: September 2005

Standardized tests are designed to measure broad goals. But many professionals have been concerned with the lack of fairly specific matches between items (or objectives) on a test and the curriculum (instruction). This study assessed the differences in standardized test scores resulting from curricular differences in two school systems. The degree of curriculum‐test match for reading and math in grades 3 and 6 was based on ratings of that match by qualified district personnel. Further, results of using different textbook series were analyzed. The dependent variables of test and subtest scores were analyzed using a two‐factor MANCOVA where textbook series and school personnel ratings were the two factors, and pretest scores and percent eligible for Aid to Families with Dependent Children (AFDC) were the covariates. None of the multivariate F tests were significant at the .05 level. It was concluded that neither the curricular match as judged by district personnel or the textbook series used had a significant impact on standardized test scores.
A REJOINDER TO ALBANESE, “THE CORRECTION FOR GUESSING: A FURTHER ANALYSIS OF ANGOFF AND SCHRADER”
Authors: ANGOFF WILLIAM H. , SCHRADER WILLIAM B.
Published: September 2005

Albanese is critical of the Angoff‐Schrader (1984) study for several reasons, mainly, the fact that it draws some of its data from volunteer student‐subjects. If this criticism is valid, and if we therefore decide to exclude all studies based on volunteer subjects, we necessarily discontinue virtually all experimentation. Albanese recognizes this, but offers no workable substitute for experimental studies. We find, on the other hand, no evidence that the use of volunteer subjects impaired the soundness of our conclusions.

Albanese offers an index of his own construction for use in evaluating the Angoff‐Schrader data, and in applying it, comes to a different interpretation from that given by the authors. However, we find serious statistical flaws in his index when applied to our data. For example, we observe that when it is applied to small samples it yields uninterpretable results. After careful consideration of Albanese's reanalyses and interpretations, we stand firm in our original conclusion that formula scores are essentially invariant under different testing directions.
THE IMPACT OF ITEM DELETION ON EQUATING CONVERSIONS AND REPORTED SCORE DISTRIBUTIONS
Authors: DORANS NEIL J.
Published: September 2005

A formal analysis of the effects of item deletion on equating/scaling functions and reported score distributions is presented. There are two components of the present analysis: analytical and empirical. The analytical decomposition demonstrates how the effects of item characteristics, test properties, individual examinee responses, and rounding rules combine to produce the item deletion effect on the equating/scaling function and candidate scores, In addition to demonstrating how the deleted item's psychometric characteristics can affect the equating function, the analytical component of the report examines the effects of not scoring versus scoring all options correct, the effects of re‐equating versus not re‐equating, and the interaction between the decision to re‐equate or to not re‐equate and the scoring option chosen for the flawed item. The empirical portion of the report uses data from the May 1982 administration of the SA T, which contained the circles item, to illustrate the effects of item deletion on reported score distributions and equating functions. The empirical data verify what the analytical decomposition predicts.
DOES THE RASCH MODEL REALLY WORK FOR MULTIPLE CHOICE ITEMS? NOT IF YOU LOOK CLOSELY
Authors: DIVGI D. R.
Published: September 2005

This paper discusses various issues involved in using the Rasch model with multiple choice tests. By presenting a modified test that is much more powerful, the value of Wright and Panchapakesan's test as evidence of model fit is shown to be questionable. According to the new test, the model failed to fit 68% of the items in the Anchor Test Study. Effects of such misfit on test equating are demonstrated. Results of some past studies purporting to support the Rasch model are shown to be irrelevant, or to yield the conclusion that the Rasch model did not fit the data. Issues like “objectivity” and consistent estimation are shown to be unimportant in selection of a latent trait model. Thus, available evidence shows the Rasch model to be unsuitable for multiple choice items.
THE CHOICE OF SCALE FOR EDUCATIONAL MEASUREMENT: AN IRT PERSPECTIVE
Authors: YEN WENDY M.
Published: September 2005

Two methods of constructing equal‐interval scales for educational achievement are discussed: Thurstone's absolute scaling method and Item Response Theory (IRT). Alternative criteria for choosing a scale are contrasted. It is argued that clearer criteria are needed for judging the appropriateness and usefulness of alternative scaling procedures, and more information is needed about the qualities of the different scales that are available. In answer to this second need, some examples are presented of how IRT can be used to examine the properties of scales: It is demonstrated that for observed score scales in common use (i.e., any scores that are influenced by measurement error), (a) systematic errors can be introduced when comparing growth at selected percentiles, and (b) normalizing observed scores will not necessarily produce a scale that is linearly related to an underlying normally distributed true trait.
AN EXAMINATION OF THE ASSUMPTION THAT THE EQUATING OF PARALLEL FORMS IS POPULATION‐INDEPENDENT
Authors: ANGOFF WILLIAM H. , COWELL WILLIAM R.
Published: September 2005

Linear conversions were developed relating scores on two recent forms of the homogeneous GRE Quantitative Test (GRE‐Q) and the specially constituted heterogeneous GRE Verbal‐plus‐Quantitative Test (GRE‐V+Q), using randomly equivalent groups of about 13, 000 taking each form. Specially defined homogeneous subpopulations were then identified, and conversions between scores on the two forms were again calculated, this time based on l000‐case samples drawn at random from the subpopulations. Finally, in order to develop empirical measures of equating error, I00 samples of 1, 000 cases each were drawn at random from the two total groups and used to calculate 100 conversions between scores on the two forms. The conversions based on the specially selected subpopulations were then compared with the total‐group conversions and evaluated in terms of the empirical standard errors. The results showed that the conversions for the subpopulations agreed with the total‐group conversion quite satisfactorily for the GRE‐Q and almost as well for the GRE‐V+Q. It was concluded that the data clearly support the assumption of population independence for homogeneous tests, but not quite so clearly for heterogeneous tests.
MEASURING THE ORGANIZATIONAL ASPECTS OF WRITING ABILITY
Authors: BENTON STEPHEN L. , KIEWRA KENNETH A.
Published: September 2005

The present study assessed the relationship among holistic writing ability, the Test of Standard Written English (TSWE), and the following tests of organizational ability: anagram solving, word reordering, sentence reordering, and paragraph assembly. Based upon a sample of 105 undergraduate students, the main findings were that writing ability, as measured by the holistic method of scoring, was significantly correlated with performance on the TSWE and the four tests of organizational ability. A composite score on all four organizational tests was found to have the highest zero‐order correlation with the measure of writing ability. A stepwise regression analysis, with the measure of writing ability as the criterion, also indicated that the composite score explained a significant proportion of the variance beyond that explained by the TSWE. The results are discussed in terms of the Kintsch and van Dijk model of strategic discourse processing, which suggests that different organizational strategies operate at the levels of words, sentences, and paragraphs. It is concluded that tests assessing organizational strategies ought to be included in assessments of writing ability.
SAT Differential Item Performance for Nine Handicapped Groups
Authors: Bennett Randy Elliot , Rock Donald A. , Kaplan Bruce A.
Published: September 2005

The purpose of this study was to identify broad classes of items that behave differentially for handicapped examinees taking special, extended‐time administrations of the Scholastic Aptitude Test (SA T). To identify these item classes, the performance of nine handicapped groups and one nonhandicapped group on each of two forms of the SAT was investigated through a two‐stage procedure. The first stage centered on the performance of item clusters. Individual items composing clusters showing questionable performance were then examined. This two‐stage procedure revealed little indication of differentially functioning item classes. However, some notable instances of differential performance at the item level were detected, the most serious of which affected visually impaired students taking the braille edition of the test.
A Comparison of Three Variations on a Standard‐Setting Method
Authors: Norcini John J. , Lipner Rebecca S. , Langdon Lynn O. , Strecker Carolyn A.
Published: September 2005

The purpose of this study was to determine whether two variations on the typical Angoff group standard‐setting process would produce sufficiently consistent results to recommend their use. Judgments obtained from a group of experts during a meeting were compared with judgments gathered from the same group before and after the meeting. The results indicate that differences between passing scores obtained with the three variations are relatively small, but those gathered before the meeting were less consistent than ratings gathered during and after the meeting. These results imply that judgments gathered after an initial traditional group‐process session can provide an efficient alternative mechanism for setting cutting scores using the Angoff method.

This research was supported by The American Board of Internal Medicine, but does not necessarily reflect its opinions or policies.