bluuebunny/crossref_metadata_2025 huggingface dataset:
THE APPLICATION OF A FACTORIAL DESIGN TO THE STUDY OF CULTURAL BIAS IN GENERAL CULTURE ITEMS ON THE NATIONAL TEACHER EXAMINATION1
Authors: MEDLEY DONALD M. , QUIRK THOMAS J.
Published: September 2005

This article describes a series of studies performed with the National Teacher Examinations which were designed to study the relationship between the cultural content of special sets of general culture test items and the performance of blacks and whites on these experimental items. Significant differences between the performance of blacks and whites were found in terms of black, modern, and traditional test items. A replication of the study with the same test items, and also with a different group of test items, is also described.
AN EXPLORATORY ANALYSIS OF PERFORMANCE ON THE SAT
Authors: WAINER HOWARD
Published: September 2005

Methods of exploratory data analysis are used to decompose data tables that portray the performance of various ethnic groups on the Scholastic Aptitude Test (SAT). These analyses show the size and structure of the differences in performance among the ethnic groups studied, the nature of changes across time, and the interactions between ethnic group membership and time. Suggestions regarding the clear display of results from data in this form are made and illustrated.
EDUCATIONAL DIAGNOSTIC ASSESSMENT
Authors: BEJAR ISAAC I.
Published: September 2005

A strong demand currently exists for testing instruments that are capable of providing more informative and diagnostic results than typical tests offer. This paper reviews approaches that have been proposed for educational diagnostic assessment. Two major approaches are identified: (a) deficit assessment, which focuses on weaknesses of the student, and (b) error analysis, which focuses on the kinds of errors the student commits. This paper also reviews recent work related to diagnostic assessment that is based on the integration of methods from cognitive psychology and artificial intelligence. It is concluded that the development of powerful diagnostic instruments may require a reexamination of existing psychometric models and possibly the development of alternative ones. It is also pointed out that the traditional approach to the specification of content in terms of static taxonomies may not be appropriate given the dynamic and sequential nature of diagnostic assessment. Finally, it is noted that the psychometric and content demands of diagnostic assessment all but require test admininstration by computer.
THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT
Authors: MESSICK SAMUEL
Published: September 2005

Because school learning entails not just accretion of knowledge but the structuring and restructuring of knowledge and cognitive skills, the conception and construction of educational achievement measures must be cast in developmental terms. And because student characteristics as well as social and educational experiences influence current performance, the interpretation and implications of educational achievement measures must be relative to intrapersonal and situational contexts. These points imply a strategy of comprehensive assessment in context that focuses on the processes and structures involved in subject‐matter competence as moderated in performance by personal and environmental influences. This article addresses in detail both the nature of developing competence and its measurement in terms of context‐dependent task performance. Construct‐irrelevant task difficulty that might jeopardize the meaning of test scores as well as construct‐irrelevant influences that might jeopardize implications for action are taken into account via the comprehensive measurement of relevant contextual factors. Comprehensive assessment in context thus facilitates valid interpretations of the meaning and implications of ability and achievement scores in particular instances, thereby lightening the interpretive and ethical burdens on test users and enhancing the validity of test use.
A META ANALYSIS OF THE VALIDITY OF PREDICTORS OF PERFORMANCE IN LAW SCHOOL
Authors: LINN ROBERT L. , HASTINGS C. NICHOLAS
Published: September 2005

Although there is considerable evidence that the Law School Admission Test (LSAT) and the undergraduate grade‐point average (UGPA) have a useful degree of predictive validity, there is also a large variation in the magnitude of the coefficients across schools. Understanding this variation has important implications for the use and interpretation of results of a validity study conducted at an individual school. A meta analysis of the validity results and data on applicants to 154 law schools was conducted in an effort to better understand this observed variation. The standard deviation (SD) on the LSAT and the correlation between the LSAT and UGPA for accepted students at each law school accounted for 58.5% of the between‐school variance in the multiple correlations of these two predictors with first‐year average grade in law school. Sampling error accounted for an additional 12% of the variance. Hence, only a small fraction of the between‐school variability in validities remains to be explained by other statistical artifacts of situational specificity factors. Mean validities and 90% credibility values for four adjustment procedures are reported as are the mean observed validities for different combinations of predictors.
TAXONOMY OF VOCATIONAL INTERESTS OF ACADEMIC MAJOR AREAS
Authors: JACKSON DOUGLAS N. , HOLDEN RONALD R. , LOCKLIN RALPH H. , MARKS EDMOND
Published: September 2005

This study appraised the degree to which student profiles on the Jackson Vocational Interest Survey (JVIS) for different academic majors could be clustered in a meaningful way. From an initial sample of 10,134 students, a matrix of mean scores for 131 academic majors on each of 34 JVIS basic interest scales was computed. This matrix was subjected to a singular value decomposition with subsequent orthogonal and oblique rotations of 17 reference axes. The 17 clusters so defined reflected distinct sets of academic major fields, with separate clusters for majors in engineering, computer science, performing arts, communicative arts, human services, and others. Male and female groups entered into the definition of every cluster. Based on the salient representations of academic majors on these reference axes, modal cluster profiles were computed and decomposed into five orthogonal higher‐order dimensions. The implications of these findings for the psychology of occupational choice, career development, and vocational interest measurement are outlined.
IQ GAINS AND THE BINET DECREMENTS
Authors: FLYNN JAMES R.
Published: September 2005

Thorndike compiled Stanford‐Binet data which made it appear that children aged 6 and under have made greater IQ gains than older children and that this pattern dominated the whole period from 1932 to 1971–72. Therefore, he sought causal factors likely to affect preschoolers more than others, for example, TV in general and educational TV in particular. A wide array of data show that the atypical gains of young children are either an artifact of sampling error or totally antedate 1947, ruling out TV as an age‐specific factor. This data also suggest that Americans have gained about 12 IQ points from 1932 to 1972 with verbal gains being a point lower and performance gains a point higher.
MEASURING INSTRUCTIONAL VALIDITY: A REPORT OF AN INSTRUCTIONAL VALIDITY STUDY FOR THE ALABAMA HIGH SCHOOL GRADUATION EXAMINATION
Authors: HARDY ROY A.
Published: September 2005

Prior to the initial administration of the Alabama High School Graduation Exam, this study was conducted to determine to what extent competencies to be measured by the exam were being taught in the public schools of Alabama.

Teachers of grades 7, 8, 9, and 10 were asked to report the proportion of students in their classes who had received instruction on each competency. The survey identifies a number of competencies that were not being extensively taught and provides insight as to possible reasons certain competencies are not uniformly taught.
ISSUES IN ITEM BANKING
Authors: MILLMAN JASON , ARTER JUDITH A.
Published: September 2005

An item bank is defined as a relatively large collection of easily accessible test questions. A wide variety of item bank schemes that meet this relatively unrestricted definition is illustrated. Advantages and disadvantages of item banking and the conditions under which item banks have the most potential value are identified. An extensive list of questions to be asked in designing item banking systems is provided. The following five questions were singled out for further discussion: How many items should be in the bank? Should users develop their own item collections or use the collections of others? How should the items be classified? Should items be calibrated? Will each test have different items or will the same test be administered to all?
ITEM BANKS: WHAT, WHY, HOW
Authors: WRIGHT BENJAMIN D. , BELL SUSAN R.
Published: September 2005

In this paper, we discuss curricular implications of item banking and its practical value to teachers and students, and list a variety of working banks with their sources. We also review the psychometric basis of item banking, outline a family of computer programs for accomplishing banking, and give the equations necessary to build a bank. We conclude with a discussion of item quality control and examples of items that misfit because of miskeying, guessing, or carelessness.
A COMPREHENSIVE MICROCOMPUTER SYSTEM FOR CLASSROOM TESTING
Authors: NITKO ANTHONY J. , HSU TSE‐CHI
Published: September 2005

This paper describes a comprehensive system of microcomputer programs that was developed as a tool for teachers to use for (a) improving instruction and classroom tests, and (b) evaluating and reporting student learning. The programs implement many of the suggestions made in standard educational measurement textbooks for how to use data from tests to improve instructional decision‐making in the classroom and to improve quality of classroom testing. The paper describes the three main components of the system: (a) a student data‐base with associated information retrieval and computational programs, (b) a test item analysis and evaluation component that can be used for guiding instruction, and (c) a test item banking and test assembly component. In the course of the description, several microcomputer program design considerations are discussed.
EFFICIENCY OF LINEAR EQUATING AS A FUNCTION OF THE LENGTH OF THE ANCHOR TEST
Authors: BUDESCU DAVID
Published: September 2005

One of the most widely used methods for equating multiple parallel forms of a test is to incorporate a common set of anchor items in all its operational forms. Under appropriate assumptions it is possible to derive a linear equation for converting raw scores from one operational form to the others. The present note points out that the single most important determinant of the efficiency of the equating process is the magnitude of the correlation between the anchor test and the unique components of each form. It is suggested to use some monotonic function of this correlation as a measure of the equating efficiency, and a simple model relating the relative length of the anchor test and the test reliability to this measure of efficiency is presented.
MULTIPLE‐CHOICE VERSUS FREE‐RESPONSE: A SIMULATION STUDY
Authors: FRARY ROBERT B.
Published: September 2005

Responses to a 40‐item test were simulated for 150 examinees under free‐response and multiple‐choice formats. The simulation was replicated three times for each of 30 variations reflecting format and the extent to which examinees were (a) misinformed, (b) successful in guessing free‐response answers, and (c) able to recognize with assurance correct multiple‐choice options that they could not produce under free‐response testing. Internal consistency reliability (KR20) estimates were consistently higher for the free‐response score sets, even when the free‐response item difficulty indices were augmented to yield mean scores comparable to those from multiple‐choice testing. In addition, all test score sets were correlated with four randomly generated sets of unit‐normal measures, whose intercorrelations ranged from moderate to strong. These measures served as criteria because one of them had been used as the basic ability measure in the simulation of the test score sets. Again, the free‐response score sets yielded superior results even when tests of equal difficulty were compared. The guessing and recognition factors had little or no effect on reliability estimates or correlations with the criteria. The extent of misinformation affected only multiple‐choice score KR20′s (more misinformation—higher KR20′s). Although free‐response tests were found to be generally superior, the extent of their advantage over multiple‐choice was judged sufficiently small that other considerations might justifiably dictate format choice.
ESTABLISHING MINIMUM STANDARDS FOR ESSAYS: BLIND VERSUS INFORMED RÉVIEWS
Authors: CROSS LAWRENCE H. , FRARY ROBERT B. , KELLY PATRICIA P. , SMALL ROBERT C. , IMPARA JAMES C.
Published: September 2005

This study was undertaken to evaluate two procedures for establishing a minimum performance standard for the essay subtest of the National Teacher Examinations (NTE) Communication Skills test. Twenty public school teachers and 20 teacher educators were randomly assigned to either a “blind” or “informed” review panel. Both panels were directed to evaluate the same set of 12 sample essays. Those assigned to the informed panel were apprised of the scores previously awarded to each essay and were asked to decide upon the lowest score they would judge to be acceptable for a beginning teacher. Those assigned to the blind panel were not informed of the scores and were asked to judge whether each essay was at least minimally acceptable for a beginning teacher. Following their blind reviews, the scores were revealed to this group, and they were also asked to indicate the lowest score they would judge to be acceptable. No significant mean differences were observed for the standards associated with blind and informed reviews, but a significant mean difference was observed between the standards set by teachers and teacher educators; the former being more stringent. Based upon criteria other than mean differences, it was concluded that the preferred procedure for setting standards on essays should involve a blind review followed by an informed review. This strategy was subsequently implemented in a state‐wide validation study in Virginia, and the results of this full‐scale study are presented along with results reported for a study using informed reviews in another state.
EFFECTS OF INCORPORATING HUMOR IN TEST ITEMS
Authors: MCMORRIS ROBERT F. , URBACH SANDRA L. , CONNOR MICHAEL C.
Published: September 2005

Two matched forms of a 50 item multiple‐choice grammar test were developed. Twenty items designed to be humorous were included in one form. Test forms were randomly assigned to 126 eighth graders who received the test plus alternate forms of a questionnaire. Inclusion of humorous items did not affect grammar scores on matched humorous/nonhumorous items nor on common post‐treatment items, nor did inclusion affect results of anxiety measures. Students favored inclusion of humor on tests, judged effects of humor positively, and estimated humorous items to be easier. Humor did not lower performance but was sought by the students. Potential for more valid and humane measurement is discussed.
OPTIMAL NUMBER OF OPTIONS: AN INVESTIGATION OF THE ASSUMPTION OF PROPORTIONALITY
Authors: BUDESCU DAVID V. , NEVO BARUCH
Published: September 2005

One assumption common to all models for determining the optimal number of options per item (e. g., Lord, 1977) is that total testing time is proportional to the number of items and the number of options per item. Therefore, under this assumption given a fixed testing time, the test can be shortened or lengthened by deleting or adding a proportional number of options. The present study examines the validity of this assumption in three tests which were administered with 2, 3, 4, and 5 options per item. The number of items attempted in the first 10 and 15 minutes of the testing session and the time needed to complete the tests were recorded. Thus, the rate of performance for both fixed time and fixed test length was analyzed. A strong and consistently negative relationship between rate of performance and the number of options was detected in all tests. Thus, the empirical results did not support the assumption of proportionality. Furthermore, the data indicated that the method by which options are deleted can play a role in this context. A more realistic assumption of generalized proportionality, proposed by Grier (1976), was supported by the results from a Mathematical Reasoning test, but was only partially supported for a Vocabulary and a Reading Comprehension test.
TASK SPECIFICATION CHART: A KEY TO A BETTER UNDERSTANDING OF TEST RESULTS
Authors: BIRENBAUM MENUCHA , SHAW DORIS J.
Published: September 2005

A task specification chart (TSC) that integrates the content facets and the procedural steps of a specified task is suggested as a tool for designing a test and for interpreting its results. An instructional unit for adding and subtracting fractions was used to demonstrate the design and application of the TSC. To evaluate its efficacy, a test based on the TSC was administered to two independent samples. Accounts of the variance of item difficulty indices and errors in students’ responses were used as the criteria for this evaluation. The results indicated that a very high percentage of variance in item difficulty indices was accounted for by item characteristics representing different task components. The typology of errors constructed on the basis of the TSC proved to be an efficient tool for identifying erroneous rules of operation underlying students' response patterns on the test.
MAXIMUM LIKELIHOOD AND BAYESIAN PARAMETER ESTIMATION IN ITEM RESPONSE THEORY
Authors: LORD FREDERIC M.
Published: September 2005

Advantages and disadvantages of joint maximum likelihood, marginal maximum likelihood, and Bayesian methods of parameter estimation in item response theory are discussed and compared.
THE ROLE OF RELIABILITY IN CRITERION‐REFERENCED TESTS
Authors: KANE MICHAEL T.
Published: September 2005

In discussion of the properties of criterion‐referenced tests, it is often assumed that traditional reliability indices, particularly those based on internal consistency, are not relevant. However, if the measurement errors involved in using an individual's observed score on a criterion‐referenced test to estimate his or her universe scores on a domain of items are compared to errors of an a priori procedure that assigns the same universe score (the mean observed test score) to all persons, the test‐based procedure is found to improve the accuracy of universe score estimates only if the test reliability is above 0.5. This suggests that criterion‐referenced tests with low reliabilities generally will have limited use in estimating universe scores on domains of items.
DEMONSTRATING THE UTILITY OF THE STANDARDIZATION APPROACH TO ASSESSING UNEXPECTED DIFFERENTIAL ITEM PERFORMANCE ON THE SCHOLASTIC APTITUDE TEST
Authors: DORANS NEIL J. , KULICK EDWARD
Published: September 2005

The standardization method for assessing unexpected differential item performance or differential item functioning (DIF) is introduced. The principal findings of the first five studies that have used this approach on the Scholastic Aptitude Test are presented.
Curricular Differences and Unidimensionality of Achievement Test Data: An Exploratory Analysis
Authors: Phillips S. E. , Mehrens William A.
Published: September 2005

The purpose of this study was to investigate whether a linear factor analytic method commonly used to investigate violation of the item response theory (IRT) unidimensionality assumption is sensitive to measurable curricular differences within a school district and to examine the possibility of differential item performance for groups of students receiving different instruction. For grades 3 and 6 in reading and mathematics, personnel from two midwestern school systems that regularly administer standardized achievement tests identified the formal textbook series used and provided ratings of test‐instructional match for each school building (classroom). For both districts, the factor analysis results suggested no differences in percentages of variance for large first factors and relatively small second factors across ratings or series groups. The IRT analyses indicated little, if any, differential item performance for curricular subgroups. Thus, the impact of factors that might be related to curricular differences was judged to be minor.
Note About Possible Bias Resulting When Under‐Statisticized Studies are Excluded from Meta‐Analyses
Authors: Stanley Julian C.
Published: September 2005

Reviews and meta‐analyses of research on a given topic may exclude' a sizable percentage of reports because they do not lend themselves to the type of summarizing procedures used. If the excluded articles contain relevant information, this may bias the conclusions of the analysis. It seems likely that, when computing statistics from their data, researchers will need to consider this aspect. A simple illustration of how that can sometimes be done readily is presented. A robust correlation coefficient easily computable from published data is shown to indicate a sizable relationship that is contrary to the main conclusion of a meta‐analysis.
Grouping Continuous Data in Discrete Intervals: Information Loss and Recovery
Authors: Shaw Dale G. , Huffman Michael D. , Haviland Mark G.
Published: September 2005

Information loss occurs when continuous data are grouped in discrete intervals. After calculating the squared correlation coefficients between continuous data and corresponding grouped data for four population distributions, the effects of population distribution, number of intervals, and interval width on information loss and recover), were examined.
Assessing Partial Knowledge in Vocabulary
Authors: Smith Richard M.
Published: September 2005

This study reports an attempt to assess partial knowledge in vocabulary. Fifty multiple‐choice vocabulary items were constructed so that the incorrect choices followed the stages of vocabulary acquisition defined by O'Connor (1940). Ability estimates based on Rasch dichotomous and polychotomous models were compared to determine if there were any gains in validity or reliability as a result of using the polychotomous scoring model rather than the dichotomous scoring model. An attempt was also made to determine the appropriateness of O'Connor's stage theory of vocabulary acquisition for predicting the type of errors that examinees of differing ability would make on the test items. The results indicate that the reliability and concurrent validity of the polychotomous scoring of a subset of items that fit the polychotomous scoring model were significantly higher than those for dichotomous scoring of the same subset of items. The results also indicate moderate support for O'Connor's theory of vocabulary acquisition.
Validation of Cognitive Sensitivity for Item Response Curves
Authors: Tatsuoka Kikumi K.
Published: September 2005

This study sought a scientific way to examine whether item response curves are influenced systematically by the cognitive processes underlying solution of the items in a procedural domain (addition of fractions). Starting from an expert teacher's logical task analysis and prediction of various erroneous rules and sources of misconceptions, an error diagnostic program was developed. This program was used to carry out an error analysis of test performance by three samples of students. After the cognitive structure of the subtasks was validated by a majority of the students, the items were characterized by their underlying subtask patterns. It was found that item response curves for items in the same categories were significantly more homogeneous than those in different categories. In other words, underlying cognitive subtasks appeared to systematically influence the slopes and difficulties of item response curves.
Differential Grading Standards Revisited
Authors: Strenta A. Christopher , Elliott Rogers
Published: September 2005

The differential grading standards described by Goldman and Widawski (1976) exist in the same magnitude and in roughly the same order a decade later in a different kind of institution. Major fields that attract as majors students who score higher on SATs employ stricter grading standards. These differential grading standards serve to attenuate the correlation between SAT scores and grades, which is, even in highly selective institutions, substantial.
The Answer Key as a Source of Error in Examinations for Professionals
Authors: Norcini John J.
Published: September 2005

There has been a growing emphasis on written examinations that assess the ability of physicians and teachers to make decisions in the absence of protocols for action—a crucial aspect of professional competence. A characteristic of such tests is controversy, even among experts, about what constitutes the correct response to some of the items. This paper studied the impact of variability in answer keys, constructed using the aggregate method, on total errors of measurement. Results indicated that several scorers made a sizable contribution to reduction in measurement error and that scorers or groups of scorers who each developed the answer key for a subset of items produced better results than a single group that developed the answer key for all items. Implications of judgment tests scored using the aggregate method are described for teacher and physician certification.
A Practitioner's Guide to Computation and Interpretation of Reliability Indices for Mastery Tests
Authors: Subkoviak Michael J.
Published: September 2005

From the perspective of teachers and test makers at the district or state level, current methods for obtaining reliability indices for mastery tests like the agreement coefficient and kappa coefficient are quite laborious. For example, some methods require two test administrations, whereas single administration approaches involve complex statistical procedures and require access to appropriate computer software. The present paper offers practitioners tables from which agreement and kappa coefficients can be read directly. Further‐more, because these indices differ from traditional reliability coefficients, the issue of what constitutes acceptable values of agreement and kappa coefficients is also addressed
Defining Score Scales in Relation to Measurement Error
Authors: Kolen Michael J.
Published: September 2005

Scale scores for educational tests can be made more interpretable by incorporating score precision information at the time the score scale is established. Methods for incorporating this information are examined that are applicable to testing situations with number‐correct scoring. Both linear and nonlinear methods are described. These methods can be used to construct score scales that discourage the overinterpretation of small differences in scores. The application of the nonlinear methods also results in scale scores that have nearly equal error variability along the score scale and that possess the property that adding a specified number of points to and subtracting the same number of points from any examinee's scale score produces an approximate two‐sided confidence interval with a specified coverage. These nonlinear methods use an arcsine transformation to stabilize measurement error variance for transformed scores. The methods are compared through the use of illustrative examples. The effect of rounding on measurement error variability is also considered and illustrated using stanines
Multiple True‐False Items: A Study of Interitem Correlations, Scoring Alternatives, and Reliability Estimation
Authors: Albanese Mark A. , Sabers Darrell L.
Published: September 2005

Intercorrelations among multiple true‐false items were examined to determine to what extent each true‐false option can be treated as independent. Results from 157 health science students and 170 medical students showed that correlations between true‐false options associated with the same stem were from 2.6 to 7.0 times larger than those from different stems. This suggests that results from previous research indicating that each true‐false option could be treated as an independent item cannot be generalized to other tests and examinee populations without supporting evidence. Four scoring methods were explored which varied chance success levels and scoring for partial knowledge. The results showed that scoring methods incorporating partial knowledge were more reliable and possessed greater concurrent and predictive validity than those minimizing chance success. Methods for computing reliability estimates were compared and suggestions were offered regarding practical use
Models of Decisionmaking Processes for Multiple‐Choice Test Items: An Analysis of Spatial Ability
Authors: Wilcox Rand R. , Wilcox Karen Thompson
Published: September 2005

Latent class models of decisionmaking processes related to multiple‐choice test items are extremely important and useful in mental test theory. However, building realistic models or studying the robustness of existing models is very difficult. One problem is that there are a limited number of empirical studies that address this issue. The purpose of this paper is to describe and illustrate how latent class models, in conjunction with the answer‐until‐correct format, can be used to examine the strategies used by examinees for a specific type of task. In particular, suppose an examinee responds to a multiple‐choice test item designed to measure spatial ability, and the examinee gets the item wrong. This paper empirically investigates various latent class models of the strategies that might be used to arrive at an incorrect response. The simplest model is a random guessing model, but the results reported here strongly suggest that this model is unsatisfactory. Models for the second attempt of an item, under an answer‐until‐correct scoring procedure, are proposed and found to give a good fit to data in most situations. Some results on strategies used to arrive at the first choice are also discussed
A Note on the Application of Multiple Matrix Sampling to Standard Setting
Authors: Norcini John J. , Shea Judy A. , Ping James C.
Published: September 2005

In many of the methods currently proposed for standard setting, all experts are asked to judge all items, and the standard is taken as the mean of their judgments. When resources are limited, gathering the judgments of all experts in a single group can become impractical. Multiple matrix sampling (MMS) provides an alternative. This paper applies MMS to a variation on Angoff's method (1971) of standard setting. A pool of 36 experts and 190 items were divided randomly into 5 groups, and estimates of borderline examinee performance were acquired. Results indicated some variability in the cutting scores produced by the individual groups, but the variance components were reasonably well estimated. The standard error of the cutting score was very small, and the width of the 90% confidence interval around it was only 1.3 items. The reliability of the final cutting score was.98
A New Approach to Avoiding Problems of Scale in Interpreting Trends in Mental Measurement Data
Authors: Braun Henry I.
Published: September 2005

Extracting policy‐relevant information from large national surveys of educational achievement is ordinarily a nontrivial task. It is made more treacherous when the data are expressed on scales that are not uniquely determined. The paper begins with a critical analysis of a recent attempt to interpret the findings on reading achievement obtained by the National Assessment of Educational Progress (NAEP). It then describes a new approach to the quantification and interpretation of change and demonstrates its appropriateness for repeated cross‐sectional designs such as NAEP. Limitations imposed by the survey design and the nature of the measurements are highlighted
Invariance of Item Characteristic Functions With Variations in Instructional Coverage
Authors: Miller M. David , Linn Robert L.
Published: September 2005

An assumption of item response theory is that a person's score is a function of the item response parameters and the person's ability. In this paper, the effect of variations in instructional coverage on item characteristic functions is examined. Using data from the Second International Mathematics Study (1985), curriculum clusters were formed based on teachers’ ratings of their students’ opportunities to learn the items on a test. After forming curriculum clusters, item response curves were compared using signed and unsigned sum of squared differences. Some of the differences in the item response curves between curriculum clusters were found to be large, but better performance was not necessarily related to greater opportunity to learn. The item response curve differences were much larger than differences reported in prior studies based on comparisons of black and white students. Implications of the findings for applications of item response theory to educational achievement test data are discussed
The Effect of Inappropriate Omissions on Formula Scores: A Simulation Study
Authors: Frary Robert B.
Published: September 2005

Responses to a 50‐item, four‐choice test were simulated for 1,000 examinees under conventional formula‐scoring instructions. One hundred ninety‐two simulation runs reflected variations in the average level o f item difficulty, the extent to which examinees tended to omit inappropriately (when the formulascoring directions recommended guessing), the extent to which they were misinformed (classified correct answers as distractors), the extent to which they guessed contrary to the formula‐scoring instructions, the extent to which examinee ability and tendency to omit inappropriately were correlated, the examinee ability level at which misinformation was most prevalent, and the extent to which item difficulty was related to the probability that an examinee would be misinformed. For each examinee, formula scores and expected formula scores were determined allowing and not allowing inappropriate omissions. Under certain conditions, failure to guess as recommended by the formula‐scoring instructions produced nontrivial proportions o f examinees with expected score losses o f one or more points. These conditions were a test o f at least moderate difficulty, a low level for the tendency to be misinformed, and at least a moderate level for the tendency to omit inappropriately.
Analysis of Differential Item Functioning in the NAEP History Assessment
Authors: Zwick Rebecca , Ercikan Kadriye
Published: September 2005

The Mantel‐Haenszel approach for investigating differential item functioning was applied to U.S. history items that were administered as part o f the National Assessment o f Educational Progress, On some items, blacks, Hispanics, and females performed more poorly than other students, conditional on number‐right score. It was hypothesized that this resulted, in part, from the fact that ethnic and gender groups differed in their exposure to the material included in the assessment. Supplementary Mantel‐Haenszel analyses were undertaken in which the number o f historical periods studied, as well as score. was used as a conditioning variable. Contrary to expectation, the additional conditioning did not lead to a reduction in the number o f DIF items. Both methodological and substantive explanations for this unexpected result were explored.
Measuring Thinking Skills Through Classroom Assessment
Authors: Stiggins Richard J. , Griswold Maggie Miller , Wikelund Karen Reed
Published: September 2005

The classroom assessment procedures o f 36 teachers in grades 2 to 12 were studied in depth to determine the extent to which they measure students” higher order thinking skills in mathematics, science, social studies, and language arts. A wide variety o f assessment documents were analyzed, teachers were observed asking oral questions in their classrooms, and each teacher was interviewed. The results revealed that paper‐and‐pencil assessment documents were dominated by recall questions across all grade levels. However, inference was assessed also, especially in mathematics. Oral questions tended to tap recall too, with analysis and inference reflected to some extent. Across grades, subjects, and forms o f assessment, comparison and evaluation questions were rare. Although these teachers had been trained to teach thinking skills to some extent, they were less often trained to assess such skills. Those who were trained tended to ask a higher proportion o f thinking skills questions than those who were not. The training implications o f the results are discussed.
Multidimensional Equating
Authors: Hirsch Thomas M.
Published: September 2005

Equatings were performed on both simulated and real data sets using the common‐examinee design and two abilities for each examinee (i.e., two dimensions). Item and ability parameter estimates were found by using the Multidimensional Item Response Theory Estimation (MIRTE) program. The amount of equating error was evaluated by a comparison of the mean difference and the mean absolute difference between the true scores and ability estimates found on both tests for the common examinees used in the equating. The results indicated that effective equating, as measured by comparability o f true scores, was possible with the techniques used in this study. When the stability o f the ability estimates was examined, unsatisfactory results were found.
Scoring Constructed Responses Using Expert Systems
Authors: Braun Henry I. , Bennett Randy Elliot , Frye Douglas , Soloway Elliot
Published: September 2005

The use of constructed‐response items in large scale standardized testing has been hampered by the costs and difficulties associated with obtaining reliable scores. The advent of expert systems may signal the eventual removal of this impediment. This study investigated the accuracy with which expert systems could score a new, nonmultiple‐choice item type. The item type presents a faulty solution to a computer programming problem and asks the student to correct the solution. This item type was administered to a sample of high school seniors enrolled in an Advanced Placement course in Computer Science who also took the Advanced Placement Computer Science (APCS) examination. Results indicated that the expert systems were able to produce scores for between 82% and 95% of the solutions encountered and to display high agreement with a human reader on the correctness of the solutions. Diagnoses of the specific errors produced by students were less accurate. Correlations with scores on the objective and free‐response sections of the APCS examination were moderate. Implications for additional research and for testing practice are offered.
Parameter Recovery in the Graded Response Model Using MULTILOG
Authors: Reise Steve P. , Yu Jiayuan
Published: September 2005

The graded response model can be used to describe test‐taking behavior when item responses are classified into ordered categories. In this study, parameter recovery in the graded response model was investigated using the MULTILOG computer program under default conditions. Based on items having five response categories, 36 simulated data sets were generated that varied on true θ distribution, true item discrimination distribution, and calibration sample size. The findings suggest, first, the correlations between the true and estimated parameters were consistently greater than 0.85 with sample sizes of at least 500. Second, the root mean square error differences between true and estimated parameters were comparable with results from binary data parameter recovery studies. Of special note was the finding that the calibration sample size had little influence on the recovery of the true ability parameter but did influence item‐parameter recovery. Therefore, it appeared that item‐parameter estimation error, due to small calibration samples, did not result in poor person‐parameter estimation. It was concluded that at least 500 examinees are needed to achieve an adequate calibration under the graded model.
A Description of What Happens When an Examinee Takes a Multiple‐Choice Reading Comprehension Test
Authors: Farr Roger , Pritchard Robert , Smitten Brian
Published: September 2005

This study investigated the strategies a group of college students used to complete a portion of a standardized reading comprehension test. Twenty‐six students were randomly assigned to either an introspective interview, in which the subjects explained to a researcher what they were doing and thinking as they read the test passages and answered the multiple‐choice questions, or a retrospective interview, in which the students completed the test without interruption and then recounted for the researcher how they had gone about the task. Data analysis resulted in the identification of three broad categories of processing behavior: an overall approach to the test task, reading strategies, and test‐taking strategies. In addition, difficulties encountered by the subjects were identified. Results indicate that the common element in each subject's approach to the test was a focus on getting to the questions as quickly as possible and then using the questions to direct a search of the passage to locate the best possible information to answer the questions. The implications of these results for better understanding the relationship between test‐taking behavior and reading are discussed.
The Differential Impact of Curriculum on Aptitude Test Scores
Authors: Angoff William H. , Johnson Eugene G.
Published: September 2005

A sample of 22, 923 students who had taken the SAT and the GRE General Test was classified by the four general undergraduate fields of study and by sex. The authors performed several analyses to determine the degree of differential impact that sex and field of study might have on GRE‐Verbal, GRE‐Quantitative, and GRE‐Analytical scores after controlling on SAT‐Verbal and SAT‐Mathematical scores. They found, first, that the correlations of SAT‐Verbal with GRE‐Verbal scores and SAT‐Mathematical with GRE‐Quantitative scores were extremely high, .86 in the total sample and ranging from the low to middle .80s in the eight subgroups. The impact of curriculum and sex, after controlling on SAT scores, was found to be low on GRE‐ Verbal scores but relatively high on GRE‐Quantitative scores, with students in heavily quantitative fields enjoying an advantage over their peers in less quantitative fields of study. The impact was moderate on GRE‐Analytical scores. Further studies designed to “purify” the fields of study and include only clearly verbal fields and clearly mathematical fields showed small additional impact. An additional study indicated a generally slight effect of the institution attended on GRE‐Quantitative scores after controlling for sex, major field of study, and initial ability.
Gender Differences in Multiple‐Choice Tests: The Role of Differential Guessing Tendencies
Authors: Ben‐Shakhar Gershon , Sinai Yakov
Published: September 2005

The present study focused on gender differences in the tendency to omit items and to guess in multiple‐choice tests. It was hypothesized that males would show greater guessing tendencies than females and that the use of formula scoring rather than the use of number of correct answers would result in a relative advantage for females. Two samples were examined: ninth graders and applicants to Israeli universities. The teenagers took a battery of five or six aptitude tests used to place them in various high schools, and the adults took a battery of five tests designed to select candidates to the various faculties of the Israeli universities. The results revealed a clear male advantage in most subtests of both batteries. Four measures of item‐omission tendencies were computed for each subtest, and a consistent pattern of greater omission rates among females was revealed by all measures in most subtests of the two batteries. This pattern was observed even in the few subtests that did not show male superiority and even when permissive instructions were used. Correcting the raw scores for guessing reduced the male advantage in all cases (and in the few subtests that showed female advantage the difference increased as a result of this correction), but this effect was small. It was concluded that although gender differences in guessing tendencies are robust they account for only a small fraction of the observed gender differences in multiple‐choice tests. The results were discussed, focusing on practical implications.
A Comparison of Two Procedures for Computing IRT Equating Coefficients
Authors: Baker Frank B. , Al‐Karni Ali
Published: September 2005

In order to equate tests under Item Response Theory (IRT), one must obtain the slope and intercept coefficients of the appropriate linear transformation. This article compares two methods for computing such equating coefficients–Loyd and Hoover (1980) and Stocking and Lord (1983). The former is based upon summary statistics of the test calibrations; the latter is based upon matching test characteristic curves by minimizing a quadratic loss function. Three types of equating situations: horizontal, vertical, and that inherent in IRT parameter recovery studies–were investigated. The results showed that the two computing procedures generally yielded similar equating coefficients in all three situations. In addition, two sets of SAT data were equated via the two procedures, and little difference in the obtained results was observed. Overall, the results suggest that the Loyd and Hoover procedure usually yields acceptable equating coefficients. The Stocking and Lord procedure improves upon the Loyd and Hoover values and appears to be less sensitive to atypical test characteristics. When the user has reason to suspect that the test calibrations may be associated with data sets that are typically troublesome to calibrate, the Stocking and Lord procedure is to be preferred.
Stepping Up Test Score Conditional Variances
Authors: Woodruff David
Published: September 2005

In Woodruff (1990), I derived estimates for the conditional standard error of measurement in prediction (CSEMP), the conditional standard error of estimation (CSEE), and the conditional standard error of prediction (CSEP). My original estimates assume that the conditional residual error score variances and the conditional residual true score variances, obtained from the regression of an observed score onto a parallel observed score, obey the same step‐up rules as do the marginal error score variance and the marginal true score variance. The present article derives alternative estimates for the various test score conditional variances that do not depend on these assumptions.
On the Reliability of Testlet‐Based Tests
Authors: Sireci Stephen G. , Thissen David , Wainer Howard
Published: September 2005

If a test is constructed of testlets, one must take into account the within‐testlet structure in the calculation of test statistics. Failing to do so may yield serious biases in the estimation of such statistics as reliability. We demonstrate how to calculate the reliability of a testlet‐based test. We show that traditional reliabilities calculated on two reading comprehension tests constructed of four testlets are substantial overestimates.
A Strategy for Examining the Validity of Job Analysis Data
Authors: Colton Dean A. , Kane Michael T. , Kingsbury Carole , Estes Carmen A.
Published: September 2005

A strategy for examining the validity of job analysis inventory data is introduced. This strategy involves the testing of hypotheses regarding expected consistencies within the data, and between the data and independent sources of information. Although the purpose of the job analysis is to obtain an empirical description of work patterns, some relationships among the data can be predicted with a high degree of confidence, and these predictions can then be used to test the validity of the job analysis data. An example investigates the validity of data collected as part of a job analysis for nurses.