bluuebunny/crossref_metadata_2025_split huggingface dataset:
THE VALUES OF ABILITY TESTING: IMPLICATIONS OF MULTIPLE PERSPECTIVES ABOUT CRITERIA AND STANDARDS*
Authors: Messick Samuel 
Published: August 2014

ABSTRACTIn response to an invitation from the National Council on Measurement in Education to review the National Academy of Sciences report on Ability Testing, this paper explores the ideological consequences of the report's tacit acceptance of a pervasive institutional perspective. By focussing on selection and classification as the major functions of ability testing in both employment and education and by embracing job performance and grades as primary criteria, the report puts implicit priority on productivity and efficiency as the primary social values to be served by testing — tempered, of course, by the legal requirements of equal opportunity. As a consequence, the conflict between productivity and parity is given central status in the report along with a call for flexible decision rules to accommodate both, but at the expense of simultaneously downplaying other values important in the workplace and especially in education. This paper examines the utility of systematically contrasting multiple value perspectives as a means of reducing the likelihood that any single perspective will dominate our assumptions and methodologies.
THE EFFECT OF THE POSITION OF AN ITEM WITHIN A TEST ON ITEM RESPONDING BEHAVIOR: AN ANALYSIS BASED ON ITEM RESPONSE THEORY
Authors: Kingston Neal M. , Dorans Neil J. 
Published: August 2014

ABSTRACTThe research described in this paper deals solely with the effect of the position of an item within a test on examinee's responding behavior at the item level. For simplicity's sake, this effect will be referred to as practice effect when the result is improved examinee performance and as fatigue effect when the result is poorer examinee performance. Item response theory item statistics were used to assess position effects because, unlike traditional item statistics, they are sample invariant. In addition, the use of item response theory statistics allows one to make a reasonable adjustment for speededness, which is important when, as in this research, the same item administered in different positions is likely to be affected differently by speededness, depending upon its location in the test.Five types of analyses were performed as part of this research. The first three types involved analyses of differences between the two estimations of item difficulty (b), item discrimination (a), and pseudoguessing (c) parameters. The fourth type was an analysis of the differences between equatings based on items calibrated when administered in the operational section and equatings based on items calibrated when administered in section V. Finally, an analysis of the regression of the difference between b's on item position within the operational section was conducted. The analysis of estimated item difficulty parameters showed a strong practice effect for analysis of explanations and logical diagrams items and a moderate fatigue effect for reading comprehension items. Analysis of other estimated item parameters, a and c, produced no consistent results for the two test forms analyzed.Analysis of the difference between equatings for Form 3CGR1 reflected the differences between estimated b's found for the verbal, quantitative, and analytical item types. A large practice effect was evident for the analytical section, a small practice effect, probably due to capitalization on chance, was found for the quantitative section, and no effect was found for the verbal section.Analysis of the regression of the difference between b's on item position within the operational section for analysis of explanations items showed a rather consistent relationship for Form ZGR1 and a weaker but still definite relationship for Form 3CGR1.The results of this research strongly suggest one particularly important implication for equating. If an item type exhibits a within‐test context effect, any equating method, e.g., IRT based equating, that uses item data either directly or as part of an equating section score should provide for administration of the items in the same position in the old and new forms. Although a within‐test context effect might have a negligible influence on a single equating, a chain of such equatings might drift because of the systematic bias.
SAMPLING VARIANCES AND COVARIANCES OF PARAMETER ESTIMATES IN ITEM RESPONSE THEORY
Authors: Lord Frederic M. , Wingersky Marilyn S. 
Published: August 2014

ABSTRACTThis paper develops a possible method for computing the asymptotic sampling variance‐covariance matrix of joint maximum likelihood estimates in item response theory when both item parameters and abilities are unknown. For a set of artificial data, results are compared with empirical values; also with the variance‐covariance matrices found by the usual formulas for the case where the abilities are known, or where the item parameters are known. The results are consistent with the conjecture that the new method is asymptotically correct except for errors due to grouping.
CHARACTERISTICS OF OUTSTANDING YOUNG ARTISTS: ARTS (1980–81) AND SCHOLARS IN THE ARTS (1979–80)
Authors: Ward William C. 
Published: August 2014

ABSTRACTQuestionnaires were sent to the most outstanding young artists participating in the 1980–81 ARTS program and in its predecessor, the 1979–80 Scholars in the Arts. The students provided information about (a) their educational plans; (b) awards and other recognition they had received; (c) reasons for entering the program and reactions to their experience in it; (d) family background; and (e) their career aspirations. In general, participants were individuals with serious educational and career plans involving continued work in their discipline, and they reported their experience in the program to have been a positive and beneficial one.
DEVELOPING HOMOGENEOUS TOEFL SCALES BY MULTIDIMENSIONAL SCALING
Authors: Oltman Philip K. , Stricker Lawrence J. 
Published: August 2014

ABSTRACTA recent multidimensional scaling analysis of item response data for the TOEFL® identified clusters of items in the test sections and suggested that these clusters might be more homogeneous and more distinct than their parent sections, and hence better suited for diagnostic use. The present study explored the feasibility and value of using such cluster scores.The original analysis was based on all the item responses (choosing one of the four alternatives, or omitting or not reaching an item). That analysis was repeated, this time using the traditional scoring of item responses as correct or incorrect. Scores were then obtained for the within‐section clusters identified in the new analysis. The new dimensions and clusters were very similar to the old ones. The scores for the clusters and the test sections had similar internal‐consistency reliabilities and intercorrelations for the total sample, but diverged inconsistently for high‐scoring and low‐scoring examinees. Major conclusions were that (a) the cluster scores have no clear cut advantage over the section scores for diagnosis or other purposes and (b) taking account of particular incorrect responses yields little or no information beyond that provided by considering only whether the responses are correct or incorrect.
ANALYSIS OF THE REVISED STUDENT DESCRIPTIVE QUESTIONNAIRE: PHASE II: PREDICTIVE VALIDITY OF ACADEMIC SELF‐REPORT
Authors: Freeberg Norman E. , Rock Donald A. , Pollack Judith 
Published: August 2014

ABSTRACTAn initial study phase dealing with the suitability of the revised (1985) Student Descriptive Questionnaire (SDQ) assessed the accuracy of student self‐report for that instrument and found it to be of sufficient accuracy to fulfill its intended purposes. This second phase of study examines the adequacy of the revised SDQ in terms of the predictive validity of its student academic self‐report information against a criterion of first‐year college achievement (GPA).Findings indicate that the validities are consonant with earlier study results using the original version of the SDQ, as well as with other similar self‐report instruments used with college applicants. This was evident in overall magnitudes of the validities obtained, the precision of prediction in multiple regression equations, and in differences between validities for school‐reported and student‐reported high school academic achievement (as GPA and class rank). Multiple regression correlations and their associated regression weights were used to contrast validities for school‐ and student‐report prediction models. These results indicated slight but consistent superiority for the school‐reported information. In multiple prediction based on either school‐ or student‐reported information, apparent differences were found: (1) in the relative contributions of the predictor variables (SAT‐V, SAT‐M, and high school academic achievement); (2) in the relative importance of the predictors for subgroups based on race and, to a lesser extent, on sex; and (3) in differing roles for high school GPA and class rank as academic predictor variables.Other predictive validities were explored for SDQ information involving students' academic self‐concept and student‐reported grades in separate high school subject‐matter areas. Differential validities based on groups defined by high school academic curriculum (i.e., math/science vs. non‐math/science) in conjunction with intended choice of college academic major were also examined.
PREDICTION OF GRADES IN COLLEGE MATHEMATICS COURSES AS A COMPONENT OF THE PLACEMENT VALIDITY OF SAT‐MATHEMATICS SCORES
Authors: Bridgeman Brent , Wendler Cathy 
Published: August 2014

ABSTRACTBecause the Scholastic Aptitude Test Mathematical Score (SAT‐M) is a measure of general mathematical aptitude that is not closely linked to the specific skills that are prerequisites for a particular course or series of courses, it should not be the only information considered in making placement decisions. Nevertheless, if it can be shown that the SAT‐M score is a reasonably good predictor of success in particular mathematics courses, then it may have a role to play as one of the pieces of information to be considered. The question of the predictive validity of SAT‐M performance was addressed by collecting grades in freshman mathematics courses from 10 colleges. Compared to tests that were specifically designed for placement purposes, the SAT‐M score was a relatively poor predictor of grades in most courses. Even after correcting for the considerable range restriction that may occur when within‐course scores are analyzed, coefficients were typically only in the mid‐.30s (compared to corrected coefficients for a local placement test that ranged from the high .40s to the low .60s). Nevertheless, the SAT‐M score significantly improved predictions from high school grade‐point average alone, especially for calculus courses.In courses at all levels, grades of males and females were very similar, but the SAT‐M scores of males were significantly above the scores of females. Gender differences for predicting grades in particular mathematics courses generally could be eliminated or greatly reduced by considering high school GPA together with SAT‐M scores. Three subscores (Algebra, Insight, and Routine) were generated from the November 1985 SAT‐M test but they had no differential utility in prediction. No practically significant trait‐treatment interactions were demonstrated.For algebra versus precalculus course placement decisions, a weighted composite of SAT‐M score and self‐reported high school grades and courses taken correctly predicted 64 percent to 71 percent of the actual placements. For precalculus versus calculus decisions, about 80 percent of the actual placements could be predicted.
APPRAISING THE COGNITIVE FEATURES OF SUBJECT TESTS
Authors: Emmerich Walter 
Published: August 2014

ABSTRACTThe study's aim was to develop a procedure that could be used to appraise the cognitive features of subject (achievement) tests. Cognitive taxonomies and an accompanying coding scheme were developed and applied to the GRE Subject Tests in Psychology and Literature in English. The cognitive profiles of these two tests were found to be strikingly different, indicating that the taxonomies were operating as intended. Ways of improving and extending the procedure are discussed.
A COMPARISON OF MULTIPLE‐CHOICE AND CONSTRUCTED FIGURAL RESPONSE ITEMS
Authors: Martinez Michael E. 
Published: August 2014

ABSTRACTIn contrast to multiple‐choice test questions, figural response items call for constructed responses and rely upon figural material, such as illustrations and graphs, as the response medium. Figural response questions in various science domains were created and administered to a sample of 4th‐, 8th‐, and 12th‐grade students. Item and test statistics from parallel sets of figural response and multiple‐choice questions were compared. Figural response items were generally more difficult, especially for questions that were difficult (p<.5) in their constructed‐response forms. Figural response questions were also slightly more discriminating and reliable than their multiple‐choice counterparts, but had higher omit rates. The paper addresses the relevance of guessing to figural response items and the diagnostic value of the item type. Plans for future research on figural response items are discussed.
THE THREE STAGE DEVELOPMENT OF THE NTE SCHOOL PSYCHOLOGIST SPECIALTY AREA TEST FROM A JOB ANALYSIS PERSPECTIVE
Authors: DeMauro Gerald E. 
Published: August 2014

ABSTRACTThe utilization of survey data and expert judgment in test development is described in the three stages of developing the NTE School Psychologist Specialty Area Test. The first stage is described in the paper entitled “Knowledge Areas Important to School Psychology.” A survey of the membership of the National Association of School Psychologists helped determine knowledge areas and job activities important to the professional functions of an entry‐level school psychologist. Expert committee judgments were then used to link these knowledge areas to job domains. The second stage, described in the paper entitled “Specifying the Content Domain of the NTE School Psychologist Specialty Area Test” reviews how the findings of the first stage were used by the Committee of Examiners to establish content specifications and the representation of component knowledge areas for the test. The findings were also used to establish a job context for each test question. Finally, the “Follow Up: Specifying the Content and Context of NTE School Psychologist Items for Item Writers” documents how the directions of the Committee of Examiners were transmitted to item writers. The three papers, taken together, provide a step by step process of item development based on expert judgment.
SEX DIFFERENCES IN THE PERFORMANCE OF HIGH‐SCORING EXAMINEES ON THE SAT®‐M
Authors: Gallagher Ann M. 
Published: August 2014

ABSTRACTPerformance of high‐scoring males and females on the mathematics section of three forms of the College Board's Scholastic Aptitude Test (SAT‐M) was examined to determine how item content, solution strategy, and speededness differentially affect performance. The mathematical and verbal sections of the SAT were also compared for similarities in the performance patterns of high scorers.Items on three forms of the SAT‐M were classified on the basis of content and potential solution strategies. Male and female performance in the top 5 percent was examined using the Mantel‐Haenszel Differential Item Functioning (MH‐DIF) procedure. A main effect was found for item‐solution strategy, but not for item content. Males tended to outperform females on items requiring logical estimation or insight even when subjects were matched on years of mathematics course taking and average mathematics grades.Conventional measures indicated that the test was not differentially speeded. However, omit rates indicated that females omitted a greater proportion of items requiring estimation. Different patterns by gender were found on the mathematical and verbal sections of the test.
EXPERIMENTAL STUDY OF THE EFFECTS OF CALCULATOR USE ON THE ADVANCED PLACEMENT CALCULUS EXAMINATIONS1
Authors: Morgan Rick , Stevens Joe 
Published: August 2014

ABSTRACTAdvanced Placement Calculus examinations were administered to nearly 7,000 students from more than 400 high schools in order to determine the impact of calculator use. Both the Calculus AB and Calculus BC experimental examinations had two sections. Section I items were designed so that a calculator was not needed to arrive at the correct solution. Approximately half of the students were permitted to use calculators on this section. Section II items were developed to require a calculator to arrive at the correct solution. All students were permitted calculator use for these items. Analyses indicated that calculator use resulted in a substantial improvement in performance for 3 of the 40 Section I items. Those with calculators capable of graphics or the taking of derivatives performed better on both sections of the exams than those with scientific calculators. The internal consistency reliability of the Section I items was higher for students not permitted calculator use. A pattern of negative D‐DIF values for the Section II items indicated that these items were differentially more difficult for females than the Section I items. The D‐DIF value of only one Section I item appeared to be substantially impacted by calculator use. Total test scores were found to be higher for those permitted to use calculators, for males in comparison to females, and for those reporting the use of advanced calculators in comparison to those reporting the use of scientific calculators.
DEVELOPMENT OF PROCEDURES FOR RESOLVING IRREGULARITIES IN THE ADMINISTRATION OF THE LISTENING COMPREHENSION SECTION OF THE TOEFL TEST
Authors: Way Walter D. , McKinley Robert L. 
Published: August 2014

ABSTRACTThe purpose of this study was to develop and evaluate two procedures to be used in determining whether examinees in a given test center are affected by a testing irregularity on the Listening Comprehension section of the Test of English as a Foreign Language (TOEFL). One approach employs analysis of covariance (ANCOVA) on Listening Comprehension (Section 1) means using scores on Structure and Written Expression (Section 2) and scores on Reading and Vocabulary (Section 3) as covariates. The second procedure entails a Bayesian approach that uses prior information collected about performance at the center in question. Analyses using these two procedures were carried out using both simulated data and data from actual testing irregularities.The results of this study support the following conclusions. First, both the ANCOVA and Bayesian procedures appear to provide useful information related to the effects of testing irregularities on Section 1 of the TOEFL test; therefore, both procedures should be incorporated into the operational procedures for resolving testing irregularities. Second, the two procedures will usually agree about the effects of testing irregularities, although differences between the procedures may occur in situations where the Bayesian procedure indicates an effect of one scaled score point and the ANCOVA procedure does not indicate a statistically significant difference between the irregularity and comparison groups at some specified level. Finally, results based on the two procedures should be interpreted with caution, particularly in situations where the two procedures produce discrepant results.
AUTOMATED ITEM SELECTION USING ITEM RESPONSE THEORY*
Authors: Stocking Martha L. , Swanson Len , Pearlman Mari 
Published: August 2014

ABSTRACTThis paper presents a new heuristic approach to interactive test assembly that is called the successive item replacement algorithm. This approach builds on the work of van der Linden (1987a) and van der Linden and Boekkooi‐Timminga (1989) in which methods of mathematical optimization are combined with Item Response Theory (IRT) to construct tests from larger collections or pools of items. This new approach is contrasted with two more formal models as well as other heuristic approaches that appear in the literature. An experiment using quasi‐realistic data is performed that serves to illustrate the differences in approaches for typical practical test construction applications.
EFFECTS OF GRADUATE COURSEWORK ON THE GRE QUANTITATIVE SCORE FOR RECENT AND NONRECENT COLLEGE GRADUATES
Authors: Hale Gordon A. , Rock Donald A. , Longford Nicholas T. 
Published: August 2014

ABSTRACTPrevious research has shown that the mean quantitative score on the Graduate Record Examinations (GRE) General Test or GRE quantitative score, declines with age of the examinee, while the mean GRE verbal score remains relatively constant. It is assumed that the age‐related decline in quantitative score is due, at least in part, to an increase in time away from formal academic work. A hypothesis that follows from this observation is that taking formal quantitative coursework, such as that provided in graduate school, should cause nonrecent graduates' mean GRE quantitative score to rebound to a level that is closer to that of recent graduates.To test this hypothesis, recent and nonrecent college graduates who were currently enrolled in graduate programs in the social sciences, for whom preadmission GRE scores were available, and who had some quantitative coursework in those programs, were identified. Recent graduates were those who had graduated one year or less before taking the GRE, and nonrecent graduates were those who had graduated (or last received postgraduate schooling) seven years or more before taking the GRE. All students were administered an abbreviated General Test.The results failed to support the hypothesis; the difference between 264 recent and 66 nonrecent graduates' quantitative performance on the second (abbreviated) GRE test was just as pronounced, relative to scale, as was the difference on the first (preadmission) GRE test. Possible reasons for these results are discussed.