bluuebunny/crossref_metadata_2025_split huggingface dataset:
COGNITIVE PROCESSING REQUIREMENTS OF CONSTRUCTED FIGURAL RESPONSE AND MULTIPLE‐CHOICE ITEMS IN ARCHITECTURE ASSESSMENT
Authors: Martinez Michael E. , Katz Irvin R. 
Published: August 2014

ABSTRACTContrasts between constructed response items and stem‐equivalent multiple‐choice counterparts have yielded only a few weak generalizations. Such comparisons typically have involved averaging item characteristics, and this aggregation has masked differences in statistical properties at the item level. Moreover, even aggregated format differences have not been explained in terms of differential cognitive processing demands of the items. In this paper, we examine item‐level differences between figural response items and their multiple‐choice counterparts in the domain of architecture. The figural response item format is an assessment form that uses figural material (such as, graphs, illustrations, and diagrams) as item stimuli and the medium through which knowledge and skill are demonstrated. We first examine item‐level format differences in difficulty and then ask whether there are corresponding differences in the cognitive processing requirements of the items that can account for the psychometric differences. After finding evidence for these connections, we propose that differences in processing requirements and concomitant psychometric properties might be systematic and predictable. These analyses are important in a larger sense in that they shed light on aspects of construct validity that are frequently neglected and they touch the interface of the usually segregated psychometric and cognitive methodologies.
THE VARIANCE OF RASCH ABILITY ESTIMATES FROM PARTIALLY‐KNOWN ITEM PARAMETERS
Authors: Mislevy Robert J. 
Published: August 2014

ABSTRACTA closed‐form approximation is given for the variance of examinee proficiency estimates in the Rasch model for dichotomous items, under the condition that only estimates, rather than true values, of item difficulty parameters are available. The term that must be added to the usual response‐sampling variance is inversely proportional to both the number of examinees in the item calibration sample and the length of the test. Illustrative numerical values suggest that the impact of uncertainty about Rasch item parameters on subsequent estimates of examinee proficiencies is less than has been observed in the 2‐ and 3‐parameter logistic IRT models.
DIF DETECTION AND DESCRIPTION: MANTEL‐HAENSZEL AND STANDARDIZATION1,2
Authors: Dorans Neil J. , Holland Paul W. 
Published: August 2014

ABSTRACTAt the Educational Testing Service, the Mantel‐Haenszel procedure is used for differential item functioning (DIF) detection and the standardization procedure is used to describe DIF. This report describes these procedures. First, an important distinction is made between DIF and Impact, pointing the need to compare the comparable. Then, these two contingency table DIF procedures are described in some detail, first in terms of their own origins as DIF procedures, and then from a common framework that points out similarities and differences. The relationship between the Mantel‐Haenszel procedure and IRT models in general and the Rasch model, in particular, is discussed. The utility of the standardization approach for assessing differential distractor functioning is described. Several issues in applied DIF analyses are discussed including inclusion of the studied item in the matching variable, and refinement of the matching variable. Future research topics dealing with the matching variable, the studied variable and the group variable are also discussed.
COMBINING MULTIPLE‐CHOICE AND CONSTRUCTED RESPONSE TEST SCORES: TOWARD A MARXIST THEORY OF TEST CONSTRUCTION
Authors: Wainer Howard , Thissen David 
Published: August 2014

ABSTRACTFuture assessment instruments will probably be composed of a combination of different types of questions. Even though different kinds of questions require different scoring procedures there may be a need, for some purposes, to have those different scores combined as a composite. In this account we describe how mixtures of such scores might be efficaciously combined. Or, if no post hoc adjustment is desired, we provide two characterizations of measurement effectiveness to aid in making unadjusted score combinations efficient.
EFFECTS OF AMOUNT OF TIME ALLOWED ON THE TEST OF WRITTEN ENGLISH
Authors: Hale Gordon A. 
Published: August 2014

ABSTRACTThis study examined students' essay performance on topics from the Test of Written English (TWE®) under two time limits—30 minutes, as on the current TWE, and 45 minutes. In the main groups of the study, each student wrote an essay on one topic under one time limit and on another topic under the other time limit (with orders counterbalanced). The correlation between scores for 30‐ versus 45‐minute conditions was relatively high and approached the parallel‐form reliability of the task, as indicated by data from students who wrote essays on separate topics under the same time limits. Thus, the provision of additional time apparently had little effect on the standing of the students in relation to each other. Both parallel‐form reliability and interrater reliability were approximately the same for the 30‐ and 45‐minute conditions. Mean scores on the 6‐point TWE scale were found to be higher by about 1/4 to 1/3 point under the 45‐minute condition than the 30‐minute condition, indicating that provision of additional time produced a modest but reliable increase in scores. The magnitude of the effect was roughly comparable for students of low versus high proficiency, and for students in intensive English programs versus students in academic coursework. Responses to a questionnaire indicated that the students regarded 45 minutes as more sufficient for accomplishing the task than 30 minutes. The results are discussed in relation to the literature on time effects and to practical implications for the Test of Written English.
SCALAR ANALYSIS OF THE TEST OF WRITTEN ENGLISH
Authors: Henning Grant 
Published: August 2014

ABSTRACTThe present research was conducted to explore the psychometric characteristics of the Test of Written English (TWE) rating scale. Rasch model scalar analysis methodology was employed with more than 4,000 scored essays across two elicitation prompts to gather the following information about the TWE rating scale and rating process:
the position and size of the interval on the overall latent trait that could be attributed to behavioral descriptors accompanying each possible integer scoring step on the TWE scale
the standard error of estimate associated with each possible transformed integer rating
the fit of rating scale steps and individual rated essays to a unidimensional model of writing ability and, concurrently, the adequacy of such a model, including the proportion of misfitting essays as a portion of all essays analyzed
the fit of individual readers to a unidimensional model of writing ability and to the expectations of a chi‐square contingency test of independence of readers and ratings assigned, along with information on some characteristics of misfitting readers
comparative scalar information for two distinct TWE elicitation prompts, including nonparametric tests of the independence of readers and scale steps assigned and the feasibility of equating of scales.
Results suggested that the intervals between TWE scale steps were surprisingly uniform and that the size of the intervals was appropriately larger than the error associated with assignment of individual ratings. The proportion of positively misfitting essays was small (approximately 1% of all essays analyzed) and was approximately equal to the proportion of essays that required adjudication by a third reader. This latter finding, along with the low proportion of misfitting readers detected, provided preliminary evidence of the feasibility of employing Rasch rating scale analysis methodology for the equating of TWE essays prepared across prompts. Some information on characteristics of misfitting readers was presented that could be useful in the reader training process.
SEX DIFFERENCES IN PROBLEM‐SOLVING STRATEGIES USED BY HIGH‐SCORING EXAMINEES ON THE SAT‐M
Authors: Gallagher Ann M. 
Published: August 2014

ABSTRACTAn item classification scheme developed by Gallagher (1990) was refined, resulting in a more accurate prediction of sex differences in performance on the mathematical sections of the College Board's Scholastic Aptitude Test (SAT). When Differential Item Functioning (DIF) procedures were performed on item data for examinees scoring above 650, the majority of items that were flagged as favoring males required the use of mathematical insight, whereas all the items flagged as favoring females required standard algorithmic solutions.Structured interviews were conducted with students (25 males and 22 females) in this score range to determine the nature of differences in strategy use. A classification scheme was developed for strategies that paralleled the item classification categories. There was substantial overlap in strategies used by males and females; however, analyses of strategy types used across all items indicated that females were more likely than males to use algorithmic strategies and males were more likely than females to use insightful strategies. It should be noted that these findings constitute a generalization across a group of subjects and items, and that there were several individual instances of males using more algorithmic strategies than females and females using more insightful strategies than males.Questionnaire data gathered from students who participated in interviews indicated a positive relationship for both males and females between SAT‐mathematical performance and positive attitudes toward mathematics (e.g., liking mathematics as a subject and recognizing its usefulness to their adult lives). The use of algorithmic strategies, however, was correlated with negative attitudes toward mathematics (e.g., mathematics being difficult and not being relevant to their lives). Implications of these and other findings are discussed.
ELIGIBILITY ISSUES AND COMPARABLE TIME LIMITS FOR DISABLED AND NONDISABLED SAT® EXAMINEES
Authors: Ragosta Marjorie , Wendler Cathy 
Published: August 2014

ABSTRACTThis project used data from test administration timing records, the College Board's Scholastic Aptitude Test (SAT) history file, and a survey questionnaire to investigate two issues: comparable testing time and eligibility for special test accommodations for SAT examinees with disabilities.Comparable testing time for disabled examinees was found to be between 1½ and 2 times that for nondisabled examinees. Time limits in that range would assure that approximately equal percentages of disabled and nondisabled students would complete each section of the SAT The exception was blind students using braille or cassette versions of the test, who required between double and triple time.Eligibility for special test accommodations is tied to severity of disability and documentation of disability. Some levels of disability could be distinguished for students with sensory disabilities, based on self‐reports and confirmed by the presence or absence of individualized education programs (IEPs). For example, IEPs were reported for 80 percent of totally blind candidates, 50 percent of legally blind candidates, and only 28 percent of visually impaired examinees who were not considered legally blind. Similarly, 90 percent of hearing‐impaired students using American Sign Language (ASL) reported having IEPs, compared to 76 percent of those using total communication (English and sign) and 42 percent of those for whom English was the preferred communication mode. Severity of disability could not be defined for physically disabled or learning‐disabled examinees, except on the basis of IEPs. However, it was difficult to isolate the need for test accommodations based on school practices (e.g., IEPs or special schools or classes), since examinees eligible for special test administrations were often tested under standard conditions.Alternatives to the current eligibility policy are discussed, including a change to school‐based criteria and the use of individualized testing programs. A change to empirically derived testing times is also discussed.
A METHOD FOR SEVERELY CONSTRAINED ITEM SELECTION IN ADAPTIVE TESTING
Authors: Stocking Martha L. , Swanson Len 
Published: August 2014

ABSTRACTThis paper examines the current state of attempts to incorporate expert test construction practices into a computerized adaptive testing paradigm. A new methodology for incorporating large numbers of constraints on adaptive test item selection is described. This methodology builds on the conventional, that is, linear, test construction practices employed by expert test specialists. The incorporation of such practices is necessary if computerized adaptive testing is to become a serious competitor to conventional paper‐and‐pencil testing in the future. The successful application of this new methodology to the design of an adaptive test in a real‐life setting is demonstrated by an example.
A PSYCHOMETRICALLY SOUND COGNITIVE DIAGNOSTIC MODEL: EFFECT OF REMEDIATION AS EMPIRICAL VALIDITY
Authors: Tatsuoka Kikumi K. , Tatsuoka Maurice M. 
Published: August 2014

ABSTRACTThe purpose of this study was to validate the results of cognitive diagnoses using the rule‐space model and to demonstrate the usefulness of cognitive diagnoses for instruction. The results of the study strongly indicated that the rule‐space model can effectively diagnose students' knowledge states and can point out ways for remediating their errors quickly with minimum effort.It was also found that the designing of instructional units for remediation can be effectively guided by the rule‐space model, because the determination of all possible ideal item‐score patterns, given an incidence matrix, is based on a tree structure of cognitive attributes, knowledge states, and items.
PREPUBLISHING THE ESSAY TOPICS FOR A TEACHER LICENSING TEST: A SMALL‐SCALE SIMULATION
Authors: Powers Donald E. , Fowles Mary E. , Farnum Marisa , Aberle Dale J. , Collins Grace E. , Bly Mary , Palmquist Michael , Lytle Richard , Mather Susan , Pollock Nancy 
Published: August 2014

ABSTRACTThe computer‐based Praxis I: Academic Skills Assessment component of The Praxis Series: Professional Assessments for Beginning Teachers™ will include a measure of prospective teachers' writing proficiency. Current plans call for making available to test takers (before they take the exam) a pool of approximately 80 essay topics, from which the actual test topics will be drawn initially.The objectives of the study described here were to predict the impact of this policy on the ways in which test takers will prepare for the test and to estimate the effect of this new practice on examinees' eventual test performance. To make these determinations, a small‐scale study was undertaken in collaboration with writing instructors at four different colleges or universities. Within the context of undergraduate writing courses at these institutions, instructors designed and conducted brief “coaching” programs in response to a challenge to provide any advice, instruction, tricks, etc., that might help their students achieve a minimally acceptable score on the test. A pool of 10 topics was disclosed and used in the coaching. Upon completion of the coaching, all students wrote two essays—one on a disclosed topic and one on a comparable, previously unseen topic.Comparisons revealed very little difference between students' subsequent performances on these two types of topics. Performances on the two topics did not correlate differentially with any of three available indicators of academic ability/achievement. However, students believed that their writing skills were better revealed by writing on disclosed topics than on topics that had not been previously seen. In short, the study uncovered little to suggest that plans for disclosing a pool of Praxis I essay topics should be modified.
RELIABILITY OF THE TEST OF SPOKEN ENGLISH REVISITED
Authors: Boldt R. F. 
Published: August 2014

ABSTRACTThe Test of Spoken English (TSE) is an internationally administered instrument for assessing nonnative speakers' proficiency in speaking English. The research foundation of the TSE® examination is currently described in the Manual for Score Users. This publication describes technical characteristics of the test, including such psychometric characteristics as level of difficulty, reliability, and validity. Consistent with the ETS Standards for Quality and Fairness, the Manual refers to two sources of variation other than the achievement being measured: interrater reliability and internal consistency. Because the reported data on both issues were drawn from a 1980 study of reliabilities based on two raters, newer data and more extensive analyses were needed.The present study uses data from recent administrations of the TSE examination. Analysis of variance examined the effects of scale, section, examinee, and rater, as well as the interactions of these factors. Reliabilities were reported for item, section, and scale scores. Common and unique variance estimates were developed for each scale score for which a section was rated. Estimates of the effects of altering section lengths suggested that some sections should be lengthened and others omitted if reliability were to be maximized. Other suggestions were offered for improving reliability.
STRATEGY USE ON MULTIPLE‐CHOICE AND FREE‐RESPONSE ITEMS: AN ANALYSIS OF SEX DIFFERENCES AMONG HIGH SCORING EXAMINEES ON THE SAT‐M
Authors: Gallagher Ann , Mandinach Ellen 
Published: August 2014

ABSTRACTTwenty‐four students who scored 650 or more on the SAT‐M were asked to think aloud while solving 13 mathematics items in either multiple‐choice or free‐response format. Strategies students used to solve the items were classified as either algorithmic or insightful. Data analyses indicated that items in the free‐response format were significantly more difficult for females than multiple choice items. No significant difference was found for males. Females were more likely than males to use algorithmic strategies in both the multiple‐choice and free‐response formats, with no significant difference in types of strategies used in either format. Males were more likely to use insightful strategies in the multiple‐choice format than in the free‐response format. Finally, on multiple‐choice items, both males and females used options to detect calculation errors, however with females using this strategy somewhat more than males.
TOWARD A STABLE DIAGNOSTIC REPRESENTATION OF STUDENTS' ERRORS IN ALGEBRA
Authors: Birenbaum Menucha , Kelly Anthony E. , Tatsuoka Kikumi K. , Gutvirtz Yaffa 
Published: August 2014

ABSTRACTDiagnoses of students' performance on procedural mathematical tasks need to display a certain level of stability and robustness if they are to be used as the basis for remediation, particularly with computer‐delivered instruction. The purpose of this study was to compare two diagnostic approaches for describing students' errors in algebra – a bug analysis and a rule‐space analysis – with the goal of investigating the relative stability of the diagnoses derived from these approaches. Consistent with the findings of recent studies, a relatively large number of bugs were unstable; stable bugs tended to be infrequent. In contrast, the results of the rule‐space analysis yielded relatively more stable diagnoses. The results were discussed in light of their consequences for designing remediation.
DISTRIBUTIONS OF ACTFL RATINGS BY TOEFL SCORE RANGES
Authors: Boldt R. F. , Larsen‐Freeman D. , Reed M. S. , Courtney R. G. 
Published: August 2014

ABSTRACTThe purpose of this study was to align verbal descriptions of test takers' language performance with distributions of the numerical scores they received on the three sections (Listening Comprehension, Structure and Written Expression, and Reading Comprehension and Vocabulary) of the Test of English as a Foreign Language (TOEFL).The study used the American Council of Teaching Foreign Languages (ACTFL) descriptors as anchors for the TOEFL scores. The ACTFL Proficiency Guidelines, from which the descriptors were taken, are widely used.Qualified English as a Second Language (ESL) instructors rated their students' listening, reading, and writing proficiency using the ACTFL descriptors. Very shortly thereafter, the ratees took the TOEFL. Students' ACTFL ratings were quantified and cross‐tabulated with TOEFL section scores.Results were analyzed as follows: First, the need to adjust for rater difficulty was investigated. Second, typical descriptive data were developed. Third, several schemes were used to quantify the verbal descriptors from the ACTFL Guidelines. Fourth, the ratings and test scores were correlated. Fifth, the issue of the differential validity of the measures of speaking, listening, and reading was examined. Sixth, several estimates of the reliability of the ratings were obtained. Finally, the distributions of ratings at levels of the TOEFL section scores were developed. These latter distributions can be helpful in interpreting TOEFL scores in terms of language performance. Although there was no one‐to‐one correspondence between the TOEFL score level and the ACTFL rating level, the correlations between ratings and scores were substantial.
COMBINATION OF CONDITIONAL LOG‐LINEAR STRUCTURES
Authors: Kim Sung‐Ho 
Published: August 2014

ABSTRACTThere has been increasing attention to the fine structure of abilities underlying task performance (Haertel and Wiley (in press)). One of the useful approaches for this is by use of graphical models to represent relationships among abilities and test items. Building a large graphical model is always an issue. Restrictions upon experiments and data collection, among others, may result in parts of the large model. Or it may be convenient for us to build parts of the large models first, and then try to combine those parts into a larger model. This paper derives, confined to categorical variables only, a theory which may be useful in combining conditional graphical models into a larger one. The main result of the paper is that we can see partial information about a true log‐linear structure (LLS) from its conditional LLSs and use the information in trying to guess the true LLS, assuming that the true LLS is graphical. An application of the result is illustrated using a simulated data set.
FAMILY CHARACTERISTICS AS PREDICTORS OF INDIVIDUAL DIFFERENCES IN CHICANO CHILDREN'S EMERGENT SCHOOL READINESS
Authors: Laosa Luis M. 
Published: August 2014

ABSTRACTThis study of normal young children in 100 two‐parent Chicano households of widely varied socioeconomic levels assessed the ability of selected family sociodemographic and intellectual characteristics to predict individual differences in the children's performance on a measure of emergent school readiness. Data were collected longitudinally in the home at ages 30, 42, and 48 months. Emergent school readiness was measured using the Preschool Inventory, administered at each of the last two longitudinal points. The family characteristics examined were (a) specific sociodemographic variables (household financial income, mother's and father's schooling attainment levels, occupational status, and the family size, sibling constellation, and home language) and (b) the mother's and father's performance on the Culture Fair Intelligence Test. Contemporaneous and time‐lagged zero‐order and partial correlations and multiple regressions assessed the unique, shared, and shifting strengths of these hypothesized family predictors of child performance. Together these predictors accounted for 50% of the variance in child performance.