bluuebunny/crossref_metadata_2025_split huggingface dataset:
AN INVESTIGATION OF IRT‐BASED ASSEMBLY OF THE TOEFL TEST
Authors: Chyn Susan , Tang K. Linda , Way Walter D. 
Published: August 2014

ABSTRACTThe current study, carried out jointly by Test Development and Statistical Analysis staff at Educational Testing Service (ETS), investigated the feasibility of the Automated Item Selection (AIS) procedure for the Test of English as a Foreign Language (TOEFL). IRT‐based statistical specifications were developed. Two TOEFL® test forms were assembled using AIS and statistical and content‐related properties were evaluated. The results of this study show that the statistical consistency (parallelism) of the tests assembled using AIS would appear to be superior to the consistency of tests assembled using traditional test‐assembly procedures. The results of the study also provided strong evidence that AIS‐assembled TOEFL tests can successfully meet the IRT‐based specifications. Test Development staff observed visible gains in efficiency in item selection for Section 1 and Section 2 and the potential for time gains in Section 3. The implications of AIS for pool management were explored.
PASS‐FAIL RELIABILITY FOR TESTS WITH CUT SCORES: A SIMPLIFIED METHOD
Authors: Breyer F. Jay , Lewis Charles 
Published: August 2014

ABSTRACTA single‐administration classification reliability index is described that estimates the probability of consistently classifying examinees to mastery or nonmastery states as if those examinees had been tested with two alternate forms. The procedure is applicable to any test used for classification purposes, subdividing that test into two half‐tests, each with a cut score, where the sum of the two half‐test cut scores is equal to the cut score for the total test. The application of this pass‐fail consistency index to binary scored objective tests, nonbinary scored performance tests, and tests containing both binary and nonbinary scored questions is presented. A calculation example is provided together with look‐up tables.
RELATIONSHIPS BETWEEN DIFFERENTIAL PERFORMANCE ON MULTIPLE‐CHOICE AND ESSAY SECTIONS OF SELECTED AP® EXAMS AND MEASURES OF PERFORMANCE IN HIGH SCHOOL AND COLLEGE
Authors: Bridgeman Brent , Morgan Rick 
Published: August 2014

ABSTRACTStudents with high scores (top third) on the essay portion of an Advanced Placement Examination and low scores (bottom third) on the multiple‐choice portion of the same examination were compared with students whose performance showed the opposite pattern (top third on the multiple‐choice questions and bottom third on the essay questions). Across examinations in different subject areas (history, English, and biology), students who were relatively strong in the essay format and weak in the multiple‐choice format were about as successful in their college courses as students whose performance showed the opposite pattern, especially in those courses where grades are typically not determined by multiple‐choice tests. Students who scored high on the multiple‐choice portion and low on the essay portion performed relatively well on other multiple‐choice tests, especially the verbal section of the SAT. Across several ethnic/racial groups, males tended to receive relatively high scores on the multiple‐choice portion of the AP United States History Examination while females received higher scores on the essays than on the multiple‐choice questions. Among females whose best language was not English, scores were substantially higher on the essay portion of the history examination; among males in this group, scores were slightly higher for the multiple‐choice questions. Because the population of students who take Advanced Placement Examinations is exceptionally able, generalizations to less able populations are not warranted.
AN INVESTIGATION OF PROPOSED REVISIONS TO SECTION 3 OF THE TOEFL TEST
Authors: Schedl Mary A. , Thomas Neal , Way Walter D. 
Published: August 2014

ABSTRACTThe TOEFL testing program is currently exploring a change in Section 3 of the TOEFL® test that would replace the vocabulary subpart with additional reading comprehension questions. This change has been proposed by internal test development specialists and is supported by external experts in the field of English as a second language. The purpose of this study was to investigate the proposed revision to Section 3 in terms of the length and timing that would be necessary to address concerns of test speededness of the section. The study was carried out using an experimental design with test length and testing time defined as independent variables, and examinee test performance defined as the dependent variable. In addition, several psychometric issues relating to the proposed revision to Section 3 were investigated as part of the study.The results of the study supported the implementation of a revised TOEFL Section 3 consisting of five reading passages with a total of 50 items. The results of the study also suggested that a total testing time of no less than 55 minutes should be allowed for the revised TOEFL Section 3. Additional psychometric analyses indicated that the current TOEFL score scale can be maintained with the revised Section 3, and that the proposed revisions will not appreciably affect the reliability and validity of Section 3 of the TOEFL test.
EQUITY ASSESSMENT FOR POLYTOMOUSLY SCORED ITEMS: A TAXONOMY OF PROCEDURES FOR ASSESSING DIFFERENTIAL ITEM FUNCTIONING1
Authors: Dorans Neil J. , Potenza Maria T. 
Published: August 2014

ABSTRACTEducational reform efforts have led to increased use of alternatives to the traditional binary–scored multiple choice item. Many of the stimuli employed by these alternative assessments yield complex responses that require complex scoring rules. Some of these new item types can be polytomously–scored. Differential item functioning (DIF) assessment is a form of equity assessment. DIF assessment attempts to identify items for which subpopulations of examinees exhibit performance differentials that are inconsistent with the performance differentials typically seen for those subpopulations on collections of items that purport to measure a common construct. Any DIF technique can be evaluated in terms of how well it meets certain statistical and practical criteria. These DIF assessment criteria need to be attended to before we can conclude that the items associated with alternative forms of assessment can be adequately tested for DIF. DIF methodology is well–defined for traditional, binary–scored multiple–choice items. This paper provides a classification scheme of DIF procedures for binary–scored items that is applicable to new DIF procedures for polytomously–scored items. In the process, a formal development of a polytomous version of a binary DIF technique is presented. Finally, several polytomous DIF techniques are evaluated in terms of statistical and practical criteria.
A CASE FOR ADJUSTING SUBJECTIVELY RATED SCORES IN THE ADVANCED PLACEMENT TESTS*
Authors: Longford Nicholas T. 
Published: August 2014

ABSTRACTThe purpose of this report is to present a case for adjusting the scores for free response items in the Advanced Placement tests. Using information about the rating process from the reliability studies, administrations of the Advanced Placement test for three subject areas are analyzed, and it is shown that the minimum squared error score adjustments proposed by Longford (1993) result in changed AP grades for an appreciable percentage of the examinees.
ALTERNATIVE MODES OF ASSESSMENT, UNIFORM STANDARDS OF VALIDITY1
Authors: Messick Samuel 
Published: August 2014

ABSTRACTIn contrast to multiple choice, alternative modes of assessment afford varying degrees of openness in the allowable responses. Prominent among the alternatives is the assessment of performance, sometimes in its own right where the issue is the quality of the particular performance per se, but more often as a vehicle for the assessment of knowledge, skill, or other attributes. Because inferences about score meaning in construct terms and about the action implications of that meaning are fundamentally similar in the alternative assessment modes (despite surface differences), the same standards of validity apply to all educational and psychological measurement. These standards are addressed in terms of content, substantive, structural, generalizability, external, and consequential aspects of construct validity.
ESTIMATING THE EFFECTS OF TEST LENGTH AND TEST TIME ON PARAMETER ESTIMATION USING THE HYBRID MODEL
Authors: Yamamoto Kentaro 
Published: August 2014

ABSTRACTWhen individuals perform tasks, they differ from each other not only in their ability to perform the tasks correctly, but also in their speed. Even though the traditional indicator of test speededness, missing responses, clearly indicates a lack of time to respond (thereby indicating the speededness of the test), it is inadequate for evaluating speededness in a multiple‐choice test scored as number correct and underestimates test speededness. Conventional IRT parameter estimation ignores the mixture of random responses during calibration; consequently, estimated parameters are biased.The HYBRID model (Yamamoto, 1989) was extended (Yamamoto, 1990) to characterize when each examinee switches from an ability‐based response strategy to a strategy of responding randomly. The model has allowed us to evaluate test speededness by estimating the proportions of examinees who switch strategies at any possible point in the test. The estimated IRT parameters based on the HYBRID model were more accurate than the ordinary IRT‐only analysis.With the extended HYBRID model applied to the data taken from an experimental form of TOEFL®, we found that 1) the test length had a small impact on the proportion of the examinees affected by the speededness of the test, 2) a greater proportion of examinees were affected by speededness of a test with a 50‐minute time limit than a test with a 55‐ or 60‐minute time limit, and 3) the difference in the proportions of examinees affected by speededness of tests under 55‐ and 60‐minute time limits was small. However, nearly 20 percent of the examinees were affected by speededness after completing 80 percent of the test. In other words, the last 20 percent of the responses of 20 percent of the examinees did not represent their true ability.
APPLICATION OF THE MANTEL‐HAENSZEL PROCEDURE TO COMPLEX SAMPLES OF ITEMS
Authors: Allen Nancy L. , Donoghue John R. 
Published: August 2014

ABSTRACTThis Monte Carlo study examined the effect of complex sampling of items on the measurement of differential item functioning (DIF) using the Mantel‐Haenszel procedure. Data were generated using a 3PL IRT model according to the balanced incomplete block design used in the National Assessment of Educational Progress (NAEP). The length of each block of items and the number of DIF items in the matching variable were varied, as was the difficulty, discrimination, and presence of DIF in the studied item. Block, booklet, pooled booklet and extra‐information analyses were compared to a complete data analysis using the transformed log‐odds on the delta scale. The pooled booklet approach is recommended for use when items are selected for examinees according to a BIB design. This study has implications for DIF analyses of other complex samples of items, such as computer administered testing or another complex assessment design.
A COMPARISON OF THE PERFORMANCE OF GRADUATE AND UNDERGRADUATE SCHOOL APPLICANTS ON THE TEST OF WRITTEN ENGLISH
Authors: Zwick Rebecca , Thayer Dorothy T. 
Published: August 2014

ABSTRACTThe performance of graduate and undergraduate school applicants on the Test of Written English (TWE®) was compared for each of 66 data sets, dating from 1988 to 1993. The analyses compared the average TWE score for graduates and undergraduates after matching examinees on the TOEFL® total score. The main finding was that, for matched examinees, undergraduate TWE means were higher than graduate means in 63 of the 66 data sets. Although these standardized mean differences (SMDs) never exceeded 0.3 of a TWE score point (with standard errors that were typically between 0.01 and 0.02), the results are noteworthy because they give a different picture than do simple comparisons of means for unmatched graduates and undergraduates, which show higher mean TWE scores for graduate applicants in the majority of cases. Of the nine SMDs exceeding 0.2, eight were in Region 1 (Asia and the Pacific) between October 1988 and May 1992. In evaluating these findings, the effects of the examinees' intended field of graduate study were investigated. For groups matched on TOEFL score, applicants to graduate programs in the physical and biological sciences tended to have lower TWE means than undergraduates and graduates in the social sciences. Graduate students in the physical sciences tended to have lower TWE means than matched graduates in the humanities. Graduate‐undergraduate differences in TWE performance may result in part from a greater concentration of graduate applicants in scientific fields. Another hypothesis is that undergraduates are more likely to have recently participated in intense English writing instruction. Region 1 may show larger graduate‐undergraduate differences because of greater demographic disparities in that region between these two groups of examinees.
MODELING THE MIXTURE OF IRT AND PATTERN RESPONSES BY A MODIFIED HYBRID MODEL1
Authors: Yamamoto Kentaro , Everson Howard T. 
Published: August 2014

ABSTRACTThis study demonstrates the utility of a HYBRID psychometric model, which incorporates both item response theoretic and latent class models, for detecting test speededness. The model isolates where in a sequence of test items examinee response patterns shift from ones providing reasonable estimates of ability to those best characterized by a random response pattern. The study applied the HYBRID model to three distinct data sets: (1) simulated data representing the performance of 3,000 examinees on a 70‐item test; (2) data from a statewide field test (n=5997 of a 40 item reading comprehension test with a fixed time limits; and (3) data from a study of urban university students (n=752) who took a similar 40‐item reading comprehension test under varying time limits. The HYBRID model successfully identified “switch points” in examinee's item response patterns in all three data sets. This paper discusses applications of the model used to detect speededness and to provide adjusted estimates of item parameters and examinee abilities.
GENDER DIFFERENCES IN COLLEGE MATHEMATICS GRADES AND SAT‐M SCORES: A REANALYSIS OF WAINER AND STEINBERG
Authors: Bridgeman Brent , Lewis Charles 
Published: August 2014

ABSTRACTWainer and Steinberg (1992) showed that within broad categories of first‐year college mathematics courses (e.g., calculus) men had substantially higher average scores on the mathematics section of the SAT (SAT‐M) than women who earned the same letter grade. In calculus courses, the median difference over the five letter grades was about 38 points on the 200‐800 SAT‐M scale. However, three aspects of their analyses may lead to unwarranted conclusions. First, they focused primarily on differences in SAT‐M scores given course grades when the more important question for admissions officers is the difference in course grades given scores on the predictor. Second, they failed to account for differences among calculus courses. Because different calculus courses may be differentially selected by men and women (e.g., calculus for engineers vs calculus for liberal arts students), and because these courses may have different grading standards, the way Wainer and Steinberg aggregated information without regard to the specific course taken could exaggerate gender differences. The reanalysis, taking distinctions among courses into consideration, suggests that their estimates were too large by about 10 points in calculus courses, although their estimates for pre‐calculus courses were confirmed. Most important, Wainer and Steinberg focused on the use of SAT‐M as an isolated indicator; such use is contrary to professional recommendations. The reanalysis indicated that a more appropriate composite indicator made up of both SAT‐M and high school grade point average demonstrated minuscule gender differences for both calculus and pre‐calculus courses.
CONTROLLING ITEM EXPOSURE CONDITIONAL ON ABILITY IN COMPUTERIZED ADAPTIVE TESTING
Authors: Stocking Martha L. , Lewis Charles 
Published: August 2014

ABSTRACTThe interest in the application of large‐scale adaptive testing for secure tests has served to focus attention on issues that arise when theoretical advances are made operational. Many such issues have more to do with changes in testing conditions rather than testing paradigms. One such issue is that of insuring item and pool security in the continuous testing environment made possible by the computerized administration of a test, as opposed to the more periodic testing environment typically used for linear paper‐and‐pencil tests. In the continuous testing environment of adaptive testing, methods have been developed in the past to use the computer to control the rate at which particular items are exposed to test‐takers. These methods have typically employed randomization schemes, sometimes in reference to a particular target distribution of test‐taker ability. This paper presents a new method of controlling the exposure rate of items conditional on the ability level of an individual test‐taker. The properties of such conditional control on the exposure rates of items, when used in conjunction with a particular adaptive testing algorithm, are explored through five studies with simulated data.
PERSISTENCE IN SCIENCE OF HIGH‐ABILITY MINORITY STUDENTS, PHASE V: COMPREHENSIVE DATA ANALYSIS
Authors: Grandy Jerilee 
Published: August 2014

ABSTRACTConcern over the underrepresentation of minorities in mathematics, science, and engineering led to this longitudinal study begun by Thomas Hilton in 1986 under a grant from the National Science Foundation. The purpose of the study was to investigate why some high‐ability minority students follow through with their plans to become scientists or engineers, while others with the same plans do not.Data came from three sources: (1) 1985 SAT files of a sample of minority students planning to major in math, science, and engineering and scoring above 550 on the SAT mathematics test, (2) a detailed survey questionnaire completed in 1987, and (3) a status survey in 1990.Hilton reported results of the first four phases of the project, including multiple regression analyses. This report of Phase V describes the results of causal modeling using LISREL to determine the direct and indirect effects of gender, SES, high school variables, and college variables on student status five years after high school.Most important to persistence in science and engineering were the type of college attended (2‐year or 4‐year), minority support systems early in college, and commitment to science or engineering by the sophomore year. The role of other variables and their effects on commitment and persistence are discussed in technical detail. Important gender differences are also discussed. The report concludes with a list of practical recommendations for counselors and heads of intervention programs both at the high school and college levels.
AN ANALYSIS OF FACTORS AFFECTING THE DIFFICULTY OF DIALOGUE ITEMS IN TOEFL LISTENING COMPREHENSION
Authors: Nissan Susan , DeVincenzi Felicia , Tang K. Linda 
Published: August 2014

ABSTRACTOne of the item types in the Listening Comprehension Section of the TOEFL® test is the Dialogue. Because the Dialogue item pool needs to have an appropriate balance of items at a range of difficulty levels, test developers have examined items at various difficulty levels in an attempt to identify their features. In this study, the authors created a classification system for certain item features, classified a sample of the current Dialogue item pool, and conducted data analyses in an attempt to characterize the features of easy and difficult Dialogue items. They results of the analyses indicate that, of the features studied, five were significant: the presence of infrequent oral vocabulary, the sentence pattern of the utterances in the stimulus, the presence of negatives in the stimulus, the necessity of making an inference to answer the item, and the roles of the speakers in the stimulus.
SMOOTHED STANDARDIZATION ASSESSMENT OF TESTLET LEVEL DIF ON A MATH FREE‐RESPONSE ITEM TYPE1
Authors: Lyu C. Felicia , Dorans Neil J. , Ramsay James O. 
Published: August 2014

ABSTRACTA smoothed version of standardization, which merges kernel smoothing with the traditional standardization DIF approach, was used to examine DIF for student produced response (SPR) items on the SAT I Math at both the item and testlet levels. This nonparametric technique avoids model misspecification problems. It also has less sampling errors because of smoothing. Results from the smoothed item‐level DIF analysis showed that regular multiple choice items have more variability in DIF values than SPRs. The testlet DIF analysis indicated that White examinees may exceed comparable African American examinees by 14‐19 scaled score units on average. Differences between the smoothed standardization and the traditional standardization are small and most likely due to the use of the true score as a matching variable in the smoothed standardization approach.
A STUDY OF WRITING TASKS ASSIGNED IN ACADEMIC DEGREE PROGRAMS
Authors: Hale Gordon , Taylor Carol , Bridgeman Brent , Carson Joan , Kroll Barbara , Kantor Robert 
Published: August 2014

ABSTRACTWriting tasks assigned in 162 undergraduate and graduate courses in several disciplines at eight universities were collected. Using a sample of the assignments, key dimensions of difference were identified, and a classification scheme based on those dimensions was developed. Application of the classification scheme provided data on the prevalence of various types of assignments and, for essay tasks, showed the degree to which the assignments were characterized by each of several features. Differences in the kinds of writing tasks assigned in different groups of disciplines were examined.
STANDARD ERRORS OF THE KERNEL EQUATING METHODS UNDER THE COMMON‐ITEM DESIGN
Authors: Liou Michelle , Cheng Philip E. , Johnson Eugene G. 
Published: August 2014

ABSTRACTThe present research derives simplified formulae for computing the standard error of the frequency estimation method for equating score distributions that are continuized using a uniform or Gaussian kernel function (Holland, King & Thayer, 1989; Holland & Thayer, 1987). The simplified formulae are applicable to equating both the observed‐ and smoothed‐score distributions (Rosenbaum & Thayer, 1987). Results from two empirical studies indicate that the simplified formulae work reasonably well for samples with moderate sizes, say one thousand examinees.