bluuebunny/crossref_metadata_2025_split huggingface dataset:
COMPUTER ANALYSIS OF THE TOEFL TEST OF WRITTEN ENGLISH
Authors: Frase Lawrence T. , Faletti Joseph , Ginther April , Grant Leslie 
Published: August 2014

ABSTRACTThis project had three main objectives: (1) to establish a database of essays written by different language groups on a variety of topics for the Test of Written English (TWE) that can be used in future research; (2) to summarize, analyze, and compare the linguistic properties of those essays; and (3) to determine how the TWE performance of language groups relates to essay styles. As part of the first objective of this project we created a database of 1,737 essays, a data matrix of essay variables, files containing sorted phrases and vocabularies of different language groups, and files of the common and unique vocabulary items for each pair of language groups.The essay sample consisted of TWE essays from five language groups, including Arabic, Chinese, English (including native‐English and nonnative‐English), and Spanish speakers. Essays from English‐speaking students in the United States were collected and scored to provide a baseline with which to compare essays by students who speak English as a second language (ESL).This report includes the analysis of 106 variables for each essay along with summary analyses, such as correlation, analyses of variance, discriminative analysis, and factor analysis. It also presents information on the accuracy and cost of data entry and on the accuracy of the text analysis programs, as well as extensive data on vocabulary. A program that assesses content was developed for this project.Results show that topic differences affected some essay variables, but these effects were generally felt equally by the different language groups. Factor analysis replicated work showing that major distinguishing features of academic writing include a nominal style, passive constructions, and complexity of sentence structure. Discriminant analysis suggest three features that might distinguish the performance of different language groups — directness, expressiveness, and academic stance. Nevertheless, a linguistic analysis of the accuracy of underlying text analysis programs shows that for some word classes implicated in defining “academic” or “expressive” styles, cautions are needed in interpreting program outputs. Two variables that can be measured unambiguously by computer — number of words and the average length of words — taken together are quite predictive of TWE essay scores of non‐English speakers (multiple R > .8).
CORRESPONDENCES BETWEEN ACT™ AND SAT® I SCORES
Authors: DORANS NEIL J. 
Published: August 2014

ABSTRACTCorrespondences between ACT™ and SAT® I scores are presented from a conceptual framework that distinguishes among three kinds of correspondences, namely, equating, scaling, and prediction. Construct similarity plays an important role in determining the nature and degree of correspondence that can be achieved. This report also maintains that statistical indices in conjunction with rational considerations are needed to determine whether the highest level of correspondence attainable between scores from two admission tests is the conceptual and statistical exchangeability sought by equating, the distributional similarity achieved by scaling, or the association attained by prediction.Relationships among the different scales of the ACT and SAT I are described in the context of the conceptual framework developed herein. Sums of scores, composites of scores, and individual scores are examined. Different types of correspondences between different sets of scores from these two admission tests are amenable to different interpretations.
WRITING ASSESSMENT IN ADMISSION TO HIGHER EDUCATION: REVIEW AND FRAMEWORK
Authors: BRELAND HUNTER M. , BRIDGEMAN BRENT , FOWLES MARY E. 
Published: August 2014

ABSTRACTA comprehensive review was conducted of writing research literature and writing test program activities in a number of testing programs. The review was limited to writing assessments used for admission in higher education. Programs reviewed included ACT, Inc.'s ACT™ program, the California State Universities and Colleges (CSUC) testing program, the College Board's SAT® program, the Graduate Management Admissions Test® (GMAT®) program, the Graduate Record Examinations® (GRE®) test program, the Law School Admission Test® (LSAT®) program, the Medical College Admission Test® (MCAT®) program, and the Test of English as a Foreign Language™ (TOEFL®) testing program. Particular attention was given in the review to writing constructs, fairness and group differences, test reliability, and predictive validity. A number of recommendations are made for research on writing assessment.
EXAMINEES' ATTITUDES ABOUT THE TOEFL‐CBT, POSSIBLE DETERMINANTS, AND RELATIONSHIPS WITH TEST PERFORMANCE
Authors: Stricker Lawrence J. , Wilder Gita Z. 
Published: August 2014

ABSTRACTThe principal aim of this study was to assess examinees' acceptance of the TOEFL‐CBT, and its associations with possible determinants of this acceptance and with test performance. A secondary goal was to evaluate differences in the pattern of results for examinees from different countries. A questionnaire concerning attitudes about the test, familiarity with computers, and other relevant variables was administered to TOEFL‐CBT examinees at large testing centers in Buenos Aires, Cairo, and Frankfurt. Attitudes about the TOEFL‐CBT were moderately positive and had similar patterns of relationships in the three countries: slight or moderate with test performance, moderate with attitudes about admissions tests in general, slight with computer anxiety and test anxiety, and minimal with other variables.
STUMPING E‐RATER: CHALLENGING THE VALIDITY OF AUTOMATED ESSAY SCORING
Authors: Powers Donald E. , Burstein Jill C. , Chodorow Martin , Fowles Mary E. , Kukich Karen 
Published: August 2014

ABSTRACTFor this study, various writing experts were invited to “challenge” e‐rater – an automated essay scorer that relies on natural language processing techniques – by composing essays in response to Graduate Record Examinations (GRE®) Writing Assessment prompts with the intention of undermining its scoring capability. Specifically, using detailed information about e‐rater's approach to essay scoring, writers tried to “trick” the computer‐based system into assigning scores that were higher or lower than deserved. E‐rater's automated scores on these “problem essays” were compared with scores given by two trained, human readers, and the difference between the scores constituted the standard for judging the extent to which e‐rater was fooled. Challengers were differentially successful in writing problematic essays. Expert writers were more successful in tricking e‐rater into assigning scores that were too high than in duping e‐rater into awarding scores that were too low. The study provides information on ways in which e‐rater, and perhaps other automated essay scoring systems, may fail to provide accurate evaluations, if used as the sole method of scoring in high‐stakes assessments. The results suggest possible avenues for improving automated scoring methods.
TEACHER CLASSROOM PRACTICES AND STUDENT PERFORMANCE: HOW SCHOOLS CAN MAKE A DIFFERENCE
Authors: Wenglinsky Harold 
Published: August 2014

ABSTRACTQuantitative studies of school effects have generally supported the notion that the problems of U.S. education lie outside of the school. Yet such studies neglect the primary venue through which students learn, the classroom. The current study explores the link between classroom practices and student academic performance by applying multilevel modeling to the 1996 National Assessment of Educational Progress in mathematics. The study finds that the effects of classroom practices, when added to those of other teacher characteristics, are comparable in size to those of student background, suggesting that teachers can contribute as much to student learning as the students themselves.
SCORING ALTERNATIVES FOR INCOMPLETE COMPUTERIZED ADAPTIVE TESTS1
Authors: Way Walter D. , Gawlick Lisa A. , Eignor Daniel R. 
Published: August 2014

ABSTRACTThe purpose of this study was to investigate an alternative method of scoring incomplete adaptive tests in the context of Educational Testing Service (ETS®) CAT programs. The method under study is based on a procedure originally developed for tests administered as part of the computerized Armed Services Vocational Aptitude Battery. It involves the use of regression‐based “penalty functions” to adjust scores on incomplete adaptive tests based on the number of items answered. The results of the study support the viability of the proposed method in operational settings. Cross‐validation impact analyses indicated that the regression‐based penalty functions that were developed from different CAT item pools resulted in similar levels of score adjustment for the adaptive tests included in the study. Applications of the method to real candidate data indicated that scores for candidates with incomplete tests would be slightly higher if penalty functions—rather than proportional scoring—were used to make the score adjustments.
EFFECTS OF SCREEN SIZE, SCREEN RESOLUTION, AND DISPLAY RATE ON COMPUTER‐BASED TEST PERFORMANCE
Authors: Bridgeman Brent , Lennon Mary Louise , Jackenthal Altamese 
Published: August 2014

ABSTRACTComputer‐based tests administered in established commercial testing centers typically have used monitors of uniform size running at a set resolution. Web‐based delivery of tests promises to greatly expand access, but at the price of less standardization in equipment. The current study evaluated the effects of variations in screen size, screen resolution, and presentation delay on verbal and mathematics scores in a sample of 357 college‐bound high school juniors. The students were randomly assigned to one of six experimental conditions—three screen display conditions crossed with two presentation rate conditions. The three display conditions were: a 17‐inch monitor set to a resolution of 1024 × 768, a 17‐inch monitor set to a resolution of 640 × 480, and a 15‐inch monitor set to a resolution of 640 × 480. Items were presented either with no delay or with a five‐second delay between questions (to emulate a slow Internet connection).No significant effects on math scores were found. Verbal scores were higher, by about a quarter of a standard deviation (28 points on the SAT® scale), with the high‐resolution display.
MONITORING FACULTY CONSULTANT PERFORMANCE IN THE ADVANCED PLACEMENT ENGLISH LITERATURE AND COMPOSITION PROGRAM WITH A MANY‐FACETED RASCH MODEL
Authors: Engelhard George , Myford Carol M. 
Published: August 2014

ABSTRACTThe purpose of this study was to examine, describe, evaluate, and compare the rating behavior of faculty consultants who scored essays written for the Advanced Placement English Literature and Composition (AP®ELC) Exam. Data from the 1999 AP ELC Exam were analyzed using FACETS (Linacre, 1998) and SAS. The faculty consultants were not all interchangeable in terms of the level of severity they exercised. If students' ratings had been adjusted for severity differences, the AP grades of about 30 percent of the students would have been different from the one they received. Almost all the differences were one grade or less. Adjusting ratings for faculty consultant severity differences would not impact some student subgroups more than others.
EXPLORING VARIABILITY IN JUDGING WRITING ABILITY IN A SECOND LANGUAGE: A STUDY OF FOUR EXPERIENCED RATERS OF ESL COMPOSITIONS
Authors: Erdosy M. Usman 
Published: August 2014

ABSTRACTVariability in judgments of ESL compositions is inherent in the view that raters are “readers” with prior experiences. Such a view, however, obliges researchers to understand how personal background and professional experience influence both scoring procedures and scoring criteria. These issues were explored by asking four raters to construct scoring criteria while assessing corpora of 60 TOEFL essays without the aid of a scoring rubric, and to discuss their procedures and criteria in follow‐up interviews. The study revealed key points in the decision‐making process, where raters' behavior diverged, and examined the impact of prior experience on these. The identification of such divergences, and potential explanations for them, were undertaken to lay the foundations for a principled explanation of rater variability.
AN ANALYSIS OF TOEFL CBT WRITING PROMPT DIFFICULTY AND COMPARABILITY FOR DIFFERENT GENDER GROUPS
Authors: Breland Hunter , Lee Yong‐Won , Najarian Michelle , Muraki Eiji 
Published: August 2014

ABSTRACTThis investigation of the comparability of writing assessment prompts was conducted in two phases. In an exploratory Phase I, 47 writing prompts administered in the computer‐based Test of English as a Foreign Language™ (TOEFL® CBT) from July through December 1998 were examined. Logistic regression procedures were used to estimate prompt difficulty and gender effects. A panel of experts reviewed selected prompts, and a taxonomy of prompt characteristics was developed and related to prompt difficulty and gender differences. In Phase II, 87 prompts administered from July 1998 through March 2000 were analyzed. All of the prompts used in Phase I, together with 40 new prompts, were analyzed using the larger Phase II database. Recommendations are made for statistical quality control procedures to identify less comparable prompts.
TRANSFER BETWEEN VARIANTS OF QUANTITATIVE ITEMS
Authors: Morley Mary E. , Bridgeman Brent , Lawless René R. 
Published: August 2014

ABSTRACTThis study investigated the transfer of solution strategies between close variants of quantitative reasoning questions. Pre‐ and posttests were obtained from 406 college undergraduates, all of whom took the same posttest; pretests varied such that one group of participants saw close variants of one set of posttest items while other groups saw close variants of other sets. Some participants also saw items that looked similar to some posttest questions but were mathematically different. Between pre‐ and posttests, some participants viewed solution rationales explaining their incorrect answers. Students at all ability levels performed better on close variants, suggesting there was some positive transfer, but the presence of appearance variants interfered with this transfer. This study suggests an important first step toward understanding how tests and item vats can be designed to capitalize on the extent to which students set up test‐taking strategies.
USING SCORE EQUITY ASSESSMENT TO EVALUATE THE EQUATABILITY OF THE HARDEST HALF OF A TEST TO THE TOTAL TEST
Authors: Dorans Neil J. , Zeller Karin 
Published: August 2014

ABSTRACTIn an article published in the spring 2003 issue of Harvard Educational Review, Roy Freedle stated that the SAT® is both culturally and statistically biased. Freedle proposed a solution to this bias, which involved using a half‐test made up of the most difficult items culled from complete SAT examination. His claims, which garnered national attention, were based on serious errors in his analysis. In Dorans and Zeller (2004), we demonstrated that the effects Freedle reported are reduced substantially when the data are analyzed correctly. Here, we describe a sound way of assessing whether the current SAT scoring procedure is fair, and we examine what happens when we subject the Freedle's hard half‐test to a score‐equity assessment.
AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0
Authors: Attali Yigal , Burstein Jill 
Published: August 2014

ABSTRACTThe e‐rater® system has been used by ETS for automated essay scoring since 1999. This paper describes a new version of e‐rater (v.2.0) that differs from the previous one (v.1.3) with regard to the feature set and model building approach. The paper describes the new version, compares the new and previous versions in terms of performance, and presents evidence on the validity and reliability of scores produced by the new version.
INVESTIGATING THE UTILITY OF ANALYTIC SCORING FOR THE TOEFL ACADEMIC SPEAKING TEST (TAST)
Authors: Xi Xiaoming , Mollaun Pam 
Published: August 2014

ABSTRACTThis study explores the utility of analytic scoring for the TOEFL® Academic Speaking Test (TAST) in providing useful and reliable diagnostic information in three aspects of candidates' performance: delivery, language use, and topic development.G studies were used to investigate the dependability of the analytic scores, the distinctness of the analytic dimensions, and the variability of analytic score profiles. Raters' perceptions of dimension separability were also obtained.Based on the phi coefficients and standard errors of measurement (SEMs), the dependability of analytic scores averaged across six tasks and double ratings was acceptable for both operational and practice settings. However, scores averaged across two tasks and double ratings were not reliable enough for operational use.Correlations among the analytic scores by task were high, but those between delivery and topic development were lower, and these results were corroborated by raters' perceptions. When averaged across tasks or task types (two or more tasks), correlations among the analytic scores were very high, and the profiles of scores were flat.The utility of analytic scoring is discussed, and both score dependability and whether analytic scores provide diagnostic information beyond that provided by holistic scores are considered.
LINKING COMPETENCIES IN EDUCATIONAL SETTINGS AND MEASURING GROWTH
Authors: von Davier Alina A. , Carstensen Claus H. , von Davier Matthias 
Published: August 2014

ABSTRACTMeasuring and linking competencies require special instruments, special data collection designs, and special statistical models. The measurement instruments are tests or tests forms, which can be used in the following situations: The same test can be given repeatedly; two or more parallel tests forms (i.e., forms intended to be similar in difficulty and content and that still need to equated) can be given at different time points; or two or more test forms that may be less parallel to various degrees can be given at different points in time. In some circumstances, the goal of the analysis is to make the scores of parallel test forms comparable across different time points and different samples of relatively similar ability (horizontal equating). In other situations, we aim at the comparability of scores of tests forms that are not parallel, and although they are intended to measure the same competencies, they are of different difficulties and are taken by samples that show large differences in ability (vertical linking). In other cases, we want to evaluate the change in competencies over time (with or without covariates) for the same individuals measured by the same instrument or by different instruments (longitudinal linking). This paper will briefly discuss a variety of techniques for relating scale scores from different data collections points and will then discuss models for measuring growth. Each of these areas is a large field in itself, and each has a potential strong impact on educational policies such as the No Child Left Behind Act (for example, the vertical linking of state or national assessments and longitudinal studies are a potential basis for informative analyses for the policy makers), on the life of students and parents (equating of achievement tests), or on the life of professionals (equating of licensure tests). In these times when more and more standardized testing is used for assessing competencies in different domains nationally and internationally, we are also discovering more challenges in ensuring that the process and the results are fair and accurate. In turn, these challenges and these new social implications open the door toward more research in support of fair assessments, both in improving upon the test construction process and in advancing the statistical methods involved.
WEIGHTING PROCEDURES AND THE CLUSTER FORMING ALGORITHM FOR DELETE‐K JACKKNIFE VARIANCE ESTIMATION FOR INSTITUTIONAL SURVEYS
Authors: Qian Jiahe 
Published: August 2014

ABSTRACTWeighting and variance estimation are two statistical issues involved in survey data analysis for large‐scale assessment programs such as the Higher Education Information and Communication Technology (ICT) Literacy Assessment. Because survey data are always acquired by probability sampling, to draw unbiased or almost unbiased inferences for the populations, weights are required in making use of estimators such as a Horvitz‐Thompson type. Variance estimation provides the basis for reporting errors. The weighting procedure generates weights based on statistical principles that are consistent with the sampling design. The estimation of the variance from survey data uses the delete‐k jackknife resampling replicate (JRR) approach, which can be adapted for variant institutional sampling designs and for dissimilarity in institute conditions. To form clusters of k cases, a merge‐dilute algorithm is proposed. The algorithm merges the cases of different groups into a queue and then allocates the cases of the queue to form homogeneous clusters of required sizes. The new algorithm is applied to the ICT sample from an institute taking the 2004 fall trial assessment.
NOTETAKING STRATEGIES AND THEIR RELATIONSHIP TO PERFORMANCE ON LISTENING COMPREHENSION AND COMMUNICATIVE ASSESSMENT TASKS
Authors: Carrell Patricia L. 
Published: August 2014

ABSTRACTUtilizing a pre‐ and posttest research design, with an instructional intervention of good practices in notetaking, the notes taken by examinees during a computer‐based listening comprehension test prior to and following the instructional intervention were examined for particular notetaking strategies. Questionnaires probed perceptions of the frequency of use and helpfulness of the notetaking strategies. Also investigated were relationships between the strategies and performance on a multiple‐choice listening comprehension measure and integrated listening/writing and listening/speaking tasks and whether the brief instructional intervention affected examinees' notes and performance on the three tasks.Results showed that examinees made little use of efficiency and marked organizational notetaking strategies. While the intervention had no effect on examinees' notetaking strategies or on their task performance, students' posttest notetaking strategies and task performance were significantly related to their pretest notetaking strategies and task performance. Moreover, notetaking and test performance were moderately related: Two notetaking strategies that most consistently related to performance on all three tasks were the number of content words in the notes and the number of test answers recorded in the notes. Efficiency strategies represented a cognitively complex category worthy of further investigation.
AN INITIAL INVESTIGATION OF A MODIFIED PROCEDURE FOR PARALLEL ANALYSIS
Authors: Liu Ou Lydia , Rijmen Frank , Kong Nan 
Published: August 2014

ABSTRACTParallel analysis has been well documented to be an effective and accurate method for determining the number of factors to retain in exploratory factor analysis. Despite its theoretical and empirical advantages, the popularity of parallel analysis has been thwarted by its limited access in statistical software such as SPSS and SAS, especially in software that analyzes ordinal data. Among the few commonly used procedures, the Hayton, Allen, and Scarpello (2004) procedure requires manually computing the mean of eigenvalues from at least 50 replications. The O'Connor (2000) procedure overcomes that limitation, yet it has difficulties dealing with random missing data. To address these technical issues of parallel analysis for ordinal variables, we adapted and modified the O'Connor procedure to provide an alternative that best approximates the ordinal data by factoring in the frequency distributions of the variables (e.g., the number of response categories and the frequency of each response category per variable). Our procedure has a slightly different theoretical rationale from O'Connor's as well as a practical advantage in dealing with missing data.