bluuebunny/crossref_metadata_2025 huggingface dataset:
DEVELOPMENT, SELECTION AND VALIDATION OF A SET OF COGNITIVE AND LINGUISTIC ATTRIBUTES FOR THE SAT I VERBAL: ANALOGY SECTION1
Authors: Buck Gary , VanEssen Tom , Tatsuoka Kikumi , Kostin Irene , Lutz Donna , Phelps Matthew
Published: August 2014

The purpose of this study is to use the rule space procedure to develop a general set of cognitive and linguistic attributes that account for performance on the Analogy Section of the SAT I Verbal. The overall research strategy was to develop a set of cognitive and linguistic attributes on one form of the Analogy Section, the experimental form, and then to examine the generalizability of these by applying the same set of attributes to a different form of the test, the generalizability form. Results produced a set 13 cognitive and linguistic attributes, with 6 interaction attributes, that were able to classify 97% of test‐takers into their latent knowledge states on the experimental form; the same set of attributes were able to classify 90% of test takers into their latent knowledge states on the generalizability form. The analysis provides a strong indication of the important determinants of success on the SAT Analogy task, and provides evidence that these are relatively stable across forms.
GRE ANALYTICAL REASONING ITEM STATISTICS PREDICTION STUDY
Authors: Boldt R. F.
Published: August 2014

Chalifour and Powers (1989) noted that the ability to predict item statistics might be used to reduce the volume of item pretesting (Mislevy, Sheehan, & Wingersky, 1993). It could also lead to better control of test statistics, such as item difficulty distributions, and to improved test specifications. Research, such as the previously cited study, is needed to examine the prospects for attaining these benefits. That study examined prediction of GRE analytical reasoning item statistics. Linear regression was used by these authors, but they surmised that non‐linear techniques might provide somewhat better prediction of item statistics.

Chalifour and Powers had amassed item statistics on a very large sample of analytical reasoning items. That sample was used in the present study.

For the present study, predictions were generated using a type of neural net, a technique that can accommodate a wide variety of non‐linear relationships, though at a cost of requiring the calculation of many constants in the prediction function. This technique did indeed provide more accurate predictions of item difficulty and item discrimination in an estimation sample. However, when the functions developed in the estimation sample were cross validated in a fresh sample, the advantages noted in the estimation sample disappeared. Only by including expert judgements of items difficulty with other predictors was the accuracy of prediction improved.

The variables that produced the above results had been selected by Chalifour and Powers (1989) for their efficacy in linear prediction. The study reported here used a “genetic algorithm”, which provides a quasi‐random search of the predictor set to find an optimal set of predictors. The search used the validity of neural nets computed during operation of the algorithm to evaluate the efficacy of prediction. Hence, this technique sought variables sets that were optimal when a neural net was to be used.

The genetic algorithm accomplished some improved prediction of item difficulties in the estimation, but no improvement with regard to discrimination. When the variable sets and the related nets were evaluated in the validation sample, any advantage gained by the search was lost. The validities of predictions in the validation sample were approximately equal whether or not the genetic algorithm had been used in developing the prediction functions.

Examination of the root‐mean squares of the discrepancies between predicted and actual values for item difficulties (and discriminations) revealed no advantage for the neural net. This examination used the validation sample.

In sum, application of the complex and computer‐intensive neural nets and genetic algorithms revealed no advantage over linear methods for predicting item difficulty and item discrimination statistics for GRE analytical reasoning items.
COMPUTER ANALYSIS OF THE TOEFL TEST OF WRITTEN ENGLISH
Authors: Frase Lawrence T. , Faletti Joseph , Ginther April , Grant Leslie
Published: August 2014

This project had three main objectives: (1) to establish a database of essays written by different language groups on a variety of topics for the Test of Written English (TWE) that can be used in future research; (2) to summarize, analyze, and compare the linguistic properties of those essays; and (3) to determine how the TWE performance of language groups relates to essay styles. As part of the first objective of this project we created a database of 1,737 essays, a data matrix of essay variables, files containing sorted phrases and vocabularies of different language groups, and files of the common and unique vocabulary items for each pair of language groups.

The essay sample consisted of TWE essays from five language groups, including Arabic, Chinese, English (including native‐English and nonnative‐English), and Spanish speakers. Essays from English‐speaking students in the United States were collected and scored to provide a baseline with which to compare essays by students who speak English as a second language (ESL).

This report includes the analysis of 106 variables for each essay along with summary analyses, such as correlation, analyses of variance, discriminative analysis, and factor analysis. It also presents information on the accuracy and cost of data entry and on the accuracy of the text analysis programs, as well as extensive data on vocabulary. A program that assesses content was developed for this project.

Results show that topic differences affected some essay variables, but these effects were generally felt equally by the different language groups. Factor analysis replicated work showing that major distinguishing features of academic writing include a nominal style, passive constructions, and complexity of sentence structure. Discriminant analysis suggest three features that might distinguish the performance of different language groups — directness, expressiveness, and academic stance. Nevertheless, a linguistic analysis of the accuracy of underlying text analysis programs shows that for some word classes implicated in defining “academic” or “expressive” styles, cautions are needed in interpreting program outputs. Two variables that can be measured unambiguously by computer — number of words and the average length of words — taken together are quite predictive of TWE essay scores of non‐English speakers (multiple R > .8).
CORRESPONDENCES BETWEEN ACT™ AND SAT® I SCORES
Authors: DORANS NEIL J.
Published: August 2014

Correspondences between ACT™ and SAT® I scores are presented from a conceptual framework that distinguishes among three kinds of correspondences, namely, equating, scaling, and prediction. Construct similarity plays an important role in determining the nature and degree of correspondence that can be achieved. This report also maintains that statistical indices in conjunction with rational considerations are needed to determine whether the highest level of correspondence attainable between scores from two admission tests is the conceptual and statistical exchangeability sought by equating, the distributional similarity achieved by scaling, or the association attained by prediction.

Relationships among the different scales of the ACT and SAT I are described in the context of the conceptual framework developed herein. Sums of scores, composites of scores, and individual scores are examined. Different types of correspondences between different sets of scores from these two admission tests are amenable to different interpretations.
WRITING ASSESSMENT IN ADMISSION TO HIGHER EDUCATION: REVIEW AND FRAMEWORK
Authors: BRELAND HUNTER M. , BRIDGEMAN BRENT , FOWLES MARY E.
Published: August 2014

A comprehensive review was conducted of writing research literature and writing test program activities in a number of testing programs. The review was limited to writing assessments used for admission in higher education. Programs reviewed included ACT, Inc.'s ACT™ program, the California State Universities and Colleges (CSUC) testing program, the College Board's SAT® program, the Graduate Management Admissions Test® (GMAT®) program, the Graduate Record Examinations® (GRE®) test program, the Law School Admission Test® (LSAT®) program, the Medical College Admission Test® (MCAT®) program, and the Test of English as a Foreign Language™ (TOEFL®) testing program. Particular attention was given in the review to writing constructs, fairness and group differences, test reliability, and predictive validity. A number of recommendations are made for research on writing assessment.
EXAMINEES' ATTITUDES ABOUT THE TOEFL‐CBT, POSSIBLE DETERMINANTS, AND RELATIONSHIPS WITH TEST PERFORMANCE
Authors: Stricker Lawrence J. , Wilder Gita Z.
Published: August 2014

The principal aim of this study was to assess examinees' acceptance of the TOEFL‐CBT, and its associations with possible determinants of this acceptance and with test performance. A secondary goal was to evaluate differences in the pattern of results for examinees from different countries. A questionnaire concerning attitudes about the test, familiarity with computers, and other relevant variables was administered to TOEFL‐CBT examinees at large testing centers in Buenos Aires, Cairo, and Frankfurt. Attitudes about the TOEFL‐CBT were moderately positive and had similar patterns of relationships in the three countries: slight or moderate with test performance, moderate with attitudes about admissions tests in general, slight with computer anxiety and test anxiety, and minimal with other variables.
STUMPING E‐RATER: CHALLENGING THE VALIDITY OF AUTOMATED ESSAY SCORING
Authors: Powers Donald E. , Burstein Jill C. , Chodorow Martin , Fowles Mary E. , Kukich Karen
Published: August 2014

For this study, various writing experts were invited to “challenge” e‐rater – an automated essay scorer that relies on natural language processing techniques – by composing essays in response to Graduate Record Examinations (GRE®) Writing Assessment prompts with the intention of undermining its scoring capability. Specifically, using detailed information about e‐rater's approach to essay scoring, writers tried to “trick” the computer‐based system into assigning scores that were higher or lower than deserved. E‐rater's automated scores on these “problem essays” were compared with scores given by two trained, human readers, and the difference between the scores constituted the standard for judging the extent to which e‐rater was fooled. Challengers were differentially successful in writing problematic essays. Expert writers were more successful in tricking e‐rater into assigning scores that were too high than in duping e‐rater into awarding scores that were too low. The study provides information on ways in which e‐rater, and perhaps other automated essay scoring systems, may fail to provide accurate evaluations, if used as the sole method of scoring in high‐stakes assessments. The results suggest possible avenues for improving automated scoring methods.
TEACHER CLASSROOM PRACTICES AND STUDENT PERFORMANCE: HOW SCHOOLS CAN MAKE A DIFFERENCE
Authors: Wenglinsky Harold
Published: August 2014

Quantitative studies of school effects have generally supported the notion that the problems of U.S. education lie outside of the school. Yet such studies neglect the primary venue through which students learn, the classroom. The current study explores the link between classroom practices and student academic performance by applying multilevel modeling to the 1996 National Assessment of Educational Progress in mathematics. The study finds that the effects of classroom practices, when added to those of other teacher characteristics, are comparable in size to those of student background, suggesting that teachers can contribute as much to student learning as the students themselves.
SCORING ALTERNATIVES FOR INCOMPLETE COMPUTERIZED ADAPTIVE TESTS1
Authors: Way Walter D. , Gawlick Lisa A. , Eignor Daniel R.
Published: August 2014

The purpose of this study was to investigate an alternative method of scoring incomplete adaptive tests in the context of Educational Testing Service (ETS®) CAT programs. The method under study is based on a procedure originally developed for tests administered as part of the computerized Armed Services Vocational Aptitude Battery. It involves the use of regression‐based “penalty functions” to adjust scores on incomplete adaptive tests based on the number of items answered. The results of the study support the viability of the proposed method in operational settings. Cross‐validation impact analyses indicated that the regression‐based penalty functions that were developed from different CAT item pools resulted in similar levels of score adjustment for the adaptive tests included in the study. Applications of the method to real candidate data indicated that scores for candidates with incomplete tests would be slightly higher if penalty functions—rather than proportional scoring—were used to make the score adjustments.
EFFECTS OF SCREEN SIZE, SCREEN RESOLUTION, AND DISPLAY RATE ON COMPUTER‐BASED TEST PERFORMANCE
Authors: Bridgeman Brent , Lennon Mary Louise , Jackenthal Altamese
Published: August 2014

Computer‐based tests administered in established commercial testing centers typically have used monitors of uniform size running at a set resolution. Web‐based delivery of tests promises to greatly expand access, but at the price of less standardization in equipment. The current study evaluated the effects of variations in screen size, screen resolution, and presentation delay on verbal and mathematics scores in a sample of 357 college‐bound high school juniors. The students were randomly assigned to one of six experimental conditions—three screen display conditions crossed with two presentation rate conditions. The three display conditions were: a 17‐inch monitor set to a resolution of 1024 × 768, a 17‐inch monitor set to a resolution of 640 × 480, and a 15‐inch monitor set to a resolution of 640 × 480. Items were presented either with no delay or with a five‐second delay between questions (to emulate a slow Internet connection).

No significant effects on math scores were found. Verbal scores were higher, by about a quarter of a standard deviation (28 points on the SAT® scale), with the high‐resolution display.
MONITORING FACULTY CONSULTANT PERFORMANCE IN THE ADVANCED PLACEMENT ENGLISH LITERATURE AND COMPOSITION PROGRAM WITH A MANY‐FACETED RASCH MODEL
Authors: Engelhard George , Myford Carol M.
Published: August 2014

The purpose of this study was to examine, describe, evaluate, and compare the rating behavior of faculty consultants who scored essays written for the Advanced Placement English Literature and Composition (AP®ELC) Exam. Data from the 1999 AP ELC Exam were analyzed using FACETS (Linacre, 1998) and SAS. The faculty consultants were not all interchangeable in terms of the level of severity they exercised. If students' ratings had been adjusted for severity differences, the AP grades of about 30 percent of the students would have been different from the one they received. Almost all the differences were one grade or less. Adjusting ratings for faculty consultant severity differences would not impact some student subgroups more than others.
EXPLORING VARIABILITY IN JUDGING WRITING ABILITY IN A SECOND LANGUAGE: A STUDY OF FOUR EXPERIENCED RATERS OF ESL COMPOSITIONS
Authors: Erdosy M. Usman
Published: August 2014

Variability in judgments of ESL compositions is inherent in the view that raters are “readers” with prior experiences. Such a view, however, obliges researchers to understand how personal background and professional experience influence both scoring procedures and scoring criteria. These issues were explored by asking four raters to construct scoring criteria while assessing corpora of 60 TOEFL essays without the aid of a scoring rubric, and to discuss their procedures and criteria in follow‐up interviews. The study revealed key points in the decision‐making process, where raters' behavior diverged, and examined the impact of prior experience on these. The identification of such divergences, and potential explanations for them, were undertaken to lay the foundations for a principled explanation of rater variability.
AN ANALYSIS OF TOEFL CBT WRITING PROMPT DIFFICULTY AND COMPARABILITY FOR DIFFERENT GENDER GROUPS
Authors: Breland Hunter , Lee Yong‐Won , Najarian Michelle , Muraki Eiji
Published: August 2014

This investigation of the comparability of writing assessment prompts was conducted in two phases. In an exploratory Phase I, 47 writing prompts administered in the computer‐based Test of English as a Foreign Language™ (TOEFL® CBT) from July through December 1998 were examined. Logistic regression procedures were used to estimate prompt difficulty and gender effects. A panel of experts reviewed selected prompts, and a taxonomy of prompt characteristics was developed and related to prompt difficulty and gender differences. In Phase II, 87 prompts administered from July 1998 through March 2000 were analyzed. All of the prompts used in Phase I, together with 40 new prompts, were analyzed using the larger Phase II database. Recommendations are made for statistical quality control procedures to identify less comparable prompts.
TRANSFER BETWEEN VARIANTS OF QUANTITATIVE ITEMS
Authors: Morley Mary E. , Bridgeman Brent , Lawless René R.
Published: August 2014

This study investigated the transfer of solution strategies between close variants of quantitative reasoning questions. Pre‐ and posttests were obtained from 406 college undergraduates, all of whom took the same posttest; pretests varied such that one group of participants saw close variants of one set of posttest items while other groups saw close variants of other sets. Some participants also saw items that looked similar to some posttest questions but were mathematically different. Between pre‐ and posttests, some participants viewed solution rationales explaining their incorrect answers. Students at all ability levels performed better on close variants, suggesting there was some positive transfer, but the presence of appearance variants interfered with this transfer. This study suggests an important first step toward understanding how tests and item vats can be designed to capitalize on the extent to which students set up test‐taking strategies.
USING SCORE EQUITY ASSESSMENT TO EVALUATE THE EQUATABILITY OF THE HARDEST HALF OF A TEST TO THE TOTAL TEST
Authors: Dorans Neil J. , Zeller Karin
Published: August 2014

In an article published in the spring 2003 issue of Harvard Educational Review, Roy Freedle stated that the SAT® is both culturally and statistically biased. Freedle proposed a solution to this bias, which involved using a half‐test made up of the most difficult items culled from complete SAT examination. His claims, which garnered national attention, were based on serious errors in his analysis. In Dorans and Zeller (2004), we demonstrated that the effects Freedle reported are reduced substantially when the data are analyzed correctly. Here, we describe a sound way of assessing whether the current SAT scoring procedure is fair, and we examine what happens when we subject the Freedle's hard half‐test to a score‐equity assessment.
AUTOMATED ESSAY SCORING WITH E‐RATER® V.2.0
Authors: Attali Yigal , Burstein Jill
Published: August 2014

The e‐rater® system has been used by ETS for automated essay scoring since 1999. This paper describes a new version of e‐rater (v.2.0) that differs from the previous one (v.1.3) with regard to the feature set and model building approach. The paper describes the new version, compares the new and previous versions in terms of performance, and presents evidence on the validity and reliability of scores produced by the new version.
INVESTIGATING THE UTILITY OF ANALYTIC SCORING FOR THE TOEFL ACADEMIC SPEAKING TEST (TAST)
Authors: Xi Xiaoming , Mollaun Pam
Published: August 2014

This study explores the utility of analytic scoring for the TOEFL® Academic Speaking Test (TAST) in providing useful and reliable diagnostic information in three aspects of candidates' performance: delivery, language use, and topic development.

G studies were used to investigate the dependability of the analytic scores, the distinctness of the analytic dimensions, and the variability of analytic score profiles. Raters' perceptions of dimension separability were also obtained.

Based on the phi coefficients and standard errors of measurement (SEMs), the dependability of analytic scores averaged across six tasks and double ratings was acceptable for both operational and practice settings. However, scores averaged across two tasks and double ratings were not reliable enough for operational use.

Correlations among the analytic scores by task were high, but those between delivery and topic development were lower, and these results were corroborated by raters' perceptions. When averaged across tasks or task types (two or more tasks), correlations among the analytic scores were very high, and the profiles of scores were flat.

The utility of analytic scoring is discussed, and both score dependability and whether analytic scores provide diagnostic information beyond that provided by holistic scores are considered.
LINKING COMPETENCIES IN EDUCATIONAL SETTINGS AND MEASURING GROWTH
Authors: von Davier Alina A. , Carstensen Claus H. , von Davier Matthias
Published: August 2014

Measuring and linking competencies require special instruments, special data collection designs, and special statistical models. The measurement instruments are tests or tests forms, which can be used in the following situations: The same test can be given repeatedly; two or more parallel tests forms (i.e., forms intended to be similar in difficulty and content and that still need to equated) can be given at different time points; or two or more test forms that may be less parallel to various degrees can be given at different points in time. In some circumstances, the goal of the analysis is to make the scores of parallel test forms comparable across different time points and different samples of relatively similar ability (horizontal equating). In other situations, we aim at the comparability of scores of tests forms that are not parallel, and although they are intended to measure the same competencies, they are of different difficulties and are taken by samples that show large differences in ability (vertical linking). In other cases, we want to evaluate the change in competencies over time (with or without covariates) for the same individuals measured by the same instrument or by different instruments (longitudinal linking). This paper will briefly discuss a variety of techniques for relating scale scores from different data collections points and will then discuss models for measuring growth. Each of these areas is a large field in itself, and each has a potential strong impact on educational policies such as the No Child Left Behind Act (for example, the vertical linking of state or national assessments and longitudinal studies are a potential basis for informative analyses for the policy makers), on the life of students and parents (equating of achievement tests), or on the life of professionals (equating of licensure tests). In these times when more and more standardized testing is used for assessing competencies in different domains nationally and internationally, we are also discovering more challenges in ensuring that the process and the results are fair and accurate. In turn, these challenges and these new social implications open the door toward more research in support of fair assessments, both in improving upon the test construction process and in advancing the statistical methods involved.
WEIGHTING PROCEDURES AND THE CLUSTER FORMING ALGORITHM FOR DELETE‐K JACKKNIFE VARIANCE ESTIMATION FOR INSTITUTIONAL SURVEYS
Authors: Qian Jiahe
Published: August 2014

Weighting and variance estimation are two statistical issues involved in survey data analysis for large‐scale assessment programs such as the Higher Education Information and Communication Technology (ICT) Literacy Assessment. Because survey data are always acquired by probability sampling, to draw unbiased or almost unbiased inferences for the populations, weights are required in making use of estimators such as a Horvitz‐Thompson type. Variance estimation provides the basis for reporting errors. The weighting procedure generates weights based on statistical principles that are consistent with the sampling design. The estimation of the variance from survey data uses the delete‐k jackknife resampling replicate (JRR) approach, which can be adapted for variant institutional sampling designs and for dissimilarity in institute conditions. To form clusters of k cases, a merge‐dilute algorithm is proposed. The algorithm merges the cases of different groups into a queue and then allocates the cases of the queue to form homogeneous clusters of required sizes. The new algorithm is applied to the ICT sample from an institute taking the 2004 fall trial assessment.
NOTETAKING STRATEGIES AND THEIR RELATIONSHIP TO PERFORMANCE ON LISTENING COMPREHENSION AND COMMUNICATIVE ASSESSMENT TASKS
Authors: Carrell Patricia L.
Published: August 2014

Utilizing a pre‐ and posttest research design, with an instructional intervention of good practices in notetaking, the notes taken by examinees during a computer‐based listening comprehension test prior to and following the instructional intervention were examined for particular notetaking strategies. Questionnaires probed perceptions of the frequency of use and helpfulness of the notetaking strategies. Also investigated were relationships between the strategies and performance on a multiple‐choice listening comprehension measure and integrated listening/writing and listening/speaking tasks and whether the brief instructional intervention affected examinees' notes and performance on the three tasks.

Results showed that examinees made little use of efficiency and marked organizational notetaking strategies. While the intervention had no effect on examinees' notetaking strategies or on their task performance, students' posttest notetaking strategies and task performance were significantly related to their pretest notetaking strategies and task performance. Moreover, notetaking and test performance were moderately related: Two notetaking strategies that most consistently related to performance on all three tasks were the number of content words in the notes and the number of test answers recorded in the notes. Efficiency strategies represented a cognitively complex category worthy of further investigation.
AN INITIAL INVESTIGATION OF A MODIFIED PROCEDURE FOR PARALLEL ANALYSIS
Authors: Liu Ou Lydia , Rijmen Frank , Kong Nan
Published: August 2014

Parallel analysis has been well documented to be an effective and accurate method for determining the number of factors to retain in exploratory factor analysis. Despite its theoretical and empirical advantages, the popularity of parallel analysis has been thwarted by its limited access in statistical software such as SPSS and SAS, especially in software that analyzes ordinal data. Among the few commonly used procedures, the Hayton, Allen, and Scarpello (2004) procedure requires manually computing the mean of eigenvalues from at least 50 replications. The O'Connor (2000) procedure overcomes that limitation, yet it has difficulties dealing with random missing data. To address these technical issues of parallel analysis for ordinal variables, we adapted and modified the O'Connor procedure to provide an alternative that best approximates the ordinal data by factoring in the frequency distributions of the variables (e.g., the number of response categories and the frequency of each response category per variable). Our procedure has a slightly different theoretical rationale from O'Connor's as well as a practical advantage in dealing with missing data.
EXAMINING TEST ITEMS FOR DIFFERENTIAL DISTRACTOR FUNCTIONING AMONG STUDENTS WITH LEARNING DISABILITIES
Authors: Middleton Kyndra , Laitusis Cara Cahalan
Published: August 2014

This study examined whether distractor choices functioned differently for students without learning disabilities than they functioned for students with learning disabilities who received no accommodation, students with learning disabilities who received a read‐aloud accommodation, and students with learning disabilities who received some form of accommodation other than read‐aloud. The study's purpose was twofold: (a) to examine the results of the DDF analysis to determine whether the distractors functioned differently for the various groups of students and (b) to aid in determining whether the test may be modified for those students with learning disabilities by removing a distractor while maintaining adequate test validity and information.
EVALUATING THE EFFECTIVENESS OF A FULL‐POPULATION ESTIMATION METHOD
Authors: Braun Henry , Zhang Jinming , Vezzu Sailesh
Published: August 2014

At present, although the percentages of students with disabilities (SDs) and/or students who are English language learners (ELL) excluded from a NAEP administration are reported, no statistical adjustment is made for these excluded students in the calculation of NAEP results. However, the exclusion rates for both SD and ELL students vary substantially across jurisdictions at a given administration, and, in some cases, have changed substantially over time within a jurisdiction. Consequently, comparisons of performance based on reported NAEP scores may indeed be biased by differential exclusion and identification practices.

Using only NAEP data, this report investigates plausible explanations for the observed heterogeneity among jurisdictions in exclusion rates. It also examines the operating characteristics of a particular class of methods that carry out statistical adjustments to NAEP's reported scores to address the possible bias due to differential exclusion rates. The final results of such adjustments are termed full‐population estimates (FPEs). The conclusions are that there is both a strong likelihood of bias and that neither the current NAEP procedure nor the FPE methodologies constitutes an ideal solution. The former because it assumes that all excluded students could not meaningfully participate in NAEP, and the latter because they implicitly assume that all students could obtain a proper NAEP score.
COMPARISON OF MULTIDIMENSIONAL ITEM RESPONSE MODELS: MULTIVARIATE NORMAL ABILITY DISTRIBUTIONS VERSUS MULTIVARIATE POLYTOMOUS ABILITY DISTRIBUTIONS
Authors: Haberman Shelby J. , von Davier Matthias , Lee Yi‐Hsuan
Published: August 2014

Multidimensional item response models can be based on multivariate normal ability distributions or on multivariate polytomous ability distributions. For the case of simple structure in which each item corresponds to a unique dimension of the ability vector, some applications of the two‐parameter logistic model to empirical data are employed to illustrate how, at least for the example under study, comparable results can be achieved with either approach. Comparability involves quality of model fit as well as similarity in terms of parameter estimates and computational time required. In both cases, numerical work can be performed quite efficiently. In the case of the multivariate normal ability distribution, multivariate adaptive Gauss‐Hermite quadrature can be employed to greatly reduce computational labor. In the case of a polytomous ability distribution, use of log‐linear models permits efficient computations.
AN EVALUATION OF STATISTICAL STRATEGIES FOR MAKING EQUATING FUNCTION SELECTIONS
Authors: Moses Tim
Published: August 2014

Nine statistical strategies for selecting equating functions in an equivalent groups design were evaluated. The strategies of interest were likelihood ratio chi‐square tests, regression tests, Kolmogorov‐Smirnov tests, and significance tests for equated score differences. The most accurate strategies in the study were the likelihood ratio tests and the significance tests for equated score differences.
COGNITIVELY BASED ASSESSMENT OF, FOR, AND AS LEARNING: A FRAMEWORK FOR ASSESSING READING COMPETENCY
Authors: O'Reilly Tenaha , Sheehan Kathleen M.
Published: August 2014

This paper presents the rationale and research base for a reading competency model designed to guide the development of cognitively based assessment of reading comprehension. The model was developed from a detailed review of the cognitive research on reading and learning and a review of state standards for language arts. A survey of the literature revealed three key areas of reading competency: prerequisite reading skill, model building skill, and applied comprehension skill. Prerequisite reading skill is the ability to read text accurately and fluently. Model building skill is the ability to construct meaning from either decoded text or spoken language. Applied comprehension skill is the ability to use and apply the information contained in text for some particular purpose. The framework is discussed in terms of 7 key principles that have implications for the design of a modern assessment of reading.
HOW DO RATERS FROM INDIA PERFORM IN SCORING THE TOEFL IBT™ SPEAKING SECTION AND WHAT KIND OF TRAINING HELPS?
Authors: Xi Xiaoming , Mollaun Pam
Published: August 2014

This study investigated the scoring of the Test of English as a Foreign Language™ Internet‐based Test (TOEFL iBT™) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on‐site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.‐based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non‐Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees.
TOWARD A DEFINITION OF VERBAL REASONING IN HIGHER EDUCATION
Authors: Burton Nancy W. , Welsh Cynthia , Kostin Irene , van Essen Thomas
Published: August 2014

This paper briefly summarizes the literatures of reading and reasoning in the last quarter century, focusing mainly on the disciplines of cognitive science, cognitive developmental psychology, linguistics, and educational psychology. These literatures were synthesized to create a framework for defining verbal reasoning in higher education. Eight general cognitive and metacognitive operations were identified (including, for example, evaluating discourse, seeking and solving problems, and monitoring one's comprehension). Several dimensions underlying these operations on which individual skills may vary were identified (such as breadth of understanding, precision of understanding, or familiarity and facility). Finally, these ideal descriptions of verbal reasoning are applied to the assessment of verbal reasoning for selection in higher education. Problems in measurement and unanticipated consequences of measurement are discussed.
JACKKNIFING TECHNIQUES FOR EVALUATION OF EQUATING ACCURACY
Authors: Haberman Shelby J. , Lee Yi‐Hsuan , Qian Jiahe
Published: August 2014

Grouped jackknifing may be used to evaluate the stability of equating procedures with respect to sampling error and with respect to changes in anchor selection. Properties of grouped jackknifing are reviewed for simple‐random and stratified sampling, and its use is described for comparisons of anchor sets. Application is made to examples of item response theory (IRT) true‐score equating in which two‐parameter logistic and general partial credit models are employed.
LINKING PARAMETER ESTIMATES DERIVED FROM AN ITEM RESPONSE MODEL THROUGH SEPARATE CALIBRATIONS
Authors: Haberman Shelby J.
Published: August 2014

A regression procedure is developed to link simultaneously a very large number of item response theory (IRT) parameter estimates obtained from a large number of test forms, where each form has been separately calibrated and where forms can be linked on a pairwise basis by means of common items. An application is made to forms in which a two‐parameter logistic model is applied to dichotomous items and a general partial credit model is applied to polytomous items.
DEFINING MATHEMATICS COMPETENCY IN THE SERVICE OF COGNITIVELY BASED ASSESSMENT FOR GRADES 6 THROUGH 8
Authors: Graf Edith Aurora
Published: August 2014

This report makes recommendations for the development of middle‐school assessment in mathematics, based on a synthesis of scientific findings in cognitive psychology and mathematics education. The focus is on background research, rather than test specifications or example tasks. Readers interested in early development and pilot efforts associated with the Cognitively Based Assessment of, for, and as Learning (CBAL) project in mathematics (for which this review helped provide a theoretical foundation) should consult Graf, Harris, Marquez, Fife, and Redman (2009). The organization of the report is motivated by the evidence‐centered design (ECD) approach to assessment developed by Mislevy and colleagues (e.g., see Mislevy, Steinberg, & Almond, 2003). The first section consists of a broad literature review that characterizes mathematical competency with respect to both content and process. Subsequent sections discuss: how to model mathematical competency at the middle school level, the kinds of evidence that reflect the level of student competency and support future learning, and how to design tasks that elicit the target evidence.
LINKING ERRORS IN TREND ESTIMATION IN LARGE‐SCALE SURVEYS: A CASE STUDY
Authors: Xu Xueli , von Davier Matthias
Published: August 2014

One of the major objectives of large‐scale educational surveys is reporting trends in academic achievement. For this purpose, a substantial number of items are carried from one assessment cycle to the next. The linking process that places academic abilities measured in different assessments on a common scale is usually based on a concurrent calibration of adjacent assessments using item response theory (IRT) models. It can be conjectured that the selection of common items has a direct effect on the estimation error of academic abilities due to item misfit, small changes in the common items, position effect, and other sources of constructirrelevant changes between measurement occasions. Hence, the error due to the common‐item sampling could be a major source of error for the ability estimates. In operational analyses, generally two sources of error are accounted for in variance estimation: student sampling error and measurement error. A double jackknifing procedure is proposed to include a third source of the estimation error, the error due to common‐item sampling. Three different versions of the double jackknifing were implemented and compared. The data used in this study were item responses from Grade 4 students who took the NAEP 2004 and 2008 math long‐term trend (LTT) assessments. These student samples used in this study are representative samples of Grade 4 student population in 2004 and in 2008 across the US. The results showed that these three double jackknifing approaches resulted in similar standard error estimates that were slightly higher than the estimates from the traditional approach, regardless of whether an item sampling scheme was used or items were dropped at random.
MODELING NONIGNORABLE MISSING DATA WITH ITEM RESPONSE THEORY (IRT)
Authors: Rose Norman , von Davier Matthias , Xu Xueli
Published: August 2014

Large‐scale educational surveys are low‐stakes assessments of educational outcomes conducted using nationally representative samples. In these surveys, students do not receive individual scores, and the outcome of the assessment is inconsequential for respondents. The low‐stakes nature of these surveys, as well as variations in average performance across countries and other factors such as different testing traditions, are contributing factors to the amount of omitted responses in these assessments. While underlying reasons for omissions are not completely understood, common practice in international assessments is to employ a deterministic treatment of omissions, either as missing observations or as responses that are considered wrong. Both approaches appear problematic. In this project, we analyzed the effects of treating omitted responses either as missing or as wrong, as is done in the majority of international studies, and compared these data‐treatment solutions to model‐based approaches to treating omitted responses. The two types of model‐based approaches used in this study are: (a) extensions of multidimensional item response theory (IRT) with an additional dimension based on response indicator variables defined and calibrated together with the set of items containing the observed responses and (b) multidimensional, multiple‐group IRT models with a grouping variable representing the within‐country stratification of respondents by the amount of omitted responses. These two model‐based approaches were compared on the basis of simulated data and data from about 250,000 students from 30 Organisation for Economic Co‐operation and Development (OECD) Member countries participating in an international large‐scale assessment.
THE VALUE OF THE STUDIED ITEM IN THE MATCHING CRITERION IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSIS
Authors: Tan Xuan , Xiang Bihua , Dorans Neil J. , Qu Yanxuan
Published: August 2014

The nature of the matching criterion (usually the total score) in the study of differential item functioning (DIF) has been shown to impact the accuracy of different DIF detection procedures. One of the topics related to the nature of the matching criterion is whether the studied item should be included. Although many studies exist that suggest the studied item should always be included in the criterion, the validity of this statement for models other than the Rasch model has not been studied. This study evaluates the effect of including/excluding the studied item in the matching criterion for situations that mimic real testing situations where the assumptions of the Rasch model are violated. A simulation study was conducted where the effect of including/excluding the studied item in the matching criterion was studied relative to different magnitudes of DIF and different group ability distributions, for data that follow the two‐parameter logistic (2PL) item response theory (IRT) and multidimensional item response theory (MIRT) models. Results from the study show that including the studied item leads to less biased DIF estimates and more appropriate Type I error rate, especially when the group ability distributions are different. Systematic bias positioning DIF estimation in favor of the high ability group was consistently found across all simulated conditions when the studied item was excluded from the matching criterion.
PRINCIPLES AND PRACTICES OF TEST SCORE EQUATING
Authors: Dorans Neil J. , Moses Tim P. , Eignor Daniel R.
Published: August 2014

Score equating is essential for any testing program that continually produces new editions of a test and for which the expectation is that scores from these editions have the same meaning over time. Particularly in testing programs that help make high‐stakes decisions, it is extremely important that test equating be done carefully and accurately. An error in the equating function or score conversion can affect the scores for all examinees, which is both a fairness and a validity concern. Because the reported score is so visible, the credibility of a testing organization hinges on activities associated with producing, equating, and reporting scores. This paper addresses the practical implications of score equating by describing aspects of equating and best practices associated with the equating process.
THE EFFECTIVENESS OF FEEDBACK FOR L1‐ENGLISH AND L2‐WRITING DEVELOPMENT: A META‐ANALYSIS
Authors: Biber Douglas , Nekrasova Tatiana , Horn Brad
Published: August 2014

This research project undertook a review and synthesis of previous research on the effectiveness of feedback for individual writing development. The work plan was divided into two main phases. First, we surveyed all available studies that have investigated the effectiveness of writing feedback, including both quantitative and qualitative research, for students who have learned English as a first language (L1‐English), students who have learned English as a second language (L2‐English), and students who have learned second languages other than English. The results of this survey are described in a narrative overview of previous research pertaining to the role of feedback in the development of writing proficiency. The survey also identified the major theoretical constructs used in this research domain, providing the basis for subsequent statistical analysis.

Second, we built on this survey to carry out a meta‐analysis of empirical studies in this research area. The goal of the meta‐analysis was to provide a quantitative investigation of the extent and ways in which feedback has been effective, summarizing the findings of previous quantitative studies that have employed suitable statistical measures. Several analytical steps were required for the meta‐analysis: developing a coding rubric; analyzing the research design and adequacy of reporting in studies to determine if they were suitable for inclusion; coding each study for all relevant research design factors; computing effect sizes for each study; and analyzing and interpreting the general patterns that hold across this set of studies.

The meta‐analysis compared the gains in writing development with respect to several different kinds of feedback. Overall, feedback was found to result in gains in writing development. Beyond that, there were several predictable findings (e.g., that written feedback is more effective than oral feedback for writing development) and several other more noteworthy trends (e.g., that peer feedback is more effective than teacher feedback for L2‐English students; commenting is more effective than error location; and in general, focus on form and content seems to be more effective than an exclusive focus on form).