bluuebunny/crossref_metadata_2025 huggingface dataset:
LARGE SAMPLE TESTS FOR COMPARING REGRESSION COEFFICIENTS IN MODELS WITH NORMALLY DISTRIBUTED VARIABLES
Authors: von Davier Alina A.
Published: August 2014

The analysis of regression coefficients is an important issue in different scientific areas, mostly because conclusions about the relationship between variables, like causal interpretations, are drawn based on these coefficients. This paper focuses on the description of the null hypothesis of invariance of regression coefficients for multidimensional stochastic regressors. In this study, it is assumed that the variables have a joint normal distribution with unknown expectation and unknown positive definite covariance matrix. In this context, it is shown that the null hypothesis contains special parameter points, called singular and stationary parameter points, that influence the distribution of the commonly used test statistics under the null hypothesis. Three large sample statistics—the Wald test, the likelihood ratio test, and the efficient score test—are compared when testing this nonlinear null hypothesis. The results of a simulation study are presented. The goal of the simulations is to check the distributions of the three statistics for finite sample sizes and at a stationary point of the null hypothesis. Another aim is to compare the empirical values of the three statistics to one another, for different parameter constellations. It is shown that all three statistics present deviations from the expected chi‐squared distribution at this special parameter point. However, any of the three statistical tests might be used for testing the hypothesis of the invariance of the regression coefficients since they remain asymptotically conservative at the stationary points of the null hypothesis.
LINKING SCORES FROM TESTS OF SIMILAR CONTENT GIVEN IN DIFFERENT LANGUAGES: SPANISH LANGUAGE PAA™ AND ENGLISH LANGUAGE SAT® I
Authors: Cascallar Alicia S. , Dorans Neil J.
Published: August 2014

Score linkages between the Verbal and Math sections of the SAT® I: Reasoning Test and the corresponding sections of the new version of a Spanish‐language admissions test, the Prueba de Aptitud Académica® (PAA™), were investigated. A bilingual group design was employed. A language proficiency measure (ESLAT) was used to define the bilingual group and as a predictor variable. Prediction and scaling for concordance results were compared. Results indicated that for both single (PAA Verbal or PAA Math to the corresponding SAT I scores) and composite (PAA−V+M to SAT I−V+M and PAA−V+M+ESLAT to SAT I−V+M) score linkage, prediction is preferable to concordance. Comparison of prediction and concordance results for composite scores versus single construct scores indicates that when PAA Verbal is combined with PAA Math to form a composite, predictions of this composite are better than for Verbal alone but worse than predictions for Math alone.
A VERTICAL SCHEME OF BUNDLING NAEP BOOKLETS
Authors: Deng Hui , Ferris Jim , Hombo Catherine
Published: August 2014

A new scheme of bundling National Assessment of Educational Progress (NAEP) booklets, vertical bundling, was proposed in this study to overcome the limitations of the traditional methodology on booklet pairing balance and bundle length choice. This study compared the vertical scheme with the horizontal scheme through simulations using the 2002 NAEP spiral plans for Reading and Writing assessments. The results showed that the vertical scheme not only had flexibility with respect to bundle length, it also demonstrated potential for reducing the required number of different bundles, decreasing book wastage, and improving balance of within‐session booklet pairings. On the other hand, it inflated within‐session booklet duplication to some extent compared to the horizontal approach, a relatively small trade‐off for improved balance of booklet pairings.
SIMULATION STUDIES APPLYING POSTERIOR PREDICTIVE MODEL CHECKING FOR ASSESSING FIT OF THE COMMON ITEM RESPONSE THEORY MODELS
Authors: Sinharay Sandip , Johnson Matthew S.
Published: August 2014

Model checking in item response theory (IRT) is an underdeveloped area. No universal model‐checking tool exists in this field. Theposterior predictive model‐checkingmethod (Guttman, 1967; Rubin, 1981, 1984) is a popular Bayesian model‐checking tool because of its simplicity, strong theoretical basis, obvious intuitive appeal, and ability to provide graphical evidence. This study applies the posterior predictive model‐checking method to assess the fit of the popular IRT models (Rasch model, 2PL model, and 3PL model) for tests with dichotomous items. A series of simulation studies applying the method yields promising results. An important issue with the application of the posterior predictive model‐checking method is the choice ofdiscrepancy measures(which are like thetest statisticsin classical hypothesis testing). This article examines the performance of a number of discrepancy measures for assessing different aspects of fit of the common IRT models and suggests simple summarization of the results. Simple graphical summaries provide useful insight about the fit of the models. Theodds ratios, corresponding to the responses of the examinees to pairs of items, appear to be powerful discrepancy measures in detecting several types of misfits of the IRT models. The observed score distribution and biserial correlation coefficients are also found useful as discrepancy measures on a number of occasions. A companion article applying the posterior predictive method to real data examples shows that the method has the potential to become very useful in psychometrics.
DESIGN AND ANALYSIS IN A COGNITIVE ASSESSMENT
Authors: Yan Duanli , Mislevy Robert J. , Almond Russell G.
Published: August 2014

There is growing interest in educational assessments that coordinate substantive considerations, learning psychology, task design, and measurement models. This paper concerns an analysis of responses from an assessment of mixed‐number subtraction that was created by Kikumi Tatsuoka (1983) in light of cognitive analyses of students' problem solutions. In particular, we fit a binary‐skills multivariate latent class model to the data and compare results to those obtained with an item response theory model and a modified latent class model suggested by model criticism indices. Markov chain Monte Carlo (MCMC) techniques are used to estimate the parameters in the model in a Bayesian framework that integrates information from substantive theory, expert judgment, and empirical data.
BEYOND ESSAY LENGTH: EVALUATING E‐RATER®'S PERFORMANCE ON TOEFL® ESSAYS
Authors: Chodorow Martin , Burstein Jill
Published: August 2014

This study examines the relation between essay length and holistic scores assigned to Test of English as a Foreign Language™ (TOEFL®) essays by e‐rater®, the automated essay scoring system developed by ETS. Results show that an early version of the system, e‐rater99, accounted for little variance in human reader scores beyond that which could be predicted by essay length. A later version of the system, e‐rater01, performs significantly better than its predecessor and is less dependent on length due to its greater reliance on measures of topical content and of complexity and diversity of vocabulary. Essay length was also examined as a possible explanation for differences in scores among examinees with native languages of Spanish, Arabic, and Japanese. Human readers and e‐rater01 show the same pattern of differences for these groups, even when effects of length are controlled.
EXPLORING ITEM CHARACTERISTICS THAT ARE RELATED TO THE DIFFICULTY OF TOEFL DIALOGUE ITEMS
Authors: Kostin Irene
Published: August 2014

The purpose of this study is to explore the relationship between a set of item characteristics and the difficulty of TOEFL® dialogue items. Identifying characteristics that are related to item difficulty has the potential to improve the efficiency of the item‐writing process The study employed 365 TOEFL dialogue items, which were coded on 49 variables, including 5 significant variables reported in Nissan, DeVincenzi, and Tang (1996). Of the 5 significant variables in Nissan et al., 3 correlated significantly with item difficulty in this study. Another 11 met a critical probability criterion. These 11 included representatives from three broad categories of variables: 2 in the category of word‐level factors, 1 in the category of discourse‐level factors, and 8 in the category of task‐processing factors. Multiple regression analyses indicate that the variables in this study account for about 40% of the variance in item difficulty.
ACCOMMODATIONS ON HIGH‐STAKES WRITING TESTS FOR STUDENTS WITH DISABILITIES
Authors: Cahalan‐Laitusis Cara
Published: August 2014

With the addition of a writing component to many high‐stakes tests (e.g., college admission tests and high school exit exams), a review of writing accommodations for students with disabilities becomes critical. This paper reviews the accommodation policies of 20 states with high school exit exams of writing and three testing organizations that administer high‐stakes tests that assess writing. In addition, this paper reviews existing research on testing accommodations, specifically those that focus on writing assessments. A section on assistive technologies for writing is also included, because these technologies are becoming more widely used and are likely to be requested as testing accommodations. Finally, recommendations for needed research are provided.
JOINT AND CONDITIONAL MAXIMUM LIKELIHOOD ESTIMATION FOR THE RASCH MODEL FOR BINARY RESPONSES
Authors: Haberman Shelby J.
Published: August 2014

The usefulness of joint and conditional maximum‐likelihood is considered for the Rasch model under realistic testing conditions in which the number of examinees is very large and the number is items is relatively large. Conditions for consistency and asymptotic normality are explored, effects of model error are investigated, measures of prediction are estimated, and generalized residuals are developed.
APPLICATION OF THE STOCHASTIC EM METHOD TO LATENT REGRESSION MODELS
Authors: von Davier Matthias , Sinharay Sandip
Published: August 2014

The reporting methods used in large scale assessments such as the National Assessment of Educational Progress (NAEP) rely on alatent regression model.The first component of the model consists of ap‐scale IRT measurement model that defines the response probabilities on a set of cognitive items inpscales depending on ap‐dimensional latent trait variableθ= (θ1, …θp). In the second component, the conditional distribution of this latent trait variableθis modeled by a multivariate, multiple regression on a set of predictor variables, which are usually based on student, school and teacher variables in assessments such as NAEP.

In order to fit the latent regression model using the maximum (marginal) likelihood estimation technique, multivariate integrals have to be evaluated. In the computer program MGROUP used by ETS for fitting the latent regression model to data from NAEP and other sources, the integration is currently done either by numerical quadrature (for problems up to two dimensions) or by an approximation of the integral. CGROUP, the current operational version of the MGROUP program used in NAEP and other assessments since 1993, is based on Laplace approximation that may not provide fully satisfactory results, especially if the number of items per scale is small.

This paper examines the application of stochastic expectation‐maximization (EM) methods (where an integral is approximated by an average over a random sample) to NAEP‐like settings. We present a comparison of CGROUP with a promising implementation of the stochastic EM algorithm that utilizes importance sampling. Simulation studies and real data analysis show that the stochastic EM method provides a viable alternative to CGROUP for fitting multivariate latent regression models.
MULTIPLE PERSONALITY DOMAINS IN RELATION TO OCCUPATIONAL CHOICE AND PERFORMANCE AMONG ESTABLISHED TEACHERS
Authors: Emmerich Walter , Rock Donald A. , Trapani Catherine S.
Published: August 2014

Does personality impact differently on occupational choice and occupational performance? In a study of established teachers, interests, the Five‐Factor Model, and attributions were examined in relation to occupational choice and performance. Attributions were assessed using a new instrument designed for teachers. Choice of teaching specialty was found to be most strongly associated with interest in the arts and sciences, openness to experience, and internal attributions in response to a positive classroom event. On the other hand, quality of teaching assessed by the National Board of Professional Teaching Standards (NBPTS) was associated (negatively) with business interests and with a tendency to externalize blame in response to a problematic teaching situation. Study findings suggested the hypothesis that broad traits located relatively high in a personality hierarchy influence trait matching in selecting an occupation, whereas more specific motivational/dynamic factors located lower in the personality hierarchy influence performance within the chosen niche. The report includes supplemental evidence on the construct validity of the personality measures for assessing teachers.
IDENTIFYING THE WRITING TASKS IMPORTANT FOR ACADEMIC SUCCESS AT THE UNDERGRADUATE AND GRADUATE LEVELS
Authors: Rosenfeld Michael , Courtney Rosalea , Fowles Mary
Published: August 2014

The authors conducted a large‐scale survey to confirm that the writing skills being assessed in the GRE® General Test can be linked to writing tasks that were judged to be important by graduate faculty from a variety of subject areas and across a wide range of institutions at both the graduate and undergraduate levels. The results obtained in this study provide an additional source of validity evidence for using the GRE Analytical Writing Assessment when making admission decisions for graduate school and are also useful in evaluating its relevance for use as an outcomes measure for upper‐division undergraduates.
COMPARISON OF UNIDIMENSIONAL AND MULTIDIMENSIONAL APPROACHES TO IRT PARAMETER ESTIMATION
Authors: Zhang Jinming
Published: August 2014

It is common to assume during statistical analysis of a multiscale assessment that the assessment has simple structure or that it is composed of several unidimensional subtests. Under this assumption, both the unidimensional and multidimensional approaches can be used to estimate item parameters. This paper theoretically demonstrates that these two approaches are the same if the joint maximum likelihood method is used to estimate parameters. However, they are different from each other if the marginal maximum likelihood method is applied. A simulation study is then conducted in this paper to further compare the unidimensional and multidimensional approaches with marginal maximum likelihood method. The simulation results indicate that when the number of items is small the multidimensional approach provides more accurate estimates of item parameters, while the unidimensional approach prevails if the test length is long enough. Further, the impact of violation of the simple structure assumption is also investigated. Specifically the correlation coefficient between subscales will be highly overestimated when the simple structure assumption is violated.
AN EVALUATION OF THE KERNEL EQUATING METHOD: A SPECIAL STUDY WITH PSEUDOTESTS CONSTRUCTED FROM REAL TEST DATA
Authors: von Davier Alina A. , Holland Paul W. , Livingston Samuel A. , Casabianca Jodi , Grant Mary C. , Martin Kathleen
Published: August 2014

This study examines how closely the kernel equating (KE) method (von Davier, Holland, & Thayer, 2004a) approximates the results of other observed‐score equating methods—equipercentile and linear equatings. The study used pseudotests constructed of item responses from a real test to simulate three equating designs: an equivalent groups (EG) design and two non‐equivalent groups with anchor test (NEAT) designs, one with an internal anchor and another with an external anchor. To compare results, the study sets the equating function in the EG design as the equating criterion. In these examples, the KE results were very close to the results from the other equating methods. Moreover, in almost all situations investigated, the KE results were closer to the equating criterion.
STRATEGIES IN RESPONDING TO THE NEW TOEFL READING TASKS
Authors: Cohen Andrew D. , Upton Thomas A.
Published: August 2014

This study describes the reading and test‐taking strategies that test takers used in the Reading section of the LanguEdge courseware (ETS, 2002a). These materials were developed to familiarize prospective respondents with the new TOEFL®. The investigation focused on strategies used to respond to more traditional single selection multiple‐choice formats (i.e., Basic Comprehension and Inferencing questions) and the new selected‐response (multiple selection, drag and drop) Reading to Learn items. The latter were designed to simulate the academic task of forming a comprehensive and coherent representation of an entire text, rather than focusing on discrete points in the text. Verbal report data were collected from 32 students, representing four language groups (Chinese, Japanese, Korean, and Other) doing the Reading section tasks from the LanguEdge courseware materials. Students were randomly assigned to two of the six reading subtests, each consisting of a 600–700 word text with 12–13 items. Subjects' verbal report accompanying items representing each of the ten item types was evaluated to determine strategy use. The findings provide insights into the response behaviors prompted by the reading tasks on the new TOEFL.
USING PAST DATA TO ENHANCE SMALL‐SAMPLE DIF ESTIMATION: A BAYESIAN APPROACH
Authors: Sinharay Sandip , Dorans Neil J. , Grant Mary C. , Blew Edwin O. , Knorr Colleen M.
Published: August 2014

The application of the Mantel‐Haenszel test statistic (and other popular DIF‐detection methods) to determine DIF requires large samples, but test administrators often need to detect DIF with small samples. There is no universally agreed upon statistical approach for performing DIF analysis with small samples; hence there is substantial scope of further work on the problem. One advantage of a Bayesian approach over a frequentist approach is that the former can incorporate, in the form of a prior distribution, existing information on the inference problem at hand; a prior distribution often leads to improved estimation, especially for small samples. Further, for any operational test, a huge volume of past data is available, and for any item appearing in a present test, there is a high chance that a number of similar items have appeared on past operational administrations of the test. Therefore, ideally, it will be possible to use that past information as a prior distribution in a Bayesian DIF analysis. This paper discusses how to perform such an analysis. The suggested Bayesian DIF analysis method is shown to be an improvement over the existing methods in a realistic simulation study.
SUBSCORES FOR INSTITUTIONS
Authors: Haberman Shelby J. , Sinharay Sandip , Puhan Gautam
Published: August 2014

Recently, there has been an increasing level of interest in reporting subscores. This paper examines the issue of reporting subscores at an aggregate level, especially at the level of institutions that the examinees belong to. A series of statistical analyses is suggested to determine when subscores at the institutional level have any added value over the total scores. The methods are applied to two operational data sets. For the data under study, the results provide little support in favor of reporting subscores for either examinees or institutions.
COMPARISON OF MULTISTAGE TESTS WITH COMPUTERIZED ADAPTIVE AND PAPER‐AND‐PENCIL TESTS
Authors: Rotou Ourania , Patsula Liane , Steffen Manfred , Rizavi Saba
Published: August 2014

Traditionally, the fixed‐length linear paper‐and‐pencil (P&P) mode of administration has been the standard method of test delivery. With the advancement of technology, however, the popularity of administering tests using adaptive methods like computerized adaptive testing (CAT) and multistage testing (MST) has grown in the field of measurement in both theory and practice. In practice, several standardized tests have sections that include only set‐based items. To date, there is no study in the literature that compares these testing procedures when a test is completely set‐based under various item response theory (IRT) models. This study investigates the measurement precision of MST compared to CAT and compared to P&P tests for the one‐, two‐, and three‐parameter logistic (1‐, 2‐, and 3PL) models when the test is completely set‐based. Results showed that MST performed better for the 2‐ and 3PL models than an equivalent‐length P&P test in terms of reliability and conditional standard error of measurement. In addition, findings showed that MST performed better for the 1‐ and 2PL models than for an equivalent‐length CAT test. For the 3PL model, MST and CAT performed about the same.
AN EQUIPERCENTILE VERSION OF THE LEVINE LINEAR OBSERVED‐SCORE EQUATING FUNCTION USING THE METHODS OF KERNEL EQUATING
Authors: von Davier Alina A. , Fournier‐Zajac Stephanie , Holland Paul W.
Published: August 2014

In the nonequivalent groups with anchor test (NEAT) design, there are several ways to use the information provided by the anchor in the equating process. One of the NEAT‐design equating methods is the linear observed‐score Levine method (Kolen & Brennan, 2004). It is based on a classical test theory model of the true scores on the test forms to be equated and on the anchor test (Levine, 1955). The Levine linear method does not yet have an equipercentile analogue, and no version of kernel equating (KE), introduced in von Davier, Holland, and Thayer (2004b), approximates the Levine linear method. Nevertheless, the Levine observed‐score equating method is often computed in practical applications for comparison purposes because it is sometimes more accurate than other linear equating methods (Petersen, Marco, & Stewart, 1982). In situations when a linear equating function is not satisfactory, an equipercentile version of the Levine function may be desirable. This paper proposes a general method for constructing hybrid equating functions that combine linear and nonlinear equating functions in a systematic way that preserves the symmetry required of equating functions (Dorans & Holland, 2000). The general method is then applied to combine the linear Levine observed‐score equating function with a nonlinear equipercentile equating function derived using the poststratification equating (PSE) assumptions within the KE framework. An easily computed approximation to the resulting PSE‐Levine equipercentile equating function is illustrated on data from a special study and compared to the results from the traditional equating functions. The special data set includes a criterion equating function, and the closeness of the PSE‐Levine function to this criterion indicates that such hybrid equating functions may be useful in practice.
KERNEL AND TRADITIONAL EQUIPERCENTILE EQUATING WITH DEGREES OF PRESMOOTHING
Authors: Moses Tim , Holland Paul
Published: August 2014

The purpose of this study was to empirically evaluate the impact of loglinear presmoothing accuracy on equating bias and variability across chained and post‐stratification equating methods, kernel and percentile‐rank continuization methods, and sample sizes. The results of evaluating presmoothing on equating accuracy generally agreed with those of previous presmoothing studies, suggesting that less parameterized presmoothing models are more biased and less variable than highly parameterized presmoothing models and raw data. Estimates of standard errors of equating were most accurate when based on large sample sizes and score‐level data that were not sparse. The accuracy of standard error estimates was not influenced by the correctness of the presmoothing model. The accuracy of estimates of the standard errors of equating differences was also evaluated. The study concludes with some detailed comparisons of how the kernel and traditional equipercentile continuization methods interacted with data that were presmoothed to different degrees.
HIERARCHICAL GENERAL DIAGNOSTIC MODELS
Authors: von Davier Matthias
Published: August 2014

This paper introduces multilevel extensions for the general diagnostic model (GDM) following recent developments on extensions of latent class analysis (LCA) to hierarchical models. The GDM is based on LCA as well as discrete latent trait models and may be viewed as a general modeling framework for confirmatory multidimensional item response models.

The multilevel extensions presented in this paper enable one to check the impact of clustered data, such as data for students within schools in large scale educational surveys, on the structural parameter estimates of the GDM. Moreover, the multilevel version of the GDM allows study of differences in skill distributions across these clusters.
REFINEMENT OF A BIAS‐CORRECTION PROCEDURE FOR THE WEIGHTED LIKELIHOOD ESTIMATOR OF ABILITY
Authors: Zhang Jinming , Lu Ting
Published: August 2014

In practical applications of item response theory (IRT), item parameters are usually estimated first from a calibration sample. After treating these estimates as fixed and known, ability parameters are then estimated. However, the statistical inferences based on the estimated abilities can be misleading if the uncertainty of the item parameter estimates is ignored. Instead, estimated item parameters can be regarded as covariates measured with error. Along the line of this measurement‐error‐model approach, asymptotic expansions of the maximum likelihood estimator (MLE) and weighted likelihood estimator (WLE) of ability were derived by Zhang, Xie, Song, and Lu (2007). In this paper, we propose an estimator of an ability parameter based on the asymptotic formula of the WLE. A simulation study shows that the new estimator effectively reduces the bias of the MLE or WLE of ability caused by the uncertainty of the item parameter estimates not taken into account.
EXAMINATION OF FATIGUE EFFECTS FROM EXTENDED‐TIME ACCOMMODATIONS ON THE SAT REASONING TEST™
Authors: Laitusis Cara Cahalan , Morgan Deanna L. , Bridgeman Brent , Zanna Jennifer , Stone Elizabeth
Published: August 2014

This study examined operational data from the SAT Reasoning Test™ to determine if students who tested under extended‐time conditions were suffering from excessive fatigue relative to students who tested under standard‐time conditions. Excessive fatigue was defined by significant (a) increases in differential item functioning (DIF) and (b) decreases in item completion rates, for items at the end of testing compared to the beginning of testing. Both of these factors were examined by comparing the performance of students who tested under standard time to students who tested with extended time on items administered early (section position 2 or 3) and different items administered late (section position 8, 9, or 10) during the 10‐section test administration. The sample included students with learning disabilities and/or Attention‐Deficit/Hyperactivity Disorder (ADHD) who tested with extended time (time and a half or double time) and students without disabilities who tested under standard‐time conditions. Analyses were conducted on the critical reading and writing sections of the SAT® and examined item difficulty as well as item completion rates. Results indicated few changes in levels of DIF (early in the test compared to late in the test). In addition, item completion rates for students who received extra time were comparable to (or in some cases higher than) test‐takers without disabilities who tested under standard time on both early and late sections.
AN ILLUSTRATION OF THE USE OF MARKOV DECISION PROCESSES TO REPRESENT STUDENT GROWTH (LEARNING)
Authors: Almond Russell G.
Published: August 2014

Over the course of instruction, instructors generally collect a great deal of information about each student. Integrating that information intelligently requires models for how a student's proficiency changes over time. Armed with such models, instructors can filter the data—more accurately estimate the student's current proficiency levels—and forecast the student's future proficiency levels. The process of instructional planning can be described as a partially observed Markov decision process (POMDP). Recently developed computer algorithms can be used to help instructors create strategies for student achievement and identify at‐risk students. Implementing this vision requires models for how instructional actions change student proficiencies. The general action model (also called the bowtie model) separately models the factors contributing to the success or effectiveness of an action, proficiency growth when the action is successful, and proficiency growth when the action is unsuccessful. This class of models requires parameterization, and this paper presents two: a simple linear process model (suitable for continuous proficiencies) and a birth‐and‐death process model (for proficiency scales expressed as ordered categorical variables). Both models show how to take prerequisites and zones of proximal development into account. The filtering process is illustrated using a simple artificial example.
ANALYTIC SCORING OF TOEFL® CBT ESSAYS: SCORES FROM HUMANS AND E‐RATER®
Authors: Lee Yong‐Won , Gentile Claudia , Kantor Robert
Published: August 2014

The main purpose of the study was to investigate the distinctness and reliability of analytic (or multitrait) rating dimensions and their relationships to holistic scores and e‐rater® essay feature variables in the context of the TOEFL® computer‐based test (CBT) writing assessment. Data analyzed in the study were analytic and holistic essay scores provided by human raters and essay feature variable scores computed by e‐rater (version 2.0) for two TOEFL CBT writing prompts. It was found that (a) all of the six analytic scores were not only correlated among themselves but also correlated with the holistic scores, (b) high correlations obtained among holistic and analytic scores were largely attributable to the impact of essay length on both analytic and holistic scoring, (c) there may be some potential for profile scoring based on analytic scores, and (d) some strong associations were confirmed between several e‐rater variables and analytic ratings. Implications are discussed for improving the analytic scoring of essays, validating automated scores, and refining e‐rater essay feature variables.
INVESTIGATING THE CRITERION‐RELATED VALIDITY OF THE TOEFL® SPEAKING SCORES FOR ITA SCREENING AND SETTING STANDARDS FOR ITAS
Authors: Xi Xiaoming
Published: August 2014

Although the primary use of the speaking section of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT Speaking test) is to inform admissions decisions at English medium universities, it may also be useful as an initial screening measure for international teaching assistants (ITAs). This study provides criterion‐related validity evidence for the use of TOEFL iBT Speaking for ITA screening and evaluates the effectiveness of using the scores for teaching assistantship (TA) assignment classification.

Four universities participated in this study. Local ITA‐screening tests or instructor recommendations were used as the criterion measures. Relationships between the TOEFL Speaking test and the local ITA tests were explored through observed and disattenuated correlations. These relationships were moderately strong, supporting the use of the TOEFL Speaking test for ITA screening. However, the strengths of the relationship between the TOEFL Speaking test and the local ITA tests were found to be somewhat different across universities depending on the extent to which the local test engaged and evaluated nonlanguage abilities. Implications of these findings are discussed.

Binary and ordinal logistic regressions were used to investigate how effective TOEFL Speaking scores were in separating students into distinct TA assignment categories. At all four universities, TOEFL Speaking scores were significant predictors of students' TA assignments and were fairly accurate in classifying students for TA assignments. ROC curves were used to determine TOEFL Speaking cut scores for TA assignments at each university that would minimize false positives (i.e., true nonpasses classified as passes).

The results have considerable potential value in providing guidance on using the TOEFL iBT Speaking scores for ITA screening.
LINKING FOR THE GENERAL DIAGNOSTIC MODEL
Authors: Xu Xueli , von Davier Matthias
Published: August 2014

Three strategies for linking two consecutive assessments are investigated and compared by analyzing reading data for the National Assessment of Educational Progress (NAEP) using the general diagnostic model. These strategies are compared in terms of marginal and joint expectations of skills, joint probabilities of skill patterns, and item parameter estimates. The results indicate that fixing item parameter values at their previously calibrated values is sufficient to establish a comparable scale for the subsequent year.
COMPARING ALTERNATIVE KERNELS FOR THE KERNEL METHOD OF TEST EQUATING: GAUSSIAN, LOGISTIC, AND UNIFORM KERNELS
Authors: Lee Yi‐Hsuan , von Davier Alina A.
Published: August 2014

The kernel equating method (von Davier, Holland, & Thayer, 2004) is based on a flexible family of equipercentile‐like equating functions that use a Gaussian kernel to continuize the discrete score distributions. While the classical equipercentile, or percentile‐rank, equating method carries out the continuization step by linear interpolation, in principle the kernel equating methods could use various kernel smoothings to replace the discrete score distributions.

This paper expands the work of von Davier et al. (2004) in investigating alternative kernels for equating practice. To examine the influence of different kernel functions on the equating results, this paper focuses on two types of kernel functions: the logistic kernel and the continuous uniform distribution (known to be the same as the linear interpolation). The Gaussian kernel is used for reference. By employing an equivalent‐groups design, the results of the study indicate that the tail properties of kernel functions have great impact on the continuized score distributions. However, the equated scores based on different kernel functions do not vary much, except for extreme scores.

The results presented in this paper not only support the previous findings on the efficiency and accuracy of the existing continuization methods, but also enrich the information on observed‐score equating models.
A DEVELOPMENTAL WRITING SCALE
Authors: Attali Yigal , Powers Don
Published: August 2014

This report describes the development of grade norms for timed‐writing performance in two modes of writing: persuasive and descriptive. These norms are based on objective and automatically computed measures of writing quality in grammar, usage, mechanics, style, vocabulary, organization, and development. These measures are also used in the automated essay scoring system e‐rater® V.2. Norms were developed through a large‐scale data collection effort that involved a national sample of 170 schools, more than 500 classes from 4th, 6th, 8th, 10th, and 12th grades and more than 12,000 students. Personal and school background information was also collected. These students wrote (in 30‐minute sessions) up to 4 essays (2 in each mode of writing) on topics selected from a pool of 20 topics.

The data allowed us to explore a range of questions about the development and nature of writing proficiency. Specifically, this paper provides a description of the trajectory of development in writing performance from 4th grade to 12th grade. The validity of a single developmental writing scale is examined through a human scoring experiment and a longitudinal study. The validity of the single scale is further explored through a factor analysis (exploratory and confirmatory) of the internal structure of writing performance and changes in this structure from 4th grade to 12th grade. The paper also explores important factors affecting performance, including prompt difficulty, writing mode, and student background (gender, ethnicity, and English language background).
ROBUSTNESS OF A VALUE‐ADDED ASSESSMENT OF SCHOOL EFFECTIVENESS
Authors: Braun Henry , Qu Yanxuan , Trapani Catherine
Published: August 2014

This paper reports on a study conducted to investigate the consistency of the results between 2 approaches to estimating school effectiveness through value‐added modeling. Estimates of school effects from the layered model employing item response theory (IRT) scaled data are compared to estimates derived from a discrete growth model based on the analysis of transitions along an ordinal developmental scale. The data were extracted from the longitudinal records maintained in the Early Childhood Longitudinal Study—Kindergarten Cohort (ECLS‐K) archive for students remaining in the same school from the beginning of kindergarten through the end of Grade 3. The results of different comparisons indicated that the estimates from the 2 approaches are moderately consistent.
TIGHT BUT LOOSE: SCALING UP TEACHER PROFESSIONAL DEVELOPMENT IN DIVERSE CONTEXTS
Authors: Wylie E. Caroline , Goe Laura , Leusner Dawn Marie , Lyon Christine J. , Tocic Cynthia , Wylie E. Caroline , Cleland Donna , Gannon Maureen , Ellsworth Judith , Heritage Margaret , Maher Jeff , Mardy Diane , Popham W. James , Snodgrass Donna , Taylor Gayle , Thompson Marnie , Wiliam Dylan
Published: August 2014

This series of papers was originally presented as a symposium at the annual meetings of the American Educational Research Association (AERA) and the National Council on Measurement in Education (NCME) held between April 9, 2007, and April 13, 2007, in Chicago, IL. The authors represent school districts and departments of education across the United States, as well as researchers at Cleveland State University, Educational Testing Service (ETS), the Institute for Education in London, and the University of Wyoming at Laramie. All of the current ETS staff, along with Dylan Wiliam and Marnie Thompson, worked at ETS for several years on an iterative research and development program, out of which grew the Keeping Learning on Track® (KLT) program.

These papers represents the thinking about the theory behind the KLT program, describes the range of contexts used to implement the program, and illustrates the inherent tensions between the desire to maintain fidelity to a theory of action and the need to demonstrate flexibility in order to accommodate local situations. Papers 2 through 6 present descriptions of five implementations in chronological order.
RESPONSE TO ASSESSMENT FEEDBACK: THE EFFECTS OF GRADES, PRAISE, AND SOURCE OF INFORMATION
Authors: Lipnevich Anastasiya A. , Smith Jeffrey K.
Published: August 2014

This experiment involved college students (N= 464) working on an authentic learning task (writing an essay) under 3 conditions: no feedback, detailed feedback (perceived by participants to be provided by the course instructor), and detailed feedback (perceived by participants to be computer generated). Additionally, conditions were crossed with 2 factors of grade (receiving grade or not) and praise (receiving praise or not). Detailed feedback specific to individual work was found to be strongly related to student improvement in essay scores, with the influence of grades and praise more complex. Overall, detailed, descriptive feedback was found to be most effective when given alone, unaccompanied by grades or praise. The results have implications for theory and practice of assessment.
EFFECTS OF CALCULATOR AVAILABILITY ON GRE® QUANTITATIVE QUESTIONS
Authors: Bridgeman Brent , Cline Frederick , Levin Jutta
Published: August 2014

In order to estimate the likely effects on item difficulty when a calculator becomes available on the quantitative section of the Graduate Record Examinations® (GRE®‐Q), 168 items (in six 28‐item forms) were administered either with or without access to an on‐screen four‐function calculator. The forms were administered as a special research section at the end of operational tests, with student volunteers randomly assigned to the calculator or no‐calculator groups. Usable data were obtained from 13,159 participants. Test development specialists were asked to rate which items they thought would become easier with a calculator. In general, the specialists were successful in identifying the items with relatively large calculator effects, though even these effects were quite small. An increase of only about four points in the percent correct should suffice for the items identified as likely to show calculator effects with no adjustment needed for the majority of the items. Introduction of a calculator should have little or no effect on gender and ethnic differences.
SAMPLE‐SIZE REQUIREMENTS FOR AUTOMATED ESSAY SCORING
Authors: Haberman Shelby J. , Sinharay Sandip
Published: August 2014

Sample‐size requirements were considered for automated essay scoring in cases in which the automated essay score estimates the score provided by a human rater. Analysis considered both cases in which an essay prompt is examined in isolation and those in which a family of essay prompts is studied. In typical cases in which content analysis is not employed and in which the only object is to score individual essays to provide feedback to the examinee, it appears that several hundred essays are sufficient. For application of one model to a family of essays, fewer than 100 essays per prompt may often be adequate. The cumulative logit model was explored as a possible replacement of the linear regression model usually employed in automated essay scoring; the cumulative logit model performed somewhat better than did the linear regression model.
THE ASSESSMENT OF INFORMATION LITERACY: A CASE STUDY
Authors: Katz Irvin R. , Elliot Norbert , Attali Yigal , Scharf Davida , Powers Donald , Huey Heather , Joshi Kamal , Briller Vladimir
Published: August 2014

This study presents an investigation of information literacy as defined by the ETS iSkills™ assessment and by the New Jersey Institute of Technology (NJIT) Information Literacy Scale (ILS). As two related but distinct measures, both iSkills and the ILS were used with undergraduate students at NJIT during the spring 2006 semester. Undergraduate students (n = 331), first through senior years, took the iSkills and submitted portfolios to be judged by the ILS. First‐year students took the Core iSkills assessment, which was designed to provide administrators and faculty with an understanding of the information and communication technology (ICT) literacy of a student doing entry‐level coursework (n = 155). Upper classmen took the more difficult Advanced iSkills assessment, appropriate for rising juniors (n = 176). Across all class levels, iSkills scores varied as expected. First‐year basic skills writing students performed at lower levels than first‐year students enrolled in traditional composition and cultural history courses; seniors performed at higher levels than sophomores and juniors. Because the NJIT ILS scores were designed to be curriculum sensitive, portfolio scores did not similarly follow grade levels. Analyses revealed weak correlations between portfolio and Core iSkills scores and moderate correlations between portfolio and Advanced iSkills scores. As two associated yet distinct systems of inquiry designed to explore undergraduate student performance, the ETS iSkills assessment and the NJIT ILS—taken both individually and together—yield important information regarding student performance.
COMPARING MULTIPLE‐GROUP MULTINOMIAL LOG‐LINEAR MODELS FOR MULTIDIMENSIONAL SKILL DISTRIBUTIONS IN THE GENERAL DIAGNOSTIC MODEL
Authors: Xu Xueli , von Davier Matthias
Published: August 2014

The general diagnostic model (GDM) utilizes located latent classes for modeling a multidimensional proficiency variable. In this paper, the GDM is extended by employing a log‐linear model for multiple populations that assumes constraints on parameters across multiple groups. This constrained model is compared to log‐linear models that assume separate sets of parameters to fit the distribution of latent variables in each group of a multiple‐group model. Estimation of these constrained log‐linear models using iterative weighted least squares (IWLS) methods is outlined and an application to NAEP data exemplifies the differences between constrained and unconstrained models in the presence of larger numbers of group‐specific proficiency distributions.

The use of log‐linear models for the latent skill space distributions using constraints across populations allows for efficient computations in models that include many proficiency distributions.
DEVELOPMENT OF APPROXIMATIONS TO POPULATION INVARIANCE INDICES
Authors: Liu Jinghua , Zhu Xiaowen
Published: August 2014

The purpose of this paper is to explore methods to approximate population invariance without conducting multiple linkings for subpopulations. Under the single group or equivalent groups design, no linking needs to be performed for the parallel‐linear system linking functions. The unequated raw score information can be used as an approximation. For other linking functions that are nonparallel‐linear, linking only needs to be conducted for the total population. The difference of the standardized mean differences between each subpopulation and the total population across the old form and the new form can be used as an approximation of population invariance. Under the nonequivalent groups with anchor test design, conducting separate subpopulation linking and comparing them to the total population linking may still be the best way to estimate population invariance.
DIF DETECTION WITH SMALL SAMPLES: APPLYING SMOOTHING TECHNIQUES TO FREQUENCY DISTRIBUTIONS IN THE MANTEL‐HAENSZEL PROCEDURE
Authors: Yu Lei , Moses Tim , Puhan Gautam , Dorans Neil
Published: August 2014

All differential item functioning (DIF) methods require at least a moderate sample size for effective DIF detection. Samples that are less than 200 pose a challenge for DIF analysis. Smoothing can improve upon the estimation of the population distribution by preserving major features of an observed frequency distribution while eliminating the noise brought about by irregular data points. This study applied smoothing techniques to frequency distributions and investigated the impact of smoothed data on the Mantel‐Haenszel (MH) DIF detection in small samples. Eight sample‐size combinations were randomly drawn from a real data set to make the study realistic and were replicated 80 times to produce stable results. The population DIF results were used as the criteria to evaluate sample estimates using root‐mean square difference (RMSD), bias analysis, and Type II error rate. Loglinear smoothing was found to provide slight to moderate improvements in MH DIF estimation with small samples.
APPLYING CONTENT SIMILARITY METRICS TO CORPUS DATA: DIFFERENCES BETWEEN NATIVE AND NON‐NATIVE SPEAKER RESPONSES TO A TOEFL® INTEGRATED WRITING PROMPT
Authors: Deane Paul , Gurevich Olga
Published: August 2014

For many purposes, it is useful to collect a corpus of texts all produced to the same stimulus, whether to measure performance (as on a test) or to test hypotheses about population differences. This paper examines several methods for measuring similarities in phrasing and content and demonstrates that these methods can be used to identify population differences between native and non‐native speakers of English in a writing task.