bluuebunny/crossref_metadata_2025_split huggingface dataset:
COMPARISON OF UNIDIMENSIONAL AND MULTIDIMENSIONAL APPROACHES TO IRT PARAMETER ESTIMATION
Authors: Zhang Jinming 
Published: August 2014

ABSTRACTIt is common to assume during statistical analysis of a multiscale assessment that the assessment has simple structure or that it is composed of several unidimensional subtests. Under this assumption, both the unidimensional and multidimensional approaches can be used to estimate item parameters. This paper theoretically demonstrates that these two approaches are the same if the joint maximum likelihood method is used to estimate parameters. However, they are different from each other if the marginal maximum likelihood method is applied. A simulation study is then conducted in this paper to further compare the unidimensional and multidimensional approaches with marginal maximum likelihood method. The simulation results indicate that when the number of items is small the multidimensional approach provides more accurate estimates of item parameters, while the unidimensional approach prevails if the test length is long enough. Further, the impact of violation of the simple structure assumption is also investigated. Specifically the correlation coefficient between subscales will be highly overestimated when the simple structure assumption is violated.
AN EVALUATION OF THE KERNEL EQUATING METHOD: A SPECIAL STUDY WITH PSEUDOTESTS CONSTRUCTED FROM REAL TEST DATA
Authors: von Davier Alina A. , Holland Paul W. , Livingston Samuel A. , Casabianca Jodi , Grant Mary C. , Martin Kathleen 
Published: August 2014

ABSTRACTThis study examines how closely the kernel equating (KE) method (von Davier, Holland, & Thayer, 2004a) approximates the results of other observed‐score equating methods—equipercentile and linear equatings. The study used pseudotests constructed of item responses from a real test to simulate three equating designs: an equivalent groups (EG) design and two non‐equivalent groups with anchor test (NEAT) designs, one with an internal anchor and another with an external anchor. To compare results, the study sets the equating function in the EG design as the equating criterion. In these examples, the KE results were very close to the results from the other equating methods. Moreover, in almost all situations investigated, the KE results were closer to the equating criterion.
STRATEGIES IN RESPONDING TO THE NEW TOEFL READING TASKS
Authors: Cohen Andrew D. , Upton Thomas A. 
Published: August 2014

ABSTRACTThis study describes the reading and test‐taking strategies that test takers used in the Reading section of the LanguEdge courseware (ETS, 2002a). These materials were developed to familiarize prospective respondents with the new TOEFL®. The investigation focused on strategies used to respond to more traditional single selection multiple‐choice formats (i.e., Basic Comprehension and Inferencing questions) and the new selected‐response (multiple selection, drag and drop) Reading to Learn items. The latter were designed to simulate the academic task of forming a comprehensive and coherent representation of an entire text, rather than focusing on discrete points in the text. Verbal report data were collected from 32 students, representing four language groups (Chinese, Japanese, Korean, and Other) doing the Reading section tasks from the LanguEdge courseware materials. Students were randomly assigned to two of the six reading subtests, each consisting of a 600–700 word text with 12–13 items. Subjects' verbal report accompanying items representing each of the ten item types was evaluated to determine strategy use. The findings provide insights into the response behaviors prompted by the reading tasks on the new TOEFL.
USING PAST DATA TO ENHANCE SMALL‐SAMPLE DIF ESTIMATION: A BAYESIAN APPROACH
Authors: Sinharay Sandip , Dorans Neil J. , Grant Mary C. , Blew Edwin O. , Knorr Colleen M. 
Published: August 2014

ABSTRACTThe application of the Mantel‐Haenszel test statistic (and other popular DIF‐detection methods) to determine DIF requires large samples, but test administrators often need to detect DIF with small samples. There is no universally agreed upon statistical approach for performing DIF analysis with small samples; hence there is substantial scope of further work on the problem. One advantage of a Bayesian approach over a frequentist approach is that the former can incorporate, in the form of a prior distribution, existing information on the inference problem at hand; a prior distribution often leads to improved estimation, especially for small samples. Further, for any operational test, a huge volume of past data is available, and for any item appearing in a present test, there is a high chance that a number of similar items have appeared on past operational administrations of the test. Therefore, ideally, it will be possible to use that past information as a prior distribution in a Bayesian DIF analysis. This paper discusses how to perform such an analysis. The suggested Bayesian DIF analysis method is shown to be an improvement over the existing methods in a realistic simulation study.
SUBSCORES FOR INSTITUTIONS
Authors: Haberman Shelby J. , Sinharay Sandip , Puhan Gautam 
Published: August 2014

ABSTRACTRecently, there has been an increasing level of interest in reporting subscores. This paper examines the issue of reporting subscores at an aggregate level, especially at the level of institutions that the examinees belong to. A series of statistical analyses is suggested to determine when subscores at the institutional level have any added value over the total scores. The methods are applied to two operational data sets. For the data under study, the results provide little support in favor of reporting subscores for either examinees or institutions.
COMPARISON OF MULTISTAGE TESTS WITH COMPUTERIZED ADAPTIVE AND PAPER‐AND‐PENCIL TESTS
Authors: Rotou Ourania , Patsula Liane , Steffen Manfred , Rizavi Saba 
Published: August 2014

ABSTRACTTraditionally, the fixed‐length linear paper‐and‐pencil (P&P) mode of administration has been the standard method of test delivery. With the advancement of technology, however, the popularity of administering tests using adaptive methods like computerized adaptive testing (CAT) and multistage testing (MST) has grown in the field of measurement in both theory and practice. In practice, several standardized tests have sections that include only set‐based items. To date, there is no study in the literature that compares these testing procedures when a test is completely set‐based under various item response theory (IRT) models. This study investigates the measurement precision of MST compared to CAT and compared to P&P tests for the one‐, two‐, and three‐parameter logistic (1‐, 2‐, and 3PL) models when the test is completely set‐based. Results showed that MST performed better for the 2‐ and 3PL models than an equivalent‐length P&P test in terms of reliability and conditional standard error of measurement. In addition, findings showed that MST performed better for the 1‐ and 2PL models than for an equivalent‐length CAT test. For the 3PL model, MST and CAT performed about the same.
AN EQUIPERCENTILE VERSION OF THE LEVINE LINEAR OBSERVED‐SCORE EQUATING FUNCTION USING THE METHODS OF KERNEL EQUATING
Authors: von Davier Alina A. , Fournier‐Zajac Stephanie , Holland Paul W. 
Published: August 2014

ABSTRACTIn the nonequivalent groups with anchor test (NEAT) design, there are several ways to use the information provided by the anchor in the equating process. One of the NEAT‐design equating methods is the linear observed‐score Levine method (Kolen & Brennan, 2004). It is based on a classical test theory model of the true scores on the test forms to be equated and on the anchor test (Levine, 1955). The Levine linear method does not yet have an equipercentile analogue, and no version of kernel equating (KE), introduced in von Davier, Holland, and Thayer (2004b), approximates the Levine linear method. Nevertheless, the Levine observed‐score equating method is often computed in practical applications for comparison purposes because it is sometimes more accurate than other linear equating methods (Petersen, Marco, & Stewart, 1982). In situations when a linear equating function is not satisfactory, an equipercentile version of the Levine function may be desirable. This paper proposes a general method for constructing hybrid equating functions that combine linear and nonlinear equating functions in a systematic way that preserves the symmetry required of equating functions (Dorans & Holland, 2000). The general method is then applied to combine the linear Levine observed‐score equating function with a nonlinear equipercentile equating function derived using the poststratification equating (PSE) assumptions within the KE framework. An easily computed approximation to the resulting PSE‐Levine equipercentile equating function is illustrated on data from a special study and compared to the results from the traditional equating functions. The special data set includes a criterion equating function, and the closeness of the PSE‐Levine function to this criterion indicates that such hybrid equating functions may be useful in practice.
KERNEL AND TRADITIONAL EQUIPERCENTILE EQUATING WITH DEGREES OF PRESMOOTHING
Authors: Moses Tim , Holland Paul 
Published: August 2014

ABSTRACTThe purpose of this study was to empirically evaluate the impact of loglinear presmoothing accuracy on equating bias and variability across chained and post‐stratification equating methods, kernel and percentile‐rank continuization methods, and sample sizes. The results of evaluating presmoothing on equating accuracy generally agreed with those of previous presmoothing studies, suggesting that less parameterized presmoothing models are more biased and less variable than highly parameterized presmoothing models and raw data. Estimates of standard errors of equating were most accurate when based on large sample sizes and score‐level data that were not sparse. The accuracy of standard error estimates was not influenced by the correctness of the presmoothing model. The accuracy of estimates of the standard errors of equating differences was also evaluated. The study concludes with some detailed comparisons of how the kernel and traditional equipercentile continuization methods interacted with data that were presmoothed to different degrees.
HIERARCHICAL GENERAL DIAGNOSTIC MODELS
Authors: von Davier Matthias 
Published: August 2014

ABSTRACTThis paper introduces multilevel extensions for the general diagnostic model (GDM) following recent developments on extensions of latent class analysis (LCA) to hierarchical models. The GDM is based on LCA as well as discrete latent trait models and may be viewed as a general modeling framework for confirmatory multidimensional item response models.The multilevel extensions presented in this paper enable one to check the impact of clustered data, such as data for students within schools in large scale educational surveys, on the structural parameter estimates of the GDM. Moreover, the multilevel version of the GDM allows study of differences in skill distributions across these clusters.
REFINEMENT OF A BIAS‐CORRECTION PROCEDURE FOR THE WEIGHTED LIKELIHOOD ESTIMATOR OF ABILITY
Authors: Zhang Jinming , Lu Ting 
Published: August 2014

ABSTRACTIn practical applications of item response theory (IRT), item parameters are usually estimated first from a calibration sample. After treating these estimates as fixed and known, ability parameters are then estimated. However, the statistical inferences based on the estimated abilities can be misleading if the uncertainty of the item parameter estimates is ignored. Instead, estimated item parameters can be regarded as covariates measured with error. Along the line of this measurement‐error‐model approach, asymptotic expansions of the maximum likelihood estimator (MLE) and weighted likelihood estimator (WLE) of ability were derived by Zhang, Xie, Song, and Lu (2007). In this paper, we propose an estimator of an ability parameter based on the asymptotic formula of the WLE. A simulation study shows that the new estimator effectively reduces the bias of the MLE or WLE of ability caused by the uncertainty of the item parameter estimates not taken into account.
EXAMINATION OF FATIGUE EFFECTS FROM EXTENDED‐TIME ACCOMMODATIONS ON THE SAT REASONING TEST™
Authors: Laitusis Cara Cahalan , Morgan Deanna L. , Bridgeman Brent , Zanna Jennifer , Stone Elizabeth 
Published: August 2014

ABSTRACTThis study examined operational data from the SAT Reasoning Test™ to determine if students who tested under extended‐time conditions were suffering from excessive fatigue relative to students who tested under standard‐time conditions. Excessive fatigue was defined by significant (a) increases in differential item functioning (DIF) and (b) decreases in item completion rates, for items at the end of testing compared to the beginning of testing. Both of these factors were examined by comparing the performance of students who tested under standard time to students who tested with extended time on items administered early (section position 2 or 3) and different items administered late (section position 8, 9, or 10) during the 10‐section test administration. The sample included students with learning disabilities and/or Attention‐Deficit/Hyperactivity Disorder (ADHD) who tested with extended time (time and a half or double time) and students without disabilities who tested under standard‐time conditions. Analyses were conducted on the critical reading and writing sections of the SAT® and examined item difficulty as well as item completion rates. Results indicated few changes in levels of DIF (early in the test compared to late in the test). In addition, item completion rates for students who received extra time were comparable to (or in some cases higher than) test‐takers without disabilities who tested under standard time on both early and late sections.
AN ILLUSTRATION OF THE USE OF MARKOV DECISION PROCESSES TO REPRESENT STUDENT GROWTH (LEARNING)
Authors: Almond Russell G. 
Published: August 2014

ABSTRACTOver the course of instruction, instructors generally collect a great deal of information about each student. Integrating that information intelligently requires models for how a student's proficiency changes over time. Armed with such models, instructors can filter the data—more accurately estimate the student's current proficiency levels—and forecast the student's future proficiency levels. The process of instructional planning can be described as a partially observed Markov decision process (POMDP). Recently developed computer algorithms can be used to help instructors create strategies for student achievement and identify at‐risk students. Implementing this vision requires models for how instructional actions change student proficiencies. The general action model (also called the bowtie model) separately models the factors contributing to the success or effectiveness of an action, proficiency growth when the action is successful, and proficiency growth when the action is unsuccessful. This class of models requires parameterization, and this paper presents two: a simple linear process model (suitable for continuous proficiencies) and a birth‐and‐death process model (for proficiency scales expressed as ordered categorical variables). Both models show how to take prerequisites and zones of proximal development into account. The filtering process is illustrated using a simple artificial example.
ANALYTIC SCORING OF TOEFL® CBT ESSAYS: SCORES FROM HUMANS AND E‐RATER®
Authors: Lee Yong‐Won , Gentile Claudia , Kantor Robert 
Published: August 2014

ABSTRACTThe main purpose of the study was to investigate the distinctness and reliability of analytic (or multitrait) rating dimensions and their relationships to holistic scores and e‐rater® essay feature variables in the context of the TOEFL® computer‐based test (CBT) writing assessment. Data analyzed in the study were analytic and holistic essay scores provided by human raters and essay feature variable scores computed by e‐rater (version 2.0) for two TOEFL CBT writing prompts. It was found that (a) all of the six analytic scores were not only correlated among themselves but also correlated with the holistic scores, (b) high correlations obtained among holistic and analytic scores were largely attributable to the impact of essay length on both analytic and holistic scoring, (c) there may be some potential for profile scoring based on analytic scores, and (d) some strong associations were confirmed between several e‐rater variables and analytic ratings. Implications are discussed for improving the analytic scoring of essays, validating automated scores, and refining e‐rater essay feature variables.
INVESTIGATING THE CRITERION‐RELATED VALIDITY OF THE TOEFL® SPEAKING SCORES FOR ITA SCREENING AND SETTING STANDARDS FOR ITAS
Authors: Xi Xiaoming 
Published: August 2014

ABSTRACTAlthough the primary use of the speaking section of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT Speaking test) is to inform admissions decisions at English medium universities, it may also be useful as an initial screening measure for international teaching assistants (ITAs). This study provides criterion‐related validity evidence for the use of TOEFL iBT Speaking for ITA screening and evaluates the effectiveness of using the scores for teaching assistantship (TA) assignment classification.Four universities participated in this study. Local ITA‐screening tests or instructor recommendations were used as the criterion measures. Relationships between the TOEFL Speaking test and the local ITA tests were explored through observed and disattenuated correlations. These relationships were moderately strong, supporting the use of the TOEFL Speaking test for ITA screening. However, the strengths of the relationship between the TOEFL Speaking test and the local ITA tests were found to be somewhat different across universities depending on the extent to which the local test engaged and evaluated nonlanguage abilities. Implications of these findings are discussed.Binary and ordinal logistic regressions were used to investigate how effective TOEFL Speaking scores were in separating students into distinct TA assignment categories. At all four universities, TOEFL Speaking scores were significant predictors of students' TA assignments and were fairly accurate in classifying students for TA assignments. ROC curves were used to determine TOEFL Speaking cut scores for TA assignments at each university that would minimize false positives (i.e., true nonpasses classified as passes).The results have considerable potential value in providing guidance on using the TOEFL iBT Speaking scores for ITA screening.
LINKING FOR THE GENERAL DIAGNOSTIC MODEL
Authors: Xu Xueli , von Davier Matthias 
Published: August 2014

ABSTRACTThree strategies for linking two consecutive assessments are investigated and compared by analyzing reading data for the National Assessment of Educational Progress (NAEP) using the general diagnostic model. These strategies are compared in terms of marginal and joint expectations of skills, joint probabilities of skill patterns, and item parameter estimates. The results indicate that fixing item parameter values at their previously calibrated values is sufficient to establish a comparable scale for the subsequent year.
COMPARING ALTERNATIVE KERNELS FOR THE KERNEL METHOD OF TEST EQUATING: GAUSSIAN, LOGISTIC, AND UNIFORM KERNELS
Authors: Lee Yi‐Hsuan , von Davier Alina A. 
Published: August 2014

ABSTRACTThe kernel equating method (von Davier, Holland, & Thayer, 2004) is based on a flexible family of equipercentile‐like equating functions that use a Gaussian kernel to continuize the discrete score distributions. While the classical equipercentile, or percentile‐rank, equating method carries out the continuization step by linear interpolation, in principle the kernel equating methods could use various kernel smoothings to replace the discrete score distributions.This paper expands the work of von Davier et al. (2004) in investigating alternative kernels for equating practice. To examine the influence of different kernel functions on the equating results, this paper focuses on two types of kernel functions: the logistic kernel and the continuous uniform distribution (known to be the same as the linear interpolation). The Gaussian kernel is used for reference. By employing an equivalent‐groups design, the results of the study indicate that the tail properties of kernel functions have great impact on the continuized score distributions. However, the equated scores based on different kernel functions do not vary much, except for extreme scores.The results presented in this paper not only support the previous findings on the efficiency and accuracy of the existing continuization methods, but also enrich the information on observed‐score equating models.
A DEVELOPMENTAL WRITING SCALE
Authors: Attali Yigal , Powers Don 
Published: August 2014

ABSTRACTThis report describes the development of grade norms for timed‐writing performance in two modes of writing: persuasive and descriptive. These norms are based on objective and automatically computed measures of writing quality in grammar, usage, mechanics, style, vocabulary, organization, and development. These measures are also used in the automated essay scoring system e‐rater® V.2. Norms were developed through a large‐scale data collection effort that involved a national sample of 170 schools, more than 500 classes from 4th, 6th, 8th, 10th, and 12th grades and more than 12,000 students. Personal and school background information was also collected. These students wrote (in 30‐minute sessions) up to 4 essays (2 in each mode of writing) on topics selected from a pool of 20 topics.The data allowed us to explore a range of questions about the development and nature of writing proficiency. Specifically, this paper provides a description of the trajectory of development in writing performance from 4th grade to 12th grade. The validity of a single developmental writing scale is examined through a human scoring experiment and a longitudinal study. The validity of the single scale is further explored through a factor analysis (exploratory and confirmatory) of the internal structure of writing performance and changes in this structure from 4th grade to 12th grade. The paper also explores important factors affecting performance, including prompt difficulty, writing mode, and student background (gender, ethnicity, and English language background).
ROBUSTNESS OF A VALUE‐ADDED ASSESSMENT OF SCHOOL EFFECTIVENESS
Authors: Braun Henry , Qu Yanxuan , Trapani Catherine 
Published: August 2014

ABSTRACTThis paper reports on a study conducted to investigate the consistency of the results between 2 approaches to estimating school effectiveness through value‐added modeling. Estimates of school effects from the layered model employing item response theory (IRT) scaled data are compared to estimates derived from a discrete growth model based on the analysis of transitions along an ordinal developmental scale. The data were extracted from the longitudinal records maintained in the Early Childhood Longitudinal Study—Kindergarten Cohort (ECLS‐K) archive for students remaining in the same school from the beginning of kindergarten through the end of Grade 3. The results of different comparisons indicated that the estimates from the 2 approaches are moderately consistent.