bluuebunny/crossref_metadata_2025_split huggingface dataset:
OVERESTIMATION OF LPI RATINGS FOR NATIVE‐KOREAN SPEAKERS IN THE TOEIC TESTING CONTEXT: SEARCH FOR EXPLANATION
Authors: Wilson Kenneth M. , Stupak Steven A. 
Published: August 2014

ABSTRACTRegression‐based guidelines have been developed for predicting ratings of speaking proficiency in English as a foreign language (EFL), as assessed using the Language Proficiency Interview (LPI) procedure, from scores on the TOEIC® (Test of English for International Communication) Test. These guidelines, which reflect the regression of LPI rating on TOEIC scores in a combined sample composed of native speakers of Japanese, French, Spanish, and Arabic, respectively, were found to overestimate LPI rating when applied to data for native speakers of Korean. It was hypothesized that this outcome would be expected if development of EFL speaking proficiency (as assessed by the LPI procedure) tends to lag relatively more behind development of the proficiencies that are assessed by the TOEIC Test, for native‐Korean‐speaking EFL learners than for demographically comparable EFL learners in other national/linguistic settings (at least those represented in the study sample). An indirect assessment of this hypothesis was undertaken in a series of exploratory analyses involving section‐ and total‐score means for native‐language groups on the familiar, three‐part version of the Test of English as a Foreign Language™ (TOEFL®). Findings were interpreted as being generally, albeit indirectly, supportive of the developmental lag hypothesis, as well as of working assumptions underlying introduction and analysis of TOEFL native‐language reference group means as group analytic variables. Needed lines of inquiry are suggested.
COMPARING CONDITIONAL AND MARGINAL DIRECT ESTIMATION OF SUBGROUP DISTRIBUTIONS
Authors: von Davier Matthias 
Published: August 2014

ABSTRACTMany large‐scale assessment programs in education utilize “conditioning models” that incorporate both cognitive item responses and additional respondent background variables relevant for the population of interest. The set of respondent background variables serves as a predictor for the latent traits (proficiencies/abilities) and is used to obtain a conditional prior distribution for these traits. This is done by estimating a linear regression, assuming normality of the conditional trait distributions given the set of background variables. Multiple imputations, or plausible values, of trait parameter estimates are used in addition to or, better, on top of the conditioning model—as a computationally convenient approach to generating consistent estimates of the trait distribution characteristics for subgroups in complex assessments. This report compares, on the basis of simulated and real data, the conditioning method with a recently proposed method of estimating subgroup distribution statistics that assumes marginal normality. Study I presents simulated data examples where the marginal normality assumption leads to a model that produces appropriate estimates only if subgroup differences are small. In the presence of larger subgroup differences that cannot be fitted by the marginal normality assumption, however, the proposed method produces subgroup mean and variance estimates that differ strongly from the true values. Study II extends the findings on the marginal normality estimates to real data from large‐scale assessment programs such as the National Assessment of Educational Progress (NAEP) and the National Adult Literacy Survey (NALS). The research presented in Study II shows differences between the two methods that are similar to the differences found in Study I. The consequences of relying upon the assumption of marginal normality in direct estimation are discussed.
EFFECT OF FEWER QUESTIONS PER SECTION ON SAT® I SCORES
Authors: Bridgeman Brent , Trapani Catherine , Curley Edward 
Published: August 2014

ABSTRACTThe impact of allowing more time for each question on SAT® I: Reasoning Test scores was estimated by embedding sections with a reduced number of questions into the standard 30‐minute equating section of two national test administrations. Thus, for example, questions were deleted from a verbal section that contained 35 questions to produce forms that contained 27 or 23 questions. Scores on the 23‐question section could then be compared to scores on the same 23 questions when they were embedded in a section that contained 27 or 35 questions. Similarly, questions were deleted from a 25‐question math section to form sections of 20 and 17 questions. Allowing more time per question had a minimal impact on verbal scores, producing gains of less than 10 points on the 200–800 SAT scale. Gains for the math score were less than 30 points. High‐scoring students tended to benefit more than lower‐scoring students, with extra time creating no increase in scores for students with SAT scores of 400 or lower. Ethnic/racial and gender differences were neither increased nor reduced with extra time.
A HISTORICAL PERSPECTIVE ON THE CONTENT OF THE SAT®
Authors: Lawrence Ida M. , Rigol Gretchen W. , Essen Thomas Van , Jackson Carol A. 
Published: August 2014

ABSTRACTThis paper provides a historical perspective on the content of the SAT®. The review begins at the beginning, when the first College Board SAT (the “Scholastic Aptitude Test”) was administered to 8,040 students on June 23, 1926. At that time, the SAT consisted of nine subtests: definitions, arithmetical problems, classification, artificial language, antonyms, number series, analogies, logical inference, and paragraph reading. Over the years, the SAT has evolved in the way it measures what is now referred to as verbal and mathematical “reasoning.” With each redesign of the SAT, a variety of considerations were taken into account, including fairness issues, scaling issues, cost, public perception, face validity, changes in the test‐taking population, changes in patterns of test preparation, and changes in the college admissions process. This paper describes the reasons for the various changes while emphasizing that the value of SAT scores rests on the test's high technical quality, and on the assumption that scores would maintain their meaning over time.
APPLYING THE ONLINE SCORING NETWORK (OSN) TO ADVANCED PLACEMENT PROGRAM® (AP®) TESTS
Authors: Zhang Yuli (Lilly) , Powers Donald E. , Wright Wendi , Morgan Rick 
Published: August 2014

ABSTRACTThis project explored the feasibility of using the ETS Online Scoring Network (OSN) to score selected Advanced Placement Program® (AP®) tests. In particular, the quality of scores obtained from traditional large‐scale readings was compared with the quality of scores obtained from OSN scoring. (For the former, readers convene in a central location to evaluate student responses in a paper‐based format; for the latter, they work alone from remote locations to evaluate responses that are transmitted via the Internet.) The study also obtained readers' reactions to OSN training and to OSN capabilities for monitoring reader performance online.The results, based on more than 11,000 test takers who had taken either the AP English Language and Composition exam or the AP Calculus AB exam, revealed little if any difference between scores assigned at traditional central readings and those assigned when readers used OSN. The lack of differences applied to (a) score level and variability, (b) interreader agreement, and (c) passing rates on the two AP exams that were investigated. Study participants did, however, suggest several areas in which improvements could be made to OSN procedures.
EFFECT OF EXTRA TIME ON GRE® QUANTITATIVE AND VERBAL SCORES
Authors: Bridgeman Brent , Cline Frederick , Hessinger James 
Published: August 2014

ABSTRACTThe purpose of this study was to test the assumption that the Graduate Record Examinations (GRE®) General Test is a measure of academic reasoning abilities in which speed of responding plays at most a minor role. In addition to completing the operational GRE General Test, participants each completed a research version of either the GRE verbal or quantitative test within either the standard time limit or within one‐and‐a‐half times the standard time limit. Scores obtained from 15,948 examinees indicate that extra time added about seven points (on the 200‐800 score scale) to examinees' verbal scores and seven points to their quantitative scores. Scores under the different timing conditions were generally comparable across gender and ethnic groups, but quantitative scores were slightly higher for lower‐ability examinees who had more time.
VARIANCE COMPONENT TESTING IN GENERALIZED LINEAR MIXED MODELS
Authors: Sinharay Sandip , Stern Hal S. 
Published: August 2014

ABSTRACTGeneralized linear mixed models (GLMM) are used in situations where a number of characteristics (covariates) affect a nonnormal response variable and the responses are correlated. For example, in a number of biological applications, the responses are correlated due to common genetic or environmental factors. In many applications, the magnitude of the variance components corresponding to one or more of the random effects are of interest, especially the point null hypothesis that one or more of the variance components are zero. This work reviews a number of approaches for estimating the Bayes factor comparing the models with and without the random effects in question. The computations involved with finding Bayes factors for these models pose many challenges, and we discuss how one can overcome them.We perform a comparative study of the different approaches to compute Bayes factors for GLMMs by applying them to two different data sets. The first example employs a probit regression model with a single variance component to data from a natural selection study on turtles. The second example uses a disease mapping model from epidemiology, a Poisson regression model with two variance components being used to analyze the data. The importance sampling method is found to be the method of choice to compute the Bayes factor of interest for these problems. Chib's method is also found to be efficient.
CALIBRATION OF POLYTOMOUS ITEM FAMILIES USING BAYESIAN HIERARCHICAL MODELING
Authors: Johnson Matthew S. , Sinharay Sandip 
Published: August 2014

ABSTRACTFor complex educational assessments, there is an increasing use of item families, which are groups of related items. However, calibration or scoring for such an assessment requires fitting models that take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families with multiple‐choice items. This paper extends the model to take into account item families with polytomous items and designs a Markov chain Monte Carlo (MCMC) algorithm for the Bayesian estimation of the model parameters. The hierarchical model, which accounts for the dependence structure inherent among the items, implicitly defines the family response function for the score categories. This paper suggests a way to combine the family response functions over the score categories to obtain a family score function, which is a quick graphical summary of the expected score of an individual with a certain ability on an item randomly generated from an item family. This paper also suggests a method for the Bayesian estimation of the family response function and family score function. This work is a significant step towards building a tool to analyze data involving item families and may be very useful practically, for example, in automatic item generation systems that create tests involving item families.
NOTES ON LINEAR EQUATING METHODS FOR THE NON‐EQUIVALENT‐GROUPS DESIGN
Authors: von Davier Alina A. 
Published: August 2014

ABSTRACTThis paper focuses on three observed‐score methods for linear equating in a Non‐Equivalent‐groups Anchor Test (NEAT) design—Tucker, Levine observed‐score, and chain. The methods and their assumptions are described, and theoretical conditions under which the three methods give identical results are presented.
WHAT IS QUANTITATIVE REASONING? DEFINING THE CONSTRUCT FOR ASSESSMENT PURPOSES
Authors: Dwyer Carol Anne , Gallagher Ann , Levin Jutta , Morley Mary E. 
Published: August 2014

ABSTRACTIn order to create fair and valid assessments, it is necessary to be clear about what is to be measured and how the resulting data should be interpreted. For a number of historical and practical reasons described in this paper, adequately detailed statements with both a theoretical and empirical base do not currently exist for the construct of quantitative reasoning for use in assessments. There is also no adequate explanation of the important differences between assessments that measure quantitative reasoning constructs and those that are intended to measure achievement in related mathematical content areas.The literature in psychology, psychometrics, philosophy, and education, while containing much that is relevant to the construct of quantitative reasoning, unfortunately does not provide materials that can be used in research and development to address such practical issues or to explain the basic nature of quantitative reasoning assessments. This paper briefly discusses the importance and use of constructs and the quantitative reasoning and standards literature. It then presents a statement about the construct of quantitative reasoning for assessment purposes within a construct validity framework that includes both a definition of the construct and threats to valid score interpretation. These threats are based on related but distinguishable constructs and other types of construct‐irrelevant variance in the assessments.
BAYESIAN ITEM FIT ANALYSIS FOR DICHOTOMOUS ITEM RESPONSE THEORY MODELS
Authors: Sinharay Sandip 
Published: August 2014

ABSTRACTDetecting item fit for common dichotomous item response theory (IRT) models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. This paper employs theposterior predictive model checkingmethod (Guttman, 1967; Rubin, 1981, 1984), a popular Bayesian model checking tool, to examine item fit for common dichotomous item response theory models. An item fit plot, comparing the observed and predicted proportion correct scores of examinee groups (with groups based on raw scores), promises to be useful in real applications. This paper also suggests how to obtain posterior predictive p‐values for the χ2‐type test statistics of Orlando and Thissen (2000) comparing the observed and predicted proportion correct scores for different raw score groups. A number of simulation studies and real data applications examine the effectiveness of the suggested item fit diagnostics. The suggested p‐values seem to have low Type I error rate, low false alarm rate, and adequate power, indicating that researchers might find the methods more acceptable than the existing item fit measures.
A SIMULATION STUDY TO EXPLORE CONFIGURING THE NEW SAT® CRITICAL READING SECTION WITHOUT ANALOGY ITEMS
Authors: Liu Jinghua , Feigenbaum Miriam , Cook Linda 
Published: August 2014

ABSTRACTThis study explored possible configurations of the new SAT® critical reading section without analogy items. The item pool contained items from SAT verbal (SAT–V) sections of 14 previously administered SAT tests, calibrated using the three‐parameter logistic IRT model. Multiple versions of several prototypes that do not contain analogy items were assembled. Item statistics and test statistics for the simulated forms were compared to the average of 13 forms of the SAT–V. These statistics included: IRT scaled score reliability, scaled score standard error of measurement, conditional scaled score standard error of measurement, r‐biserial, and equated deltas. The results indicated that it is possible to maintain measurement precision for the new SAT critical reading section without analogy items, but it may be necessary to modify the distribution of item difficulty in order to obtain adequate precision at the ends of the score scale.
A COMPARISON OF TWO MODELS FOR COGNITIVE DIAGNOSIS
Authors: Yan Duanli , Almond Russell , Mislevy Robert 
Published: August 2014

ABSTRACTDiagnostic score reports linking assessment outcomes to instructional interventions are one of the most requested features of assessment products. There is a body of interesting work done in the last 20 years including Tatsuoka's rule space method (Tatsuoka, 1983), Haertal and Wiley's binary skills model (Haertal, 1984; Haertal & Wiley, 1993), and Mislevy, Almond, Yan, and Steinberg's Bayesian inference network (Mislevy, Almond, Yan, & Steinberg, 1999). Recent research has resulted in major breakthroughs for the use of a parametric IRT model for performing skills diagnoses. Hartz, Roussos, and Stout (2002) have developed an identifiable and flexible model called the fusion model.This paper compares the fusion model and the Bayesian inference network in the design and analysis of an assessment, including theQ‐matrix, and then compares other related models, including rule space, item response theory (IRT), and general multivariate latent class models. The paper also attempts to characterize the kinds of problems for which each type of measurement model is well‐suited. A general Bayesian psychometric framework provides common language, making it easier to appreciate the differences.In addition, this paper explores some of the strengths and weaknesses of each approach based on a comparative analysis of a cognitive assessment, the mixed number subtraction data set. In this case, both the fusion model and Bayesian network approaches yield similar performance characteristics and also seem to pick up on different characteristics.
ELICITED SPEECH FROM GRAPH ITEMS ON THE TEST OF SPOKEN ENGLISH™
Authors: Katz Irvin R. , Xi Xiaoming , Kim Hyun‐Joo , Cheng Peter C‐H. 
Published: August 2014

ABSTRACTThis research applied a cognitive model to identify item features that lead to irrelevant variance on the Test of Spoken English™ (TSE®). The TSE is an assessment of English oral proficiency and includes an item that elicits a description of a statistical graph. This item type sometimes appears to tap graph‐reading skills—an irrelevant construct; TSE raters report that many examinees perform worse on this item type than they do on the other 11 items in the test. We adapted a cognitive theory of graph comprehension to predict the degree to which TSE graph items tap irrelevant skills such as graph reading. Through analyses of existing TSE data as well as an experiment, we show how the theory provides specific, empirically justified recommendations on the construction of graph items that minimize the influence of extraneous skills.
A UNIFIED APPROACH TO IRT SCALE LINKING AND SCALE TRANSFORMATIONS
Authors: von Davier Matthias , von Davier Alina A. 
Published: August 2014

ABSTRACTThis paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non‐Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean‐mean, mean‐var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed‐item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then a rewriting of the log‐likelihood function together with an appropriately implemented maximization procedure of the log‐likelihood function under linear (or nonlinear restrictions) will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models like the generalized partial credit model.
A GRAPHICAL APPROACH TO ITEM ANALYSIS
Authors: Livingston Samuel A. , Dorans Neil J. 
Published: August 2014

ABSTRACTThis paper describes an approach to item analysis that is based on the estimation of a set of response curves for each item. The response curves show, at a glance, the difficulty and the discriminating power of the item and the popularity of each distractor, at any level of the criterion variable (e.g., total score). The curves are estimated by Gaussian kernel smoothing, a weighted moving average process with a parameter that can be varied at the user's discretion. The response curve for the correct answer can be accompanied by curves indicating a confidence region. The response curves also form the basis for estimating item statistics for any group of examinees for which the distribution of the criterion variable is known.
NATIONAL BOARD FOR PROFESSIONAL TEACHING STANDARDS BIAS‐REDUCTION TRAINING: IMPACT ON ASSESSORS' AWARENESS
Authors: Wylie E. Caroline , Szpara Michelle Y. 
Published: August 2014

ABSTRACTThis study is an in‐depth investigation of the National Board for Professional Teaching Standards (NBPTS) bias‐reduction training, from the perspective of assessors. The research examined how successful the bias training was in guiding assessors to recognize their biases and to identify actions to be used to reduce the impact of bias on their scoring decisions. The study focused on portfolio video entries to allow for a greater exploration of bias, since assessors are provided with a visual and aural depiction of the candidates and their students.
ASSESSING COGNITIVE ACHIEVEMENT GROWTH DURING THE KINDERGARTEN AND FIRST GRADE YEARS
Authors: Rock Donald A. , Pollack Judy M. , Weiss Michael 
Published: August 2014

ABSTRACTThis study attempts to identify different patterns of cognitive growth in kindergarten and first grade associated with selected subpopulations. The results are based upon a nationally representative sample of fall kindergartners who were retested in the spring of their kindergarten year and then again in the fall and spring of their first grade year. Of special interest here is the estimation of the growth rates of subpopulations of children that are often considered to be at risk, educationally and/or economically. This study also investigates whether or not the absence of formal schooling during the summer differentially impacts subpopulations who are “at risk” because they may lack a strong educational support system in the home.
COMPARABILITY OF TOEFL CBT WRITING PROMPTS: RESPONSE MODE ANALYSES
Authors: Breland Hunter , Lee Yong‐Won , Muraki Eiji 
Published: August 2014

ABSTRACTEighty‐three Test of English as a Foreign Language™ (TOEFL®) CBT writing prompts that were administered between July 1998 and August 2000 were examined in order to identify differences in scores that could be attributed to the response mode chosen by examinees (handwritten or word processed). Differences were examined statistically using polytomous logistic regression. An English language ability (ELA) variable was developed from the multiple‐choice components of the TOEFL examination and used as a matching variable. Although there was little observed difference in mean writing scores, when examinees were matched on English language ability, small differences were observed in effect sizes consistently favoring the handwritten response mode. The difference favoring the handwritten response mode occurred for all of the writing prompts analyzed; however, the differences for individual writing prompts were small. This difference suggests a general effect for response mode.