bluuebunny/crossref_metadata_2025_split huggingface dataset:
AUTOMATED TOOLS FOR SUBJECT MATTER EXPERT EVALUATION OF AUTOMATED SCORING
Authors: Williamson David M. , Bejar Isaac I. , Sax Anne 
Published: August 2014

ABSTRACTAs automated scoring of complex constructed‐response examinations reaches operational status, the process of evaluating the quality of resultant scores, particularly in contrast to scores of expert human graders, becomes as complex as the data itself. Using a vignette from the Architectural Registration Examination (ARE), this paper explores the potential utility of classification and regression trees (CART) and Kohonen self‐organizing maps (SOM) as tools to facilitate subject matter expert (SME) examination of the fine‐grained (feature level) quality of automated scores for complex data, with implications for the validity of the resultant scores. The paper explores both supervised and unsupervised learning techniques, the former being represented by CART (Breiman, Friedman, Olshen, & Stone, 1984) and the latter by SOM (Kohonen, 1989). Three applications comprise this investigation, the first of which suggests that CART can facilitate efficient and economical identification of specific elements of complex solutions that contribute to automated and human score discrepancies. The second application builds on the first by exploring CART's value for efficiently and accurately automating case selection for human intervention to ensure score validity. The final application explores the potential for SOM to reduce the need for SMEs in evaluating automated scoring. While both supervised and unsupervised methodologies examined were found to be promising tools for facilitating SME roles in maintaining and improving the quality of automated scoring, such applications remain unproven and further studies are necessary to establish the reliability of these techniques.
TOEIC®/LPI RELATIONSHIPS IN ACADEMIC AND EMPLOYMENT CONTEXTS IN THAILAND
Authors: Wilson Kenneth M. , Nagara Supalak Komarakul Na , Woodhead Robert 
Published: August 2014

ABSTRACTThe Test of English for International Communication (TOEIC®) is an ETS‐developed measure of proficiency in English as a second language (ESL) or foreign language (EFL), designed to assess aspects of English language listening comprehension (LC) and reading comprehension (RC). In general samples of native speakers of Japanese, Korean, French, Spanish, and Arabic, respectively, scores on the TOEIC have been found to be strongly related to performance on the Language Proficiency Interview (LPI) procedure, a widely recognized direct measure of speaking proficiency. In this exploratory study, TOEIC/LPI relationships in subgroups (e.g., by gender, educational level, organizational setting at time of assessment) were assessed within a general national/linguistic sample—native speakers of Thai who are involved in operational assessments under the aegis of the TOEIC representative agency in Thailand, the Center for Professional Assessment. Observed relationships were relatively strong and positive in the total sample and within each of the subgroups studied. Observed coefficients were noticeably larger in subgroups of females than in corresponding subgroups of males and for a sample of university students than for examinees tested in places of employment. Means of observed LPI rating for the respective subgroups were consistent with expectations based on TOEIC scores. Further research is needed to assess the extent to which the subgroup findings may tend to be generalizable in the Center (Thailand) context or elsewhere.
STATISTICAL AND MEASUREMENT PROPERTIES OF FEATURES USED IN ESSAY ASSESSMENT
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTStatistical and measurement properties are examined for features used in essay assessment to determine the generalizability of the features across populations, prompts, and individuals. Data are employed from TOEFL® and GMAT® examinations and from writing for CriterionSM.
COMPARABILITY OF TOEFL CBT WRITING PROMPTS FOR DIFFERENT NATIVE LANGUAGE GROUPS
Authors: Lee Yong‐Won , Breland Hunter , Muraki Eiji 
Published: August 2014

ABSTRACTThis study has investigated the comparability of computer‐based testing (CBT) writing prompts in the Test of English as a Foreign Language™ (TOEFL®) for examinees of different native language backgrounds. A total of 81 writing prompts introduced from July 1998 through August 2000 were examined using a three‐step logistic regression procedure for ordinal items. An English language ability (ELA) variable was created by summing the standardized TOEFL Reading, Listening, and Structure scale scores. This ELA variable was used to match examinees of East Asian (Chinese, Japanese, and Korean) and European (German, French, and Spanish) language groups. Although about one third of the 81 prompts were initially flagged because of statistically significant group effects, the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect.
EXAMINING FREEDLE'S CLAIMS ABOUT BIAS AND HIS PROPOSED SOLUTION: DATED DATA, INAPPROPRIATE MEASUREMENT, AND INCORRECT AND UNFAIR SCORING
Authors: Dorans Neil J. , Zeller Karin 
Published: August 2014

ABSTRACTIn the Spring 2003 issue of Harvard Educational Review, Roy Freedle stated that the SAT® is both culturally and statistically biased, and he proposed a solution to ameliorate this bias. His claims, which garnered national attention, were based on serious errors in his analysis. We begin our analyses by assessing the psychometric properties of Freedle's recommended hard half‐test that he thinks should form the basis for the supplemental SAT score he proposes to report. Next we demonstrate his justification for a score based on this half‐test is based on a flawed analysis. The numbers in his critical Table 2 do not represent what he claims they do. We then demonstrate what occurs when current data are used with both the correct scoring and the incorrect scoring proposed by Freedle. When the table is constructed correctly using current data, the effects that Freedle reported are reduced substantially in magnitude to the point where they do not warrant any of the corrective actions he proposes. We conclude our analysis of Freedle's claims with DIF analyses of our own that are inconsistent with his DIF analyses of old SAT test editions he used as his initial motivation for correcting the SAT scoring procedure.
THE BEST LINEAR PREDICTOR FOR TRUE SCORE FROM A DIRECT ESTIMATE AND SEVERAL DERIVED ESTIMATES
Authors: Haberman Shelby J. , Qian Jiahe 
Published: August 2014

ABSTRACTStatistical prediction problems often involve both a direct estimate of a true score and covariates of this true score. Given the criterion of mean squared error, this study determines the best linear predictor of the true score given the direct estimate and the covariates. Results yield an extension of Kelley's formula for estimation of the true score to cases in which covariates are present. The best linear predictor is a weighted average of the direct estimate and of the linear regression of the direct estimate onto the covariates. The weights depends on the reliability of the direct estimate and on the multiple correlation of the true score with the covariates. One application of the best linear predictor is to approximate the human true score from the observed holistic score of an essay and from essay features derived from a computer analysis.
POPULATION INVARIANCE AND LINEAR EQUATING FOR THE NON‐EQUIVALENT GROUPS DESIGN
Authors: von Davier Alina A. , Han Ning 
Published: August 2014

ABSTRACTThis study investigates the population sensitivity of the commonly used linear equating methods in the Non‐Equivalent‐groups with an Anchor Test (NEAT) design: the Tucker, the Levine observed‐score, and the chain linear methods. For a detailed analysis of the subject, we apply three distinctive approaches to a real data set from a NEAT design: a) the RMSD index for the NEAT design of von Davier, Holland, and Thayer (2004); b) the parallel‐linking system of Dorans and Holland (2000), and c) the pseudo‐NEAT design approach of von Davier, Holland, and Thayer (2003).The data set is used to illustrate the derivations on male and female subpopulations. Comparisons of the results obtained using linear and equipercentile equating are also presented. For this particular data set the Levine function seems to vary less across subpopulations than the other methods; the Tucker method seems to vary the most.
USER'S GUIDE FOR SCORIGHT (VERSION 3.0): A COMPUTER PROGRAM FOR SCORING TESTS BUILT OF TESTLETS INCLUDING A MODULE FOR COVARIATE ANALYSIS
Authors: Wang Xiaohui , Bradlow Eric T. , Wainer Howard 
Published: August 2014

ABSTRACTSCORIGHT is a very general computer program for scoring tests. It models tests that are made up of dichotomously or polytomously rated items or any kind of combination of the two through the use of a generalized item response theory (IRT) formulation. The items can be presented independently or grouped into clumps of allied items (testlets) or in any combination of the two. When there are testlets, the program assesses the degree of local dependence and adjusts the estimates accordingly. The estimation is accomplished within a fully Bayesian framework using Markov chain Monte Carlo procedures, which allows the easy calculation of many important characteristics of the scores that are not available with other methods. The current version of SCORIGHT, version 3.0, includes a new module that allows the user to include covariates in the analysis.
COGNITIVE DIAGNOSIS FOR NAEP PROFICIENCY DATA
Authors: Xu Xueli , von Davier Matthias 
Published: August 2014

ABSTRACTMore than a dozen statistical models have been developed for the purpose of cognitive diagnosis. These models are supposed to extract a much finer level of information from item responses than traditional unidimensional item response models. In this paper, a general diagnostic model (GDM) was used to analyze a set of simulated sparse data and real data from National Assessment of Educational Progress (NAEP) assessments. The purpose of this study was to find out (a) whether the parameters can be recovered for a sparse data matrix in the framework of the GDM and (b) how to estimate group characteristics for large survey data, such as NAEP, in the framework of the GDM. The results of the simulation study show that GDM parameters can be recovered satisfactorily. The GDM under single group and multiple group assumptions were employed to fit NAEP assessment data. The results under these two assumptions and between the GDM and the statistics derived using the operational NAEP model were compared. The findings indicate that it is possible to conduct cognitive diagnosis for NAEP proficiency data.
ADAPTIVE TECHNOLOGIES
Authors: Shute Valerie J. , Zapata‐Rivera Diego 
Published: August 2014

ABSTRACTThis paper describes research and development efforts related to adaptive technologies, which can be combined with other technologies and processes to form an adaptive system. The goal of an adaptive system, in the context of this paper, is to create an instructionally sound and flexible environment that supports learning for students with a range of abilities, disabilities, interests, backgrounds, and other characteristics. After defining key terms and establishing a rationale for adaptation, we present a general framework to organize adaptive technologies. We then describe experts' thoughts on what to adapt and how to adapt and conclude with a summary of key challenges and potential futures of adaptive technologies.
AN ENTROPY‐BASED MEASURE OF DEPENDENCE BETWEEN TWO GROUPS OF RANDOM VARIABLES
Authors: Kong Nan 
Published: August 2014

ABSTRACTIn multivariate statistics, the linear relationship among random variables has been fully explored in the past. This paper looks into the dependence of one group of random variables on another group of random variables using (conditional) entropy. A new measure, called the K‐dependence coefficient or dependence coefficient, is defined using (conditional) entropy. This paper shows that the K‐dependence coefficient is a measure of the degree of dependence of one group of random variables on another group of random variables. The dependence measured by the K‐dependence coefficient includes both linear and nonlinear dependencies between two groups of random variables. Therefore, the K‐dependence coefficient measures the total degree of dependence and not just the linear component of the dependence between the two groups of random variables. Furthermore, the concept of the K‐dependence coefficient is extended by defining the partial K‐dependence coefficient and the semipartial K‐dependence coefficient. Properties of partial and semipartial K‐dependence coefficients are also explored.
IMPACT OF GROUP WORK AND EXTENDED ESSAY WRITING ON ONLINE ADVANCED PLACEMENT PROGRAM® STUDENT PERFORMANCE
Authors: Handwerk Phil 
Published: August 2014

ABSTRACTOnline high schools are growing significantly in number, popularity, and function. However, little empirical data has been published about the effectiveness of these institutions. This research examined the frequency of group work and extended essay writing among online Advanced Placement Program® (AP®) students, and how these tasks may have impacted their AP exam performance. Hierarchical, set‐wise regression models found little to no impact of group work or extended essay writing on multiple‐choice, free response, or composite scores across 9 AP exams administered from 2003–2005. These models included student sex, race, and year in school as independent variables; covariates were the number of previous AP exams taken by the student and PSAT/NMSQT® scores. Implications for online providers and for future research of online high schools are discussed.
MONOTONE PROPERTIES OF A GENERAL DIAGNOSTIC MODEL
Authors: Xu Xueli 
Published: August 2014

ABSTRACTMonotonicity properties of a general diagnostic model (GDM) are considered in this paper. Simple data summaries are identified to inform about the ordered categories of latent traits. The findings are very much in accordance with the statements made about the GPCM (Hemker, Sijtsma, Molenaar, & Junker, 1996, 1997). On the one hand, by fitting a GDM with equal slopes across items, the observed total score X+ = Σj:qjk=1 xj demonstrates the monotone likelihood ratio (MLR) property in individual coordinate at the cost of losing model fit. On the other hand, fitting a GDM without slope restriction increases the model fit but by sacrificing MLR property. Trade‐offs between these two situations should be considered in practice.
AN ASSESSMENT FOR LEARNING SYSTEM CALLED ACED: DESIGNING FOR LEARNING EFFECTIVENESS AND ACCESSIBILITY
Authors: Shute Valerie J. , Hansen Eric G. , Almond Russell G. 
Published: August 2014

ABSTRACTThis paper reports on a 3‐year, NSF‐funded research and development project called ACED: Adaptive Content with Evidence‐based Diagnosis. The purpose of the project was to design, develop, and evaluate an assessment for learning (AfL) system for diverse students, using Algebra I content related to geometric sequences (i.e., successive numbers linked by a common ratio). A key feature of the project was its dual focus on both learning effectiveness and accessibility. The project consisted of two studies. Study 1 (N = 268) experimentally evaluated the impact on learning of two ACED features: (a) elaborated task feedback and (b) adaptive sequencing of tasks. Study 2 examined the accessibility of ACED for students with visual disabilities (N = 4; two blind and two low‐vision)—in a condition involving both adaptive sequencing and elaborated feedback—using a variety of accessibility features. Available features included voicing of test content (via synthesized and prerecorded audio), talking tactile graphics, font enlargement, screen magnification, and so on. Together these studies lay groundwork for the design of AfL systems that can enhance the learning of individuals with and without disabilities.
MIXTURE DISTRIBUTION DIAGNOSTIC MODELS
Authors: von Davier Matthias 
Published: August 2014

ABSTRACTThis paper introduces the mixture general diagnostic model (MGDM), an extension of the general diagnostic model (GDM). The MGDM extension allows one to estimate diagnostic models for multiple known populations as well as discrete unknown, or not directly observed mixtures of populations. The GDM is based on developments that integrate located latent class models; multiple classification latent class models; and discrete, multidimensional item response models into a common framework. Models of this type express the probability of a response vector as a function of parameters that describe the individual item response variables in terms of required skills and of indirectly observed (latent) skill profiles of respondents. The skills required for solving the items are, as in most diagnostic models, represented as a design matrix that is often referred to as a Q‐matrix. This Q‐matrix consists of rows describing, for each item response, what combination of skills is needed to succeed or to obtain partial or full credit. The hypothesized Q‐matrix is either the result of experts rating items of an existing assessment (retrofitting) or comes directly out of the design of the assessment instrument, in which it served as a tool to design the items.The MGDM takes the GDM and integrates it into the framework of discrete mixture distribution models for item response data (see von Davier & Rost, 2006). This increases the utility of the GDM by allowing the estimation and testing of models for multiple populations. The MGDM allows for complex scale linkages that make assessments comparable across populations and makes it possible to test whether items function the same in different subpopulations. This can be done with known subpopulations (defined by grade levels, cohorts, etc.), as well as with unknown subpopulations that need to be identified by the model. In both cases, MGDMs make it possible to determine whether different sets of item‐by‐skill parameters and/or different skill distributions have to be assumed for different subpopulations. This amounts to a generalized procedure that can be used to test for differential item functioning (DIF) on one item or on multiple‐response variables using multiple‐group or mixture models. This procedure enables testing DIF models against models that allow additional skills for certain items in order to account for differences between subpopulations.
TIME REQUIREMENTS FOR THE DIFFERENT ITEM TYPES PROPOSED FOR USE IN THE REVISED SAT®
Authors: Bridgeman Brent , Laitusis Cara Cahalan , Cline Frederick 
Published: August 2014

ABSTRACTThe current study used three data sources to estimate time requirements for different item types on the now current SAT Reasoning Test™. First, we estimated times from a computer‐adaptive version of the SAT® (SAT CAT) that automatically recorded item times. Second, we observed students as they answered SAT questions under strict time limits and recorded the amount of time taken for each question. Finally, we asked high school students to record the amount of time taken for test subsections that were composed of items of a single type. The rules of thumb used by test developers were quite accurate in rank ordering the item types from least to most time‐consuming, but the time actually spent was generally higher than assumed in the rules of thumb.
GROWTH IN READING PERFORMANCE DURING THE FIRST FOUR YEARS IN SCHOOL
Authors: Rock Donald A 
Published: August 2014

ABSTRACTThis study addressed concerns about the potential for differential gains in reading during the first 2 years of formal schooling (K‐1) versus the next 2 years of schooling (1s‐3rd grade). A multilevel piecewise regression with a node at spring 1st grade was used in order to define separate regressions for the two time periods. Empirical Bayes estimates of individual growth curves were developed, and smoothed estimates were plotted for selected demographic groups.
AN ALTERNATIVE DATA COLLECTION DESIGN FOR EQUATING WITH VERY SMALL SAMPLES
Authors: Puhan Gautam , Moses Tim , Grant Mary , McHale Fred 
Published: August 2014

ABSTRACTA single group (SG) equating design with nearly equivalent test forms (SiGNET) design was developed by Grant (2006) to equate small volume tests. The basis of this design is that examinees take two largely overlapping test forms within a single administration. The scored items for the operational form are divided into mini‐tests called testlets. An additional testlet is created but not scored for the first form. If the scored testlets are Testlets 1‐6 and the unscored testlet is Testlet 7, then the first form is composed of Testlets 1‐6, the second form is composed of Testlets 2‐7, and Testlets 2‐6 are common to both test forms. They are administered as a single administered form, and when a sufficient number of examinees have taken the administered form for an SG equating, the second form (Testlets 2‐7) is equated to the first form (Testlets 1‐6) using SG equating. As evident, there are at least two merits of the SiGNET design over the nonequivalent groups with anchor test (NEAT) design. First, it facilitates the use of an SG equating design, which has the least random equating error, and second, it allows for the accumulation of sufficient data to equate the second form. Since the examinees scores are based on only the first form (i.e., the operational form), the two forms can be administered until sufficient data are collected to equate the second form. This study compared equatings under the SiGNET and NEAT designs and found reduced bias and error for the SiGNET design in very small sample size situations (e.g., N = 10 or 15). Implications for practice using the SiGNET design are also discussed.
MODEL‐BASED WEIGHTING AND COMPARISONS
Authors: Qian Jiahe 
Published: August 2014

ABSTRACTIn survey research, sometimes the formation of groupings, or aggregations of cases on which to make an inference, are of importance. Of particular interest are the situations where the cases aggregated carry useful information that has been transferred from a sample employed in a previous study. For example, a school to be included in the sample of the High School Effectiveness (HSES) study must contain one or more cases transferred from the National Educational Longitudinal Study of 1988 (NELS:88). To calculate the aggregation inclusion probabilities, this study investigated three statistical models and, based on these models, derived the school weights for the HSES study. This study also assessed the effects of weighting by comparing the statistics yielded from different sets of weights: (a) those from an empirical population database and (b) those from data generated from simulation based on the principles of a superpopulation. Both categorical data and continuous variables were analyzed in the comparison.
AUTOMATED SCORING OF SHORT‐ANSWER OPEN‐ENDED GRE® SUBJECT TEST ITEMS
Authors: Attali Yigal , Powers Don , Freedman Marshall , Harrison Marissa , Obetz Susan 
Published: August 2014

ABSTRACTThis report describes the development, administration, and scoring of open‐ended variants of GRE® Subject Test items in biology and psychology. These questions were administered in a Web‐based experiment to registered examinees of the respective Subject Tests. The questions required a short answer of 1‐3 sentences, and responses were automatically scored by natural language processing methods, using the c‐rater™ scoring engine, immediately after participants submitted their responses. Participants received immediate feedback on the correctness of their answers, and an opportunity to revise their answers. Subsequent human scoring of the responses allowed an evaluation of the quality of automated scoring. This report focuses on the success of the automated scoring process. A separate report describes the feedback and revision results.