bluuebunny/crossref_metadata_2025_split huggingface dataset:
SCORE COMPARABILITY FOR LANGUAGE MINORITY STUDENTS ON THE CONTENT ASSESSMENTS USED BY TWO STATES
Authors: Young John W. , Holtzman Steven , Steinberg Jonathan 
Published: August 2014

ABSTRACTIn this research investigation of score comparability for language minority students (English language learners [ELLs] and former English language learners), we examined 3 indicators of score comparability (reliability, internal test structure, and differential item functioning) for 4th and 8th grade students who took the NCLB‐mandated content assessments in English‐language arts and mathematics in 2 different U. S. states. Overall, for the 8 assessments we examined, a high degree of score comparability was found for ELLs and former ELLs, when compared with native English speakers. The results from this study showed that although the assessments from the 2 states differed somewhat with respect to the 3 indicators, a high degree of score comparability was found for both states' content assessments.
FIT OF ITEM RESPONSE THEORY MODELS: A SURVEY OF DATA FROM SEVERAL OPERATIONAL TESTS
Authors: Sinharay Sandip , Haberman Shelby J. , Jia Helena 
Published: August 2014

ABSTRACTStandard 3.9 of theStandards for Educational and Psychological Testing(American Educational Research Association, American Psychological Association, & National Council for Measurement in Education, 1999) demands evidence of model fit when an item response theory (IRT) model is used to make inferences from a data set. We applied two recently suggested methods for assessing goodness of fit of IRT models—generalized residual analysis (Haberman, 2009) and residual analysis for assessing item fit (Bock & Haberman, 2009)—to several operational data sets. We assessed the practical significance of misfit whenever possible. This report summarizes our findings. Though evidence of misfit of the IRT model was found for all the data sets, the misfit was not always practically significant.
THE RELATIONSHIP BETWEEN RATERS' PRIOR LANGUAGE STUDY AND THE EVALUATION OF FOREIGN LANGUAGE SPEECH SAMPLES
Authors: Winke Paula , Gass Susan , Myford Carol 
Published: August 2014

ABSTRACTThis study investigated whether raters' second language (L2) background and the first language (L1) of test takers taking the TOEFL iBT® Speaking test were related through scoring. After an initial 4‐hour training period, a group of 107 raters (mostly of learners of Chinese, Korean, and Spanish), listened to a selection of 432 speech samples that 72 test takers (native speakers of Chinese, Korean, and Spanish) produced. We analyzed the rating data using a multifaceted Rasch measurement approach to uncover potential biases in the rating process. In addition, 26 of the raters participated in stimulated recall sessions, during which they watched videos of themselves rating. Using the video as a prompt, we asked them to discuss and explain their rating processes at the time of rating. The results from our bias interaction analyses revealed that matches between the raters' L2 and the test takers' L1 resulted in some of the raters assigning ratings that were significantly higher than expected. As a whole, raters with Spanish as an L2 were significantly more lenient toward test takers who had Spanish as an L1, and raters with Chinese as an L2 were significantly more lenient toward test takers who had Chinese as an L1. Analyses of the qualitative data, assisted by the program QSR NVivo 8, revealed information concerning the raters' awareness of their biases.
THE SINGLE GROUP WITH NEARLY EQUIVALENT TESTS (SIGNET) DESIGN FOR EQUATING VERY SMALL VOLUME MULTIPLE‐CHOICE TESTS
Authors: Grant Mary C. 
Published: August 2014

ABSTRACTThe single group with nearly equivalent tests (SiGNET) design proposed here was developed to address the problem of equating scores on multiple‐choice test forms with very small single‐administration samples. In this design, the majority of items in each new test form consist of items from the previous form, and the new items that were administered as unscored items in the previous form. Each form is equated using data from examinees who took the previous form. As the equating is a single‐group design, with the 2 forms having a large number of overlapping items, the size of the equating sample can be much smaller than in other designs.
CONSTRUCTED‐RESPONSE MATHEMATICS TASKS STUDY
Authors: Fife James H. , Graf Edith Aurora , Ohls Sarah 
Published: August 2014

ABSTRACTSix tasks, selected from assessments administered in 2007 as part of the Cognitively Based Assessment of, for, and as Learning (CBAL) project, were revised in an effort to remove difficulties with the tasks that were unrelated to the construct being assessed. Because the revised tasks were piloted on a different population from the original tasks, it was not possible to make direct comparisons between the performance of the revised tasks and that of the original tasks, other than to make a qualitative assessment of whether or not the nonconstruct difficulties had, in fact, been removed. But we were able to pilot between 2 and 4 versions of each revised task, and we could compare the performance of our pilot sample on the various versions of each task. For Mix It Up, we prepared 2 nonparallel versions—the first attempted to preserve the construct‐related difficulty of the original while removing the nonconstruct‐related ambiguities, and the second was intended to be an easier task that measured the same skills and abilities. For Fruit Drink and Paste we created 4 versions, carefully varying different aspects of the language while keeping other aspects constant. For the 2 tasks from Bigfoot, we varied 2 features independently, creating 2 versions of each feature and therefore 4 versions of each task. Finally, for Forest Carbon, we created 4 versions, varying from unscaffolded to carefully scaffolded. Because the revision of each task was its own experiment, the analysis of each task, and our conclusions from that analysis, are described separately.
EQUIVALENCY OF THE DINA MODEL AND A CONSTRAINED GENERAL DIAGNOSTIC MODEL
Authors: von Davier Matthias 
Published: August 2014

ABSTRACTThis report shows that the deterministic‐input noisy‐AND (DINA) model is a special case of more general compensatory diagnostic models by means of a reparameterization of the skill space and the design (Q‐) matrix of item by skills associations. This reparameterization produces a compensatory model that is equivalent to the (conjunctive) DINA model, and is valid for all types of complex structure Q‐matrices, not only for trivial cases. This equivalency uses the GDM as the basis, is not based on recent developments of diagnosis models such as G‐DINA or LCDM. Model equivalency is a topic of some relevance as soon as researchers want to draw conclusions derived from any particular model‐based estimates. It can be shown that for multidimensional models, there are often multiple ways to specify different sets of latent variables and their relationships to observed variables. This report goes beyond showing that multiple versions of a design matrix lead to the same model‐based conditional probability space; it shows that a conjunctive diagnostic classification model can be expressed as a constrained special case of a compensatory diagnostic modeling framework.
THE SENSITIVITY OF PARAMETER ESTIMATES TO THE LATENT ABILITY DISTRIBUTION
Authors: Xu Xueli , Jia Yue 
Published: August 2014

ABSTRACTEstimation of item response model parameters and ability distribution parameters has been, and will remain, an important topic in the educational testing field. Much research has been dedicated to addressing this task. Some studies have focused on item parameter estimation when the latent ability was assumed to follow a normal distribution, whereas others have utilized nonparametric or semiparametric techniques to substitute the normal ability assumption. However, both approaches have their limitations. A normal ability assumption is not flexible enough to reflect possible deviations from symmetry, whereas the nonparametric and semiparametric techniques used to capture possible nonnormal features of the latent ability have difficulty in reaching satisfactory estimates for certain quantities of the ability distribution such as quantiles. Hence a continuous generalized skew normal (GSN) distribution was applied in this study to better capture the possible underlying asymmetric ability distribution. In addition, simultaneous estimation of both the item parameters and the distributional parameters was employed. The performance of the GSN was compared with the normal ability assumption in terms of item parameter and distributional parameter recoveries, based on a series of simulation studies. The results showed that (a) under the Rasch model, both the item parameter estimates and the distributional parameter estimates are robust to the misspecification of the ability distribution, and (b) under the two‐parameter logistic model, although the distributional parameter estimates are fairly robust, the item parameter estimates are slightly more sensitive to the misspecification of the ability distribution, especially when the underlying ability distribution is highly skewed.
A GENERAL PROCEDURE TO ASSESS THE INTERNAL STRUCTURE OF A NONCOGNITIVE MEASURE—THE STUDENT360 INSIGHT PROGRAM (S360) TIME MANAGEMENT SCALE
Authors: Ling Guangming , Rijmen Frank 
Published: August 2014

ABSTRACTThe factorial structure of the Time Management (TM) scale of the Student 360: Insight Program (S360) was evaluated based on a national sample. A general procedure with a variety of methods was introduced and implemented, including the computation of descriptive statistics, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA). Overall, the results indicated that the TM scale measured multidimensional constructs of TM with 5 factors. The paper concludes with a discussion of several issues concerning the wording of items and residual dependencies, as well as future directions for research.
HOW DOES THE KNOWLEDGE OF SUBGROUP MEMBERSHIP OF EXAMINEES AFFECT THE PREDICTION OF TRUE SUBSCORES?
Authors: Haberman Shelby J. , Sinharay Sandip 
Published: August 2014

ABSTRACTSubscores are reported for several operational assessments. Haberman (2008) suggested a method based on classical test theory to determine if the true subscore is predicted better by the corresponding subscore or the total score. Researchers are often interested in learning how different subgroups perform on subtests. Stricker (1993) and Livingston and Rupp (2004) found that the mean difference between the subgroups was not the same for the different subscores. We suggest new methods to investigate whether the quality of prediction of the true subscore improves if the investigator knows about the subgroup membership. We applied our suggested method to data from 4 operational testing programs. We found that the quality of prediction of the true subscore does not improve if the investigator knows the subgroup membership. We also found that whether the subscores have value added does not depend on the subgroups.
EVALUATION OF THE E‐RATER® SCORING ENGINE FOR THE GRE® ISSUE AND ARGUMENT PROMPTS
Authors: Ramineni Chaitanya , Trapani Catherine S. , Williamson David M. , Davey Tim , Bridgeman Brent 
Published: August 2014

ABSTRACTAutomated scoring models for the e‐rater® scoring engine were built and evaluated for the GRE® argument and issue‐writing tasks. Prompt‐specific, generic, and generic with prompt‐specific intercept scoring models were built and evaluation statistics such as weighted kappas, Pearson correlations, standardized difference in mean scores, and correlations with external measures were examined to evaluate the e‐rater model performance against human scores. Performance was also evaluated across different demographic subgroups. Additional analyses were performed to establish appropriate agreement thresholds between human and e‐rater scores for unusual essays and the impact of using e‐rater on operational scores. The generic e‐rater scoring model with operational prompt‐specific intercept for the issue‐writing task and prompt‐specific e‐rater scoring model for the argument writing task were recommended for operational use. The two automated scoring models were implemented to produce check scores at a discrepancy threshold of 0.5 with human scores.
EFFECTIVE PRACTICES FOR DEVELOPING THE LITERACY SKILLS OF ENGLISH LANGUAGE LEARNERS IN THE ENGLISH LANGUAGE ARTS CLASSROOM
Authors: Turkan Sultan , Bicknell Jerome , Croft Andrew 
Published: August 2014

ABSTRACTThis paper is a review of literature presenting instructional strategies—based on normative as well as empirical arguments—which have proven to be effective in envisioning whatallteachers need to know and be able to do to teach English language arts (ELA) to English language learners (ELLs). The studies selected for review address what is particular to teaching ELA to ELLs. The paper is divided into two main sections: (a) teachers' linguistic practices and (b) teachers' pedagogical practices. In the first section, we report on the studies that analyze teachers' understanding of linguistics and present implications for their instruction of ELLs. Three areas of effective practice are emphasized based on the particular aspects of teaching ELA to ELLs. The first area is that teachers should recognize that literacy skills in ELLs' native languages might influence the ways in which ELLs process linguistic information in English. The second area highlights the argument that teachers should find ways to facilitate ELLs' mastery of academic vocabulary. The third area covers the significance of enhancing ELLs' metacognitive reading skills. In the second section, on teacher pedagogical practices, we discuss two broad pedagogical skills that emerge from both the normative and empirical studies reviewed and are closely related: (a) the teachers' ability to help ELLs construct meaning from the texts or speech represented in the ELA classroom and (b) the teachers' ability to engage ELLs in actively learning to read and write. The paper ends with a summary and a brief reflective statement on the limitations of the review of the literature.
EXAMINING THE IMPACT OF DRIFTED POLYTOMOUS ANCHOR ITEMS ON TEST CHARACTERISTIC CURVE (TCC) LINKING AND IRT TRUE SCORE EQUATING
Authors: Li Yanmei 
Published: August 2014

ABSTRACTIn a common‐item (anchor) equating design, the common items should be evaluated for item parameter drift. Drifted items are often removed. For a test that contains mostly dichotomous items and only a small number of polytomous items, removing some drifted polytomous anchor items may result in anchor sets that no longer resemble mini‐versions of the new and old test forms. In this study, the impact of drifted polytomous anchor items on the test characteristic curve (TCC) linking and item response theory (IRT) true score equating for a test containing only a small number of polytomous items was investigated. Simulated tests were constructed to mimic a real large‐scale test. The magnitude of the item parameter drift, anchor length, number of drifted polytomous items in the anchor set, and the ability distributions of the groups taking the old form and new form were manipulated. Results suggest that anchor length and number of drifted polytomous items had a relatively large impact on the linking and equating results. The accuracy of linking and equating results were affected by the magnitude of item parameter drift. The ability distributions of the groups had little effect on the linking and equating results. In general, excluding drifted polytomous anchor items resulted in an improvement in equating results.
A NOTE ON EXPLAINING AWAY AND PARADOXICAL RESULTS IN MULTIDIMENSIONAL ITEM RESPONSE THEORY
Authors: van Rijn Peter W. , Rijmen Frank 
Published: August 2014

ABSTRACTHooker and colleagues addressed a paradoxical situation that can arise in the application of multidimensional item response theory (MIRT) models to educational test data. We demonstrate that this MIRT paradox is an instance of the explaining‐away phenomenon in Bayesian networks, and we attempt to enhance the understanding of MIRT models by placing the paradox in a broader statistical modeling perspective.
WHAT CAN REPEATED CROSS‐SECTIONAL STUDIES TELL US ABOUT STUDENT GROWTH?
Authors: Almond Russell G. , Sinharay Sandip 
Published: August 2014

ABSTRACTTo answer questions about how students' proficiencies are changing over time, educational researchers are looking for data sources that span many years. Clearly, for answering questions about student growth, a longitudinal study—in which a single sample is followed over many years—is preferable to repeated cross‐sectional samples—in which a separate sample is taken every year. Repeated cross‐sectional studies, such as the National Assessment of Educational Progress (NAEP), however, are often readily available. Repeated cross‐sectional studies conflate several sources of variability (differences in the initial status of individuals, individual differences in the growth curves, and individual‐by‐measurement‐occasion differences) in ways that are not easily separated. Although repeated cross‐sectional studies can provide information about the growth of the averages, the growth of the averages corresponds to the average of the growth curves only in very restricted circumstances. This paper reviews the literature on modeling growth with an eye to characterizing the limitations of repeated cross‐sectional studies and understanding the sensitivity of the results to key decisions (particularly, choices of cut points). In most cases, repeated cross‐sectional studies should be used to confirm and contextualize the results of more targeted longitudinal studies.
TOPICAL TRENDS IN A CORPUS OF PERSUASIVE WRITING
Authors: Heilman Michael , Madnani Nitin 
Published: August 2014

ABSTRACTMany writing assessments use generic prompts about social issues. However, we currently lack an understanding of how test takers respond to such prompts. In the absence of such an understanding, automated scoring systems may not be as reliable as they could be and may worsen over time. To move toward a deeper understanding of responses to generic issue prompts, we analyzed topical trends in test takers' responses and correlated these trends with those found in the news. We found evidence that many trends are similar across essays and the news. However, we also observed some interesting differences. Based on these analyses, we make recommendations in this paper for developers of writing assessments and automated scoring systems.
STATISTICAL REPORT OF 2011 CBAL™ MULTISTATE ADMINISTRATION OF READING AND WRITING TESTS
Authors: Fu Jianbin , Wise Maxwell 
Published: August 2014

ABSTRACTIn the Cognitively Based Assessment of, for, and as Learning (CBAL™) research initiative, innovative K‐12 prototype tests based on cognitive competency models are developed. This report presents the statistical results of the 2 CBAL Grade 8 writing tests and 2 Grade 7 reading tests administered to students in 20 states in spring 2011. Specifically, classical item statistics including rater reliabilities for human‐scored items, item p+ values, item‐total correlations, item response times, item omit and not‐reached rates, system error rates, differential item functioning, interscore correlations, and reliabilities of subscores and total scores are reported. In addition, t‐tests, multiple comparisons, correlations, and mixed models are used to examine the factors influencing test scores, including test form, test order, and various background variables at the student, teacher, and school levels. The results show that these 4 tests performed reasonably well.
EXPLORING ALTERNATIVE TEST FORM LINKING DESIGNS WITH MODIFIED EQUATING SAMPLE SIZE AND ANCHOR TEST LENGTH
Authors: Wang Lin , Qian Jiahe , Lee Yi‐Hsuan 
Published: August 2014

ABSTRACTThe purpose of this study was to evaluate the combined effects of reduced equating sample size and shortened anchor test length on item response theory (IRT)‐based linking and equating results. Data from two independent operational forms of a large‐scale testing program were used to establish the baseline results for evaluating the results from two alternative designs. Under the two alternative designs, two simulated conditions were created from the original data. Under one condition, we reduced the equating sample size (from about 2,000 to about 1,000) per anchor item and shortened the anchor test length (by half) per equating sample. Under the other condition, we reduced the sample size (from about 2,000 to about 1,000) per anchor item only. A complete grouped jackknife replication method was used to estimate the standard errors of the linking and equating procedures from 100 jackknife replicate samples; the complete procedures included IRT calibrations, item parameter scaling, and IRT true score equating. The findings from a comparison of the results from the two simulated conditions and the baseline results showed that neither alternative design had any practical impact on the linking and equating results for either test form.
A CRITERION TO EVALUATE THE INDIVIDUAL RAW‐TO‐SCALE EQUATING CONVERSIONS
Authors: Guo Hongwen , Puhan Gautam , Walker Michael 
Published: August 2014

ABSTRACTIn this study we investigated when an equating conversion line is problematic in terms of gaps and clumps. We suggest using the conditional standard error of measurement (CSEM) to measure the scale scores that are inappropriate in the overall raw‐to‐scale transformation.
DIMENSIONALITY ANALYSIS OF CBAL™ WRITING TESTS
Authors: Fu Jianbin , Chung Seunghee , Wise Maxwell 
Published: August 2014

ABSTRACTThe Cognitively Based Assessment of, for, and as Learning (CBAL™) research initiative is aimed at developing an innovative approach to K‐12 assessment based on cognitive competency models. Because the choice of scoring and equating approaches depends on test dimensionality, the dimensional structure of CBAL tests must be understood. The purpose of this study was to investigate the dimensionality of 4 CBAL Grade 8 writing tests. Each of the 4 tests focused on one of the following writing genres: persuasive letter/memorandum, research‐based report (pamphlet), persuasive essay, and literature essay. Dimensionality was investigated using exploratory and confirmatory factor analyses. The results show multidimensionality and support subscore structures and bifactor task models for all 4 tests.
POSTSTRATIFICATION EQUATING BASED ON TRUE ANCHOR SCORES AND ITS RELATIONSHIP TO LEVINE OBSERVED SCORE EQUATING
Authors: Chen Haiwen (Henry) , Livingston Samuel A. 
Published: August 2014

ABSTRACTThis paper presents a new equating method for the nonequivalent groups with anchor test design: poststratification equating based on true anchor scores. The linear version of this method is shown to be equivalent, under certain conditions, to Levine observed score equating, in the same way that the linear version of poststratification equating is equivalent to Tucker equating. Some issues related to this result are discussed.