bluuebunny/crossref_metadata_2025_split huggingface dataset:
THE CBAL READING ASSESSMENT: AN APPROACH FOR BALANCING MEASUREMENT AND LEARNING GOALS
Authors: Sheehan Kathleen M. , O'Reilly Tenaha 
Published: August 2014

ABSTRACTNo Child Left Behind has highlighted the need for new types of assessments that not only provide high‐quality evidence about what students know and can do, but also help to move learning forward. This paper describes a linked set of formative and summative reading assessments designed to address the tradeoffs inherent in these two goals. Targeted skills include the full range of competencies underlying proficient reading at the middle‐school level, including both lower‐order skills such as oral reading fluency and decoding, and higher‐order skills such as the ability to integrate and synthesize information from multiple texts. Data collected in pilot administrations of two prototype test forms are presented. Analyses suggest that this new approach yields acceptable measurement properties while simultaneously addressing crucial learning outcomes. This paper was presented June 23, 2009 at the National Conference on Student Assessment, Los Angeles, CA.
CBAL: RESULTS FROM PILOTING INNOVATIVE K–12 ASSESSMENTS
Authors: Bennett Randy Elliot 
Published: August 2014

ABSTRACTCBAL, an acronym for Cognitively Based Assessment of, for, and as Learning, is a research initiative intended to create a model for an innovative K–12 assessment system that provides summative information for policy makers, as well as formative information for classroom instructional purposes. This paper summarizes empirical results from 16 CBAL summative assessment pilots involving almost 10,000 online administrations conducted between 2007 and 2010. The results suggest that, on average, the CBAL initiative was successful in building innovative assessments in reading, writing, and mathematics that worked as intended. However, there was considerable variation in the functioning of test forms, especially in writing and math. That variation might suggest that the knowledge needed to produce high‐quality forms of these innovative tests in a replicable and scalable manner is not yet at hand. Further, although the results described offer a significant start, many critical questions must be answered before CBAL assessments (or ones like them) are ready for high‐stakes operational use.
COMPUTER‐ADAPTIVE TESTING FOR STUDENTS WITH DISABILITIES: A REVIEW OF THE LITERATURE
Authors: Stone Elizabeth , Davey Tim 
Published: August 2014

ABSTRACTThere has been an increased interest in developing computer‐adaptive testing (CAT) and multistage assessments for K‐12 accountability assessments. The move to adaptive testing has been met with some resistance by those in the field of special education who express concern about routing of students with divergent profiles (e.g., some students with math‐based learning disabilities may have difficulty with basic computation but not high level problem solving) and poor performance on early test questions. This paper consists of a literature review focusing on adaptive testing issues for students with disabilities in the K‐12 sector. While it is clear that there are issues that will present obstacles to administering accountability tests adaptively to students with disabilities, this synthesis of research and policy developments with respect to this topic will be useful both for development of research agendas and to inform states that are currently using or are considering moving to CAT.
A DIFFERENTIAL WORD USE MEASURE FOR CONTENT ANALYSIS IN AUTOMATED ESSAY SCORING
Authors: Attali Yigal 
Published: August 2014

ABSTRACTThis paper proposes an alternative content measure for essay scoring, based on the difference in the relative frequency of a word in high‐scored versus low‐scored essays. The differential word use (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high‐scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE® issue), across different prompts. Evaluation results across four GRE and TOEFL® tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt‐specific content analysis measures of e‐rater®. Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e‐rater features. This measure provides a way to conduct a task‐level content analysis that is related to the prompt‐specific content analysis of e‐rater but does not require prompt‐specific training. Regression results for the prompt‐level DWU show that it can be used as a replacement to prompt‐specific content‐vector analysis (CVA) features.
IMPROVING TEST SCORE REPORTING: PERSPECTIVES FROM THE ETS SCORE REPORTING CONFERENCE
Authors: Zapata‐Rivera Diego , Zwick Rebecca , Hullman Jessica , Rhodes Rebecca , Rodriguez Fernando , Shah Priti , Zwick Rebecca , Sklar Jeffrey C. , Zapata‐Rivera Diego 
Published: August 2014

ABSTRACTThis volume includes 3 papers based on presentations at a workshop on communicating assessment information to particular audiences, held at Educational Testing Service (ETS) on November 4th, 2010, to explore some issues that influence score reports and new advances that contribute to the effectiveness of these reports. Jessica Hullman, Rebecca Rhodes, Fernando Rodriguez, and Priti Shah present the results of recent research on graph comprehension and data interpretation, especially the role of presentation format, the impact of prior quantitative literacy and domain knowledge, the trade‐off between reducing cognitive load and increasing active processing of data, and the affective influence of graphical displays. Rebecca Zwick and Jeffrey Sklar present the results of the Instructional Tools in Educational Measurement and Statistics for School Personnel (ITEMS) project, funded by the National Science Foundation and conducted at the University of California, Santa Barbara to develop and evaluate 3 web‐based instructional modules intended to help educators interpret test scores. Zwick and Sklar discuss the modules and the procedures used to evaluate their effectiveness. Diego Zapata‐Rivera presents a new framework for designing and evaluating score reports, based on work on designing and evaluating score reports for particular audiences in the context of the CBAL (Cognitively Based Assessment of, for, and as Learning) project (Bennett & Gitomer, 2009), which has been applied in the development and evaluation of reports for various audiences including teachers, administrators and students.
ADVANCING THE QUALITY AND EQUITY OF EDUCATION FOR LATINO STUDENTS: A WHITE PAPER
Authors: Young John W. , Lakin Joni , Courtney Rosalea , Martiniello María , Adler Rachel , Blood Ian , Burrus Jeremy , DiCrecchio Nicole , Elliott Diane , Miller Susanne 
Published: August 2014

ABSTRACTThis white paper provides an overview of the issues that affect the quality and equity of education in grades K–16 for Latino students in the United States. This paper is organized chronologically to reflect the typical educational timeline for students in the United States, and we focused on several key transition points in the educational timeline: from elementary school to middle school, from middle school to high school, from high school to college, and from a two‐year college to a four‐year college. Each of these transition points can pose challenges for every student, but in this paper, we focused specifically on the unique challenges that Latino students encounter. It is our hope that this paper will provide an impetus to action and will serve as a guide for future directions for collective research efforts on the critical issues regarding the education of Latino students in the 21st century.
MODELING CHANGE IN LARGE‐SCALE LONGITUDINAL STUDIES OF EDUCATIONAL GROWTH: FOUR DECADES OF CONTRIBUTIONS TO THE ASSESSMENT OF EDUCATIONAL GROWTH
Authors: Rock Donald A. 
Published: August 2014

ABSTRACTThis paper provides a history of ETS's role in developing assessment instruments and psychometric procedures for measuring change in large‐scale national assessments funded by the Longitudinal Studies branch of the National Center for Education Statistics. It documents the innovations developed during more than 30 years of working with longitudinal studies based on national samples. In addition to describing these innovations, it provides their rationale and presents some empirical results in their support.
A REVIEW OF ETS DIFFERENTIAL ITEM FUNCTIONING ASSESSMENT PROCEDURES: FLAGGING RULES, MINIMUM SAMPLE SIZE REQUIREMENTS, AND CRITERION REFINEMENT
Authors: Zwick Rebecca 
Published: August 2014

ABSTRACTDifferential item functioning (DIF) analysis is a key component in the evaluation of the fairness and validity of educational tests. The goal of this project was to review the status of ETS DIF analysis procedures, focusing on three aspects: (a) the nature and stringency of the statistical rules used to flag items, (b) the minimum sample size requirements that are currently in place for DIF analysis, and (c) the efficacy of criterion refinement. The main findings of the review are as follows:• The ETS C rule often displays low DIF detection rates even when samples are large.• With improved flagging rules in place, minimum sample size requirements could probably be relaxed. In addition, updated rules for combining data across administrations could allow DIF analyses to be performed in a broader range of situations.• Refinement of the matching criterion improves detection rates when DIF is primarily in one direction but can depress detection rates when DIF is balanced. If nothing is known about the likely pattern of DIF, refinement is advisable.Each of these findings is discussed in detail, focusing on the case of dichotomous items.
DOES PREEQUATING WORK? AN INVESTIGATION INTO A PREEQUATED TESTLET‐BASED COLLEGE PLACEMENT EXAM USING POSTADMINISTRATION DATA
Authors: Gao Rui , He Wei , Ruan Chunyi 
Published: August 2014

ABSTRACTIn this study, we investigated whether preequating results agree with equating results that are based on observed operational data (postequating) for a college placement program. Specifically, we examined the degree to which item response theory (IRT) true score preequating results agreed with those from IRT true score postequating and from observed score equating. Three academic subjects were examined in this study: analyzing and interpreting literature, American government, and college algebra. The findings suggested that differences between equating results from IRT true score preequating and postequating varied from subject to subject. In general, IRT true score postequating agreed with IRT true score preequating for most of the forms for a test subject. Any difference among the equating results can be attributed to the way through which items were pretested, contextual/order effects, or the violation of IRT assumptions.
A NOTE ON THE CHOICE OF AN ANCHOR TEST IN EQUATING
Authors: Sinharay Sandip , Haberman Shelby , Holland Paul , Lewis Charles 
Published: August 2014

ABSTRACTAnchor tests play a key role in test score equating. We attempt to find, through theoretical derivations, an anchor test with optimal item characteristics. The correlation between the scores on a total test and on an anchor test is maximized with respect to the item parameters for data satisfying several item response theory models. Results suggest that under these models, the minitest, the traditionally used anchor test, is not optimal with respect to anchor‐test‐to‐total‐test correlation; instead, an anchor test with items of medium difficulty, the miditest, seems to be the optimum anchor test. This finding agrees with the empirical findings of Sinharay and colleagues that the miditest mostly has higher anchor‐test‐to‐total‐test correlation compared to the minitest and mostly performs as well as the minitest in equating.
IDENTIFYING SPEECH ACTS IN E‐MAILS: TOWARD AUTOMATED SCORING OF THE TOEIC® E‐MAIL TASK
Authors: Felice Rachele De , Deane Paul 
Published: August 2014

ABSTRACTThis study proposes an approach to automatically score the TOEIC® Writing e‐mail task. We focus on one component of the scoring rubric, which notes whether the test‐takers have used particular speech acts such as requests, orders, or commitments. We developed a computational model for automated speech act identification and tested it on a corpus of TOEIC responses, achieving up to 79.28% accuracy. This model represents a positive first step toward the development of a more comprehensive scoring model. We also created a corpus of speech act‐annotated native English workplace e‐mails. Comparisons between these and the TOEIC data allow us to assess whether English learners are approximating native models and whether differences between native and non‐native data can have negative consequences in the global workplace.
EXAMINING LINGUISTIC CHARACTERISTICS OF PARAPHRASE IN TEST‐TAKER SUMMARIES
Authors: Burstein Jill , Flor Michael , Tetreault Joel , Madnani Nitin , Holtzman Steven 
Published: August 2014

ABSTRACTThis annotation study is designed to help us gain an increased understanding of paraphrase strategies used by native and nonnative English speakers and how these strategies might affect test takers' essay scores. Toward that end, this study aims to examine and analyze the paraphrase and the types of linguistic modifications used in paraphrase in test‐taker responses and differences that may exist between native and nonnative English speakers. We are also interested in how these factors might influence final essay score. Outcomes discussed in this report can be used to inform the development of new e‐rater® scoring engine features that capture information related to paraphrase, specifically in nonnative speaker responses to the TOEFL® exam integrated writing task.
FROM BIOLOGY TO EDUCATION: SCORING AND CLUSTERING MULTILINGUAL TEXT SEQUENCES AND OTHER SEQUENTIAL TASKS
Authors: Sukkarieh Jana Z. , von Davier Matthias , Yamamoto Kentaro 
Published: August 2014

ABSTRACTThis document describes a solution to a problem in the automatic content scoring of the multilingual character‐by‐character highlighting item type. This solution is language independent and represents a significant enhancement. This solution not only facilitates automatic scoring but plays an important role in clustering students' responses; consequently, it has a nontrivial impact on the refinement of the items and/or their scoring guidelines. Furthermore, though designed for a specific problem, the proposed solution is general enough for any educational task that can be transformed into a sequential one. To name a few: It can be used for a set of actions expected from a student in simulations or learning trajectories as projected by a teacher, inside an intelligent tutoring system, or even in a game—or it can simply be used for a set of student clicks, button selections, or keyboard hits expected to reach a correct answer. This solution provides flexibility for existing automatic‐scoring techniques and potentially could provide more flexibility if coupled with statistical data‐mining techniques.
LENGTH OF TEXTUAL RESPONSE AS A CONSTRUCT‐IRRELEVANT RESPONSE STRATEGY: THE CASE OF SHELL LANGUAGE
Authors: Bejar Isaac I. , VanWinkle Waverely , Madnani Nitin , Lewis William , Steier Michael 
Published: August 2014

ABSTRACTThe paper applies a natural language computational tool to study a potential construct‐irrelevant response strategy, namely the use of shell language. Although the study is motivated by the impending increase in the volume of scoring of students responses from assessments to be developed in response to the Race to the Top initiative, the data for the study were obtained from the GRE® Analytical Writing measure. The functioning of the shell detection computational tool was first evaluated by applying it to a corpus of over 200,000 issue and argument essays and by means of a study to evaluate whether the shell language score agreed with the characterization of shell by two scoring experts. It was concluded that the computational tool worked well. The tool was then used to select essays for rescoring to determine whether the presence of shell language had had an effect on the operational scores they received. We found no evidence that such an effect was present. However, we did find a leniency effect in the operational scores. That is, the essays that were rescored as part of this project received a lower score compared to the operational score. The validity implications of these results are discussed.
IDENTIFYING THE MOST IMPORTANT 21ST CENTURY WORKFORCE COMPETENCIES: AN ANALYSIS OF THE OCCUPATIONAL INFORMATION NETWORK (O*NET)
Authors: Burrus Jeremy , Jackson Teresa , Xi Nuo , Steinberg Jonathan 
Published: August 2014

ABSTRACTTo identify the most important competencies for college graduates to succeed in the 21st century workforce, we conducted an analysis of the Occupational Information Network (O*NET) database. O*NET is a large job analysis operated and maintained by the U.S. Department of Labor. We specifically analyzed ratings of the importance of abilities (52 ratings), work styles (16 ratings), skills (35 ratings), and knowledge (33 ratings) to succeed in one's occupation. First, we conducted descriptive analyses. Next, data were split into 2 sets, according to the theoretical structure proposed by the O*NET content model, and principal component analyses (PCAs) were run on each dataset. The PCAs identified 15 components: problem solving, mechanical skills, service orientation, cultural literacy, business literacy, science literacy, civic literacy, information processing, athleticism, visual acuity, fluid intelligence, communication skills, teamwork, achievement/innovation, and attention to detail/near vision. Components were then ranked in importance using the mean component scores over all occupations. A comparison of this ranking with previous 21st century competencies frameworks suggested that 5 competencies stand out as important for most occupations: problem solving (e.g., complex problem solving), fluid intelligence (e.g., category flexibility), teamwork (e.g., cooperation), achievement/innovation (e.g., persistence), and communication skills (e.g., oral expression). Consistent with this conclusion, a correlation of component scores with wages found that 4 of these 5 competencies were strongly related to wages, with the exception being teamwork.
EXAMINING THE EFFICACY OF A TIME MANAGEMENT INTERVENTION FOR HIGH SCHOOL STUDENTS
Authors: Burrus Jeremy , Jackson Teresa , Holtzman Steven , Roberts Richard D. , Mandigo Terri 
Published: August 2014

ABSTRACTThe current paper reports the results of 2 quasiexperimental studies conducted to examine the efficacy of a new time management intervention designed for high school students. In both studies, there was no difference between the treatment and control groups in improvement in self‐reported time management skills as a result of the intervention. However, the treatment group reported significantly greater improvement than the control group for secondary outcomes such as stress (Studies 1 and 2), anxiety (Studies 1 and 2), depression (Study 1), and knowledge of time management strategies (Study 1). Additionally, advisor ratings of student time management skills were higher for the treatment than for the control group in Study 2. Implications and suggestions for improving the intervention are discussed.
CONSTRUCTED‐RESPONSE DIF EVALUATIONS FOR MIXED‐FORMAT TESTS
Authors: Moses Tim , Liu Jinghua , Tan Adele , Deng Weiling , Dorans Neil J. 
Published: August 2014

ABSTRACTIn this study, differential item functioning (DIF) methods utilizing 14 different matching variables were applied to assess DIF in the constructed‐response (CR) items from 6 forms of 3 mixed‐format tests. Results suggested that the methods might produce distinct patterns of DIF results for different tests and testing programs, in that the DIF methods' results might be similar for tests with multiple‐choice (MC) and CR scores that are similar in their measurement characteristics but would exhibit larger variations for tests with MC and CR scores having more distinct measurement characteristics. Impact measures of the MC and CR scores appeared to be a useful basis for indicating the scores' measurement similarity, for predicting the variations of DIF results from using these scores as matching variables, and possibly for indicating the most appropriate DIF method and matching variable for a particular test. The results are described in terms of their implications for research and practice.
COLLABORATIVE PROBLEM SOLVING AND THE ASSESSMENT OF COGNITIVE SKILLS: PSYCHOMETRIC CONSIDERATIONS
Authors: von Davier Alina A. , Halpin Peter F. 
Published: August 2014

ABSTRACTCollaboration is generally recognized as a core competency of today's knowledge economy and has taken a central role in recent theoretical and technological developments in education research. Yet, the methodology for assessing the learning benefits of collaboration continues to rely on educational tests designed for isolated individuals. Thus, what counts as evidence of learning does not correspond to current best practices for teaching, and it does not reflect what students are ultimately expected to be able to do with their knowledge. The goals of this paper are to give an overview of the research conducted in several fields of work related to collaboration, propose a framework for the assessment of cognitive skills (such as science or math) through collaborative problem‐solving tasks, and propose several statistical approaches to model the data collected from collaborative interactions. This research contributes to the knowledge needed to support a new generation of assessments based on collaboration.
Volcanic stratigraphy of the Quaternary La Garrotxa Volcanic Field (north‐east Iberian Peninsula)
Authors: BOLÓS XAVIER , PLANAGUMÀ LLORENÇ , MARTÍ JOAN 
Published: August 2014

ABSTRACTThe monogenetic Quaternary La Garrotxa Volcanic Field forms part of the Catalan Volcanic Zone (north‐east Iberian Peninsula), one of the alkaline volcanic provinces of the European rift system. It harbours more than 50 basaltic monogenetic cones that range in age from the Middle Pleistocene to the Early Holocene and include cinder and scoria cones, lava flows, tuff rings and maars. This study is the result of extensive fieldwork, including the study of ephemeral outcrops and the stratigraphic logging of new water wells and geotechnical drill holes, also taking into account existing information gathered by recent geophysical studies that have applied shallow geophysical methods to establish the substrate geology of this volcanic field. We have obtained a comprehensive volcanic stratigraphy of the area that identifies the products of each single eruption, their relative stratigraphy and their surface area. This volcanic stratigraphy constitutes an essential tool for understanding the evolution of this volcanic field and for establishing a correct volcanic hazard assessment for the area, but it also provides a precise reference for the Quaternary tephrochronology of the lake sediments in neighbouring areas.