National Evaluation of Title III Implementation Supplemental Report: Exploring Approaches to Setting English Language Proficiency Performance Criteria and Monitoring English Learner Progress. Draft
Author: Cook, Gary
Date: 2012-02-00
Type: Numerical/Quantitative Data
Subjects: Civil Rights,Elementary Secondary Education,Federal Legislation,Civil Rights Legislation,State Standards,Second Language Learning,Criteria,Standard Setting,Educational Legislation,English (Second Language),Language Proficiency,Accountability,Technical Assistance,Guidelines,Data Analysis,Research Methodology,Academic Achievement,Benchmarking,Matrices,Achievement Gains,Decision Making

The Elementary and Secondary Education Act (ESEA), as amended by the No Child Left Behind Act of 2001 inaugurated important changes in assessment and accountability for English Learner (EL) students. Specifically, Title III of the law required states to develop or adopt English-language proficiency (ELP) standards aligned with language demands of academic content standards. An annually administered ELP assessment based on those standards was also required by the ESEA (NCLB 2002). Title III also instituted new accountability requirements for districts and states. These new EL accountability provisions required states to define criteria for progress in learning English, establish a performance standard for English proficiency, and set annually increasing performance targets for the number and percentage of ELs meeting these criteria. As has been well documented, the new law's requirements exceeded the technical capacity of many states and districts to comply with it (Abedi 2004; Government Accountability Office 2006). Over the past several years, empirical research with more rigorous ELP assessments, systematic technical assistance efforts, and federal guidance have helped to reduce confusion and increase coherence in state Title III accountability systems (Abedi 2007; Linquanti and George 2007; Cook et al. 2008; Federal Register 2008). Nevertheless, a significant need remains to develop the capacity of state and technical assistance providers to utilize empirical data for performance standard setting and accountability policy development in these areas. Even as ESEA reauthorization draws closer, prevailing civil rights laws and sustained focus on improving EL education suggest these policy-making needs will grow in importance, particularly given broad adoption of Common Core State Standards and establishment of related multistate academic and ELP assessment consortia. This document is intended to contribute to that capacity development by describing and illustrating several empirical methods and conceptual/theoretical rationales to help state policy-makers, standard-setting panels, and the technical advisory panels and assistance providers to (1) determine a meaningful ELP performance standard; (2) establish a realistic, empirically anchored time frame for attaining a given ELP performance standard; and (3) take into account an EL's ELP level when setting academic progress and proficiency expectations. This is &quot;by design&quot; a technical document intended to assist those charged with providing empirical information germane to developing or revising EL accountability models, using ELP and academic assessments. Chapter I positions data-analysis methods illustrated in the report within a larger deliberative process of setting meaningful, ambitious, and realistic performance standards and accountability criteria for EL students. The chapter offers guidelines for enacting best practices in standard setting, and highlights limitations in using empirical data. Chapter II illustrates three methods (decision consistency, logistic regression, and descriptive box plots) for analyzing empirical data to assist policy-makers in determining an ELP performance standard for English Learners. Chapter III illustrates methods for conducting empirical analyses to inform setting expected time frames for EL students to attain the ELP performance standard. Finally, Chapter IV discusses two methods for taking into account an EL's ELP level when setting academic progress and proficiency expectations, and one method that explicitly ignores it. First, progressive benchmarking methods are illustrated that adjust either EL students' content achievement scale scores or their weight (individual &quot;count&quot;), based on each student's ELP level relative to their initial ELP level and time in the state school system. Second, an indexed progress method utilizes ELs' ELP growth as a proxy for English language arts performance on a weighted, time-sensitive basis for more newly-arrived ELs who enter the state's school system at lower initial ELP levels. Third, a status and growth accountability matrix method credits both a predetermined level of student academic growth as well as attainment of academic proficiency, without considering an English Learner's ELP level. Each method is carefully described and applied using the same education agency's sample data set. All the approaches presented in this document--many of which have been employed by the principal authors in working with states on their EL accountability systems--are intended to stimulate discussion and further exploration of additional methods among state data analysts, technical assistance providers, and researchers. The ultimate goal is to support the development and regular use of empirical methods that inform ambitious, realistic, and meaningful performance standards and accountability policies, which will foster EL students' linguistic and academic progress and attainment. Appended are: (1) Decision Consistency Method; (2) Education Agency 1; (3) Education Agency 2; (4) Education Agency 3; (5) Event History Analysis; and (6) Education Agency 1. Individual chapters contain footnotes. (Contains 76 exhibits.)