A Study of the Effectiveness of a School Improvement Intervention (Success in Sight). Final Report. NCEE 2012-4014
Author: Wilkerson, Stephanie B.
Date: 2012-02-00
Type: Numerical/Quantitative Data
Subjects: Evidence,Experimental Groups,Intervention,Elementary Education,Educational Improvement,Mathematics Achievement,Academic Achievement,Teacher Surveys,Control Groups,Reading Achievement,Program Effectiveness,Educational Legislation,Federal Legislation,School Effectiveness,Low Achievement,Decision Making,Data,Participative Decision Making,Leadership,Effect Size,Comparative Analysis,Educational Indicators,Federal Programs,Educational Strategies,Correlation,Fidelity

Success in Sight focuses on the interrelated parts of an education system. This systemic school improvement intervention is designed to address schools' specific needs while building their capacities to plan, implement, and evaluate school improvement practices. It is intended to help schools, leadership teams, and teachers systemically engage in continuous school improvement practices to advance the learning of all students (Cicchinelli et al. 2006). Schools have used Success in Sight over the past 11 years to facilitate school improvement efforts. However, there have been no cluster randomized trials to provide causal evidence regarding its effectiveness in improving student and teacher outcomes. Therefore, the main purpose of this study was to provide unbiased estimates of the impact of Success in Sight on student academic achievement in reading or mathematics. The achievement outcome areas of reading and mathematics were chosen for this study based on the NCLB mandate that all students should be proficient in reading and mathematics by 2014. Additionally, all states assess reading and mathematics achievement in grades 3-5, which are the focus of this study. The study also sought to provide an unbiased estimate of the effects of Success in Sight on teacher capacity for school improvement practices related to data-based decisionmaking, purposeful community, and shared leadership. This study's target population was low- to moderate-performing elementary schools located in states served by McREL under its Regional Educational Laboratory (REL) contract from the U.S. Department of Education's Institute of Education Sciences (IES) and Comprehensive Center grant from the U.S. Department of Education's Office of Elementary and Secondary Education. This study was the first cluster randomized trial to examine the effectiveness of Success in Sight on primary outcomes--student achievement in reading and mathematics--and intermediate teacher outcomes--city for school improvement practices in data-based decisionmaking, purposeful community, and shared leadership. The results of the benchmark analyses revealed that Success in Sight did not have a statistically significant impact on student achievement in reading or mathematics or on teacher capacity for school improvement practices in data-based decisionmaking, purposeful community, or shared leadership. Although this study used rigorous methodology, readers should consider findings in the context of its limitations. One limitation is that the study used a volunteer sample of low- to moderate-performing schools in Minnesota and Missouri. Therefore, the results do not generalize to schools that differ systematically from this specific sample of schools. In addition, because the study assessed only reading and mathematics at grades 3-5 using state assessments, the study's findings are not generalizable to other content areas, grades, or assessments. Furthermore, the study findings do not generalize to schools that implement Success in Sight for more than two years. The study also had limitations related to how teacher capacity outcomes were measured. Data from the teacher practice impact analyses were based entirely on teacher self-report collected through an online survey. Appended are: (1) Regional Educational Laboratory Central firewall procedures; (2) Power analyses; (3) Response rates by time point, measure, and experimental group; (4) Data collection instruments; (5) Rationale for cross-state data aggregation and z-score approach; (6) Content review of state assessments; (7) Development and description of the teacher survey measuring teacher capacity for school improvement practices; (8) Calculation of effect sizes; (9) Procedures for handling missing data; (10) Procedures to control for multiple comparisons; (11) Meta-analytic methods for combining state-specific impact estimates; (12) Comparisons of the local context for treatment and control schools; (13) Raw means and standard deviations; (14) Variance components estimates and intraclass correlations; (15) Supporting tables for impact analyses of primary outcomes; (16) Analytic models for sensitivity analyses of primary outcomes; (17) Supporting tables for sensitivity analyses for primary outcomes; (18) Supporting tables for impact analyses of secondary outcomes; (19) Analytic model for sensitivity analyses for secondary outcomes; (20) Supporting tables for sensitivity analyses for impact analyses of secondary outcomes; (21) Analytic model for exploratory analysis; and (22) Supporting tables for exploratory analysis. (Contains 25 figures, 62 tables and 53 footnotes.)