Differential Impacts of Intensive District-Level Technical Assistance on Student Achievement: A Study of California's District Assistance and Intervention Teams
Author: Strunk, Katharine O.
Date: 2012-00-00
Type: Reports - Research
Subjects: Quasiexperimental Design,Intervention,Federal Legislation,Achievement Gains,Academic Achievement,Technical Assistance,Student Characteristics,Accountability,Outcomes of Education,Comparative Analysis,Educational Change,Mathematics Achievement,Scores,English Language Learners

Building on Strunk, McEachin, and Westover (2011), the authors examine whether or not District Assistance and Intervention Teams (DAITs) have a differential impact on student performance across school and student characteristics. They use a quasi-experimental design to examine the impacts of DAITs on student achievement on math and English Language Arts (ELA) California Standards Tests (CSTs) based on preliminary evidence from the first two years of the DAIT intervention. They are specifically interested in how the treatment effect varies by grade-level, schools' Title 1 and No Child Left Behind (NCLB) accountability status, and students' prior achievement. Although increases in average achievement are important outcomes of technical assistance for low-performing districts, it is equally important to know who is actually benefitting from the assistance and whether the neediest students and schools within the lowest performing districts are also making significant achievement gains. The intent of this analysis is to isolate the impact of DAITs on student outcomes, specifically for different groups of students in different kinds of schools. First, to do this, the authors want to compare student outcomes in districts that received DAITs to some untreated set of students. A clear comparison group is the students in PI3 districts that received non-DAIT TA. This comparison solves the issue of bias stemming from the potential impacts of other reforms or trends that impact all PI3 students, but brings with it another set of potential biases: because the DAIT intervention was assigned to districts with the lowest performing students, there is reason to think that the inherent differences between students in DAIT districts and those without will impact eventual outcomes. The authors use a five-year panel from California's student-level administrative dataset (from 2005-6 to 2009-10) that tracks approximately 4.9 million students, enrolled in 9,000 schools and 1,000 districts, across each of the five years. The 4.9 million students in the dataset represent the students the authors were able to match longitudinally over the 2005-6 to 2009-10 school years, constituting approximately 83% of the total population. Results indicate that the significant increases in math achievement found in Strunk, McEachin, and Westover (2011) are not necessarily consistent across students and schools, and the intervention does not always appear to affect the neediest students and schools within the lowest performing districts. The authors' results also indicate that while average treatment effects often provide useful and tangible answers to policy questions, they often mask important variation among populations of interest. (Contains 7 tables.)