STEM Learning Opportunities Providing Equity (SLOPE): An Investing in Innovation (i3) Grant. Final Evaluation Report
Author: Gallagher, Carole
Date: 2015-09-30
Type: Reports - Research
Subjects: STEM Education,Educational Opportunities,Middle School Students,Secondary School Mathematics,Intervention,Program Effectiveness,Program Evaluation,Academic Achievement,Algebra,Fidelity,Program Implementation,Compliance (Legal),Models,Outcome Measures,Teacher Surveys,Active Learning,Student Projects,Units of Study,Coaching (Performance),Faculty Development,Time Factors (Learning),Randomized Controlled Trials

This five-year evaluation examined the effectiveness of a promising middle-school mathematics intervention funded through an Investing in Innovation (i3) development grant. Evaluation objectives were to: (1) study the impact of an intervention aimed at increasing the academic achievement of students in Algebra I--a gate-keeping course--as measured by student performance on an end-of-year state test in mathematics; and (2) better understand the relationship between intervention impact and implementation fidelity, as measured by levels of compliance by teachers with the study protocol. The intervention was piloted in Year 2 of the grant (2011-12 school year) that was followed by a two-year [randomized control trial] RCT in grant years 3 (2012-13 school year) and 4 (2013-14 school year). Data collected in the RCT years were focused on impact and exploratory analyses, respectively. For the RCT component, 70 Grade 8 Algebra I teachers were recruited from 15 school districts across California. Randomization, conducted by WestEd in spring 2012, was conducted at the teacher level. Students were assigned to classrooms without knowledge of the group membership of teachers (treatment vs. control), using each district's routine placement policies. Fidelity of implementation study was monitored by collecting systematically information from teachers assigned to the treatment condition throughout the course of the study. The contrast of interest was performance on a standardized Algebra I test by students assigned to classrooms taught by treatment teachers compared to performance by students assigned to classrooms taught by control teachers. The final analytic sample for the 2012-13 cohort included 1,384 students assigned to 28 treatment teachers and 1,088 students assigned to 27 control teachers. None of the contrasts showed a statistically significant difference at the 0.05 level. Students who were assigned to classrooms taught by treatment teachers did not perform differently in relation to those assigned to classrooms taught by control teachers. Overall findings from the implementation study indicated that great variability emerged in the ways in which teachers implemented the intervention. The threshold for fidelity was reached with only one component (Instructional Unit #1) of the four studied (three instructional units and professional coaching). The following appendices are included: (1) Logic Model: SLOPE (DEV11) v.13; (2) Teacher Background Survey; (3) Interpreting Intervention Impact through the Lens of Implementation Fidelity: Findings from a Federally Funded Evaluation--Paper Presented at the Annual Meeting of the American Educational Research Association (Chicago, Illinois, April 19, 2015); (4) Implementation Survey for Air Traffic Control; (5) 2012-2013 Measuring Fidelity of Implementation for Algebra I Drop-in Units: DEV11 (SLOPE); (6) Teacher-Level Participation in i3 SLOPE Evaluation (2011-2014); and (7) Findings from Evaluator Study of Implementation: Implementation Year 1.