Measuring the Implementation Fidelity of the Response to Intervention Framework in Milwaukee Public Schools. REL 2017-192
Author: Ruffini, Stephen J.
Date: 2016-11-00
Type: Reports - Research
Subjects: Program Implementation,Fidelity,Response to Intervention,Public Schools,Program Effectiveness,Intervention,Scoring Rubrics,Elementary Education,School Districts,Educational Improvement,Teaching Methods,Institutional Characteristics,Student Diversity,Data Collection,Decision Making,Evaluation Methods,Progress Monitoring,Cultural Relevance,Leadership,Cooperation,Reliability,At Risk Students

Many schools identified by states as needing improvement through their Elementary and Secondary Education Act waivers have selected Response to Intervention (RTI), a three-tiered instruction program sometimes referred to as tiered levels of instruction, as one of their main strategies for improving school performance and closing achievement gaps. Yet research on the effects of tiered interventions in school settings is thin. Most studies that show strong impacts have focused on small samples of schools where the leaders of small group instruction (tier 2 instructors) were employed by the intervention developers, thereby allowing the developers to pay close attention to the quality of implementation and to give direct guidance on tier 2 instruction. Studies of these same interventions that involve more schools and that use school staff to lead the small group instruction often find smaller effects. Several factors may explain why the larger studies produce smaller effects; one such factor may be that little effort was made by the schools to monitor implementation systematically and use implementation information as the basis for improvements\. Prior to this study Regional Educational Laboratory Midwest and partners affiliated with the former National Center on Response to Intervention worked with Milwaukee Public Schools to develop a research-based rubric for rating school-level implementation of RTI. The rubric was coupled with a data dashboard that analyzes ratings, displays results at various levels of aggregation, and identifies RTI components that are being implemented inadequately and require improvement. National Center on Response to Intervention staff successfully trained 22 of the district's school improvement coaches to use the rubric when rating implementation during school visits and to enter the ratings into the data dashboard. The current study analyzed ratings by district staff employed as school improvement coaches and who volunteered for the study. In 2014/15 these school improvement coaches visited 70 district schools that serve students in grades K-5. The coaches examined documents and interviewed school staff on implementation of RTI. Based on the information they gathered during a school visit, the coaches rated the schools' implementation of RTI using a 33-item rubric. Ratings for two schools were incomplete, leaving 68 schools in the sample. Analyses focused on the reliability of the RTI implementation rubric, average implementation ratings across the 68 schools, and correlations between aggregate ratings and school characteristics. Key findings include the following: (1) Ratings of the same schools made independently by two school improvement coaches employed by the district showed a high degree of consistency, even after accounting for chance (Cohen's kappa interrater reliability estimates range from 0.71 to 0.85 for the various components); (2) Ratings across the 33 indicators in the implementation fidelity rubric showed a high degree of consistency (alpha = 0.94), and the consistency of ratings on indicators for the six key RTI components fell in the adequate or good range (alphas for key components ranging from 0.70 to 0.85); (3) Two years after rolling out RTI, all 68 schools had made progress toward implementing the framework, and 53 percent of schools were found to be implementing it with fidelity; (4) Some 69 percent of schools had yet to implement the multitiered instruction component with adequate fidelity (especially the tier 3 subcomponent), and 49 percent had yet to implement the evaluation component with adequate fidelity. These components were subsequently identified as priority areas for additional school-level professional development and coaching within the district; and (5) Several school characteristics showed statistically significant relationships with implementation ratings for RTI components. Specifically, higher-performing schools and schools with higher percentages of teachers with advanced credentials, higher staff retention rates, smaller percentages of economically disadvantaged students, and lower student suspension rates showed stronger implementation of RTI than did schools without these characteristics. The following are appended: (1) The district's implementation fidelity monitoring system for the Response to Intervention framework; and (2) Data collection and methodology. [For the summary companion report, &quot;Measuring the Implementation Fidelity of the Response to Intervention Framework in Milwaukee Public Schools. Stated Briefly. REL 2017-192,&quot; see ED570889.]