The Implementation and Effects of a Multi-Year Continuous Improvement Research Project in Rural Secondary Schools: Montana Partnership Comparison Study 2
Author: Grindal, Todd
Date: 2020-08-00
Type: Reports - Research
Subjects: Middle Schools,High Schools,Rural Schools,School Turnaround,Program Design,Program Implementation,Educational Improvement,Reading Tests,Reading Achievement,Reading Difficulties,Outcomes of Education,Middle School Students,High School Students,Language Arts,College Entrance Examinations,Reading Skills,Scores,Student Characteristics

This project took place at four Montana middle and high schools in three rural settings located across the state. Study participants included students and teachers located in four middle and high schools. Each of the schools had large percentages of struggling readers, American Indian students, English learners, and low-income students. Comparison schools were selected from the 11 other Montana middle and high schools that had participated in the Montana Striving Readers Program, a previous literacy focused school improvement initiative. The study team worked with teams of 8-12 school administrators, instructional coaches, reading intervention teachers, and general education teachers to implement a set of quick turnaround interventions using a Plan Do Study Act (PDSA) process. These PDSA teams worked with SRI and MT-OPI to design and implement two 6-week PDSA cycles in each school year. During each PDSA cycle, team members implemented small alterations in their school practices or processes, referred to as &quot;change ideas,&quot; which were designed to advance student literacy skills and focused on students who struggled with reading. PDSA team members were encouraged to select change ideas that would support struggling readers in particular, with the idea that these supports would help all learners access core classroom instruction and content. Teams met weekly to review data on relevant teacher practices and make changes to implementation. At the end of the cycle, the teams reviewed the data on student outcomes and developed a plan for the next PDSA cycle. School teams engaged in six PDSA cycles across the 4-year project. The study team used propensity scores to test the effect of participation in the PDSA cycles on student literacy outcomes. The propensity score is the predicted probability of participating in a treatment based on a set of potentially confounding covariates (i.e., student demographic and disability characteristics, baseline score) using logistic regression. Propensity score analysis attempts to equalize the mean values of potentially confounding observed covariates in the treatment and comparison groups, assuring that differences in outcomes between the treatment and covariate effect are not the result of differences in mean values of those covariates. Comparison schools were selected from the 11 other Montana middle and high schools that had participated in the Montana Striving Readers Program, a previous literacy focused school improvement initiative. Students in the comparison schools received literacy instruction through the Montana Striving Readers Program, without the additional supports and strategies provided through the partnership's PDSA approach. Student outcome data from the treatment schools and comparison schools were used to answer research questions regarding differences in student literacy/reading achievement. Data on the overall reading ability of middle school students were collected via fall and spring Istation Indicators of Progress Advanced Reading assessments as well as the spring administration of the Smarter Balanced Assessment Consortium English language arts assessment. The study team used data from the ACT Reading assessment conducted in the spring of grades 10 and 11 to measure the reading skills of high school students. Researchers used a two-level model with students at level one and classrooms at level two to compare outcomes in literacy and achievement of students in treatment and comparison schools. To match improvement and comparison schools, the study used nearest neighbor criteria based on school and average student characteristics, and propensity score matching to arrive at a comparison group of students from the comparison schools that was matched to students in the four improvement schools. The study did not find evidence of a statistically significant association between a school's participation in the intervention and students' literacy skills.