Impact Evaluation of &quot;ACCESS&quot;: A Culture Creating Effective Systems for Success
Author: Askew, Karyl
Date: 2019-06-00
Type: Reports - Evaluative
Subjects: Program Evaluation,Grants,Educational Innovation,Educational Improvement,Technology Integration,Curriculum Design,Middle School Mathematics,Academic Achievement,Mathematics Achievement,Reading Achievement,Rural Schools,Program Implementation,Middle School Students,College Entrance Examinations,Standardized Tests,High School Students,Outcomes of Education,Scores,Mathematics Tests,Reading Tests,Program Descriptions,Distance Education,Fidelity,Intervention,Educational Practices,Instructional Design,Sustainability

&quot;ACCESS&quot;, A Culture Creating Effective Systems for Success, was a three-year Investing in Innovation (i3) development grant funded by the Office of Innovation and Improvement, U.S. Department of Education. The &quot;ACCESS&quot; grant led the way for a district-wide redesign of educational practices based on the novel approach of embedding a culture of technology-based education (tech culture) within rural schools and the broader community, with an intentional effort to record, evaluate, refine, and disseminate an effective model for implementation across the state and nation. The impact evaluation included two studies that examined the effect of &quot;ACCESS&quot; on the academic achievement of students in grades 5-10 as measured by NC standardized End-of-Grade assessments and the American College Testing (ACT) standardized test. The middle grades study (Study 1) used a comparative short interrupted time series (CSITS) design to assess the impact of the intervention on mathematics and reading achievement in grades 5 through 8 after two years of program exposure. The high school study (Study 2) used a quasi-experimental design (QED) to assess the impact of ACCESS on ACT composite scores taken in grade 10 after two years of program exposure. For both studies, we compared the outcomes of ACCESS participants with matched samples of non-participants in other NC school districts. Propensity score matching (PSM) was used to match ACCESS samples with comparison students at baseline; baseline equivalence was established all pre-test assessment measures. Both studies met What Works Clearinghouse (WWC) Group Design Standards with Reservations. The results indicated no statistically significant differences in achievement between the &quot;ACCESS&quot; treatment group and the business-as-usual comparison groups as measured by the ACT composite scores taken in grade 10 or NC End-of-Grades 5-8 standardized reading and mathematics exams. To contextualize the findings, researchers considered the duration of students' exposure to &quot;ACCESS&quot;, alignment between technology integration and NC standardized assessments, and contextual factors that may have narrowed the divide between MCS and other school districts from which comparison schools/samples were drawn. The report concludes with suggestions for future research and implications for education policy. [Funding for this report came from the U.S. Department of Education under its Investing in Innovation (i3) initiative within the Office of Innovation and Improvement through Grant U411C140128 to Montgomery County Schools.]