Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18. Appendix
Author: Jaciw, Andrew P.
Date: 2020-11-18
Type: Reports - Research
Subjects: Faculty Development,Science Instruction,Science Achievement,Elementary School Teachers,Elementary School Science,Pedagogical Content Knowledge,Academic Standards,Alignment (Education),Teacher Attitudes,National Standards,Educational Opportunities,Educational Environment,Program Effectiveness,Fidelity,Program Implementation,Achievement Tests,Content Area Writing,Outcomes of Education,Elementary School Students,Grade 4,Grade 5,Mathematics Tests,Language Tests,English,Science Tests,Measurement,Student Attitudes,Teacher Surveys,Student Surveys,Administrator Surveys,Principals,Hierarchical Linear Modeling,Benchmarking,Sample Size

These appendices accompany the report &quot;Final Report of the i3 Impact Study of Making Sense of SCIENCE, 2016-17 through 2017-18.&quot; Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment. It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd's Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students' science achievement through improving science instruction. Under this grant, WestEd and Empirical Education also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale-up study of Making Sense of SCIENCE. The impact evaluation, which is the focus of this report, was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016-17 and 2017-18 school years. The study randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual (&quot;control&quot;) group, which received the professional learning (delayed-treatment) after the study ended. The study found that Making Sense of SCIENCE had a positive impact on teacher content knowledge (effect size = 0.56, p = .006) and a positive impact on a holistic scale of teacher pedagogical content knowledge (effect size = 0.41, p = 0.026). The study also yielded positive and significant impacts on the amount of time teachers spent on science instruction (effect size = 0.40, p = 0.015) and on the emphasis that teachers placed on NGSS-aligned instructional practices, with statistically significant effect sizes ranging from 0.40 to 0.49 standard deviations. This suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice. In regard to student science achievement, the study did not find statistically significant results for the full sample of students (effect size = 0.06, p = 0.494), or for the sample of students in the bottom third of incoming math and English Language Arts achievement, with effect sizes of 0.22 standard deviations (p = 0.099) and 0.073 standard deviations (p = 0.567), respectively. Notably, with the exception of one negative effect size, additional analyses on student achievement using different measures and samples yielded positive, but not statistically significant, effect sizes ranging from 0.02 to 0.12 standard deviations. Evaluators offer three potential contributors to the findings of limited impact on student science achievement. First, given the timing of the study in relation to the release of the NGSS in 2013, finding a suitable NGSS-aligned student science assessment was a challenge. Second, most study schools and districts had not yet adopted NGSS-aligned curricula and did not have access to NGSS-aligned curriculum resources. Third, the sample of teachers was unstable across the two years, with the percentage of teachers leaving the school congruous to the percentage observed at the national level. [For the full report, see ED609253; for the research summary, see ED609256.]