The Middle School Algebra Readiness Initiative: An Evaluation of Teacher Outcomes and Student Mathematics Achievement and Gains
Author: Hixson, Nate
Date: 2012-10-00
Type: Reports - Evaluative
Subjects: Middle Schools,Algebra,Secondary School Mathematics,Mathematics Instruction,Intervention,Grade 6,Grade 7,Grade 8,Computer Software,Pedagogical Content Knowledge,Middle School Teachers,Middle School Students,Mathematics Achievement,Achievement Gains,Program Effectiveness,Teaching Methods,Student Characteristics,Standardized Tests,Scores,Program Implementation,Fidelity,Mathematics Curriculum,Faculty Development

We conducted an evaluation study of the Middle School Algebra Readiness Initiative, a middle school mathematics intervention that was implemented in two West Virginia school districts during the 2011-2012 school year. In participating middle schools, the Carnegie Learning MATHia® software intervention and accompanying classroom curriculum were used as a total replacement for the districts' alternative mathematics curriculum for Grades 6, 7, and 8. A cohort of teachers was trained by Carnegie Learning in mathematical content and pedagogy as well as in the proper implementation of the software and classroom curriculum materials. Our evaluation tested five hypotheses. Our first was related to the impact of the initiative on teacher-level outcomes, specifically teachers' content and pedagogical knowledge in the areas of patterns, functions, and algebra. This hypothesis was tested by using a pretest/posttest assessment of teacher knowledge. We used the research-validated, &quot;Learning Mathematics for Teaching&quot; (LMT) assessment. Our statistical analysis of teacher pretest/posttest differences revealed that, for the 20 teachers who completed both a pretest and posttest, there was only a marginal gain. This gain was not statistically significant. As such, we rejected our first hypothesis. The remaining four study hypotheses tested the impact of the initiative on students' mathematical achievement and year-to-year mathematics gains as measured by the Grade 6, 7, or 8 mathematics subtest of the West Virginia Educational Standards Test 2 (WESTEST 2). We used propensity score matching (PSM) to match students in a variety of implementation scenarios to select a comparison group of students. The comparison groups for our investigations of Hypotheses 2-4, included students who used the MATHia software/curriculum at various levels of implementation, matched to their grade-level peers who used some other curriculum during the 2011-2012 school year. For Hypothesis 5, we compared students who used the MATHia program for at least 1 hour per week-meeting the vendor's definition of &quot;adequate use&quot;--to a comparison group of students who used the program for less time. In all cases, we rigorously matched the two groups of students using a variety of covariates including sex, free/reduced-price lunch eligibility, special education eligibility, grade, and prior academic achievement in both mathematics and reading/language arts. We then conducted two student-level analyses. First, we examined mean differences in students' standardized test scores and mathematics gains, determining if the treatment or comparison group scores differed by a statistically significant margin. Second, we used linear regression to determine, after controlling for the aforementioned covariates, what level of impact the treatment had on student achievement and gains. We found in most cases that students who were in the treatment group underperformed when compared with their grade-level peers who used an alternate curriculum. With a few exceptions, the differences were statistically significant. However, the results of the linear regressions illustrated that, after controlling for important covariates, the negative relationship among treatment and student achievement/gains was relatively small, but still statistically significant. Several limitations impair our ability to make conclusions based on these results. Most critically, we had very little information about the degree to which the teachers and students implemented the intervention components with fidelity. We know very little about the quality or content of the training provided by Carnegie Learning. We experienced considerable attrition among educators in the period between the pretest and posttest LMT administration. Only 55% of teachers completed both assessments. We also received very different numbers of students from Carnegie Learning and the school districts when we requested this information for our analyses. Finally, we found that very few students met the implementation criteria recommended by Carnegie Learning. This finding, in particular, points to a potential lack of fidelity in implementation, which makes us very reluctant to allow this evaluation to stand as a fair trial of the efficacy of the MATHia software/curriculum. In fact, we recommend strongly against using our report in this manner. It should be seen as an evaluation of an entire initiative rather than any curriculum or software program alone. In light of these and other limitations described in this report, we make only two recommendations. First, we suggest future program implementations of this type take substantial measures to collect critical qualitative implementation data so that the results of quantitative analyses can be more readily interpreted. This can be accomplished, among other strategies, by devoting greater resources to the program evaluation component of such projects. Second, in districts where similar programs are currently underway or in the planning stages, we recommend continuous monitoring and technical assistance to ensure that the program components are delivered as intended. Doing so may help prevent a potential negative impacts on student outcomes.