EVALUATION OF THE WATERFORD EARLY MATH & SCIENCE PROGRAM FOR KINDERGARTEN: First-Year Implementation in Five Urban Low-Income Schools

Evaluation of the Waterford Early Math & Science Program for Kindergarten: First-Year Implementation in Five Urban Low-Income Schools Citation: Powers, P., & Price-Johnson, C. (2007). Evaluation of the Waterford Early Math & Science Program for Kindergarten: First-Year Implementation in Five Urban Low-Income Schools. Creative Research Associates. Background: The Waterford Early Math & Science (WEMS) program is a comprehensive educational software program designed to build math and science skills and concepts in grades K-2, alone or to supplement existing curricula. The program’s capability to individualize lessons, assess and track student progress, and reteach lessons is aimed at keeping potentially “at risk” students at grade level. Purpose: The present evaluation of the Waterford Early Math & Science program is the first independent study of its effectiveness. Setting: The study was carried out in five low-income, largely Hispanic schools during the 2005-06 school year. Study Sample: This report covers the 22 kindergartens (345 students) from a larger sample of 59 K-2 classrooms (923 students) in the five schools. Intervention: Treatment classrooms were provided with four to six computers loaded with the Waterford Early Math & Science program. The teachers received initial and on-going training in its use and were instructed to give every student at least four 22-minute sessions on the program each week. Research Design: The study used an experimental design in which classrooms were assigned through a process of stratified random selection to the treatment (Waterford Math & Science program) or control group. Control or Comparison Condition: Control classrooms were to use the district curricula and any supplemental programs or educational software already in use, except for Waterford Early Math & Science. Data Collection and Analysis: Data collected included pretest data on student achievement in math and science in the fall, and posttest data on the same in the spring. The Stanford Achievement Test (SAT10 Form A, SESAT1 and SESAT2) provided the pre-post measures for math and the environment (science). Usage data stored in each computer provided the measure of exposure to the program. Usage was filtered at 1100 minutes corresponding to six months of 45 minutes a week in order to see program effects on student achievement. Statistical analyses included paired and independent sample t tests, analysis of covariance, and chi-square tests for non-parametric measures. Treatment and control groups were also analyzed by subgroups of gender, ethnicity, primary home language, English language learner (ELL) status, pretest achievement quartile, and by program dosage. Classroom observations of a sample of treatment and control classrooms, focus groups of the treatment teachers, principal interviews and surveys of all treatment and control teachers provided data on the level of implementation of the program, equivalency of conditions, and attitudes toward various aspects of the program. Findings: The most significant findings for kindergarten were: • Qualitative data showed the attitudes of the treatment classroom teachers and principals to be very favorable toward the program; • The general level of implementation was as expected for first-year implementation, with moderately high use of the software and limited use of supplementary materials; • WEMS students significantly outperformed the control students in math and the environment tests; • Program effect sizes were moderate; • When compared with their counterparts in the control classes, the WEMS students made significantly greater gains in math and the environment tests in the case of boys, girls, Hispanics, Spanish home language students, and ELL students; • WEMS English home language students and English-proficient (non-ELL) students also made significantly higher gains in the environment test than their control counterparts; • WEMS Spanish home language students and WEMS ELL students made significantly greater gains than other WEMS and control students and scored above the national mean on both tests; • Kindergartners at all levels of program usage made greater gains than the controls on both tests. Native American, African American and white children showed greater gains in the WEMS classrooms than in the controls, but their samples were too small for results to be conclusive. Conclusion: The WEMS program appeared to benefit all children in kindergarten, particularly Hispanics, students whose primary home language was Spanish, and English language learners. 1. Teacher and principal attitudes Focus groups, interviews, conversations with the teachers during class observations, and surveys were used to determine the attitudes of teachers and principals toward various aspects of the WEMS program. The attitudes investigated concerned WEMS training and support, technical issues, implementation, the quality and content of the program, and student response and achievement. Attitudes toward technology, technical aspects Teachers appreciated how easy the program was to use and how little direct maintenance it required once it was running. They especially liked that the children could manage the rotations themselves. The teachers noted that the kids seemed to really love the program. One principal noted, “It’s like having four other teachers in the room who never lose their patience. ” Attitudes toward the quality and content of the program Teachers and principals agreed on the high quality of the WEMS program. Technology was seen as one more opportunity to meet the diverse needs of the students. Teachers and principals specifically mentioned the comprehensive nature of the program, the quality of the graphics and sound, the ability to track student progress, and the variety of activities in different modalities such as music. They liked that it was self-paced, had different levels, could print reports, and engaged the students extremely well. They also mentioned its usefulness for vocabulary development and listening skills. The movies and science lessons were judged to be especially good. According to the teachers and principals, the testing, diagnostic assessments and tracking of student progress put it above other computer-based and internet instructional resources. Teachers especially appreciated the integration of math and science. Many teachers welcomed the additional exposure and expressed their frustration at not being able to do more science activities in the classroom because of time constraints. Attitudes toward student engagement with WEMS All teachers and principals in the surveys, observations, focus groups and interviews noted the enthusiastic response of the children to the program. Students picked up how to use the computer quickly and motivation remained high, even at the end of the year, in contrast to other computerbased instructional programs. The program engaged students who were not usually engaged. Students would ask: “Why haven’t I gone yet? ” “This is interesting. ” “When is it my turn? ” Because of the high level of interest, teachers found WEMS a good motivational tool as well as an instructional program. Students from other classes sometimes came in and wanted to use the program but were denied. 2. Achievement Determining the effect of the WEMS program on student achievement was the main purpose of this evaluation. Qualitative data was collected from teachers and principals from focus groups, interviews, conversations, and surveys. In general, findings suggested that the effect of the program was positive, with teachers who scheduled heavy use of WEMS responding more positively. Quantitative data relating to program effects consisted of results from the Stanford Achievement Tests in math and the environment and records of usage of the WEMS program for each student. Achievement scores were compared across the treatment and control groups and by subgroups. Select results from these analyses are presented below. Gains by School In math, kindergartners in WEMS classes made greater gains than controls at all five schools, and WEMS students had significant pre to posttest gains at all schools (see Figure 1). On the environment test, results were similar. Students in the WEMS classes made greater gains than controls at all schools (see Figure 2). Figure 1. Kindergarten SAT10 Math Gains