The Results of a Randomized Control Trial 

Evaluation of the SPARK Literacy Program 

September 2015 

Curtis J. Jones 
SPARK Evaluation i | P a g e

A special thanks to the following individuals who have personally been integral to the successful implementation of 

this project: 

Kris Bischoff, Boys & Girls Clubs of Greater Milwaukee 

Tia Hatchett, Boys & Girls Clubs of Greater Milwaukee 

Dave Knutson, Boys & Girls Clubs of Greater Milwaukee 

Patricia Marcus, Boys & Girls Clubs of Greater Milwaukee 

Joseph Schmidlkofer, Boys & Girls Clubs of America 

Nuntiata Buck, Milwaukee Public Schools 

Kristin Kappelman, Milwaukee Public Schools 

Deborah Kuether, Milwaukee Public Schools 

Marc Sanders, Milwaukee Public Schools 

Kai Tai Chan, University of Wisconsin, Milwaukee 

Rachel Lander, University of Wisconsin, Milwaukee 

Ruth Short, University of Wisconsin, Milwaukee 

Rachel Westrum, University of Wisconsin, Milwaukee 

Grant Sim, University of Wisconsin, Madison 

Eva Chen, Westat 

Jennifer Hamilton, Westat 

A very special thanks to the following individuals who supported the development of this report: 

Michael Christian, Education Analytics 

Rob Meyer, Education Analytics 

Andrew Rice, Education Analytics 

Rachel Lander, University of Wisconsin, Milwaukee 

Rachel Westrum, University of Wisconsin, Milwaukee 

For more information about the evaluation of SPARK, please contact: 

Curtis Jones, Ph.D. 

University of Wisconsin, Milwaukee 

jones554@uwm.edu

This work has been supported by the U.S. Department of Education’s Investing in Innovation program, through 

U396C100694. The opinions expressed are those of the authors and do not represent views of the U.S. Department 

of Education. 
SPARK Evaluation ii | P a g e

Summary Results 

The purpose of this report is to presents the results of a two-year randomized control trial 

evaluation of the SPARK literacy program. SPARK is an early grade literacy program developed 

by Boys & Girls Clubs of Greater Milwaukee. In 2010, SPARK was awarded an Investing in 

Innovation (i3) Department of Education grant to further develop the program and test its impact 

in seven Milwaukee Public Schools (MPS). In the fall of 2013, 286 students were randomly 

selected to receive SPARK for two years (2013-14 and 2014-15 school years) and 290 selected 

as control students. Overall attrition rates ranged from 33.9% to 36.3%, consisting primarily of 

students who moved away during the two study years. Differential attrition rates between 

SPARK and control students ranged from 1.96% to 2.31%. These overall and differential 

attrition rates are classified as “Low” according to standards developed by the What Works 

Clearinghouse. 

The results demonstrate that SPARK had statistically significant positive impacts on 

reading achievement (Hedges’s g = .23), literacy (Hedges’s g = .35), and regular school day 

attendance (Hedges’s g = .25). SPARK students were absent from school 5.8 fewer times than 

control students and were 27% less likely to be chronically absent from school. The benefit of 

SPARK was greater for students who started the program with greater need for literacy 

instruction. The impact of SPARK on reading achievement and literacy for students in the lower 

half of literacy, according to the Phonological Awareness Literacy Screening (PALS) was large, 

Hedges’s g = .36 and .66 respectively. Further, while only 10% of control students below literacy 

benchmark at the start of the study met benchmark at the conclusion of the study, 62% of 

SPARK participants who started below benchmark eventually met benchmark. Taken together, 

these results suggest that SPARK is meeting its goal of supporting the efforts of MPS schools 

and parents to teach students to read. 
SPARK Evaluation iii | P a g e

Table of Contents 

BACKGROUND OF THE SPARK PROGRAM ........................................................................................................... 1

The SPARK Model............................................................................................................................................................... 1

In-school tutoring ............................................................................................................................................................. 2

Family Engagement ......................................................................................................................................................... 3

EVALUATION METHODS ................................................................................................................................................. 4

ANALYSIS PLAN ................................................................................................................................................................... 4

Outcome Measures ............................................................................................................................................................... 4

Modeling Strategy................................................................................................................................................................. 5

ATTRITION ............................................................................................................................................................................... 7

SAMPLE CHARACTERISTICS ......................................................................................................................................... 7

Initial Sample ......................................................................................................................................................................... 7

Final Sample ........................................................................................................................................................................... 8

PARTICIPATION .................................................................................................................................................................. 10

RESULTS .................................................................................................................................................................................. 12

Reading Achievement – MAP ........................................................................................................................................ 12

Literacy – PALS .................................................................................................................................................................. 13

School Attendance .............................................................................................................................................................. 15

CONCLUSIONS ..................................................................................................................................................................... 16

REFERENCES ......................................................................................................................................................................... 17 
SPARK Evaluation 1 | P a g e

Background of the SPARK Program 

The Milwaukee Public Schools (MPS), a district serving over 80,000 students, faces a 

significant challenge to teach its students how to read and write. According to the Wisconsin 

Knowledge and Concepts Examination (WKCE) only 15% of MPS students were proficient in 

reading (2011) compared to 35% statewide. The results of the WKCE are consistent with results 

of the National Assessment of Educational Progress (NAEP) and the ACT, which show that 

MPS students struggle with literacy throughout their education; only 15% of 4th grade MPS 

students are proficient in reading (NAEP, 2011) and 14% of MPS 11th graders scored at least 21 

on the ACT Reading Test, the benchmark identified for college readiness (special analysis). The 

results of the NAEP further shows that there are significant achievement gaps for minority and 

low-income students; 39% of 4th grade white MPS students are proficient in reading, compared 

to 7% of black and 15% of Hispanic students and 7% of 4th grade low-income (free/reduced 

lunch participants) MPS students are proficient in reading, compared to 48% of non-low-income 

students. 

These statistics demonstrate that the need for increased literacy opportunities in the 

Milwaukee area is urgent, and that this need is even more pronounced for low-income and 

minority students. SPARK was created in 2005 by Boys & Girls Clubs of Greater Milwaukee 

(BGCGM) to address this need. In 2010, SPARK received a Department of Education Investing 

in Innovation (i3) grant award to further develop the program and expand it to seven additional 

low-income and minority Milwaukee elementary schools. 

The SPARK Model 

By using both in-school tutoring and family engagement, SPARK works both to develop 

he literacy skills of early-grade students and to support families as they learn to support the 

iteracy development of their students. SPARK students are also encouraged to participate in 

fter-school club activities. It is through this multi-modal strategy that SPARK seeks to have a 

asting impact on students and to prepare them to succeed beyond their participation in SPARK. 

t

l

a

l
SPARK Evaluation 2 | P a g e

In-school tutoring 

The tutoring component of SPARK is loosely based on the Reading Recovery program. 

Reading Recovery focuses on in-school tutoring with lesson plans written, and assessments, 

analyzed by the licensed teachers themselves, and has been found to be effective in developing 

student literacy skills (Pinnell, Lyons, DeFord, Bryk, & Seltzer, 1994). For the in-school tutoring 

component of the current study, SPARK students were pulled out of non-core classes during the 

school day for 30 minutes, up to three times per week, for two years. Each tutoring session 

included five activities: 

• The Familiar Activity is a brief element that gets the student ready for learning by 

reviewing a skill they have recently learned. 

• Word play is a key element in the lesson where students receive targeted, differentiated 

instruction on foundational reading skills including phonics and phonemic awareness. 

Word plan is individualized to focus on students’ needs. It is centered on two main 

activities: Word Sorts and Making Words. These activities combine both constructivist 

learning and structured instruction. Each of these activities focus on specific skills and 

tutors are explicit with students about the lesson’s foci. Word Sorts involve students 

sorting words into various categories to increase their understanding of the structure of 

sounds and letters. Making Words involves students using different letters to make words 

and provides a structured way for students to learn how the sounds of language are put 

together. Students also read phonics-based books during Word Play time and do 

enrichment activities to cement their understanding of the focus skills. 

• During every SPARK lesson, students spend time Reading a book at their instructional 

level. Before reading the book, they do a book walk to familiarize themselves with the 

content and vocabulary of the book. Students read both fiction and non-fiction books. As 

students read, tutors use a variety of strategies to help students decode and make meaning 

of text. Students use graphic organizers to build comprehension skills. 

• Students spend time each lesson Writing sentences connected to their Word Play skill or 

their instructional reading book. Tutors help students correctly spell the words in their 

sentence(s). Elkonin boxes are a central piece of SPARK writing and used to help 

students encode words. Elkonin boxes are an instructional method used in early 

elementary grades to build phonological awareness by segmenting words into individual 

sounds/boxes. 

• The lesson ends with a brief opportunity for students to hear their tutor read a book. 

At each site a program manager, who is also a certified teacher, oversees and supports the 

tutors. Tutors participate in all-program training at the beginning of the year focused on the 

implementation of the lesson, how to develop a lesson plan, and how to administer and use 

literacy assessments. At the site level, other individualized training opportunities are developed 
SPARK Evaluation 3 | P a g e

throughout the year as needs arise. Tutors are formally observed at least once monthly by their 

program manager and receive feedback following these observations. 

Assessment is an important component of SPARK. All students are assessed with the 

Phonological Awareness Literacy Screening (PALS)1 at the beginning and end of the school 

year. The PALS is used to determine each student’s needs and help create individual lesson 

plans. Running Records2 are also used to track student progress throughout a student’s 

participation in SPARK. 

Family Engagement 

SPARK seeks to have a lasting impact on students by engaging families in the literacy 

evelopment of students. Involving families in tutoring programs can improve children’s 

cademic knowledge, skills and confidence (Bryan, 2005; Little, 2009). Encouraging family 

nvolvement in educational programs traditionally focuses on families attending events, 

eceiving information from staff, volunteering (Epstein, 2001), and generally exhibiting “good 

arent” behaviors (Li, 2010). Getting to know families and the ways that their lives are 

tructured outside of the educational setting may lead to a reciprocal relationship that can 

ncrease involvement (Graue & Hawkins, 2010). The family component of a program is not only 

o make families aware of the program’s mission and goals but also to empower families in their 

hildren’s learning both in the program and at home. 

To execute the family engagement component, each site has a parent partner who works 

ith each participating student’s family. Their work is designed to bridge the divide between 

chool and home by translating literacy concepts, educating families about a variety of literacy 

ctivities, and validating the literacy practices already happening in the home. Parent partners 

elp families see how they already are incorporating literacy into their children’s lives and show 

arents how to promote literacy more effectively. Parent partners stay connected with families 

hrough a monthly newsletter, monthly family events at each site, phone calls, and emails. These 

ommunications are designed to keep families aware of student progress in SPARK, help 

amilies promote literacy at home, and address any school attendance issues that arise during the 

rogram. Parent partners also conduct home visits for all students twice during the summer 

 

https://pals.virginia.edu/rd-background.html 

https://www.readinga-z.com/helpful-tools/about-running-records/ 

d

a

i

r

p

s

i

t

c

w

s

a

h

p

t

c

f

p

1

2
SPARK Evaluation 4 | P a g e

between their first and second year of participation and as needed during the school year. These 

visits are viewed as opportunities to connect with families in their own space and learn about the 

literacy activities already taking place in the home. 

Evaluation Methods 

The Office of Socially Responsible Evaluation in Education (SREed) at the University of 

Wisconsin Milwaukee utilized a randomized control trial selection framework at the student 

level to isolate the impact of SPARK. Informed consent was obtained from 576 parents for their 

students to participate in the study. A random sample of kindergarten, 1st, and 2nd grade 

students in seven MPS schools was selected in October and November of 2013 to participate. 

286 students were randomly selected as SPARK participants and 290 as control students. 

Stratification was done by school and grade level within school. The specific number of students 

selected to receive SPARK within each strata was determined both by the number of consented 

students and the capacity to serve students within each site. Students with a reading-related IEP 

or who were English Language Learners, were not eligible to participate in the evaluation but 

were eligible to receive tutoring. All other students were eligible to participate. 

Analysis plan 

Outcome Measures 

The evaluation tested the impact of SPARK across three domains: reading achievement, 

iteracy, and school attendance. The tutoring component of SPARK was viewed as potentially 

aving an impact on both reading achievement and literacy, while the family engagement 

omponent was viewed as potentially having an impact on school attendance. All three outcomes 

isted below are collected by MPS and were provided directly to the evaluation by MPS for the 

urpose of evaluating SPARK. 

eading achievement (Measures of Academic Progress (MAP) Reading Assessment) – The 

AP is a norm-referenced, adaptive assessment of reading achievement. The technical reference 

anual3 reports an internal marginal reliability of .95. Test-retest reliabilities were reported as 

etween .76 and .89. It also is reported to have high concurrent validity with a variety of other 

l

h

c

l

p

R

M

m

b

 

3 www.apsrc.net/images/interior/nwea%20resources/nwea_technicalmanual.pdf 
SPARK Evaluation 5 | P a g e

reading assessments including the Iowa Test of Basic Skills and the Stanford 9 achievement test. 

MPS administers the MAP each fall, winter, and spring to all students. 

Literacy (Phonological Awareness Literacy Screening (PALS)) – The PALS is a criterionreferenced, teacher-administered assessment of literacy. The technical reference manual4

 reports 

an internal reliability of between .76 and .83. Inter-rater reliabilities were reported as 

approximately .92. Test-retest reliabilities were reported as between .92 and .96. It also was 

reported to have both concurrent and predictive validity with a variety of other reading 

assessments. MPS began administering the PALS to all kindergarten and first grade students in 

the 2013-14 school year. MPS began administering the PALS to second grade students in the 

2014-2015 school year. Administering the PALS to third grade students was optional but MPS 

decided to do so in all SPARK schools so that the PALS could be used as an outcome in the 

evaluation for all grade levels. 

Regular School Data Attendance – Chronic absenteeism (missing more than 10% of school days) 

is a problem that disproportionally impacts poor and minority students and predicts both 

academic and social problems later in students’ educations. (Chang & Romero, 2008). Because 

of this, reducing absences is one of the goals of SPARK. MPS tracks the number of days 

students are absent from school. The total number of absences for both the 2013-14 and 2014-

2015 were provided directly to the evaluation by MPS to measure the impact SPARK had on 

school attendance. Baseline attendance data were not available for the evaluation. 

Modeling Strategy 

Separate generalized linear statistical models with robust standard error estimators were 

used to compare spring 2015 MAP, spring 2015 PALS, and attendance (number of absences for

both program years combined) of participants and controls for current first, second, and third 

grade students. The results of these grade level models were then pooled to estimate the overall 

impact of SPARK on each domain. 

Table 1 presents the covariates used in each of the grade specific models for each 

outcome. Covariates were chosen based on their availability and predictive validity for each 

outcome. Baseline PALS were not available for 3rd grade students because MPS did not require 

schools to administer PALS to 2nd grade students. Because of this MAP Math scores were 

 

4 https://pals.virginia.edu/pdfs/rd/tech/K_Tech_Ref_2014_B.pdf 
SPARK Evaluation 6 | P a g e

included as an additional covariate in 3rd grade MAP and PALS models. Non-predictive 

covariates were removed from each model. School was included as a fixed factor in all models to 

account for the clustering of data within schools. Other student characteristics such as ethnicity, 

gender, and IEP status were not found to uniquely predict outcomes so were not included in any 

models. Students with missing data were excluded from each analysis. All models were done 

using both standardized and unstandardized outcomes. No adjustments were made to 

significance levels because only one outcome was used for each of the three domains (literacy, 

reading achievement, and school attendance). 

The potential that SPARK has a differential impact on students with different levels of 

baseline literacy and reading achievement was explored by including the interaction terms of 

SPARK with baseline PALS (for 1st and 2nd grade models) and with baseline MAP reading (for 

3

rd grade models) in each model. When the interaction terms were found to be significant, 

additional models were done splitting the sample in half by baseline literacy and achievement 

scores. 

Table 1: Factors included in each outcome model 

Outcomes 

2-year 

cohort 

SPARK School 

Fall 

2013 

PALS 

Fall 

2013 

MAP 

Reading 

Fall 

2013 

MAP 

Math 

F/R 

lunch 

School 

x 

PALS 

School 

x MAP 

Reading 

School 

x MAP 

Math 

PALS K - 1st x x x x x x 

1st - 2nd x x x x x x 

2nd - 3rd* x x x x x x 

MAP K - 1st x x x x x x 

1st - 2nd x x x x x x 

2nd - 3rd* x x x x x x 

Attendance K - 1st x x x x x 

1st - 2nd x x x x x 

2nd - 3rd* x x x x x

* MPS did not administer the PALS with 2nd grade students in the 2013-2014 school year. 
SPARK Evaluation 7 | P a g e

Attrition 

In the fall of 2013, a total of 576 students across seven schools consented to participate in 

the SPARK program and evaluation. Selections were made in October and November, after fall 

assessments had been completed. 286 students were randomly selected as SPARK participants 

and 290 as control students. Table 2 presents the samples used in each analysis and the 

corresponding attrition rates. Depending on the analysis, overall attrition rates ranged from 

33.9% to 36.3%. These include the 187 students who moved away during the two years of the 

study and additional students with missing data. Differential attrition rates between SPARK 

participants and control students ranged from 1.96% to 2.31%. These overall and differential 

attrition rates are classified as “Low” according to the What Works Clearinghouse Procedures 

and Standards Handbook Version 3.0. 5 There was no replacement of students that dropped from 

the evaluation. 

Table 2: Attrition rates for each analysis of the impact of SPARK 

Moved 

away 

Missing 

data 

Final 

sample 

Attrition 

rate 

MAP Control 290 96 6 188 35.2%

SPARK 286 91 3 192 32.9%

Total 576 187 9 380 34.0%

PALS Control 290 96 12 182 37.2%

SPARK 286 91 10 185 35.3%

Total 576 187 22 367 36.3%

Attendance Control 290 96 5 189 34.8%

SPARK 286 91 3 192 32.9%

Total 576 187 8 381 33.9%

Sample Characteristics 

Initial Sample 

Of the 576 consented students, 205 (36%) were in kindergarten, 214 (37%) in first, and 

157 (27%) in second. 549 (95%) were eligible for free/reduced lunch, 459 (80%) were African 

 

5 http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_standards_handbook.pdf 
SPARK Evaluation 8 | P a g e

American, 71 (12%) Hispanic, 291 (50.5%) female, and 51 (9%) had an IEP for 

speech/language. English language learners and students with learning disabilities, cognitive 

impairments, and emotional disabilities were excluded from the study but were able to receive 

SPARK tutoring. Table 3 presents the baseline literacy (PALS) and achievement scores (MAP) 

for the total consented sample of 576 students. 

Table 3: Fall 2013 MAP and PALS scores – Total sample 

Control 

MAP Reading RIT MAP Math RIT PALS 

Grade Mean SD n Mean SD n Mean SD n 

K 142.9 9.1 104 138.2 9.8 92 64.0 21.5 93

1st 155.1 10.6 108 155.8 12.2 94 48.3 19.3 95

SPARK 

2nd 163.4 12.4 78 167.7 10.3 75

Total 153.0 13.4 290 153.0 16.1 261 56.1 21.8 188

K 140.4 10.6 101 137.4 9.2 89 60.1 22.6 89

1st 155.3 10.9 106 157.0 12.5 101 50.0 17.3 101

Total 

2nd 165.8 13.4 79 170.1 10.7 76

Total 152.9 15.4 286 154.2 17.0 266 54.7 20.5 190

K 141.6 9.9 205 137.8 9.5 181 62.1 22.1 182

1st 155.2 10.7 214 156.4 12.3 195 49.2 18.3 196

2nd 164.6 12.9 157 168.9 10.5 151

Total 153.0 14.4 576 153.6 16.6 527 55.4 21.2 378

Final Sample 

By the spring of the 2014-2015 school year, after attrition, the characteristics of the 

resulting sample of 389 students are presented in Table 4 below. The final sample consisted 

primarily of low-income and minority students. Table 5 presents the baseline PALS and MAP 

scores for the final sample. Table 6 presents final scores on the PALS, MAP, and school 

absences. 
SPARK Evaluation 9 | P a g e

Table 4: Final sample of study participants 

Control SPARK Total

138

137

114

61

Grade Level K-1st 

1st-2nd 

2nd 3rd – 

72

63

59

66

74

55

School Brown 27 34

Cass 21 25 46

Clarke 25 21 46

81st 27 24 51

Engleburg 

Rogers Street 

Sherman 

32

31

31

34

32

25

66

63

56

295

63

31

196

Race/Ethnicity Black 

Hispanic 

Other 

149

29

16

146

34

15

Gender Female 96 100

Male 98 95 193

17

372

366

F/R Lunch Eligible No 

Yes 

9

185

8

187

IEP No 182 184

Yes 12 11 23

Total 194 195 389

Table 5: Fall 2013 MAP and PALS scores – Final sample 

MAP Reading RIT MAP Math RIT PALS 

Grade Mean SD n Mean SD n Mean SD n 

Control K 143.9 9.0 72 139.0 9.0 69 66.6 20.7 70

1st 156.2 9.9 63 156.0 11.5 59 51.3 18.3 60

2nd 163.1 11.3 59 167.7 9.6 58

Total 153.7 12.8 194 153.4 15.6 186 59.5 21.0 130

SPARK K 141.4 11.3 66 137.9 9.7 64 59.7 22.3 64

1st 156.0 11.7 74 156.8 13.0 74 50.7 17.4 74

2nd 164.4 12.5 55 168.5 10.2 54

Total 153.4 15.0 195 153.8 16.5 192 54.9 20.2 138

Total K 142.7 10.2 138 138.5 9.4 133 63.3 21.6 134

1st 156.1 10.9 137 156.5 12.3 133 51.0 17.8 134

2nd 163.7 11.9 114 168.1 9.8 112

Total 153.6 13.9 389 153.6 16.1 378 57.2 20.7 268
SPARK Evaluation 10 | P a g e

Table 6: Spring 2015 MAP, PALS, and school absences – Final sample 

MAP Reading RIT PALS Absences 

Grade Mean SD n Mean SD n Mean SD n 

Control K 173.7 13.2 71 39.0 17.8 71 32.8 28.5 71

1st 176.5 15.3 61 51.8 20.0 59 27.4 24.4 61

2nd 188.7 12.5 59 66.6 15.5 55 25.0 21.1 59

Total 179.2 15.1 191 51.3 21.1 185 28.7 25.2 191

SPARK K 173.0 8.9 65 40.5 13.3 64 29.8 26.5 65

1st 181.1 15.2 74 61.1 13.7 70 22.8 17.1 74

2nd 189.8 14.8 54 68.0 16.9 53 20.7 14.2 54

Total 180.8 14.8 193 56.0 18.5 187 24.6 20.4 193

Total K 173.3 11.3 136 39.7 15.7 135 31.4 27.5 136

1st 179.0 15.4 135 56.8 17.4 129 24.9 20.7 135

2nd 189.2 13.6 113 67.3 16.1 108 23.0 18.2 113

Total 180.0 14.9 384 53.7 19.9 372 26.6 22.9 384

Participation 

SPARK students received an intensive amount of services across the two program years 

while control group students received the “business as usual” reading instruction provided by 

MPS. The average SPARK student received 122.5 tutoring sessions (SD = 27.3). Figure 1 

presents the distribution of tutoring sessions students received. It is important to note that three 

students did not receive any tutoring sessions, as they were misidentified by program staff as 

control students. These students were still included as SPARK participants in subsequent 

analyses. Parents of SPARK students were also engaged, with the average family attending three 

family events (Range 0 to 10, SD = 2.2), receiving 32 parent contacts (Range 0 to 69, SD = 13.9) 

and 2.4 home visits (Range 0 to 8, SD = 1.4). Figures 2 and 3 presents the distribution of parent 

contacts and home visits per student. Again, the three misidentified students did not receive any 

parent engagement services as part of SPARK. 
SPARK Evaluation 11 | P a g e

Figure 1: Distribution of tutoring sessions received by SPARK students 

Figure 2: Distribution of parent contacts per SPARK student 
SPARK Evaluation 12 | P a g e

65

53

31

25

13

3 3 2

0 1 2 3 4 5 6 7

Figure 3: Number of home visits per student 

Results 

Reading Achievement – MAP 

SPARK was found to significantly impact student reading achievement. The overall, 

unstandardized effect of SPARK on the MAP was 2.8 RIT score points (Table 7). This 

corresponds to an effect size (Hedges g) of .23. Effect sizes can be converted into Cohen’s 

(1977) U3 improvement index, which estimates the difference in mean percentile rank between 

intervention and control students. The effect size of .23 suggests an improvement index of 9.1%. 

Put another way, the average SPARK student would rank at the 59th percentile among the control 

group. 

The pooled interaction term of SPARK participation and baseline PALS scores/MAP 

scores was significant (t = 2.17, p < .05) suggesting that SPARK differentially impacted MAP 

scores for students with different starting literacy and achievement levels. Based on this, models 

predicting MAP Reading scores were done separately for students starting in the bottom half and 

top half of PALS scores (for K-1) and MAP Reading scores (2nd grade). The results of these 

models demonstrate that SPARK had a much larger impact on the reading achievement of lessliterate students. The unstandardized effect of SPARK on these students was 4.4 RIT score 
SPARK Evaluation 13 | P a g e

points on the MAP (Table 7). This corresponds to a large effect size (Hedge’s g) of .36 and an 

improvement index of 14.1%. 

Table 7: Effects of SPARK on MAP Reading scores 

B SE t-stat p-value 

Unstandardized Overall K 1.99 1.35 1.47

First Grade 6.04 1.81 3.33

2nd Grade 1.23 1.70 0.73

Overall 2.80 0.67 4.15 <.001 

Low K 2.56 2.21 1.15

First Grade 9.00 2.88 3.12

2nd Grade 2.65 3.13 0.84

Overall 4.39 1.11 3.97 < .001 

High K 

First Grade 

1.21

6.59

1.68

2.53

0.72

2.60

2nd Grade 1.94 2.67 0.73

Overall 2.66 0.84 3.18 < .01 

Standardized Overall K 0.18 0.12 1.47

First Grade 0.39 0.12 3.33

2nd Grade 0.09 0.12 0.73

Overall 0.23 0.06 3.97 <.001 

Low K 0.23 0.20 1.16

First Grade 0.59 0.19 3.12

2nd Grade 0.20 0.23 0.85

Overall 0.36 0.09 3.82 < .001 

High K 

First Grade 

0.11

0.43

0.15

0.16

0.72

2.60

2nd Grade 0.14 0.20 0.73

Overall 0.23 0.07 3.07 < .01 

Literacy – PALS 

SPARK was found to also significantly impact student literacy. The overall, 

unstandardized effect of SPARK was 5.7 PALS scale points (Table 8). This corresponds to an 

effect size (Hedge’s g) of .35 and an improvement index of 13.7%. The pooled interaction term 

of SPARK participation and baseline PALS scores/ MAP scores was again significant (t = 5.61, 

p < .001) suggesting that SPARK differentially impacted PALS scores for students with different 
SPARK Evaluation 14 | P a g e

starting literacy and achievement levels. Based on this, PALS models were also done separately 

for students starting in the bottom half and top half of PALS scores (for K-1st grade) and MAP 

Reading scores (2nd grade). The results of these models demonstrate that SPARK again had a 

much larger impact on the literacy of less-literate students. The unstandardized effect of SPARK 

on PALS for these students was 10.5 points (Table 8). This corresponds to an effect size 

(Hedge’s g) of .66 and an improvement index of 24.5%. 

Table 8: Effects of SPARK on PALS scores 

B SE t-stat p-value 

Unstandardized Overall K 

First Grade 

2nd Grade 

5.37

9.62

2.53

2.00

2.47

2.37

2.69

3.89

1.07

Overall 5.69 0.99 5.76 <.001 

Low K 

First Grade 

2nd Grade 

7.49

19.83

2.18

2.91

3.76

5.15

2.57

5.28

0.42

Overall 10.47 1.46 7.19 < .001 

High K 

First Grade 

2nd Grade 

-0.79

1.05

3.74

2.23

2.26

1.36

-0.35

0.46

2.75

Overall 2.21 0.92 2.41 < .01 

Standardized Overall K 

First Grade 

2nd Grade 

0.34

0.55

0.16

0.13

0.14

0.15

2.69

3.89

1.07

Overall 0.35 0.06 5.69 <.001 

Low K 

First Grade 

2nd Grade 

0.48

1.14

0.14

0.19

0.22

0.32

2.57

5.28

0.42

Overall 0.66 0.09 7.09 < .001 

High K 

First Grade 

2nd Grade 

-0.05

0.06

0.23

0.14

0.13

0.08

-0.35

0.46

2.74

Overall 0.13 0.06 2.37 < .05 

 As an additional analysis of the PALS, scores were converted into literacy benchmarks, 

indicating whether students were on track in their literacy development. Table 9 presents the 
SPARK Evaluation 15 | P a g e

cross tabulation of baseline and post-SPARK benchmark status for first and second grade control 

and SPARK students. This table shows that there were 41 (21 SPARK and 20 control) students 

that started the evaluation below benchmark, while at post, there were 87 (35 SPARK and 52 

control). The most telling finding here is that nearly all (18) of the 20 control students that started 

below benchmark finished below benchmark, while most SPARK students, 13 out of 21, that 

started below benchmark finished at or above benchmark. That 62% of SPARK and only 10% of 

control students who started below benchmark on the PALS were able to catch up demonstrates 

the sizable impact that SPARK had on students who needed more literacy support. 

Table 9: Cross-tabulation of students meeting PALS benchmark before and after SPARK 

Fall 2013 

Spring 2015 (Post SPARK) 

Below 

benchmark 

Met Total benchmark

Control Below benchmark 18 2 20

(Pre SPARK) Met benchmark 34 74 108

SPARK Below benchmark 8 13 21

Met benchmark 27 85 112

Total Below benchmark 26 15 41

Met benchmark 61 159 220

Total 87 174 261

School Attendance 

Finally, SPARK was found to also significantly impact school attendance. The overall, 

unstandardized, effect of SPARK was 5.8 absences (Table 10), indicating that SPARK students 

had 5.8 fewer absences than control students. This corresponds to an effect size (Hedge’s g) of 

.25 and an improvement index of 9.9%. The pooled interaction term of SPARK participation and 

baseline PALS scores/ MAP scores was not a significant predictor of school absences (t = 1.04, p

> .05). Therefore, attendance models were not conducted separately for students with low and 

high baseline literacy levels. 

Rates of chronic absenteeism (defined as missing at least 10% of school days) of SPARK 

and control students were then compared to further explore how these findings impact students. 

While 54 (28.6%) control students were chronically absent from school, only 40 (20.8%) 
SPARK Evaluation 16 | P a g e

SPARK students were. Based on this, SPARK students were 27% less likely to be chronically 

absent from school than control students. 

Table 10: Effects of SPARK on Attendance (Number of absences) 

B SE t-stat p-value 

Unstandardized K 5.6 4.3 1.31

First Grade 5.3 3.7 1.45

2nd Grade 6.6 4.0 1.63

Overall 5.8 1.9 3.00 <.01 

Standardized K 0.20 0.15 1.31

First Grade 0.26 0.18 1.45

2nd Grade 0.36 0.22 1.62

Overall 0.25 0.08 3.31 <.001 

Conclusions 

576 students in the Milwaukee Public Schools participated in a randomized control trial 

of SPARK, an early grade literacy program that combines one-on-one tutoring with family 

engagement to impact the literacy development of students. During the two study years, about 

34% of students left SPARK because they moved away. The results of the evaluation of SPARK 

on the remaining 380 (188 control and 192 SPARK) students found that SPARK had statistically 

significant positive impacts on reading achievement, literacy, and regular school day attendance. 

SPARK students were absent from school, on average, 5.8 fewer times than control students. The 

benefit of SPARK was greater with students who started the program with greater literacy 

instruction needs; while only 10% of control students below literacy benchmarks at the start of 

the study met benchmark at the conclusion of the study, 62% of SPARK participants who started 

below benchmark met benchmark at the end of the study. Taken together, these results suggest 

that SPARK is meeting its goal of addressing the challenge of helping MPS schools and parents 

teach students how to read. 
SPARK Evaluation 17 | P a g e

References 

Bryan, J. (2005). Fostering educational resilience and achievement in urban schools through 

school-family-community partnerships. Professional School Counseling, 8(3), 219. 

Chang, H. and Romero, M. (September, 2008). Present, Engaged, and Accounted For: The 

Critical Importance of Addressing Chronic Absence in the Early Grades. National Center 

for Children in Poverty. Retrieved online at: 

http://www.nccp.org/publications/pdf/text_837.pdf 

Cohen, J. (1977). Statistical Power Analysis for the Behavioral Sciences (revised edition). New 

York: Academic Press. 

Graue, E. and Hawkins, M. (2010).”I always feel like they don’t know anything about us”: 

Diverse families talk about their relations with school. In Miller-Marsh, M. and TurnerVorbeck, T. (Eds.) (Mis)Understanding Families: Learning from Real Families in our 

Schools (pp.109-125) New York, NY: Teachers College Press. 

Little, P. M.. (2009). Supporting Student Outcomes through Expanded Learning Opportunities. 

Harvard Family Research Project. Retrieved online at: http://www.hfrp.org/out-ofschool-time/publications-resources/supporting-student-outcomes-through-expandedlearning-opportunities 

Epstein, J.L. (2001). School, Family, and Community Partnerships: Preparing Educators and 

Improving Schools. Boulder, CO: Westview Press. 

Li, G. (2010). Social class, culture and “good parenting”: Voices of low-SES families. In MillerMarsh, M. and Turner-Vorbeck, T. (Eds). (Mis)Understanding Families: Learning from 

real families in our schools (pp.162-178). New York, NY: Teachers College Press. 

Pinnell, G. S., Lyons, C. A., DeFord, D. E., Bryk, A., & Seltzer, N. (1994). Reading Research 

 Quarterly, 29, 8-39.