1 

Summer Advantage Final Report 

Final Report for the Summer Advantage 

Program in the Roaring Fork School 

District

APA Consulting 

May 2017 
Summer Advantage Final Report

2

Acknowledgements

This work was funded by Mile High United Way and the Corporation for National and Community 

Service’s Social Innovation Fund grant. The report was produced in cooperation with Summer Advantage 

and Summit 54.

Special thanks

Augenblick, Palaich and Associates (APA) is a Denver-based research and consulting firm. Founded in 

1983, APA has worked in all 50 states addressing key education issues including school finance, educator 

effectiveness, early childhood education, and program evaluation. APA would like to thank Summer 

Advantage and Summit 54 for selecting APA to serve as the external evaluator for this study, and Mile 

High United Way for their efforts as the intermediary for the Corporation for National and Community 

Service’s Social Innovation Fund grant that makes this evaluation possible.

Principal authors:

Jack Hill, Dale DeCesare, and Abby McClelland.

Recommended citation:

Augenblick, Palaich and Associates, (2016). Final Five Year Evaluation Report for the Summer Advantage 

Program in Colorado’s Roaring Fork School District. Denver, Colorado.

The contents of this document are solely the responsibility of Augenblick, Palaich and Associates and do 

not represent the official views of the Corporation for National and Community Service’s Social 

Innovation Fund or Mile High United Way.

This document may be reproduced in whole or part without restrictions as long as Augenblick, Palaich 

and Associates is credited for the work. Upon request, the contents of this document will be made 

available in alternate formats to serve accessibility needs of persons with disabilities.

Funding:

This report is based upon work supported by the Social Innovation Fund (SIF), a key White House 

initiative and program of the Corporation for National and Community Service (CNCS). The Social 

Innovation fund combines public and private resources to grow the impact of innovative, communitybased solutions that have compelling evidence of improving the lives of people in low-income 

communities throughout the United States.
Summer Advantage Final Report 

3 

Table of Contents

Final Report for the Summer Advantage Program in the Roaring Fork School District................................1

Executive Summary.......................................................................................................................................5

Program Description ................................................................................................................................. 5

Prior Research and Study Design ..............................................................................................................5

Level of Evidence Targeted .......................................................................................................................5

Evaluation Measures and Analysis............................................................................................................6

Overall Findings.....................................................................................................................................6

Program Implementation......................................................................................................................6

Student Outcomes................................................................................................................................7

Introduction ..................................................................................................................................................8

Overview of the Implementation Monitoring Analysis.............................................................................8

Overview of the Impact Analysis...............................................................................................................9

Summer Advantage.....................................................................................................................................11

History.....................................................................................................................................................11

Programming...........................................................................................................................................11

Partnerships............................................................................................................................................12

Goals........................................................................................................................................................12

Implementation Analysis ............................................................................................................................15

Background .............................................................................................................................................15

Research Questions and Activities..........................................................................................................15

Research Findings ...................................................................................................................................17

Performance on Program Objectives..................................................................................................18

Conclusions.................................................................................................................................................30

Implementation Analysis Conclusions....................................................................................................30

Impact Analysis Conclusions...................................................................................................................31
Summer Advantage Final Report

4

List of Tables

Table 1: Implementation Analysis Questions – p. 15

Table 2: Implementation Analysis Program Objectives to be Monitored – p. 16

Table 3: Student Recuriting Objectives were Fully Met – p. 17

Table 4: Summer Advantage Continues to Meet Enrollment Goals – p. 17

Table 5: 2015 Completion and Attendance Rates Remain High – p. 18

Table 6: Demographic Characteristics of Summer 2015 SA Students, by Completion Status – p. 19

Table 7: Objectives to Recruit Qualified Staff were Fully Met – p. 19

Table 8: Objectives About Training Teachers were Partially Met – p. 22

Table 9: Objectives About Building Program-District Relationships were Fully Met – p. 24

Table 10: Objectives About Assessing Students were Fully Met – p. 25

Table 11: Objectives About Providing Students with Appropriate Daily Academic Instruction and 

Enrichment were Fully Met – p. 25

Table 12: Objectives About Ensuring that Teachers and Staff Use Culturally Competent Curriculum were 

Partially Met – p. 26

Table 13: Objectives About Ensuring Maintenance of Faculty-to-Student Rations were Fully Met – p. 28

Table 14: Objectives about Parental Participation in the Program were Fully Met – p. 28

Table 15: Breakdown of Student Sample by Demographic Factors – p. 30

Table 16: The Post-Match Groups of SA and Comparison Students are Comparable – p. 32

Table 17: Confirmatory and Exploratory Research Questions – p. 33

Table 18: Summer Advantage Participation is Associated with Higher DIBELS Scores – p. 34

Table 19: Kindergarten Participation in SA is Associated with Higher DIBELS Next Scores – p. 35

Table 20: Preliminary Effect Sizes – p. 36
Summer Advantage Final Report 

5 

Executive Summary 

Program Description 

Summer Advantage (SA) is a voluntary, five-week intensive summer learning program that provides 

summer enrichment to elementary school students to help mitigate summer learning loss. The program, 

which has the explicit goal of holding student literacy scores steady throughout the summer months, 

targets low-income students, who tend to experience the steepest declines in learning over the 

summer. By reducing summer learning loss, the program expects to improve student literacy outcomes. 

On average, SA delivers services to more than 750 kindergarten through fourth grade students in the 

Roaring Fork School District (RFSD) which serves a mountain community in Colorado’s western slope. 

There is no cost to students to attend the programs, but the program uses different funding sources to 

underwrite the costs of serving students. The program receives funding for some of these students 

through Mile High United Way (MHUW) Social Innovation Fund (SIF) funding. The Colorado State Board 

of Education has identified RFSD as a “Closing the Achievement Gap District,” where raising the 

academic achievement of non-white students is a high priority. 

Prior Research and Study Design 

Augenblick, Palaich and Associates (APA), the independent evaluator for Summer Advantage, conducted 

this rigorous, five-year evaluation study as part of a SIF grant through Mile High United Way. The 

Summer Advantage program is modeled off of the Building Education Leaders for Life (BELL) summer 

program in Boston and New York, which was evaluated by the Urban Institute with a randomized 

controlled trial. There have been no previous formal studies of Summer Advantage’s adaptation and 

implementation of the BELL program in Colorado. APA’s five-year evaluation of the program is intended 

both to document the fidelity of implementation of the Summer Advantage program in the Roaring Fork 

Valley School District and to evaluate the impact of the program on student literacy outcomes at the 

end of the summer. 

Level of Evidence Targeted 

APA’s five-year study targeted a “moderate” level of student impact evidence – as defined by CNCS 

evaluation guidelines – and was conducted from 2012-2017, with student impact data collected for the 

2014, 2015, and 2016 summer sessions. This evaluation was unable to pursue a “strong” level of 

evidence as defined by CNCS, because this standard requires that programs be studied using random 

assignment or a state or national level multi-site study. In this case implementing a randomized 

controlled trial which randomly assigned certain children to participate in the Summer Advantage 

program and others to receive no such support was impractical and ethically objectionable to both the 

program and school district involved. Similarly, a multi-site study was impossible given the scope of the 

Summer Advantage program in the Roaring Fork Valley. Given the availability of strong quasiexperimental methods to conduct education research, this evaluation expects to establish a moderate 

level of evidence using a propensity score matching approach.
Summer Advantage Final Report 

6 

Evaluation Measures and Analysis 

To measure the fidelity with which the program is implemented according to its stated plans, APA relies 

on document review, interviews with program and district leaders, focus groups with program teachers, 

and a site visit each summer to gather data on program implementation fidelity. For the program’

impact analysis, APA is examining the impact on participating students using the district-administered 

literacy assessment, Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Next.1 The analysis of these 

assessment scores involves using propensity score matching to generate a comparison group of students 

with similar demographics and spring assessment scores (the pre-test) who did not participate in 

Summer Advantage. The spring assessment scores for both the comparison group and the Summer 

Advantage student group are gathered and analyzed. After the summer program is delivered, the 

analysis then uses a Hierarchical Linear Model to estimate the program effect on both student groups’

fall literacy scores (the post-test). 

This report combines findings across three SA summer sessions. Data are pooled so that results from the 

2014 summer cohort are combined with results from the 2015 and 2016 SA cohorts.

Overall Findings 

Program Implementation 

Studying program implementation fidelity is a critical element of any education program evaluation. The 

goal of studying implementation fidelity is to understand the key elements that make up a particular 

program, then to document how those elements were actually implemented on the ground as planned. 

These steps are essential to establishing a logic chain that verifies a program’s key components are 

linked to outcomes that the evaluation attempts to measure, including impacts on teachers or students. 

APA conducted a full implementation study of the SA program in 2013. This study, which included a 

series of focus groups with teachers and principals, interviews with district leaders, extensive reviews of 

program documents, and site visits to the Summer Advantage program, found that the program was 

implemented overall in accordance with its intended design. Since the full implementation study in 

summer 2013, APA has conducted implementation monitoring in 2014, 2015, and 2016. This monitoring 

included similar, but less intensive data gathering each year to verify that no significant modifications 

were made to the program. APA’s implementation monitoring included all SA students in Glenwood 

Springs, Sopris, and Basalt elementary schools.2 

Over the three years of implementation monitoring, APA found that Summer Advantage was 

implemented with fidelity every summer and met the program’s implementation goals. These included 

 

1

 Although the Summer Advantage program assesses participating students using the STAR assessment 

during the program, STAR assessment scores were not available for students who did not participate in 

Summer Advantage. For this reason, the impact evaluation focuses only on the outcome of fall DIBELS 

scores. 

2

 The Glenwood and Sopris school sites are combined during the summer, to form the Glenwood/Sopris 

Summer Advantage site. 
Summer Advantage Final Report

7

identifying and recruiting students who met program criteria; recruiting, hiring, and training qualified 

staff; building strong relationships with school and district-level leaders; identifying schools; teaching 

academic content and providing enrichment experiences using an appropriate curriculum aligned to 

district objectives; maintaining appropriate instructor/student ratios in classrooms; and maintaining 

clearly communicated program expectations with parents. Achieving these goals reflects the program’s 

significant success in building connections with parents and education and community leaders. It also 

reflects the program’s continued ability to effectively communicate expectations and implementation 

goals to staff at all levels and to obtain staff buy-in for maintaining program fidelity.

Student Outcomes

Gathering and analyzing data across three summer sessions, APA found a statistically significant and 

positive impact of SA on student reading performance for kindergarteners and 1st graders. Kindergarten 

students who participated in SA scored 15 percentage points higher on fall assessments than similar 

students who did not attend Summer Advantage. First grade students who attended SA scored six 

percentage points higher on fall assessments than similar students who did not attend Summer 

Advantage. 
Summer Advantage Final Report 

8 

Introduction 

This is the final Mile High United Way (MHUW) Social Innovation Fund (SIF) evaluation report for 

Summer Advantage (SA), sponsored by Summit 54 in Colorado’s Roaring Fork Valley. This report has two 

main components: (1) an implementation monitoring review of SA over four summers (including the 

program’s implementation in summer 2013, 2014, 2015, and 2016) and (2) an impact analysis of SA 

(looking at the impacts of the 2014, 2015 and 2016 years of the program).3 This report presents the 

implementation monitoring and impact evaluation components of the study separately because they 

involve different research questions and, to some extent, different student samples. The final section of 

the report presents conclusions, recommendations, and anticipated next steps for the evaluation. 

SA is a voluntary, five-week intensive summer learning program that provides summer enrichment to 

elementary school students to help mitigate summer learning loss. The program, which has the explicit 

goal of holding student literacy scores steady throughout the summer months, targets low-income 

students, who tend to experience the steepest declines in learning over the summer months. By 

reducing summer learning loss, the program expects to improve third grade literacy outcomes. SA 

delivers services to more than 750 kindergarten through fourth grade students in the Roaring Fork 

School District (RFSD) which serves a mountain community in Colorado’s western slope. Some of these 

students have their SA attendance funded through MHUW SIF and some have their attendance funded 

through other means. The Colorado State Board of Education has identified RFSD as a “Closing the 

Achievement Gap District,” where raising the academic achievement of non-white students is a high 

priority. 

 

WHAT IS SUMMER ADVANTAGE (SA)? 

SA is a five-week summer reading and math program designed to reduce summer learning loss, 

and to provide enrichment particularly for children from lower income backgrounds. SA receives 

funding support from external organizations including Summit 54 and Mile High United Way 

through the Social Innovation Fund. 

APA Consulting (APA), an independent evaluation and research firm founded in Denver in 1983, 

conducted a five-year evaluation of SA, funded through MHUW SIF. This report covers all five years of 

APA’s evaluation activities. Year 5 evaluation activities focused on kindergarten through third grade 

students from three of RFSD’s four public elementary schools: Glenwood Springs Elementary, Sopris 

Elementary, and Basalt Elementary. 

Overview of the Implementation Monitoring Analysis 

APA’s evaluation of SA builds on findings from prior research on the Building Education Leaders for Life 

(BELL) summer programs (Chaplin & Capizzano, 2006), and is also informed by the yearly 

3

The implementation monitoring at times includes data pertaining to SA programming across the entire Roaring 

Fork School District. However, one school (Crystal River) and a grade level (fourth) were not included in APA’s

impact evaluation. The impact analysis looks only at students in kindergarten through third grades at Basalt, 

Glenwood Springs, and Sopris Elementary Schools. While impact evaluation findings cover the 2014-2016 years of 

the SA program, the implementation evaluation and monitoring findings cover the 2013-2016 program years. 
Summer Advantage Final Report 

9 

implementation study APA completed in 2013. The 2013 implementation study used data collected in 

2012 and 2013 in the Roaring Fork School District. The study found that SA has a clear theory of change4

and is implemented with fidelity. In the subsequent years including 2014-2016 APA did not conduct full 

implementation studies but instead conducted implementation monitoring to track the extent to which 

the program maintained overall fidelity to its intended implementation model. 

APA’s implementation monitoring analysis examined SA’s summer implementation at sites serving 

children from three RFSD elementary schools. APA conducted focus groups with teachers, teaching 

assistants (TAs), SA administrative staff, and SA national program staff and a site visit to the RFSD school 

district during each summer to observe program operations. APA also continued to monitor SA internal 

data related to program implementation. 

Consistent with findings from APA’s 2013 implementation study, APA found that SA continues to be 

implemented with fidelity in Colorado. The SA program from 2014-2016 was overall implemented in 

accordance with organizational expectations and with the implementation goals outlined in the contract 

between Summit 54 and MHUW. Specifically, SA was able to recruit, hire, and train staff; identify at least 

three schools from which to recruit students (in 2015 and 2016 there were two SA program sites serving 

children from three RFSD schools); identify and recruit students who met SA criteria (e.g. students from 

low-income households, students at risk of summer learning loss, students whose families could meet 

SA attendance and parental involvement requirements); assess students using the STAR assessment5; 

build strong relationships with school- and district-level leaders; build fidelity to the SA program model 

among site program managers, teachers, and program staff; provide academic content as well as 

enrichment experiences. These achievements reflect the program’s significant success in building 

connections with parents, educators, and community leaders in the Roaring Fork Valley. They also 

reflect the program’s continued ability to communicate expectations and implementation goals to staff 

at all levels, and to instill a high level of staff buy-in to the program. 

Overview of the Impact Analysis 

In 2014, APA conducted the first year of a multiyear analysis of SA’s impacts on student academic 

performance. APA’s impact analysis compares the reading skills of students who attended SA to those of 

similar students who did not attend SA. APA’s 2014 Year 1 impact analysis established processes and 

procedures that the research team continued to use throughout the multiyear study. This facilitated 

APA’s pooling of data from across years of the SA program to build up larger sample sizes of 

participating students. 

 

4 A theory of change provides a model (usually a visual diagram, or “logic model”) of the assumptions about how 

the SA program achieves expected outcomes. 

5

 The Summer Advantage program uses the STAR assessment for internal progress monitoring over the course of 

the program. The school district does not use the STAR assessment. Because STAR scores are available only for 

students who participated in Summer Advantage, the impact evaluation uses the DIBELS assessment administered 

by the district as the outcome measure.
Summer Advantage Final Report

10

In Year 1, the APA research team was able to successfully establish processes for exchanging and 

managing sensitive data files with the district; matching students to identify a balanced treatment and 

comparison sample; and creating and applying statistical models to analyze student performance data.

Schools in the Roaring Fork School District administer the Dynamic Indicators of Basic Early Literacy Skills 

(DIBELS) Next assessment as part of their compliance with Colorado’s Reading to Ensure Academic 

Development (READ) Act (CO HB 12-1238). The assessment provides a tool to measure student reading 

progress across districts. Schools also administer Northwest Evaluation Association (NWEA) Measures of 

Academic Progress (MAP) assessments, which APA uses to match students who did and did not 

participate in SA.

To achieve a moderate level of evidence of program impacts on student achievement, APA compared 

DIBELS performance between SA students and similar, non-SA students in RFSD. To do this, APA used a 

sampling technique called Propensity Score Matching (PSM). After completing the sampling and 

controlling for student characteristics, school location, and prior student performance, the researchers 

compared the fall DIBELS scores of SA and non-SA students. Researchers analyzed DIBELS scores of 

students from three out of the four RFSD elementary schools that participated in SA during the summers 

of 2014, 2015, and 2016 (the fourth school, Crystal River, was outside of the scope of the current 

evaluation). APA’s analysis for this final report includes data pooled from the 2014-2016 program years.
Summer Advantage Final Report

11

Summer Advantage

SA is a voluntary summer learning program for students. In 2016, SA completed its fifth year in RFSD. 

This section describes SA’s program, partnerships, history, and goals.

History

Summer Advantage USA is a national program first developed in Indiana. SA was originally modeled on

the BELL summer program out of Boston and New York. The BELL program operates primarily in urban 

areas with high populations of academically underserved children.

In 2005, researchers worked with the Urban Institute to evaluate the BELL program. The authors 

performed a randomized controlled trial (RCT) study of a sample of 1,087 children in first through 

seventh grades who applied to participate in the BELL summer programs in New York and Boston in 

summer 2005 (Chaplin & Capizzano, 2006). The researchers were able to collect complete test score 

data from 835 of the sample participants, amounting to a response rate of 78 percent for both the 

control group and the treatment group. The study authors were able to demonstrate a strong level of 

evidence of the BELL program’s effectiveness. They found an improvement in reading test scores 

equivalent to approximately two additional months of reading instruction for the BELL participants when

compared to their control group peers (Chaplin & Capizzano, 2006). Additionally, the researchers found 

that the program, as implemented, had an impact on summer learning activities and parental

involvement in reading (Chaplin & Capizzano, 2006).

SA is an adaption of the BELL program, with key differences in design and setting. In particular, (1) SA 

decreases the summer school dosage of the BELL program from six weeks to five, due to districtprogram agreements; (2) SA expands the model to a new geographic region (Colorado); (3) It operates in 

a western state, where school districts enjoy a higher degree of local control than in some other parts of 

the U.S.; (4) The SA program includes kindergarteners; and (5) it operates in a rural mountain setting, as 

opposed to the primarily urban setting where the BELL program has typically operated. 

Programming

SA takes place over five weeks during the summer, operating five days a week for six and half hours a 

day. On Monday through Thursday, students receive both literacy and math instruction in the morning 

followed by two hours of enrichment activities in the afternoon, including art, drama, and physical 

education. On Fridays, SA brings in inspirational speakers, takes students on field trips, or engages 

students in other activities. 

One significant component of the SA program is its culture of academic achievement. Staff members are

committed to the idea that all children can achieve academic success. Staff refer to SA participants as 

“scholars.” SA also emphasizes parental involvement, encouraging parents to support their children at 

home and requiring high student attendance rates. Parents are also encouraged to participate in SA field 

trips and enrichment activities.
Summer Advantage Final Report

12

SA employs one academic teacher and one TA for every 24 scholars. All staff undergo training prior to 

the start of the program. Each academic teacher is required to have a minimum of three years teaching 

experience, as well as a teaching license. Enrichment teachers must show content knowledge in the area 

they teach and must have previous experience working with children, but do not have to be certified 

teachers. TAs are generally college-age students working towards education degrees. They usually have 

previous experience with children or are looking to gain practical experience in elementary education. 

SA uses a nationally recognized summer school curriculum for kindergarten through eighth grade 

students. The program includes math and reading, along with weekly themed activities that build on 

each other. The program also allows for periodic assessments of student progress.

Partnerships

SA maintains multiple community partnerships. The program has built these partnerships with the help 

of Summit 54, a nonprofit organization focused on building community supports to benefit low-income 

students and help them to attend and graduate college. The founders of SA, Tony and Terri Caine, live in 

the nearby community of Aspen, which has a robust network of donors and community organizations,

and are able to take advantage of their many relationships to build community support for SA. 

Along with these community partnerships, SA also has partnerships RFSD school and district leaders. SA

works closely with each school principal and with the district as a whole and has a memorandum of 

understanding (MOU) outlining the terms and expectations of its relationship with the district. This MOU 

establishes the expectations that schools will help SA recruit and review student applicants; that RFSD 

will provide facilities for the SA program; and that RFSD will provide transportation for SA student 

attendees for the duration of the program.

Further, SA benefits from partnership grant funding. In 2012, Summit 54 received a $50,000 grant from 

the Colorado Department of Education to provide breakfast and lunch every day of the SA program. SA 

also receives a grant of approximately $236,000 per year (from 2012 through summer 2017) from the 

SIF program administered in Colorado by Mile High United Way. Summit 54 provides matching funds as 

part of SIF matching expectations.

Goals

Researchers focusing on differential opportunities as an underlying cause of an academic achievement 

gap have focused their attention on children’s learning experiences outside the formal school year such 

as experiences during summer. Barbara Heyns first documented that student achievement gaps did not 

remain stable over the summer months, but expanded at a faster rate than during the school year 

(Heyns, 1978). In other words, schools likely act as a stabilizing force for achievement gaps and when 

children are not in school, those gaps widen (Id.). This initial finding of expanding achievement gaps 

during summer months out of school has been replicated by a number of researchers (Alexander, 

Entwisle & Olsen, 2007; Downey, Von Hippel & Broh, 2004; Phillips & Chin, 2004). 
Summer Advantage Final Report

13

All students lose some amount of learning during the summer, when they are less likely to be actively 

engaged in learning and structured schoolwork (Fairchild & Boulay, 2002). This learning loss includes 

both factual knowledge and procedural knowledge, such as how to learn or participate in a classroom 

setting (Id.). Overall, students lose about one month of learning or about one tenth of a standard 

deviation of their spring test scores (Cooper, Nye, Charlton, Lindsay & Greathouse, 1996). Students tend 

to lose more learning in math than in reading, perhaps because of the greater opportunity for parents to 

maintain reading progress over the summer (Id.). The larger learning loss experienced by lessadvantaged children over summer breaks likely occurs because children with lower socioeconomic 

status experience fewer learning opportunities at home and when not in school than students of higher 

socioeconomic status (Alexander, Entwisle & Olsen, 2001). These learning opportunities could include 

opportunities such as going to museums or the zoo with parents, reading at home, or attending an 

educational summer camp. 

Further research into the effect of summer school on expanding achievement gaps shows that the 

expansion of the gap over the summer is likely more profound than changes in achievement gaps over 

the course of the school year. A study in Baltimore found that even when black and white students had 

similar achievement scores when starting first grades, their test score changes over the summer created 

an achievement gap of about half a standard deviation in only two years (Entwisle & Alexander, 1992). 

This means that even if students have an equal opportunity during the school year to keep pace with 

their fellow students, differences in summer experiences can still create a dramatic achievement gap in 

relatively little time. This finding suggests that summer interventions to prevent learning loss are 

especially important during early grades to avoid an ever-broadening achievement gap (Schacter, 2003). 

Summer programs, like SA, can “mitigate learning loss and could even produce achievement gains” (p. 

xiii). 

WHAT IS SUMMER LEARNING LOSS? 

Summer learning loss is a well-documented phenomenon wherein students lose math and 

literacy skills during their time away from school. It disproportionately affects lower-SES 

students, who tend to spend their summers in lower-quality environments, having fewer 

enriching experiences than their higher-SES peers. It contributes to the larger income-based 

achievement gap in the U.S. education system. Summer programs seek to decrease this loss. 

With this research base in mind, SA makes itself open to all RFSD students, but prioritizes students from 

lower-income backgrounds. In this way, SA aims to reduce summer learning loss among lower-income 

students, “leveling the playing field” of education, and decreasing the income-based achievement gap. 

SA’s theory of change is that learning loss can be reduced for students when they are provided with 

high-quality instruction throughout the summer months. The SA theory of change, illustrated below, 

outlines SA’s inputs, activities, outputs, and intended impacts. The theory of change is an iterative tool, 

meaning that it provides a framework for planning, implementation, and evaluation of SA, both 

throughout the MHUW SIF grant term and beyond. If SA is implemented with fidelity, then its projected 

outputs should translate to positive outcomes for scholars.
Summer Advantage Final Report 

THEORY OF CHANGE 

INPUTS 

Certified/licensed administrators, including a summer principal and AP; high-quality, certified 

teachers, TAs, and staff; Houghton Mifflin Harcourt Summer Success program; existing 

evidence; community partnership; school partnership, including transportation and facility 

use; expertise of external evaluators; and Community Grant support. 

14 

ACTIVITIES 

Program operates 6.5 hours/day, 5 days/week; has highly qualified, certified teachers and TAs 

lead each class (2-to-24 staff-to-scholar ratio); encourages parental engagement through 

parent/teacher meetings, workshops, field trips, and other activities; uses nationally 

recognized summer reading and math curricula; and includes enrichment activities like art, 

music, drama, gym, field trips, and service projects. 

OUTPUTS 

Outputs include the teachers and TAs trained in Houghton Mifflin Summer Success Math and 

Reading curricula; students recruited to SA; students who complete SA; rate of attendance for 

students who complete SA; staff-to-scholar ratio; teachers who return for subsequent SA 

programs; and students who return for subsequent SA programs. 

SHORT-TERM OUTCOMES 

Program produces positive changes in pre-test to 

post-test performances; reduces summer 

learning loss, as measured by CBLA exam; 

improves student performances; allows SA staff 

to gain knowledge and skill in implementation of 

Houghton Mifflin curriculum; provides further 

theoretical and practical foundation in early 

literacy development; and allows students and 

teachers to develop a culture of success. 

LONG-TERM OUTCOMES 

Program improves third grade literacy outcomes; 

reduces achievement gaps within participating 

Roaring Fork Valley School District schools; 

improves academic outcomes for students who 

participate in more than one year of SA; and 

allows teachers as well as TAs to use what they 

learn in SA to improve their teaching for 

successive cohorts of the SA program. 

IMPACTS 

Evaluation supports replication of SA Program; evaluation makes meaningful contribution 

to the literature on summer intervention programs; SA reduces summer learning loss. 
Summer Advantage Final Report 

15

Implementation Analysis 

This section discusses APA’s complete SA implementation analysis, covering the implementation analysis 

background, key questions and activities (methods), and findings over the 2013-2016 summer program 

years. 

Background 

APA conducted a full implementation study of the SA program in 2013. This study was designed to help 

determine if SA was implemented with consistency and fidelity to its stated program model. 

WHAT IS THE PURPOSE OF AN IMPLEMENTATION ANALYSIS? 

An implementation analysis helps determine if a program is being implemented with consistency 

and fidelity to its stated program model or design. It usually involves researching a program’s 

stated operation plans, reviewing program materials, interviewing involved parties, conducting 

observations of programming and operations, and synthesizing data through qualitative analysis. 

To conduct a full implementation analysis of SA in RFSD in 2013, APA (1) reviewed key program 

materials, such as training manuals and program guides; (2) conducted a summer 2013 site visit to RFSD 

to directly observe program operations; (3) conducted focus groups with SA academic teachers and 

enrichment teachers; and (4) conducted interviews with district leaders, school principals, SA program 

leaders, and SA program managers. APA analyzed the data produced through these efforts to produce 

an implementation study report. The report, completed in November 2013, found that SA 

• Launched in a new state (Colorado) and setting (rural mountain community) with consistent 

implementation across schools and with fidelity to program expectations and intended design; 

• Met student enrollment, attendance, and programming dosage goals; 

• Met goals for summer school teacher training and curriculum delivery; 

• Recruited staff who met program expectations for experience, motivation, and attendance; 

• Offered a curriculum that teachers and staff perceived to be well-structured, appropriate, and 

aligned with district academic expectations; and 

• Met internal evaluation goals for teachers, with each teacher receiving at least two formal 

observations, followed by feedback from SA staff. 

Since the full implementation study in summer 2013, APA has conducted implementation monitoring in 

2014, 2015, and 2016. APA’s implementation monitoring sample includes all SA students in 

Glenwood/Sopris and Basalt elementary schools (including fourth graders and students who were later 

released from the program for failing to meet attendance requirements). Over the three years of 

implementation monitoring, APA found that Summer Advantage was implemented with fidelity every 

summer. 

 Research Questions and Activities 

Implementation monitoring activities in 2016 were similar to previous summers and included: 
Summer Advantage Final Report 

16

• A site visit in summer 2016 to directly observe program operations; 

• Four focus groups with a total of 37 SA teachers (two focus groups for Sopris and Glenwood 

Springs teachers, who shared a program site, and two for Basalt Elementary teachers); 

• Focus groups with 22 SA TAs; 

• A program document review; 

• An interview with Summit 54 staff members; and 

• Informal interviews with SA national program staff. 

APA used research questions to guide the implementation analysis, including both student-level 

questions and program-level questions. Table 1, below, lists the questions. 

Table 1: Implementation Analysis Questions 

Question Level 

1. How many students complete the program? Student 

2. What are the attendance rates for students who complete the program? Student 

3. How many scholars return to the program in subsequent years? Student 

4. What are the attendance rates for teachers? Program 

5. What is the intended student-to-teacher ratio, and is this ratio achieved? Program 

6. What implementation challenges do program staff face? Program 

7. Are staff trained according to program expectations? Program 

8. Is the program able to recruit staff whose qualifications meet program 

requirements? 

Program 

9. What, if any, other summer academic programs exist in the district? Program 

Working with Summit 54 and SA, APA identified nine program objectives that could be monitored in for 

fidelity or quality of implementation. The table below provides a description of each objective and 

specific goals associated with that objective. 
Summer Advantage Final Report 

17

Table 2: Implementation Analysis Program Objectives to be Monitored 

Program Objective Measure

A. Recruit students 1. Recruit 500 students in one district. 

2. Recruit high need students. 

B. Recruit qualified staff 1. Recruit staff with a passion for working in high-need student settings. 

2. Recruit academic teachers who have at least three years of experience 

teaching, hold teacher licensure, and have at least a BA. 

3. Recruit enrichment teachers who have at least three years of experience 

teaching and have at least a BA. 

4. Achieve a near 100 percent teacher attendance rate during the program. 

C. Train teachers 1. Teachers complete online training modules. 

2. Teachers participate in a mandatory week-long professional training 

session prior to the program start. 

3. Teachers view training sessions as useful and high-quality. 

4. Teachers are evaluated at least twice during the program. 

D. Build program-district 

relationships 

1. Achieve buy-in and support for the program from district leaders. 

2. Achieve buy-in and support for the program from school principals. 

E. Assess students pre- and 

post-program 

1. Assess students at the beginning and end of the program using the STAR. 

F. Provide students with 

appropriate daily academic 

instruction and enrichment. 

1. Students miss no more than 2-3 days over the course of the program. 

2. Program staff call home for absent students. 

3. Program releases students who do not meet attendance expectations. 

4. Students participate actively in class. 

5. Students participate in field trips and enrichment activities. 

G. Ensure that teachers and 

staff use culturally competent 

curriculum. 

1. Curriculum is consistent with district expectations. 

2. Curriculum is consistent with student cultural needs. 

H. Ensure maintenance of 

faculty-to-student ratios. 

1. Minimum of 1 teacher and 1 TA per 24 students, with a goal of 19 

students per classroom. 

I. Encourage parent 

participation. 

1. Parents are required to attend a mandatory parent orientation meeting 

prior to their children being accepted into the program. 

2. Parents sign contracts with SA. 

3. Parents attend mid-summer parent-teacher conferences. 

4. Parents participate and sign nightly reading logs. 

5. Parents are encouraged to attend end-of-session graduation with lunch. 

During the course of each summer’s implementation evaluation monitoring, APA investigated each of 

the objectives listed in the table above. Information and findings in each area are provided in the 

“Research Findings” section, below. 

Research Findings 

Overall, APA finds that Summit 54’s summer SA program was implemented in accordance with 

organizational expectations and with the implementation goals outlined in the contract between 

Summit 54 and MHUW. The program was able to: 

• Identify and recruit students who meet SA criteria; 

• Recruit, hire, and train qualified staff; 

• Build strong relationships with school and district-level leaders; 

• Identify at least three schools from which to recruit and serve students; 

• Assess students using the STAR; 
Summer Advantage Final Report 

18

• Build strong fidelity to the SA program among site managers, teachers, and program staff; 

• Teach academic content and provide enrichment experiences to participating students; 

• Use an appropriate curriculum aligned to district objectives; 

• Maintain appropriate instructor/student ratios in Summer Advantage classrooms; and 

• Maintain clearly communicated program expectations with parents. 

This list of achievements reflects the program’s significant success across multiple summers in building 

connections with parents as well as education and community leaders in the Roaring Fork Valley. It also 

reflects the program’s continued ability to effectively communicate expectations and implementation 

goals to staff at all levels, and to instill a high level of buy-in from these staff to maintain program 

fidelity. The “Performance on Program Objectives” subsection, below, details implementation analysis 

findings in regards to each of the objectives (A through I) listed in Table 2, above. 

Performance on Program Objectives 

Objective A. Recruit Students 

Table 3: Objective A was Fully Met 

Measure Partially Met Fully Met Notes 

1. Recruit 500 students 

in one district. 

Accomplished 3 out of 4 

years 

2. Recruit at least 51% 

high-need students. 

Achieved this objective 

every year 

Source: APA analysis, 2013-2016

The two measures for this objective were for SA to (1) recruit 500 students in one district and (2) recruit 

high-need students. As Table 4, below, illustrates, SA surpassed its student recruitment goal in three of 

the four years. (In 2013, SA had a goal of recruiting 400 students). SA fell 45 students short of its goal in 

2016. 

Table 4: Summer Advantage Continues to Meet Enrollment Goals 

School Grade 2013 

Enrollment 

2014 

Enrollment

2015 

Enrollment

2016 

Enrollment 

Basalt K 76 42 41 43

1 55 59 56 37

2 57 47 57 37

3 39 55 43 42

4 - - - - 

Glenwood/Sopris K 85 85 70 58

1 113 75 84 63

2 61 78 56 70

3 70 61 69 65

4 56 45 36 50

Total 415 547 512 465 

Source: SA 2013-2016
Summer Advantage Final Report 

19

When looking at this table, it is important to note that not all students actually started or completed the 

program. In each year, several students were “dismissed” from the program for various reasons. Such 

reasons might include lack of initial participation (i.e. enrolling but never showing up), infrequent or 

erratic attendance in the program, or other circumstances signifying a student’s inability to fulfill the 

expectations of the program. The SA program does expect some attrition each year. 

The figures in Table 5, below, show the numbers of students in 2013 to 2016 who enrolled in and 

completed SA.

Table 5: Completion and Attendance Rates Remain High 

Year 2013 2014 2015 2016 

Basalt Average Attendance 97% 96% 93% 92% 

Sopris/Glenwood Average Attendance 93% 93% 91% 86% 

Total Average Attendance 95% 94% 92% 88% 

Source: APA analysis, 2013-2016

Table 5 reflects the fact that SA undertook significant effort to enroll students for the summer 2016 

program. Teachers and school principals at Basalt, Sopris, and Glenwood elementary schools made 

strong recruitment efforts. Teachers donated time to develop lists of students whom they identified as 

having the most academic needs. In many cases, teachers sent letters home to inform the parents about 

SA as a summer opportunity. SA staff also held meetings with parents to inform them about SA. For 

children designated as high-need who were not yet enrolled in SA, school staff made phone calls to 

families and sent letters home, using Spanish translators if necessary to communicate with parents. In 

2016, SA did not meet the first measure of Objective A (Recruit Students), which is to recruit 500 

students. These recruitment tactics were similar to ones used in previous years. 

The second measure of Objective A (Recruit Students) is the recruitment of high-need students. SA 

targets “at-risk” students, where a student’s eligibility for free and reduced-priced lunch (FRL) is used as 

a proxy for at-risk status. SA aims to recruit at least 51 percent at-risk students. SA targets these 

students in an effort to help close the achievement gap that exists between students from more and less 

socioeconomically privileged backgrounds. 

Based on enrollment data, the program operated every year in accordance with national SA

expectations with regard to identification and recruitment of high-need students to the program. Using 

district-provided information, APA found that roughly 84 percent of students who attended SA in 2016 

were FRL-eligible. Table 6, below, shows SA enrollment data by student demographic groups, including 

test scores, gender, racial and ethnic group, special population, FRL eligibility, and elementary school 

attended during the regular school year. 
Summer Advantage Final Report 

20

Table 6: Demographic Characteristics of Summer 2016 SA Students, by Completion Status 

Completed 

Treatment 

Started Treatment But 

Did not Complete

Fall DIBELS

Composite 176 115

NWEA Spring Math 178 176

Spring DIBELS

Composite 215 139

Male 49% 53%

Hispanic 85% 76%

White 14% 24%

Other Race 1% 0%

IEP 9% 12%

English 24% 29%

Spanish 76% 71%

Other Language 0% 0%

Fully English 

Proficient 11% 0%

Limited English 

Proficient 60% 67%

Not English 

Proficient 29% 33%

FRL 84% 71%

Basalt 39% 30% 

Sopris/Glenwood 61% 70% 

N 480 32

Source: RFSD, 2016 

Objective B: Recruit Qualified Staff 

Table 7: Objective B was Fully Met 

Measure Partially Met Fully Met Notes 

1. Recruit staff with a 

passion for working in 

high-need settings. 

SA staff recruitment 

activities seek out 

passionate, motivated 

individuals who often 

return year after year. 

2. Recruit academic 

teachers with 3+ years 

teaching experience, 

teacher licensure, and at 

least a BA. 

The SA hiring process is 

rigorous and competitive, 

resulting in high-quality 

teachers. Focus group 

data indicates academic 

teachers continue to 

meet experience and 

licensure requirements. 

3. Recruit enrichment 

teachers with 3+ years 

teaching experience and 

at least a BA. 

The SA hiring process is 

rigorous and competitive, 

resulting in high-quality 

teachers. 
Summer Advantage Final Report 

21

Measure Partially Met Fully Met Notes 

4. Achieve a near 100 

percent teacher 

attendance rate during 

the program. 

Though APA does not 

have program data on 

staff attendance rates, SA 

staff and program leaders 

in focus groups and 

interviews expressed high 

awareness of attendance 

expectations and very low 

rates of absenteeism. 

Source: APA analysis, 2013-2016

A review of program materials and interviews with program leaders indicates that SA continues to utilize 

a rigorous process to screen teacher applicants for the program. The organization’s national team, 

headquartered in Indiana, has significant involvement with the interviewing and screening of applicants 

and has a dedicated staff person to oversee recruitment activities. 

The program also has an innovative approach to help build continuity into the hiring process and to tap 

into the skills and experience of teachers with a history of success working for the program. The 

program accomplishes this by operating a team of paid program alumni who are brought together each 

spring to assist with the screening process. This team is made up primarily of former SA teachers who 

have been recommended by school or district partners to serve in this capacity. The national SA

program then trains these alumni on what to look for when reviewing prospective teacher candidate 

resumes and how to properly screen candidates. Once candidates pass through the screening process, 

they can enter an interview process, which is also overseen by national SA staff. As a result of these 

efforts there have been no reported issues with teacher recruitment over the past several years of the 

program’s operation. 

The measures for Objective B (recruit qualified staff) are that recruited staff should: (1) have a passion 

for working in high-needs settings; (2) have at least a BA, have at least three years of teaching 

experience, and hold teacher licensure (academic teachers); (3) have at least a BA and at least three 

years teaching experience (enrichment teachers); and (4) achieve a near 100 percent attendance rate 

during the program. Teacher Assistants (TAs) are also expected to have some level of experience and 

engagement with education. For example, TAs may be undergraduate students pursuing education 

majors, or they may have past experience working as tutors or in summer programs. SA allows recent 

high school graduates who are attending college in the fall to potentially also serve as TAs. APA’s

interviews with SA teachers highlighted how important TAs are to the program, especially in terms of 

providing classroom support. Many program teachers indicated that TAs play a critical role in providing 

consistency for students, managing student behavior, preparing classroom materials, and supporting 

day-to-day activities. 
Summer Advantage Final Report 

22

Since initial SA program implementation, staff recruitment has been facilitated by the fact that word 

about the program has spread to more teachers and staff in the district as well as to potential TA staff. 

When the program initially launched in 2012, it faced several recruitment hurdles, including a short time 

frame for recruitment activities, difficulty recruiting teachers and TAs to work in a mountain setting with 

few affordable housing options for staff, and a need to attract some of the district’s most qualified 

teachers to work during the summer, when many educators view summer as a time to recharge and be 

away from the classroom. 

After several years of program implementation in RFSD, SA has developed a positive reputation that 

enhances its ability to recruit teachers and TAs. The program is viewed by teachers as worthwhile and 

desirable to work for, and has proved effective in attracting its teachers to return in successive 

summers. Since SA calls for evaluations of all teachers each summer, recruiters can look at teacher 

evaluation records from previous summers to help select the highest-performing returning teacher 

candidates. 

SA continues to offer SA teachers and TAs levels of pay that are generally viewed as fair and attractive. 

In addition, the program provides subsidized housing opportunities for SA staff who come from outside 

the Roaring Fork Valley. Non-local staff can take advantage of subsidized housing in Colorado Mountain 

College dormitories, or they can stay at a local hotel at a reduced price. These financial and housing 

incentives are viewed as important to attracting and retaining quality staff. 

Teachers in APA’s focus groups consistently expressed a strong passion for working with students over 

the summer to improve academic skills, especially high need students whose other summer learning 

opportunities are extremely limited. At both Sopris and Basalt, teachers consistently mentioned working 

outside of their contract hours in order to carry out program expectations, such as to have conferences 

with parents. Teachers across both schools strongly supported the goals and efforts of the program and 

were willing to work out of contract hours to support SA. Additionally, administrators across program 

years indicated that they were typically starting work on SA several months before the beginning of the 

program in order for the program to have a successful start. One administrator suggested they had put 

in 30 out-of-contract hours prior to the start of the program. Having such strong buy-in from staff is 

impressive, and it demonstrates the program’s success in recruiting staff that are passionate about their 

work. It also has contributed over the past several program years to more consistent program 

implementation across school sites. 

Focus group data also indicate academic and enrichment teachers were licensed teachers with multiple 

years of experience in the teaching profession. Many teachers also taught at RFSD during the schoolyear, which gave them familiarity with students and with district expectations. Several TAs said that 

working at SA gave them the opportunity to see if teaching is a career they wanted to pursue as a career 

full time. 
Summer Advantage Final Report 

23

While many teachers reported that they enjoy the experience of working for SA and that they would like 

to return to SA in future years, in previous years’ teachers also reported a feeling of fatigue. These 

teachers explained that, after several years of teaching both during the school year and during the 

summer (for SA), they may want to take summer breaks. However, in 2016, no teachers reported a 

feeling of fatigue. In previous years, teachers and teaching assistants have suggested several ways in 

which the program can help continue to make itself an attractive employment prospect, including: 1) 

provide more opportunity for teacher input into which schools they are assigned to accommodate 

commuting and other concerns; 2) provide teacher assistants and teachers more input into the grade 

level where they are assigned to teach during the program; and 3) consider providing added pay for 

returning staff to recognize the value of their prior experience in the program. 

In both 2015 and 2016, teachers interviewed expressed a desire for additional clarifications during the 

hiring process. In particular, some dual (enrichment and academic) teachers had misconceptions about 

which cohorts of students they would be working with. In some cases, in 2016, some TA’s also indicated 

in focus groups that they were not made aware of their classroom assignment until immediately prior to 

the start of the program, and that more advance notification would be desirable. These issues initially 

affected staff ability to prepare lesson plans and materials. And while the interview process was 

generally reported as being clear, some teachers indicated a lack of clarity around job expectations, 

especially around scheduling requirements for parent-teacher conferences. 

With regard to attendance (a measure of Objective B: Recruit Quality Staff), teacher focus groups agreed 

that the program clearly conveyed expectations. Teachers were uniformly aware of the program’s strict 

attendance policy (for both teachers and students) and agreed that these expectations were critical to 

the success of this type of program, which operates over a very compressed time frame. In general, 

interview and focus group participants were impressed with the organization, scope, and goals of the SA 

program, and had a high level of overall buy-in to the program’s expectations and requirements, 

including consistent teacher attendance. 

Objective C: Train Teachers 

Table 8: Objective C Was Partially Met 

Measure Partially Met Fully Met Notes 

1. Teachers complete online modules. All teachers completed online 

modules. 

2. Teachers participate in a mandatory weeklong professional training session prior to 

program start. 

All teachers participated in 

mandatory training. 

3. Teachers view training sessions as useful and 

high-quality. 

Some teachers expressed 

frustration with training 

redundancy for returning 

staff. 

4. Teachers are evaluated at least twice during 

the program. 

Teacher evaluations were 

conducted. 

Source: APA analysis, 2013-2016
Summer Advantage Final Report 

24

Teachers overall across the years of APA’s implementation monitoring expressed that they received 

adequate training to meet program expectations. In several years, some TAs and teachers indicated that 

more training for TAs on classroom and student behavior management techniques would strengthen 

TAs’ contributions. Also in several years those staff who returned to SA after teaching in the program in 

prior summers, indicated that having to repeat the same online training modules was repetitive, 

however these returning staff also indicated that they appreciated having all of the SA staff together on 

one site for training. 

The measures for Objective C (Train Teachers) are that: (1) teachers complete online training modules, 

(2) teachers participate in a mandatory week-long professional training session prior to the program 

start, (3) teachers view training sessions as useful and high-quality, and (4) teachers are evaluated at 

least twice during the program. 

To achieve the measure that teachers complete online training modules, all SA teachers are required to 

complete an online course that includes a series of modules. Teachers must also pass online tests 

related to each module prior to starting their assignments. Some returning staff expressed that the 

modules could be “repetitive.” However, new staff frequently expressed that the modules were a 

valuable means of providing familiarization with the program. Returning teachers were very 

appreciative of the program’s efforts to reduce the redundancy of the online models by offering 

“refreshers” instead of full modules and by allowing people to “test out of” certain modules. 

The second and third measures of Objective C (train teachers) are that: (2) teachers participate in a 

mandatory week-long professional training prior to the program start; and that (3) teachers view 

training sessions as useful and high-quality. Across program years all, or nearly all teachers participated 

in training. Some expressed a desire to differentiate training for returning teachers versus new teachers. 

They also expressed a desire for more training on program logistics (e.g. managing parent teacher 

conferences and using reading logs) during the training week. New teachers in 2016 indicated that, 

while they received general curriculum training, they could have used more training on day-to-day 

program operations. 

In 2016, the character development program “Building Dreams” was replaced by a new program called 

“Crew”. Crew was generally well received by the teachers. Crew was an extension of a program run 

during the school year in RFSD, and district teachers working in SA were therefore familiar with the 

program. However, if a teacher did not teach in RFSD during the school year, they reported receiving no 

training on how Crew functioned. Regardless, teachers appreciated the flexibility the program offered 

them compared to Building Dreams. 

With regard to the fourth measure of Objective C, that teachers are evaluated at least twice during the 

program, teachers across program years generally spoke positively about the evaluation process. 
Summer Advantage Final Report 

25

Teachers were concerned about evaluation observations occurring too quickly, before routines could be 

established, and/or at awkward times. Additionally, teachers were worried that evaluations could be 

rushed because program leaders were sometimes working under tight deadlines. Teachers did, 

however, uniformly appreciate the collective feedback that was periodically given out to the whole staff. 

This feedback, they noted, offered valuable reminders of program goals and expectations. They also 

perceived the process to be appropriately transparent. 

During site visits and interviews across program years, APA evaluators observed that program leaders, 

teachers, and site program managers at each school had clear and consistent understanding of the SA 

program’s expectations. Site program managers were trained in classroom observation and in using a 

common observation rubric. All teachers understood that they were to be evaluated at least twice over 

the course of the five-week program, that such evaluations were conducted with fidelity across program 

years, and that teachers received feedback based on these evaluations. Teachers also understood that 

evaluation data could be used in hiring decisions in future years. 

Objective D: Build Program-District Relationships

Table 9: Objective D Was Fully Met 

Measure Partially Met Fully Met Notes 

1. Achieve buy-in and support for 

the program from district 

leaders. 

Summit 54 has cultivated buy-in and 

support from RFSD, which continues 

to provide space to operate, 

transportation, and other 

maintenance and logistical supports. 

2. Achieve buy-in and support for 

the program from school 

principals. 

School leaders had positive 

impressions of SA and provided 

important resources to SA. 

Source: APA analysis, 2013-2016

Summit 54 co-founder Terri Caine, along with Summit 54 staff, have cultivated collaboration between 

RFSD and SA. Over the course of the past several years, RFSD and SA renewed a comprehensive MOU to 

help guide their collaboration. SA continues to make strong efforts to maintain a close rapport with 

RFSD leaders. RFSD continues to support SA by granting access to facilities, custodial services, food, and 

transportation. RFSD also provides financial support for the Crystal River Elementary School site 

(necessary since the Crystal River site does not receive MHUW SIF funding.) The relationship between SA 

and RFSD has also encouraged RFSD leaders to take active interest in supporting APA’s evaluation. RFSD

has been responsive to data collection needs, providing APA with timely, evaluation-related data 

including student assessment data. RFSD also kept APA and SA apprised of potential policy changes 

including changes in assessments used by the district that could affect evaluation activities. 

In conversations with APA, school and district leaders credited SA with filling a critical gap in RFSD’s 

education services for children – a gap created several years ago when budget cuts forced RFSD to 
Summer Advantage Final Report 

26

eliminate district summer school programming. The gap in summer education services means that 

children with working parents are often left unsupervised over the summer. With SA, students can get 

structured summer programming, and school leaders continue to indicate over the course of APA’s 

implementation monitoring that SA students tend to have fewer behavior problems at the start of the 

school year and need to spend less time re-learning school expectations. 

School principals continue to be an invaluable resource for SA, providing services to the program such as 

bilingual translation services to help recruit students and families during the regular school year. 

Objective E: Assess Students Pre- and Post-Program 

Table 10: Objective E Was Fully Met 

Measure Partially Met Fully Met Notes 

1. Assess students at 

the beginning and end 

of the program using 

STAR. 

SA began using STAR in 

summer 2014. 

Source: APA analysis, 2013-2016

In 2014, SA switched from the ITBS assessment to the STAR assessment for internal measures of student 

academic growth. Teachers and program staff have indicated support for the use of the STAR 

assessment and the data it produces. Principals expressed that STAR better aligned to the RFSD school 

year curriculum. Teachers indicated the STAR assessment data is useful in helping differentiate 

instruction and that the system produces extremely timely reports that provide immediate feedback to 

educators. 

Objective F: Provide Students with Appropriate Daily Academic Instruction and Enrichment 

Table 11: Objective F Was Fully Met 

Measure Partially Met Fully Met Notes 

1. Students miss no 

more than 2-3 days over 

the program. 

Attendance expectations 

were clear to students 

and families. 

2. Program staff call 

home for absent 

students. 

Staff called home for 

absences, usually on the 

same day as the 

absence. 

3. Program releases 

students who do not 

meet attendance 

expectations. 

Students who failed to 

show up for more than 

2-3 days were released 

from SA. 

4. Students participate 

actively in class. 

Curriculum components 

encourage student 

participaton and 

engagement. 

5. Students participate 

in field trips and 

enrichment activities. 

Students had weekly 

opportunities for field 

trips and enrichment. 

Source: APA analysis, 2013-2016
Summer Advantage Final Report 

27

SA strives to provide students with rigorous academic learning and enrichment experiences throughout 

the summer. The first three measures of Objective F are that: (1) students miss no more than two to 

three days over the course of SA; (2) SA staff call home for absent students; and (3) SA releases students 

who do not meet attendance expectations. SA has clear expectations about participation. Students are 

not permitted to miss more than two to three days throughout the five-week program, although 

exceptions are made for illnesses or extraordinary circumstances. This translates into an expectation of 

around 88 to 92 percent overall attendance. If a student misses more than two to three days without an 

extenuating circumstance, then SA managers may release that student from SA. 

Across the two summer SA sites involved in APA’s evaluation, the overall student attendance rate was 

92 percent. This meets the program’s expectations.6

 APA believes part of this success can be attributed 

to program staff actively following up with families in instances of student tardiness or absence, usually 

on the same day of the absence. 

The final two measures of Objective F are that: (4) students participate actively in class; and (5) students 

participate in field trips and enrichment activities. The program has selected curricular materials that 

include student participation as an important classroom component. Across program years, teachers in 

APA’s focus groups also consistently indicated that student participation is a priority for their 

instruction. APA classroom observations during site visits, while not a representative sample of all the 

classes taught, suggest that student engagement and participation in class is indeed a high priority for 

teachers in the program. Teachers also expressed satisfaction with the flexibility and variety of field trips 

in 2016. Administrative staff had made it a priority to plan out Friday field trips in advance in 2016. 

Teachers noticed this change and commented that field trips felt more organized than in previous years. 

 

6 This number excludes those students who were “released” according to SA protocol. 
Summer Advantage Final Report 

28

Objective G. Ensure that Teachers and Staff Use Culturally Competent Curriculum 

Table 12: Objective G was Partially Met 

Measure Partially Met Fully Met Notes and concerns 

1. Curriculum is consistent with 

district expectations. 

Houghton Mifflin summer school curriculum is 

valued and aligns with RFSD expectations. 

However, some teachers expressed concerns 

about alignment with Common Core. 

2. Curriculum is consistent with 

student cultural needs. 

With regard to the academic curriculum, overall 

alignment with student cultural needs is viewed 

as satisfactory. Some staff expressed concern 

that the phonics curriculum could be more 

alinged with Common Core. 

With regard to the Crew program instituted in 

2016, staff were happy overall that it replaced 

Building Dreams, which was implemented in 

2015. However, non-RFSD teachers felt the 

Crew program was unoragnized. This may have 

been at least partially due to the fact that they 

had less familiarity with it. 

Source: APA analysis, 2013-2016

SA continues to administer the Houghton Mifflin curriculum. RFSD leaders and school principals 

indicated that this curriculum is seen as valuable and appropriate to the student population. It is the 

same curriculum RFSD used in its own summer school programs before they were eliminated. Teachers 

have expressed that the curriculum is generally effective, though in past years they were concerned 

about its alignment with Common Core standards. Teachers in past years also expressed concern with 

phonics components of the curriculum, which they suggested were not as effective for bilingual 

students and were outdated. 

In 2016, several curriculum issues that were identified in APA’s prior year implementation monitoring 

appear to have been fixed. For instance, in one prior summer teachers reported that there were not 

enough material kits for all students, but in 2016, SA purchased new material kits and most students 

were able to have their own materials. One teacher did report being several books short in their 

classroom. 

One issue that emerged in 2016 were challenges involving use of technology to support the curriculum. 

Some teachers in APA’s focus groups stated that they were forced to handle technological issues 

themselves, particularly involving usage of projectors, speakers and computers to implement the 

curriculum. In some cases, teachers were forced to bring their own devices. However, SA teachers also 

reported that SA was very responsive in addressing curriculum issues from previous years. They 

therefore expressed confidence that these new curriculum issues would be resolved by the following 

summer. 
Summer Advantage Final Report 

29

The summer of 2016 featured the implementation of the Crew program, a new character development 

program for students that was chosen to replace the Building Dreams character development program 

that was implemented in 2015. Building Dreams experienced several implementation challenges that 

APA identified in its 2015 implementation monitoring efforts, and as a result the program moved 

towards selection and implementation of a new program. Although some teachers in APA’s 2016 focus 

groups still had concerns about the new Crew program, it was viewed far more favorably than Building 

Dreams. The Crew curriculum builds off of a character development program already in place during the 

school year at RFSD. One of the major differences between Crew and Building Dreams is that Building 

Dreams required that the program be implemented during breakfast, which teachers in 2015 reported 

as creating significant implementation challenges that sometimes interfered with children having 

sufficient time to eat. Under the Crew program in 2016, teachers report that students are able to eat 

without any program taking place at the same time. Several teachers described Crew as “one of the 

better parts of the day.” Other teachers felt Crew lacked direction and described it as “wasted time.” 

Overall however, returning teachers who experienced both programs indicated that Crew was clearly an 

upgrade from Building Dreams. 

Objective H: Ensure Maintenance of Faculty-to-Student Ratios 

Table 13: Objective H Was Fully Met 

Measure Partially Met Fully Met Notes 

1. Minimum of 1 teacher 

and 1 TA per 24 students, 

with a goal of 19 

student/class. 

SA consistently aims to 

maintain ratios at or 

below 2:24. Site visit spot 

check observations 

indicated appropriate 

class sizes. 

Source: APA analysis, 2013-2016

In general, over the four years of APA’s implementation evaluation and monitoring, SA consistently met 

its program goal to maintain classroom ratios at or below 2:24 teachers to students and to provide a 

minimum of one teacher and one TA per classroom. However, in summer 2016, several classes opened 

with 26 students. APA inquired as to the reasons for this challenge, and program site leaders indicated 

that they did not have control over the student enrollment distribution across grades, and that creating 

smaller class sizes was challenging when some grades turned out to have higher enrollment levels. 

Other staffing concerns included the desire to have a special education assistant/interventionist to 

support student needs, the potential need for additional technological staffing assistance, and the need 

for added nursing support. At Basalt, the nurse during the academic year was volunteering her time to 

help out during the SA program, however, SA site leaders did have to fill this role at times when the 

nurse was unavailable. 
Summer Advantage Final Report 

30

Objective I: Parents Participate in Program 

Table 14: Objective I Was Fully Met 

Measure Partially Met Fully Met Notes 

1. Parents are required to attend a mandatory 

parent orientation meeting prior to their 

children being accepted into SA. 

Parents whose children 

participated in SA attended the 

orientation. 

2. Parents sign contracts with SA. Parents signed contracts during 

“parent night” at the start of the 

program. 

3. Parents attend mid-summer parent-teacher 

conferences. 

Parents attended conferences, 

though feedback received 

indicated that the timing of the 

conferences was not always 

optimal. 

4. Parents participate in and sign reading logs. Parents consistently read with 

their students, as evidenced by 

their signing student reading logs. 

5. Parents are encouraged to attend end-ofsession graduation with lunch. APA does not have specific data on 

parent attendance levels at the 

end-of-session graduation, but 

program leaders indicated that 

parents were strongly encouraged 

to attend. 

Source: APA analysis, 2013-2016

SA expects parents to play an active role in supporting their children during the summer by ensuring as 

close to 100 percent attendance as possible, by attending parent-teacher conferences during the SA 

program, by reading to their children over the course of the summer, and by signing reading logs over 

the course of the summer so the SA program has assurance that reading is taking place at home. Parents 

sign a contract prior to the start of the SA program each summer, expressing their commitment to 

making sure their children are present and prepared for the program. Parents must attend at least two 

meetings (one parent night where the contract is signed, and one parent-teacher conference during the 

course of the program). 

The program’s adherence and enforcement of these requirements was strong and consistent over all 

four years of APA’s implementation evaluation and monitoring. APA observed through SA teacher and 

staff feedback that, at least in part due to this consistently strong implementation, the program has 

been able to build a “network” amongst parents, students and teachers around the importance of 

keeping students academically engaged over the summer, particularly involving reading. SA has allowed 

for a new line of communication to be formed. For instance, one teacher at Sopris indicated that SA has 

allowed her to develop relationships with students before they start in her class the following fall. 

Furthermore, numerous teachers and school principals across multiple years of APA’s evaluation 

indicated that SA had a noticeable, positive effect on how ready students are to learn at the start of the 

next school year. These teachers and principals express that students who do not attend SA are more 

likely to require a period of behavior adjustment once school resumes in the fall, which is a drain on 

instructional time and on teachers’ ability to get classes off to a fast start at the beginning of each school 
Summer Advantage Final Report 

31

year. Students attending SA in the summer are much more likely to come to school ready to learn on 

day one of the school year, according to multiple teachers and principals that APA interviewed over the 

course of the past four years. 

Network building was also seen through enhanced parent engagement for those parents whose children 

attend SA. SA teachers who teach in RFSD during the regular school year have noted that parent 

participation and engagement is enhanced for those parents whose children participate in the program. 

Teachers and principals also indicated that parents of SA students appear to be more comfortable 

joining their child during lunch or recess and that this increased participation of parents also affords 

teachers more opportunities to have informal conversations with the parents that can help keep them 

apprised of their child’s progress in school over time. Several teachers also noted the fact that many 

parents would “hang out” when picking up their kids, and that this provided them an opportunity to 

engage with each other. Teachers indicated this is added engagement across parents enhances parents’

ability to network amongst themselves, which helps strengthen and broaden the overall sense of 

community at the school both during the SA program and the following school year. 

Since families are offered scholarships to participate in SA, SA makes it clear that a strong parent 

commitment is expected or the scholarship slot may be given to another student. Parents are expected 

to read with their child every night for at least 30 minutes and sign reading logs (turned in every 

Monday). Parents are also expected to help with homework over the course of the program. Teachers, 

principals, and program managers over the course of APA’s four-year implementation evaluation and 

monitoring efforts expressed that, as result of the program’s clearly communicated expectations, 

parents understood what was required of them and their children. Teachers told APA that parents were 

typically active and responsive and made sure that their child’s homework was complete. Parents also 

consistently read with their students, as evidenced by their signing student reading logs consistently 

over the multiple years of APA’s study. 

Impact Analysis 

The final impact results presented in this study come an analysis of pooled data from the 2014, 2015, 

and 2016 SA program years. The analysis plan for this evaluation compares students who participated in 

SA to similar students from RFSD who did not participate in SA. The group of comparison students is 

constructed using propensity score matching (PSM), a statistical technique which uses the demographic 

characteristics and pre-test scores of participants to match them to non-SA participants with similar 

demographics and pre-test scores. Comparing these groups allows APA researchers to estimate the 

effect of the SA group by comparing the differences in outcomes between SA participants and similar 

non-SA participants. 

The analysis using the two matched groups compares the changes in student test scores between the 

spring assessment, which occurs before students attend Summer Advantage, and the fall assessment 
Summer Advantage Final Report 

32

after they have completed the program. To look at score changes, APA examined student scores on the 

DIBELS Next reading assessment, which measures literacy skills. APA used a multi-level analysis 

technique called hierarchical linear modeling (HLM) to control for the school-level characteristics of the 

different schools attended by students in the treatment and comparison group. 

Background 

All of the research questions in the impact analysis compare SA students to similar non-SA students 

identified using PSM techniques. The same group of comparison students was used for the analysis of 

each of the research questions listed below. APA identified the comparison group using a PSM that 

employed demographic variables and the students’ DIBELS Next pre-test scores from the spring 

immediately prior to the SA program. Each comparison group student is uniquely matched to a SA 

participant student with similar demographic characteristics and pre-test scores. Because the resulting 

treatment and comparison groups are similar, any differences in outcomes between the two groups are 

likely to be attributable to SA participation, rather than to other differences between the groups of 

students. APA used the same PSM technique for the previous impact analyses. 

After generating the comparison group, APA ran hierarchical linear modeling (HLM) regressions 

comparing outcomes between comparison group students and SA participant students. HLM is used 

when the outcome is measured at the individual level, but the treatment (i.e. SA program) is provided to 

groups of students. 

Sampling Eligibility 

For all research questions, students were included in the analysis sample only if they took the DIBELS 

Next exam in the spring prior to their participation (or non-participation) in SA and also in the fall 

immediately following their participation (or non-participation) in SA. Furthermore, students were 

required to participate in SA with a dosage level of at least 80 percent to be included in the treatment 

group. This means that students who did not complete the program or completed less than 80 percent 

of the program were not included in the impact study.7 There was a total of 925 eligible students (236 in 

2016, 398 in 2015, and 291 in 2014). Student included in the sample were pooled over three years. 

Constructing Comparison Groups 

APA used PSM techniques to create comparison groups of non-participating students for each year of 

the analysis. In PSM, a propensity score is generated for each student – both SA participants and nonparticipants – describing the student’s probability of attending SA. In order to identify the comparison 

group using PSM, APA gathered demographic and test score information for all students in RFSD who 

were in kindergarten through second grade prior to their participation (or non-participation) in SA. As in 

previous years, propensity scores were computed looking at a student’s gender, primary home 

 

7

 This means that the analysis uses a treatment on the treated framework, rather than an intent to treat 

framework. Because SA does not collect outcome data on students who have less than 80 percent 

participation and exits them from the program, an intent to treat analysis is not possible. 
Summer Advantage Final Report 

33

language, race, special education status, free or reduced lunch status, and spring DIBELS score. After 

computing a propensity score for each student, APA then matched students based on their propensity 

scores using a 1-to-1 nearest-neighbor matching estimator, without replacement. In this method, each 

treatment student is matched with the comparison student with the most similar propensity score, 

based on their background characteristics. The matched comparison student is then removed from the 

pool of potential matches, so that every treatment student is matched with a unique comparison 

student. This leads to equal sized treatment and comparison groups. Note that this PSM cannot account 

for student characteristics that are not captured by available data, including unobserved factors such as 

personal motivation or parental engagement. 

One of the primary variables used to match students in the PSM was students’ composite score on the 

DIBELS Next assessment in the spring before their summer of SA participation (or non-participation). 

Because the DIBELS Next composite score includes different subcomponents depending on the grade 

level of the student being tested, the composite scores for students in different grades are on different 

scales and are not directly comparable. To ensure comparability of DIBELS Next scores across grade 

levels, APA converted the raw composite scores into z-scores, standardizing within grade level. Because 

of this standardization, a kindergarten student who scored in the 90th percentile of her peers would 

have a similar z-score to a second-grade student scoring in the 90th percentile of her peers, even though 

the students’ raw composite scores would be quite different. Both the match and the analysis used this 

standardized measure of a student’s spring literacy skills. The conversion of DIBELS Next scores was not 

necessary for the match, as each student in the treatment group was always matched with a comparison 

group in the same grade level. The conversation was used to compare literacy gain by grades. 

After completing the match, APA tested the quality of the PSM by calculating the standardized mean 

difference of each demographic variable for the treatment and comparison groups. This standardized 

mean comparison is a common tool for measuring differences between groups.8 When reviewing the 

quality of a match, the absolute standardized mean difference should be no larger than 0.25 and 

preferably less than 0.1. Table 16, below, details the characteristics of the 2016 treatment and 

comparison groups generated by the PSM, showing that the two groups are very similar in terms of raw 

demographics and the standardized mean differences between the groups fall below the recommended 

thresholds. (Details of the match balance for previous years of data can be found in preliminary reports 

for those years of the program.) 

Table 16. The post-match groups of 2016 Summer Advantage and comparison students are 

comparable. 

Summer Advantage 

Students 

Comparison 

Students 

Standardized Mean 

Difference 

 

8

 This statistic is calculated by determining the raw difference between the treatment and control groups on a 

certain variable, then dividing the result by the standard deviation of the pooled group on that same variable. For 

example, a standardized mean difference of 0.05 would indicate that SA students’ mean for that variable is 0.05 

standard deviations higher than the mean of the matched comparison students. 
Summer Advantage Final Report 

34

Summer Advantage 

Students 

Comparison 

Students 

Standardized Mean 

Difference 

Fall DIBELS 

Composite 166.78 165.39 0.00 

NWEA Spring 

Math 181.48 188.03 -0.04 

Spring DIBELS 

Composite 202.44 203.75 -0.07 

Male 0.49 0.49 0.04 

Hispanic 0.86 0.86 0.19 

White 0.12 0.13 -0.21 

Other Race 0.01 0.02 0.05 

IEP 0.08 0.08 0.04 

English 0.20 0.18 -0.14 

Spanish 0.79 0.79 0.13 

Other 

Language 0.00 0.01 0.07 

Fully English 

Proficient 0.28 0.29 -0.05 

Limited 

English 

Proficient 

0.46 0.50 0.10 

Not English 

Proficient 0.72 0.70 -0.08 

Statistical Analysis 

After completing the propensity score match and ensuring that the treatment and comparison groups 

were comparable, APA developed a statistical model to analyze each of the research questions, listed 

below in Table 17. For each research question, the statistical analysis included a set of control variables 

at the student level and another set of control variables at the school level. These controls are intended 

to isolate the effect of SA on student outcomes from the effects of other variables, including 

demographics and past student performance. The student-level control variables are: 

• Race; 

• Gender; 

• Home language; 

• IEP status; 

• Socio-economic status (FRL status); 

• School of attendance; 

• Spring DIBELS Next scores (pre-test reading achievement); and 

• Spring MAPS math score (pre-test math achievement). 
Summer Advantage Final Report 

35

The following sections include the coefficients on the SA participation variables and the statistically 

significant coefficients on key variables of interest. These coefficients describe the relationship between 

the variable and the outcome assessment scores. 

Research Questions and Activities 

The impact analysis is intended to explore whether SA has a positive impact on student learning. All 

years of this evaluation have focused on the same research questions, reviewed in Table 17. 

Table 17: Confirmatory and Exploratory Research Questions 

Research Question Question Level Type of Question 

C1: Does participation in Summer Advantage have an impact on 

student literacy for students who participated compared to a district 

sample as measured by standardized assessments? 

Student, 

Program Impact: Confirmatory 

E1: Does the number of years of participation in Summer Advantage 

moderate impacts on student literacy, such that students with 

previous participation experience greater impact than students with 

no previous participation? 

Program Impact: Exploratory 

E2: Does Summer Advantage participation impact Kindergarten 

student performance on standardized literacy assessments 

compared to non-program participants? 

Student, 

Program Impact: Exploratory 

Research Findings 

Questions 1 and 3: Does participation in SA have an impact on student literacy for 

students who participated compared to a district sample, as measured by standardized 

assessments? 

This research question compares on students’ standardized DIBELS composite score from the fall 

following SA participation (or non-participation), adjusting for the variables listed above. APA conducted 

separate analyses by grade level to answer this question, both in order to answer the research question 

about kindergarten performance and because previous years’ analyses had found significant differences 

in program effect by grade level. In order to isolate the effect of participating in one year of SA, this 

analysis excluded students participating in their second or third summer. This means that if a student 

participated in SA in 2014, 2015, and 2016, only their 2014 scores were included in this analysis. 

Does SA improve student performances on standardized literacy assessments? 

APA finds that participation in only one year of SA provides a statistically significant and positive 

effect on student reading performance for kindergarten and first grade students. For 

kindergarten students, students who participated in Summer Advantage scored 15 percentage 

points higher on fall assessments than similar students who did not attend Summer Advantage. 

First grade students who attended Summer Advantage scored 6 percentage points higher on fall 

assessments than similar students who did not attend Summer Advantage.
Summer Advantage Final Report 

36

When analyzing all three years of pooled data, APA found a statistically significant difference between 

the fall DIBELS scores for students who did and did not participate in SA in kindergarten and first grade. 

Statistical significance means that APA is confident that the mean difference between DIBELS scores for 

SA participants and non-participants is not due to random chance, but is a real difference. Although 

participation in the SA program had a significant effect for kindergarten and first grade students, 

findings for the effects of second and third grade participation on literacy rates were inconclusive. 

On average, kindergarten students who participated in SA scored 7.14 points higher on the fall DIBELS 

assessment than comparable, non-SA students, controlling for other demographic characteristics and 

students’ DIBELS pre-test scores. This is equivalent to an effect size of 0.195, meaning that for 

kindergarteners, participation in SA is associated with scoring 0.195 standard deviations higher on the 

fall DIBELS assessment. Similarly, first grade students who participated in SA scored, on average, 11.47 

points higher on the fall literacy assessment, again controlling for other demographic characteristics and 

students’ DIBELS pre-test scores. For first graders, this is equivalent to an effect size of 0.142, meaning 

that first graders who participate in SA score 0.142 standard deviations higher on the fall DIBELS 

assessment. 

The kindergarten score increase is equivalent to an effect size of 0.195 and the first grade test score 

increase is equivalent to an effect size of 0.142. These effect sizes are equivalent to the effect sizes seen 

by other summer programs in the literature review cited above. 

Kindergarten students who participated in SA gained 11 percentile points over the course of the 

summer. Kindergarten students who participated in the program were in the 44th percentile in the 

spring and were in the 55th percentile in the fall. In comparison, kindergarten students who did not 

participate in the program were in the 49th percentile in the spring but fell to the 45th percentile in the 

fall, losing 4 percentile points. This means that in comparison to students who did not attend SA, 

kindergarten students who attended gained 15 percentile points. 

First grade students who participated in SA stayed stable in terms of academic percentiles over the 

course of the summer, starting and ending the summer in the 50th percentile. In comparison, first grade 

students who did not participate in the program started the summer in the 50th percentile but ended 

the summer in the 44th percentile, losing 6 percentile points. This means that in comparison students 

who did not attend SA, first grade students who attended gained 6 percentile points. 

Table 18: SA Participation is Associated with Higher DIBELS Scores in Kindergarten and First Grade 

 Kindergarten 1st Grade 2nd Grade 3rd Grade 

SA Treatment 7.14 

*** 

11.47

*** 0.27 -0.93 

* p < .1 **p < .05 *** p < .01 

Source: APA analysis 
Summer Advantage Final Report 

37

Previous years’ analyses also failed to find a significant effect for second graders. To investigate this, APA 

spoke with researchers evaluating similar academic programs focused on low-income students in this 

grade range for the SIF program. Those researchers shared with APA that although they had found 

significant program effects in a range of grade levels, they also had failed to find significant effects for 

second grade using the DIBELS assessment. This suggests that the inconclusive effects in second and 

third grade may be a function of the components of the DIBELS assessments for those grade levels, 

rather than an actual lack of program effect on students in those grades. Further research using a variety 

of student outcome measures, beyond the DIBELS Next, may pick up program effects that the DIBELS 

Next is currently failing to capture. That examination of other student outcomes would be important 

before concluding that the SA program does not have an impact on students in those grades. 

Question 2: Does the number of years of participation in SA moderate impacts on 

student literacy, such that students with previous participation experience greater impact 

than students with no previous participation? 

 

9

 Because the effect of participating in the program for the second or third year is equivalent to the 

effect of participating in the program for the first year, this means that students who participated in 

their second or third year of the program as rising 2nd or 3rd graders did not experience a score increase. 

Do students who have spent multiple summers in SA have greater literacy improvements than 

students who have only spent one summer in SA? 

For students who participate in multiple years of Summer Advantage, the benefit of the second 

or third year of participation is equivalent to the benefit from the first year of participation. In 

other words, participating in multiple years of Summer Advantage has an additive effect. 

Looking across data from the program in 2014, 2015, and 2016 summers, there were 205 students who 

attended two years of SA and 50 students who attended three years of SA. 

Using these 255 students, APA conducted an analysis to examine the impact on students from 

participating in a second or third year of SA. This analysis found that there is an additive impact on fall 

DIBELS scores from participating in a second or third year. This means that, on average, students 

participating in their second or third year of SA showed an increase in DIBELS scores equivalent to the 

increase in DIBELS scores for students participating in their first year of the program.9 This is important 

because it indicates that SA may be teaching students more than basic classroom behavior skills and is 

additionally imparting specific academic knowledge that can continue to benefit a student even if he or 

she has participated in the program in a previous summer. There is no evidence that participation in 

additional years of SA has a greater effect than the first year of participation. Nor is there evidence that 

participation in additional years has less effect than the first year of participation. 
Summer Advantage Final Report 

38

Conclusions 

APA’s evaluation over the past five years of the program finds that the SA program is filling a significant 

gap in the district by providing academic-based summer services that are otherwise almost non-existent. 

Teachers and district administrators consistently noted over the course of the study that there were no 

viable summer academic alternatives for students in the region apart from some athletic programs or 

short-term summer day-camps (almost all of which are fee-based). SA therefore is providing a unique

research-supported, subsidized academic and enrichment program which would otherwise not be 

available to over 30 percent of the district’s elementary-aged student population. 

The Roaring Fork School District and its surrounding community both continue to express strong support 

for the program, with the district providing infrastructure, maintenance, and other logistical support and 

community organizations assisting in hosting and supporting Friday field trip enrichment activities for 

students. While filling a vital need within the community, SA has also produced statistically significant 

positive results for students in reading. SA not only provides participating students with an enriching 

summer activity, but also with high quality instruction that produces quantifiable academic benefits. 

Implementation Analysis Conclusions 

With regard to fidelity of SA program implementation, APA conducted monitoring and evaluation efforts 

over four summers (2013, 2014, 2015, and 2016). This four-year analysis found that SA was overall able 

to achieve its implementation goals and operated with fidelity and consistency in the mountain rural 

setting of the Roaring Fork Valley. APA’s implementation monitoring included data reviews, interviews, 

focus groups, and observations of administrative staff, teachers, and program leaders. This research 

indicated that SA is continuing to operate with fidelity and with general consistency across years and 

across school sites. 

The program has been able to effectively serve and recruit students while garnering strong student, 

teacher, school leader, district, and parent buy-in to the program. High attendance standards were set 

and upheld among the students, teachers and staff throughout the course of the study. 

Over the five years of implementation analysis, SA was also able to effectively respond to a variety of 

concerns and issues raised each year, and the program proved to be nimble in addressing the needs of 

teachers and TAs. In fact, teachers, site leaders, principals, and district leaders report that the SA has 

created a program that is conducive to learning and that teachers and TAs enjoy being a part of. 

The program also met and clearly communicated rigorous goals to teachers and other staff members 

around training, evaluation, and expectations for family involvement. As a result, teachers demonstrate 

a high level of support for the program’s curriculum, structure, and goals, and are strongly invested in 

helping the program continue to grow, improve, and succeed over time. 
Summer Advantage Final Report

39

Impact Analysis Conclusions

APA’s impact study examined the relationship between SA participation and subsequent student 

performance on the DIBELS Next literacy assessment. APA found a statistically significant impact of SA 

on student reading performance for kindergarten and first grade students. 

Examined by grade level, APA found that participation of kindergarten students in only one year of SA 

provides a statistically significant and positive effect on kindergarten student reading. Although APA’s 

research question focused only on kindergarten students, APA also explored grade-level specific 

relationship for other students. APA found a statistically significant relationship between first grade and 

participation in SA. However, APA found statistically inconclusive results for participation during second 

grade or third grade. This does not mean that there is not a statically significant relationship for second 

and third grade participation in SA. However, APA is limited by only having five years of data. As SA 

continues, the program will gather more data and significant relationships will be easier to detect.

Additionally, APA examined if participation in multiple years of SA produced greater results. APA found 

no evidence that participation in additional years of Summer Advantage has a greater effect than the 

first year of participation. Nor is there evidence that participation in additional years has less effect than 

the first year of participation. However, the sample size for this specific research question was very 

small. Only 50 students attended SA for 3 years. This question should be reexamined once more 

students have completed multiple years of SA. 

The detection of statistically significant relationships between kindergarten and first grade student 

participation in SA is very promising for SA. APA suggest continuing to evaluate the program in future 

years in order to determine if this positive effect is also present for children in other grades. Regardless, 

the results of the impact study justify and support the existence of SA in RFSD. SA clearly provides a 

positive and education based summer alternative for students. 
Summer Advantage Final Report 

40

References 

Alexander, K.L., Enwisle, D.R., and Olson, L.S. (2001). Keep the faucet flowing: Summer learning and 

home environment. American Educator, Vol. 25, No. 3, p. 10 - 15. 

Alexander, K.L., Enwisle, D.R., and Olson, L.S. (2007). Lasting consequences of the summer learning gap. 

American Sociological Review, Vol. 72, p. 167 - 180. 

Annie E. Casey Foundation, Unequal Opportunities for School Readiness. 

http://www.aecf.org/upload/publicationfiles/fact_sheet2.pdf. 

Chaplin, D. & Capizzano, J. (2006). Impacts of a summer learning program: A random assignment study 

of Building Educated Leaders for Life (BELL). The Urban Institute. 

Cooper, H., Nye, B., Charlton, K., Lindsay, J., & Greathouse, S. (1996). The effects of summer vacation on 

achievement test scores: A narrative and meta-analytic review. Review of Educational Research, 

Vol. 66, No. 3, p. 227 - 268. 

Downey, D.B., von Hippel, P.T., & Broh, B.A. (2004). Are schools the great equalizer? Cognitive inequality 

during the summer months and the school year. American Sociological Review, Vol. 69, p. 613 - 

635. 

Entwisle, D. R., & Alexander, K. L. (1992). Summer setback: Race, poverty, school composition, and 

mathematics achievement in the first two years of school. American Sociological Review, 72-84. 

Fairchild, R., & Boulay, M. (2002, November). What if summer learning loss were an education policy 

priority. In Presentation for the 24th Annual APPAM Research Conference, November (Vol. 9). 

Hernandez, D.J. (2011). Double Jeopardy: How Reading Skills and Poverty Influence High School 

Graduation. Baltimore, MD: Annie E. Casey Foundation. 

Heyns, B. (1978). Summer learning and the effects of schooling (pp. 227-268). New York: Academic 

Press. 

Heyns, B. (1987). Schooling and cognitive development: Is there a season for learning? Child 

Development, Vol. 58, No. 5, p. 1151–60.

McCombs, J.; Augusting, C.; Schwartz, H.; Bodilly, S.; McInnis, B., Lichter, D.; and Cross, A. RAND 

Corporation. (2011). Making Summer Count: How Summer Programs Can Boost Children’s 

Learning. Retrieved 3/29/16 from http://www.rand.org/pubs/monographs/MG1120.html.

Rubin, D. B., & Thomas, N. (2000). Combining propensity score matching with additional adjustments for 

prognostic covariates. Journal of the American Statistical Association, 573–585. 

Schacter, J. (2003). Preventing summer reading declines in children who are disadvantaged. Journal of 

Early Intervention, 26(1), 47-58. 
Summer Advantage Final Report 

41

Stuart, E. A. (2007). Estimating Causal Effects Using School-Level Data Sets. Educational Researcher, 

36(4), 187-198. http://doi.org/10.3102/0013189X07303396

Song, M., & Herman, R. (2010). Critical Issues and Common Pitfalls in Designing and Conducting Impact 

Studies in Education Lessons Learned From the What Works Clearinghouse (Phase I). Educational 

Evaluation and Policy Analysis, 32(3), 351–371. doi:10.3102/0162373710373389 

Terzian, M and Moore, K. What Works for Summer Learning Programs for Low-Income Children and 

Youth: Preliminary Lessons from Experimental Evaluations of Social Interventions. (2009). 

What Works Clearinghouse, (2011). Procedures and standards handbook, Version 2.1. Washington, DC: 

Institute of Education Sciences, U.S. Department of Education.