Evaluation Report Brief 

Mile High United Way: Summit 54

Office of Research and Evaluation, Corporation for National and Community Service June 2018 

nationalservice.gov/research 1 

What is the community challenge? 

Summer learning loss is a well-documented phenomena 

wherein students lose math and literacy skills during their time 

away from school. It disproportionately affects students from 

lower socioeconomic status (SES) background, who tend to 

have fewer summer enrichment experiences than their higherSES peers. It continues to contribute to the larger income-based 

achievement gap in the U.S. education system. 

Program At-a-Glance 

CNCS Program: Social Innovation Fund

Intervention: Summer Advantage 

Subgrantee: Summer 54 

Intermediary: Mile High United Way 

Focus Area(s): Youth Development/Education 

Focus Population(s): K-4th grade, prioritize those 

from lower-income background 

Community(ies) Served: Roaring Fork School 

District, Colorado 

What is the promising solution?

The Summer Advantage is a voluntary, five-week intensive 

summer learning program that provides summer enrichment to 

elementary school students to help mitigate summer learning 

loss. The program, which has the explicit goal of holding student literacy scores steady throughout the 

summer, consists of the following core components: 

▪ For four days of the week, students receive three hours of literacy and math instruction in the morning and 

two hours of enrichment activities in the afternoon. On Fridays, the program offers inspirational speakers, 

field trips, and other activities. 

▪ Highly qualified and certified teachers and staff are hired and trained to lead each class (2-24 staff-tostudent ratio). 

▪ The program uses the Summer Success Math and Reading curriculum (Houghton Mifflin) for academic 

enrichment instruction. 

▪ The program has high expectation of student attendance and parental engagement through parent/teacher 

meetings, workshops, field trips, and other activities. 

What was the purpose of evaluation?

The evaluation of Summer Advantage by Augenblick, Palaich and Associates (APA) was conducted from 

2012-2017, with student impact data collected for 2014-2016 summer sessions. 

The report includes an implementation evaluation, exploring whether Summer Advantage implemented the 

program with fidelity to its stated model. The findings are based on data from multiple sources, including 

document review, interviews with program and district leaders, focus groups with program teachers, and a 

site visit each summer to gather data on program implementation fidelity. 

The report also includes an impact evaluation, using a quasi-experimental design with propensity score 

matching, examining whether reading skills of students in Summer Advantage improved significantly more 

than those of similar students not served by the program. The impact analysis includes data from the summer 

2014-2016 sessions and includes a total of 925 Summer Advantage students and an equal number of similar 

comparison students. The impact evaluation sought to answer three research questions: 
2 

1. Does participation in Summer Advantage have an impact on student literacy for students who 

participated compared to a district sample as measure by standardized assessment? 

2. Does the number of years of participation in Summer Advantage moderate impacts on student literacy, 

such that students with previous participation experience greater impact than students with no 

previous participation? 

3. Does Summer Advantage participation impact Kindergarten student performance on standardized 

literacy assessment compared to non-program participants? 

What did the evaluation find?

The implementation evaluation found Summer Advantage was implemented with fidelity. The program was 

able to: 

1. identify and enroll students who met program 

criteria, 

2. recruit, hire, and train qualified staff, 

3. build strong relations with school and district-level 

leaders, 

4. teach academic content using an appropriate 

curriculum aligned to district objectives, and 

5. maintain clearly communicated program 

expectations with parents. 

The impact study found a statistically significant impact of 

Summer Advantage on student reading performance for kindergarten and first grade students. Examined by 

grade level, APA found that participation of kindergarten students in only one year of Summer Advantage 

provides a statistically significant and positive effect on kindergarten student reading. 

Additionally, APA found that for students who participate in multiple years of Summer Advantage, the 

benefit of the second or third year of participation is equivalent to the benefit from the first year of 

participation. In other words, participating in multiple years of Summer Advantage has an additive effect. 

Evaluation At-a-Glance 

Evaluation Design(s): Impact evaluation (quasiexperimental design with propensity score 

matching) and implementation evaluation 

Study Population: Kindergarten through 4th grade 

(Independent) Evaluator(s): Augenblick, Palaich 

and Associate (APA) 

This Evaluation’s Level of Evidence*: Preliminary 

*SIF and AmeriCorps currently use different definitions of levels of evidence.

Notes on the evaluation

When examining the multiple year effect of Summer Advantage on participating students, the sample size for 

students who attended the program for three years was only 50. Giving the small sample size, this research 

question should be reexamined once more students have completed multiple years of Summer Advantage. 

How is Summit 54 using the evaluation findings to improve?

The detection of statistically significant relationship between kindergarten and first grade student participation 

in Summer Advantage is promising. APA has suggested Summit 54 to continue its evaluation of the program 

in order to determine if this positive effect is also present for children in other grades. 

The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service (CNCS), combines public and private resources to grow the impact of 

innovative, community-based solutions that have compelling evidence of improving the lives of people in low-income communities throughout the U.S. The SIF invests in three 

priority areas: economic opportunity, healthy futures, and youth development. 

The content of this brief was drawn from the full evaluation report submitted to CNCS by the grantee/subgrantee. The section of the brief that 

discusses evaluation use includes contribution of the grantee/subgrantee. All original content from the report is attributable to its authors. 

To access the full evaluation report and learn more about CNCS, please visit nationalservice.gov/research. 

Office of Research and Evaluation, Corporation for National and Community Service June 2018

nationalservice.gov/research