Evaluation Report 

Impact Evaluation 

PYs 2017-18 and 2018-19 

Executive Summary 

This report provides an analysis and evaluation of the City of Lakes AmeriCorps program, 

specifically if the program is accelerating academic language growth of middle school LTELs. The 

method of analysis was a comparative growth analysis of standardized test scores using 

students served versus a control group. Results of the comparative growth analysis indicate that 

CoL services are accelerating academic language growth of students served. Recommendations 

based on these findings include further identifying the specific characteristics of students for 

which the program is successful, as well as which components of the program have the highest 

effect on this positive outcome. It is also recommended that the program examine its 

classification of dosing, and potentially redefine that based on the timing of the WIDA ACCESS 

test. 

Background and Purpose 

As a program that receives less than $500,000 annually in CNCS funds, City of Lakes AmeriCorps 

(CoL) utilizes an internal evaluation process. Program Director Lisa Lambert oversees all 

components of the evaluation, working closely with the district’s Research Evaluation 

Assessment and Accountability Department, who completes quantitative analyses for the 

program. For the 2017-18 and 2018-19 program years, CoL conducted an Impact evaluation which 

fits into the Preliminary Evidence tier. 

City of Lakes AmeriCorps has been serving middle school English Learner students since 2011. 

Our program deepened that focus to Long-Term English Learners in 2014 as the unique needs of 

that student group came to light as a result of national research. All activities outlined in our 

logic model (Appendix A) are designed to address our identified need, namely that Long-Term 

English Learners (LTELs) are not succeeding academically on par with their peers. LTELs are 

students with 5 or more years of Limited English Proficiency status. To be classified as an LTEL, a 

student has to have been enrolled in ESL services for at least five school years, during which they 

attended at least 135 days of school per year. 

As a group, LTELs tend to have lower grades, lower standardized test scores, and face a higher 

risk of dropping out than non-LTELs. This difference begins due to a language barrier, but after 

years of falling behind, many LTELs also develop habits of disengagement in school (Laurie 

Olsen, 2014), which further exacerbates the problem. 76% of the 1,182 middle school English 

Learners (ELs) in Minneapolis Public Schools (or 898 students) are LTELs. 91% of Minneapolis 

Public Schools EL students are scoring as not proficient on standardized tests, with 36% of them 

scoring in the "Falling Behind" categories of growth. 
 

 

 

 

 

 

City of Lakes AmeriCorps' Theory of Change is that by providing small group evidence-based 

English language instruction and support, supplemented by after-school programming that 

utilizes literacy skills in engaging ways, and by involving parents, LTELs in grades 6 -8 will 

significantly improve their academic outcomes. Participation in the CoL program will result in 

increased engagement in school as well as increased academic language skills, as evidenced by 

teacher and student surveys, pre- to post- test assessments, and a comparison of standardized 

test scores. 

Existing research: Previous year’s impact results give us reason to believe our program’s 

approach is working, because they consistently show a positive effect from the City of Lakes 

program. CoL began using English 3D in PY 2015-16. At the end of that program year, an ANOVA 

test of significance completed by our district's REAA department of showed that overall average 

growth on MCA Reading scores was higher for CoL students than for eligible (but not served) 

students. This is also true when we looked at percentile rankings of scores. Students served by 

CoL had an average percentile ranking growth of 3.84 on MCA Reading scores; the average 

percentile ranking growth of middle school English Learners overall was 3.28. 

In PY 2016-17, our district piloted use of the FAST assessment (Formative Assessment System for 

Teachers) to look at student growth over the course of the year. This assessment ranks a 

student as Low, Some or High Risk of not meeting grade level standards by the end of the year. 

For students served by CoL, the number of students ranked at High Risk fell from 74% in Fall to 

53% in Spring. Comparatively, districtwide, that number went from 39% in Fall up to 41% in Spring. 

Purpose: This impact evaluation focused on determining whether the City of Lakes program was 

having an impact on academic language acquisition of the target students, namely middle school 

Long-Term English Learners. The evaluation focused on middle school Long-Term English 

Learners in Minneapolis Public School for both the served and comparison groups. 

Results of this evaluation are being shared with the Corporation for National and Community 

Service at the end of the 2018-19 grant year. Results are also being shared with program 

management staff and other internal stakeholders to assess effectiveness of the CoL program. 

The scope of the evaluation will look at data from all CoL sites over the 2017-18 and 2018-19 

program years. It will analyze overall effect from all activities on a standardized measure of 

academic language growth, the WIDA ACCESS. 

Evaluation Methods and Data Analysis 

Questions The evaluation will address the following key question: 

1) Do students participating in the CoL program have higher growth in academic English 

skills than students who are not receiving our services? 
 

 

 

 

Evaluation Design: This evaluation is an Impact Evaluation, designed to examine how well the 

program is meeting the stated effect of increasing academic language acquisition for the target 

group of students. 

Data Collection 

Program data of who is served by City of Lakes is collected from members with their first quarter 

reports, due in early December. Members report the first name, last name, grade, and student ID 

number of any students receiving the full range of program services (Academic Language 

instruction, push-in support, and parent communication). Students added to caseloads 

throughout the year are done so as they join, with a deadline of being added by the next 

quarterly report due date. All reports from members are submitted through Google Reporting 

Spreadsheets. These spreadsheets are only accessible to members logging in with MPS login 

info, and once entered, are not editable by members. The MPS Research Evaluation Assessment 

and Accountability Department (REAA) oversees data collection and management of 

standardized test scores both for CoL students and for the comparison group. 

Comparative Analysis: To ensure that CoL is accelerating growth for LTELs, comparative growth 

on standardized test scores is evaluated between CoL caseload students and a matched 

comparison group. This is measured through WIDA ACCESS scores as well as through another 

standardized test, FAST (Formative Assessment System for Teachers). WIDA ACCESS is our 

district's measure of language proficiency for ELs in MPS and is administered annually to all 

students receiving ESL services. 

Growth targets have been recently developed for WIDA ACCESS and are based on English 

Learners students showing higher growth in earlier years, then slowing a bit in later years. This is 

a typical trend for English Learners, whose language skills develop quickly as they are newly 

learning the language, then tend to stall as they move from conversational language 

development to more academic language development. All growth targets are set based on a 

student achieving full language proficiency within seven years. An accountability index of 1 

indicates that a student met their target; anything less means they fell short of their target, and 

anything more means that exceeded their target. 

We anticipated seeing students receiving higher amounts of CoL services meeting their WIDA 

ACCESS accountability index score (score of 1.0 or higher) when compared to a match sampling 

group. To evaluate this, we looked at both the percentage of students who received a 1.0 or 

higher Accountability index (are on track with growth), and at the mean accountability index for 

students (sum of student change divided by target change). With FAST scores, we also looked at 

the percentage of students meeting their growth targets. 

To determine growth comparison using WIDA ACCESS or other standardized test scores, an 

evaluator from the district's REAA (Research Evaluation Assessment and Accountability) Dept. 

completed an analysis of growth scores for COL participants (treatment) vs. eligible students 

(control). This evaluation involved a comparison of growth from students who received services 

from a CoL member to students who did not receive services from a CoL member but shared the 

following characteristics: is in middle school, is an LTEL (student with 5 or more years Limited 

English Proficiency status), and attends a CoL site as their school. Data to identify these students 
came from Google Reporting forms submitted by CoL members on students they had served, 

and from district databases with student demographic and test score information. 

Timeline & Logistics: 

Evaluation Element Timeline Conducted By 

Clear definition of eligible 

students and identification of 

matching variables 

August 2018 Program Director Lisa Lambert 

and Evaluator Melody JacobsCassuto 

Members trained in data 

collection elements 

August/September 2017, 

August/September 2018 

Program Director Lisa Lambert 

Caseload students and 

comparison group identified 

September 2017, September 

2018 

Members, Program Director Lisa 

Lambert, and Evaluator Melody 

Jacobs-Cassuto 

Data checks to ensure 

accurate and timely member 

report submissions 

2017 – October and 

December 

2018 – April, October, and 

December 

2019 – April and July 

CoL Program Staff 

WIDA ACCESS administered February/March 2018 and 

February/March 2019 

Members and partner ESL 

teachers 

Data analysis of WIDA 

ACCESS growth 

July-September 2018 and July 

– September 2019 

REAA Evaluator Melody JacobsCassuto 

Evaluation Report created October and November 2019 Program Director Lisa Lambert 

Internal stakeholders 

meeting with evaluation data 

reviewed and continuous 

improvement decisions made 

December 2019 Program Director Lisa Lambert 

Budget 

The evaluation budget for our program is $4,000 per year, which covers approximately 80 hours 

of evaluation time from our district’s REAA Department. REAA provided this estimate of time 

and the rate of $50 per hour to complete this evaluation. This allotted budget covers both the 

Impact Evaluation and Process/Implementation Evaluations being completed by December 2019. 

This amount is less than 1% of our overall yearly budget. Since much of the data collection and 

analysis was done online and with equipment provided to members by their site (ie computers), 

no additional budget was required for supplies. The greatest material needed was time -- both 

by members in doing data collection and by Program Director Lisa Lambert who completed data 

analysis. Time to complete these items is included in position descriptions for both members and 

the Program Director, and no additional budget was needed to support these activities. 
 

Results 

For both PY2017-18 and PY2018-19, in comparing growth of students receiving CoL services with 

a control group of middle school LTELs attending the same schools who did not receive CoL 

services, data trends tell us that students receiving CoL services make more growth than 

students in the control group. 

At first look of overall ACCESS growth targets for 2017-18, students in the control (no CoL 

services) group appear to be higher, with 36% meeting or exceeding their growth targets 

compared to 34% of students who received CoL services. However, when broken down by 

program dosage, students receiving either a Medium or High dosage of CoL services met their 

growth targets at a higher rate than students receiving no services or a Low dosage of services. 

This data indicates that there is a minimum dosage required for effect, but that once that 

minimum dosage is met, student growth is positively affected. 
 

The same is true when looking at the mean accountability index for students (the proportion of 

the ACCESS growth target met). Students receiving CoL services in PY2017-18 grew average of 

60.1% of their growth target, while students not receiving services grew an average of 49.9% of 

their target. In PY2018-19, students receiving CoL services grew an average of 53% of their target 

compared to average of 44% for those not receiving services. 
This trend continues when comparing the average Accountability Index for students served by 

CoL to a wider range that includes the district and the state. As the below chart illustrates, the 

Accountability Index for the Control group hovers near MPS and State results for both PY2017-18 

and PY2018-19, whereas the Accountability Index for students served by CoL in significantly 

higher. 

Mean Accountability 

Index 

Year MPS State Control Group CoL Served Group 

Middle School PY2017-18 53.30 50.58 49.9 60.1 

PY 2018-19 45.50 42.26 44.0 53.0 

Beyond ACCESS, this result is further enforced when we examine the effect on FAST scores. In both 

program years, students receiving CoL services met their winter growth targets at a higher rate than 

the control group of students not receiving CoL services. 
 

 

 

Cohesively, these results indicate that students participating in the CoL program have higher growth 

in academic English skills than students who are not receiving CoL services. 

Limitations: 

This study does have potential limitations. Pre-existing differences between the intervention and 

comparison groups at the outset of the intervention may have led to inaccurate estimates of the 

program’s effects, either by underestimating for some groups or overestimating for others. The groups 

were based on sharing three characteristics: being in middle school, being a LTEL, and attending a school 

where CoL services were offered. Groups were not propensity matched further to account for other 

characteristics such as home language, receiving other ESL services, or qualifies for free or reduced-price 

lunch, to name a few. Matching based on the additional characteristics could illustrate exactly what 

effect CoL services have for each specific population. 

Another limitation is the timing of the WIDA ACCESS. This assessment is given in late February and 

throughout March, while our program continues to serve students through May. The dosage we use in 

reporting a student’s level of received services as Low, Medium or High is based on services received for 

the entire program year. However, utilizing the ACCESS as a measure of academic language growth 

really only measures growth based on the amount of services received through February or early March. 

Students who started the program late, and are therefore reported as having a Low or Medium dosage, 

may show less growth than they actually had during the course of the entire school year based on the 

early timing of the test. A pre- and post-test is given to CoL students to fully measure their academic 

language growth for the entirety of the program year; however, it is not given to the comparative group 

of students due to the increase in testing burden it presents to those students. Therefore, WIDA ACCESS 

remains the one standardized measure of language growth we can use for both control and served 

groups. 

Conclusion and Recommendations 

The City of Lakes program is demonstrating positive effects on advancing middle school LTEL’s

academic achievement. Students receiving CoL services have shown higher growth on both 

WIDA ACCESS and FAST standardized tests than their fellow middle school LTEL peers who are 

not receiving CoL services. Moving forward, it will be important to further identify the specific 

characteristics of students for which the program is successful, as well as which components of 

the program have the highest effect on this positive outcome. It is also recommended that the 

program examine its classification of dosing, and potentially redefine that based on the timing 

of the WIDA ACCESS test. 

Appendixes –

A. Logic Model 

B. Example data collection tool - English 3D Logs 
References 

Olsen, Laurie (2014). “Meeting the Unique Needs of Long-Term English Language Learners: A 

Guide For Educators,” National Education Association, published March 2014.

http://www.laurieolsen.com/uploads/2/5/4/9/25499564/lo_nea_ltel_publication.pdf