Des Moines Public Schools (DMPS) AmeriCorps

AmeriCorps Literacy Tutoring to K-5th Grade Students 

Program at DMPS Elementary Schools

Short-Term Impact Study

Evaluation Report

Prepared by: 

W. Todd Abraham, PhD and Elena Polush, PhD

Independent Contractors

September 25, 2019
i

TABLE OF CONTENTS

EXECUTIVE SUMMARY .......................................................................................................... 1

INTRODUCTION......................................................................................................................... 4

Program Description ................................................................................................................ 4

Prior Research........................................................................................................................... 4

Problem Statement.................................................................................................................... 5

Theory of Change...................................................................................................................... 5

SHORT-TERM IMPACT EVALUATION STUDY ................................................................. 5

Purpose and Main Questions................................................................................................... 5

At the student level................................................................................................................ 6

At the school level.................................................................................................................. 6

Evaluation Design ..................................................................................................................... 6

Sample Description ............................................................................................................... 7

Table 1. Unidentified Students Receiving AC Interventions.............................................. 7

Table 2. Transfer Students Receiving Interventions........................................................... 8

Data Sources.............................................................................................................................. 8

Contextual Data .................................................................................................................... 8

School-Level Data ................................................................................................................. 9

AmeriCorps Member Data .................................................................................................. 9

Student-Level and Outcomes Data.................................................................................... 10

Table 3. Conditions by Grade ........................................................................................... 10

Table 4. Assessments by Grade ........................................................................................ 11

General Analytic Approach ................................................................................................... 11

Quantitative Methods......................................................................................................... 11

Qualitative Methods............................................................................................................ 12

RESULTS .................................................................................................................................... 12

Evaluation Question 1A.......................................................................................................... 12

Kindergarteners.................................................................................................................. 13

Figure 1. AC Contact vs. No Contact in AC Schools (K) ................................................ 14

1

st Graders ........................................................................................................................... 14

Figure 2. AC Contact vs. No Contact in AC Schools (1st) ............................................... 14

2

nd Graders .......................................................................................................................... 15

Figure 3. AC Contact vs. No Contact in AC Schools (2nd)............................................... 15

Table 5. AC Support vs. No Support within AC Schools (K-2

nd).................................... 16
ii

3

rd Graders........................................................................................................................... 17

Figure 4. AC Contact vs. No Contact in AC Schools (3rd)............................................... 17

4

th Graders........................................................................................................................... 17

Figure 5. AC Contact vs. No Contact in AC Schools (4th)............................................... 18

5

th Graders........................................................................................................................... 18

Figure 6. AC Contact vs. No Contact in AC Schools (5th)............................................... 19

Table 6. AC Contact vs. No Exposure within AC Schools (3rd-5

th)................................. 20

Evaluation Question 1A: General Findings.......................................................................... 21

Evaluation Question 1B.......................................................................................................... 21

Kindergarteners.................................................................................................................. 21

Figure 7. AC Service vs. No AC Service in AC Member Schools (K)............................ 22

1

st Graders ........................................................................................................................... 22

Figure 8. AC Service vs. No AC Service in AC Member Schools (1st)........................... 22

2

nd Graders .......................................................................................................................... 23

Figure 9. AC Service vs. No AC Service in AC Member Schools (2nd) .......................... 23

Table 7. AC Contact by Grade in Schools with AC Members (K-2) ............................... 24

3

rd Graders........................................................................................................................... 25

Figure 10. AC Service vs. No AC Service in AC Member Schools (3rd)......................... 25

4

th Graders........................................................................................................................... 25

Figure 11. AC Service vs. No AC Service in AC Member Schools (4th)......................... 26

5

th Graders........................................................................................................................... 26

Figure 12. AC Service vs. No AC Service in AC Member Schools (5th)......................... 26

Table 8. AC Contact by Grade in Schools with AC Members (3rd-5

th)............................ 27

Evaluation Question 1B: General Findings.......................................................................... 28

Evaluation Question 2A.......................................................................................................... 28

Kindergartners.................................................................................................................... 28

Figure 13. Growth by Number of AC Interventions Received (K) .................................. 29

1

st Graders ........................................................................................................................... 29

Figure 14. Growth by Number of AC Interventions Received (1st) ................................. 29

2

nd Graders .......................................................................................................................... 30

Figure 15. Growth by Number of AC Interventions Received (2nd)................................. 30

Table 9. Growth by Number of AC Interventions Received (K-2) .................................. 31

3

rd Graders........................................................................................................................... 32

Figure 16. Growth by Number of AC Interventions Received (3rd)................................. 32
iii

4

th Graders........................................................................................................................... 32

Figure 17. Growth by Number of AC Interventions Received (4th)................................. 33

5

th Graders........................................................................................................................... 33

Figure 18. Growth by Number of AC Interventions Received (5th)................................. 33

Table 10. Growth by Number of AC Interventions Received (3rd-5

th)............................. 34

Evaluation Question 2A: General Findings.......................................................................... 35

Evaluation Question 2B.......................................................................................................... 35

Table 11. Matching AC and Non-AC Schools ................................................................. 36

Table 12. Baseline Equivalence across Matched Schools................................................ 36

Kindergartners.................................................................................................................... 36

Figure 19. AC Schools vs. Non-AC Schools (K) ............................................................. 37

1

st Graders ........................................................................................................................... 37

Figure 20. AC Schools vs. Non-AC Schools (1st) ............................................................ 37

2

nd Graders .......................................................................................................................... 38

Figure 21. AC Schools vs. Non-AC Schools (2nd) ........................................................... 38

Table 13. AC Service vs. No AC Service in Matched Schools (K-2

nd)............................ 39

3

rd Graders........................................................................................................................... 40

Figure 22. AC Schools vs. Non-AC Schools (3rd)............................................................ 40

4

th Graders........................................................................................................................... 40

Figure 23. AC Schools vs. Non-AC Schools (4th)............................................................ 41

5

th Graders........................................................................................................................... 41

Figure 24. AC Schools vs. Non-AC Schools (5th)............................................................ 41

Table 14. AC Service vs. No AC Service in Matched Schools (3rd-5

th)........................... 42

Evaluation Question 2B: General Findings.......................................................................... 43

Evaluation Question 3A.......................................................................................................... 43

Table 15. Variance Reduction due to AC Member Presence ........................................... 44

Evaluation Question 3A: General Findings.......................................................................... 44

Evaluation Question 3B.......................................................................................................... 45

Evaluation Question 4A.......................................................................................................... 45

Table 16. Intervention Efficacy by Identification Method (K-2) ..................................... 46

Table 17. Intervention Efficacy by Identification Method (3rd-5

th).................................. 47

Evaluation Question 4A: General Findings.......................................................................... 47

Evaluation Question 4B.......................................................................................................... 48

Table 18. Intervention Efficacy by Delivery Method (K-2)............................................. 49
iv

Table 19. Intervention Efficacy by Delivery Method (3rd-5

th).......................................... 50

Evaluation Question 4B: General Findings.......................................................................... 50

CONCLUSIONS ......................................................................................................................... 51

Limitations............................................................................................................................... 52

Ethical Considerations............................................................................................................ 53

RECOMMENDATIONS............................................................................................................ 54

REFERENCES............................................................................................................................ 55

APPENDICES............................................................................................................................. 56

Appendix A: Theory of Change............................................................................................. 56

Appendix B: AmeriCorps Member Survey.......................................................................... 57

Appendix C: AmeriCorps Member Survey Results (October 2018).................................. 58

Appendix D: AmeriCorps Member Survey Results (April 2019) ...................................... 62

Appendix E: General Analytic Model................................................................................... 68
1

EXECUTIVE SUMMARY

In 2018-2019, Des Moines Public Schools (DMPS) served over 33,000 students in Des Moines, 

Iowa, a diverse, urban city of approximately 205,000 people. Over 73% of DMPS students were 

low-income qualifying free or reduced-price lunch, over 57% were minorities, over 19% were 

English Language Learners, and over 14% received Special Education services. According to the 

Annie E. Casey Foundation (2011), 16% of children who are not reading proficiently in 3rd grade 

drop out of school, compared to only 4% of their proficient peers. Long-term impacts of early 

reading difficulties can be devastating (Lane, 2014). 

Recognizing the critical importance of reading and literacy skills for students’ future success, 

DMPS set a district goal to achieve proficiency in reading among 100% of K-5 students. 

Unfortunately, the number of students who are not currently reading proficiently in K-5

th grade 

within the DMPS district is high. For example, only 60.6% of DMPS 3

rd grade students are 

proficient in reading. Similar proficiency rates for grades K-5 in literacy at DMPS elementary 

schools demonstrate the need for successful intervention to reach the 100% proficient goal. To 

address the disparity in proficiency rates, AmeriCorps members provided evidence-based Lexia 

Core5 Reading tutoring to K-5

th students who were struggling to achieve or maintain proficiency 

during the 2018-2019 school year. All of the 38 DMPS elementary schools utilized Lexia Reading 

Core5, an evidence-based, personalized technology-based intervention adopted by DMPS to 

support academic instruction and student learning. The program goal was to place AmeriCorps 

members in 29 (76%) of the DMPS elementary schools during the 2018-2019 school year. 

Although some members left their positions, 17,651 separate intervention services were provided to 

2,613 K-5 students at 20 DMPS schools.

To evaluate the efficacy of AmeriCorps intervention services and factors that potentially alter 

efficacious intervention delivery, the current study examined student outcomes regarding reading 

proficiency across multiple within-case and across-case comparisons. In attempts to isolate various 

components of potential influence at the individual and school levels, the current evaluation addressed 

a series of interrelated questions to determine whether AmeriCorps member service 1) impacted 

student reading proficiency and proficiency growth, 2) provided an added benefit to students in need of 

additional support, 3) provided an added benefit to schools that house AmeriCorps members, and 4) 

was influenced by differences in implementation fidelity within each member placement. Specific 

questions addressed and findings relevant to each question are presented below.

 Do students receiving AmeriCorps intervention services demonstrate growth in reading

proficiency that differs from students who do not receive intervention services?

 Do students receiving AmeriCorps intervention services demonstrate reading proficiency

growth that differs from same-grade students who do not receive intervention services but

attend schools where AmeriCorps services are provided to other grades?

Findings from comparisons between students who received AmeriCorps intervention support 

and students in the same grades, within the same schools, who did not receive intervention 

services, findings were generally consistent. As one might expect, students in need of services 

were generally performing poorly in relation to students who did not require services. Given 

that service provision was determined by decrements in reading proficiency within the Lexia 

Core 5 system, these findings are not surprising. However, in nearly all grades, students 
2

receiving AmeriCorps services maintained growth in reading proficiency that was comparable 

to their peers who did not require intervention supports. These findings suggest that 

AmeriCorps service provides a positive benefit to students who receive services. However, 

findings comparing students in schools served by AmeriCorps members across grades that were 

either served or not served by an AmeriCorps member did not reveal differential patterns of 

proficiency growth as a function of AmeriCorps member presence in the building versus actual 

service provision to students in a particular grade. 

 Does reading proficiency growth differ as a function of intervention frequency among students

who receive AmeriCorps intervention services?

 Do AmeriCorps services provide added benefits in schools with service members in

comparison to non-AmeriCorps schools with the same existing services?

Comparison of reading proficiency growth among students in schools where AmeriCorps 

members provided services and schools that did not have an AmeriCorps member indicated 

that reading proficiency growth was approximately consistent across schools, whether the 

school had an AmeriCorps service member or not. Although these findings suggest little benefit 

to having an AmeriCorps member in the building, comparisons of students receiving support 

services with regard to amount of service received painted a different picture. In schools where 

AmeriCorps members provided support, the amount of support services was consistently 

related to reading proficiency growth. While the amount of service received is necessarily tied 

to proficiency decrements (those falling behind require more support), findings indicated that 

increases in intervention frequency reduced proficiency deficits among students who received 

support services. That is, although a student receiving 10 interventions would be expected to 

score lower on a proficiency assessment than a student receiving no interventions, that same 

student who received 10 interventions did not continue to fall further behind a student who 

received only 5 interventions. These findings suggest that AmeriCorps intervention services 

help to maintain reading proficiency growth among those students who require member 

support. 

 Does presence of an AmeriCorps member provide increased benefits to students through

impact on variability in reading proficiency relative to students in buildings without

AmeriCorps services?

 Does growth in reading proficiency differ as a function of whether students who are flagged for

support by the Lexia Core5 program receive intervention services from AmeriCorps members?

Findings were limited with regard to added benefits due to unavailable data regarding 

individual student need for support in non-AmeriCorps buildings and also by somewhat limited 

information about other services potentially available in all buildings. Generally, AmeriCorps 

member presence did account for small amounts of variance in students initial proficiency 

assessments, but compelling evidence with regard to proficiency growth did not emerge.
3

 Do AmeriCorps member hours/intervention loads or methods used for identifying students in

need of intervention differentially relate to reading proficiency growth?

 Does modality of intervention delivery differentially relate to growth in reading proficiency?

Findings related to differential methods of identifying students in need of support services or 

differential modalities of delivering interventions consistently demonstrated that the efficacy of 

the AmeriCorps interventions did not differ across important differences in implementation. 

Specifically, use of flagging or rostering demonstrated no consistent effects across grades with 

regard to altered reading proficiency growth, whereas delivery of intervention services in oneon-one or small group settings were generally equally efficacious. These findings do seem 

promising in that lack of influences allows for greater flexibility, a common theme among 

members, in how AmeriCorps members are able to best deliver intervention services to the 

students who need additional support. 

Collectively, the findings in this evaluation point to evidence that AmeriCorps member service 

does provide a benefit to students who receive services in that those students do not continue to fall 

further behind their peers over the course of the academic year. Specifically, parallel growth among 

those who did and did not receive AmeriCorps support, and decreased decrements in proficiency due 

to increase intervention experiences among those who did receive services clearly demonstrate that 

the services provided by AmeriCorps members are benefiting students. Overwhelming evidence of 

added benefit to schools due to AmeriCorps service was less compelling due largely to limitations on 

school-level programming and support data and, more importantly, to relatively little variance in 

reading proficiency growth across buildings. 

Feedback obtained from AmeriCorps members consistently indicated being isolated, unintegrated, 

or unrecognized by school staff. Members also indicated that they often found their position and 

responsibilities unclear in the eyes of school staff. As AmeriCorps members intend to continue service 

provisions in schools, it seems wise to clearly delineate what members are in the schools to do and to 

develop a better system to integrate members into the school culture early during their service 

tenure. 

AmeriCorps members also indicated a need for better training to feel more confident with the 

material and systems they are expected to use. More structured training will also instill a greater 

degree of implementation fidelity aiding the disentanglement of member variability in service 

provision from efficacy of the services provided.

Subsequent efforts to evaluate the efficacy of AmeriCorps members’ service using student 

outcomes would be wise to identify and collect data elements that account for extraneous sources of 

student-level variability. Future evaluation efforts might continue to use contextual measures, but 

individual measures of socioeconomic and family dynamics factors would be much more 

informative with regard to prediction of individual assessment scores. An increased emphasis on 

individual student indicators related to student outcomes would benefit future evaluation efforts 

regardless of whether quasi-experimental, matching, statistical control, or randomized designs are 

employed. 
4

INTRODUCTION

Program Description 

During the 2018-2019 school year, the AmeriCorps Literacy Tutoring to K-5th Grade Students 

program (hereinafter referred to as the program) in the Des Moines Public Schools (DMPS) focused 

on reading literacy. The purpose of the program was to provide at-risk K-5th grade students at the 

DMPS elementary schools with additional Lexia Reading Core5 scripted interventions delivered by 

AmeriCorps (AC) members aimed at accelerating these students’ mastery of foundational literacy 

skills. 

DMPS includes 38 elementary schools. While the program goal was to serve 29 (76%) 

elementary schools, the AmeriCorps members provided services to 21 (55%) elementary schools 

during the 2018-2019 school year. Two AC members left at the beginning of the school year. Their 

school assignments were not replaced. Although three other members left and their assignments 

were not filled, they provided partial services. Among schools that the AmeriCorps members 

served, 17 (85%) also received Title I funds. All of the 38 DMPS elementary schools utilized Lexia 

Reading Core5, an evidence-based, personalized technology-based intervention adopted by DMPS 

to support academic instruction and student learning. 

Prior Research 

In 2018-2019, Des Moines Public Schools (DMPS) served over 33,000 students in Des Moines, 

Iowa, a diverse, urban city of approximately 205,000 people. Over 73% of students were lowincome (qualifying free or reduced-price Lunch), over 57% were minorities, over 19% were 

English Language Learners, and over 14% received Special Education services. According to the 

Annie E. Casey Foundation (2011), 16% of children who are not reading proficiently in 3rd grade 

drop out of school, compared to only 4% of their proficient peers. Long-term impacts of early 

reading difficulties can be devastating (Lane, 2014). Further, a combination of factors including 

high absenteeism, low achievement, and discipline problems predict whether a student will drop out 

of school. Students with greater levels of such risks are less likely to graduate from high school 

(Balfanz, Herzog, et al., 2007; Kurlaender, Reardon, et al., 2008). 

The empirical evidence does not support the adage, “Just wait, they will catch up” (Foorman, et. 

al., 1997). Rather, intervening early is the key to helping struggling students before they fall too far 

behind (Haager, Klingner, & Vaughn, 2007). Research demonstrates that tutoring effectively 

improves students’ growth in literacy. Pullen, Lane, and Monaghan (2010) argue that tutoring 

programs are promising interventions for struggling readers and appropriate for implementation by 

trained classroom volunteers. Evidence-based reading interventions (1) provide universal screening 

and quality instruction, (2) engage students in learning, and (3) facilitate student academic growth 

(Haager, Klingner, & Vaughn). 

Lexia Core5 Reading is a 3-step personalized learning model that is student-centered and supports 

student independent work at their own pace to develop foundational reading skills 

(https://www.lexialearning.com/why-lexia/research-proven).
5

Problem Statement 

Recognizing the critical importance of reading and literacy skills for students’ future success, the 

DMPS district goal was to achieve proficiency in reading among 100% of K-5 students. Unfortunately, 

the number of students who were not reading proficiently in K-5th grade within the DMPS district was 

high. For example, only 60.6% of DMPS' 3rd grade students were proficient in reading. Similar 

proficiency rates for grades K-5 in literacy at DMPS elementary schools demonstrated the need for 

successful intervention to reach the 100% proficient goal. To address this disparity in proficiency rates 

21 AmeriCorps members provided evidence-based Lexia Core5 Reading tutoring to K-5th students who 

were struggling to achieve or maintain proficiency during the 2018-2019 school year. Lexia Core5 

Reading was aligned with classroom instruction and district and state standards. 

Theory of Change 

The program implementation guiding assumptions include: 

 The AmeriCorps program recruits and engages members that provide sustainable connections 

to the DMPS schools for future years. 

 The AmeriCorps members provide positive experiences when serving students and staff 

members in the DMPS schools. 

 Students gain more confidence with reading after working with the AmeriCorps members 

(tutors). 

 Students will demonstrate more growth than predicted growth outcomes/benchmarks because 

of the intervention support by the AmeriCorps members (tutors). 

These assumptions undergird the program theory of change (see Appendix A) that provided the basis 

for this evaluation. 

This report describes the evaluation of the DMPS AmeriCorps Literacy Tutoring to K-5th Grade 

Students program during the 2018-2019 school year of (09/01/2018 – 05/30/2019). The evaluation 

implementation activities were consistent with activities outlined in the evaluation plan provided 

August 30, 2018. The report first describes the evaluation purpose and main questions, design, ethical 

considerations, and limitations. The body of the report presents analytic findings specific to each 

evaluation question. The report concludes with interpretation of findings, remaining 

challenges/questions, and recommendations. 

SHORT-TERM IMPACT EVALUATION STUDY 

Purpose and Main Questions 

Consistent with the evaluation plan the program evaluation examined two aims to assess (1) 

potential beneficial influences of the AmeriCorps members’ services on students’ individual academic 

growth and (2) added benefits to the school-student support system due to AmeriCorps members’ 

services. Primary questions relevant to each of the evaluation aims included both student- and schoollevel outcomes, specifically: 
6

At the student level

(1) Does school year reading growth among students who receive AmeriCorps member 

intervention differ systematically from semester/school year reading growth among students 

who do not receive AmeriCorps intervention service? 

(2) Does school year reading growth among students differ systematically as a function of how 

much AmeriCorps service students receive? 

At the school level

(3) Do schools with AmeriCorps members providing services demonstrate higher student 

proficiency in reading and provide increased benefits to students in comparison to schools 

without AmeriCorps service members?

(4) Do characteristics of AmeriCorps members or the modality of service delivery influence the 

effectiveness of member services on student literacy?

Although the primary evaluation questions were based on student outcomes at both the individual 

and school level, process-related data (i.e., formative evaluation) were also collected to assess and 

describe program implementation. These data were collected to inform decisions for potential 

improvements and to better support AmeriCorps members’ experiences that ultimately relate to student 

learning. Specifically, AmeriCorps members were invited to share their perspectives about experiences 

providing tutoring to students and potential impacts on students 

Evaluation Design 

The evaluation design employed both within-case and across-case methods at both the student and 

school levels to assess influences of AmeriCorps members’ services on student reading literacy. 

Within-case approaches examined literacy at the student level where growth across the single school 

year on the same literacy metrics assessed repeatedly (i.e., FAST or MAP) within each student served 

as the outcome of interest. In addition, focus on students within the same schools served as a withincase assessment of building-level proficiency as it related to AmeriCorps member service. Student 

proficiency was also examined across different students (groups of students) and across schools to 

assess added beneficial influences on student literacy related to AmeriCorps member service. 

Given complexities involving the multiple academic services or interventions provided at DMPS 

schools during school hours, the ability of students to transition from needing service to not needing 

service and back, different methods of identifying students in need of service, as well as differences in 

service providers, modality of service provision, and general school characteristics, traditional 

comparison group designs were not possible. Rather, the evaluation design involved multiple 

comparisons that attempt to hold certain confounding influences constant within each comparison 

under the explicit understanding that no single comparison is optimal. Although each of the 

comparisons described below are confounded with some factor, or set of factors, (e.g., individual 

differences, school characteristics, AmeriCorps member characteristics, service modality, etc.), the 

complete set of comparisons attempts to assemble a body of compelling evidence that yields valid 

conclusions about the effectiveness and added benefit of the AmeriCorps member intervention services 

on student literacy. 
7

Sample Description

The evaluation sample of students included 2,667 students in grades K-5 who attended schools 

with AmeriCorps members providing Lexia interventions and K-5 students in all other DMPS schools. 

Student rosters included 12 entries for students receiving Competent Private Instruction (CPI), who 

were excluded from the current evaluation study. Among students receiving AmeriCorps intervention 

services, 54 students experiencing 75 interventions (see Table 1) did not have a corresponding State ID 

for linkage to demographic and other data. These cases were also eliminated from analyses.

Table 1. Unidentified Students Receiving AC Interventions

Of the remaining students receiving AC intervention services, 10 transferred during the academic 

year to another school where they also received AC services. To avoid double counting these students 

in analyses, the data record for the school in which these students received the majority of their AC 

intervention services was retained. In the event that number of interventions was equal across schools, 

the school at the end of the year was retained (see Table 2). The resulting sample of AC students 

included 17,651 interventions conducted with 2,613 individuals. Similar steps were taken to ensure 

demographic data was available for non-AC students, resulting in a comparison sample of 13,097 

students in grades K-5 within 17 DMPS schools. 
8

Table 2. Transfer Students Receiving Interventions

Data Sources 

 

Contextual Data 

Although data were obtained for both students and buildings (see below), indices of socioeconomic 

conditions are not collected in the DMPS system. In order to account for potential influences due to 

socioeconomic factors, the evaluation design included contextual data obtained from the U. S. Census 

Bureau’s American Community Survey (ACS). The ACS is an ongoing survey that collects various 

types of demographic information, including socioeconomic indices, for use at the block group level. 

The ACS design does not sample all residents of a particular area but does continually collect 

information within that area. Data are available each year and as estimates across multi-year periods. 

To avoid sparse block group data around schools in more industrialized (non-residential) blocks, 

single-year 2017 ACS estimates were aggregated to the census tract level, with tracts usually 

containing five block group areas. Each tract included the area surrounding only one DMPS building to 

reflect a ‘school neighborhood’ assessment of socioeconomic conditions. 

Although not identical to other indices of neighborhood socioeconomic adversity and disadvantage, 

the measure created for use in the current evaluation included similar contextual-level constructs. 

Specifically, ACS estimates for the percentage of families below the poverty level, percentage of 

unemployed adults, percentage of adults with less than a high school education, percentage of families 

receiving public assistance, and percentage of single female-headed households were compiled. In 

addition, percentage of non-White population and per capita income were included to reflect racial 

diversity, and family income adjusted for family size. Preliminary analyses indicated moderate-to-high 

correlations among the ACS indicators. An exploratory factor analysis indicated that a single factor 

could account for 60% of the total variance in the ACS indicators across school neighborhoods. As 
9

such, the ACS indicators were standardized and combined into a single Poverty and Disadvantage 

Index (PDI) that demonstrated adequate internal consistency (Cronbach’s  = .91). As evidence of 

validity, PDI scores at the census tract level accounted for 49% of the total variance in 2018 Iowa 

Reading Assessment scores when aggregated to the building level. 

School-Level Data 

 

Building-level data were provided by DMPS as part of the certified enrollment information 

collected each year. In addition to racial/ethnic decomposition, indices at the school level included 

percentage of students eligible for free or reduced lunch, percentage of students receiving special 

education services, and percentage of ELL students. To avoid small cell issues due to crossing 

ethnicities within schools by intervention status, race/ethnicity categories were essentially collapsed 

into a percentage of non-White students index. Preliminary analyses indicated reasonably consistent 

but relatively small relationships between state assessment scores and percentage of ELL, Special 

Education, and Free/Reduced Lunch students. Because ethnicity, at the school level, did not 

consistently related to assessment scores, the index was excluded from subsequent analyses. 

AmeriCorps Member Data 

AmeriCorps staff provided data related to service loads for each AmeriCorps member assigned to 

an AC service school. Primary information included the number of hours per week and months per 

year each member provided service. Information was also provided to index the specific grades AC 

members served within their assigned school. In addition, indicators were created to reflect whether 

members provided service in a one-on-one or group format and the primary method used (rosters vs. 

Lexia flagging) to identify students in need of AC member services. Finally, counts of the total number 

of interventions delivered and the total number of unique students served were also provided. 

To gather contextual insights about the AmeriCorps tutoring program implementation, AmeriCorps 

members were asked to (1) describe methods to select students used at their schools (i.e., through 

Lexia flags and/or roster provided by school staff) and (2) share their experiences and perspectives as 

Lexia tutors regarding potential impacts on students learning to read. A short survey was developed 

and administered to the 2017-2018 AmeriCorps members as part of building the AmeriCorps program 

process evaluation capacity. Hence, for the purpose of this evaluation this survey was utilized to 

provide immediate feedback and to inform appropriate choices for potential improvements that could 

be made during the 2018-2019 school year and in the future. The survey consisted of five open-ended 

questions. The survey instrument is provided in Appendix B. 

The survey was administered online via the Quatrics survey software platform during the second 

third of both the fall (October 2018) and spring (April 2019) semesters. At the beginning of the school 

year 21 members were recruited and assigned to the DMPS elementary schools. Two members left 

shortly after being assigned. One of these two member assignments was replaced. Three other 

members left later. The initial survey administered during the second third of the fall semester was sent 

to 18 members, with 16 members (89%) responding. At the second administration, close to the end of 

the school year, the 2018-2019 AmeriCorps cohort consisted of 17 members, all of whom completed 

the second survey. 

Survey data were analyzed in an aggregate format employing data analysis methods described 

below. The generated results were presented in a briefing format (see Appendix C and Appendix D) 
10

and shared with the AmeriCorps staff and members during their regular monthly meetings, utilizing 

the stakeholder participatory strategy (Mertens & Wilson, 2012). The intent was to engage the 

AmeriCorps members in making sense of their group overall perspectives, and to collectively reflect 

on potential solutions, strategies that could be used to address issues, and concerns shared individually 

when completing the survey. Consistent with the literature in stakeholder engagement in evaluation, it 

could be argued that the survey results discussion contributed to the AmeriCorps staff members’ 

efforts to build a sense of community and gain appreciation of evaluation as the tool for improvement 

among the AmeriCorps members (Chen, 2015). 

Student-Level and Outcomes Data 

 

Student data were provided by DMPS for each individual (N = 15,710) in the evaluation study. 

Although some students were missing data for various measures (e.g., attendance, ELL status, etc.) the 

amount of missing data was relatively small. Demographic measures included race/ethnicity, gender, 

ELL status, IEP level, home language, Section 504 status, gifted and talented status, indices of 

behavior problems and attendance, as well as an indicator of ‘at-risk’ status. Preliminary analyses 

indicated expected small cells within grade in each school for many of the demographic measures. In 

addition to small cells for some indicators, others, including Title I reading services and Early 

Intervention Service status were not observed in the analysis sample. Preliminary analyses did not 

reveal consistent relationships between previous year assessment scores and ethnicity (non-White) or 

gender. Much more consistent moderate associations were observed between assessment scores and 

home language (English or not), free/reduced lunch eligibility, ELL status, IEP level, and attendance 

(days attended / days enrolled). 

As shown in Table 3 below, AmeriCorps members intervened with substantial numbers of students 

in each grade. However, it is also clear that intervention efforts were more concentrated in the earlier 

grades with approximately 28% of all DMPS kindergarteners receiving intervention services but only 

approximately 6% of district 5th graders receiving AmeriCorps support. 

Table 3. Conditions by Grade

Primary student outcomes consisted of standardized reading assessments administered three times 

during the school year. For K-1st students, FAST assessments were administered in early fall (9/17-

9/28), winter (01/22-2/01) and spring (5/06-5/17). Students in 2nd-5th grades completed MAP 

assessments at similar intervals in fall (9/04-9/14), winter (12/03-12/14) and the end of the academic 

year (4/22-5/03). Although the evaluation plan intended to supplement analyses of the FAST/MAP 

assessments, Iowa state assessment scores were not available before the timing of this report (expected 

release in mid-October 2019). In addition to assessment scores for outcomes analysis, DMPS also 

provided assessment scores for use as statistical controls. In April 2019, English Language Learners 

completed the English Language Proficiency Assessment for the 21st Century (ELPA21). 

Unfortunately, this assessment was only available for 3,161 students (20.1%) district wide (and 
11

obviously related to ELL status), so ELPA21 scores were not used in subsequent analyses. Finally, 

where available, DMPS provided scores from the previous year for both the Iowa Reading Assessment 

and FAST/MAP assessments obtained at year-end. Scores on both measures were highly correlated 

with each other and highly correlated to MAP/FAST scores obtained in fall of 2018. 

As shown in Table 4 below, current kindergarten students did not complete assessments the prior 

year, limiting the ability to include baseline proficiency in the analysis. Current 1st and 2nd graders did 

complete FAST assessments the previous spring. Similarly, current 3rd graders had completed a 

previous MAP assessment at the end of their 2nd grade year but had not yet completed the Iowa 

Reading Assessment. Finally, current 4th and 5th graders had completed both the MAP and Iowa 

Assessment the previous year. Consistent with the pattern of assessments available for statistical 

control, outcomes for current K-1st graders included scores on the FAST assessment, whereas students 

in 2nd-5th grade all completed MAP assessments during the 2018-2019 year. 

Table 4. Assessments by Grade 

General Analytic Approach

Quantitative Methods 

A first consideration with regard to analysis involved the differential patterns of assessments depict 

in Table 4. In addition to complications inherent in trying to establish comparable scales across grades, 

use, and availability of different assessments across the student cohorts presented analytic challenges. 

Rather than employing methods to standardize assessment scores across grades and including grade as 

a predictor (interaction effect) in analytic models, students within each grade were analyzed separately. 

In addition to reducing model complexity, such an approach allows for a more fine-grained 

examination of potential differential influences of AC member support across students at different 

proficiency development stages. As indicated above, preliminary analyses of contextual, school, and 

student characteristics revealed reasonably consistent relationships with assessment scores obtained the 

previous academic year. Attempts to match students via propensity or other matching methods were 

not generally successful due to small cells in some schools, relatively few observations with a 
12

particular characteristic, and the need to match both within grade and within school (or schools that 

were matched pairs). For example, in the largest cohort of students that received AC intervention 

services (kindergarteners), cells in some schools fell below 10 students in either the intervention (e.g., 

Greenwood = 8; Stowe = 4) or comparison (e.g., Howe = 4; McKinley = 6) conditions. In the absence 

of successful matching opportunities, the general analysis, instead, included important factors as 

statistical controls. 

The general modeling strategy remained consistent with the analysis plan proposed in that repeated 

assessments on the FAST/MAP were modeled as a function of time in a typical growth curve approach 

(see Appendix E). Based on preliminary findings regarding the trajectory of reading proficiency 

growth over time in a baseline model without statistical controls, the function of time was included as 

both a linear and quadratic effect. As fixed effects, these two components of time most appropriately 

captured growth as a linear increase that flattened out (reached asymptote) at year end. Due to the 

availability of only three assessments, the random components of the model allowed for only the 

intercept (starting point) and linear rate of change to vary across individuals. While the quadratic 

component was included in the models as a fixed effect (with corresponding interactions), the rate of 

quadratic change (flattening) was considered stable across individuals. At the school level, both 

intercepts and linear slopes were allowed to vary, while the quadratic component was not. Somewhat 

surprisingly, significant variability in the school-level intercept and slope was not always statistically 

significant. Although more complicated than necessary in some situations, inclusion of both random 

effects at the school-level did provide appropriate control for school-level nesting and was therefore 

retained. Finally, control variables at both the individual and school level were included based on 

preliminary findings. Once again, these control variables were not always statistically significant 

predictors of FAST/MAP growth, but they were consistently included in models to allow for consistent 

interpretation of findings across all modeling results. 

Qualitative Methods 

The qualitative data from the AmeriCorps member survey (i.e., members’ responses to the survey 

four open-ended questions) were analyzed by identifying common themes and providing representative 

quotations (i.e., the members’ verbatim statements) that support themes (Saldana & Omasta, 20187). 

While the AmeriCorps member survey design was qualitatively driven aimed at providing members 

with the opportunity to reflect on and describe their experiences as Lexia tutors at the beginning and at 

the end of the school year, the members’ feedback about the methods to select students through Lexia 

flags and/or rosters provided by school staff, were summarized using frequencies. 

RESULTS 

Evaluation Question 1: (Does semester/academic year growth among students who receive 

AmeriCorps member intervention differ systematically from 

semester/academic year growth among students who do not receive 

AmeriCorps intervention service?

Evaluation Question 1A

Do students receiving AmeriCorps intervention services demonstrate growth in reading 

proficiency that differs from students who do not receive intervention services? 
13

To address the first evaluation question, analyses were conducted comparing yearly growth of 

FAST/MAP scores across students who did or did not receive AmeriCorps intervention services. 

Importantly, students in the comparison group were limited to those students attending the same 

schools as students who received AmeriCorps services. As discussed above, matching methods proved 

difficult to implement with a high degree of accuracy and without extensive loss of cases (e.g., 

matching 7 interventions students to 80 non-intervention students), therefore, use of statistical controls 

was employed in attempts to mitigate influences due to baseline differences in proficiency between the 

two groups of students. Of primary interest in these analyses is whether the treatment indicator (Group) 

influences the growth intercept (fall 2018 score) or interacts with time to demonstrate differential rates 

of growth related to the need for AmeriCorps service Results are presented separately for each grade 

below. 

Kindergarteners

Across the 21 AC schools, no intervention service was provided to kindergarten students in four 

buildings (Phillips, Willard, Windsor, or Wright). These schools were eliminated from analysis, 

resulting in 17 schools that included intervention cases. Across schools, approximately 725 students 

received interventions and had scores on the 2018-2019 FAST assessment; compared to approximately 

470 students who did not receive AC intervention services in the same schools. In the absence of 

statistical controls via assessment scores from the prior year, baseline equivalence on the fall 2018 

assessment was not tenable. Perhaps not surprisingly, because intervention services were already 

available prior to assessment, those students who received AC intervention services demonstrated 

significantly lower reading proficiency in early fall than those who would not require services 

(Cohen’s d = 0.51). Interestingly, follow-up analyses indicated significantly baseline differences in 

only 7 of the 17 AC schools. 

Among Kindergarteners, results revealed the expected linear effect of time and squared time 

indicating growth but flattening over the academic year. The significant effect of Group indicated a 

significantly lower average starting point (2.28 points) among students who received AC interventions 

throughout the year. AC students also finished the year in spring significantly lower (2.35 points) on 

average than did students who would not require intervention services. Although these findings might 

be expected, results did not indicate moderation of the linear and non-linear time components. Lack of 

group moderation of the time effects indicates that AC service is keeping the trajectory of growth 

among AC students consistent with non-AC students (see Figure 1 and Table 5). That is, students who 

need AC intervention service start and end the year with lower proficiency than those who do not need 

services, but students receiving services demonstrate proficiency growth that is comparable to other 

kindergarteners, suggesting that students receiving intervention services are not falling further behind 

their peers. 
14

Figure 1. AC Contact vs. No Contact in AC Schools (K)

1

st Graders

First grade students completed the FAST at the end of their kindergarten year. Analysis of baseline 

scores still indicated a significant difference between students who would receive AC services and 

those who would not, but the magnitude of the difference (Cohen’s d = 0.25) suggested a smaller 

disparity in reading proficiency at the start of the current school year. Across the 21 AmeriCorps 

schools, two buildings (Phillips and Willard) did not provide intervention services to 1st grade students 

and were dropped from analysis. Results indicated a similar pattern of growth in that reading 

proficiency increased across the year reaching a plateau by year-end. As shown in Figure 2 (also see 

Table 5), students receiving AC intervention services started fall significantly lower (1.51 points) than 

did students who would not receive intervention services. Group membership interacted with the linear 

time component indicating that 1st graders who received intervention services demonstrated shallower 

growth in proficiency over the year in comparison to students who did not receive AmeriCorps 

services. The lower starting point in fall and slower growth over time culminated average proficiency 

levels that were significantly lower (5.02 points) among service recipients at the end of the school year. 

Figure 2. AC Contact vs. No Contact in AC Schools (1st) 
15

2

nd Graders

Three AmeriCorps schools (Morris, Phillips, and Willard) did not provide intervention services to 

2

nd grade students and were dropped from analysis. Consistent with first grade students, 2nd graders 

demonstrated a small (Cohen’s d = 0.38), but statistically significant decrement in reading proficiency 

on their previous year’s FAST score. As shown in Figure 3 (also see Table 5), 2nd graders also started 

and ended the year with proficiency scores that were significantly lower (2.51 and 3.08 points, 

respectively) as a function of receiving AC intervention services. Interestingly, overall growth among 

2

nd grade students did not demonstrate a quadratic flattening. Consistent with findings from 

kindergarten students, group membership did not interact with time, indicating that those students who 

received AmeriCorps intervention services maintained reading proficiency growth comparable to their 

peers who received no intervention service. 

Figure 3. AC Contact vs. No Contact in AC Schools (2nd) 
16

Table 5. AC Support vs. No Support within AC Schools (K-2nd) 
17

3

rd Graders

Three schools (Brubaker, Jackson, and Willard) did not provide services to 3rd grade students and 

were eliminated from analysis. Consistent with student in earlier grades, those 3rd graders who would 

receive AC services demonstrated significantly lower levels of reading proficiency (Cohen’s d = 0.31) 

at the end of their 2nd grade year, compared to students who would not require intervention services in 

the current year. Analysis results (see Table 6) indicated expected growth across time with both linear 

increase and nonlinear flattening over the school year. Students receiving AmeriCorps services both 

started and ended the year with reading proficiency levels that were significantly lower (2.48 and 2.92 

points, respectively) on average, than their peers who did not require intervention services. However, 

receipt of service did not alter the growth trajectory, indicating that students who received services 

continued to progress at a rate similar to students who did not receive services (see Figure 4). 

Figure 4. AC Contact vs. No Contact in AC Schools (3rd) 

4

th Graders

AmeriCorps services were not provided for 4th grade students in 7 schools (Brubaker, Findley, 

Howe, Jackson, Morris, Willard, and Wright) that were removed from analysis. Among 4th grade 

students, pervious-year MAP and Iowa assessment scores demonstrated significant, and moderate 

(Cohen’s d = 0.46 and d = 0.51, respectively) decrements as a function of eventual AC service receipt. 

The consistent pattern of increasing and flattening growth was observed (see Table 6), as was a 

significant average difference (1.79 points) in fall MAP scores. However, interactions between group 

membership and the linear (p < .05) and nonlinear (p = .07) components of time (see Table 6) suggest 

differential growth in reading proficiency as a function of AC service receipt. Specifically, those 

students receiving intervention services gained significantly more (2.31 points per assessment) than did 

those students who did not receive services. Although the quadratic interaction suggests a more rapid 

flattening of growth among AC service recipients, students receiving services finished the year only 

slightly lower (0.73 points) than did those receiving no services. Importantly, as shown in Figure 5, the 

nonsignificant difference in spring suggests that, even though service receivers started the year lower 

than their peers, those who received AC intervention support caught up to their peers by year-end. 
18

Figure 5. AC Contact vs. No Contact in AC Schools (4th) 

5

th Graders

AmeriCorps intervention services were not provided to 5th grade students in 9 (Brubaker, Capitol 

View, Greenwood, Howe, Jackson, Morris, River Woods, Walnut Street, and Willard) of the 17 

schools with AmeriCorps members. Within the remaining eight schools, the number of students that 

received AC intervention services was also small (approximately 145) in comparison to students in 

earlier grades. Consistent with 4th graders, 5th grade students scored significantly lower in reading 

proficiency at the end of the previous year on both the MAP and Iowa assessments (Cohen’s d = 0.47 

and d = 0.48, respectively). Among 5th grade students, only the linear effect of time was statistically 

significant, indicating continued growth across the academic year. Students receiving AmeriCorps 

services, once again, started the year significantly lower (2.11 points) than comparison students. 

Although the trajectories shown in Figure 6 seem to suggest differential growth due to AmeriCorps 

intervention status, lack of statistically significant interactions between group and time indicate that 

students who received intervention services did demonstrate growth that is comparable to students who 

did not receive intervention services over the academic year. Consistent with a lower starting point in 

fall and parallel growth across the school year, students who received AmeriCorps serviced did end the 

year with significantly lower (1.78 points) levels of reading proficiency than did their peers who did 

not receive AmeriCorps services. 
19

Figure 6. AC Contact vs. No Contact in AC Schools (5th) 
20

Table 6. AC Contact vs. No Exposure within AC Schools (3rd-5th) 
21

Evaluation Question 1A: General Findings

 

Although all grades demonstrated initial differences in reading proficiency, whether via fall FAST 

assessments for kindergarteners or via previous year MAP/IA assessment measures, the general pattern 

of findings with regard to AmeriCorps service provision was reasonably consistent across grades. 

Regardless of condition or grade, all students demonstrated yearly gain in reading proficiency scores. 

In most cases, the pattern of that gain reflected a linear increase that flattened out by year’s end. 

Among kindergarteners, and students in 2nd, 4th, and 5th grades, the receipt of AmeriCorps intervention 

services did not alleviate initial decrements in reading proficiency but receipt of service did result in 

rates of growth that did not differ from students who did not receive AmeriCorps services. These 

findings suggest a maintenance effect in which initial lower performance does not decrease further 

over the course of the school year. In two grades, receipt of intervention services did alter the rate of 

growth in reading proficiency over time. However, the impacts appear to have occurred in different 

directions. Specifically, 1st grade students who received AmeriCorps services demonstrated a slowed 

rate of growth relative to their peers, whereas 3rd grade students who received intervention services 

appear to catch up to their peers by the end of the year with regard to reading proficiency. 

Evaluation Question 1B

Do students receiving AmeriCorps intervention services demonstrate reading proficiency 

growth that differs from same-grade students who do not receive intervention services but 

attend schools where AmeriCorps services are provided to other grades? 

To address Question 1B, analyses compared FAST/MAP growth among students who received 

AmeriCorps intervention services to FAST/MAP growth among same-grade students who attended an 

AC school but where services were not available in their grade. The comparison group was limited to 

students who attended schools with an AmeriCorps member assigned, but where the AC member did 

not provide services to a particular grade. For example, 1st grade students receiving AmeriCorps 

intervention services in AC schools were compared to 1st grade students in other AC schools where 

services were not provided to 1st grade students. Due to difficulties described above regarding 

matching methods, statistical controls were again employed to mitigate influences due to both 

individual- and building-level factors. Of primary interest in these analyses is whether the treatment 

indicator (Grade) influences the growth intercept (fall 2018 score) or interacts with time to demonstrate 

differential rates of growth related to the presence of an AmeriCorps service provider, in the absence of 

actual service provision. Results are presented separately for each grade below. 

Kindergarteners

Across all AC schools, four buildings (Phillips, Willard, Windsor, and Wright) provided AC 

services but did not intervene with kindergarten students. Baseline reading proficiency among 

kindergartners based on fall 2018 FAST scores indicated approximate equivalence (Cohen’s d = 0.15) 

among students in schools that provided services (approximately 1200) and students in schools that did 

not (approximately 220). As shown in Figure 7 below (also see Table 7), kindergarten students 

demonstrated significant linear growth over time that did not flatten by year-end. Actual provision of 

service by AmeriCorps members did not influence initial proficiency, the rate of proficiency growth, or 

final levels of proficiency at the end of the school year. 
22

Figure 7. AC Service vs. No AC Service in AC Member Schools (K) 

1

st Graders

Among AC schools, two buildings (Phillips, Willard) provided AC services but did not intervene 

with 1st grade students. Students in schools providing intervention services (approximately 1260) did 

not differ (Cohen’s d = 0.07) in average levels of reading proficiency from students (approximately 

100) in AC schools that did not provide 1st grade intervention services based on spring FAST scores 

during the previous year. Although students receiving AC services started the year slightly lower (2.00 

points) on average than did students in non-service schools, the difference was not statistically 

significant. Linear and nonlinear time effects indicated growth and leveling over the academic year. 

The marginal time by grade interaction suggests slightly slower growth among students receiving AC 

services in comparison to students in AC schools that did not provide service to 1st grade students. As 

shown in Figure 8 below (also see Table 7), the decreased growth among students in schools that 

provided services resulted in average reading proficiency levels that were significantly lower (9.01 

points) than levels for students in AC schools that did not provide 1st grade interventions. 

Figure 8. AC Service vs. No AC Service in AC Member Schools (1st) 

 
23

2

nd Graders

Three schools (Morris, Phillips, Willard) included an AmeriCorps member who did not provide 

intervention services with 2nd grade students. Average FAST scores in spring of 2018 were 

approximately equal (Cohen’s d = 0.10) between 2nd graders who attended AC schools and received 

interventions and those attending AC schools without AC service in 2nd grade. In fall 2018, those 

receiving services did not differ in average MAP scores (-0.23 points) from those in AC schools 

without 2nd grade service. Only a linear effect of time was observed, indicating overall growth across 

the academic year. Lack of time by grade interactions (see Table 7) indicates that students receiving 

AC services progressed at a rate comparable to students in AC schools that did not provide 2nd grade 

services (see Figure 9). Approximately equal starting points, coupled with parallel growth resulted in 

nearly equivalent average MAP scores at year-end. 

Figure 9. AC Service vs. No AC Service in AC Member Schools (2nd) 
24

Table 7. AC Contact by Grade in Schools with AC Members (K-2)
25

3

rd Graders

Three schools (Brubaker, Jackson, Willard) provided AmeriCorps services but did not intervene 

with 3rd grade students. Third grade students did not differ in average MAP scores at the end of 2018 

(Cohen’s d = 0.07). Students in AC schools who received intervention services began the year slightly, 

but not significantly, lower (0.82 points) than students in AC schools that did not provide 3rd grade 

services. The significant linear effect of time indicated growth across the academic year. Absence of a 

linear time by grade interaction (see Table 8) indicated approximately equal growth between students 

receiving AC services and students in schools where services were not provided in 3rd grade. The 

marginal quadratic time by grade interaction suggests a slightly greater degree of slowing in growth 

among students receiving services (see Figure 10), but average MAP scores did not differ at the end of 

the school year. 

Figure 10. AC Service vs. No AC Service in AC Member Schools (3rd) 

4

th Graders

Seven schools (Brubaker, Findley, Howe, Jackson, Morris, Willard, and Wright) provided 

AmeriCorps services but did not intervene with 4TH grade students. Average reading scores on both 

the MAP (Cohen’s d = 0.07) and IA Assessment (Cohen’s d = 0.01) in spring 2018 did not differ 

between students receiving intervention services and students in non-service AC schools. A linear 

effect of time indicated growth over the academic year with only a slight flattening by year’s end. 

Neither of the time by service interactions were significant (see Table 8), indicating parallel growth 

between those who received services and those who did not. However, as shown in Figure 11, students 

in AC schools that received intervention services did finish the year marginally lower (-2.32 points) 

than did students in AC schools that did not provide services in 4th grade. 
26

Figure 11. AC Service vs. No AC Service in AC Member Schools (4th) 

 

5

th Graders

Eight schools (Brubaker, Greenwood, Howe, Jackson, Morris, River Woods, Walnut, Willard) 

provided AmeriCorps services but did not intervene with 5th grade students. Average scores on both 

Iowa and MAP assessments were nearly identical in spring 2018 (Cohen’s d = 0.06 and d = 0.04, 

respectively). Students in AC schools that provided services to 5th grade started the year slightly (-0.82 

points), but not significantly lower than students in AC schools without 5th grade services. Consistent 

with earlier grades, significant linear growth and slight flattening over the course of the year was 

observed (see Table 8). Once again, results indicated no time by service interactions indicating 

approximately parallel growth across 5th students who did receive AC services and those who did not 

receive services in schools where AC members were present. Consistent with equal starting points and 

parallel growth, Figure 12 shows that 5th grade students in AC schools that provided services finished 

the year only slightly lower (-0.95 points) on average than did students in AC schools where no 

services were provided in 5th grade. 

Figure 12. AC Service vs. No AC Service in AC Member Schools (5th) 
27

Table 8. AC Contact by Grade in Schools with AC Members (3rd-5th) 
28

Evaluation Question 1B: General Findings

Across grades, results were generally consistent with regard to whether AC member service 

influences reading proficiency relative to AC member presence. In kindergarten, 2nd, 3rd, and 5th grade, 

students receiving AC intervention services demonstrated growth parallel to students in the same 

grades who did not receive AC services, even though an AmeriCorps member was present in the 

building. Although not statistically significant for 4th graders, both 1st and 4th graders who received 

AmeriCorps intervention services demonstrated decreased growth relative to same-grade students in 

schools where an AC member was present, but did not provide services in those grades. Collectively, 

these findings suggest that actual service provision does not result in differential growth (for most 

grades) relative to the mere presence of an AmeriCorps member providing services to other grades in 

the building. One possible explanation for the lack of findings specific to intervention delivery 

involves the process/mechanism by which AC members are assigned to work with grades in particular 

schools. For example, students in a particular school might be in need of intervention support in certain 

grades, while students in other grades in the same school do not demonstrate need for services. 

Unfortunately, information relative to grade assignment or service allocation within buildings was not 

available for this evaluation. 

Evaluation Question 2: Does semester/academic year growth among students differ systematically as a 

function of how much AmeriCorps service students receive? 

Evaluation Question 2A

Does reading proficiency growth differ as a function of intervention frequency among 

students who receive AmeriCorps intervention services? 

To address Question 2A, analyses were limited only to students within AC schools, and only to 

students who received at least one intervention service within each grade. Number of interventions 

received ranged from a single occurrence to 115 sessions (M = 6.77, SD = 11.35). Due to the severe 

skew in the number of interventions received, counts were transformed (natural log) for analysis. 

Primary focus in these analyses involved the influence of the number of interventions (INT in Table 9) 

as a predictor of initial reading proficiency in fall 2018, as well as a moderator of the growth trajectory 

components (time and time squared). Influences due to the number of interventions would reflect 

differential starting points or altered growth in reading proficiency as a function of increasing 

frequency of intervention contact. 

Kindergartners

Among kindergarteners, growth in reading proficiency over time followed the expected linear 

increase and flattening pattern consistent with previous findings. The number of AC interventions 

experienced did relate negatively to the first FAST assessment in fall 2018, such that each additional 

intervention received corresponded to lower average FAST scores. Number of AC interventions also 

interacted with both the linear and non-linear components of time but in opposite signs (see Table 9). 

As the number of interventions increased, linear growth slowed but quadratic growth increased 

resulting in a ‘catch-up’ effect. Although students receiving multiple interventions finished the year 

with significantly lower average FAST scores relative to students receiving only a single intervention, 

the difference in spring 2019 FAST scores decreased as students received more AC intervention 

support (see Figure 13). 
29

Figure 13. Growth by Number of AC Interventions Received (K) 

1

st Graders

Results among 1st graders were reasonably consistent with regard to growth over time. Once again, 

the number of AC interventions related negatively to fall 2018 FAST scores. Although the interaction 

between time and number of interventions received suggested a similar catching up pattern (see Figure 

14), neither the linear nor the quadratic interaction achieved statistical significance (see Table 9). 

Finally, the number of interventions received did continue to relate negatively and significantly to 

average FAST scores in spring 2019, such that those receiving more interventions finished the year 

lower on average than did students receiving fewer interventions. 

Figure 14. Growth by Number of AC Interventions Received (1st) 
30

2

nd Graders

Growth in reading proficiency among 2nd graders only demonstrated a linear increase over time 

without a significant flattening at the end of the year. Number of interventions received related 

negatively to MAP scores at the beginning of the year. Although neither of the time by intervention 

count interactions was statistically significant (see Table 9), the effect estimates (i.e., negative linear 

interaction and positive quadratic interaction) were consistent with a maintenance pattern (see Figure 

15), such that greater invention appears to reduce proficiency disparities. Although those students 

receiving multiple interventions appear to be maintaining, average MAP scores at the end of the year 

were significantly lower among those receiving greater intervention support. 

Figure 15. Growth by Number of AC Interventions Received (2nd)
31

Table 9. Growth by Number of AC Interventions Received (K-2)
32

3

rd Graders

Results for 3rd graders were strikingly consistent with those for younger students, in that the growth 

in reading proficiency increased and leveled out during the year. Number of interventions received 

corresponded with lower average MAP scores in fall 2018. Once again, the interactions with time were 

not statistically significant (see Table 10) but the signs of the coefficients were consistent with the 

maintenance pattern discussed above (see Figure 16). Although patterns do suggest some recovery 

with greater intervention service, average MAP scores were still significantly lower at the end of the 

year among students who continued to receive AC support. 

Figure 16. Growth by Number of AC Interventions Received (3rd) 

4

th Graders

Among 4th graders, the trajectory of growth in MAP scores was consistent with previous findings. 

However, the number of interventions received did not relate to average MAP scores early in the 

school year. Once again, the interactions with time were consistent with maintenance effects but not 

statistically significant (see Table 10). The marginally significant interaction between the number of 

interventions received and the linear component of time suggests a larger disparity in MAP scores at 

the end of the year. As shown in Figure 17, this marginal interaction did result in a small, but 

statistically significant, difference in average MAP scores on the spring 2019 assessment (see Figure 

17). 
33

Figure 17. Growth by Number of AC Interventions Received (4th)

5

th Graders

Students in 5th grade demonstrated only a linear growth component over time. Neither the number 

of interventions received, nor the interactions with time were statistically significant (see Table 10). 

Although Figure 18 appears consistent with the pattern of maintenance seen in the earlier grades, the 

coefficients related to the interaction terms suggest a reversed pattern. Specifically, the positive 

interaction with time suggests a steeper rate of growth per number of interventions received, while the 

negative quadratic interaction suggests a faster flattening in growth as the number of interventions 

increases. Importantly, the average difference in MAP scores at the end of the year as a function of 

increased intervention contact was statistically significant but the magnitude of the difference at yearend appears to be reducing as a function of AC supports provided (see Figure 18). 

Figure 18. Growth by Number of AC Interventions Received (5th)
34

Table 10. Growth by Number of AC Interventions Received (3rd-5

th)
35

Evaluation Question 2A: General Findings 

In each of the earlier grades (K – 3

rd), the number of AmeriCorps interventions received related 

significantly to initial levels of reading proficiency in fall 2018. This finding is not terribly surprising 

in that intervention services were being provided for students before the fall assessment. Interestingly, 

the number of interventions received did not relate to fall assessment scores for students in the later 

grades (4th and 5th). Although not consistently statistically significant, interactions between the number 

of interventions received and time were also very similar across K-4 students. In each grade, the 

number of interventions related negatively with time, but positively with the quadratic component of 

time, suggesting a pattern in with greater intervention contact appears to reduce disparity in reading 

proficiency. Importantly, greater intervention frequency was consistently related to lower reading 

proficiency at year-end, but the magnitude of the difference in proficiency scores appears to shrink 

slightly as a function of continued intervention service. Contrary to this general pattern, 5th grade 

students demonstrated reversed relationships between service frequency and growth, such that increase 

intervention contact appears to continue flattening the trajectory of reading proficiency growth. 

Evaluation Question 2B 

Do AmeriCorps services provide added benefits in schools with service members in 

comparison to non-AmeriCorps schools with the same existing services? 

A first step in addressing Question 2B involved identifying schools that were as similar as possible 

with the exception of an AmeriCorps member. Information provided regarding services/initiatives in 

non-AmeriCorps schools were limited to indicators of Title I funding and whether the school was a 

School for Rigor. All of the DMPS schools were Schools of Rigor and the majority were Title I 

funded. Importantly, data provided by DMPS indicated that no students within schools were receiving 

Title I reading support. Finally, as indicated earlier, all DMPS schools were using the Lexia Core5 

program during the 2018-2019 year. Due to somewhat limited information regarding other readingrelated supports at non-AmeriCorps schools, schools were matched on indices that attempted to equate 

specific building composition factors as well as neighborhood poverty and disadvantage. Because of 

the small number of schools, school characteristics described above were combined to obtain 

Mahalanobis distances reflecting an aggregate distance of each school from the ‘middle’ of all schools. 

Mahalanobis distance and PDI scores based on ACS data (see above), were used to propensity score 

match pairs of schools where one school provided AmeriCorps services and the other did not. 

Table 11 provides propensity and distance information for each AmeriCorps school and 

corresponding matched non-AmeriCorps school for those buildings that could be closely matched. As 

shown in Table 12, matched buildings did not differ on the PDI or Mahalanobis distance measure, as 

expected, based on the propensity matching criteria. However, matched schools also did not differ in 

average school-level scores on the MAP, FAST, or Iowa assessments obtained for students in the prior 

academic year. Although based on limited service information, these results suggest that the matching 

approach did result in building pairs that were approximately comparable with regard to the primary 

student outcomes of interest. Of primary interest in the analyses reported below is whether 

AmeriCorps services at the building level (AC in Table 13) relate to initial levels of reading 

proficiency or moderate the growth in reading proficiency over time in relation to initial proficiency 

and growth in matched buildings that do not have an AmeriCorps member in the building. In all 

analyses, if AmeriCorps services were not provided within the target grade, both the AmeriCorps 

school and its matched non-AmeriCorps school were eliminated from the analysis. 
36

Table 11. Matching AC and Non-AC Schools

Table 12. Baseline Equivalence across Matched Schools

Kindergartners

Among kindergarteners, average FAST scores did not differ significantly across AC and non-AC 

schools. Linear and quadratic effects of time indicated growth that flattened out over the academic 

year. Lack of a significant linear time by AC interaction indicated approximately equal growth over the 

year but a statistically significant quadratic interaction with AC status suggests greater slowing of 

growth within the AC schools (see Table 13). Importantly, as shown in Figure 19, even with greater 

decline in growth over time, kindergarten students in AC schools did not differ significantly (-0.23 

points) in average FAST scores from kindergarteners in non-AC schools. 
37

Figure 19. AC Schools vs. Non-AC Schools (K) 

1

st Graders

Average FAST scores also did not differ at the beginning of the year between students in AC 

schools and students in Non-AC schools among 1st graders. Consistent with the general growth 

trajectory observed earlier, linear and nonlinear effects indicated growth and leveling off over the 

academic year. Lack of a linear by AC status interaction indicated parallel growth over time but the 

significant nonlinear time by AC status interaction suggests more rapid flattening of growth among 1st

graders in AC schools (see Table 13). More rapid flattening did correspond with slightly lower (3.73 

points) average FAST scores at the end of the year among students in AC schools (see Figure 20), but 

the difference was not statistically significant. 

Figure 20. AC Schools vs. Non-AC Schools (1st) 
38

2

nd Graders

Second grade students did not differ in average MAP assessment scores at the beginning of the 

year. Linear time effects indicated growth over the year but no evidence of flattening in the absence of 

a quadratic time component (see Table 13). Neither of the time by AC status interactions were 

significant, indicating nearly identical growth trajectories in MAP scores across AC and Non-AC 

schools. As shown in Figure 21, consistent with approximately equal starting points and nearly 

identical growth, average MAP scores at year-end did not differ between students in AC schools and 

those in non-AC schools. 

Figure 21. AC Schools vs. Non-AC Schools (2nd) 
39

Table 13. AC Service vs. No AC Service in Matched Schools (K-2nd) 
40

3

rd Graders

Although not statistically significant, 3rd grade students in AC schools started the year with slightly 

higher (1.18 points) average MAP scores than did students in non-AC schools. Linear and nonlinear 

effects of time indicate the typical growth and leveling off pattern over the academic year. Interactions 

between AC status and both components of time suggest differential effects (see Table 14). First, the 

linear interaction with AC status indicates a significantly slower rate of growth among students in AC 

schools, relative to students in Non-AC schools. However, the nonlinear interaction with AC status 

indicates that 3rd grade students in AC schools do not level off as fast as do students in Non-AC 

schools. As shown in Figure 22, this decrease in the rate at which growth slows overcomes the slower 

rate of growth over time, such that 3rd grade students in AC schools finish the year with slightly, but 

not significantly, higher average (0.50 points) MAP scores than students in Non-AC schools. 

Figure 22. AC Schools vs. Non-AC Schools (3rd) 

4

th Graders

Among 4th grade students, average MAP scores at the beginning of the year did not differ across 

AC and Non-AC schools. Linear and quadratic time effects demonstrate growth and leveling off over 

the academic year. Neither time component interacted with AC status (see Table 14) indicating nearly 

identical growth trajectories over time culminating in approximately equal (1.06 point difference) 

average MAP scores by the end of the year. Although not statistically significant, 4th grade students in 

AC schools demonstrated slightly higher average MAP scores than did students in Non-AC schools at 

the end of the school year.
41

Figure 23. AC Schools vs. Non-AC Schools (4th) 

5

th Graders

Fifth grade students in AC schools started the year with slightly, but not significantly lower 

average (1.33 points) MAP scores than did students in Non-AC schools. Time effects indicated linear 

growth but no significant slowing over the year. Interactions between time components and AC status 

mirrored those observed among 3rd graders in that the linear growth in MAP scores was slower for 5th

grade students in AC schools than 5th grade students in Non-AC schools. However, the marginally 

significant quadratic interaction reflected less reduction in growth among students in the AC schools 

(see Table 14). This reduced slowing, as shown in Figure 24, resulted in AC students still scoring 

lower on average (3.22 points) at year-end than Non-AC students, but the difference was not 

statistically significant. 

Figure 24. AC Schools vs. Non-AC Schools (5th) 
42

Table 14. AC Service vs. No AC Service in Matched Schools (3rd-5th) 
43

Evaluation Question 2B: General Findings 

The general pattern of results presented above is consistent in both the lack of relationships 

between AC member status and reading proficiency at both the beginning of the year and over time. In 

cases where AC member presence was related to proficiency growth, the relationship involved the 

quadratic component of time indicating a more rapid decline in linear growth among students in AC 

schools. Even though students in some grades demonstrated this slowing of growth, average levels of 

reading proficiency did not differ significantly across AC and non-AC schools in any grade. Although 

speculative, results for students in 3rd and 4th grade suggest possible gains among students in schools 

served by AmeriCorps members, relative to their peers in non-AmeriCorps schools. In the absence of 

compelling statistical support, clear evidence of an additive benefit of AmeriCorps member presence is 

not available. However, it is important to note that schools largely drive the selection process of 

AmeriCorps. Specifically, of the 11 non-AmeriCorps matched schools in the previous analyses, almost 

half (n = 5; 45.5%) indicated no interest in having an AmeriCorps member in the building. In the 

absence of additional information, a possible reason for lack of apparent additive benefits of member 

presence could reflect the use of alternative supports at non-AmeriCorps schools that could not be used 

in the matching process. 

Evaluation Question 3: Do schools with AmeriCorps members providing services demonstrate higher 

student proficiency and provide increased benefits to students in comparison 

to schools without AmeriCorps service providers? 

Evaluation Question 3A 

Does presence of an AmeriCorps member provide increased benefits to students through 

impact on variability in reading proficiency relative to students in buildings without 

AmeriCorps services? 

To examine whether presence of an AmeriCorps member impacted variability in student reading 

proficiency growth, a school-level model was estimated within each grade. Because the presence of a 

single AmeriCorps member reflects a school-level characteristic, only building level covariates were 

included in the model. Student assessments on either the FAST (K and 1st) or the MAP (grades 3-5) 

were modeled with a linear growth curve. The student-level model excluded the quadratic effect for 

time for simplicity, but also because only the intercept and linear growth terms were allowed to vary 

across schools. Building-level covariates included total enrollment, Title I funding, percentage of 

Asian, African American, Hispanic/Latino, Native American, Mixed Race, and Pacific Islander 

students, as well as percentage of ELL and special education students, and the percentage of students 

eligible for free/reduced lunch. Other possible controls, including use of Lexia Core 5 and 

identification as a school for rigor were constants across the DMPS schools. Analyses were conducted 

within grade level and only included AmeriCorps schools serving the corresponding grade. 

Comparison schools reflected all non-AmeriCorps schools but, again, only included students in the 

corresponding grade within each analysis. 

The baseline model included the growth model with random components for intercepts and linear 

slopes at both the student and building level. The comparison model added both the main effect of AC 

Member presence (i.e., prediction of the growth intercept), and the interaction between AC Member 

presence and linear time (i.e., prediction of the growth slope). Because interest centered on buildinglevel influences, no student-level covariates or controls were included in the models. As such, 

reductions in variability were not expected for the student-level intercept and slope. Given that most of 
44

the variability in growth occurred at the student level, most reductions in growth parameter variance at 

the student level were zero or negative (essentially zero), consistent with the exclusion of student-level 

variables (Singer & Willett, 2003; Kreft & de Leeuw, 1998; Snijders & Bosker, 1999). 

As shown in Table 15, formal deviance tests did not indicate significant reductions in variance at 

the school-level due to presence of an AmeriCorps member within each corresponding grade. 

However, in some cases, member presence did correspond to either a marginal model improvement or 

a non-trivial reduction in growth variability between schools. Aside from the anomalous results among 

4

th grade students, AC member presence corresponded with approximately a 4% reduction in intercept 

variance (i.e., FAST or MAP scores in fall 2019) across buildings. Slope variances were not so 

consistent with relatively little influence in most grades, with the exception of larger reductions in 

building-level variability in grades 1 and 3. 

Table 15. Variance Reduction due to AC Member Presence 

Evaluation Question 3A: General Findings 

 Findings revealed suggestive evidence that availability of AmeriCorps services did slightly 

reduce building-level variability in reading proficiency; however, such influences were generally 

limited to the initial assessments obtained in fall 2018. Consistent with potential issues involving other 

available services described in the general findings above, it is possible that the impact of AmeriCorps 

member presence is muted due to alternative supports available in other non-AmeriCorps schools. In 

addition, and perhaps more importantly, results from the specific analyses for Evaluation Question 3A 

and findings from the previous analyses presented above are consistent in terms of variability in the 

growth of reading proficiency. Although the general modeling strategy consistently included random 

components for the intercept and linear slope at the building level, these components were not always 

statistically significant. While there was non-zero variability at the school level in most analyses, the 

bulk of variance in reading proficiency growth clearly occurs at the individual level. That is, variation 

in starting points and growth occurs largely between individual students, with substantially less 

variability occurring across schools. As such, the findings directly above might be expected if 

individual student growth is facilitated and maintained by different support services within school 

buildings. 
45

Evaluation Question 3B 

Does growth in reading proficiency differ as a function of whether students who are flagged 

for support by the Lexia Core5 program receive intervention services from AmeriCorps 

members? 

This comparison intended to compare reading proficiency growth among students flagged by Lexia 

who received intervention services in AmeriCorps schools and students flagged by Lexia who did not 

receive intervention services in AmeriCorps schools. Of primary importance, this set of planned 

comparisons would equate students on the need for intervention services using the same metric which 

would then allow a rigorous comparison of whether AmeriCorps intervention was successful in 

comparison to students at non-AmeriCorps schools at the same level of support need. Although the 

Lexia Core5 program was in use within all DMPS schools, flagging data were not available for use in 

this evaluation. 

Evaluation Question 4: Do characteristics of AmeriCorps members or the modality of service delivery 

influence the effectiveness of services? 

Evaluation Question 4A 

Do AmeriCorps member hours/intervention loads or methods used for identifying students in 

need of intervention differentially relate to reading proficiency growth? 

AmeriCorps member hours only varied in blocks of 20, 30, 35, and 40 with relatively few members 

working more than 30 hours per week. Although not entirely consistent across all placements, 

members who worked more hours generally did so across a larger number of grades. For example, only 

one 20-hour member (~12%) served all six grades whereas three 40-hour members (75%) did so. 

Similar disparities exist with regard to hours of service in that individual members might provide far 

fewer or far more interventions than other members at the same hourly commitment. For example, one 

35-hour member provided more than 1200 interventions while another 35-hour member provided 

fewer than 400. While this disparity could indicate quicker or more prolonged intervention, it could 

also reflect duration of assignment at a particular school. Alternatively, differential intervention loads 

could reflect greater intervention frequency with fewer students relative to delivering fewer 

interventions to a larger group of students. Given the difficulties inherent in disentangling service time, 

service breadth, and service volume, analysis focused only possible impacts of intervention efficacy 

due to use of different identification methods. 

Although most AmeriCorps members indicated using some combination of identification methods, 

either relying on rosters or relying on flagging within the Lexia program, AmeriCorps staff provided 

information about the most commonly used method where available. Among the 21 DMPS schools 

that included an AmeriCorps member, primary identification method was not available for four 

buildings (Monroe, Walnut, Willard, or Windsor). Of the remaining 17 schools, most (10; 58.8%) 

relied primarily on Lexia Flagging, with fewer (7; 41.2%) relying primarily on roster identification for 

intervention services. Of primary interest in these analyses is whether identification method (Method in 

Table 16) influences initial levels of reading proficiency in fall 2018 or growth in reading proficiency 

across the school year. Findings indicating differences in proficiency or differences in proficiency 

growth would suggest that primary identification method is an important factor to consider in terms of 

intervention efficacy. 
46

Across the earlier grades (K-2nd), AmeriCorps members relied primarily on Lexia flagging 

(approximately 66% vs. 34%) to identify students in need of intervention services. As shown in Table 

16, identification method was not statistically significantly related to initial levels of reading 

proficiency or growth in reading proficiency over time. At the end of the school year, differences in 

average reading proficiency were not statistically significant but also mixed. Specifically, average 

reading proficiency among kindergartners at the end of the year was slightly (3.89 points) higher if 

students were identified via flagging. Alternatively, average reading proficiency was slightly higher 

(2.94 points) among 1st graders if identification relied primarily on rosters. Among 2nd graders, no 

apparent difference in reading proficiency was observed across flagging and roster identification 

methods. 

Table 16. Intervention Efficacy by Identification Method (K-2)

In later grades (3rd-5th), results were generally consistent. Although average levels of reading 

proficiency at the beginning of the year did not differ significantly as a function of identification 

method, the negative association across grades (see Table 17) suggests that students identified 

primarily by Lexia flagging were scoring lower than were students identified via rosters in fall 2018. 

Consistent with the lack of statistically significant associations, identification method did not interact 

with time, indicating that growth in reading proficiency across the school year did not differ whether 

identification relied on flagging or roster methods. Importantly, results in each grade revealed no 

significant differences in average reading proficiency at the end of the school year as a function of 

identification method.
47

Table 17. Intervention Efficacy by Identification Method (3rd-5th)

Evaluation Question 4A: General Findings 

With the exception of a suggestive relationship between lower initial reading proficiency among 

students identified vial Lexia flagging in grades 3-5, findings were generally consistent with regard to 

identification methods. Students across grades did not differ in rate of growth or average levels of 

reading proficiency at the end of the year, suggesting that use of Lexia flagging or rosters for 

intervention identification is equally efficacious. While the results point to no differences based on 

identification method, it is important to consider that most AmeriCorps members did indicate using 

some combination of both approaches. In addition, survey responses by AmeriCorps members (see 

Appendix D) indicate that use of both identification methods actually increased by the end of the 

academic year. Unfortunately, data available for this evaluation only included the primary 

identification method used by each AmeriCorps member. Though speculative, it is possible that 

meaningful differences could emerge if finer delineation were possible to isolate AmeriCorps members 

who relied solely on rosters or solely on Lexia flagging. Of course, the lack of findings could also be 

reflective of accurate identification of students in need of intervention support via both Lexia flagging 

and rosters. As AmeriCorps members indicated in the AmeriCorps survey (see Appendix C), most of 

the students who end up on rosters are also flagged by the Lexia system. 
48

Evaluation Question 4B 

Does modality of intervention delivery differentially relate to growth in reading proficiency? 

To examine potential differences in intervention efficacy as a function of whether students received 

services in single or group formats, AmeriCorps staff provided information corresponding to each AC 

member. Of primary interest is whether interventions occurred in one-to-one or small group settings. 

While each AmeriCorps member likely engaged in both modes of intervention delivery, the primary 

mode of delivery was indexed. Across 20 AmeriCorps schools for which intervention data were 

available, the majority of intervention delivery (n = 13; 65% of schools) occurred in small groups. As a 

preliminary step in the analysis, intervention modality was crossed with identification method (see 

directly above) to avoid singularities that would yield redundant findings. In grades K-2 phi 

coefficients were generally small (.29 - .50) indicating that use of one identification methodology was 

not consistently redundant with mode of intervention delivery. Although identification method and 

delivery mode were not singular, all of the group delivery interventions among kindergartners occurred 

with AmeriCorps members who relied primarily on flagging for identification. 

As shown in Table 18, results across the earlier grades were somewhat mixed with regard to 

delivery modality influences. Delivery method was not significantly related to initial levels of reading 

proficiency in fall 2018 for students in any of the three grades. A significant interaction between 

delivery method and time was observed among kindergarteners indicating slower linear growth in 

proficiency if interventions were primarily delivered in small group settings. Although not statistically 

significant, the estimate of the delivery method by time interaction was consistent among 1st graders, 

suggesting a similar potential decrease in intervention efficacy due to delivery method. The single 

interaction between delivery method and the quadratic component of time among 1st graders suggests a 

recovery of the slight decrease in linear growth across the school year. Although delivery method did 

relate to components of growth in reading proficiency in some cases, no significant differences in 

average reading proficiency were observed at the end of the year across students who received 

individual or group-based intervention services. 
49

Table 18. Intervention Efficacy by Delivery Method (K-2)

 

 

Influences on reading proficiency and growth in proficiency due to modality of intervention 

delivery were not observed among students in 3rd or 4th grade (see Table 19). Delivery of interventions 

via one-on-one or small group modes did not relate to initial levels of reading proficiency in fall 2018, 

linear/nonlinear growth in reading proficiency over time, or final levels of reading proficiency in 

spring 2019 for students in these grades. Results regarding initial levels of reading proficiency were 

consistent among 5th grade students. However, intervention delivery mode did interact with both the 

linear (p = .06) and nonlinear (p = .01) components of time. The negative interaction with the linear 

component of time indicates that 5th grade students who received intervention services in small groups 

demonstrated slower proficiency growth in comparison to 5th grade students who received one-on-one 

intervention services. The positive interaction with the nonlinear component of time indicates that 

those 5th graders who received interventions in group settings were actually more likely to accelerate 

proficiency growth across the school year, such that no average difference in reading proficiency 

remained at year-end between students receiving interventions under different delivery modalities. 
50

Table 19. Intervention Efficacy by Delivery Method (3rd-5th) 

Evaluation Question 4B: General Findings 

 Delivery modality did not demonstrate consistent statistically reliable relationships with 

reading proficiency. Across grades, students receiving AmeriCorps interventions in one-on-one 

settings did not differ in initial reading proficiency, growth in reading proficiency, or reading 

proficiency at the end of the school year. Results did reveal suggestive evidence that the youngest 

students (K and 1st grade) and the older students (5th graders) might benefit more from individualized 

intervention delivery in terms of linear growth. However, it is important to balance small potential 

gains from providing one-on-one interventions with these students against the findings that mode of 

intervention delivery does not appear to relate to differences in year-end reading proficiency. While 

individualized delivery might be beneficial in terms of faster proficiency growth, intervention delivery 

in small group settings does not result in detrimental effects on overall proficiency development. 

Interestingly, AmeriCorps members were split in their responses (see Appendices C and D) about the 

impacts on learning of using either modality, with some members stressing the importance of one-onone attention that teachers cannot provide, while other members stressed the importance of interactive 

learning provided in small group settings. 
51

CONCLUSIONS

The first evaluation question sought to determine whether AmeriCorps member services impacted 

student growth in reading proficiency across the academic year. In comparisons between students who 

received AmeriCorps intervention support and students in the same grades, within the same schools, 

who did not receive intervention services, findings were generally consistent. As one might expect, 

students in need of services were generally performing poorly in relation to students who did not 

require services. Given that service provision was determined by decrements in reading proficiency 

within the Lexia Core 5 system, these findings are not surprising. However, in nearly all grades, 

students receiving AmeriCorps services maintained growth in reading proficiency that was comparable 

to their peers who did not require intervention supports. These findings suggest that AmeriCorps 

service provides a positive benefit to students who receive services. However, findings comparing 

students in schools served by AmeriCorps members across grades that were either served or not served 

by an AmeriCorps member did not reveal differential patterns of proficiency growth as a function of 

AmeriCorps member presence in the building versus actual service provision to students in a particular 

grade. 

The second evaluation question addressed whether the amount of AmeriCorps service received 

related to differential reading proficiency growth. An obvious comparison to address this question 

focused on comparison reading proficiency growth among students in schools where AmeriCorps 

members provided services (service > 0) and schools that did not have an AmeriCorps member 

(service = 0). Findings were generally consistent in that reading proficiency growth was approximately 

consistent across schools, whether the school had an AmeriCorps service member or not. Although 

these findings suggest little benefit to having an AmeriCorps member in the building, comparisons of 

students receiving support services with regard to amount of service received painted a different 

picture. In schools where AmeriCorps members provided support, the amount of support services was 

consistently related to reading proficiency growth. While the amount of service received is necessarily 

tied to proficiency decrements (those falling behind require more support), findings indicated that 

increases in intervention frequency reduced proficiency deficits among students who received support 

services. That is, although a student receiving 10 interventions would be expected to score lower on a 

proficiency assessment than a student receiving no interventions, that same student who received 10 

interventions did not continue to fall further behind a student who received only 5 interventions. These 

findings point strongly to efficacy of AmeriCorps interventions at maintaining reading proficiency 

growth among those students who require member support. 

The third evaluation question intended to examine whether AmeriCorps service member presence 

resulted in an added benefit to students/schools in comparison to schools that did not have an 

AmeriCorps member. Findings were limited with regard to added benefits due to unavailable data 

regarding individual student need for support in non-AmeriCorps buildings and also by somewhat 

limited information about other services potentially available in all buildings. Generally, results were 

consistent in that AmeriCorps member presence did account for small amounts of variance in students 

initial proficiency assessments, but compelling evidence with regard to proficiency growth did not 

emerge. 
52

The final evaluation question focused on potential moderators of intervention efficacy that might 

result from characteristics of the AmeriCorps member or variability in how the AmeriCorps members 

provided service in their respective buildings. Member characteristics were limited to workloads, 

intervention counts, and other factors that were largely difficult to isolate. Findings related to 

differential methods of identifying students in need of support services or differential modalities of 

delivering interventions consistently demonstrated that the efficacy of the AmeriCorps interventions 

did not differ across important differences in implementation. Specifically, use of flagging or rostering 

demonstrated no consistent effects across grades with regard to altered reading proficiency growth, 

whereas delivery of intervention services in one-on-one or small group settings were generally equally 

efficacious. These findings do seem promising in that lack of influences allows for greater flexibility, a 

common theme among members, in how AmeriCorps members are able to best deliver intervention 

services to the students who need additional support. 

Collectively, the findings in this evaluation point to evidence that AmeriCorps member service 

does provide a benefit to students who receive services in that those students do not continue to fall 

further behind their peers over the course of the academic year. Specifically, parallel growth among 

those who did and did not receive AmeriCorps support, and decreased decrements in proficiency due 

to increase intervention experiences among those who did receive services clearly demonstrate that the 

services provided by AmeriCorps members are benefiting students. Overwhelming evidence of added 

benefit to schools due to AmeriCorps service was less compelling due largely to limitations on schoollevel programming and support data and, more importantly, to relatively little variance in reading 

proficiency growth across buildings. 

Limitations 

As described in the evaluation design section above, the inability to cleanly delineate intervention 

and comparison groups was known prior to conducting the evaluation. Although the evaluation plan 

aimed to account for this limitation by conducting multiple different comparisons to isolate potential 

confounding factors, additional unforeseen challenges did arise during the evaluation; these additional 

challenges reflect the bulk of limitations that will be discussed. 

A primary limitation with regard to student data involved access to Lexia flagging of students in 

schools that did not include AmeriCorps members. Planned as a comparison of interest in the 

evaluation (see 3B above), this comparison would have provided useful information for equating 

students across AmeriCorps and Non-AmeriCorps schools with regard to their potential need for 

intervention services. In addition, availability of such data would have provided an index of student 

need on a consistent metric that could have been employed at both the individual student and 

school/building level. 

A second important limitation centers on the ability to disentangle the assignment of AmeriCorps 

members from demonstrated student need. For example, an AmeriCorps member may be vital for 

providing services to 3rd grade students in one school but an AmeriCorps member in another school 

may be more needed in a different grade. Without some index of how determinations are made to 

distribute AmeriCorps member service loads within schools, it remains unclear whether students in 

grades without service do not have access to services because they are generally not needed (i.e., 3rd

graders in School A are all progressing). 
53

A final major limitation involves fidelity of AmeriCorps service use and availability of alternative 

services within schools. The current evaluation did not have strong data related to implementation 

fidelity (identification method and delivery mode, aside), and there was virtually no data available with 

regard to competing/complementing services in either AmeriCorps or non-AmeriCorps schools. Based 

on comments obtained from AmeriCorps members (see Appendix D), individual classrooms varied in 

their use of the Lexia Core 5 program (some did not use it at all). In some assignments, AmeriCorps 

members were often serving a different function (e.g., behavioral interventionists) than intended. 

Finally, members acknowledge occasional difficulty in locating their intervention (flagged) students 

because those students were already involved in other (non-AmeriCorps) intervention supports. These 

fidelity concerns are not inherently the responsibility of AmeriCorps members but such variability 

almost certainly impacts the efficacy of intervention services at the classroom/building level. Without 

consistent indicators of the degree to which AmeriCorps members are dealing with these external 

issues, influences on student outcomes cannot be separated out to isolate AmeriCorps intervention 

service versus other services versus other duties. 

Ethical Considerations 

Dr. Abraham and Dr. Polush, independent evaluation and research consultants (the authors of this 

report), conducted this evaluation. Dr. Abraham and Dr. Polush completed training in the protection of 

human research participants through the Collaborative Institutional Training Initiative (CITI). A Data 

Sharing Agreement enacted July 9, 2019 between DMPS and the two consultants governed use and 

sharing of student assessment and demographic data for the purpose of this evaluation. 

All student assessment and demographic data were provided by DMPS staff authorized to access 

student information stored in the district infinite campus (IC) system and/or the AmeriCorps projectspecific database. Specifically, DMPS staff (1) de-identified students’ personal information to ensure 

privacy and (2) made data required for the purpose of the evaluation available via a secured Box site 

created for the project evaluation only and jointly maintained by DMPS AmeriCorps staff members 

and the evaluator-contractors. Student identification numbers linked by the DMPS staff members 

involved in this project evaluation for matching subsequent AmeriCorps student and member data sets 

necessary for the repeated measures design of this evaluation. 

AmeriCorps members’ participation in the online survey was voluntary. Prior to survey 

administration the evaluator (Dr. Polush) attended the AC members’ regular meeting and explained the 

survey, its purpose, voluntary participation, confidentiality, access to the survey data/responses, and 

potential sensitive issues. The evaluator also answered questions. Responses from members who 

participated in the survey administered at the beginning and at the end of the 2018-2019 school year 

were kept confidential. Only the DMPS AmeriCorps staff members and the two evaluator-consultants 

had access to the survey raw data. The survey results were presented in aggregate form so that no 

personal information was identifiable. Finally, throughout the conduct of the evaluation, the evaluators 

referred to the best practices outlined by the National Center for Education Statistics SLDS in 

Technical Brief 3, “Statistical Methods for Protecting Personally identifiable Information in Aggregate 

Reporting” (http://nces.ed.gov/pubs2011/2011603.pdf) to minimize, to the greatest extent possible, any 

risk that individuals could be identified. 
54

RECOMMENDATIONS 

Recommendations for moving forward based on the findings of this evaluation are complicated to 

some degree by a plan to replace the Lexia Core 5 system with another system, FAST BRIDGE, in 

DMPS schools during the 2020-2021 year. Although this evaluation focused on the benefits of 

AmeriCorps members providing structured interventions, those interventions were connected to the 

Lexia system, the corresponding curriculum, and the methods Lexia provided to monitor proficiency 

progress. Without further knowledge of capabilities for monitoring and intervening through use of the 

FAST BRIDGE system, addressing future actions based on the current Lexia system seems of little 

utility. Rather, the recommendations discussed below are intended to be broad enough to adapt to a 

new system and also specific to future evaluation efforts to examine efficacy of AmeriCorps member 

service, regardless of the specific intervention system in place at that time. 

Feedback obtained from AmeriCorps members (see Appendices C and D) consistently indicated 

that there were difficulties with uptake at the start of the school year. In many cases, members 

indicated being isolated, unintegrated, or unrecognized by school staff. Connected to weak integration 

in the school building, members indicated that they often found their position/responsibilities unclear 

in the eyes of school staff. For example, one member explicitly noted that initial duties to get students 

and teachers set up in Lexia continued long into the academic year, taking time away from providing 

interventions. As AmeriCorps members intend to continue service provisions in schools, it seems wise 

to clearly delineate what members are there to do and to develop a better system to integrate members 

into the school culture early during their service tenure. 

AmeriCorps members also indicated a desire for better training in the Lexia system prior to 

performing intervention services. This need for training likely transcends specific systems but does 

reflect a need among AmeriCorps members to feel more confident with the material they are expected 

to use. In addition to bolstering confidence/efficacy, more structured training in the relevant student 

service system will also instill a greater degree of implementation fidelity, as members would no 

longer feel the need to figure things out on their own. This collateral improvement in implementation 

fidelity would also start to disentangle individual member variability in service provision from efficacy 

of the services provided, one aspect that was difficult to do in the current evaluation. 

Perhaps the most prominent finding in this intervention was that most of the variability in reading 

proficiency and reading proficiency growth occurred across individual students; not across different 

schools. Because of such substantial individual variability, subsequent efforts to evaluate the efficacy 

of AmeriCorps members’ service using student outcomes would be wise to identify and collect data 

elements that account for those extraneous sources of variability. Although the current evaluation 

employed covariates in all analyses to attempt to statistically control for individual-level influences on 

proficiency, student-level data were limited to demographic information collected as part of the state 

reporting system. The current evaluation intended to include information about socioeconomic context 

beyond free/reduced lunch status (the only SES-relevant index collected through state reporting) and 

this information did relate to variability in student scores. Although future evaluation efforts might 

continue to use contextual measures, individual (family-level) measures would be much more 

informative with regard to prediction of individual assessment scores. An increased emphasis on 

individual student indicators related to student outcomes would benefit future evaluation efforts 

regardless of whether quasi-experimental, matching, statistical control, or randomized designs are 

employed. 
55

REFERENCES

Balfanz, R., Herzog, L., & Mac Iver, D. J.(2007) Preventing student disengagement and keeping 

students on the graduation path in urban middle-grades schools: Early identification and effective 

interventions, Educational Psychologist, 42(4), 223-235, DOI: 10.1080/00461520701621079

Chen, H. T. (2015). Practical program evaluation: Theory-driven evaluation and the integrated 

evaluation perspective (2nd ed.). Thousand Oaks, CA: Sage. 

Foorman, B. R., Francis, D.J., Winikates, D., Mehta, P., Schatschneider, C., & Fletcher, J. M. (1997) 

Early interventions for children with reading disabilities, Scientific Studies of Reading, 1(3), 255-

276, DOI: 10.1207/s1532799xssr0103_5 

Haager, D., Klingner, J., & Vaughn, S. (Eds.). (2007). Evidence-based reading practices for response 

to intervention. Baltimore, MD: Paul H Brookes. 

Kreft, I. G. G., & de Leewu, J. (1998). Introducing multilevel modeling. Thousand Oaks, CA: Sage. 

Mertens, D. M., & Wilson, A. T. (2012). Program evaluation theory and practice: A comprehensive 

guide. New York, NY: The Guilford Press. 

Saldana, J. & Omasta, M. (2018). Qualitative research: Analyzing life. Thousand Oaks, CA: Sage. 

Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis. New York: Oxford University 

Press. 

Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced 

multilevel modeling. Thousand Oaks, CA: Sage. 
56

APPENDICES 

Appendix A: Theory of Change
57

Appendix B: AmeriCorps Member Survey 

Introduction 

Greetings! Please share with us your reflective thoughts on your experiences serving as the DMPS 

AmeriCorps Lexia Tutor at the beginning of this 2018 fall semester by completing this brief survey. 

This survey includes one multiple choice question and five open-ended questions. It should take 

approximately 30 minutes to complete. 

Your participation in this survey is voluntary. Your responses will be kept confidential. 

We will summarize survey responses in an aggregate format. That means that your and other 

AmeriCorps Lexia Tutors' feedback will be combined so that no personal information will be 

identifiable. 

We appreciate in advance your time and insights!

Please complete this brief survey by [date] at the end of the day. 

Questions 

Q1: How were your students selected for tutoring sessions? (Please select ONE option.) 

 Students selected only through Lexia flags: 

Please explain your process of student selection in more detail. 

 Students selected only through roster provided by school staff 

Please explain your process of student selection in more detail. 

 Students selected through both Lexia flags and roster 

Please indicate your percentages of use of each method to select students, for example, 

30% roster and 70% Lexia flag. 

Q2 How is your service as a Lexia tutor most impactful to students? (Please describe.) 

Q3: What have been the most challenging experiences as a Lexia tutor thus far (i.e. during the 

beginning of this 2018 Fall Semester)? (Please describe.) 

Q4: How have you dealt with these challenges? (Please describe.) 

Q5: Please use this space to provide additional insights that could help the DMPS staff better 

understand your service to our students as a Lexia Tutor and make improvements. Thanks much in 

advance. 
58

Appendix C: AmeriCorps Member Survey Results (October 2018)
59
60
61
62

Appendix D: AmeriCorps Member Survey Results (April 2019) 
63
64
65
66
67
68

Appendix E: General Analytic Model

Linear Multilevel Model Specification1

Level 1: Time 

At the first level of the model, repeated identical assessments (Y), the FAST or MAP assessments in 

this evaluation, were specified as outcomes due to time (t) nested within student (i). Coding of time 

is somewhat arbitrary in terms of months vs. weeks, etc. as is the setting of the ‘intercept’ parameter 

(0) that can be specified at any point where the effect of time is set to zero. For example, coding 

time as t = 0, 1, 2 for three assessments sets the intercept as the initial score on the assessment. 

Alternatively, coding time as t = -2, -1, 0 sets the intercept as the final assessment at Time 3. Change 

over time in the repeated measures is modeled as a linear (or nonlinear) function of time (t) and 

captured by the slope parameter (1). Because time is not a perfect predictor of observed scores on 

the repeated assessments, unexplained variance is captured by a residual (e). The residual estimation 

can involve time-specific components (shown here as the more general model) or be estimated in a 

restricted form to reflect equality of the residual variance across time. 

Yti = 0ti + 1ti(Time) + eti

Level 2: Student 

At the second level of the model, the growth parameters of the Level 1 model (0 and 1) were 

estimated as outcomes that vary across Level 2 units, individual students in this case. Variability in 

the intercept term (r0i) indicates that individual students differ in their initial level (if t1 = 0) or final 

level (if t3 = 0) of the measured assessment. Variability in the slope term (r1i) indicates that the rate of 

change in assessment scores differs across individual students. Each of the Level 1 growth 

components has an overall average that reflects the average intercept (00) and average slope (10) 

across all students. 

0i = 00 + r0i

1i = 10 + r1i

Variables of interest at the student level enter the model at this point becoming potential predictors of 

individual variability in both starting (or ending) points and rates of change. For example, a student’s 

degree of absenteeism or previous year’s proficiency might serve to predict lower starting (or 

ending) levels of proficiency on the FAST/MAP assessments and also predict the rate of change, 

indicating that higher absenteeism or lower baseline proficiency corresponds to less improvement (a 

flatter slope) in proficiency over the full school year. Entry of predictors at the student level only 

slightly modifies the Level 2 equations: 

0i = 00 + 01(Days Absent) + r0i

1i = 10 + 11(Days Absent) + r1i

 

1 Notation consistent with most MLM sources 
69

Linear Multilevel Model Extension

The general growth model described above extends directly to a third level, reflecting either school or 

AmeriCorps service member in the current analyses. The extension changes in only notation at Levels 

1 and 2 where the additional j subscript reflects school/member. 

Level 1: Time 

Equations and parameter interpretations remain the same as described above, with the addition that 

repeated assessments (t) are nested within students (i) who are nested within school/member (j). 

Ytij = 0tij + 1tij(Time) + etij

Level 2: Student 

Once again, model equations and parameter interpretations remain unchanged. 

0ij = 00j + r0ij

1i = 10j + r1ij

Level 3: School 

At Level 3, the average growth parameters of the Level 2 model (00j and 10j) were estimated as 

outcomes that vary across Level 3 units, schools or AmeriCorps members in this case. Variability in 

the average intercept (U00j) indicates that the average starting (or ending) point for students within 

schools/members differs across schools/members. Similarly, variability in the average slope (U10j) 

indicates that the average rates of change within each school/member vary across schools/members. 

Each of the Level 2 average growth components are averaged across Level 3 units 

(schools/members) to obtain an overall intercept (000) and an overall slope (100), reflecting the 

average starting (or ending) point and average rate of change for all students across all 

schools/members. 

00j = 000 + U00j

10j = 100 + U10j

Level 2 variables of interest at the school or AmeriCorps member level enter the model at this point 

becoming potential predictors of school/member variability in starting (or ending) points, and rates 

of change. For example, a member’s delivery modality might predict higher average starting (or 

ending) levels of proficiency on the FAST/MAP assessments and also predict the average rate of 

change, indicating that one-to-one delivery relates to better improvement (a steeper slope) in 

proficiency over the full school year. Entry of predictors at the school/member level results in similar 

modification of the Level 3 equations: 

00j = 000 + 001(Service) + U00j

10j = 100 + 101(Service) + U10j