Washington Reading Corps 

Impact Evaluation 

2019

FINAL REPORT AND RECOMMENDATIONS

Mission360 Consulting, LLC 

www.mission360consulting.com

ESD - STATE OF WASHINGTON 
Mission360 Consulting, LLC 1211 Summit Ave, Saint Paul, MN 55015  Landline: 651-547-7710  Cell: 612-770-1249

Washington Reading Corps Program Impact Evaluation 

July 2019 

Authors 

This study was conducted by researchers from Mission360 Consulting, LLC; JEMM Consulting, LLC; and 

Parsimony, Inc.: 

 Patrice Tetta, M.S., Mission360 Consulting, LLC; 

 Jordan Baker, M.S., Parsimony, Inc.; and 

 Amanuel Medhanie, Ph.D., Parsimony, Inc. 

The researchers had support from Caroline Cochran, Ph.D., JEMM Consulting, LLC. 

For citations and reference to this publication, please use the following: 

Tetta, P., Baker, J., & Medhanie, A. (2019). Washington Reading Corps Impact Evaluation Final Report 

and Recommendations, Washington Employment Security Department, Olympia, WA. 

Please note: the conclusions and recommendations presented in this report reflect the perspectives of 

the researchers listed above, not necessarily those of the Washington Reading Corps. 

Acknowledgements 

The researchers would like to thank the following individuals and organizations for their assistance in 

conducting this study: 

● The Washington State Office of Superintendent of Public Instruction, which provided the 

student data (not individually-identified) necessary to conduct the analyses. 

● The individual Site Supervisors who graciously gave their time for interviews. 

● The Washington Reading Corp staff who provided the assistance we needed to carry out the 

research efficiently and effectively. 

● Researchers at RMC Research Corporation, who developed the original study design upon 

which this evaluation work relied. 
Mission360 Consulting, LLC 1211 Summit Ave, Saint Paul, MN 55015  Landline: 651-547-7710  Cell: 612-770-1249

Table of Contents 

EXECUTIVE SUMMARY .............................................................................................................1

Purpose of the Study ............................................................................................................... 1

Study Design.......................................................................................................................... 1

Key Findings ..........................................................................................................................2

Recommendations .................................................................................................................. 3

WASHINGTON READING CORPS IMPACT EVALUATION.............................................................4

About the Washington Reading Corps .......................................................................................4

About the Evaluation Study......................................................................................................8

Study Design........................................................................................................................ 10

Conclusions and Implications..................................................................................................28

Recommendations ................................................................................................................29

APPENDIX A: DATA FILES AND VARIABLES USED ...................................................................32

APPENDIX B: STUDENT DESCRIPTIVE STATISTICS AND AVERAGE MONTHS OF GROWTH .......34

Overall ................................................................................................................................ 34

Gender ................................................................................................................................ 35

Race/Ethnicity ...................................................................................................................... 36

Free or Reduced-Price Lunch (FRPL) Status .............................................................................. 38

English Learner (EL)/Limited English Proficiency Status ............................................................. 39

Grade Level..........................................................................................................................40

APPENDIX C: WRC SITE SUPERVISOR INTERVIEW QUESTIONS ...............................................41

APPENDIX D: WRC SITE SUPERVISOR INTERVIEW CONTENT ANALYSIS ..................................45

APPENDIX E: WRC LOGIC MODEL............................................................................................57
 Washington Reading Corps Impact Evaluation, July 2019 Page 1 

EXECUTIVE SUMMARY 

In 2018, Washington Reading Corps (WRC), a statewide literacy-support program, hired Mission360 

Consulting, LLC to conduct an external impact evaluation that would examine the overall effect of fulltime WRC literacy tutors placed in Kindergarten through 4th grade classrooms in public schools across 

Washington State. 

Purpose of the Study 

The goal of this study was to measure differences in literacy-skill and reading growth between two 

groups of Washington State public school students in grades K through 4: 

● A treatment group, who received WRC tutoring in combination with other services provided 

through the Washington State Learning Assistance Program - English Language Arts (LAP 

ELA), and 

● A comparison group, who received other LAP ELA services, but no WRC tutoring. 

The study was designed to provide statistical evidence of the impact of WRC tutoring on reading 

growth compared to what would have happened in the absence of the program. The study focused on 

students identified as needing “Tier 2” reading interventions (students not making expected progress 

with standard literacy and reading instruction, assessed as needing targeted intervention to meet 

benchmarks). The study also measured the impact that WRC tutors have on the culture of reading in 

the schools and communities they serve. 

The two research questions investigated were: 

1. To what extent do months of growth differ between Tier 2 students served by a combination of 

WRC tutoring and other LAP ELA services and Tier 2 students served only by other LAP ELA 

services? 

2. To what extent does the work of WRC Members impact the culture of reading in the school and 

in the local community? 

Study Design 

The mixed-methods evaluation included two components: 

● A quasi-experimental design study that used quantitative program-level and student-level 

data from Washington State’s CEDARS longitudinal database to test whether schools at which 

WRC Members were placed showed statistically-significant differences in student reading 

growth (as measured by changes in assessment scores) compared with similar schools that did 

not have WRC Members. 

and 

● A case study that used qualitative data collected from phone interviews with staff at WRC 

service sites to assess and describe WRC’s impact on the culture of reading at the site and in the 

community. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 2 

Key Findings 

This study resulted in three key findings: 

1) No statistically-significant differences in months of reading growth were found 

between students who received a combination of WRC tutoring and other LAP ELA 

services and students who received only other LAP ELA services.

To examine the relationship between student-level characteristics and school-level 

characteristics and student reading growth, several statistical models were fit on a sample of 

5,457 students. The results of the models suggested that neither receiving a combination of 

WRC tutoring and other LAP ELA services nor receiving only other LAP ELA services was a 

significant predictor of students’ reading growth. 

2) The effect of WRC tutoring on reading growth does not differ significantly by student 

demographics, such as income status (as indicated by eligibility for the federal Free 

or Reduced-Price Lunch program) or ethnicity. 

Through the use of statistical models, the researchers assessed whether reading growth 

differed among different demographic groups for students who received a combination of WRC 

tutoring and other LAP ELA services compared with students that received only other LAP ELA 

services. The results of the models suggested no significant differences in the effect of the 

combination of WRC tutoring and other LAP ELA services compared with only other LAP ELA 

services by either student income status (eligibility for Free or -Reduced- Price Lunch program) 

or ethnicity. 

3) Site staff at schools where WRC Members are placed perceive a significant, positive 

impact for WRC on the culture of reading at their schools. 

Site Supervisors interviewed expressed a high regard for the WRC program and for the 

members placed at their sites. Their praise for the program was consistent and effusive. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 3 

Recommendations 

The researchers are making the following recommendations to WRC, so that the organization can more 

clearly drive and measure impact for WRC tutoring. 

Recommendations for Program Implementation and Data Collection

1) Standardize the WRC model and the assessments used to measure reading growth. 

2) Create and require the use of an electronic system (a) for WRC site staff to report on all member 

activities and (b) for individual WRC Members and community-based volunteers to report on 

tutoring provided, including electronic entry of assessment data to minimize errors in the entry and 

reporting of student assessment scores. 

3) Work with the administrators of the CEDARS longitudinal database to create clearer ways to 

distinguish which students receive WRC tutoring, which students receive only other LAP ELA 

interventions, and which students receive both. 

4) Work to identify a control group of students who receive neither WRC tutoring nor other reading 

interventions in order to more clearly demonstrate the impact of WRC tutoring. 

Recommendations for Future Study 

1) The researchers strongly recommend that WRC establish core tutoring model and pilot that model 

with a subset of program sites that agree to implement the model with fidelity. A quasiexperimental study could be designed to evaluate the impact of the standard model compared with 

the existing, non-standardized tutoring services WRC Members provide. 

2) The researchers also recommend either (a) changing the way WRC and other LAP ELA services are 

flagged in the CEDARS longitudinal data system or (b) creating some other reliable, electronic 

method to identify the specific interventions provided at the student level and then reimplementing substantially the same study. 

3) In addition, the researchers recommend conducting a broader survey of site staff at all WRC sites, 

which would include questions from the interview tool created for the case study (described earlier) 

as well as a range of other topics. Responses from these surveys could be used to determine areas 

of deeper exploration and identify a subset of site staff to interview to gather further information to 

guide program improvement and/or future areas of study. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 4 

WASHINGTON READING CORPS IMPACT EVALUATION 

About the Washington Reading Corps 

Background, History, and Need 

WRC is a dynamic statewide literacy-support program designed to support the development of “school 

readiness” skills for children in early learning centers and improved academic performance in reading 

for elementary school students. 

WRC harnesses the talent, energy, and passion of AmeriCorps Members to work in alignment with 

schools and community-based organizations to provide literacy-focused interventions individually 

and/or in small groups, including: 

● Evidence-based literacy skills-building activities and support for low-income and English 

Learner (EL) preschool children who do not meet age-appropriate benchmarks for literacy 

development. 

● Evidence-based reading tutoring for low-income students (those eligible for Free or ReducedPrice Lunch) and EL students in Kindergarten through Grade 4 who do not meet grade-level 

benchmarks in reading. 

Both preschool children and K-4 students served are those considered to be “Tier 2” in most MultiTiered Systems of Supports (that is, they struggle with literacy or reading but do not experience the 

most severe learning challenges). 

The first program of its kind in the nation, WRC was created in 1998 through an unprecedented 

collaboration between four public agencies: the Washington Governor's office; the Washington State 

Commission for National and Community Service; the Washington Employment Security Department, 

which houses the Washington Service Corps (WSC); and the Office of the Superintendent of Public 

Instruction (OSPI), Washington’s State Education Agency. 

These four agencies established WRC to advance the state's vision for literacy: to ensure all children 

and youth are able to effectively read, write, speak, think, create, problem solve, and reflect in order to 

participate in a democratic, multicultural society. 

Now jointly led by WSC and OSPI, WRC provides literacy tutoring to students who are not meeting 

grade-level standards in reading. This collaborative governance helps ensure that all WRC 

programming not only is fully aligned with the state’s literacy development guidelines, reading 

standards, and comprehensive plans to improve student reading proficiency, but is also deeply rooted 

in the vision and goals of national service. 

Since its inception, WRC has worked to address the significant challenges in Washington State related 

to children’s literacy development and young students’ reading proficiency and the impact of low 

literacy and reading proficiency levels on later academic success. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 5 

Decades’ worth of research (Hernandez, 20121; Lesnick et al, 20102; Snow, Burns & Griffin, 19983) shows 

that 4th grade reading proficiency is critical for school success and that a solid foundation of literacy 

skills before kindergarten entry is crucial to achieving grade-level reading proficiency by 4th grade. This 

research indicates that children who enter kindergarten with lagging literacy skills stay behind their 

peers and continue to struggle not only with reading but also with other core academic subjects that 

demand reading proficiency (Bridges, et al, 20044

). 

Without significant intervention, 75% of students who are poor readers in elementary school remain 

poor readers in high school (U.S. Department of Education, 19995

) putting them on “the high school 

dropout track”: in a great many cases, [dropping out] is the result of not being able to read proficiently 

as early as 4th grade (Feister, 20106).

The students who struggle most with reading are disproportionately poor children of color (Feister, 

20137). In 2017, disparities in performance on the National Assessment of Educational Progress (NAEP) 

—a.k.a. “The Nation’s Report Card”— were wide: the average reading scale score for all 4th grade 

students was 221 points, with a 28-point gap between low-income students and higher-income 

students, and a 37-point gap between students who are English Learners (EL) and those who are not. 8

National trends are echoed in Washington, where the majority of low-income and EL 4th graders lag 

significantly behind their peers: on the 2017 NAEP, the average reading scale score for all 4th grade 

students in Washington, 223 points, was slightly higher than the national average, but the disparities 

between student groups were larger: The state had a 32-point gap in 4th-grade reading scale scores 

between low-income students (FRPL eligible) and higher-income students not FRPL eligible) and a 48-

point gap between EL and non-EL students. 

In addition, in the 2017-18 academic year, only 42.8% of low-income students (those eligible for the 

federal Free or Reduced-Price Lunch program) met or exceeded grade-level standards in English 

Language Arts (ELA), compared with 74.5% of students who were not low income and only 14% of 

English Learners met or exceeded grade-level ELA standards, compared with 64.5% of students who 

 

1 Hernandez, D. (2012). Double Jeopardy: How Third Grade Reading Skills and Poverty Influence High School Graduation. Baltimore, MD: The 

Annie E. Casey Foundation. https://www.aecf.org/resources/double-je opardy/ 

2

Lesnick, J., George, R., Smithgall, C., & Gwynne, J. (2010). Reading on grade level in third grade: How is it related to high school performance 

and college enrollment? Baltimore, MD: Annie E. Casey Foundation. https://www.aecf.org/resources/reading-on-gra de-level-in-third-gra dehow-is-it-related-to-high-school-perf/

3 Snow, C. E., Burns, S. M., & Griffin, P. Editors. (1998). Preventing Reading Difficulties in Young Children, Chapter 4: Predictors of Success and 

Failure in Reading. National Research Council, National Academy of Sciences. Courtesy of National Academy Press.

4 Bridges, M., Fuller, B., Rumberger, R., & Tran, L. (2004). Preschool for California’s children: Promising benefits, unequal access. Berkeley, CA: 

Policy Analysis for California Education (PACE).

5 U.S. Department of Education, America Reads Challenge. (July 1999). Start Early, Finish Strong: How to Help Every Child Become a Reader. 

Available at www2.ed.gov/pubs.

6

Feister, L. (2010). Early Warning! Why Reading by the End of Third Grade Matters. KIDS COUNT Special Report. Baltimore, MD: Annie E. 

Casey Foundation.

7

Feister, L. (2013). Early warning confirmed: A research update on third-grade reading. Baltimore, MD: Annie E. Casey Foundation.

8

https://www.nationsreportcard.gov/dashboa rds/achievement_ga ps.aspx 
 Washington Reading Corps Impact Evaluation, July 2019 Page 6 

were not identified as English Learners (State of Washington Office of Superintendent of Public 

Instruction9).

WRC works to provide additional support for students from pre-K through 4th grade to help them make 

gains in reading growth that will enable them to meet or exceed grade-level ELA standards. 

Programming and Services 

To address the need for literacy and reading improvement among low-income and EL children and 

students who are below literacy and reading benchmarks, WRC engages in a variety of activities each 

year, including: 

● Providing leadership, guidance, training, technical assistance, and program coordination for 

sites across the state. 

● Placing AmeriCorps Members at program sites across the state (including high-needs10 early 

childhood education programs, elementary schools, and community-based organizations) to 

provide evidence-based literacy skills-building activities and interventions. 

● Coordinating with site-based program staff who are responsible for overseeing the service of 

the Members placed at their sites) to provide Member supervision and support, site-specific 

literacy and tutoring training, and assessment of individual students’ strengths and needs. 

● Offering other resources, primarily through regional Educational Service Districts, including 

technical assistance from literacy specialists and data from Washington State’s Comprehensive 

Education Data and Research System (CEDARS) longitudinal database.11

The literacy activities and reading tutoring WRC Members provide directly address asses sed student 

needs and are aligned with state standards and best practices in literacy development and reading 

support. Schools at which Members are placed are identified through a formal Request for Application 

process and priority for Member placement is given to schools with the highest percentages of lowincome (FRPL-eligible) and/or EL students. 

In the 2017–2018 program year, WRC enrolled 178 AmeriCorps members (158 full-time and 20 halftime) to provide (a) evidence-based literacy skills-building activities for low-income and English EL 

children from age 3 to 5 and (b) evidence-based reading tutoring for low-income and EL students from 

kindergarten through Grade 4. Members served in approximately 100 high-need early learning centers 

and elementary schools throughout Washington State and provided tutoring to 711 children ages 3 to 5 

and 4,154 Kindergarten to grade 4 students. Most WRC sites are elementary schools and vary greatly in 

geographic location, available resources, grades served, staffing structure, and how they support their 

WRC Member(s). 

Of the Kindergarten through 4th grade students served through WRC the program that year, 2,901 

completed the program (either met benchmarks and exited the program or participated for at least 6 

 

9 All state-level data cited is from the state’s Report Card, accessed at 

https://washingtonstatereportcard.ospi.k12.wa.us/ReportCard/ViewSchoolOrDistrict/103300

10 “High-needs” elementary schools are those that enrollments of at least 60% low-income students, with low-income defined by eligibility for 

the federal Free or Reduced-Price Lunch program under the Richard B. Russell National School Lunch Act [20 USCS § 1021 (11)].

11 Through the CEDARS longitudinal database, OSPI collects, stores, and reports on student enrollment, completion, and proficiency data. 

Data from CEDARS was used for the quantitative analyses in this study.
 Washington Reading Corps Impact Evaluation, July 2019 Page 7 

weeks and/or through the end of the school year). Of those 2,901 students, 1,199 met benchmarks or 

made at least 11 months of growth in reading by the end of the program year. 

WRC Theory of Change and Logic Model 

WRC has articulated a theory of change to guide its work: 

By recruiting, training, placing, and supporting AmeriCorps members who (a) provide evidence-based 

tutoring to children who do not demonstrate age-appropriate language and literacy skills and to students 

who are below grade level in reading proficiency – particularly those who are low income or English 

Learners – and (b) engage parents in their children’s early learning and academic growth, WRC will 

increase school readiness, accelerate growth toward reading proficiency, and improve parents’ capacity to 

support learning at home for participating children. 

This theory of change is supported by a logic model that outlines the needs the program is addressing, 

described the inputs and activities the program uses to address identified needs, and articulates the 

specific outputs and outcomes the organizations projects to achieve as a result of the inputs and 

activities. WRC’s full logic model is included in the appendices of this report. 

As shown in that document, in the short term, WRC anticipates achieving the following outcomes at 

the sites at which AmeriCorps members are placed: 

(a) Among the children and students who complete WRC tutoring, improvements in: 

● Reading attitude and behaviors, measured by post-program surveys. 

● Language and literacy skills, measured by pre- and post-program 

assessments. 

● Reading proficiency or months of reading growth, measured by preand post-program assessments. 

(b) Increases in family participation in literacy nights and similar activities, measured by parent 

surveys and event sign-ins. 

(c) Reports by WRC Members that they intend to pursue career pathways into education, measured 

by surveys. 

In the medium term, WRC anticipates that school sites served by WRC Members will see increases in 

the percentage of children who meet or exceed age-appropriate benchmarks in language and literacy 

and grade-level benchmarks in reading. Over the long term, these improvements will increase college 

readiness and college and career success for WA students. 

Prior Evidence of Effectiveness for WRC’s Model 

In 2015, an external outcome evaluation was conducted by RMC Research Corporation to assess the 

impact of WRC tutoring on student reading outcomes. The evaluation used a hierarchical linear model 

(HLM) to assess WRC’s impact on student reading outcomes. Specifically, the evaluators tested the 

impact that certain site-level characteristics (supervisor helpfulness, tutoring group size, length of 

employment for WRC staff, total school enrollment) and student-level characteristics (average minutes 

per tutoring session, average tutoring days per week) had on student reading outcomes. The model 

utilized data exclusively on students that received WRC tutoring. 

 Washington Reading Corps Impact Evaluation, July 2019 Page 9 

● Alternative practices for which the district is able to provide evidence that the practices meet 

the definition of promising, research-based, or evidence-based. 

The interventions included in the LAP ELA Menu of Best Practices fall into four broad categories and 

school districts may use LAP ELA funds to offer one or more interventions that fall into these 

categories: 

1) Student-centered practices and strategies, which focus on direct supports for students who have 

been identified as below benchmark. 

2) Educator-focused practices and strategies, which aim to increase educators’ capacity to accelerate 

student reading growth. 

3) Transition and readiness practices and strategies, which are geared to provide additional support 

for students at key transition points, such as the 8th-grade-to-high-school transition. 

4) Family and community practices and strategies, which focus on family and community 

engagement as the key to student reading success. 

While the LAP ELA Menu of Best Practices does not identify specific branded programs that meet the 

definitions of promising, research-based, or evidence-based, it does identify the specific best-practice 

strategies school districts can implement, and offers both success factors for districts to consider when 

planning interventions and other considerations related to students, including student demographics, 

proficiency level, age and grade level, etc. 

One of the main best practices in the “student-centered” category of the LAP ELA Menu of Best 

Practices is “Tutoring by Adults.” According to the menu “carefully selected adult tutors 14... can provide 

targeted one-on-one or small-group instruction to meet the specific needs of students.” 

The specific evidence-based strategies for this best practice include, among other strategies: 

● Targeting students who are reading below grade-level proficiency standards and who have not 

yet met grade level ELA assessments. 

● Selecting a scientifically research-based intervention model within a multi-tiered system of 

support that use individualized, diagnostic assessments to design appropriate developmental 

lessons for students. 

● Providing extensive and ongoing tutoring for all tutors that includes observation and correction 

techniques. 

● Providing one-to-one or small group tutoring, consisting of 3–6 students. 

The LAP ELA services provided by the schools included in this study would have fallen into one of the 

four broad categories listed above. It is important to note that the CEDARS database, from which the 

researchers received the student data that was analyzed, indicates only whether LAP ELA services were 

provided. CEDARS does not identify either which category of services were provided or which specific 

services students received and no other data was available to the researchers that would have identified 

the specific LAP ELA service provided. 

 

14 “Adult tutors” can be teachers, intervention specialists, paraeducators, other classified personnel, and volunteers [including 

Washington Reading Corps members] who receive specialized professional learning in foundational literacy skills and the 

state’s ELA standards. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 10

WRC Tutoring 

WRC Members provide: (a) evidence-based literacy skills-building activities and supports for lowincome and ELL preschool children who do not meet age-appropriate literacy-development 

benchmarks and (b) evidence-based reading tutoring for low-income and ELL students in kindergarten 

through grade 4 who do not meet grade-level reading benchmarks. 

These interventions are provided individually and/or in small groups of three to six students for 

preschool children and K-4 students those who are struggling with literacy or reading but are not 

experiencing the most severe learning challenges (labeled “Tier 2” in a Multi-Tiered System of Support). 

Children and students receive WRC tutoring at least 3 times weekly, for at least 15 minutes each session 

and the focus of tutoring sessions is based on assessment of individual student needs. 

WRC tutoring is considered a “best practice” approach, as defined by the LAP ELA Menu, because 

members are well-trained adults who provide research- or evidence-based tutoring within a multitiered system of support, focusing on students who have been identified through diagnostic 

assessments as being in need of Tier 2 interventions. 

It is important to note that Washington is a “local control” state, meaning that each of its 295 school 

districts makes its own decisions about a wide range of policies, processes, and programs including 

education curricula and student assessment. Consequently, some variation among sites exists in the 

format of tutoring (e.g., “push-in” provided in the classroom or “pull-out” tutoring provided outside the 

classroom), the schedule and duration of sessions, the content tutored, and the formative and 

summative assessments used. These variations are based on local program implementation logistics, 

alignment with the school’s education program, and/or on students’ identified needs. 

Study Design 

The evaluation described in this document was conducted in late 2018 through early 2019 and included 

a subset of public schools and public-school students from across Washington State. Among the 

schools included in this study, 98 offered WRC tutoring as well as other LAP ELA services and 196 

offered only other LAP ELA services. 

This mixed-methods evaluation examined the impact of WRC on schools at which WRC Members are 

placed, using two different methods: 

1) A quasi-experimental design study, in which quantitative program-level and student-level data 

was collected and analyzed to test whether schools at which WRC Members were placed to provide 

tutoring services during the 2017-18 school year showed statistically-significant differences in 

student reading growth (as measured by changes in assessment scores) compared with similar 

schools that did not have WRC Members. 

This study was designed to answer the following question: To what extent do months of growth 

differ between Tier 2 students who received a combination of WRC tutoring and other LAP ELA services 

and Tier 2 students who received only LAP ELA? 

If students in schools with WRC Members were shown to have greater reading growth than 

students in schools without WRC Members (i.e., those who received only other LAP ELA services), it 

could be inferred that the WRC program had a positive impact on student achievement. 

2) A case study, in which qualitative data about WRC Members’ activities and staff perceptions of 

WRC’s impact was collected via phone interviews with staff at WRC service sites. The Site 
 Washington Reading Corps Impact Evaluation, July 2019 Page 11

Supervisors interviewed were selected randomly from a subsample of Site Supervisors at all schools 

that participated in WRC in the 2017-18 school year. 

The interview questions were designed to answer the following question: To what extent does the 

work of WRC Members impact the culture of reading in the school and in the local community?

Interviews offered a deeper examination of the program’s impact on the school and its students 

and staff. 

Site Supervisors’ answers to these questions helped to provide context for the quantitative student 

data analyzed and also pointed to further potential areas of inquiry to better understand WRC’s 

impact on schools, students, and the local community. 

A detailed description of each of these evaluation components appears below. 

Quasi-Experimental Study of the Impact of WRC on Reading Growth 

The quasi-experimental study was designed to measure the impact of WRC on reading growth and 

answer Research Question 1: To what extent do months of growth differ between Tier 2 students who 

served by a combination of WRC tutoring and other LAP ELA services and Tier 2 students served only by 

other LAP ELA services? 

To answer this question, the study used multivariate matching to create two groups of students 15 (a 

“treatment” group of students who received a combination of WRC tutoring and other LAP ELA 

services and a control group who received only other LAP ELA services, but not WRC tutoring) and then 

to compare reading growth for the two groups. 

To provide a more in-depth analysis of impact, Research Question 1 was subdivided into two separate 

but related questions, each of which is addressed individually in the following discussion: 

● Research Question 1a: To what extent do months of growth differ between Tier 2 students 

served by WRC tutoring and LAP ELA (labelled WRC + LAP ELA in the analysis tables) and Tier 2 

students served by only LAP ELA (labelled LAP ELA only in the analysis tables)?

● Research Question 1b: To what extent does the effect of WRC tutoring on reading growth differ 

by student demographics?

Data Collection 

To answer Research Question1 (sub-questions 1a and 1b), the researchers obtained the following data 

files from the Washington State Office of Superintendent of Public Instruction (OSPI). These files 

spanned the 2017-18 school year. 

1) A file detailing enrollment and demographic information for students at 2,386 elementary schools 

in Washington State. 

2) A file indicating whether students participated in WRC tutoring and/or other LAP ELA services 

during the 2017-18 academic year. 

3) A file containing student scores on reading/literacy assessments administered by sites. 

 

15 Students included in the study were selected from among all Tier 2 students enrolled in public schools in Washington State 

during the 2017-18 academic year. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 12

4) A file indicating whether students had a Free or Reduced-Price Lunch (FRPL) flag, which denotes 

eligibility for the federal Free or Reduced-Price Lunch program (a proxy for low-income status). 

5) A file indicating whether students had an EL/LEP flag, which denotes eligibility for English 

Language Development services (an indication of status as an English Learner). 

The specific variables that were used within each data file are displayed in Appendix A. 

Because of the way data is collected and reported via the state’s CEDARS longitudinal database, within 

a given data file, it was possible for a student to have more than one record associated with them if the 

student attended multiple schools during the 2017-18 school year. To resolve this issue, each student 

was linked to a single school of primary responsibility for the academic year in the following way: 

● Only student records that had a primary school flag indicator labelled as “Yes” were kept.

● If a student was associated with multiple primary schools of responsibility, the record that had 

the earliest school enrollment date and the highest value of cumulative days present was kept. 

Applying the above restrictions reduced the data to a single record per student. 

School and Student Selection 

School Selection 

Schools were selected for inclusion in the study via multivariate matching, through which WRC schools 

were matched to comparison (non-WRC) schools that were characteristically similar. The 

characteristics on which the two groups of schools were matched are: 

1) Whether the school had at least one student in the school flagged as receiving LAP ELA 

services. 

2) The proportion of the student body with an FRPL (Free or Reduced-Price Lunch) flag, indicating 

they were low-income students eligible for the federal FRPL program. 

3) The proportion of the student body with an EL/LEP (English Learner/Limited English 

Proficiency) flag, indicating they were identified as English Learners eligible to receive English 

Language Development services. 

This matching procedure used a 1-to-2 ratio of WRC schools to comparison (non-WRC) schools and 

yielded a final sample of 294 schools, with 98 WRC and 196 comparison schools. The results of the 

multivariate matching procedure (displayed in Table 1, below) produced a sample of characteristicallysimilar WRC and comparison schools. 

Prior to matching, the characteristics of interest for WRC schools were not closely reflected by those of 

the comparison schools: WRC schools had 57.6% students with an FRPL flag, compared with 47.5% at 

comparison schools and WRC schools had 14.0% students with an EL/LEP flag, compared with 10.2% at 

comparison schools. 

After matching, the characteristics of interest for WRC schools were much more closely reflected by 

those of the comparison schools: WRC schools had 57.6% students with an FRPL flag, compared with 

57.9% at comparison schools; and WRC schools had 14.0% students with an EL/LEP flag, compared 

with 13.0% at comparison schools. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 13

Table 1: Results of Multivariate Matching for School Selection 

Pre-Matching Post-Matching

Matching Variable WRC Schools Non-WRC Schools

(Comparison)

WRC Schools Non-WRC Schools

(Comparison)

Proportion of students with FRPL flag 0.576 0.475 0.576 0.579

Proportion of students with EL/LEP flag 0.140 0.102 0.140 0.130

Numbers in the table above represent average values. 

Student Selection 

After matching WRC and comparison (non-WRC) schools, the researchers then used the multivariate 

matching process to match WRC and comparison (non-WRC) students. To be considered for 

multivariate matching, students must have met all of the following criteria: 

1) Been enrolled in Kindergarten or grade 1, 2, 3, or 4. 

2) Either attended a WRC school and received both WRC tutoring and other LAP ELA services OR 

attended a non-WRC school (selected in the previous matching process) and received only other 

LAP ELA services. 

3) Had a valid score for their baseline assessment. 

4) Had a student record that contained complete data indicating whether they received WRC and/or 

other LAP ELA services. 

Of the students whose records were examined, 10,989 fit the above criteria. The variables that were 

used for multivariate matching are displayed in Appendix B. 

Students were matched on their baseline assessment score,16 grade level, and eligibility for Free or 

Reduced-Price Lunch (FRPL). Using a 1-to-2 matching ratio of WRC students (attended a WRC school 

and received both WRC tutoring and other LAP ELA services) to comparison students (attended a nonWRC school and received only other LAP ELA services) multivariate matching yielded a final sample of 

5,457 students, with 1,840 WRC students and 3,617 comparison students. An additional 5,532 students 

were unmatched (64 WRC students and 5,468 comparison students) and were, therefore, excluded 

from the analysis. 

The results of the multivariate matching procedure (displayed in Table 2 below) produced a sample of 

characteristically-similar WRC and comparison students. 

Prior to matching, the characteristics of interest for the WRC students were not closely reflected by 

those of the comparison students. For example: 

● 71.7% of WRC students were flagged as FRPL-eligible, compared with 74.3% of comparison 

students. 

 

16 Because schools use different student assessments that contain different score ranges, the baseline assessment score 

variable was rescaled to have a mean of zero and standard deviation of 1 within each grade level and assessment combination. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 14

● 25.9% of WRC students were in Kindergarten, compared with 15.8% of comparison students. 

● 13.2% of WRC students were in Grade 4, compared with 20.9% of comparison students. 

● WRC students had an average baseline score of 0.304, compared with -0.074 for Comparison 

students. 

After matching, the characteristics of interest for the WRC students were much more closely reflected 

by those of the comparison students. For example: 

● 71.8% of WRC students were flagged as FRPL-eligible, compared with 71.4% of comparison 

students. 

● 25.6% of WRC students were in Kindergarten, compared with 22.3% of comparison students. 

● 13.5% of WRC students were in Grade 4, compared with 13.8% of comparison students. 

● WRC students had an average baseline score of 0.052, compared with 0.109 for Comparison 

students. 

Table 2: Results of Multivariate Matching for Student Selection 

Pre-Matching Post-Matching

Matching Variable WRC Students Non-WRC Students 

(Comparison)

WRC Students Non-WRC Students 

(Comparison)

FRPL Flag 0.717 0.743 0.718 0.714

Grade Level K 0.259 0.158 0.256 0.223

Grade Level 1 0.223 0.215 0.222 0.224

Grade Level 2 0.213 0.200 0.213 0.227

Grade Level 3 0.173 0.218 0.174 0.188

Grade Level 4 0.132 0.209 0.135 0.138

Assessment Pre-score (scaled) 0.304 -0.074 0.052 0.109

Numbers in the table above represent average values. 

Analysis of Data

Research Question 1a 

Before analyzing the data for Research Question 1a (To what extent do months of growth differ between 

Tier 2 students served by WRC tutoring and LAP ELA and Tier 2 students served by only LAP ELA?), the 

researchers hypothesized that they would find a positive, statistically-significant relationship between 

the school-level WRC treatment variable and student reading growth. This positive relationship would 

suggest that, on average, students who received both WRC tutoring and other LAP ELA services would 

demonstrate higher growth than students who received only other LAP ELA services. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 15

Research Question 1a was answered by fitting a series of multilevel regression models, 17 one for each 

grade, using the lme function in the nlme package for the statistical programming language R.18 

The models took the form below: 

Yij = β0j + β1jX1ij

+ ∑ βpjXpij

+ eij

Student-level

β0j = γ00 + γ01W1j + b 0j School-level

β1j = γ10 School-level

Where: 

● Yij is the reading growth score for student i in school j. 

● β0j is the average growth score for school j. 

● β1j is a coefficient reflecting the relationship between a student-level variable X1j (e.g. 

student’s ethnicity) and the growth score for that student. 

● γ00is the overall average growth score. 

● γ10is a coefficient reflecting the relationship between the school-level variable W1j (whether 

school provided WRC tutoring) and the growth score for that student. 

and 

● γ01is a coefficient reflecting the relationship between the school-level variable W1j (whether 

school provided WRC tutoring) and the growth score for that school. 

The five grade-specific multilevel regression models used to answer research question 1a included the 

following student-level variables: 

● Baseline assessment score. 

● Ethnicity. 

● Gender. 

● Eligibility for Free or Reduced-Price Lunch (FRPL). 

● Eligibility for English Language Development services (EL/LEP students). 

 

17 Multilevel regression is a statistical technique that takes into account when data are nested (e.g., students are nested within schools). 

Failing to take into account nesting can result in incorrect statistical significance tests. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical 

linear models: Applications and data analysis methods (Vol. 1). Sage.

18 R is an open-source statistical software program and much of its utility is derived by installing packages that have a set of tools (functions) 

for performing data transformations and analyses. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 16

The model also used the following school-level variables: 

● Whether the school provided both WRC tutoring and other LAP ELA services or only other 

LAP ELA services. 

● The school’s percentage of students eligible to receive Free or Reduced-Price Lunch (FRPL). 

● The school’s percentage of students eligible to receive English Language Development services 

(EL/LEP students). 

Research Question 1b 

Before analyzing the data for Research Question 1b (To what extent does the effect of WRC tutoring on 

reading growth differ by student demographics?), the researchers hypothesized that FRPL-eligible 

students and students of color who received WRC tutoring would, on average, have higher reading 

growth than their counterparts (FRPL-ineligible students and white students who received WRC 

tutoring). That is, it was anticipated that the benefits of WRC tutoring would be greater for these 

traditionally-underserved students, relative to their peers. 

Research question 1b was answered by fitting the same multilevel regression models as in question 1a, 

above, but also included an interaction term between the ethnicity variable and the WRC treatment 

variable, as well as an interaction term between the FRPL variable and the WRC treatment variable. The 

introduction of the interaction terms allows for the ability to test if differences in months of reading 

growth occur as a function of both the treatment that students receive (a combination of WRC tutoring 

and other LAP ELA services or other LAP ELA services only) as well as the student’s demographic 

characteristics (i.e., low-income status, as indicated by eligibility for the FRPL program, or ethnicity). 

The next section of this report presents the results of these analyses. 

Results 

Research Question 1a 

The answer to Research Question 1a (To what extent do months of growth differ between Tier 2 students 

served by WRC tutoring and LAP ELA and Tier 2 students served by only LAP ELA?) is that no statisticallysignificant differences in months of reading growth were found between Tier 2 students who received 

WRC tutoring and other LAP ELA services (WRC + LAP ELA) and students who received only other LAP 

ELA services (LAP ELA ONLY). 

Among WRC + LAP ELA students and LAP ELA ONLY students, a similar percentage achieved at least 

10 months of reading growth: 

● Among the 1,840 WRC + LAP ELA students in the study, 45.2% achieved at least 10 months of 

reading growth. 

● Among the 3,617 LAP ELA ONLY students, 48.5% achieved at least 10 months of reading 

growth. 

Both of these percentages are within the expected range of success for students who have been 

assessed through pre-program screening as “below benchmark,” but not “well below benchmark” – i.e., 
 Washington Reading Corps Impact Evaluation, July 2019 Page 17

Tier 2 students.19. This data, however, shows no additional impact for WRC tutoring beyond the impact 

for LAP ELA over all. This lack of a demonstrated difference does not necessarily mean WRC tutoring 

does not have an additional impact: Challenges related to the way data is entered, flagged, and 

reported (as discussed in the “Limitations” section) may be obscuring the impact the program has and 

further study is needed. 

The authors fit a total of five different grade-specific multilevel models to test research question 1a. 

The results of the models fit suggested that, while WRC tutoring was unrelated to months of reading 

growth, two specific student characteristics were negatively associated with growth in reading: (1) 

student baseline scores, and (2) student eligibility for Free or Reduced-Price Lunch (FRPL). 

After controlling for the other variables in the model, both of these characteristics were found to be 

predictive of less reading growth: 

● Students who had higher baseline assessment scores achieved fewer months of reading growth 

than students who had lower baseline assessment scores. This relationship is likely explained by 

the fact that students with higher scores at baseline tend to have less room for improvement in 

their scores. 

● Students who were FRPL-eligible achieved fewer months of reading growth compared to 

students who were ineligible for FRPL. This relationship is likely explained by the lack of reading 

resources available outside the classroom to support growth for many FRPL-eligible children. 

Other demographic characteristics such as ethnicity, gender, and eligibility for English Language 

Development services (EL/LEP students) were generally not predictive of months of growth. For nearly 

every grade-specific model, the school-level variables (percentage of students eligible for FRPL and 

percentage students eligible for English Language Development services) were not statistically 

significant, suggesting that school-level demographics were generally unrelated to reading growth. 

Table 3, below, presents the results from the multilevel models used. In the table, only cells containing 

an asterisk (“*”) indicate that a variable had a p-value of < .05 (i.e., was a statistically-significant 

predictor of months of reading growth for a particular grade level). As the table shows, the treatment 

variable WRC + LAP ELA was not statistically significant for any of the grade-level models even after 

controlling for other student and school variables. 

Table 3: Regression Summary for the Multilevel Models Assessed, Research Question 1a 

To what extent do months of growth differ between Tier 2 students served by WRC 

tutoring and LAP ELA and Tier 2 students served by only LAP ELA? 

Variable Grade K1 Grade 1 Grade 2 Grade 3 Grade 4

(Intercept) 7.80* (1.34) 10.39* (1.18) 13.26* (1.29) 13.33* (1.15) 11.08* (1.38) 

Scaled Assessment Pre-score -1.26* (0.20) -0.05 (0.17) -0.26 (0.18) -0.70* (0.23) -1.03* (0.33) 

WRC + LAP ELA 0.47 (0.78) -0.12 (0.70) 0.19 (0.76) -0.02 (0.61) 0.25 (0.78) 

American Indian/Alaskan Native -2.00* (0.78) -0.47 (1.01) 0.45 (1.06) -1.04 (1.14) -0.98 (1.34) 

 

19 DIBELS Next Benchmark Goals and Composite Score. Dynamic Measurement Group, Inc. 2010. WRC serves Tier 2 students who have 

scored “below benchmark” on assessments and who, thus, have a 40% to 60% likelihood of achieving literacy goals.
 Washington Reading Corps Impact Evaluation, July 2019 Page 18

Variable Grade K1 Grade 1 Grade 2 Grade 3 Grade 4 

Asian -1.31 (0.97) -0.51 (1.07) 1.34 (1.09) 0.19 (1.45) 0.92 (1.33) 

Black/African American -1.05 (0.77) 0.16 (0.87) -0.12 (0.90) 0.11 (1.06) -1.88 (1.33) 

Hispanic/Latino of any race(s) -0.19 (0.36) -0.90 (0.47) -0.63 (0.48) -0.60 (0.56) -0.29 (0.67) 

Native Hawaiian/Other Pacific Islander 0.02 (1.12) -1.25 (1.92) 0.29 (1.79) -1.53 (2.18) 2.64 (2.11) 

Two or More Races -0.21 (0.51) -0.60 (0.65) 0.81 (0.70) 0.46 (0.79) -0.12 (0.97) 

Male -0.27 (0.24) 0.18 (0.31) 0.01 (0.32) -0.02 (0.39) 0.38 (0.46) 

EL/LEP (yes) 0.42 (0.34) 0.65 (0.45) 0.03 (0.44) 0.31 (0.52) -1.71* (0.63) 

FRPL (yes) -0.71* (0.28) -0.92* (0.39) -0.47 (0.41) -1.16* (0.50) 0.68 (0.57) 

School Proportion FRPL-Eligible Students 1.63 (2.51) -1.76 (2.24) -4.68 (2.51) -6.12* (2.12) -3.43 (2.64) 

School Proportion EL/LEP Students (Eligible for 

English Language Development Services) 

1.56 (2.87) 3.29 (2.55) -0.41 (2.81) 4.15 (2.25) 1.47 (2.82) 

Cell values are presented in the format Estimate-Significance-Standard Error. An “*” denotes statistical significance (p < .05). 

It is important to note that the schools included in the sample used a wide variety of different 

assessments to determine reading growth for students over the school year. Specifically, it is unclear 

whether the various assessments used across schools measure the same constructs, have similar 

psychometric properties, and/or perform similarly in their capacity to measure a student’s reading 

ability at baseline. The inconsistency in and lack of clarity around the constructs, psychometric 

properties, and performance of assessments poses a threat to the validity of this study and limits the 

conclusiveness of the results. This threat is discussed in more depth in the section Limitations on Data 

Analysis for the Quasi-Experimental Study, which starts on page 20. 

In order to mitigate this threat, the researchers performed a sensitivity analysis in which the sample 

was restricted to students who took the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) 

assessment. DIBELS was the assessment administered to the largest number of both WRC and nonWRC students in the sample. Eighty schools and 1,194 students (398 WRC students, 796 non-WRC 

students) were included in this sensitivity analysis. The results of this analysis also suggested that there 

was no relationship between receiving both WRC tutoring and LAP ELA and higher reading growth 

relative to receiving only LAP ELA. 

The researchers also analyzed the association between months of growth in reading and the length of 

time that elapsed between a student’s baseline score and their final score. The researchers assessed the 

relationship between reading growth and time elapsed between assessments by including a “time 

between assessments” variable in the grade-specific multilevel models fit for research question 1a. 

Even after controlling for time between assessments, the models indicated no relationship between 

receiving both WRC tutoring and other LAP ELA services and higher months of growth in reading 

compared to receiving only LAP ELA services. Appendix B presents bar charts and tables displaying 

average months of reading growth by specific demographic characteristics. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 19

Research Question 1b 

The answer to Research Question 1b (To what extent does the effect of WRC tutoring on reading growth 

differ by student demographics?) is that the effect of WRC tutoring on reading growth did not differ by 

student demographics. 

The results for the models fit for research question 1b closely resemble the results for the results for 

research question 1a. Baseline assessment score and FRPL eligibility were again generally predictive of 

months of growth in reading, where higher baseline scores were associated with reduced growth and 

FRPL-eligible students achieved fewer months of growth compared to FRPL-ineligible students. As in 

research question 1a, receiving WRC tutoring was not a statistically-significant predictor of reading 

growth, as no statistically-significant difference in months of reading growth was found between 

students who received both WRC tutoring and LAP ELA services compared to students who received 

only LAP ELA. 

Table 4, below, presents the results from the multilevel models used to answer Research Question 1b. 

Cells containing an “*” indicate that a variable was a statistically-significant predictor of months of 

reading growth for a particular grade level. As the table shows, the treatment variable (WRC + LAP 

ELA) was not statistically significant for any of the models. 

Table 4: Regression Summary for the Multilevel Models Assessed, Research Question 1b 

To what extent does the effect of WRC tutoring on reading growth differ by student 

demographics? 

Variable Grade K1 Grade 1 Grade 2 Grade 3 Grade 4

(Intercept) 4.17* (1.37) 10.39* (1.21) 12.60* (1.31) 13.20* (1.17) 6.77* (1.67) 

Scaled Assessment Pre-score -0.96* (0.20) -0.06 (0.17) -0.23 (0.18) -0.68* (0.23) -0.91* (0.33) 

WRC + LAP ELA 0.39 (0.86) -0.28 (0.95) -0.46 (1.01) 0.55 (1.00) 0.98 (1.22) 

American Indian/Alaskan Native -1.58 (1.15) 0.93 (1.85) 3.23 (1.67) 0.41 (2.26) -0.73 (2.28) 

WRC + LAP ELA: American Indian/Alaskan Native -0.89 (1.54) -2.55 (2.21) -4.57* (2.16) -2.33 (2.60) 0.42 (2.81) 

Asian -1.12 (1.34) -0.55 (1.22) 1.01 (1.14) 0.80 (1.83) -0.47 (1.52) 

WRC + LAP ELA: Asian -0.27 (1.90) 0.40 (2.54) 4.42 (4.26) -1.45 (3.01) 5.06 (2.92) 

Black/African American 0.60 (1.06) 0.34 (0.94) -0.08 (1.00) 0.78 (1.22) -2.21 (1.48) 

WRC + LAP ELA: Black/African American -2.87 (1.53) 0.35 (2.56) -0.47 (2.34) -1.76 (2.46) 1.13 (3.31) 

Hispanic/Latino of any race(s) -0.02 (0.44) -0.49 (0.54) -0.60 (0.57) 0.08 (0.67) -0.75 (0.77) 

WRC + LAP ELA: Hispanic/Latino of any race(s) -0.52 (0.68) -1.48 (0.92) -0.26 (0.90) -1.91 (1.07) 1.21 (1.30) 

Native Hawaiian/Other Pacific Islander 0.06 (3.33) -2.00 (2.05) 0.78 (2.28) NA 3.02 (2.22) 

WRC + LAP ELA: Native Hawaiian/Other PI 0.15 (3.53) 8.31 (6.05) -1.50 (3.68) NA -5.86 (6.68) 

Two or More Races -0.46 (0.72) -0.17 (0.77) 0.43 (0.85) 0.82 (0.99) -0.64 (1.23) 

WRC + LAP ELA: Two or More Races 0.43 (1.00) -1.39 (1.42) 1.10 (1.48) -1.01 (1.65) 1.06 (1.99) 

Male -0.27 (0.23) 0.17 (0.31) -0.03 (0.32) 0.02 (0.40) 0.41 (0.46) 
 Washington Reading Corps Impact Evaluation, July 2019 Page 20

Variable Grade K1 Grade 1 Grade 2 Grade 3 Grade 4

EL/LEP (yes) 0.51 (0.34) 0.74 (0.45) 0.04 (0.44) 0.44 (0.54) -1.74* (0.63) 

FRPL (yes) -0.83* (0.36) -1.28* (0.48) -0.74 (0.49) -1.36* (0.63) 1.10 (0.69) 

School Proportion FRPL-eligible Students 1.37 (2.33) -1.79 (2.25) -4.39 (2.49) -6.26* (2.12) -3.58 (2.60) 

EL/LEP Students (eligible for English Language 

Development Services)

1.70 (2.65) 3.38 (2.57) -0.24 (2.79) 3.85 (2.25) 2.29 (2.76) 

Months Between Assessments 0.57* (0.08) 0.00 (0.00) 0.09* (0.03) NA 0.55* (0.13) 

WRC + LAP ELA: FRPL flag Y 0.29 (0.56) 1.10 (0.84) 0.99 (0.86) 0.46 (1.04) -1.85 (1.20) 

Cell values are presented in the format Estimate-Significance-Standard Error. An “*” denotes statistical significance (p < .05). NA values 

indicate insufficient data. 

Appendix B presents additional charts and tables displaying average months of reading growth by 

treatment and by specific demographic characteristics. 

Limitations on Data Analysis for the Quasi-Experimental Study 

The analysis of data on the impact of WRC on reading growth in the quasi-experimental study has 

several major limitations, each of which is described below. A discussion of recommended changes to 

address these limitations appears in the section Recommendations, which starts on on page 29.

First Limitation 

Because of variation in the way schools collect data on the key outcome of interest for this study 

(reading growth), it was difficult to set a standard measure for that outcome. 

Washington is a “local control” state, meaning that each of its 295 school districts makes autonomous 

decisions about a wide range of policies, procedures, programs and curricula, and assessment and 

evaluation. Because of the local control structure, school districts have the freedom to determine which 

assessments they use to measure student performance, and the schools in this study used a wide range 

of different tools to assess student reading growth. Assessment tools used appear to not only measure 

different constructs (e.g., fluency, comprehension, etc.) but also to have different psychometric 

properties, and to use different score ranges to report growth. 

As a form of sensitivity analysis, the authors had hoped to select a subset of WRC and LAP ELA + WRC 

students from the sample who took a common reading assessment. Because of the wide variety of 

assessments used in the schools included in this study, however, it was not possible to identify a 

common reading assessment -- or even one that was used by a majority of schools in the study. 

Nineteen different assessments were used to measure reading growth for the students included in the 

study and no individual assessment was used by more than 20 percent of the schools studied, making it 

impossible to identify a reading assessment that could be considered “common.” 

Had it been possible to identify a common reading assessment, the researchers could have matched 

students based on that assessment, thus strengthening the power to detect a significant difference in 

the treatment effect. 

To address the issue of multiple assessment score ranges (due to the schools’ use of different reading 

assessments with different scoring protocols), the researchers converted the scores for all assessments 
 Washington Reading Corps Impact Evaluation, July 2019 Page 21

to the same scale, using a z-score transformation. The baseline assessment score variable was rescaled 

to have a mean of zero and standard deviation of 1 within each grade level and assessment 

combination. 

However, to the extent that all of the assessments used by WRC schools and comparison (non-WRC) 

schools measure different constructs (i.e., phonemic awareness, vocabulary, fluency, comprehension, 

etc.), the transformation to z-scores could adversely affect the credibility of the treatment effect 

estimate. In addition, issues related to variations in the treatment provided could also adversely affect 

that credibility (see “Third Limitation,” below).

This limitation points to the need for a common reading assessment for WRC sites. 

Second Limitation 

Because of inconsistencies in the way schools recorded, reported, and flagged data in the CEDARS 

database, it was difficult to ensure (a) that the assignment of students to the “tr eatment” or “control” 

groups accurately reflected whether they had actually received either WRC tutoring and other LAP ELA 

services or only other LAP ELA services and (b) that the growth score reported for each student 

accurately reflected the growth that student made. 

These challenges included the following 

(a) Lack of clarity, detail, and consistency in the data flagging system about which students 

received WRC tutoring and other LAP ELA services and which received only other LAP ELA 

services. 

In the State’s CEDARS data system, schools may be flagged as offering WRC tutoring, other 

LAP ELA services, or both. However, because WRC tutoring is considered to be a best practice 

as defined by the LAP ELA Menu of Best Practices (described earlier), it appears that some 

schools flagged as providing both WRC tutoring AND other LAP ELA services may in fact have 

offered WRC tutoring as their LAP ELA service. It is possible, therefore, that a student whose 

data flag indicated they received both WRC tutoring and other LAP ELA services may actually 

have received only WRC tutoring rather than WRC tutoring plus a different LAP ELA 

intervention. 

In addition, it is possible that the non-WRC schools included in the study may have used their 

LAP ELA funds to offer tutoring that is essentially similar to that offered by WRC but is provided 

by an adult other than a WRC Member. 

These misspecifications call into question the treatment effect estimate produced by the 

multilevel regression models, which used these data flags to identify which students were to be 

included in the “treatment” group (students who received both WRC tutoring and other LAP 

ELA services) and in the control group (those who received only other LAP ELA services). 

(b) Inconsistency and/or errors in data entry. 

In many of the student data files, the outcome of interest for this study, months of reading 

growth, had misspecified values entered. For example, in some records, students were labelled 

as having negative months of growth, even though their assessment scores showed gains 

from baseline assessment to final assessment. The misspecified values in the months of reading 

growth field likely contributed to a confounded treatment effect estimate. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 22

This limitation points to the need for more clarity around the flags that ind icate which services students 

receive as well as a way to prevent or rectify inconsistencies and errors in data entry. 

Third Limitation 

WRC directs all sites to provide tutoring one-on-one or in small groups at least three times weekly for at 

least 15 minutes each session, for a minimum of six weeks. Not all sites, however, adhere strictly to 

those directions all the time or with every student. Not only does the dosage provided by WRC 

Members tend to vary across schools, based on local logistics and student-identified needs, because 

Washington is a local control state (as discussed earlier) considerable variation also exists in the 

structure and content of tutoring.20 Given this variability, the services defined as “treatment” for WRC 

students was inconsistent across schools and, thus, the subsequent treatment effect estimate produced 

by the multilevel regression models may be misleading. 

In addition, even if all WRC schools used a common assessment and that assessment suggested impact 

for the “treatment” of WRC tutoring, the variability across sites in what WRC tutoring consists of would 

make it difficult to identify what is driving the impact. Any one of the following aspects of WRC tutoring 

could be correlated with impact: the tutoring model (e.g., pull-out or push-in), specific literacy and 

reading tutoring strategies, or simply having another adult supporting students’ literacy development. 

This limitation points to the need for more consistency in the ways in which sites implement the WRC 

model. 

Fourth Limitation 

Because the state does not require data on students who do not receive extra intervention to be 

reported to CEDARS, it is not possible to compare students who receive WRC tutoring with students 

who do not receive any LAP ELA intervention. Because the students who receive no intervention are 

not identified, it is impossible to create a “no intervention” group for comparison purposes.

This limitation points to the need to determine a way to identify a large group of students who receive 

WRC tutoring and a large group of similar students who receive neither WRC nor other interventions. 

Case Study of the Impact of WRC on Reading Culture 

The case study was designed to measure the impact of WRC at schools where WRC Members are 

placed and answer Research Question 2: To what extent does the work of WRC Members impact the 

culture of reading in the school and in the local community? 

To address this question, the study collected qualitative data about WRC Members’ activities and staff 

perceptions from a subsample of schools that have hosted WRC Members in the past to provide 

tutoring. 

 

20 Differences in tutoring structure include whether sessions are “push-in” (i.e., tutoring is provided in the classroom, concurrent with regular 

reading instruction) or “pull-out” (i.e., students are pulled out of the classroom for tutoring session) as well as whether tutoring is provided 

individually or in small groups; differences in content include both whether or not tutoring is aligned with a particular classroom curriculum 

and which of multiple core reading skills are supported (e.g., fluency, vocabulary, or comprehension); differences in dosage include variability 

in frequency, duration, and intensity of tutoring sessions. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 23

Data Collection 

To answer Research Question 2, the researchers designed an interview tool to gather qualitative data 

from Site Supervisors about WRC Members and their service. Designed in partnership with key staff 

members from WRC, the interview tool included questions for WRC Site Supervisors (the individuals 

responsible for overseeing WRC Members’ efforts) designed to elicit information about the specific 

services provided by the WRC Members placed at their site and their perceived impact. Interview topics 

included the role of WRC Members in (a) recruiting and training literacy volunteers from the 

community, (b) facilitating literacy events and activities, and (c) participating in other activities that 

impact the school’s reading culture. The interview questions are included in the Appendices, on page 

41. 

The Site Supervisor at each WRC school in the sample was invited to participate in a one-hour 

telephone interview with a researcher from the evaluation team. Invitations were sent via email and 

included a description of the evaluation and the purpose of the interviews. 

The Site Supervisors received a copy of the interview questions in advance in order to give them an 

opportunity to prepare thoughtful responses and to gather information from other staff involved with 

WRC Member activities. Using the interview questions as a guide, the researchers gathered information 

about the Site Supervisors’ perception of the WRC Members’ impact on student reading growth, on 

reading culture in the school and local community, and on family and community engagement with the 

school, among other topics. 

Interviews were conducted in November and December of 2018 and were recorded, with the 

permission of the interviewees, then transcribed, coded, and analyzed. After completion of the four 

interviews, the audio recordings were transcribed to Word documents. The content of the interviews 

was examined and emergent themes were identified. 

School Selection 

The researchers used stratified random sampling to select four schools from a population of 31 service 

sites across the state that had at least three years’ experience hosting a WRC Member to provide 

tutoring for students. 

The stratified random sampling procedure used geographical region as the strata, selecting one school 

from each region (East, Northwest, Seattle-Tacoma-Olympia, and Southwest). 

Analysis of Data

Research Question 2 was answered by analyzing the themes that emerged in the transcripts of the 

coded interviews. Because of the small number of interviews conducted, this analysis was able to be 

completed without the use of qualitative data analysis software. Instead, the researchers reviewed the 

transcripts, coded the content by themes, and created a table that summarizes responses from each of 

the interviewees for each of the themes. This table is included in the Appendices, on page 45. 

The themes that were included in the analysis included:

● Supervisor background (responsibilities at the site, longevity with the site, experience with 

WRC, and experience supervising members). 
 Washington Reading Corps Impact Evaluation, July 2019 Page 24

● Member activities (the percentage of time members provide tutoring, other literacy services, 

family engagement activities, and work with community volunteers, and details about each of 

those activities) 

● Tutoring details (the member:student tutoring ratio, the tutoring model, and other information 

about tutoring) 

● Alternative service and partnerships (whether members engaged in alternative service, what 

that service consisted of, and what external partners the members worked with) 

● WRC impact (supervisors’ perceptions of the extent to which members helped improve student 

outcomes, helped improve the experience of community volunteers, helped increase the 

number of community volunteers, helped teachers do their job, and/or helped improve parent 

knowledge and skills). 

● Impression of WRC (supervisors’ perceptions of the WRC program as a whole and their 

impression of how WRC is perceived by teachers, students and families, and external partners). 

● Impact on external partners (supervisors’ perceptions of whether WRC Members make existing 

partnerships stronger and/or create new partnerships or programs) 

● Other comments (supervisors’ perceptions about the burden of paperwork involved in hosting a 

WRC Member, other challenges, and other positive aspects of WRC) 

Results 

The answer to Research Question 2 (To what extent does the work of WRC Members impact the culture of 

reading in the school and in the local community?) is that Site Supervisors perceive a significant, positive 

impact for WRC on the culture of reading at their schools. The evaluators found many common 

responses regarding the themes of interest in the Site Supervisor interviews.

Most importantly, all Site Supervisors expressed a high regard for the WRC program and for the 

members placed at their sites. Their praise for the program was consistent and effusive. As illustrated in 

the “Sample Quotes” in the analysis chart in Appendix D, the program is considered either extremely 

helpful or indispensable by the Site Supervisors and Site Supervisors feel that members help create a 

more “literacy friendly” environment at their sites. Furthermore, Site Supervisors also expressed the 

belief that other site staff, parents, and students hold the WRC program overall and the WRC Members 

in particular in similar high regard. 

Site Supervisors also expressed the strong belief that WRC Members help improve reading growth for 

students (although they were able to offer only limited tangible evidence of that impact). It is important 

to note, however, that supervisor perceptions of improvements in student reading growth were not 

supported by the data analysis in the larger quasi-experimental study of impact on reading growth 

(described earlier in this report), which revealed no statistically-significant difference in months of 

reading growth between students who receive both WRC tutoring and other LAP ELA services and 

those who received only other LAP ELA services. This discrepancy is discussed further in the following 

section, Limitations on the Data Analysis for the Case Study. 

In addition, three of the four Site Supervisors cited the burden of paperwork as an issue and noted that, 

while they still felt having Members is worth their time and effort, all the paperwork requirements 

create difficulties. The same three Site Supervisors each also indicated that they were aware of other 
 Washington Reading Corps Impact Evaluation, July 2019 Page 25

schools in their district that decided not to apply for a Member because of the paperwork. The fourth 

Site Supervisor didn’t address this topic.

Other consistent themes: 

● WRC Members tutor students both 1:1 and in small groups on a consistent basis, while 

community volunteers typically provide 1:1 tutoring only (usually episodic, rather than 

ongoing). 

● WRC Members are both viewed as and treated as members of the school teaching staff – by 

teachers and parents alike. 

● WRC Members provide much-needed support for teachers. By working with specific struggling 

students, they help teachers regain time for other teaching tasks, including supporting students 

with the greatest reading challenges. 

While all Site Supervisors indicated that Members spend by far the largest amount of their time in 

direct tutoring services (between 75% and 90% of their time commitment), differences were noted in 

the amount of time Members spend on: 

● Other student-focused literacy supports, such as before and after school literacy activities. 

● Family engagement activities, including family literacy nights. 

● Volunteer management and recruitment. 

Sites also differ in: 

● The primary model used for tutoring. Two Site Supervisors indicated their sites use a pull-out 

model, in which students to be tutored are removed from the classroom for tutoring, and two 

indicated a push-in model, in which tutoring occurs in the classroom during classroom 

instruction. 

● Members’ level of autonomy in and responsibility for providing tutoring. At one site, Members 

help teachers and paraprofessionals plan and implement services; at others, they implement 

only those activities that teachers direct them to carry out. 

● Members’ level of autonomy in and responsibility for providing other literacy-focused student 

and family activities. At some sites, Members drive the planning and implementation process; 

others help implement activities that are planned by school staff or groups like the PTA. 

● Whether community volunteers also provide tutoring and whether Members are involved in 

recruiting and training community volunteers or in coordinating their work. 

● Whether Members engage with community partners, create new partnerships, or strengthen 

existing ones. 

One Site Supervisor noted the impact of the program on sparking Members’ interest in the teaching 

profession. The same Site Supervisor also indicated that having WRC helped reinforce the idea that the 

district was working hard to help students, making parents more likely to vote in favor of funding levies 

than if they thought the district wasn’t adequately addressing problems (“if you perceive that nothing is 

being done”).
 Washington Reading Corps Impact Evaluation, July 2019 Page 26

One Site Supervisor also noted a performance issue with one of the Members but indicated that having 

difficulties with Members is a deviation from the norm and another Site Supervisor indicated that they 

have had to release members from service in the past, but for the most part Members fit in really well at 

the school. 

In short, the following quotes encapsulate the feelings of the Site Supervisor interviews: We love the 

Washington Reading Corps program! You're really getting a good bang for your buck! 

Site Supervisor Interview Content Analysis details by site are provided in a chart in the Appendix. The 

code numbers and descriptions (code book) appear after the analysis chart. 

Limitations on the Data Analysis for the Case Study 

The analysis of data on the impact of WRC on reading culture has several major limitations, each of 

which is described below. A discussion of recommended changes to address these limitations appears 

in the section Recommendations, which starts on page 29. 

First Limitation 

The first limitation on data analysis for the case study was the timing of the interviews, which were 

required to be completed prior to the availability of data analysis from the larger quasi-experimental 

study. As a result, questions that might have been asked about the discrepancy between Site 

Supervisors’ perceptions of impact on reading growth and the results from the quasi-experimental 

study indicating no differences between WRC and non-WRC students were not included. In the absence 

of specific questions, difficult to speculate about how Site Supervisors might reconcile the discrepancy 

between their perceptions of student reading growth and what the data suggested. 

Second Limitation 

The second limitation on this analysis was the small number of Site Supervisors interviewed (four). 

Resource limitations meant that interviewing a larger pool of Site Supervisors was not feasible at the 

time the study was conducted. Because of the small number of Site Supervisors included in the case 

study, it is difficult to extrapolate findings to the majority of WRC sites or to know the extent to which 

the perspectives expressed in the four interviews are representative of the perspectives of the majority 

of Site Supervisors across the state. 

Discussion of Limitations 

A number of possible explanations exist for the discrepancy between Site Supervisor perception of the 

impact of WRC tutoring student reading growth and what the quantitative data suggests about that 

impact: 

(1) The impact of the program was hidden by the inconsistencies in data entry, data flagging, and 

data reporting outlined earlier (the “signal” of impact could not be heard through the “noise” of 

data inconsistencies). 

This possibility points to the need for improvements in data entry, data flagging, and data 

reporting to reduce noise, so that the signal of impact is clearer. 

(2) The perception of the four Site Supervisors interviewed may not be representative of the larger 

universe of Site Supervisors and the perceptions of the larger group of Site Supervisors may be 

more mixed and may more closely reflect the outcomes of the quasi-experimental study, which 
 Washington Reading Corps Impact Evaluation, July 2019 Page 27

did not establish a statistically-significant difference in reading growth between the treatment 

group and the control group in the quantitative study. 

This possibility points to the need to survey and/or interview the larger group of Site 

Supervisors about their perceptions of impact. Results from a survey could be used to design a 

targeted interview tool that delves more deeply into the question of perceived vs actual impact 

on reading growth and that gathers additional information about the value of WRC. 

(3) Time and resource limitations precluded separate analysis comparing actual reading growth for 

students at the four schools included in the interviews with Site Supervisor perceptions. The 

four schools represented in the interviews may have had student reading-growth outcomes 

that were different from the overall average in the larger quasi-experimental study. 

This possibility points to the need to compare quantitative and qualitative analyses by site, to 

determine whether there is a link between Site Supervisors’ perceptions and the reading 

growth of students at that particular site and, again, to delve more deeply into the question of 

value beyond reading growth if no link between perception of growth and data demonstrating 

growth is found. 

(4) Because of data-flagging issues, it is difficult to know what other interventions students 

received, in addition to WRC tutoring. However, because the analysis of quantitative data didn’t 

make these issues evident until after the interviews were conducted, interview questions did 

not ask about whether WRC tutoring was the only intervention students received. As a result, it 

is unclear whether students received other tutoring in addition to WRC tutoring or other nontutoring interventions aimed at improving reading growth. This lack of clarity makes it difficult 

to know if the impact Site Supervisors say they perceived resulted from a combination of 

interventions or WRC tutoring alone. In addition, because it is not possible to compare WRC 

tutoring to “no intervention,” Site Supervisors perceptions were likely based on what they 

believed would have happened for students in the absence of any intervention. 

This possibility points to the need (a) for future surveys and/or interviews that ask more indepth and targeted questions about other interventions and (b) to resolve issues around data 

collection, as discussed earlier. It also highlights the need to conduct a study that compares 

students who receive WRC tutoring with similar students who receive neither WRC tutoring nor 

another similar tutoring intervention. Such a study would require a method of data flagging 

that clearly identifies which interventions students are receiving. 

(5) The placebo effect is common in the social sciences, as in medicine. When people believe an 

intervention is supposed to have an impact, they will perceive that impact subjectively, even in 

the absence of objective data that supports their perception. Furthermore, the supervisors may 

have answered the question about impact on student reading growth as they were thinking 

more about the overall value of the program to their site. 

This possibility points to the need to design more specific survey or interview questions about 

program value to sites. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 28

Conclusions and Implications 

Conclusions 

In summary, this study resulted in three key findings: 

1) No statistically-significant differences in months of reading growth were found between students 

who received a combination of WRC tutoring and other LAP ELA services and students who 

received only other LAP ELA services. The results of the models suggested that neither receiving a 

combination of WRC tutoring and other LAP ELA services nor receiving only other LAP ELA services 

was predictive of students’ reading growth. 

2) The effect of WRC tutoring on reading growth does not differ significantly by student 

demographics, such as income status (as indicated by eligibility for the federal Free or ReducedPrice Lunch program) or ethnicity. The results of the models suggested no significant differences in 

the effect of the combination of WRC tutoring and other LAP ELA services compared with only 

other LAP ELA services by either student income status (eligibility for Free or -Reduced- Price 

Lunch program) or ethnicity. 

3) Site staff at schools where WRC Members are placed perceive a significant, positive impact for WRC 

on the culture of reading at their schools. Site Supervisors interviewed expressed a high regard for 

the WRC program and for the members placed at their sites. Their praise for the program was 

consistent and effusive. 

While the case study indicated that site staff perceive significant value in the service WRC Members 

provide, the results of the quasi-experimental study did not directly support those perceptions and the 

quasi-experimental study did not show an impact for WRC compared with LAP ELA. 

The reason researchers were unable to find any difference in months of reading growth between 

students who received a combination of WRC tutoring and other LAP ELA services and students who 

received only other LAP ELA services, may include one or more of the following: 

1) Variability in the tools that schools use to collect data on reading growth (the key outcome of 

interest for this study). 

2) Inconsistencies in the way data was entered, collected and/or flagged that masked the treatment 

effect (i.e., an effect exists but was undetectable because of corrupted or inconsistent data 

collection, reporting, or flagging processes) because it was difficult to ensure the accurate 

assignment of students to the “treatment” or “control” groups or to verify that the growth score 

reported for each student accurately reflected the growth that the student made. 

3) Inconsistencies in the tutoring models provided, which would make it difficult to establish causation 

even if an effect were detected. 

4) The absence of a “no intervention” control group, which made it impossible to compare growth 

data for students who received WRC tutoring with students who received no additional 

intervention. 

5) An actual lack of impact for the treatment (WRC tutoring) beyond the impact of other services 

students receive. 

These challenges point to the need for further study to determine which of the factors above may be 

true. However, until the study limitations outlined earlier are resolved, it may be difficult to design a 
 Washington Reading Corps Impact Evaluation, July 2019 Page 29

study that can credibly determine the extent of WRC’s impact on student reading growth or to 

determine which aspects of WRC tutoring may be more strongly correlated with reading growth. 

Implications 

The main implication of these conclusions is that, if WRC wants to accurately measure the impact of its 

tutoring interventions, the program should work to ensure each of the following:

● Less variability in the assessment tools used to measure reading growth. The best scenario 

would be to have one reading assessment tool that all WRC sites use. 

● More clarity around the flags that indicate which services students receive. The best scenar io 

would be to have a system that clearly indicates all of the reading interventions LAP ELAfunded schools provide (both WRC tutoring and other non-WRC LAP ELA interventions). 

● A better system to rectify/prevent errors in data entry, so that values that are outside the range 

specified by assessment tools cannot be entered. Particularly if only one assessment were used 

across all sites, it should be relatively easy to create an electronic data collection system that 

would limit data entered in a specific field to a specific range of values and that could flag for 

review any instances in which post-program assessment scores were lower than baseline 

scores. 

● More consistency in the ways in which sites implement the WRC model. The best scenario 

would be to have a single tutoring model (content, structure, and dosage) that all WRC sites 

implement. 

It would also be helpful to find a way to identify a large group of students who receive WRC tutoring 

and a large group of similar students who receive neither WRC nor other interventions. This 

identification may not be possible within the state’s longitudinal database, however.

A secondary implication of these conclusions is additional -- and more specific -- input from a much 

larger pool of site staff about the perceived value of the WRC program is needed. Furthermore, it would 

be helpful to compare differences in site staff perceptions with differences, if any, in student reading 

growth by site, to determine whether any correlation exists between each Site Supervisor’s perception 

of impact and reading growth for that site. 

This exploration could help determine what other student-level changes might be possible and valuable 

to measure in a separate quasi-experimental study regarding WRC Member impact (e.g., increased in 

overall school attendance, reductions in individual student absences, increases in students’ feelings of 

connection to school, reports of improved school climate, increased parent engagement, etc.). 

Recommendations 

The researchers make the following recommendations to WRC, in order to help more clearly drive and 

measure impact for WRC tutoring. 

Recommendations for Program Implementation and Data Collection 

(1) Standardize the WRC model and the assessments used to measure reading growth: 

● Identify a single valid, reliable, and easy-to-administer reading assessment and require 

all sites at which WRC Members are placed to use that assessment for all students who 
 Washington Reading Corps Impact Evaluation, July 2019 Page 30

receive WRC tutoring. Sites could continue to use other assessment in addition to the 

required assessment. 

● Establish a core WRC tutoring model that outlines a specific content, structure, and 

dosage for tutoring services and require all sites at which WRC Members are placed to 

use that model for all students who receive WRC tutoring. 

● Establish other relevant measures of success, in addition to months of reading growth 

(e.g., reduction in student absences or disciplinary referrals, increase in student feelings 

of connection to school, etc.), create assessment tools for those measures. 

(2) Create and require the use of an electronic system (a) for WRC site staff to report on all member 

activities and (b) for individual WRC Members and community-based volunteers to report on 

tutoring provided, including electronic entry of assessment data to minimize errors in the entry 

and reporting of student assessment scores. 

The specific attributes to be tracked in such a system should be decided in collaboration 

between WRC staff and other relevant parties (e.g., a software developer, an external 

evaluation consultant, and one or more site-based staff members), but could include items and 

events such as: 

● The duration, content, and format of tutoring, tracked at each encounter between a 

WRC Member and a student and at each encounter between a community volunteer 

and a student (at sites that use community volunteers to extend WRC tutoring 

capacity). 

● The timing of and results from administration of screening and progress monitoring 

assessments, as well as the name of the assessment tool(s) used. The system should 

prevent entry of values outside a defined range to ensure the validity of the pre- and 

post-intervention scores entered. 

● Other specific activities members engage in (in addition to tutoring) and the timing and 

duration of those activities. 

(3) Work with the administrators of the CEDARS longitudinal database to create clearer ways to 

distinguish which students receive WRC tutoring, which students receive only other LAP ELA 

interventions, and which students receive both. In addition, the system should specify the LAP 

ELA intervention, so that it is clear whether that intervention is non-WRC tutoring or some 

other type of intervention. 

(4) Work with OSPI leadership to identify a control group of students who receive neither WRC 

tutoring nor other reading interventions but who are demographically similar to students who 

do receive WRC tutoring in order to more clearly demonstrate the impact of WRC tutoring. 

Recommendations for Future Study 

(1) The researchers strongly recommend that WRC establish core tutoring model (consistent 

tutoring format, structure, content, and assessments) and pilot that model with a subset of 

program sites that agree to implement the model with fidelity. With a consistent model in 

operation across multiple sites, the researchers recommend designing and implementing a 

quasi-experimental study that would evaluate the impact of the standard model compared with 

the existing, non-standardized tutoring services WRC Members provide. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 31

Implementing such a study would necessitate the recruitment of a sufficient number of WRC 

sites to participate in the pilot (ideally at least 20 sites) and recruiting a similar number of WRC 

sites that would not implement the new model but would agree to use the core model’s 

assessment tool, either in place of or in addition to any other assessments used. With the 

second group of sites as a control, it would then be possible to compare the impact of the 

standardized WRC model to the impact of WRC’s non-standardized tutoring services. 

Creating a more standardized model (as Reading Corps programs in other states have done) 

provides a clearer way not only to measure the impact of a program but also to determine 

which particular aspects of a program are having the greatest impact. 

(2) The researchers also recommend either (a) changing the way WRC and other LAP ELA services 

are flagged in the CEDARS longitudinal data system or (b) creating some other reliable, 

electronic method to identify the specific interventions provided at the student level and then 

re-implementing substantially the same study 

 Being able to identify which students received WRC tutoring, other non-WRC LAP ELA 

tutoring, and other non-tutoring LAP ELA interventions would prevent students files from 

flagged in a duplicative way, which in turn would clear up confusion about which services have 

been provided to students and significantly reduce the threats to the validity of research 

outcomes, making it more likely that a second study could detect an impact for WRC tutoring, 

particularly if a single assessment was used across all sites studied. 

(3) In addition, the researchers recommend conducting a broader survey of site staff at all WRC 

sites, which would include questions from the interview tool created for the case study 

(described earlier) as well as about the following topics, with Likert-scale response values: 

● The format, content, and dosage of tutoring provided by WRC Members and by 

community volunteers (if any). 

● School-level data on reading growth as well as on other relevant student outcomes 

(student absences, feelings of connection to school, school climate, parent 

engagement, etc.). 

● Specific ways in which site staff perceive that WRC Members add value at their site and 

in their community. 

Comparing the results of those surveys on a site-by-site basis to measures of reading growth 

could help determine correlations between specific aspects of tutoring and months of reading 

growth as well as potentially establish a correlation between actual reading growth and site 

staff perceptions of WRC impact. 

The researchers could use the responses from these surveys to determine areas of deeper 

exploration and identify a subset of site staff to interview to gather further information to guide 

program improvement and/or future areas of study. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 32

Appendix A: Data Files and Variables Used 

Table 5: Data Files and Variables Used for Analysis 

Data File Variables Used 

Enrollment and 

Demographics 

Research ID: a unique ID assigned to each student 

School Code: a unique school ID 

Race: the race/ethnicity of the student 

Gender: the gender of the student 

Grade Level: the grade level of the student 

Primary School Flag: indicator of whether the school code associated with the 

student is considered their primary school 

School Enrollment Date: the date on which the student began school 

Cumulative Days Present: the total cumulative number of days the student has been 

present and in attendance in the 2017-18 school year 

School Programs WRC flag: indicator of whether or not the student received WRC tutoring 

LAP ELA flag: indicator of whether or not the student received other LAP ELA 

services 

Growth Progress monitoring assessment: unique assessment measuring student 

performance on key reading and/or literacy outcomes 

Academic months of growth: a measure of academic growth in months based on 

progress monitoring assessments 

Beginning score: the first raw score or scale score from the progress monitoring 

assessment 

End score: the final raw score or scale score from the progress monitoring 

assessment 

FRPL Eligibility Free or Reduced-Price Lunch (FRPL) eligibility flag: indicator of whether or not a 

student is eligible for the federal Free or Reduced-Price Lunch program 

EL/LEP English Learner/Limited English Proficiency (EL/LEP) flag: indicator of whether or not 

the student is eligible to receive English Language Development services as an 

English Learner 
 Washington Reading Corps Impact Evaluation, July 2019 Page 33

Table 6: Variables Used in Multivariate Matching for Student Selection 

Matching Variable Description 

Free or ReducedPrice Lunch 

Eligibility Flag 

Flag indicating if a student is eligible to receive Free or Reduced-Price Lunch 

(serves as a proxy for low-income status) 

Grade Level Indicator of a student’s grade level with possible values K1 (Kindergarten), 1 

(1st Grade), 2 (2nd Grade), 3 (3rd Grade), 4 (4th Grade) 

Assessment Prescore (scaled) Z-score produced from assessment pre-scores as a function of student grade 

level and the unique assessment. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 34

Appendix B: Student Descriptive Statistics and Average Months of Growth 

Overall 

Treatment N Average Growth SD

LAP ELA only 3671 9.66 6.55 

WRC + LAP ELA 1857 9.23 5.73 
 Washington Reading Corps Impact Evaluation, July 2019 Page 35

Gender 

Gender Treatment N Average Growth SD

Female LAP ELA only 1727 9.70 6.32 

Female WRC + LAP ELA 885 9.15 5.76 

Male LAP ELA only 1890 9.53 6.57 

Male WRC + LAP ELA 955 9.35 5.73 
 Washington Reading Corps Impact Evaluation, July 2019 Page 36

Race/Ethnicity 

Race Treatment N Average Growth SD

White LAP ELA only 1417 9.49 6.13 

White WRC + LAP ELA 887 9.57 5.77 

American Indian/Alaska Native LAP ELA only 52 9.13 6.27 

American Indian/Alaskan Native WRC + LAP ELA 168 7.86 5.79 

Asian LAP ELA only 102 9.84 6.31 

Asian WRC + LAP ELA 33 10.24 5.42 

Black/African American LAP ELA only 183 9.97 5.87 

Black/African American WRC + LAP ELA 47 8.23 4.77 

Hispanic/Latino of any race LAP ELA only 1599 9.66 6.84 

Hispanic/Latino of any race(s) WRC + LAP ELA 545 9.22 5.65 

Native Hawaiian/Other Pacific Islander LAP ELA only 33 9.76 6.69 

Native Hawaiian/Other Pacific Islander WRC + LAP ELA 20 7.90 5.35 
 Washington Reading Corps Impact Evaluation, July 2019 Page 37

Race Treatment N Average Growth SD

Two or More Races LAP ELA only 226 9.50 6.03 

Two or More Races WRC + LAP ELA 140 9.39 6.08 
 Washington Reading Corps Impact Evaluation, July 2019 Page 38

Free or Reduced-Price Lunch (FRPL) Status 

FRPL Eligible Treatment N Average Growth SD

No LAPELA only 983 9.98 6.24 

No WRC + LAP ELA 513 9.75 5.79 

Yes LAPELA only 2634 9.48 6.53 

Yes WRC + LAP ELA 1327 9.07 5.72 
 Washington Reading Corps Impact Evaluation, July 2019 Page 39

English Learner (EL)/Limited English Proficiency Status 

English Learner Treatment N Average Growth SD

No LAP ELA only 2372 9.46 6.29 

No WRC + LAP ELA 1365 9.31 5.73 

Yes LAP ELA only 1245 9.90 6.74 

Yes WRC + LAP ELA 475 9.10 5.79 
 Washington Reading Corps Impact Evaluation, July 2019 Page 40

Grade Level 

Grade Level Treatment N Average Growth SD

K1 LAP ELA only 824 9.64 6.07 

K1 WRC + LAP ELA 455 8.61 5.15 

1 LAP ELA only 844 9.43 6.55 

1 WRC + LAP ELA 415 8.76 5.33 

2 LAP ELA only 790 10.11 6.55 

2 WRC + LAP ELA 395 10.48 5.80 

3 LAP ELA only 640 9.31 6.42 

3 WRC + LAP ELA 324 9.19 6.11 

4 LAP ELA only 519 9.48 6.75 

4 WRC + LAP ELA 251 9.41 6.54 
 Washington Reading Corps Impact Evaluation, July 2019 Page 41

Appendix C: WRC Site Supervisor Interview Questions 

WRC Site Supervisor Interview Tool 

Final (October 8, 2018) 

1. Background 

Question: Tell me about your work at XXX school: what are your primary responsibilities? 

Follow up: How long have you worked here? 

Follow up: How many WRC Members have you worked with during your time here? 

Follow up: What kinds of literacy activities have these members been engaged in? 

 Probe to determine whether the members: 

 Provide literacy tutoring to students (academic improvement); 

 Coordinate and provide literacy-focused students and family engagement activities and events; 

and/or 

 Recruit and support literacy volunteers from the community to provide literacy tutoring and 

student and family engagement activities and events. 

Follow up: What is your estimate of the percentage of the WRC Members’ time spent on each type of 

activity? 

Follow up: What, if any, other activities did members engage in? If members have not engaged in any 

WRC literacy activities, do you know what kept them from engaging in the activities? 

2. Questions about WRC Members’ Work with Students and Families

As you respond to the next few questions think broadly about all of the WRC Members you’ve worked 

with over the past XX years. 

Question: Please describe the ways in which members work directly with students and families at your 

site. 

 Probe for specific activities: provide tutoring, organize reading groups, organize family literacy 

nights, 

 Probe timing: during the school day, before and/or after school, evenings, school breaks, etc. 

Question: How have members integrated literacy into existing events or activities, either during the 

school day, before or after school, or in the evenings (parent/family events and activities)? 

 Probe for specific examples 
 Washington Reading Corps Impact Evaluation, July 2019 Page 42

 Probe for the results of the members’ literacy integration work.

Follow up: What impact or effect did the Members at your site have these events and activities? 

Question: What role have members played in creating new literacy events or activities, either during 

the school day, before or after school, or in the evenings (parent/family events and activities)? 

 Probe for specific examples 

 Probe for the impact of the new literacy events or activities. 

Follow up: To what extent are these events and activities part of the ongoing life of your organization? 

Question: What Alternative Service, if any, have members engaged in during the time they have spent 

with you? 

Follow up (if necessary): Please describe the nature of that service. 

Follow up: What partnerships or improvements in programming, if any, resulted from this Alternative 

Service? 

Question:How have members supported relationship-building efforts with external partners or created 

or maintained partnerships with outside organizations? 

Follow up: What was the nature of those efforts? 

Follow up: What ongoing partnerships or improvements in programming, if any, resulted from these 

efforts? 

3. Questions about WRC Members’ Work with Literacy Volunteers

As you respond to the next few questions, think broadly about how all of the WRC Members you’ve 

worked with over the past XX years have worked with volunteers from the community who serve at 

your site. 

Question: Please describe how members have supported and/or contributed to literacy volunteering at 

your site. 

 Probe to determine the type of support or contribution (e.g., recruited volunteers, coordinated 

volunteers, supervised volunteers, trained volunteers, mentored volunteers). 

 Probe to determine the kinds of training members provide to tutors. 

 Probe to determine the ways in which they mentor tutors. 

 Probe to determine the kind of supervision they provide to tutors. 

Question: Please describe the literacy tutoring volunteers have provided for students at your site. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 43

 Probe to determine: peer-to-peer, cross-age, adult-led tutoring 

 Probe to determine: episodic or ongoing and approximate dosage (e.g., twice weekly) 

Question: Tell me about the other literacy activities these volunteers have engaged in, in addition to 

tutoring. 

 Probe to determine which, if any, of the following volunteers have engaged in: hosting literacy 

nights, organizing books drives, creating send-home reading bags, organizing literacy focused 

events with books stores, restaurants, etc. 

 Probe to determine: what would likely have happened in the absence of WRC Member placement 

(tutor fewer students or have no tutoring, have fewer or no literacy-focused events and activities). 

4. Questions about WRC Program and Member Impact 

Question: What impact or effect did the WRC program have on literacy tutoring and other literacy 

activities at your site? 

Question: What impact or effect did the WRC program have on students, parents, volunteers, other 

site staff, and the school as a whole? 

Follow up: What impact or effect did the WRC program have on students? 

Follow up: Tell me about the evidence you see for the impact or effect on students. 

Follow up: What impact or effect did the WRC program have on parents? 

Follow up: Tell me about the evidence you see for the program’s impact or effect on parents.

Follow up: What impact or effect did the WRC program have on volunteers? 

Follow up: Tell me about the evidence you see for the program’s impact or effect on volunteers.

Follow up: What impact or effect did the WRC program have on other site staff? 

Follow up: Tell me about the evidence you see for the program’s impact or effect on other site staff.

Follow up: What impact or effect did the WRC program have on the school as a whole? 

Follow up: Tell me about the evidence you see for the program’s impact or effect on the school as a 

whole. 

Question: Were members able to implement events or activities at your site that you would have been 

unable to implement if the members had not been placed at your sites. 

Follow up: If so, tell me about those activities and events and why you would have been unable to 

implement them without the member placements. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 44

Question: What impact or effect did the WRC Members’ engagement in Alternative Service have on 

the school or community? 

Question: Tell me about your view of how the WRC program as a whole is perceived by students, 

parents and family members, volunteers, and other site staff. 

 Probe to determine what interactions, conversations, or observations lead to the conclusions about 

how the program is perceived. 

Question: Tell me about your view of how individual WRC Members are perceived by students, parents 

and family members, volunteers, and other site staff. 

 Probe to determine what interactions, conversations, or observations lead to the conclusions about 

how WRC Members are perceived. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 45

Appendix D: WRC Site Supervisor Interview Content Analysis 

Region

Theme Southwest Northwest STO East Sample Quotes

1 Supervisor Background

1a Responsibilities Reading Specialist and 

Special Ed Teacher

Title One Literacy 

Teacher

Principal Reading Intervention 

Teacher/ Assessment 

Coordinator

I am a 2nd through 6th grade reading intervention teacher. I am also the 

district assessment coordinator. I sit on the child study committees of both the 

Elementary and the Junior High School/High School building and I am still ... 

consulted as curriculum director, although at this point in time we're not really 

following a curriculum adoption.

1b Longevity 30 years in District; 10 

years as Site 

Supervisor; Part Time

4 years in District; 30 

years as a teacher; 4 

years as Site 

Supervisor in 

Bellingham, 3 years’

experience as Site 

Supervisor in another 

district

12 years in District 28 years in District
 Washington Reading Corps Impact Evaluation, July 2019 Page 46

Region 

 Theme Southwest Northwest STO East Sample Quotes 

1c WRC Experience 20 years 7 years 5 years 20 years 

1d Member Experience 30 to 40 years total 12 years total (7 

Bellingham, 5 previous 

district) 

5 30 to 40 years total 
 Washington Reading Corps Impact Evaluation, July 2019 Page 47

Region 

 Theme Southwest Northwest STO East Sample Quotes 

2 Member Activities 

2a Tutoring Percentage and 

Details 

80%

Direct tutoring, mostly 

pull-out, mostly 1:1 

75% 

Direct tutoring, mostly 

in-class, mostly 1:1 

90% 

Direct tutoring, mostly 

pull-out, small group, 

and 1:1 reading 

80%

Direct tutoring 1:1 or 

small group tutoring, 

both in-class and pullout 

[Members work with students] on the sight words and on the reading passages 

and maybe having the text pre-read ... to give them that extra support ...

We have [the Members do] push-in ... within each grade level, to get a sense of 

what are the skills and some of the vocabulary that the teacher uses 

specifically in terms of literacy. Also looking at, at some behavioral 

management skills that are used within each of those classrooms...[Members] 

are supporting the classroom teachers and the students within those 

classrooms, trying to coordinate and collaborate rather than have them be 

taught things in isolation. 

2b Other Literacy 

Percentage and Details 

5%

Before and after 

school literacy groups 

Before and after 

school literacy 

workshops for parents 

Posting literacy videos 

for parents on the 

school website 

15% 

Before/ after school 

activities (Club de la 

Cultura, book clubs)

Create literacy 

materials for 

classrooms 

Help with summer 

school literacy 

activities 

Organize library 

materials, etc. 

6%

Before and after 

school literacy focused 

activities 

Help in the library 

during recess and after 

school 

Lunch group literacy 

(reading and games) 

Organize book drives 

13% 

Before and after 

school literacy focused 

activities (book 

groups, etc.) 

Reading contests and 

book drives 

Help with HOSTS 

program and 21st

CCLC reading tutoring 

Lunch group literacy 

[Members have implemented] before and after school literacy group activities. 

We usually focus on one grade level...this year... it's going to be focused on 

second and third graders. 

[Members have implemented] literacy workshops for our parents before and 

after school. Sometimes it will just focus on one grade level, like say 

kindergarten and we invite parents to come in to learn about how to teach your 

students phonemic awareness. So show them rhyming games or different 

things like that. So they can do it with their kids at home. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 48

Region 

 Theme Southwest Northwest STO East Sample Quotes 

2c Family Engagement 

Percentage and Details 

10% 

Family nights monthly 

with two literacy 

nights a year 

Members have 

responsibility for 

hosting literacy events 

and for building 

literacy into other 

events. 

10% 

Implement literacy 

activities at all PTA 

family events, but do 

not have primary 

responsibility for 

making events 

happen. 

1%

Provide literacy 

support for family 

activities 

Do some planning and 

implementing of 

literacy activities (like 

author nights), but 

mostly build literacy 

into existing activities 

5%

Provide literacy 

support for family 

activities – two larger 

literacy events + 

monthly family events 

Members very 

involved in planning as 

well as implementing 

This year our theme is “Polar Animals” and so every station is going to ... be 

focused on different, comparing and contrasting like the Orca and the blue 

whale and the kids will have little fact sheets that they fill out and also some 

fun crafty artsy activities to go along with it. 

[Members have] done literacy workshops for our parents before and after 

school. Sometimes it will just focus on one grade level, like say kindergarten, 

and we invite parents to come in to learn about how to teach your students 

phonemic awareness ... show them rhyming games or different things like 

that... they can do with their kids at home.

WRC Members coordinate with PTA... they attend every meeting and in every 

event that PTA puts on, they are WRC Members that add an element of 

literacy support...[For a recent pumpkin-carving event], WRC created a fall 

literacy resource book that had some suggestions of different high interest 

books per grade level...and how to help parents read at home with your 

students. 

Even if it's a math night or science night, they [WRC Members] always think of 

some literacy -related...activity that ....can be a part of the family night. 

2d Volunteer Percentage 

and Details 

5%

Recruitment, some 

training, some 

coordination or 

scheduling 

Recruit at Career Fairs, 

local high school, 

college, businesses, 

and naval base 

0% 3%

Recruitment, some 

training, some 

coordination or 

scheduling 

Joint training and 

placement of 

volunteers with UW 

2%

Recruitment, some 

training, some 

coordination or 

scheduling 

[Members] go talk to [local high school] students about the possibility of 

volunteering and also about possibly being Members at some point in those 

students' lives. They go to meet with Key Club and Honor Society at our local 

high school. They try to get there a couple times a year... we get a lot of 

volunteers that are school through that. 

A WRC Member from about three years ago ... developed that [training] 

PowerPoint for our volunteers...

It takes more coordinating with the classroom teachers, like finding out what 

time are you willing to take volunteers? So [Members] do a lot of that – writing 

down the times that [teachers need volunteers] and find like, students from 

the high school that get out an hour early to go help and first grade... The first 

couple of times the volunteers come, [Members] will meet them in the office, 

show them how to sign in ... introduce them to the teacher and the student, 

and then after that it becomes kind of the students just show up and come and 

know what to do. 

3 Tutoring Details 

3a Member Tutor Ratio Mostly 1:1, some small 

group 

Volunteers do only 

one-on-one 

Mostly 1:1, some small 

group 

Members do both 1:1 

and small group daily 

during school 

Volunteers do only 

one-on-one 

Members do both 1:1 

and small group daily 

during school 
 Washington Reading Corps Impact Evaluation, July 2019 Page 49

Region 

 Theme Southwest Northwest STO East Sample Quotes 

3b Member Tutoring Model Pull-Out with some 

Push-In

Ongoing tutoring, 

mostly outside of 

reading block, but 

linked to what’s being 

taught in the 

classroom 

Members help 

teachers/paras plan 

and tutors implement 

tutoring 

Push-In

Teachers/paras plan 

and tutors implement 

tutoring during 

“intervention blocks” 

throughout the school 

day every day 

Pull-Out 

Member tutor mostly 

through pull-out in 

small groups 

Pull-out reading with 

students 

In alignment with 

classroom teachers 

[Members] are considered part of our support services and we choose really 

carefully who they work with and what they do. And so there's times that ... 

students that are really close to having grade level skills... just an extra little 

boost and so we can have WRC do a book club with a group of students that. 

Or they're students that are really, really at risk and the classroom teacher 

provides instruction then we may have WRC follow up with those same kids, 

working with specific sight words or on certain targeted blends that the 

students might not have or specific skills that are very targeted to as a third 

boost for increasing their literacy skills. 

...students sometimes resent being pulled out to do some things, but the 

mentors work really hard to make their activities engaging and they choose 

books that the kids really like so I think that the students' overall perception is 

that it's a good thing and it's a fun thing. 

3c Other Tutoring Episodic 1:1 tutoring 

provided by volunteers 

Peer tutoring provided 

by 7th and 8th graders, 

mostly literacy games 

at lunch or recess 

None Episodic 1:1 tutoring 

provided by volunteers 

Episodic 1:1 tutoring 

provided by volunteers 

 

What our WRC [Members] do is they figure out the students that are going to 

be tutored, they talked to teachers and then help make the schedule. What the 

seventh and eighth graders do is give up ... one or two of their recesses a week 

on kind of an alternating schedule so that they're not all out every recess. The 

seventh and eighth graders then to come grab their folder and go meet with 

their students. And usually it's something really simple, like they're just 

working on sight words... playing maybe concentration with the sight words 

that the kids are working ... we've got all kinds of games like Candyland, 

Chutes and Ladders that are vocabulary games or sight word games. 

4 Alternative Service and 

Partnerships 

 

4a Alternative Service Boys and Girls Clubs, 

Kitsap Public Library, 

YWCA 

None; they work for 

the district doing nonstudent things during 

breaks (organizing 

library, etc.) 

South Sound Reading 

Foundation 

Centro Integral 

Educativo para 

Latin@s En Olympia 

(CIELO) – with bilingual 

students 

No formal alternative 

service because of the 

rural location 

Members tend to 

figure out literacyrelated projects on 

their own to do during 

breaks and summer 

[In the summer, Members have] gone to the Boys and Girls Club because we 

have a lot, of course students, that have been there -- their own students and 

also other students from our school that are part of Boys and Girls Club in the 

summer. So they're able to kind of carry on tutoring with them in those 

locations...
 Washington Reading Corps Impact Evaluation, July 2019 Page 50

Region 

 Theme Southwest Northwest STO East Sample Quotes 

4b External Partners Costco, Local High 

School, Local Library, 

Washington Youth 

Academy, Bremerton 

Naval Base, Olympic 

College 

None (want Members 

to focus on direct 

tutoring) 

South Sound Reading 

Foundation 

Centro Integral 

Educativo para 

Latin@s En Olympia 

(CIELO) – with bilingual 

students 

None (remote rural 

location makes it 

difficult) 

One of the librarians is ... bringing something to do with [family literacy nights] 

... She knows our theme is the Polar regions -- animals of the polar regions -- so 

she's bringing [books and materials about that] and she also will bring things 

for parents to fill out, information to get library cards, and she hands out free 

bookmarks and gives parents information about things going on at the local 

libraries...over Christmas break and things like that.

[Members have helped strengthen partnerships and make it so district 

continues to work with organizations even when Members aren’t there]

5 WRC Impact 

5a Improve student 

outcomes 

Helps with reading 

growth and attitudes 

toward reading 

Helps with reading 

growth and attitudes 

toward reading 

Helps with reading 

growth, attitudes 

toward reading, and 

behavior management 

Helps with reading 

growth and attitudes 

toward reading 

Improves attendance 

Just making that smaller one-to-one or small-group contact with those kids 

make such a huge difference because a lot of kids just need that extra little 

push and extra little cheerleader kind of sort of. I think the relationships are 

huge, too. Just another friendly face in the hallway or another person that 

comes in the room, that cares about them... makes them happier to be at 

school, I think, and helps them learn more because of that. 

[WRC Members] help those struggling readers [who] might not be struggling so 

much that they get as much help maybe from like a Title I or a Special Ed 

teacher. So, it's kind of helps fill that gap with the students that really do need 

help and they're able to make some good progress with them because their kids 

that sometimes have only lacked being at benchmark because of lack of 

instruction or nobody reads with them at home. 

Right now, we have a list of students that need support and there is no other 

way to give them that support because ...we are not a LAP School ... we're not 

a Title I school, we don't have a reading interventionist, we don't have 

anything like that in our schools, so the only thing that we have is the 

AmeriCorps [Members]. 

Those students that are receiving extra services with Reading Corps do tend to 

make gains, maybe not up to benchmark, but for the most part they've made 

at least 2 years’ [growth] ... and... then a lot of them show an increased good 

attitude towards reading [compared to] the beginning of the year. 

5b Improve volunteer 

experience 

Makes volunteer 

experience smoother 

and helps with training 

Not applicable Makes a Difference Makes volunteer 

experience smoother 

and provides support 

for volunteers 

[Members make volunteering] more personable ... somebody is meeting them 

in the office, somebody is remembering to send them a thank you note...having 

the kids draw pictures for the volunteers...
 Washington Reading Corps Impact Evaluation, July 2019 Page 51

Region 

 Theme Southwest Northwest STO East Sample Quotes 

5c Increase volunteer 

numbers 

Helps increase the 

number of community 

volunteers 

Not applicable Helps increase the 

number of community 

volunteers 

Volunteer program is 

strong and extra 

recruitment isn’t 

needed, so doesn’t 

really have an impact 

on numbers 

[Members are] making them feel really appreciated and I think that helps bring 

those volunteers from year to year. 

[There] wouldn't be as many volunteers [if Members weren’t there to] take 

some of the burden off the classroom teachers, to have that interim person to 

figure out what that student needs and help get materials ready for them and 

you know, that's hard for teachers to do. 

5d Help teachers do their job Gives teachers more 

time; takes some 

burdens off their plate 

Gives teachers more 

time; takes some 

burdens off their plate 

Gives teachers more 

time; takes some 

burdens off their plate 

Gives teachers more 

time; takes some 

burdens off their plate 

I know the teachers and the parents both appreciate the fact that students are 

getting help because like I say, they don't necessarily fit [Title One or Special 

Ed requirements]. [Teachers] appreciate that extra time they're getting with 

instruction. It takes a lot of take the pressure off the teacher, finding time to 

work with those students...it gives them just that extra opportunity of small 

group instruction, which is invaluable. 

5e Improve parent 

knowledge and skills 

Help parents learn 

how to support 

literacy at home 

Help parents learn 

how to support 

literacy at home 

Didn’t directly address Help parents learn 

how to support 

literacy at home and 

find books to read 

with their children 

So, I think helping give them the skills to help their kids at home and then 

parents appreciate that because they're not always aware of how to, how to 

help their students. 

[Members put together] just a really nice little booklet to help parents read at 

home with [their] students, with [their] kids. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 52

Region 

 Theme Southwest Northwest STO East Sample Quotes 

6 Impression of WRC 

6a Site Supervisors Wouldn’t be able to do 

some of the 

programming without 

them 

All programs would 

still happen, but 

wouldn’t be the same, 

not as deep, not as 

literacy focused. 

Summer school would 

be a watered-down 

version 

All programs would 

still happen, but 

wouldn’t be the same, 

not as deep, not as 

literacy focused 

All programs would 

still happen, but 

wouldn’t be the same, 

not as deep, not as 

literacy focused. 

Without our Washington Reading Corps [Members], we wouldn't be able to put 

on the family literacy nights the way that we do because they take hours and 

hours and, as the years go on, I find that it's harder and harder to get any 

teachers to become a part of it... because everybody's so overloaded.

[Members are] able to offer that extra thing for families in the community that 

may otherwise not [happen]. 

You're really getting a good bang for your buck if you're paying whatever your 

$6,000 or $7,000 for your Member. We love the Washington Reading Corps 

program! 

[Despite the paperwork burdens that made several other district schools decide 

to drop WRC], it was just not even a question. All that being said, all the 

requirements and all the things that we have to do for WRC organization is 

worth it because they, they are just a really important part of our staff. 

Yes, the impact is huge I am missing them terribly this year! The whole 

community notices that difference. (School had been unable to find Members) 

There isn't a lot of time [for staff] to do extras, but with the Reading Corps 

members being there ... they actually plan the activities that we're doing, then 

basically all I have to do show up to support them. They do the heavy lifting, so 

if we didn't if we didn't have Reading Corps that wouldn't happen. 

[Members] are a critical piece of the summer school program ... I don't know if 

[summer school staff] could literally pull it off without [the Members] ... they're 

a critical, critical piece. 

6b Teachers Teachers and families 

deeply appreciate the 

work Members do 

Teachers and families 

deeply appreciate the 

work Members do 

Teachers and families 

deeply appreciate the 

work Members do 

Teachers and families 

deeply appreciate the 

work Members do 

[Teachers] are so appreciative and they show a lot of appreciation for our WRC 

Members and they just treat them as part of the team. 

I just say overall, as a general summary of our schools attitude and 

appreciation for WRC, they're most beloved. 

From staff, I, I just think they're really valued, respected members of our staff 

and, and treated as such. We sometimes will joke about like we can't control 

the pay scale but we can sure make up for it with our appreciation for what 

they do. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 53

Region 

 Theme Southwest Northwest STO East Sample Quotes 

6c Students and Families Students and families 

perceive and respect 

Members as teachers 

Students and families 

perceive them as and 

respect them as 

teachers 

Parents very 

complimentary; 

appreciate the extra 

literacy support 

Members provide 

Parents very 

complimentary; 

appreciate the extra 

the literacy events and 

activities Members do 

We put out surveys ... at our family nights of course, and we get really positive 

feedback on those things.... 

Parents ... say, gosh, I hadn't even thought about how to, you know, read a 

story and have the kids make predictions, looking for words that they might 

know ... parents appreciate [Members’ help with] that because they're not 

always aware of how to, how to help their students. 

It's funny because even though we sent out the letters that your student’s 

getting tutored by Washington Reading Corps ... they’re kind of confused 

who's helping [the student] and they just think [the Members] are either 

volunteers or their employees of the school. So, it's hard sometimes to get 

some clarity out there, what they are. 

Parents notice that their students are making progress ... especially when the 

students graduate ... and they are ecstatic...

6d Partners Partners appreciate 

the support Members 

provide for their 

programs 

Not Applicable Partners appreciate 

the support Members 

provide for their 

programs 

Not Applicable The Boys and Girls Club ... last year, they called me and said are we going to 

get some of your members this year? So for the summer. They definitely 

appreciate that extra help. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 54

Region 

 Theme Southwest Northwest STO East Sample Quotes 

7 Impact on External 

Partners 

 

7a Make existing 

partnerships stronger 

Members help deepen 

relationships with 

partners 

Not Applicable Members help deepen 

relationships with 

partners 

Not Applicable [As a result of Members work with external partners], CIELO has been more 

integrated, not only with our school but the Olympia school district and that 

has been a really fabulous partnership... having members go and help [makes 

it stronger]. 

7b Create new partnerships 

or programs 

No Not Applicable No Not Applicable [While Members strengthen and support existing partnerships, they have not 

developed new partnerships that didn’t exist previously]
 Washington Reading Corps Impact Evaluation, July 2019 Page 55

Region 

 Theme Southwest Northwest STO East Sample Quotes 

8 Other Comments 

8a Burden of paperwork Burden is high, but is 

worth it for them 

Site Supervisor knows 

of other districts that 

have dropped WRC 

because of the 

paperwork and 

reporting burdens 

Burden is high, but is 

worth it for them 

Site Supervisor knows 

of other districts that 

have dropped WRC 

because of the 

paperwork and 

reporting burdens 

Not mentioned Burden is high, but is 

worth it for them 

Site Supervisor 

understands why 

schools drop WRC 

because of the 

paperwork and 

reporting burdens 

There's several schools in our own district that used to have Washington 

Reading Corps members and they just, they've kind of dropped it because 

somebody retired, you know, and nobody was willing to take it on or because it 

is a lot of work [paperwork for application, oversight, reporting]. 

Last year...there were several schools within our district that opted not to have 

WRC people because of the amount of paperwork and requirements and all the 

other things that go along with them when, you know, when whoever 

supervises them is a teacher as well and all our commitments. 

The federal program continues to add paperwork and time requirements ... as a

Site Supervisor, I get two people and a ton of extra duties added to an already 

overly-full schedule... that would be the thing that [might drive us to] 

discontinue the program just because, again, you can only squeeze so much 

blood out of a turnip. They do need they do need to understand that site staff 

at the schools that have reading Corps in place do not get any compensation ... 

for the extra work ... to have the program in place.

This summer, trying to get two people signed up, I probably put in 20 to 30 

hours on double checking paperwork and making phone calls and tracking 

people down and getting them to the right place for fingerprints and all of 

that. And that's my time off! 

It is a great program, but in the years that I've been doing it there have just 

been more and more demands placed upon me as the Site Supervisor, and 

more and more money that comes out of our district to pay for the position, 

which again, it's well worth it, but it is maybe going to reach a point where 

there are diminishing returns. I don't know...I haven't seen that yet, but it could 

get there. 

8b Other challenges None noted, other 

than paperwork 

burden 

Had a Member who 

wasn’t a good fit and 

had 

medical/behavioral 

issues 

None noted Have had to release 

Members from service 

because they weren’t 

a good fit 

We got the WRC supervisor’s help ... but it was like putting a round peg in a 

square hole ... it was very challenging when it's not a good fit. Part of that was 

on us because my vice principal did an interview, and just felt like, well if he 

made it through the WRC screening process maybe it's going to be okay. I just 

have to say that that was just super, super challenging when it, when it wasn't 

a good fit. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 56

Region 

 Theme Southwest Northwest STO East Sample Quotes 

8c Other positives None noted None noted None noted WRC gets people 

interested in 

education careers 

Having WRC provide a 

positive impression of 

school and district and 

may help when 

parents are voting on 

funding levies 

In the 20 years that I've been [working with WRC Members] I have had two 

members that have actually gone on and become teachers and one of them is 

currently a 2nd grade teacher in this building. I have two other members that 

have gone on and become para educators or substitute teachers while they are 

finishing up a program that they are going through on their [AmeriCorps] 

education award... I know that at least one other one is working in education 

but more in a central office position and then I also happen to know that we 

have some para pros working in our district that have been AmeriCorps 

Members ... the program does get people involved in the field of education. 
 Washington Reading Corps Impact Evaluation, July 2019 Page 57

Appendix E: WRC Logic Model 

NEEDS

(Community Problems)

INPUTS

(Project Resources)

ACTIVITIES

(Core Components)

OUTPUTS

(Implementation & 

Participation)

OUTCOMES (Evidence of Change)

Short-Term 

(Participant level)

Medium-Term 

(Program/Site level)

Long-Term 

(Population level)

In WA, the % of students who 

demonstrate age- and gradeappropriate language, literacy, 

and reading skills is especially 

low for low-income and 

English Learners (EL) 

students.*

(a) Entering kindergarteners 

with age-appropriate skills: 

Language skills

 All : 41%.

 Low Income:31%.

 EL:: 25%.

Literacy skills:

 All: 60%. 

 Low Income: 42%.

 EL: 42%.

(b) 4th graders who meet or 

exceed grade-level standards 

in English Language Arts:**

 Low-income: 42%.

 Not low-income: 75%.

 EL: 14%.

 Not EL: 65%.

WA has a limited pipeline of 

qualified teachers, early 

learning and elementary 

teachers.***

OSPI and WSC staff who 

provide leadership, 

guidance, training, TA,, and 

program coordination.

AmeriCorps members who 

tutor students,, recruit and 

engage volunteer tutors, 

and provide parent 

outreach and engagement.

Community tutors who 

provide additional tutoring

Program site staff who 

provide member 

supervision, support, and 

training in site-specific 

literacy and reading 

curricula.

Regional ESD staff who 

provide TA from literacy 

specialists.

WA’s Comprehensive 

Education Data and 

Research System 

(CEDARS), which collects, 

stores, and reports data on 

student-level enrollment, 

completion, and 

proficiency.

AmeriCorps Members: 

 Provide evidence-based reading 

tutoring individually and in small 

groups to (a) to K-4

th grade 

students below benchmarks in 

literacy and reading, through 

intensive 15+-minute sessions, 3+

times a week and (b) to preschool 

children through literacy activities 

throughout the day.

 Design, implement, and/or 

provide support for activities that 

help parents support children’s 

literacy and reading proficiency.

 Recruit and manage community 

volunteers to provide additional 

tutoring.

OSPI, WRC, and Program site staff:

 Provide evidenced-based training 

and ongoing support in literacy 

concepts, best practices in 

reading tutoring, and effective 

student engagement strategies.

 Provide hands-on classroom 

experience, supervised by 

certified educators a

 Provide access to other 

professional learning.

AmeriCorps members trained & 

hours of training.

Sites receiving Member 

placements.

Low-income K-4 students who 

are enrolled, screened, and 

tutored in reading.

Low-income preschool children 

who are enrolled, screened, and 

tutored in language and 

literacy.

Hours of reading/literacy 

tutoring provided .

Enrolled participants who 

complete the program.

Completing K-4 students who 

reach grade level proficiency 

OR make at least 11 months’ 

reading growth in one academic 

year.

Completing preschool children 

who demonstrate ageappropriate language and 

literacy development.

Family members who 

participate in literacy-focused 

school or home activities.

Community volunteer tutors 

recruited and trained.

Among completing 

participants, 

improvement in:

Reading attitude and 

behaviors, measured 

by post-program 

surveys.

Language and literacy 

skills, measured by 

pre- and postprogram assessments.

Reading proficiency or 

months of reading 

growth, measured by 

pre- and postprogram assessments.

Increases in family 

participation 

measured by parent 

surveys and event 

sign-ins.

Reports by WRC 

members that they 

intend to pursue 

career pathways into 

education, measured 

by surveys.

At program sites, 

increases in:

The % of children with

age-appropriate 

language/literacy skills, 

measured by benchmark 

assessments.

The % of students who 

achieve grade-level 

reading proficiency, 

measured by performance 

on state reading 

assessments.

The % of children and 

students who read outside 

of school and families who 

support reading at home, 

measured by surveys.

A decrease in disparities in 

reading proficiency, 

measured by performance 

on state reading 

assessments.

Reports by former WRC 

members that they 

started on career 

pathways in the field of 

education, measured by 

alumni surveys.

At schools statewide, 

increases in:

The % of students who 

meet or exceed 

proficiency benchmarks 

in English Language Arts, 

measured by 

performance on state 

assessments. 

The % of students who 

graduate high school on 

time, measured by state 

high school graduation 

data.

The % of high school 

graduates who enroll in 

and successfully 

complete postsecondary 

programs, measured by 

longitudinal data.

Reports by former WRC 

members that they have 

become teachers, 

helping address the 

state’s shortage of 

qualified teachers 

measured by surveys and 

state teacher data.

* Data from 2017-18 WAKids assessment of entering kindergartners’ skills in multiple domains of school readiness, showing the % of kindergarteners who demonstrate “the skills expected of 5-year-olds” in each domain. 

** Data from 2015-16 Washington State report card (https://washingtonstatereportcard.ospi.k12.wa.us/ReportCard) summarizing results of assessments of reading proficiency for 4th grade students that show the % of 

students who met or exceed basic grade-level proficiency in English Language Arts. *** Data from a 2015 survey of WA principals showing the state’s difficulty finding qualified teachers (45% were unable to find fullycertified and appropriately-qualified teachers for all classroom positions that needed to be filled; 80% employed classroom teachers using emergency certificates or long-term substitutes; and more than 90% indicated 

that they were “struggling” or in a “crisis” mode in finding qualified candidates.