Washington Reading Corps 

Implementation and Impact Evaluation

August 2016 2015–2016 Report

Prepared for

Washington Reading Corps

Office of Superintendent of Public Instruction

Olympia, WA 98504

Washington Service Corps

Employment Security Department

Lacey, WA 98503

Prepared by

RMC Research Corporation

111 SW Columbia Street, Suite 1030

Portland, OR 97201

iii

Contents

Exhibits......................................................................................................................................................... iv

Program Overview ........................................................................................................................................ 1

Program Implementation ............................................................................................................................. 3

Program Impact .......................................................................................................................................... 19

Summary and Recommendations............................................................................................................... 27

Appendix A Evaluation Methodology................................................................................................... 31

Appendix B HLM Model ....................................................................................................................... 37

Appendix C Washington Reading Corps Member Survey With Frequencies....................................... 38

Appendix D Washington Reading Corps Site Supervisor Survey With Frequencies............................. 47

Appendix E Impact Evaluation Summary 2014–2015.......................................................................... 53

Evaluation Questions

To what extent are the Washington Reading Corps programs being implemented? ............................... 3

To what extent are the Washington Reading Corps programs providing their projected 

program outputs?......................................................................................................................................... 3

What barriers or contextual factors are influencing the Washington Reading Corps program 

implementation?........................................................................................................................................ 11

What best practices can be identified in the Washington Reading Corps program design and 

implementation?........................................................................................................................................ 13

To what extent are these changes sustainable? ....................................................................................... 15

To what extent do student assessment scores differ between those served by Washington 

Reading Corps and those not served by Washington Reading Corps?..................................................... 19

To what extent are the Washington Reading Corps programs on track to show increases in 

students achieving grade level reading proficiency?................................................................................ 19

To what extent are the Washington Reading Corps programs on track to show decreases in 

reading proficiency gaps? .......................................................................................................................... 19

To what extent are the Washington Reading Corps programs meeting the 3 CNCS National 

Performance Measures? ............................................................................................................................ 20

What is the relationship between school characteristics, Washington Reading Corps member 

characteristics, and student reading skills?............................................................................................... 21

To what extent are the Washington Reading Corps programs on track to show increases in 

students meeting reading benchmarks? ................................................................................................... 22

To what extent are the Washington Reading Corps programs on track to show improvements 

in students’ attitudes, behaviors, and self-confidence in reading? ......................................................... 23

(see also Appendix E)
iv

Exhibits

Exhibit 1 Washington Reading Corps School Demographics.................................................................. 2

Exhibit 2 Average Minutes of Daily Tutoring .......................................................................................... 5

Exhibit 3 Usefulness of Washington Reading Corps Training Institute Topics ....................................... 6

Exhibit 4 Reported Receipt of Site-Specific Training............................................................................... 7

Exhibit 5 Relevance of Training Members Received............................................................................... 8

Exhibit 6 Washington Reading Corps Participation by Grade................................................................. 9

Exhibit 7 Program Participation............................................................................................................ 10

Exhibit 8 Effectiveness of Program Elements ....................................................................................... 14

Exhibit 9 Site Sustainability of Program Elements................................................................................ 16

Exhibit 10 Members Planning to Return in 2016–2017.......................................................................... 17

Exhibit 11 Factors Associated With Washington Reading Corps Members’ Experiences ...................... 18

Exhibit 12 National Performance Measures Results............................................................................... 21

Exhibit 13 Descriptive Statistics for Predictor Variables......................................................................... 21

Exhibit 14 Achievement of Reading Benchmarks................................................................................... 22

Exhibit 15 Change in Student Attitudes.................................................................................................. 23

Exhibit 16 Reading Behavior Improvement by Grade............................................................................. 24

Exhibit 17 Change in Reading Behaviors................................................................................................. 25

Exhibit 18 Change in Other Reading Behaviors ...................................................................................... 25

Exhibit 19 Changes in Confidence in Reading......................................................................................... 26
RMC Research CorporationPortland, OR 1 

Program Overview

The Washington Reading Corps places AmeriCorps members in public schools and early learning sites 

with the goal of supporting the development of foundational and comprehensive skills of struggling 

readers from pre-Kindergarten through Grade 6. The program seeks to achieve that goal through 

effective collaboration involving schools, community organizations, families, community members, 

business and state partners, and AmeriCorps. Funded by the Corporation for National and Community 

Service (CNCS), at the state level the Washington Reading Corps (WRC) is a partnership between the 

Washington Service Corps (WSC) and the Office of Superintendent of Public Instruction (OSPI). The 

Washington Reading Corps has been supported since 1998 by the Washington State Employment 

Security Department and SERVE Washington, the Washington Commission for National and Community 

Service. 

To improve students’ reading skills, Washington Reading Corps members provide (a) one-on-one or 

small-group tutoring using research-based reading methods, (b) reading support for families through 

outreach and literacy activities, and (c) recruitment and training of community volunteers to expand the 

program’s impact. The program logic model and theory of change developed by the Washington 

Reading Corps appear in Appendix A. In 2015–2016 sites could apply to have members either at early 

learning sites or at Kindergarten through Grade 6 sites. 

In 2015–2016 the Washington Reading Corps placed 153 members (with 117 retained) in 8 early 

learning centers and 69 schools across 40 school districts in Washington State. According to the OSPI 

website,

1 the participating schools had an average total enrollment of 367 students (range 86–695). 

Data available for 66 of the 77 sites indicated that the majority of students were White or Hispanic and 

qualified for free or reduced-price meals (see Exhibit 1). Teachers had an average of 13 years of teaching 

experience (range 0–21 years).

1Data source: http://reportcard.ospi.k12.wa.us/DataDownload.aspx
2 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit 1 Washington Reading Corps School Demographics

The majority of the students in Washington Reading Corps schools were 

White or Hispanic and qualified for free or reduced-price meals. 

Hispanic 24%

(range 2–91%)

American Indian/American Native 6%

(range 0–94%)

Asian American 4%

(range 0–29%)

Black 3%

(range 0–23%) n = 66

FRPM 61%

(range 18–95%)

Special education 17%

(0–93%)

Transitional bilingual 12%

(0–58%)

White 55%

(range 0–89%)

RACE/ETHNICITY

SPECIAL PROGRAM ELIGIBILITY
RMC Research CorporationPortland, OR 3 

Program Implementation 

The implementation evaluation addressed program outputs for tutoring implementation, member

professional development and training, student tutoring participation, member involvement in family 

literacy events and volunteer recruitment, barriers and best practices, and site sustainability of 

Washington Reading Corps activities. The evaluation questions and their data sources are in Appendix A 

(Exhibit A1). Data were synthesized from various sources:

Tutoring Implementation and Outputs

Evaluation Question

To what extent are the Washington Reading Corps programs 

being implemented? 

Evaluation Question

To what extent are the Washington Reading Corps programs 

providing their projected program outputs?

Sites Served

In 2015–2016, 77 sites in Washington State participated in the Washington Reading Corps program, 

above the target of 75. These sites included 69 elementary schools and 8 early learning centers. Initially 

153 Washington Reading Corps members participated with 39 returning members (33%), and 117 

members were retained during the year. Most members (86%) began serving in September 2015 and 

the remainder (14%) began in October 2015. Sites had between 1 and 5 Washington Reading Corps 

members.

Core Reading Programs and Tutoring Programs

As part of their 2015 application for the Washington Reading Corps program, sites provided brief 

descriptions of their core reading programs and intervention materials, which included such programs as 

Good Habits, Great Readers; Engage New York; Journeys by Houghton-Mifflin; Treasures Reading 

Curriculum; Fountas and Pinnell; Imagine It; Open Court; Benchmark Literacy; and Read Well, among 

others. According to site supervisor feedback, members typically received initial training on their site’s 

core reading program from multiple staff members, including the Washington Reading Corps site 

supervisor (66%), the site’s literacy specialist (59%), the Title I and/or Learning Assistance Program 

teacher (57%), and the classroom teacher (48%).

SS

STL

MS

PD Program data

Student tracking logs

Washington Reading Corps member survey (see Appendix C)

Washington Reading Corps site supervisor survey (see Appendix D)

PD

PD
4 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

In addition to core reading instruction, students identified for Tier II services receive supplemental 

tutoring in reading through the Washington Reading Corps program. Some of the tutoring approaches 

proposed by site applicants included Leveled Literacy Intervention (Fountas and Pinnell), Read Naturally, 

Reading Mastery, Read Well, Road to the Code, Wonder Works, Early Reading Intervention, and 

Read 180. 

Tutoring Implementation

From the feedback collected from the Washington Reading Corps member survey and the site 

supervisor survey 4 main implementation findings emerged:

 The majority of Washington Reading Corps sites implemented key aspects of the tutoring

model. On a scale from not at all to to the full extent, at least 75% of Washington Reading Corps

member respondents (n = 74) said that implementation was to the full extent for (a) 20-minute

tutoring sessions with Tier II intervention students 3 times a week (75%) and (b) small-group

tutoring with no more than 6 students in each session (76%).

Almost all site supervisor respondents said members provided small-group tutoring (92%), most

(77%) said members provided one-on-one tutoring, and 70% said members implemented both

tutoring formats at their site. Roughly 75% of site supervisors reported that members were

involved in before- or after-school reading programs.

 Schools used a combination of push-in and pull-out delivery models for Washington Reading

Corps tutoring. Many site supervisor respondents (79%) indicated that members followed a

pull-out model (students leave class to work with the tutor), and 64% reported using a push-in

model (the tutor goes into classes). According to 21% of the site supervisor respondents

members were embedded in a classroom. Early learning centers used a range of delivery

models, similar to school sites.

 Site supervisors strongly endorsed the usefulness of tutoring provided by a Washington

Reading Corps member. More than 80% of Washington Reading Corps site supervisor

respondents described tutoring as very or extremely useful at their site. A majority cited data

from reading assessments or student growth in reading as evidence of the usefulness of

Washington Reading Corps tutoring. The ability to assist many more students was another noted

strength.

 The majority of Washington Reading Corps members served at their site in ways consistent

with their role. Most members had not been asked to serve in ways beyond their designated

role, but a few reported being asked to provide general assistance in the classroom (n = 8);

assist in nonreading subjects (n = 5); or fill in for recess, bus, library, or unrelated school

volunteer duties (n = 4).

Tutoring Minutes and Days Provided by Members 2012–2016

Using the student tracking logs, site supervisors (2015–2016) and members (2012–2015) recorded the 

average number of minutes per day and average number of days per week members provided tutoring 

to students in Kindergarten through Grade 6. The evaluation team compared these data across 4 years. 

As Exhibit 5 shows, on average sites exceeded the target of providing 20 minutes of tutoring per 

STL

MS SS
RMC Research CorporationPortland, OR 5 

session in 2012–2013, 2013–2014, 2014–2015, and 2015–2016.

2 Sites also provided tutoring slightly 

more than 3 days per week (average of 3.5 days in 2012–2013, 3.4 days in 2013–2014, 4 days in

2014–2015, and 4 days in 2015–2016).3

Exhibit 2 Average Minutes of Daily Tutoring

On average students received more than 20 minutes of tutoring per day.

Member Professional Development and Training

In 2015–2016 Washington Reading Corps member training included a 3-day training institute in the fall, 

monthly professional learning cluster meetings led by Washington Reading Corps members and regional 

trainings led by Educational Service Districts and Washington Reading Corps coordinators, and on-site 

training and coaching from school and early learning site staff. 

2015 Washington Reading Corps Training Institute

In October 2015 Washington Reading Corps conducted the Training Institute, a 2-day required training

for new and returning members. Speakers included school district and Educational Service District staff, 

OSPI staff, and external presenters from United Way of Chelan/Douglas Counties, Central Washington 

University, Whatcom Dispute Resolution Center, and elsewhere. Content included the fundamentals of 

reading, effective communication skills, student behavior management, cultural competency and 

inclusion, early childhood development, and early learning teaching strategies (targeted to members 

placed in early learning centers). 

2Minutes per day: n = 7,174 students in 2012–2013, n = 2,417 students in 2013–2014, n = 3,970 in 2014–2015, n = 3,550 in

2015–2016. 

3Days: n = 7,171 students in 2012–2013, n = 2,404 students in 2013–2014, n = 3,987 in 2014–2015, n = 3,551 in 2015–2016. 

MS

 

PD
6 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

For the 2015–2016 program year the Training Institute included a new presentation to members about 

best practices in recruitment and retention of community volunteers. Each member received a 

volunteer recruitment workbook with information about the volunteer program at the member’s site

and guidance on assessing teacher and staff needs for volunteers; connecting with past volunteers; and 

training, tracking, and communicating with volunteers. Other Training Institute topics included a 

member orientation to Washington Reading Corps program requirements and expectations and brief 

presentations on data collection methods, reading with English language learners, extended learning

opportunities, family literacy engagement, and reading activity toolkits. 

Washington Reading Corps members (about 43%) rated the Training Institute topics as useful or very

useful (see Exhibit 3). Whereas instruction on the fundamentals of reading was considered most useful, 

Fish! Philosophy Training received mixed reviews with 52% of members rating it as either very or 

extremely useful and 35% rating it as not or somewhat useful. Conversely, members reported that 

training on volunteer recruitment and management was least useful. 

Exhibit 3 Usefulness of Washington Reading Corps Training Institute Topics

Usefulness of Washington Reading Corps Training Institute Topics

Topic

Not

Useful

Somewhat 

Useful Useful

Very

Useful

Extremely 

Useful

I did not 

receive 

training on 

this

Fish! Philosophy Training 12% 23% 8% 24% 28% 4% 

Fundamentals of Reading 3% 12% 30% 23% 26% 5% 

Student and Behavior 

Management

7% 18% 22% 23% 19% 11% 

Cultural Competency and 

Inclusion

10% 15% 27% 19% 17% 12% 

English Language Learners 4% 14% 22% 19% 14% 27% 

Activity Toolkits 6% 12% 28% 22% 10% 21% 

Data Collection Methods 8% 21% 28% 13% 8% 22% 

Extended Learning 4% 12% 30% 12% 8% 34% 

Family Engagement 5% 14% 30% 17% 8% 26% 

Early Learning 3% 10% 26% 7% 6% 47% 

Principled Negotiation 7% 15% 22% 14% 6% 35% 

Volunteer Recruitment/Mgmt 15% 16% 30% 13% 6% 20% 

Note. n = 92–100. Colored text reflects trainings more members rated as most useful or least useful.

Professional Learning Cluster Meetings

Starting in 2015–2016 members were offered member-led professional learning cluster meetings. These 

meetings occurred monthly from December 2015 through July 2016 and included a sustainability binder 

project, member reflection project, Martin Luther King Jr. Day service project development, and several 

assignments related to professional development, such as development of an “elevator pitch” about 

member service and practice interviewing. Also added in 2015–2016 were 3 regional literacy trainings 

PD MS
RMC Research CorporationPortland, OR 7 

provided by OSPI, Washington Reading Corps, and Educational Service District staff for members in 

November 2015 and February and May 2016. Regional Washington Reading Corps coordinators 

organized trainings and activities with each regional ESD coordinator, including a LETRS training 

(ParaReading), a supplemental reading module based on the National Reading Panel’s recommendations 

for research-based instruction.4 However, in contrast to other types of professional development, the 

quality of regional professional learning cluster meetings and monthly member-led professional 

learning cluster meetings received mixed reviews by members (see Appendix C).

Site-Specific Training

In their applications, prospective sites described plans for site-based training. Each site was expected to 

train Washington Reading Corps members on certain topics. At least 62% of 110 Washington Reading 

Corps member survey respondents received a training on the site safety plan, orientation to the site, 

how students learn to read, tracking student progress, and where to direct questions. Training on 

other topics was reported less frequently (see Exhibit 4). 

Exhibit 4 Reported Receipt of Site-Specific Training

Reported Receipt of Site-Specific Training

Topic Members

Site safety plan 76% 

Orientation to site 65% 

How students learn to read (critical components of reading) 64% 

Tracking student progress 62% 

Where to direct questions 62% 

Tutoring/intervention program 58% 

Site culture (expectations for conduct) 58% 

Student behavior management 52% 

Communication plan for accessing school/site staff 45% 

Family literacy engagement plan 42% 

Strategies for diverse learners 38% 

How tutoring connects with classroom instruction 35% 

State English Language Arts (ELA) learning standards 32% 

Strategies for English Language Learners or bilingual students 29% 

Community volunteer recruitment or management plan 29% 

Note. n = 110. Colored text reflects site-specific training topics received by at least 60% of members.

Strategies for bilingual students or English language learners (ELL). About a third of site supervisor

respondents said their site provided some Washington Reading Corps member training to support

4National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel. Teaching children to 

read: an evidence-based assessment of the scientific research literature on reading and its implications for reading instruction.

Retrieved May 19, 2015, from http://www.nichd.nih.gov/publications/nrp/smallbook.htm

PD MS SS
8 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

reading skills of ELL or bilingual students. About half of those described coaching, demonstrations, or 

weekly conversations with members specifically related to supporting bilingual and ELL students. At 

several sites the ELL teacher provided training or coaching to members. At several sites members were 

provided with Guided Language Acquisition Design (GLAD) strategies from a model designed to support

ELL students in a mainstream classroom.

Ongoing supervision or coaching of members. Almost every site reported ongoing supervision or 

coaching of Washington Reading Corps members. More than 70% of site supervisor respondents rated 

ongoing coaching or supervision of members as a very or extremely effective element of the 

Washington Reading Corps program (see Exhibit 8). Site supervisors described:

 Daily check-ins or weekly or biweekly meetings (n = 23 respondents). 

 Coaching from the classroom teacher, Title I or Learning Assistance Program teacher, principal, 

or literacy coach (n = 21). 

 Member participation in staff trainings, ESD and Early Childhood Education and Assistance 

Program trainings, reading-specific trainings, or grade level collaboration meetings (n = 12). 

 Supervisor or staff observations of members during tutoring and feedback (n = 4). 

 Supervisor being housed in office space with members (n = 4). 

Quality and Relevance of Training

Members rated the quality of all types of Washington Reading Corps training they received in 

2015–2016 from any source. Of those who were offered and attended different types of training 

(n = 26–104), at least 47% of members rated the quality as very good or excellent for:

 Professional development with school staff (64%). 

 Washington Reading Corps Training Institute (51%). 

 Site-specific Washington Reading Corps trainings (47%). 

 Other professional development (47%). 

Members also rated the relevance of all of the Washington Reading Corps training and professional 

development they received in 2015–2016 (see Exhibit 5). 

Exhibit 5 Relevance of Training Members Received

The majority of the members considered the training they received to be relevant or 

very relevant to their work.

 

MS
RMC Research CorporationPortland, OR 9 

Student Participation

Washington Reading Corps Program Participation

Exhibit 6 shows the number and percentage5 of students in the targeted grades at active sites who were 

enrolled, screened, and tutored in the Washington Reading Corps program at any time during 

2012–2013, 2013–2014, 2014–2015, and 2015–2016. 

Exhibit 6 Washington Reading Corps Participation by Grade

Washington Reading Corps Participation by Grade

Year Total K G1 G2 G3 G4 G5 G6

2012–2013 7,199 28%a 18% 17% 13% 10% 10% 3% 

2013–2014 2,417 41%a 16% 13% 10% 10% 9% 1% 

2014–2015 4,008 27%a 22% 19% 14% 9% 7% 2% 

2015–2016 3,602 20% 21% 20% 15% 13% 9% 2% 

Note. Percentages may not sum to 100 due to rounding. Total number of tutoring participants and grades based on student 

tracking logs. Percentages in 2015–2016 based on 3,420 students with a grade recorded in the student tracking logs. aMay include some pre-K children.

Tutoring Program Participation and Completion

Program completion is defined as either participating in tutoring through spring or meeting the reading 

benchmark during the year and thus exiting the Washington Reading Corps program. Across all years, 

the numbers of students still receiving Washington Reading Corps tutoring at the end of the year

(defined as between May 16 and June 30 for 2012–2014 and between May 1 and June 30 for 2014–2015

and 2015–2016) and students who met the benchmark and thus completed the program earlier in the 

year6 were obtained from student tracking logs (see Exhibit 7). Because students whose benchmark 

status or program end date was not recorded were not included in this analysis, Exhibit 7 likely 

underrepresents student participation in Washington Reading Corps. 

 

5A total of 15% of student data had either no pretest date or a date outside of 2012–2014. These students were not removed 

from 2012–2013 and 2013–2014 analyses, but this finding suggests potentially broader issues with missing data or the quality 

of data entry.

6In 2012–2013 and 2013–2014 the variable for met benchmark was combined with improved at least one grade. In 

2014–2015 tracking logs, these were 2 separate variables. In 2015–2016 tracking logs included met benchmark and “Did 

student improve?” as 2 separate variables.

STL

STL
10 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit 7 Program Participation

Since 2012–2013 between 70% and 80% of Washington Reading Corps 

students were participating at year end. 

Literacy-Focused Family Activities and Events

In addition to tutoring students, Washington Reading Corps members were expected to be involved in

literacy-focused activities and events for parents and families. The Washington Reading Corps reported 

374 events through June 2016 with 27,239 attendees. Family literacy nights were common events. 

Other examples of family events included “Dr. Seuss night,” ELL Parent Night, “Books and Breakfast,” 

book fairs or book giveaways, and a spelling bee. 

Members predominantly served in Family Literacy Nights. The vast majority of Washington Reading 

Corps members surveyed (90%) said they were involved with at least 2 family literacy events during the 

academic year—primarily through family literacy nights. Other member-reported activities included 

providing literacy resources (n = 21) and organizing volunteers (n = 5). According to site supervisor

survey respondents, 95% said members attended family literacy events and at least 80% said members 

planned events, developed materials, promoted, and facilitated family literacy events. 

Almost all sites (95%) measured results of family literacy activities or events based on the number of 

families who attended or participated. Roughly half of sites measured results based on teacher or staff 

feedback after literacy events (58%) or whether more families got involved in school activities (49%). 

Finally, 36% of sites considered whether students showed more interest in reading after events. 

Community Volunteers

To help schools achieve their literacy goals, Washington Reading Corps members were expected to

support recruitment of community volunteer tutors for before- and after-school programs, enhance or 

create community volunteer recruitment systems, or develop or expand existing volunteer recognition 

program. Such activities did not always align with sites’ needs; for example, some sites already had a 

staff person in charge of community volunteer recruitment. The Washington Reading Corps’ data 

PD MS SS

PD MS SS

 
RMC Research CorporationPortland, OR 11 

indicated that by the end of June 2016 a total of 3,512 community volunteers had provided 39,787 

hours of volunteer service related to reading support across sites. 

 Most members were involved in recruiting community volunteers to tutor students according 

to 66% of site supervisors and 38% of member survey respondents. The supervisors of 33% of 

the sites reported that members assisted the site’s volunteer coordinator and at about 25% of 

sites members trained volunteers to tutor or were involved in volunteer retention 

(e.g., appreciation events). Members described other involvement such as coordinating 

volunteers (n = 7), orienting volunteers (n = 5), and interacting with volunteers at events (n = 2). 

Members indicated that recruiting volunteers and managing volunteer commitment were the 

most challenging aspects of working with community volunteers. 

 A majority of members received site training for their role with community volunteers. 

Members met with or worked with school staff (e.g., the principal, Washington Reading Corps

coordinator, site supervisor, school counselor, or volunteer liaison) regarding volunteer 

recruitment. Several site supervisors mentioned a volunteer handbook as a training resource.

About 20% of site supervisor respondents indicated that no member training was provided.

 Community volunteer hours were tracked in several ways. More than half of sites (58%) used a 

paper system to track community volunteer hours, whereas 25% used a computer spreadsheet 

and 20% used an online system. No tracking system existed at 10% of the sites. Most often

responsibility for tracking community volunteer hours was shared across multiple site staff 

(e.g., the member, site supervisor, other staff persons, and/or a volunteer coordinator). 

In addition to involvement with community tutors, members were often involved in arranging for peer 

tutoring or cross-age tutoring within their schools.

Barriers and Best Practices in Implementation

Evaluation Question

What barriers or contextual factors are influencing the 

Washington Reading Corps program implementation?

Barriers to Effectiveness

Barriers to implementation described by site supervisors included the following: 

 Certain Washington Reading Corps program requirements. Program requirements described as 

challenging included initial paperwork requirements; the need for consistent monitoring of 

student progress; site supervisor training off-site at a busy time of year; members’ volunteer 

hours requirement and restrictions on types of acceptable activities; off-site member trainings 

and other responsibilities that interrupted members’ on-site activities; and Washington Reading 

Corps access being limited to “strategic” students.

 Scheduling challenges. Some site supervisors and members reported difficulty scheduling 

pull-out tutoring that did not conflict with core instruction. Additional challenges mentioned by 

Washington Reading Corps members included lack of time allotted to pull students out for 

tutoring, not seeing the same students regularly because teachers’ schedules changed, having to 

tutor students during ‘free choice’ time when students are “more interested in playing than 

learning,” and being assigned to tutor additional students when a Washington Reading Corps

member left during the academic year.

MS SS
12 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

 Member characteristics. Some members did not appear to be interested in the work, seemed

immature or irresponsible, or had personal issues that interfered with their service. 

 Member recruitment and absenteeism. Finding strong candidates and enough candidates to fill 

member positions was described as challenging and time consuming. Additionally, member

absenteeism and departure from the Washington Reading Corps for personal or professional 

reasons were also implementation barriers noted by site supervisors. 

Implementation challenges cited by member survey respondents were categorized as either related to 

the Washington Reading Corps program or site specific. With regard to program challenges, a few

members reported issues related to training (n = 4) and meetings (n = 3). Training issues included 

training that occurred too late in academic year and school staff who were unfamiliar with the 

Washington Reading Corps. Some members felt that the frequency of professional learning cluster 

meetings took time away from serving at the site. The most common site-specific challenges included

teachers’ lack of clarity regarding member roles or early learning members not understanding their role 

is (n = 12) and scheduling conflicts (n = 11). 

Suggestions for Improvement

Suggestions to improve the Washington Reading Corps program addressed some identified barriers to 

implementation and other topics such as the following: 

 Certain Washington Reading Corps program requirements. Suggested improvements included 

reducing the number of hours or expanding types of activities counted toward members’ 

required hours and increasing options for members volunteering in the community. At a few 

sites the tutoring group size exceeded Washington Reading Corps requirements or members felt 

overwhelmed by the number of students assigned to them. Adjusting the timing of some 

program activities was another recommendation, such as having cluster meetings during school 

breaks or on non-school days and having members start on-site earlier to participate in staff 

development. Lastly, site supervisors recommended reducing or streamlining the reporting 

requirements; the amount of paperwork to initiate a member and ongoing documentation were 

considered burdensome.

 Member recruitment. Site supervisor suggestions for improvement included identifying and 

attracting more qualified candidates, selecting members pursuing a career in education, and 

reducing sites’ burdens related to summer recruitment logistics (e.g., doing background checks, 

selecting members when teachers are gone for summer).

 Professional development for members. Site supervisors’ recommended topics for additional 

member professional development included recruiting volunteers, managing student behavior 

and positive behavior training, implementing strategies to support struggling readers, and 

increasing early learning training. Conducting a midyear meeting to discuss next steps toward 

members’ future jobs or education was also suggested. In response to an open-ended survey 

question about their interest in other types of training, Washington Reading Corps members 

most frequently requested more professional development on behavior management (n = 23), 

reading strategies (n = 8), and working with ELL students (n = 8) and a variety of other student 

populations (e.g., students in poverty; students with autism or other special education needs; 

and students from a variety of cultural backgrounds; n = 5).

 Additional support from Washington Reading Corps administrators. Site supervisor suggestions 

included increasing initial communication with members about expected activities and projects; 

checking in monthly with new members from a state-level staff person; increasing contact with 

MS SS
RMC Research CorporationPortland, OR 13 

site supervisors; showcasing exemplary practices so all Washington Reading Corps sites benefit 

from successful strategies; and increasing program support to help members find alternative 

placements during school breaks.

Evaluation Question

What best practices can be identified in the Washington Reading 

Corps program design and implementation?

Practices That Support Effectiveness

Washington Reading Corps member survey respondents rated the degree to which they found specific 

program factors helpful. Overall, they found school resources more helpful than resources provided by 

the Washington Reading Corps program coordinators. Roughly 30% rated as extremely helpful

(a) support from school administration for the Washington Reading Corps program, (b) site supervisor 

involvement with the Washington Reading Corps members, and (c) lead teacher involvement with 

Washington Reading Corps members.

Site supervisor survey respondents rated the effectiveness of elements of the Washington Reading 

Corps program for helping struggling readers at their site. More than 80% of respondents rated these

program elements as extremely or very effective: “tutoring occurs in small groups of 6 or fewer 

students”; “tutoring occurs for 20 minutes at least 3 times per week”; and “students spend more time 

practicing reading skills” as a result of their Washington Reading Corps participation (see Exhibit 8). 

MS SS
14 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit 8 Effectiveness of Program Elements

A majority said program elements were extremely or very effective for struggling readers.

In response to an open-ended question, site supervisors described the following practices as helping the 

members or the tutoring program be effective in 2015–2016: 

 Coaching and training opportunities for members, including from site staff. Examples of 

coaching practices included opportunities for members to tutor with seasoned teachers and 

staff; conference with literacy staff; observe effective teachers model interventions; participate 

in staff, department, or weekly team meetings; and meet with the site supervisor regularly. 

Training practices included member training at the October Training Institute; ensuring the 

member was trained in the site’s reading program; and training in positive behavior supports, 

social thinking, and other topics. These types of coaching and training practices were described 

by many site supervisors (n = 23).

 Member characteristics and integration. Member characteristics such as eagerness, flexibility, 

competence, drive, dedication, work ethic, and organizational skills supported program 

effectiveness. Members’ involvement in the school and community and their development of 

relationships with students and staff were other effective practices. Members were described as 

fully integrated into the site community by 62% of site supervisor respondents and moderately

integrated by 19% of respondents.
RMC Research CorporationPortland, OR 15 

 Elements of the Washington Reading Corps program. Elements of the program itself were 

highlighted as effective practices, including implementation of research-based reading 

intervention methods; the augmentation of reading interventions available to students 

(including before- and after-school programs) because of members; members’ review and use of 

student data; and support and communication from Washington Reading Corps staff to sites.

Sustainability of Program Components

Evaluation Question

To what extent are these changes sustainable?

Site sustainability includes members’ efforts to recruit, engage, and maintain community volunteers to 

support reading tutoring at each site. Three additional dimensions of sustainability were assessed: 

written resources that describe implementation of the Washington Reading Corps program; 

sustainability of reading supports provided by members; and member retention, such as factors 

affecting whether members continue to serve for a second year. 

Written Resources for Implementation

Site supervisor survey respondents described written materials and resources at each site about how to: 

 Orient and train Washington Reading Corps members on site. Most frequently, written 

materials from Washington Reading Corps or OSPI were reported (e.g., supervisor handbook, 

sustainability handbook or binder, site orientation checklist). Other materials included past 

member training materials, classroom materials and curriculum guides, written school or district 

expectations, and resource books or articles on teaching literacy. Additional resources included 

new staff orientation and trainings, knowledge and experience of site staff (site supervisor, 

principal, instructional coach, etc.), and building staff development resources.

 Organize family literacy events. Written materials from past years’ events and sustainability 

binders were most frequently mentioned, along with guides from the Title I grant program. 

Other resources included staff experience, returning members’ experience, online materials, 

and partnering with groups such as the Parent-Teacher Association, an after-school reading 

club, or Boys and Girls Clubs. Eleven respondents were unsure or said their sites did not have 

written materials or resources on this topic.

 Recruit community volunteers to tutor. Most frequently mentioned written materials included

volunteer recruitment manuals, site sustainability binders, and site supervisors’ or past 

members’ notes and materials. The parent handbook, district volunteer form, and guides from 

the Title I grant program were described. Resources included the district or other volunteer 

coordinators, experience of staff members, and partnerships with United Way or the 

Parent-Teacher Association. Seventeen respondents were unsure or said their sites did not have 

written materials or resources on this topic.

Reading Support Sustainability

Site supervisor survey respondents indicated which elements would likely continue without the 

Washington Reading Corps program (see Exhibit 9). Most elements were likely to be fully or partially 

sustained at most sites: 

SS

SS
16 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

 Two thirds of respondents said data used to monitor student progress in reading would 

continue without the program. 

 Reading supports for struggling readers and personalized reading instruction would continue, 

but in a reduced way according to more than two thirds of respondents. 

 In contrast, roughly one third of respondents said one-on-one reading tutoring and tutoring for 

20 minutes at least 3 times per week would not continue without Washington Reading Corps. 

Exhibit 9 Site Sustainability of Program Elements

Tutoring elements were the least likely to be sustained across all sites. 

Note. n = 57–58. 
RMC Research CorporationPortland, OR 17 

Member Retention

Member retention beyond their first year is relevant to program sustainability because retention may 

support continuity of tutoring, family literacy events, and volunteer efforts at a site. According to the 

Corporation for National and Community Service guidelines, members are allowed to serve a total of 4 

terms and receive an education grant for the equivalent of 2 terms. About 18% of the respondents to 

the Washington Reading Corps member survey planned to return in 2016–2017, whereas slightly more 

than half did not plan to return (Exhibit 10).

Exhibit 10 Members Planning to Return in 2016–2017

Slightly more than half of Washington Reading Corps members

did not plan to return in 2016–2017. 

Survey respondents who planned to return most frequently reported that their decision was influenced 

by their enjoyment of Washington Reading Corps (n = 7), the education grant (n = 4), and the learning 

experience (n = 4). Other factors included the school culture and the feeling that they were making a 

difference. Members who did not plan to return most frequently responded that the stipend was too 

low (n = 23), they were planning to attend school in the fall (n = 18), or they could no longer receive the 

education grant (n = 5). They also referred to lack of support (n = 3) and scheduling problems (n = 3) and 

a few mentioned burnout or burdensome Washington Reading Corps requirements.

Factors associated with retention. To further explore factors related to Washington Reading Corps

member retention, RMC Research conducted an exploratory factor analysis on 16 survey items related 

to member experiences7 (see Appendix C). This analysis revealed 3 factors: meaningfulness of work 

(α = .82), satisfaction (α = .56), and affiliation (α = .88). The survey items included in these 3 factors are 

presented in Exhibit 11.

7Mesch, D.L., Tshirhart, M., Perry, J.L., & Lee, G. (1998). Altruists or Egoists? Retention in Stipended Service, Nonprofit 

Management and Leadership, 9(1), 3–21. 

 

MS
18 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit 11 Factors Associated With Washington Reading Corps Members’ Experiences

Factors Associated With Washington Reading Corps Members’ Experiences

Meaningfulness of Service Satisfaction Affiliation

I am helping others through the 

Washington Reading Corps.

I am serving the site community.

I have a good relationship with the 

students I tutor through the 

Washington Reading Corps.

I know how my service as a 

Washington Reading Corps member 

influences my students' reading 

skills.

I feel a responsibility for developing 

my students' literacy.

I feel like my skills are being utilized 

as a Washington Reading Corps 

member.

I find my service with the 

Washington Reading Corps 

interesting.

I find my service with the 

Washington Reading Corps 

challenging.

I receive quality supervision as part 

of the Washington Reading Corps.

I have been able to get to know 

other Washington Reading Corps 

members.

I have a good relationship with 

other Washington Reading Corps 

members.

I have an opportunity to socialize 

with other Washington Reading 

Corps members.

Washington Reading Corps members who indicated they were not planning to return in 2016–2017 

reported significantly less affiliation than those who planned to return and those who were unsure.

Regardless of whether they planned to return, members’ ratings on meaningfulness and satisfaction

related to Washington Reading Corps did not significantly differ. In other words, members who planned 

to return, did not plan to return, or were unsure did not differ in how meaningful or satisfying they 

found their service with the Washington Reading Corps. To further examine the relationship among 

Washington Reading Corps member characteristics (e.g., gender, previous volunteer experience, 4-year 

degree), meaningfulness of service, satisfaction, affiliation, and compensation, RMC Research designed 

a multiple logistic regression model to explore whether these factors predicted retention. Members 

with a 4-year degree were significantly more likely (p = .01) to report that they did not plan to return 

during the 2016–2017 academic year than members without a 4-year degree.

RMC Research then created a second multiple logistic regression model that only included 2 predictors:

the factor affiliation and the survey item “I receive fair benefits for my service with the Washington 

Reading Corps (e.g., stipend, education grants, etc.” In this model “fair benefits” was a significantly 

strong predictor of Washington Reading Corps member retention (p = .04, odds ratio = 2.25), whereas 

perceptions of affiliation did not significantly predict retention. This finding suggests that Washington 

Reading Corps members were 2.3 times more likely to be retained for each unit increase in members’ 

perception of fair benefits. In other words, the more the members agreed that the benefits they 

receive for their service were fair, the more likely they were to indicate they planned to return the 

following year.

RMC Research then conducted a third multiple logistic regression model to explore the relationship 

between the 5 facilitators of implementation (support from school administration, time allotted to tutor 

students, lead teacher involvement with members, Washington Reading Corps site supervisor 

involvement, and Washington Reading Corps program coordinator involvement) and member retention. 

This multiple logistic regression model revealed that none of these facilitators significantly influenced 

member retention.
RMC Research CorporationPortland, OR 19 

Program Impact 

The impact evaluation of the Washington Reading Corps examined (a) the influence of the Washington 

Reading Corps program on student reading scores, (b) the relationships between school characteristics, 

member characteristics, and student reading scores, (c) change in student reading skills, and (d) member 

perceptions of change in students’ attitudes, behaviors, and self-confidence in reading. The evaluation 

questions and their data sources are presented in Appendix A (Exhibit A1). Data were synthesized from 

several sources:

Influence of the Washington Reading Corps on Student Reading Scores

Evaluation Questions

To what extent do student assessment scores differ between those 

served by Washington Reading Corps and those not served by 

Washington Reading Corps? 

To what extent are the Washington Reading Corps programs on 

track to show increases in students achieving grade level reading 

proficiency? 

To what extent are the Washington Reading Corps programs on 

track to show decreases in reading proficiency gaps?

RMC Research addressed these 3 evaluation questions for the 2014–2015 cohort using a matched 

comparison group (see Appendix E for results). For the 2015–2016 cohort, these evaluation questions 

will be addressed using student-level data from Washington Reading Corps schools and from 

comparison schools identified in fall 2016. Analyses will occur in winter 2017 when the state assessment 

data are available.8

To analyze the impact of Washington Reading Corps on 2015–2016 participating students, RMC 

Research will use a quasi-experimental design with a 2-step propensity score matching process

(matching students and matching schools). When random assignment to experimental conditions is not 

feasible in real world settings, quasi-experimental designs provide another rigorous approach to 

exploring the causal relationship between an intervention and related outcomes. Matching Washington 

 

8This analysis is proposed in the 2016–2017 contract.

Statewide reading assessments

Program data

Student tracking logs

MS Washington Reading Corps member survey (see Appendix C)

PD

STL

RA

RA PD
20 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Reading Corps students and schools to other similar students and schools reduces sampling bias and 

leads to a more precise analysis of whether reading outcomes are related to Washington Reading Corps

participation. Identifying a matched comparison group creates the counterfactual condition: what would 

student outcomes have been if students had not participated in Washington Reading Corps? With 

matched comparison groups in place, the evaluation team can statistically test whether students’ 

reading outcomes differ significantly at schools with and without Washington Reading Corps tutoring. 

To address whether participation in Washington Reading Corps influenced student reading scores on the 

ELA Smarter Balanced Assessment, RMC Research will use cross-sectional hierarchical linear modeling 

(HLM) to account for Washington Reading Corps’ hierarchical structure, meaning that students 

(hierarchical Level 1) are nested in different schools (hierarchical Level 2). As an analytic technique, HLM 

provides a more precise estimate of individual student effects and school effects on reading scores. 

Because Washington Reading Corps is implemented in multiple schools, HLM can identify how student 

scores on the ELA Smarter Balanced Assessment may be influenced by each school’s unique 

characteristics, by each student’s unique characteristics, and by the school’s participation in Washington 

Reading Corps. 

Evaluation Question

To what extent are the Washington Reading Corps programs 

meeting the 3 Corporation for National and Community Service 

(CNCS) National Performance Measures?

The Washington Reading Corps focused on economically disadvantaged students with regard to 

Education Performance Measures 1 and 2 (ED1 and ED2). Although specific information about the 

economic status of individual students participating in the program is not available, the Washington 

Reading Corps program is typically implemented in schools with a significant percentage of students 

receiving free or reduced-price meals. Among 66 Washington Reading Corps sites with OSPI Report Card 

data, the average percentage of students receiving free or reduced-price lunch was 61% (range

18–95%). As shown in Exhibit 12, the number of students participating in active Washington Reading 

Corps sites is represented in Measure ED1. For Measure ED2, students who completed the program is 

defined as the number of students who either continued to receive services through spring or met 

benchmark during the year and exited the Washington Reading Corps program. For Measure ED5, in 

2015–2016 improved academic performance in literacy is defined as the number of students who 

received a ‘yes’ for “did student improve?” variable at any time during the school year: 92% of students 

with data were rated as showing improved academic performance in literacy.9

 

9A total of 687 students had missing data on the variable “did student improve?” and 226 were reported no improvement.

STL
RMC Research CorporationPortland, OR 21 

Exhibit 12 National Performance Measures Results

National Performance Measures Results

National Performance Measure 2015–2016 

Measure ED1 The number of economically disadvantaged students or students with 

special/exceptional needs who start in a CNCS-supported education program. 

3,605 

Measure ED2 The number of economically disadvantaged students or students with 

special/exceptional needs who complete a CNCS-supported education program. 

2,504 

Measure ED5 The number of students with improved academic performance in literacy. 2,692 

What is the relationship between school characteristics, 

Washington Reading Corps member characteristics, and student 

reading skills?

Predictors of Meeting Grade Level Reading Benchmark

To further explore whether participation in the Washington Reading Corps influences student outcomes, 

RMC Research used HLM to explore the relationship between site characteristics, Washington Reading 

Corps member perceptions, and whether or not students met grade level reading benchmarks. Using 

HLM accounts for the hierarchical structure of the Washington Reading Corps (meaning that students 

are nested in schools). HLM provides a way to determine which characteristics, by hierarchical level, are 

significantly related to student reading outcomes.

RMC Research conducted a 2-level HLM that included students (n = 2,334) at Level 1 and sites (n = 33) at 

Level 2. The final model included 6 predictors of meeting the grade level reading benchmark. Exhibit 13 

presents descriptive statistics for each of the variables included in the HLM. In contrast to 2014–2015, 

none of the predictors listed in Exhibit 13 were significantly related to students meeting the grade level 

reading benchmark. In 2014–2015 the helpfulness of the site supervisor and the tutoring group size 

were positively associated with students meeting grade level benchmarks, whereas the number of 

tutoring days per week was negatively associated with students meeting grade level benchmarks.

Exhibit 13 Descriptive Statistics for Predictor Variables

Descriptive Statistics for Predictor Variables

Predictor Mean Min Max

Helpfulness of site supervisor 3 0 4 

Tutoring group size 4 1 7 

Number of years with Washington 

Reading Corps

6.22 2 19 

Total school enrollment 403 126 668 

Average minutes per tutoring session 28.59 0 110 

Average tutoring days per week 4 0 6 

Note. Helpfulness of site supervisor 1 = not helpful, 2 = somewhat helpful, 3 = helpful, 

4 = very helpful, 5 = extremely helpful. Site n = 33, student n = 2,334. 

PD STL MS
22 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Change in Student Reading Skills

Evaluation Question

To what extent are the Washington Reading Corps programs on 

track to show increases in students meeting reading benchmarks? 

The percentages of students in the program who met reading benchmarks at any time during the year 

(fall, winter, or spring) are reported in Exhibit 14. Notably, in 2012–2013 and 2013–2014 met benchmark

was combined with gained at least one grade, meaning that students might have met either of these

criteria, whereas in 2014–2015 met benchmark and gained at least one grade were separate variables. 

In 2015–2016 tracking logs included met benchmark and “Did student improve?” as separate variables. 

Exhibit 14 Achievement of Reading Benchmarks

Roughly 40 to 50% of students enrolled in the active Washington Reading Corps 

sites met reading benchmarks during the year.

Note. Only students with reading benchmark data recorded in the student tracking logs are included. 

In 2015–2016, 548 students (15%) were missing reading benchmark data.

STL
RMC Research CorporationPortland, OR 23 

Student Attitudes, Behaviors, and Self-Confidence in Reading

Evaluation Question

To what extent are the Washington Reading Corps programs on 

track to show improvements in students’ attitudes, behaviors, and 

self-confidence in reading?

Attitudes

Washington Reading Corps members were asked 2 questions about if and how their students’ attitudes 

toward reading had changed since the beginning of the 2015–2016 school year. The majority of 

respondents (67%) reported that their students’ attitudes were much better while some reported

student attitudes were somewhat better (26%). As Exhibit 15 shows, open-ended survey responses 

further supported these results.

Exhibit 15 Change in Student Attitudes

Members reported that student attitudes toward and enjoyment of 

reading changed the most. 

MS

One of my lowest readers at the beginning of the year has 

become something of an actor in class now, while also being able 

to read his lines proficiently. The progress I've seen is incredible, 

and helping the kids find what part of reading is their favorite 

part has been a joy.

They went from "why do we have 

to?" to "what are we starting 

today?" 
24 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Behaviors

To explore the extent to which students’ behaviors toward reading changed, RMC Research analyzed 

data from the student tracking logs and from the member survey. 

Student tracking logs. Members at each site rated students in fall, winter, and spring on the extent to 

which students were engaged and participated during their tutoring sessions. RMC Research conducted

a paired samples t-test for each grade to determine whether reported student behaviors changed 

significantly between fall and spring. Across all grades except Grade 6, student engagement and 

participation significantly increased, with the greatest behavior change in Grade 1 (see Exhibit 16). 

Exhibit 16 Reading Behavior Improvement by Grade

Reading behaviors improved across all grades. 

Grade Fall Spring Change n 

K 2.89 3.27 0.38*** 361 

1 2.77 3.28 0.51*** 323 

2 2.87 3.29 0.42*** 343 

3 2.92 3.25 0.33*** 275 

4 3.04 3.42 0.38*** 214 

5 3.02 3.46 0.44*** 124 

6 2.95 3.00 0.05 20 

All Grades 2.90 3.30 0.40 1,660 

Note. 1 = disengaged (rarely participated), 2 = somewhat engaged (sometimes 

participated), 3 = mostly engaged (often participated), 4 = engaged (always 

participated). Only students with fall and spring ratings were included in the analysis.

***p = .000.

STL MS
RMC Research CorporationPortland, OR 25 

Member survey. Survey respondents estimated the percentage of students they tutored in 2015–2016 

who showed improvement across a range of reading behaviors. The vast majority of members reported 

observing change in over half of their students on all reading behaviors. Exhibit 17 shows the proportion 

of respondents who reported that more than half of their students showed improvement in each area.

Exhibit 17 Change in Reading Behaviors

All reading behaviors improved among majority of students. 

Reading Behavior Members

Demonstrating interest in new books 84% 

Putting in effort when reading 82% 

Expressing enjoyment when reading 79% 

Answering questions about what they read 76% 

Exploring different kinds of books 75% 

Reading independently 74% 

Comfort reading aloud 72% 

Choosing own books 71% 

Staying on-task while reading 70% 

Asking questions about what they read 67% 

Reading aloud fluently 66% 

Reading for longer amounts of time 66% 

Asking about word definitions 57% 

Note. Percentages represent proportion who reported that more 

than 50% of their students showed improvement. n = 97–98. 

When asked about other changes in their students’ reading behaviors, about one quarter of Washington 

Reading Corps members reported that students they tutored were more motivated (n = 24) or more 

interested in reading (n = 22) as the year progressed. Washington Reading Corps members also 

described several other ways their students’ reading behaviors changed (Exhibit 18). 

 
26 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit 18 Change in Other Reading Behaviors

Members reported observing changes in student reading behaviors. 

Confidence in Reading

Washington Reading Corps members were surveyed about whether they noticed any changes in their

students’ self-confidence in reading. About half of respondents (n = 47) noticed increased confidence in

students’ reading, while others provided specific examples presented in Exhibit 19. 

Exhibit 19

The majority of members observed increased student confidence in reading.

MS
RMC Research CorporationPortland, OR 27 

Summary and Recommendations

The findings and recommendations from the 2015–2016 implementation and impact evaluation are 

organized around (a) program implementation, (b) barriers to effectiveness, suggestions for 

improvement, and best practices, (c) sustainability, and (d) program impact. 

Program Implementation

 A total of 77 sites participated (69 elementary schools and 8 early learning centers) and 117

members were retained in the Washington Reading Corps program in 2015–2016. 

 In Kindergarten through Grade 6, 3,602 children participated and 2,504 were counted as 

completing the tutoring program; however, the number of students who completed the 

program is likely underreported due to missing data.

 The majority of Washington Reading Corps sites implemented key aspects of the tutoring 

model, such as one-on-one and small-group (with 6 or fewer students) tutoring. On average 

sites exceeded the target of 20 minutes of daily tutoring and sessions occurred 4 days a week.

 Site supervisors strongly endorsed the usefulness of having a Washington Reading Corps 

member provide tutoring to improve struggling students’ reading skills. More than 80% of 

Washington Reading Corps site supervisor respondents described tutoring as very or extremely 

useful at their site and a majority cited data from reading assessments or student growth in 

reading as evidence of usefulness. 

 Members received professional development and training from multiple sources: the 

Washington Service Corps Training Institute, member-led and regional professional learning 

cluster meetings, and their respective sites. 

 About half of members rated training from all sources as very good or excellent quality and

as relevant or very relevant to their service. 

 With respect to the Washington Service Corps Training Institute, members said the most 

useful topics included the fundamentals of reading and student and behavior 

management. The Fish! Philosophy training had the most mixed ratings.

 The quality of regional professional learning cluster meetings had more mixed ratings than 

other types of professional development. Some site supervisors and members disliked the 

schedule because it interrupted members’ on-site activities. 

 Most sites provided training and ongoing coaching to members.

 Most site supervisor respondents rated ongoing coaching or supervision of members as 

a very or extremely effective element of the Washington Reading Corps program.

 Sites did not consistently provide training across all topics, according to member

respondents.

 About a third of site supervisor respondents said their site provided some member 

training to support reading skills of English Language Learners or bilingual students.

 Members were most commonly involved in implementing family literacy nights at their sites. 

Almost all sites measured results of family literacy activities based on the number of families 

who attended; other indicators of results were teacher and staff feedback and whether more 

families got involved in school activities.

 Most members were involved in recruitment of community volunteers and most received site 

training for their role with community volunteers. 
28 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

 More than half of sites used a manual paper system to track community volunteer hours. 

Most often responsibility for tracking hours was shared across multiple site staff, including 

the member(s) and site supervisor. 

Barriers to Effectiveness, Suggestions for Improvement and Best Practices

 Several barriers to effectiveness were cited by site supervisors and members: 

 Certain Washington Reading Corps program requirements (i.e., paperwork, off-site trainings 

that interrupted activities, members’ volunteer hours requirements, restrictions on 

acceptable activities). 

 Member characteristics that interfered with service, challenges recruiting members, and 

member absenteeism. 

 Challenges scheduling pull-out tutoring. 

 Some teachers’ lack of clarity regarding member roles and some early learning members not 

understanding their role. 

 Site supervisors and members made some suggestions for improvement: 

 Modify some program requirements for members. 

 Adjust the timing of off-site trainings. 

 Reduce reporting requirements. 

 Improve the efficacy and efficiency of member recruitment. 

 Provide additional member professional development on managing student behavior, 

reading strategies, early learning, and working with ELL students and other student 

populations.

 Increase support from Washington Reading Corps administrators to sites through more 

initial communication with members about expected activities and projects, check-ins with 

new members, increased contact with site supervisors, and assistance find placements for 

members during school breaks.

 Best practices in support of program effectiveness identified by site supervisors included: 

 All elements of the Washington Reading Corps program design. In particular, 

implementation of research-based reading intervention programs, tutoring in small groups, 

tutoring for 20 minutes at least 3 times per week, increased student time spent practicing 

reading skills (including through before- and after-school programs supported by members), 

members’ use of student data, and support from Washington Reading Corps staff to sites.

 Coaching and training opportunities for members, including from site staff.

 Member characteristics and member integration with students and staff.

Sustainability

 In terms of written materials and resources for ongoing implementation of the Washington 

Reading Corps program, sites described:

 Written materials from Washington Reading Corps or OSPI and other on-site resources (staff 

trainings, experience of staff members) for orientation and training of members on site. 

 Written materials from past years’ events and sustainability binders around organization of 

family literacy events (a subset were unsure or their sites did not have written materials or 

resources). 
RMC Research CorporationPortland, OR 29 

 Volunteer recruitment manuals, site sustainability binders, parent handbook, and guides

from the Title I grant program for recruitment of community volunteers (a subset were

unsure or their sites did not have written materials or resources).

 In terms of reading support sustainability at sites, most elements were likely to be sustained,

either fully or in a reduced way. Using data to monitor student progress in reading was the

element most likely to be fully sustained. However, about one third of site supervisors said

one-on-one reading tutoring and tutoring for 20 minutes at least 3 times per week would not

continue without Washington Reading Corps.

 In terms of local sustainability through member retention, about 18% of members planned to

return for the next academic year.

Program Impact

 About 46% of participating students with data met grade level reading benchmarks during

2015–2016.

 Fully 92% of students with data were rated as showing improved academic performance in

literacy during 2015–2016.

 Site characteristics and Washington Reading Corps member perceptions did not significantly

predict whether or not students met grade level reading benchmarks in 2015–2016.

 Across all grades except Grade 6, student engagement and participation in tutoring

significantly increased over the year, with the greatest behavior change observed in Grade 1

students.

 Members consistently reported improvement in students’ reading behaviors, attitudes, and

confidence in reading. For example, more than 80% of Washington Reading Corps members

surveyed indicated that by spring 2016 their tutoring students put in more effort when reading

and were more interested in new books. A majority said students had much better attitudes

toward and enjoyment of reading. About half of members reported students showed more

confidence in reading and reading aloud.

Recommendations

RMC Research has developed recommendations to strengthen both the implementation and impact and 

the evaluation of the Washington Reading Corps program. 

To Strengthen Implementation and Impact

 Improve the efficacy and efficiency of member recruitment. In addition to logistical challenges

of recruitment, more generally site supervisors said member characteristics could be either a

barrier or a key facilitator to program implementation.

 Review program requirements for members and for site paperwork to reduce burden on

members and on site supervisors where possible.

 Provide additional training for members—for example, professional development on managing

student behavior, reading strategies, early learning, and working with ELL students and other

student populations.

 Members had mixed perceptions of the quality of professional learning clusters. Scheduling

these meetings on non-school days may reduce disruption to sites’ Washington Reading

Corps implementation.
30 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

 Consider additional support from Washington Reading Corps administrators to sites. Site 

supervisor respondents mentioned support from Washington Reading Corps staff to sites as a 

best practice and requested more ongoing communication from Washington Reading Corps

administrators to members and to site supervisors.

 Increase consistency of sites’ sustainability through written materials and resources. Although 

many sites described written materials and resources that would support sustainability of 

implementation, a subset of sites did not seem to have written materials and resources.

 Increase efforts to support members’ affiliation with other Washington Reading Corps 

members to support local sustainability by increasing member retention for a second year. 

Efforts might include, for example, structuring some social activities at the annual Training 

Institute, offering online forums for member communication, and encouraging site supervisors 

or lead teachers to support member interactions. Members’ perceived affiliation with other 

members (“I have been able to get to know other Washington Reading Corps members,” “I have 

a good relationship with other Washington Reading Corps members,” and “I have an 

opportunity to socialize with other Washington Reading Corps members”) predicted whether 

they planned to remain at their site for an additional year. 

To Strengthen Evaluation of Program Impact

 Increase efforts to support accuracy and completeness of data entry in the student tracking 

logs to strengthen impact analyses. For example, quality assurance checks by Washington 

Reading Corps staff in fall and winter may allow for more complete logs. When data is missing 

for “met benchmark,” the full impact of the Washington Reading Corps program cannot be 

described. If “start date” and “end date” were consistently entered, the effect of program 

dosage on student reading outcomes could also be explored.

 Consider collecting student tracking log data using an online data collection tool. Student 

tracking log data are currently collected 3 times a year using individual spreadsheets for each 

site. Some sites alter spreadsheets or enter incompatible data values. This method also requires 

consolidating spreadsheets into a single data file after each data collection wave. An online data 

collection tool can improve the quality of data that are collected and increase efficiency in data 

cleaning and analysis.
RMC Research CorporationPortland, OR 31 

Appendix A 

Evaluation Methodology

RMC Research initiated its independent impact and implementation evaluation of the Washington 

Reading Corps program in April 2015 under contract with the Washington State Office of 

Superintendent of Public Instruction (OSPI). The impact evaluation examined the program’s effect on 

students’ reading skills through a quasi-experimental design that compared state assessment reading 

scores of the schools implementing the Washington Reading Corps program to the scores of a matched 

sample of schools not implementing the program. Tracking logs were used to report on students 

meeting benchmarks and select sites for telephone interviews, and surveys were conducted to provide 

insight into Washington Reading Corps members’ perceptions of changes in students’ reading attitudes, 

behaviors, and self-confidence. The implementation evaluation described program outputs, barriers and 

facilitators to implementation, possible best practices, and sustainability.

Evaluation Questions and Design

The evaluation was guided by the program logic model and 8 questions originally developed by the 

Washington Reading Corps and 2 related questions proposed by RMC Research (see Exhibit A1; see also 

Exhibit A3, the logic model). The central impact question addressed by the quasi-experimental design 

was To what extent do student assessment scores differ between those served by Washington Reading 

Corps and those not served by Washington Reading Corps? Quasi-experimental designs provide a 

rigorous approach for exploring the causal relationship between an intervention and related outcomes

when random assignment is impractical. By employing this design RMC Research statistically tested 

whether sites that have Washington Reading Corps members differ significantly in student reading 

scores from similar sites without Washington Reading Corps members. If sites with Washington Reading 

Corps members have significantly higher reading achievement scores than those without Washington 

Reading Corps members, it may be inferred that the program had a positive impact on student 

achievement.10 Because the Washington Reading Corps provides individualized and small-group 

interventions, it likely impacts reading skills of students who are tutored by Washington Reading Corps 

members (rather than all students in a school). Therefore, to more rigorously test the impact of 

Washington Reading Corps on students, RMC Research used student-level data for this analysis.

 

10The primary limitation of using matched comparison groups rather than random assignment to treatment and comparison 

conditions is related to group equivalence. With matched comparison designs, we can assume that groups are equivalent only 

on known characteristics. Random assignment assures equivalence on both known and unknown characteristics.
32 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit A1 

Washington Reading Corps Evaluation Questions

Focus Evaluation Question Data Sources

Implementation 1. To what extent are the Washington Reading Corps

programs being implemented?

 Program data

 Member surveysa

 Site supervisor surveysa

Impact 2. To what extent are the Washington Reading Corps

programs meeting the 3 CNCS National 

Performance Measures?b

 Program data

 Student tracking logs

Implementation 3. To what extent are the Washington Reading Corps

programs providing their projected program 

outputs?

 Program data

 Student tracking logs

 Member and/or Site supervisor

surveysa

Impact 4. To what extent are the Washington Reading Corps

programs on track to meet their short- and 

medium-term outcomes?

a) Improvements in students’ attitudes, 

behaviors, self-confidence in reading

b) Increases in students meeting reading 

benchmarks

c) Increases in students achieving grade level 

reading proficiency, decreases in reading 

proficiency gaps

 Member surveysa; student tracking 

logs

 Student tracking logs

 State reading assessment data

Impact 5. To what extent do student assessment scores 

differ between those served by Washington 

Reading Corps and those not served by 

Washington Reading Corps? 

 Program data

 State reading assessment data

Implementation 6. What barriers/contextual factors are influencing 

the Washington Reading Corps program 

implementation?

 Member surveysa

 Site supervisor surveysa

Implementation 7. What best practices can be identified in the 

Washington Reading Corps program design and 

implementation?

 Member surveysa

 Site supervisor surveysa

Implementation 8. To what extent are these changes sustainable?

a) What program characteristics influence 

member retention?c

 Site supervisor surveysa

 Member surveysa

Impact 9. What is the relationship between school 

characteristics, Washington Reading Corps

member characteristics, and student reading 

skills?c

 Program data

 Member surveysa

 Student tracking logs

a

Source proposed by and data collected by RMC Research. bCNCS National Performance Measures include (a) number of economically 

disadvantaged students or students with special or exceptional needs who start in a CNCS-supported education program, (b) number of 

economically disadvantaged students or students with special or exceptional needs who completed participation in a CNCS-supported K–12 

education program, and (c) number of students with improved academic performance in literacy (and/or math). c

Question proposed by RMC 

Research.
RMC Research CorporationPortland, OR 33 

Data Sources

The data sharing agreement with OSPI allowed RMC Research to receive and analyze de-identified 

student-level data, which included Washington Reading Corps student tracking log data. Washington 

Reading Corps and Washington Service Corps staff provided additional program data to RMC Research 

such as excerpts from site applications, Washington Reading Corps Training Institute training materials, 

and Washington Service Corps data on family literacy events and community volunteer hours. RMC 

Research also used surveys to collect data from Washington Reading Corps members and site 

supervisors. 

Washington Reading Corps Student Tracking Log Data

Most spring 2015–2016 student tracking logs were received in July 2016. Tracking logs from 2012–2013, 

2013–2014, 2014–2015, 2015–2016 were used to address program outputs. Outputs based on tracking 

logs were summarized only for students in Kindergarten through Grade 6.

Online Washington Reading Corps Member Survey

The online Washington Reading Corps member survey included questions about professional 

development; the tutoring provided at each site; changes in students’ attitudes, behaviors, and 

confidence in reading; barriers and facilitators to program implementation; and the respondents’ 

experiences as a Washington Reading Corps member. Items related to experiences as a Washington 

Reading Corps member were developed based on research conducted on the motivational, 

demographic, and individual factors that affect the retention of stipended volunteers (Mesch, 

Tschirhart, Perry, & Lee, 1998).

11 Of the 127 Washington Reading Corps members, 126 with active email 

accounts were invited to complete an online survey via Survey Monkey, and the response rate was 87%. 

Respondents received an electronic $10 gift card from Starbucks for completing the survey (see

Appendix C). The survey was developed in April 2015, reviewed by Washington Reading Corps and 

Washington Service Corps stakeholders and by RMC Research’s internal Human Protections Committee, 

and administered to all Washington Reading Corps members in May 2015 and 2016. 

Online Washington Reading Corps Site Supervisor Survey

The online Washington Reading Corps Site Supervisor Survey included questions about implementation 

of reading tutoring at each site; member training in the site’s reading approach; facilitators and barriers 

to tutoring implementation; members’ roles with community volunteers and with family literacy events; 

and sustainability of program elements. Across 75 sites there were 71 site supervisors, including 4 who

supervised 2 sites and were invited to complete surveys about each of their 2 sites. All 71 site 

supervisors had active email accounts and were asked to complete an online survey via Survey Monkey. 

The response rate was 81% (61 respondents out of 75 possible surveys). Respondents received an 

electronic $10 gift card from Starbucks for completing the survey (see Appendix D). The survey was 

developed in April 2016, reviewed by Washington Reading Corps and Washington Service Corps 

stakeholders and by RMC Research’s internal Human Protections Committee, and administered to 

Washington Reading Corps site supervisors in May 2016.

Exhibit A2 outlines all data types used to address implementation and impact questions.

 

11Mesch, D.L., Tshirhart, M., Perry, J.L., & Lee, G. (1998). Altruists or Egoists? Retention in Stipended Service, Nonprofit 

Management and Leadership, 9(1), 3–21. 
34 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Exhibit A2 

Washington Reading Corps Evaluation Data

Focus Sources

Demographic  School (e.g., free or reduced-price lunch, student demographics, county, district)

 Student (e.g., free or reduced-price lunch, race/ethnicity, gender)

Program data and 

student tracking 

logs

 Washington Reading Corps members (e.g., gender, education level)

 Number of members trained

 Hours of training provided

 Number of sites served

 Number of tutoring hours provided by members to students

 Number of children in early learning centers and elementary schools who are enrolled, 

screened, and tutored

 Number of students who complete the tutoring program

 Number of parents and family members who participate in literacy-focused school or 

home activities

 Number of community volunteers recruited, trained, and retained

Student tracking 

logs

 Screener test scores: fall, winter, spring

 Number of grade levels gained

 Met benchmark on assessment tool

Participating and 

comparison 

schools

 State reading assessments

Member survey, 

site supervisor 

surveya

 Implementation of reading tutoring

 Suggestions for program improvement (facilitators and barriers to implementation)

 Sustainability of Washington Reading Corps program elements

 On-site training for members

 Members’ roles with family literacy events and community volunteers

 Members’ perceptions of change in students’ attitudes, behaviors, and self-confidence in 

reading

 Members’ experience with the program

 Members’ perceived quality, relevance, and utility of professional development activities

aNew data collected by RMC Research.
RMC Research CorporationPortland, OR 35 

Exhibit A3 

Logic Model Chart Developed by Washington Reading Corps

Theory of Change: By training dedicated AmeriCorps members to provide best-practice literacy tutoring to struggling readers, targeted to students’ assessed needs, Washington Reading Corps can help raise 

students' reading proficiency levels, improve their academic achievement, and, over the long term, improve their life outcomes. By helping individual children gain the skills they need for academic and life 

success, Washington Reading Corps helps foster socially engaged citizens and economically stable communities.

Project Resources

INPUTS

What we invest (# & type of 

AmeriCorps members)

Core Project Components

ACTIVITIES

What we do

Evidence of Implementation 

and Participation

OUTPUTS

Direct products from program 

activities

Evidence of Change

OUTCOMES

Short-Term

(Changes in knowledge, 

skills, attitudes, 

opinions)

Medium-Term

(Changes in behavior 

or action)

Long-Term 

(Meaningful changes)

OSPI and WSC staff to provide 

leadership, guidance, and 

technical assistance.

Washington Reading Corps

Program Coordinator to 

manage day-to-day 

implementation; recruit, 

place, and provide training 

and technical assistance to 

members; monitor fidelity of 

implementation and program 

success; and coordinate 

planning for ongoing quality 

improvement.

150 Full-time AmeriCorps 

members to provide tutoring, 

volunteer recruitment and 

coordination, and

community/parent outreach.

Early Learning Centers (ELC) 

and school site staff to 

provide member supervision, 

support, and alignment with 

core early literacy and/or 

reading curricula.

Regional ESD staff to support 

service sites in specific 

geographic areas with 

technical assistance from 

literacy specialists, in 

partnership with OSPI.

1) Serve sites that have the highest level of needs 

(high percentages of low-income children, English 

Learners, children of military personnel; low overall 

reading proficiency; and or large reading proficiency 

gaps).

2) Use a competitive application process to identify 

and select program sites based on demonstrated 

need, use of research-based strategies, and 

strongest level of commitment to Washington 

Reading Corps’ service goals and objectives.

3) Use valid, reliable literacy screening assessments to 

determine which students need support and focus 

on those that need Tier II interventions. 

4) Use valid, reliable, diagnostic, oral language fluency, 

and progress monitoring tools to assess students’ 

reading proficiency early and multiple times 

throughout the year and use results of assessments 

to target tutoring to student need. 

5) Provide tutoring individually and in small groups of 

no more than six through intensive 20-minute 

sessions at least three times a week (for preschool 

students, provide support during literacy activities 

throughout the day). All tutoring is based on proven 

literacy development practices.

6) Provide significant, structured, evidenced-based 

training and ongoing support to AmeriCorps tutors.

7) Provide ongoing technical assistance by qualified 

Literacy Specialists to ensure fidelity of 

implementation and troubleshoot/problem-solve 

challenges.

8) Use WRC members to leverage additional tutoring 

support from community volunteers.

The number of: 

1) AmeriCorps members trained 

(150 Full Time). 

2) Hours of training provided 

(minimum 36 hours). 

3) Sites served (75). 

4) Tutoring hours provided by 

members (270,000). 

5) Children in ELCs and students 

in elementary schools who are 

enrolled, screened, and 

tutored (6,000). 

6) Participants who complete the 

tutoring program (5,100). 

7) Participants in ELCs who 

improve emergent 

literacy/reading readiness 

skills and students in

elementary schools who 

increase their grade level 

proficiency or meet 

curriculum-based reading 

benchmarks (3,300). 

8) Parents or family members 

who participate in 

literacy-focused school or 

home activities (24,292). 

9) Community volunteers 

recruited, trained, and 

retained (3,000).

1) Improvements in 

students’ attitudes, 

behaviors, and 

self-confidence in 

reading, as measured 

by observation.

2) Increases in 

students’ literacy 

skills and reading 

proficiency, as 

measured by a). the 

number of children 

who meet literacy 

benchmarks on 

progress monitoring 

tools and b). the % of 

students who gain 

one grade level of 

reading proficiency 

on state assessments 

or meet 

curriculum-based 

reading benchmarks.

3) Washington Reading 

Corps members’ 

reports of 

satisfaction with the 

program as 

measured by 

Washington Reading 

Corps’ "Life After 

AmeriCorps" survey.

1) An increase in the 

number of 

Washington State 

students achieving 

grade-level reading 

proficiency, as 

measured by 

performance on 

state reading 

assessments.

2) A decrease in 

reading proficiency 

gaps, as measured by 

disaggregated 

performance data 

from state reading 

assessments.

1) Washington State 

students meet 

college-ready 

benchmarks in high 

school.

2) Students graduate 

from high school and 

enroll in and 

successfully 

complete 

postsecondary

programs.

3) As adults, students 

gain meaningful 

employment that 

pays a living wage 

and participate 

actively in civic life. 

4) Participating 

AmeriCorps 

members use the 

skills and experience 

they gain to become 

leaders in their 

chosen professions 

and in their 

communities.

RMC Research CorporationPortland, OR 37 

Appendix B 

HLM Model

Exhibits B1 and B2 present the results for the final HLM model that explored the relationship between

school-level characteristics, Washington Reading Corps member characteristics, and whether or not 

students met grade level benchmarks.

Exhibit B1 

HLM Final Estimation of Fixed Effects: Population Average Model

Fixed Effect Coefficient

Standard 

Error t-ratio

Approx. 

df p-value

Intercept1,B0

Intercept2,Y00 0.143 0.210 0.679 27 0.503 

Years in WRC,Y01 -0.030 0.002 -0.661 27 0.514 

Total Enrollment,Y02 0.002 0.045 1.168 27 0.253 

Site Supervisor,Y03 0.206 0.249 0.828 27 0.415 

Average days of the week, slope,B1

Intercept2,Y10 -0.009 0.070 -0.129 1842 0.897 

Average minutes per day, slope, B2

Intercept2,Y20 -0.007 0.006 -1.198 1842 0.231 

Average group size, slope, B3

Intercept2,Y30 0.073 0.043 1.694 1842 0.090 

Exhibit B2 

HLM Final Estimation of Fixed Effects: Population Average Model (continued)

Fixed Effect Coefficient Odds Ratio Confidence Interval

Intercept1,B0

Intercept2,Y00 0.143 1.153 (0.749, 1.775)

Years in WRC,Y01 -0.030 0.970 (0.885,1.065) 

Total Enrollment,Y02 0.002 1.002 (0.998,1.006) 

Site Supervisor,Y03 0.206 1.223 (0.738,2.047) 

Average days of the week, slope,B1

Intercept2,Y10 -0.009 0.991 (0.864,1.136) 

Average minutes per day, slope, B2

Intercept2,Y20 -0.007 0.993 (0.981,1.005) 

Average group size, slope, B3

Intercept2,Y30 0.073 1.076 (0.989,1.171) 
38 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Appendix C 

Washington Reading Corps Member Survey With Frequencies

Survey Introductory Text

Thank you taking the time to complete this survey. The survey includes questions about the professional 

development you received as a Washington Reading Corps member, what tutoring looks like at your 

site, changes you have noticed in your students’ attitudes, behaviors, and self-confidence around 

reading, barriers and facilitators to implementing the program, and a few questions about you. The 

survey will take roughly 15 minutes to complete. You will receive a $10 gift card from Starbucks for your 

participation.

Professional Development

1. If you received WRC training through the Training Institute, please rate how useful the training 

was in each of the following areas. Select only one response for each item.

Training Institute Topics

Not

Useful

Somewhat

Useful Useful

Very

Useful

Extremely

Useful

I did not

receive 

training on this

Activity Toolkits 6% 12% 28% 22% 10% 21% 

Cultural Competency and Inclusion 10% 15% 27% 19% 17% 12% 

Data Collection Methods 8% 21% 28% 13% 8% 22% 

Early Learning 3% 10% 26% 7% 6% 47% 

English Language Learners 4% 14% 22% 19% 14% 27% 

Extended Learning 4% 12% 30% 12% 8% 34% 

Family Engagement 5% 14% 30% 17% 8% 26% 

Fish! Philosophy Training 12% 23% 8% 24% 28% 4% 

Fundamentals of Reading 3% 12% 30% 23% 26% 5% 

Principled Negotiation 7% 15% 22% 14% 6% 35% 

Student and Behavior Management 7% 18% 22% 23% 19% 11% 

Volunteer Recruitment and 

Management

15% 16% 30% 13% 6% 20% 

Note. n = 92–100. 
RMC Research CorporationPortland, OR 39 

2. Please select any site specific trainings (at your WRC site or community site) you received during 

the 2015–2016 academic year. Check all that apply.

Topic

Received 

Training

Tutoring/intervention program 58% 

Tracking student progress 62% 

How tutoring connects with classroom instruction 35% 

How students learn to read (critical components of reading) 64% 

Strategies for diverse learners 38% 

Strategies for English Language Learners or bilingual students 29% 

State English Language Arts (ELA) learning standards 32% 

Community volunteer recruitment or management plan 29% 

Family literacy engagement plan 42% 

Student behavior management 52% 

Communication plan for accessing school/site staff 45% 

Orientation to site 65% 

Site culture (expectations for conduct) 58% 

Site safety plan 76% 

Where to direct questions 62% 

Note. n = 110. 

3. Please rate the quality of the WRC training and professional development you received during 

the 2015–2016 academic year.

WRC Professional 

Development or Training Poor Fair Good

Very 

Good Excellent

Not 

Offered 

Did Not 

Attend

Washington Reading 

Corps Training Institute

4% 15% 23% 29% 21% 8% 0% 

Regional professional 

learning cluster (PLC) 

meetings (primarily led by 

ESDs)

9% 29% 24% 17% 19% 1% 1% 

Monthly member-led 

professional learning 

cluster (PLC) meetings

14% 22% 29% 24% 12% 0% 0% 

Site specific WRC trainings 2% 9% 31% 23% 24% 3% 9% 

Professional development 

with site staff

4% 9% 19% 28% 37% 2% 2% 

Other: please rate here 

and describe below.

4% 8% 8% 12% 31% 31% 8% 

Note. n = 26–104. 
40 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

4. Thinking about all of the WRC training and professional development you received in 2015–16, 

how relevant was it to your work as a WRC member?

Did Not 

Attend

Not 

Relevant

Somewhat 

Relevant

Moderately 

Relevant

Very 

Relevant

Extremely 

Relevant

1% 1% 29% 30% 31% 9% 

Note. n = 101. 

5. Think about all of the training and professional development in student behavior management 

you received through WRC and at your site this year. Do you think it was sufficient?

Response Percent

Yes 55% 

No 45% 

Note. n = 100. 

6. As a WRC member, are there areas where you wish you had more training or professional 

development? If so, in which areas?

[open ended]

Community Volunteers and Family Literacy Engagement

7. Please describe your involvement with community volunteers who tutor students (e.g., calling 

volunteers from previous years, training and scheduling volunteers, recruiting volunteers, 

administering volunteer tutor surveys.)

8. Please describe any challenges recruiting or working with community volunteers.

9. Please describe your involvement with family literacy engagement.

Implementation of Reading Tutoring

10. To what extent were the following aspects of WRC tutoring implemented at your site during the 

2015–2016 academic year?

WRC Program Elements Not at all

To some 

extent

To a moderate 

extent

To the full 

extent

Not

applicable

20 minute tutoring sessions with Tier II 

intervention students 3 times a week

4% 3% 11% 75% 7% 

Small-group tutoring with no more than 

6 students in each session

3% 4% 12% 76% 5% 

One-on-one reading tutoring 4% 16% 26% 54% 0% 

Embedded pre-literacy support (Pre-K 

sites only)

12% 3% 4% 14% 67% 

Note. n = 99–100. 
RMC Research CorporationPortland, OR 41 

11. In how many family literacy events did you serve during the 2015–2016 academic year 

(e.g., Dr. Seuss Night, Books and Bears, Title I Family Literacy)?

None One Two

Three or 

more

2% 8% 31% 59% 

Student Responses to Reading

12. Thinking about the students you tutored in reading this year, approximately what percentage of 

students showed improvement in:

Student reading behavior 0% 1–25% 26–50% 51–75% 76–100% Not Applicable

Reading for longer amounts of time 0% 4% 20% 34% 33% 9% 

Reading independently 0% 5% 12% 43% 31% 8% 

Putting in effort when reading 0% 2% 9% 25% 58% 6% 

Staying on-task while reading 0% 7% 18% 53% 18% 5% 

Asking questions about what they read 0% 10% 20% 35% 33% 2% 

Answering questions about what they read 0% 7% 15% 39% 37% 2% 

Asking about word definitions 0% 14% 25% 33% 24% 4% 

Reading aloud fluently 0% 10% 16% 28% 39% 7% 

Comfort reading aloud 0% 8% 10% 36% 37% 9% 

Choosing own books 0% 8% 13% 26% 46% 7% 

Demonstrating interest in new books 0% 7% 6% 32% 52% 3% 

Exploring different kinds of books 0% 7% 12% 36% 39% 6% 

Expressing enjoyment when reading 0% 3% 16% 34% 45% 2% 

Note. n = 97–98. 

13. Thinking about the students you tutored this year, have you noticed any other changes in their 

reading behaviors? If so, please describe.

[open ended]
42 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

Student Attitudes Toward Reading

14. Thinking about the students you tutored since the beginning of this academic year, how have 

your students’ attitudes changed, if at all, toward reading?

Attitude toward reading Percent

Much worse 0% 

Somewhat worse 0% 

About the same 5% 

Somewhat better 26% 

Much better 67% 

I don’t know. 2% 

Note. n = 99. 

15. Please explain how your students’ attitudes toward reading have changed since the beginning 

of this academic year.

[open ended] 

16. Thinking about the students you tutored this year, have you noticed any changes in their 

self-confidence in reading? If so, please describe.

[open ended]

Facilitators and Barriers to Implementation

17. How helpful were the following factors this past year in implementing the WRC tutoring model? 

Factors

Not 

helpful

Somewhat 

helpful Helpful

Very 

Helpful

Extremely 

Helpful

Support from school administration for 

the WRC program

4% 6% 21% 39% 30% 

Time allotted to tutor students 1% 10% 28% 42% 19% 

Lead teacher involvement with the WRC 

members

1% 12% 28% 31% 28% 

WRC Site Supervisor involvement 4% 5% 18% 41% 32% 

WRC Program Coordinator involvement 1% 8% 30% 39% 22% 

Other 23% 0% 8% 15% 54% 

Note. n = 13–98. 

18. What challenges this past academic year, if any, did you face implementing the WRC tutoring 

model at your site?

[open ended] 
RMC Research CorporationPortland, OR 43 

WRC Member Experiences

19. We’re interested in learning more about your experience with the WRC. Please rate your level of 

agreement with the following statements as a WRC member during the 2015–16 academic 

year. 

WRC Member Experience

Strongly 

Disagree Disagree Agree

Strongly 

Agree

I am helping others through the WRC. 0% 1% 32% 67% 

I am serving the site community. 0% 0% 37% 63% 

I have a good relationship with the students I 

tutor through the WRC. 

0% 0% 14% 86% 

I know how my work as a WRC member 

influences my students’ reading skills. 

0% 0% 25% 75% 

I feel a responsibility for developing my 

students’ literacy. 

0% 0% 19% 81% 

I feel like my abilities are being utilized as a 

WRC member. 

0% 4% 39% 57% 

I feel like my skills are being utilized as a WRC 

member. 

0% 2% 44% 54% 

I find my service with the WRC challenging. 8% 18% 43% 31% 

I find my service with the WRC interesting. 0% 3% 39% 58% 

I am learning new skills from WRC that will be 

useful in the future. 

0% 8% 34% 58% 

I receive fair benefits for my service with the 

WRC (e.g., stipend, education grants, etc.)

4% 34% 42% 20% 

I receive quality supervision as part of the 

WRC.

2% 9% 37% 52% 

I have received positive feedback for my work

with the WRC. 

0% 4% 28% 68% 

I have been able to get to know other WRC 

members. 

0% 3% 44% 53% 

I have a good relationship with other WRC 

members. 

0% 5% 41% 54% 

I have an opportunity to socialize with other 

WRC members. 

0% 8% 45% 46% 

Note. n = 97–98. 

20. Have you been asked to serve in other ways during the school day that are not related to your 

role as a WRC member? If so, please describe.
44 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

21. Do you plan to return as a WRC member during the 2016–17 academic year?

Response Percent

Yes 18% 

No 66% 

I don’t know 17% 

Note. n = 100. 

22. What factors are influencing your decision about returning as a WRC member in the 2016–17 

academic year? Please explain.

[open ended] 

23. Is there anything else you would like to add about your experience as a WRC member?

[open ended] 

WRC Member Demographics

24. Have you had previous experience, not including your service with the WRC, as a volunteer?

Response Percent

Yes 90% 

No 10% 

Note. n = 100. 

25. I identify my gender as

[open ended] 

26. What is your highest level of education?

Level of Education Percent

Some high school, but did not 

graduate

0% 

High school graduate or GED 10% 

Some college, but did not 

graduate

29% 

College graduate 52% 

Other (please specify) 9% 

Note. n = 100. 
RMC Research CorporationPortland, OR 45 

27. Which grade levels did you work with as a WRC member during the 2015–16 academic year?

Grade Level Percent

Pre-kindergarten 14% 

Kindergarten 72% 

Grade 1 75% 

Grade 2 75% 

Grade 3 68% 

Grade 4 57% 

Grade 5 53% 

Grade 6 15% 

RMC Research CorporationPortland, OR 47 

Appendix D 

Washington Reading Corps Site Supervisor Survey With Frequencies

Survey Introductory Text

Thank you taking the time to complete this survey. The survey includes questions about implementation 

of members’ tutoring, community volunteer recruitment, and family literacy activities; facilitators, 

barriers, and suggestions for program improvement; sustainability of Washington Reading Corps 

activities; and your role. The survey will take about 15 minutes to complete. You will receive a $10 gift 

card from Starbucks for your participation.

Implementation of Reading Tutoring

1. How many Washington Reading Corps members serve at your site?

1 2 3 

54% 31% 15% 

Note. n = 61. 

2. At your site, with which groups does a WRC member provide tutoring or reading support? Check 

all that apply.

Pre-Kindergarten Kindergarten Grade 1 Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 

15% 77% 82% 82% 74% 79% 62% 18% 

3. What does tutoring by Washington Reading Corps (WRC) members look like at your site? Check 

all that apply.

Member 

provides 

one-on-one 

tutoring

Member 

provides 

small group 

tutoring

Member 

follows push-in 

model (tutor 

goes into 

classes)

Member follows 

pull-out model 

(students leave 

class to work 

with tutor)

Member helps 

with before- or 

after-school 

reading 

program

Member is 

embedded 

in a 

classroom

Other 

(please 

specify)

77% 92% 64% 79% 74% 21% 5% 
48 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

WRC Member Training in Site’s Reading Approach

4. How does the member get initial training about your site’s core reading program? Check all that 

apply.

From literacy 

specialist at site

From Title I and/or Learning 

Assistance Program teacher

From WRC site 

supervisor

From classroom 

teacher

Other (please 

specify)

59% 57% 66% 48% 8% 

5. Does your site provide training to members about how to support reading skills among bilingual 

students or English Language Learners? 

Response Percent

Yes 36 

No 64 

6. If yes, what kind of training is provided?

[open ended] 

7. Does your site provide ongoing supervision or coaching to the WRC member?

Response Percent

Yes 98 

No 2 

8. If yes, please describe the ongoing supervision or coaching provided to the WRC member.

[open ended] 

Facilitators and Barriers to Tutoring Implementation

9. Please rate the usefulness of WRC tutoring at your site.

Not

Useful

Somewhat

Useful Useful

Very

Useful

Extremely

Useful

For struggling students’ reading 

skills, tutoring by a WRC member is

0% 5% 13% 26% 56% 

10. Please describe what evidence you used to rate the usefulness of WRC tutoring.

[open ended] 
RMC Research CorporationPortland, OR 49 

11. Please rate how effective these elements of the Washington Reading Corps program are for 

helping struggling readers at your site. Select only 1 response per item.

Not

Effective

Somewhat

Effective Effective

Very

Effective

Extremely

Effective This didn’t occur

WRC students spend 

more time practicing 

reading skills

0% 3% 15% 32% 49% 0% 

Reading instruction is 

personalized

0% 5% 15% 30% 47% 2% 

Data is used to monitor 

student progress in 

reading

0% 7% 14% 26% 52% 2% 

Tutoring occurs for 20 

minutes at least 3 times 

per week

0% 2% 14% 32% 52% 0% 

Tutoring occurs in small 

groups of 6 or fewer 

students

0% 2% 8% 29% 58% 3% 

One-on-one reading 

tutoring occurs

2% 2% 15% 25% 47% 8% 

Embedded pre-literacy 

support occurs

0% 2% 21% 23% 28% 25% 

On-site training for WRC 

members is provided

0% 0% 21% 28% 50% 2% 

Coaching to WRC 

members from site staff 

is ongoing

0% 5% 19% 17% 58% 2% 

Off-site training for WRC 

members is provided

0% 24% 15% 30% 27% 3% 

WRC site supervisor 

supports program and 

tutoring

0% 2% 20% 19% 56% 3% 

Note. n = 56–59. Percentages may not equal 100% due to rounding.
50 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

12. What other practices helped make the WRC member(s) or the WRC tutoring program effective 

this year?

[open ended] 

13. What were some barriers to implementing the WRC tutoring program at your site?

[open ended] 

14. What suggestions do you have to improve the WRC program or the WRC member’s role?

[open ended] 

15. How integrated is the WRC member in your site community?

Not at all 

Integrated

Slightly 

Integrated

Somewhat 

Integrated

Moderately 

Integrated

Fully 

Integrated

0% 9% 10% 19% 62% 

Note. n = 58. 

WRC Member’s Role with Community Volunteers

16. What is your WRC member’s role with community volunteers who tutor students? Check all that 

apply.

Recruiting

volunteers 

to tutor

Training 

volunteers 

to tutor

Tracking 

volunteers’ 

hours

Volunteer retention 

(e.g., appreciation 

events)

Assisting 

Volunteer 

Coordinator

My site’s WRC member is 

involved in these activities:

66% 27% 36% 25% 34% 

Note. n = 59. 

Other (please specify): [open ended] 

17. What kind of training at your site does the WRC member receive for their role with community 

volunteers? 

[open ended] 

18. At your site, who tracks community volunteers’ participation (e.g., hours, frequency of 

volunteers)? Check all that apply.

WRC 

member

WRC site 

supervisor

Volunteer 

Coordinator

Other staff 

person

Tracking 

doesn’t 

happen

Who tracks community 

volunteer participation?

45% 33% 25% 35% 13% 

Note. n = 60. 
RMC Research CorporationPortland, OR 51 

19. What kind of tracking system for community volunteers does your site have?

Manual paper system Computer spreadsheet On-line system None Other (please specify)

58% 25% 20% 10% 10% 

Note. n = 59. 

WRC Member’s Role with Family Literacy Events

20. What is your WRC member’s role with family literacy activities or events? Check all that apply.

Planning 

family 

literacy 

events

Promoting 

family 

literacy 

events

Developing 

materials for 

family literacy 

events

Facilitating 

at family 

literacy 

events

Attending 

family 

literacy 

events

My site’s WRC member is 

involved in these activities:

86% 81% 80% 83% 95% 

Note. n = 59. 

Other (please specify): [open ended] 

21. What types of family literacy activities or events occurred at your site this year?

[open ended] 

22. Check all that apply. Our site measures results of family literacy activities or events based on:

Number of families who 

attend family literacy 

events

Teacher/staff 

feedback after 

literacy events

More students 

show interest in 

reading after events

More families get 

involved in school 

activities

Other 

(please 

specify)

95% 58% 36% 49% 3% 

Note. n = 59. 

Sustainability

23. What kinds of written materials or resources does your site have about how to recruit 

community volunteers to tutor students?

[open ended] 

24. What kinds of written materials or resources does your site have about how to organize family 

literacy events?

[open ended] 

25. What kinds of written materials or resources does your site have about how to orient and train 

WRC members on-site?

[open ended] 
52 Final Washington Reading Corps Impact Evaluation 2015–2016 Report

26. Imagine your site did not have the Washington Reading Corps program and member next year. 

What elements would continue at your site?

Without Washington Reading Corps, these elements would likely continue:

Without Washington Reading Corps, these 

elements would likely continue: Yes 

Yes, but in a 

reduced way No 

Reading supports for struggling readers 26% 72% 2% 

Personalized reading instruction 24% 67% 9% 

Data used to monitor student progress in 

reading

69% 31% 0% 

Reading tutoring for 20 minutes at least 3 

times per week

21% 50% 29% 

Reading tutoring in small groups of 6 or fewer 

students

22% 60% 17% 

One-on-one reading tutoring 9% 54% 37% 

Community volunteer program to tutor 

students

21% 63% 16% 

Family literacy events 34% 57% 9% 

Note. n = 57–58. 

27. Please add any other thoughts or feedback.

[open ended] 

Your Site Role

28. Please describe your site role this year. Check all that apply.

Site Role Percent

I taught students in an elementary 

classroom.

12 

I taught pre-Kindergarten children. 0 

I was the site’s Reading or Literacy 

Specialist.

38 

I was the site’s Washington Reading 

Corps site supervisor.

90 

My role was something else (please 

describe)

21 
RMC Research CorporationPortland, OR 53 

Appendix E 

Impact Evaluation Summary 2014–2015 
Washington Reading Corps

February 2016 Impact Evaluation Summary

Prepared for 

Washington Reading Corps 

Office of Superintendent of Public Instruction 

Olympia, WA 98504 

Washington Service Corps 

Employment Security Department 

Lacey, WA 98503

Prepared by 

RMC Research Corporation 

111 SW Columbia Street, Suite 1030 

Portland, OR 97201

RMC Research CorporationPortland, OR 1 

Impact Evaluation

The Washington Reading Corps places AmeriCorps members in education and community sites with the 

goal of supporting struggling readers’ foundational skills from pre-Kindergarten through Grade 6. The 

program seeks to achieve that goal through effective collaboration involving schools, families, 

community members, business and state partners, and AmeriCorps. At the state level the Washington 

Reading Corps (WRC) is a partnership between the Washington Service Corps (WSC) and the Office of 

Superintendent of Public Instruction (OSPI) that has been supported since 1998 by the Washington State 

Employment Security Department, SERVE Washington. 

Washington Reading Corps schools are expected to utilize a multitiered instructional system of support 

for reading based on individual student data. To improve students’ reading skills Washington Reading 

Corps members provide (a) one-on-one or small-group tutoring using research-based reading programs, 

(b) reading support for families through outreach and educational activities, and (c) recruitment and 

training of community volunteers to support literacy. Tutoring is intended to supplement rather than 

supplant core reading instruction. When a student is identified as needing Tier II intervention services 

due to not meeting the expected benchmarks on a universal reading screening tool, he or she receives 

one-on-one or small-group (up to 6 students) tutoring from a trained Washington Reading Corps 

member with the goal of at least 20 minutes of tutoring 3 times per week. 

This impact evaluation rigorously tested the effect of Washington Reading Corps participation on 

Grade 4 and Grade 5 students by comparing Washington Reading Corps students’ outcomes on the 

spring 2015 English Language Arts (ELA) Smarter Balanced Assessment to the outcomes of students from 

matched comparison schools that did not participate in Washington Reading Corps in 2014–2015. 

Washington Reading Corps Students 

During the 2014–2015 academic year 65 sites participated in the Washington Reading Corps program 

with 2,649 pre-Kindergarten to Grade 2 students and 1,349 students in Grade 3 through Grade 6. 

Because the younger students do not complete statewide assessments, the impact evaluation is limited 

to students in Grades 3 through Grade 6. According to OSPI data, the majority of Washington Reading 

Corps students in Grades 3 through 5 were either White (51–53%) or Hispanic (21–28%), with a lower 

percentage of White students in Grade 6 (33%). Across all grade levels 70–77% of Washington Reading 

Corps students were identified as low income, consistent with the program’s goal of prioritizing reading 

support for low-income students. 

Prior reading level varied across the Washington Reading Corps participants. RMC Research identified 

each Grade 4 and Grade 5 participant’s score and corresponding reading level on the 2014 

Measurements of Student Progress (MSP) reading assessment, which was administered in the spring 

prior to the students’ participation in Washington Reading Corps (see Exhibit 1). 
2 Washington Reading Corps Impact Evaluation Summary 

Exhibit 1 

Note. Grade 6 students were excluded from the impact analyses due to small sample size. Exhibit includes 

only students with 2014 MSP Reading scores. Due to rounding, percentages may not total 100%. 

As a Tier II intervention, students who meet reading standard might not be the intended audience for 

the Washington Reading Corps program. The fact that in spring 2014 some prospective Washington 

Reading Corps participants had MSP reading scores that met or exceeded the standard suggests possible 

differences between the MSP reading assessment and the screening tools the program sites used in fall 

2014. That is, these summative and formative assessments might have targeted different reading skills—

or the students’ reading skills might have changed over the summer between assessments. Alternately, 

the locally selected screening tools could have identified some participants who might not be the best 

candidates for the Tier II reading services offered by the Washington Reading Corps. 

As Exhibit 2 shows, after receiving Washington Reading Corps tutoring the majority of students were 

classified as Level 1 or Level 2 on the spring 2015 ELA Smarter Balanced Assessment, whereas between 

23% and 38% of Washington Reading Corps participants met or exceeded the ELA standard (Level 3 or 

Level 4). Overall, statewide percentages of students meeting the standard were lower on the ELA 

Smarter Balanced Assessment in 2015 compared to the MSP reading assessment in 2014. Statewide, 

between 52% and 58% of students in Grade 3 through Grade 6 met the ELA Smarter Balanced 

Assessment standard in 2015, and between 70% and 73% of Grade 3 through Grade 6 students met the 

MSP reading standard in 2014. 1,2 ELA Smarter Balanced Assessment scale scores and their 

corresponding reading levels are presented in Exhibit B4 in Appendix B. 

 

1http://reportcard.ospi.k12.wa.us/summary.aspx?groupLevel=District&schoolId=1&reportLevel=State&year=2014-15

2

 http://reportcard.ospi.k12.wa.us/summary.aspx?groupLevel=District&schoolId=1&reportLevel=State&year=2013-

14&yrs=2013-14

About 40% of prospective Washington Reading Corps participants were at Level 

3 or Level 4 on the MSP reading assessment standard in spring 2014.
RMC Research CorporationPortland, OR 3 

Exhibit 2 

Impact Evaluation 

The impact evaluation was guided by 3 questions: 

1 To what extent do student assessment scores differ between those served by 

Washington Reading Corps and those not served by Washington Reading Corps? 

2 To what extent are the Washington Reading Corps programs on track to show 

increases in students achieving grade level reading proficiency? 

3 To what extent are the Washington Reading Corps programs on track to show 

decreases in reading proficiency gaps? 

To analyze the impact of Washington Reading Corps on participating students (Evaluation Question 1), 

RMC Research used a quasi-experimental design with a 2-step matching process to provide more precise 

estimates of program impact than is possible without matching. Using propensity score matching RMC 

Research selected a comparison sample of Grade 4 and Grade 5 students who would have been as likely 

to receive Washington Reading Corps services as the students who did. First, schools that participated in 

Washington Reading Corps in 2014–2015 (n = 32) and enrolled students in Grades 3–6 were matched to 

similar schools that did not participate in Washington Reading Corps (n = 32). Second, Washington 

Reading Corps students in Grade 4 (n = 132) and Grade 5 (n = 122) were matched to similar students at 

their corresponding matched school. Students were matched for similarity on characteristics such as 

race and ethnicity, low income status, and MSP reading scale scores from the prior spring. The number 

of Grade 6 students who participated in Washington Reading Corps was too small (n = 83) to include in 

the analyses. 

A majority of Washington Reading Corps participants were at Level 1 or Level 2 

on the ELA Smarter Balanced Assessment in spring 2015.
4 Washington Reading Corps Impact Evaluation Summary 

RMC Research then used hierarchical linear modeling (HLM) to analyze program impact on Grade 4 and 

Grade 5 students. HLM models the outcome variable (ELA Smarter Balanced Assessment scale score or 

met standard) as a combination of student- and school-level predictor variables. Each of the predictor 

variables included in the model controls for that variable’s influence on the outcome. 

To address Evaluation Questions 2 and 3, RMC Research conducted a descriptive subgroup analysis of 

Washington Reading Corps students in Grades 3–6. Subgroups that had fewer than 10 students 

(e.g., American Indian/Alaska Native, Black/African American) were excluded from the analyses. Further 

detail on the evaluation design and analyses appears in Appendix A.

1 To what extent do student assessment scores differ between those served by 

Washington Reading Corps and those not served by Washington Reading Corps? 

After controlling for the influence of student- and school-level predictors, RMC Research identified 4 key 

findings related to program impact. 

First, Washington Reading Corps had a significant positive impact on Grade 5 students who were 

reading at Level 2 in 2014. Grade 5 students who were identified as reading at Level 2 the spring prior to 

participating in Washington Reading Corps had significantly higher ELA Smarter Balanced Assessment 

scale scores than the comparison students. After controlling for student- and school-level 

characteristics, Washington Reading Corps students’ scale scores were an estimated 27 points higher 

than comparison students. They also were estimated to be 3 times more likely to meet the ELA Smarter 

Balanced Assessment standard in 2015 than the comparison students. In contrast, Grade 5 students who 

were reading at Level 1 the spring prior to participating in Washington Reading Corps had significantly 

lower ELA Smarter Balanced Assessment scale scores than the comparison students. After controlling for 

student- and school-level characteristics, these Washington Reading Corps students’ scale scores were 

an estimated 52 points lower than comparison students. Those comparison students identified as 

Level 1 in 2014 might have received other types of reading interventions during the 2014–2015 year that 

account for this difference. 

Second, Washington Reading Corps had a significant positive impact on Grade 5 Hispanic students’ 

ELA Smarter Balanced Assessment scale scores. After controlling for student- and school-level 

characteristics, Washington Reading Corps had a positive influence on Grade 5 Hispanic students’ ELA 

Smarter Balanced Assessment scale scores. Their scale scores were an estimated 19 points higher than 

Grade 5 Hispanic students who did not participate in Washington Reading Corps. However, participation 

in Washington Reading Corps in Grade 5 did not increase Hispanic students’ likelihood of meeting the 

ELA Smarter Balanced Assessment standard relative to comparison students. 

Third, Washington Reading Corps had no significant impact on Grade 4 students’ ELA Smarter 

Balanced Assessment scale scores. Participation in Washington Reading Corps did not have a significant 

impact on ELA Smarter Balanced Assessment scale scores for Grade 4 students who were at either 

Level 1 or 2 the prior spring. One possible explanation for why Grade 5 participants reading at Level 2 in 

2014 benefited and Grade 4 participants reading at Level 2 in 2014 did not: significantly more students 

in the Grade 4 comparison group were identified as Learning Assistance Program (LAP) Reading status, 

which might have diminished the differences in reading skills between this sample of Grade 4 

Washington Reading Corps participants and their comparison group. 

Finally, Grade 4 students who were reading at Level 1 in 2014 were significantly less likely to meet the 

2015 ELA Smarter Balanced Assessment standard. Students identified as Level 1—well below the 
RMC Research CorporationPortland, OR 5 

standard—might not be best served by a Tier II intervention such as Washington Reading Corps. Nearly 

20% of Grade 4 Washington Reading Corps participants in 2014–2015 were identified as Level 1 the 

prior spring. Similar students in the comparison schools might have received other reading interventions 

during the 2014–2015 academic year that increased their likelihood of meeting the 2015 ELA Smarter 

Balanced Assessment standard. 

2 To what extent are the Washington Reading Corps programs on track to show 

increases in students achieving grade level reading proficiency? 

Evaluation Question 2 was operationalized as a description of the percentage of Washington Reading 

Corps students who met the ELA Smarter Balanced Assessment standard in spring 2015. As Exhibit 3 

shows, Grade 5 Washington Reading Corps participants had the highest percentage meeting the 

standard compared to other grades.

Exhibit 3 

Percentage of all Washington Reading Corps students who met 

ELA Smarter Balanced Assessment standard in 2015. 

Washington Reading Corps also collected data on the number of participating students who met the 

benchmark on locally selected reading screening tools in 2014–2015. Of the 4,240 student tracking logs 

that RMC Research received, 3,730 had complete benchmark data (see Exhibit 4).

Exhibit 4 

Between 40% and 55% of Washington Reading Corps participants met the grade 

level benchmark on locally selected screening tools in 2014–2015. 

 
6 Washington Reading Corps Impact Evaluation Summary 

3 To what extent are the Washington Reading Corps programs on track to show 

decreases in reading proficiency gaps? 

Washington Reading Corps staff have not yet fully identified which subgroup gaps in 

reading achievement might be most meaningful to explore. To support further 

discussion RMC Research calculated percentages for participating students who met the 

ELA Smarter Balanced Assessment standard in 2015 grouped by race and ethnicity, low 

income status, gender, and need for academic support status (sample sizes were too 

small to report for some subgroups, such as American Indian/Alaska Native students). 

These descriptive statistics—unlike the impact results described heretofore—do not 

control for other student- or school-level variables related to meeting the standard. 

Exhibit 5 presents the percentages of all Washington Reading Corps students in Grade 3 

through Grade 6 who met the ELA Smarter Balanced Assessment standard by subgroup 

and the subgroup percentages for matched Washington Reading Corps and comparison 

students in Grade 4 and Grade 5.

 

  

Exhibit 5 

Percentage of students who met the ELA Smarter Balanced Assessment standard in spring 2015. 
RMC Research CorporationPortland, OR 7 

Recommendations

Washington Reading Corps provides tutoring and other literacy supports to struggling readers from 

pre-Kindergarten through Grade 6. This summary examined the impact of Washington Reading Corps for 

a subset of 2014–2015 participants: students in Grades 4 and 5. Of note, the impact of Washington 

Reading Corps tutoring on 3,227 pre-Kindergarten to Grade 3 students could not be examined because 

participation in statewide reading assessments begins in Grade 3. Reading scores for Grade 3 were used 

to establish baseline equivalence between Grade 4 Washington Reading Corps and comparison 

students. The number of Grade 6 students who participated in Washington Reading Corps was too small 

to include in the analyses. 

By matching Washington Reading Corps schools and comparison schools and then matching individual 

Washington Reading Corps participants with similar comparison students, the impact of Washington 

Reading Corps participation on reading outcomes was estimated with high precision. Comparison 

students were similar to Washington Reading Corps participants on many characteristics that may 

influence reading outcomes, including reading scale scores on the prior year’s statewide assessment. 

This quasi-experimental design resulted, however, in some loss of data because not all 2014–2015 

schools and students had assessment scores from the prior year. 

RMC Research has 3 recommendations based on the impact findings: 

 Replicate these impact analyses with 2015–2016 Washington Reading Corps schools and 

students to more fully describe the effects of participation on reading outcomes. This 

summary had to exclude schools that field tested the ELA Smarter Balanced Assessment in 

2013–2014 and therefore did not have MSP reading scores from the spring prior to 2014–2015 

Washington Reading Corps participation for those students. However, more 2015–2016 

Washington Reading Corps schools and students can be included in future impact analyses, 

providing larger samples more fully inclusive of those who participated in Washington Reading 

Corps and with greater statistical power to detect impact. In addition, in 2015–2016 a consistent 

statewide assessment can be used to describe reading skills before and after Washington 

Reading Corps participation and to establish baseline equivalence with the comparison students.

 Continue to be intentional about student selection process. In Grade 5 Washington Reading 

Corps had a positive impact on students who were reading at Level 2 the prior spring. The 

program model is to target students needing Tier II support, and Grade 5 Level 2 students might 

benefit most from Washington Reading Corps. This supposition is further supported by the fact 

that after RMC Research controlled for student- and school-level characteristics, Grade 5 

students who were reading at Level 1 the previous spring had significantly lower scale scores on 

the ELA Smarter Balanced Assessment than the comparison students. Additionally, about 40% of 

participants had already met or exceeded the standard (Level 3 or Level 4) prior to receiving 

Washington Reading Corps services. For students in Grades 4, 5, and 6, Washington Reading 

Corps should consider recommending that sites use prior year state reading assessment data to 

identify participants. This approach would provide consistent inclusion criteria for students in 

these grades across sites and might more precisely identify students best served by the 

Washington Reading Corps. 

 Consider using LAP Reading status as a matching variable and as a covariate in future impact 

studies. RMC Research suggests including students’ LAP Reading status as part of the propensity 

score matching process and as a covariate in the HLM analyses. This approach would more 

precisely determine how LAP services mirror or interact with Washington Reading Corps 
8 Washington Reading Corps Impact Evaluation Summary 

services. For example, students who are identified for LAP Reading support services and also 

participate in Washington Reading Corps might be receiving more supplemental services than 

students who participate in Washington Reading Corps only. 
RMC Research CorporationPortland, OR 9 

Appendix A

Evaluation Methodology

Research Design 

To analyze the impact of Washington Reading Corps on participating students, RMC Research used a 

quasi-experimental design with a 2-step matching process. When random assignment to experimental 

conditions is not feasible in real world settings, quasi-experimental designs provide another rigorous 

approach to exploring the causal relationship between an intervention and related outcomes. Matching 

Washington Reading Corps students and schools to other similar students and schools reduces sampling 

bias and leads to a more precise analysis of whether reading outcomes are related to program 

participation. Identifying a matched comparison group creates the counterfactual condition. In other 

words, what would student outcomes have been if the students had not participated in Washington 

Reading Corps? With matched comparison groups in place, the evaluation team can statistically test 

whether students’ reading outcomes at schools providing Washington Reading Corps services differ 

significantly from schools that do not. If students at the former have significantly higher reading scores 

than students at the latter it can be inferred that the program had a positive impact on student reading 

outcomes. The primary limitation of using matched comparison groups is related to group equivalence. 

Using a matched comparison group, groups are assumed to be equivalent on observable characteristics 

such as family income, gender, race, ethnicity, and prior reading scores but it is not known if the groups 

differ on unobserved characteristics such as attitudes and beliefs about reading. 

Using propensity score matching RMC Research selected a comparison sample of Grade 4 and Grade 5 

students who would have been as likely to receive Washington Reading Corps services as the students 

who did. First, schools that participated in Washington Reading Corps in 2014–2015 (n = 32) that 

enrolled students in Grades 3–6 were matched to similar schools that did not participate in Washington 

Reading Corps (n = 32). Next, Washington Reading Corps students in Grade 4 (n = 132) and Grade 5 

(n = 122) were matched to similar students at their corresponding matched school. 

Sample 

To address program impact (Evaluation Question 1) RMC Research identified a subsample of 

Washington Reading Corps and comparison schools and students using propensity score matching. Use 

of this statistical technique to create comparison groups based on their observed characteristics 

(e.g., low income, reading scores) reduces sampling bias and approximates random assignment. That is, 

propensity score matching achieves a balanced sample by identifying a comparison group that would 

have been as likely to receive Washington Reading Corps had it been offered to them. The propensity 

score represents either a school’s or a student’s probability of being eligible for Washington Reading 

Corps. The program and comparison schools were matched using the publically available 2013–2014 

Demographic Information by School data file and the 2013–2014 MSP-HSPE Scores by School data file 

located on OSPI’s Washington State Report Card website 

(http://reportcard.ospi.k12.wa.us/DataDownload.aspx). Student-level data for Washington Reading 

Corps and comparison students were acquired through a data sharing agreement with OSPI. Schools and 

students were matched using 2014 data to establish baseline equivalence. To address the program 
10 Washington Reading Corps Impact Evaluation Summary 

impact on the 2014–2015 Washington Reading Corps cohort, the evaluation team needed to ensure that 

the comparison schools and students were similar on variables of interest (e.g., reading level) in the 

spring of 2014—several months before Washington Reading Corps was implemented in the participating 

schools. 

Subsample of matched schools. Using nearest neighbor propensity score matching, the evaluation 

team matched the Washington Reading Corps and comparison schools on 7 covariates that were 

hypothesized to be related to students’ scores on the 2015 ELA Smarter Balanced Assessment. Nearest 

neighbor matching pairs each Washington Reading Corps school with a comparison school that has the 

closest probability of being selected for Washington Reading Corps. For example, a Washington Reading 

Corps school with a propensity score of .0419 was matched to a comparison school with a propensity 

score of .0418—meaning that these 2 schools had a nearly identical probability of receiving Washington 

Reading Corps services had they been offered. Comparison schools were matched to Washington 

Reading Corps schools in 2013–2014 on: 

 Percent of students receiving free- or reduced-priced lunch. 

 Percent of Black/African-American students. 

 Percent of Hispanic students. 

 Percent of Native American/Alaska Native students. 

 School reading level indices for Grades 3, 4, and 5 in 2013–2014.

3

The school-level covariate balance is presented in Exhibit A1. Matched Washington Reading Corps and 

comparison schools did not differ significantly on any characteristics included in the propensity score 

matching. 

Exhibit A1

Washington Reading Corps and Comparison Subsample 

Sample Type WRC Schools Comparison Schools

Entire Sample 48 830

Matched sample 32 32

Note. Matched sample only includes schools that enrolled 

Grade 3 to Grade 6 students and that administered the 

Measures of Student Progress (MSP) in Reading in 2014. 

Subsample of matched students. Washington Reading Corps and comparison students were also 

matched using nearest neighbor propensity score matching. To identify matched comparison students, 

RMC Research used students’ 2014 MSP reading scores to establish baseline reading equivalence for 

Washington Reading Corps and comparison students. Students who participated in the ELA Smarter 

Balanced Assessment pilot in 2014 were excluded from the sampling pool. Matching on 2014 MSP 

reading scores ensured that the students in the comparison sample were at a similar baseline reading 

level to Washington Reading Corps students. Accounting for students’ 2014 MSP reading scores led to a 

more precise measurement of Washington Reading Corps’ impact on students’ ELA Smarter Balanced 

Assessment scale scores in 2015. Because of the low number of Grade 6 students that participated in 

Washington Reading Corps in 2014–2015, these students were excluded. The final sample included a 

matched sample of comparison students in Grade 4 and Grade 5 (see Exhibit A2). 

 

3Reading indices were derived from schools’ MSP reading scores.
RMC Research CorporationPortland, OR 11

Exhibit A2

Washington Reading Corps and Comparison Student Subsample 

Grade 4 Grade 5 

Sample Type WRC Comparison WRC Comparison 

Total sampling pool 380 2,158 301 2,106 

Students with 2014 MSP reading scale scores 134 1,805 122 1,749 

Matched students with 2014 MSP reading scale scores 132 132 122 122

Power Analysis. RMC Research conducted a power analysis using Optimal Design software. Each 

criterion in Exhibit A3 was entered into Optimal Design to identify the power to detect small (.20) or 

medium (.40) program effects. All HLM analyses were sufficiently powered to detect a moderate 

program effect. 

Exhibit A3

Power Analysis for Hierarchical Linear Modeling 

Grade 

Number of 

clusters α ICC 

Effect 

Size Power 

4 46 .05 .06 .20 .75 

.40 1.00 

5 42 .05 .19 .20 .30 

.40 .82 

Data 

The impact evaluation included several types of data from different sources (see Exhibit A4). 

Exhibit A4

Washington Reading Corps Impact Data 

Data Type/Data Source 

Demographic 

School (e.g., free and reduced-price lunch, student 

demographics, Title 1 status, Reading Index) 

 OSPI publically available report card data

 Washington Reading Corps

Student (e.g., free and reduced-price lunch, 

race/ethnicity, low income status) 

 OSPI student-level enrollment data

Program 

WRC Grade 3, 4, 5, and 6 participants  Washington Reading Corps student tracking logs

WRC program sites  Washington Reading Corps

Student Achievement 

2014 MSP reading Assessments  OSPI student-level data

2015 ELA Smarter Balanced Assessment 
12 Washington Reading Corps Impact Evaluation Summary 

Impact Analysis 

To address whether participation in Washington Reading Corps influenced student reading scores on the 

ELA Smarter Balanced Assessment in Grades 4 and 5, RMC Research used cross-sectional HLM rather 

than a longitudinal model for 2 reasons: (a) student outcomes in 2015 could not be compared to prior 

years without access to student growth percentile data and (b) 2015 was the first year Washington State 

administered the Smarter Balanced Assessments, which might influence trends in reading achievement. 

The cross-sectional HLM analysis is an equally rigorous analytic technique that accounts for Washington 

Reading Corps’ hierarchical structure, meaning that students (hierarchical Level 1) are nested in 

different schools (hierarchical Level 2). As an analytic technique HLM provides a more precise estimate 

of individual student effects and school effects. In other words, because Washington Reading Corps is 

implemented in multiple schools HLM can identify how student scores on the ELA Smarter Balanced 

Assessment may be influenced by each school’s unique characteristics and also by each student’s unique 

characteristics. The evaluation team included the following student- and school-level characteristics in 

each HLM model either because they were of interest to Washington Reading Corps or they were 

expected to influence student outcomes. Students’ 2014 MSP reading level was included as a covariate 

to account for the relationship between students’ reading level in 2014 and their reading level in 2015. 

Exhibit A4

School and Student Characteristics Included in Hierarchical Linear Models 

Hierarchical Level Characteristics Variable Type Values 

Level 2 School Percent free and reduced-price lunch Continuous 0%–100% 

Title I Status Dichotomous 0 = no, 1 = yes 

Number of years in WRC Continuous 0–17

WRC Participation Dichotomous 0 = no, 1 = yes 

Level 1 Student 2014 MSP reading score level Dichotomous Level 1 (0 = no, 1 = yes) 

Level 2 (0 = no, 1 = yes) 

Level 3 (0 = no, 1 = yes) 

Level 4 (0 = no, 1 = yes) 

Low Income Dichotomous 0 = no, 1 = yes 

Hispanic Dichotomous 0 = no, 1 = yes 

Outcome ELA SBA scale score Grade 4 Continuous 2131–2663

ELA SBA scale score Grade 5 Continuous 2201–2701

ELA SBA met standard Dichotomous 0 = no, 1 = yes
RMC Research CorporationPortland, OR 13

Appendix B

Supplemental Data

Exhibit B1

Washington Reading Corps Student Sample Characteristics 

Student Characteristics Grade 3 Grade 4 Grade 5 Grade 6 

Student Characteristic % n % n % n % n

American Indian or Alaskan Native 2% 13 3% 10 3% 9 5% 4 

Asian 4% 24 5% 18 7% 18 13% 11

Black 6% 34 5% 18 6% 15 12% 10

Hispanic 28% 156 26% 88 21% 56 29% 25

White 52% 297 51% 172 53% 140 36% 31

Native Hawaiian/Pacific Islander 1% 5 2% 5 1% 2 0% 0 

Two or more Races 7% 39 7% 25 8% 22 7% 6 

Low Income 68% 384 69% 232 69% 180 77% 67

Homeless 4% 20 7% 25 2% 6 2% 2 

S504 2% 12 2% 7 4% 10 0% 0 

Gifted 0% 2 1% 3 1% 3 1% 1 

Migrant 4% 20 2% 6 1% 3 2% 2 

Special Education 14% 78 16% 54 12% 31 18% 16

Title 3 Native American 0% 0 0% 0 0% 1 0% 0 

LAP Reading 29% 163 19% 62 9% 23 26% 23

Note. Grade 3 n = 568, Grade 4 n = 336, Grade 5 n =263, Grade 6 n = 87.
14 Washington Reading Corps Impact Evaluation Summary 

Exhibit B2

Matched Comparison School Characteristics 

School Characteristics Comparison WRC 

Total Enrollment 424 372

Percent American Indian or Alaskan Native 1% 3%

Percent Black 2% 3%

Percent Hispanic 24% 20%

Percent White 61% 60%

Percent 2 Or More Races 8% 8%

Percent Migrant 3% 2%

Percent Transitional Bilingual 14% 11%

Percent Special Education 13% 17%* 

Percent Free or Reduced-Priced Meals 55% 55%

Percent Section 504 1% 2%

ReadingLevelIndex.3 3 3 

ReadingLevelIndex.4 3 3 

ReadingLevelIndex.5 3 3 

Note. Schools were matched using OSPI report card data from 2013–2014 using 

Percent American Indian or Alaskan Native, Percent Black, Percent Hispanic, 

Percent Free or Reduced-Priced Meals, and all 3 Reading Level Indices. 

Comparison n= 32, Washington Reading Corps n = 32. 

*p < .05. 
RMC Research CorporationPortland, OR 15

Exhibit B3

Matched Student Characteristics (2014—2015 Cohort) 

Grade 4 Grade 5 

Student Characteristics WRC Comparison WRC Comparison 

American Indian or Alaskan Native 2% 1% 2% 1%

Black 6% 2% 4% 2%

Hispanic 28% 25% 24% 24%

White 50% 58% 55% 55%

Migrant 2% 1% 2% 6%

Special Education 16% 18% 16% 22%

Title3 Native American 0% 1% 1% 1%

LAP Reading 17% 37%* 14%* 2%

S504 2% 2% 4% 5%

Gifted 1% 2% 0% 3%

Low Income 66% 67% 71%* 57%

Homeless 5% 3% 3% 2%

2014 MSP Level 1 19% 23% 7% 6%

2014 MSP Level 2 40% 31% 51% 50%

2014 MSP Level 3 31% 31% 39% 39%

2014 MSP Level 4 11% 14% 4% 5%

2014 MSP Reading Scale Score 393 393 395 396

2015 Met Standard 23% 38%* 33% 45%* 

2015 SBA ELA Smarter Balanced 

Assessment Scale Score 

2422 2426 2464 2483*

Note. Grade 4 Washington Reading Corps n = 132, Comparison n = 132; Grade 5 Washington Reading 

Corps n = 122, Comparison n = 122.

*p < .05.

Exhibit B4

Reading Levels Aligned With Smarter ELA Smarter Balanced Assessment and MSP Reading Scale Scores 

Smarter Balanced MSP 

Reading Level Definition Grade 4 Grade 5 Grade 4 Grade 5 

Level 1 Well below standard 2131–2415 2201–2441 275–274 275–274

Level 2 Below standard 2416–2472 2442–2501 375–399 375–399

Level 3 Met standard 2473–2532 2502–2581 400–423 400–421

Level 4 Exceeds standard 2533–2663 2582–2701 424–475 422–475
16 Washington Reading Corps Impact Evaluation Summary 

Exhibit B5

Grade 4 Hierarchical Linear Model: ELA Smarter Balanced Assessment Scale Score Outcome 

Model 1 Model 2 Model 3 Model 4 Model 5 

Predictor Est. SE Est. SE Est. SE Est. SE Est. SE

Intercept 2422.88 11.28 2422.25 30.39 2323.40 64.07 2326.83 70.41 2274.24 122.65 

Low income 7.81 28.49 21.67 31.35 21.58 31.65 58.42 63.99 

Hispanic -17.44 12.83 -9.28 10.41 -9.03 10.17 -7.92 20.11 

2014 MSP Level 2 78.22 44.31 79.04 43.07 88.52 53.25 

2014 MSP Level 3 124.31** 45.37 124.50** 44.56 127.11** 45.08 

2014 MSP Level 4 178.90*** 47.04 178.69*** 46.72 185.46*** 51.62 

WRC -7.40 17.63 34.06 59.57 

Percent FRP lunch -0.74 1.06 

Title I 40.08 43.74 

Years in WRC 1.05 1.47 

WRC X Low Income -73.22 63.20 

WRC X Hispanic 3.47 22.60 

WRC X 2014 MSP Level 2 -19.50 29.57 

Deviance 3411.11 3494.49 3339.61 3329.82 3297.66 

ICC 0.06 

Note. School n = 46, student n = 268. 

***p < .001. **p < .01. *p < .05. 
RMC Research CorporationPortland, OR 17

Exhibit B6

Grade 4 Hierarchical Linear Models 1–3: ELA Smarter Balanced Assessment Met Standard Outcome 

Model 1 Model 2 Model 3 

Predictor Est. SE OR CI Est. SE OR CI Est. SE OR CI

Intercept -0.91 0.19 0.40 (.28, .59) -0.58 0.25 0.56 (.33,.93) -2.34 0.52 0.10 (.03,.27) 

Low Income -0.17 0.31 0.84 (.46,1.54) 0.05 0.35 1.04 (.52, 2.11) 

Hispanic -1.02*** 0.25 0.36 (.22,.60) -0.92** 0.28 0.40 (.23, .70) 

2014 MSP Level 2 0.81 0.56 2.25 (.75, 6.77) 

2014 MSP Level 3 2.23*** 0.51 9.28 (3.39, 25.45) 

2014 MSP Level 4 3.55*** 0.56 34.66 (11.47, 104.73) 

Note. School n = 46, student n = 268. 

***p < .001. **p <. 01. *p < .05. 
18 Washington Reading Corps Impact Evaluation Summary 

Exhibit B7

Grade 4 Hierarchical Linear Models 4 and 5: ELA Smarter Balanced Assessment Met Standard Outcome 

Model 4 Model 5 

Predictor Est. SE OR CI Est. SE OR CI

Intercept -2.01 0.55 0.13 (.04, .41) -1.27 0.61 0.28 (.08,.96) 

Low Income 0.04 0.37 1.04 (.51,2.15) -0.67 0.36 0.51 (.25,1.05) 

Hispanic -0.94* 0.29 0.39 (.22,.69) -1.07** 0.35 0.34 (.17,.69) 

2014 MSP Level 2 0.90 0.56 2.47 (.82,7.41) 0.59 0.68 1.81 (.47, 7.70) 

2014 MSP Level 3 2.36*** 0.50 10.63 (3.96,28.53) 2.33*** 0.49 10.27 (3.93, 26.82) 

2014 MSP Level 4 3.72*** 0.55 41.31 (13.90, 122.76) 3.67*** 0.54 39.28 (13.49, 114.36) 

WRC -0.93* 0.37 0.39 (.19,.82) -1.96** 0.62 0.14 (.04, .50) 

Percent FRP lunch 0.02 0.01 1.02 (1.00,1.05) 

Title I -0.15 0.44 0.86 (.36, 2.09) 

Years in WRC 0.04 0.06 1.04 (.93,1.17) 

WRC X Low Income 1.13 0.72 3.09 (.74, 12.87) 

WRC X Hispanic -0.02 0.97 0.98 (.34,2.84) 

WRC X 2014 MSP Level 2 0.65 0.71 1.92 (.48, 7.70) 

Note. School n = 46, student n = 268. 

***p < .001. **p < . 01. *p < .05 
RMC Research CorporationPortland, OR 19

Exhibit B8

Grade 5 Hierarchical Linear Model: ELA Smarter Balanced Assessment Scale Score Outcome 

Model 1 Model 2 Model 3 Model 4 Model 5 

Predictor Est. SE Est. SE Est. SE Est. SE Est. SE

Intercept 2472.86 7.28 2494.28 9.23 2412.56 17.73 2420.71 26.23 2437.64 26.73 

Low income -33.78*** 9.03 -20.78* 8.98 -19.67* 8.76 -32.86* 12.76 

Hispanic 1.29 7.51 -6.79 5.72 -6.67 6.01 -16.68** 6.02 

2014 MSP Level 2 45.97** 14.78 44.91* 14.55 29.72 15.41 

2014 MSP Level 3 120.17*** 17.72 119.21*** 17.73 119.13*** 16.85 

2014 MSP Level 4 150.21*** 18.02 149.29*** 18.02 148.44*** 17.92 

WRC -17.95 14.06 -51.51* 19.35 

Percent FRP lunch 0.05 0.37 0.00 0.38 

Title I -2.91 19.25 2.96 19.59

Years in WRC 0.21 1.32 0.03 1.27 

WRC X Low Income 23.10 17.98 

WRC X Hispanic 18.91* 11.19 

WRC X 2014 MSP Level 2 27.27* 12.72 

Deviance 2695.69 2671.61 2553.65 2534.69 2507.58

ICC 0.19 

Note. School n = 42, student n = 244. 

***p < .001. **p < .01. *p < .05. 
20 Washington Reading Corps Impact Evaluation Summary 

Exhibit B9

Grade 5 Hierarchical Linear Models 1-3: ELA Smarter Balanced Assessment Met Standard Outcome 

Model 1 Model 2 Model 3 

Predictor Est. SE OR CI Est. SE OR CI Est. SE OR CI

Intercept -0.43 0.19 0.65 (.44, .94) 0.07 0.27 1.07 (.623, 1.83) -1.50 0.76 0.22 (.05,1.04) 

Low Income -0.80** 0.26 0.45 (.27,.74) -0.70* 0.32 0.49 (.26,.93) 

Hispanic -0.01 0.34 0.99 (.50,1.95) -0.19 0.38 0.82 (.39,1.74) 

2014 MSP Level 2 0.64 0.72 1.91 (.46,7.92) 

2014 MSP Level 3 2.57*** 0.96 13.02 (3.02,56.15) 

2014 MSP Level 4 3.01** 0.96 20.46 (3.07,136.40) 

Note. School n = 42, student n = 244. 

***p < .001. **p < .01. *p < .05. 
RMC Research CorporationPortland, OR 21

Exhibit B10 

Grade 5 Hierarchical Linear Models 4 and 5: ELA Smarter Balanced Assessment Met Standard Outcome 

Model 4 Model 5 

Predictor Est. SE OR CI Est. SE OR CI

Intercept -1.21 0.96 0.30 (.04,2.09) -0.53 0.99 0.59 (.08,4.41) 

Low Income -0.72* 0.33 0.49 (.26,.93)) -1.39 0.47 0.25 (.09,.64) 

Hispanic -0.19 0.39 0.83 (.38,1.79) 0.00 0.53 1.00 (.35, 2.84) 

2014 MSP Level 2 0.55 0.68 1.74 (.45,6.66) -0.11 0.69 0.90 (.23, 3.54) 

2014 MSP Level 3 2.52*** 0.72 12.49 (3.02,51.73) 2.48** 0.76 11.99 (2.65, 54.14) 

2014 MSP Level 4 3.05** 0.96 21.16 (3.17,141.28) 3.04** 0.97 21.06 (3.14, 141.49) 

WRC -0.66 0.54 0.52 (.17,1.53) -1.79* 0.83 0.17 (.03,.89) 

Percent FRP lunch 0.00 0.02 1.00 (.97,1.04) 0.00 0.02 1.00 (.97, 1.04) 

Title I -0.17 0.85 0.85 (.15,4.78) 0.22 0.02 1.25 (.19, 8.18) 

Years in WRC 0.03 0.04 1.03 (.94,1.13) 0.01 0.04 1.01 (.93,1.10) 

WRC X Low Income 1.02 0.72 2.79 (.68, 11.5) 

WRC X Hispanic -0.42 0.73 0.66 (.16, 2.78) 

WRC X 2014 MSP Level 2 1.15* 0.55 3.19 (1.08, 9.39) 

Note. School n = 42, student n = 244. 

***p < .001. **p < . 01. *p < .05.