1 

The Tutoring Partnership: 

SIF Implementation and Outcome 

Evaluation

Final Report 

June 2017

Submitted by: 

Mallory Warner-Richter, Ashley Hirilall, 

Winnie Li, & Kathryn Tout
2 

Acknowledgements

This report is based upon work supported by the Social Innovation Fund (SIF), a key White House 

initiative and program of the Corporation for National and Community Service (CNCS). The Social 

Innovation Fund combines public and private resources to grow the impact of innovative, communitybased solutions that have compelling evidence of improving the lives of people in low-income 

communities throughout the United States. SIF support for this project was provided through Greater 

Twin Cities United Way and Generation Next. Opinions or points of view expressed in this document are 

those of the authors and do not necessarily reflect the official position of, or a position that is endorsed 

by, the Corporation, Generation Next, United Way or the Social Innovation Fund Program.

We would like to thank the staff of the Saint Paul Public Schools Foundation and WithInsight for sharing 

their understanding of the Tutoring Partnership and its evolution during the study. Thanks also go to the 

Center for Applied Research and Educational Improvement (CAREI) at the University of Minnesota. CAREI 

staff provided technical assistance throughout the project, and was a key partner in the implementation 

analysis in the first two years of this SIF project. 

Child Trends would also like to thank the staff at the Research, Evaluation, and Assessment (REA) 

departments in Saint Paul Public Schools and Minneapolis Public Schools. REA staff compiled student 

demographic and assessment records for thousands of tutored students for this study. 

Finally, we would like to thank the staff and tutors at the tutoring programs participating in this 

evaluation. They were generous with their time to complete surveys and participate in interviews. We 

could not have the depth of understanding we have without their participation. 
3 

Table of Contents

Executive Summary.......................................................................................................................... 5

Background ................................................................................................................................. 5

Description of Study ..................................................................................................................... 5

Findings ...................................................................................................................................... 7

Conclusions ................................................................................................................................. 8

Introduction .................................................................................................................................. 10

The Tutoring Partnership ............................................................................................................ 10

Tutoring as a Strategy to Address the Achievement Gap ................................................................ 10

Prior Research............................................................................................................................ 11

Theory of Change....................................................................................................................... 12

Evaluation Overview ................................................................................................................... 15

Implementation Evaluation Questions .......................................................................................... 16

Outcome Evaluation Questions .................................................................................................... 16

Implementation Study .................................................................................................................... 17

Implementation Study Design...................................................................................................... 17

Implementation Measures and Analysis ........................................................................................ 17

Implementation Findings............................................................................................................. 19

Discussion of Implementation Findings......................................................................................... 34

Outcomes Study ............................................................................................................................ 36

Study Design ............................................................................................................................. 36

Sample Description..................................................................................................................... 37

Outcome Measures..................................................................................................................... 38

Outcome Analysis....................................................................................................................... 40

Outcome Findings: Student Proficiency and Growth....................................................................... 43

Discussion of Outcome Findings................................................................................................... 51

Conclusions and Limitations ............................................................................................................ 52

Conclusions............................................................................................................................. 52

Limitations.............................................................................................................................. 53

References .................................................................................................................................... 54

Appendix A. Tutoring Partnership Staff Interview Protocol ................................................................. 56

Appendix B. Tutoring Program Staff Interview .................................................................................. 58

Appendix C. Tutor Survey ............................................................................................................... 61
4 

Appendix D. Tutoring Program Staff Survey...................................................................................... 66

Appendix E. Professional Development Workshops by SIF Years Offered............................................. 73

Appendix F. Professional Development Attendance per Strategy......................................................... 75

Appendix G. Technical Assistance Usage per Strategy ....................................................................... 78

Appendix H. Tutor Training Attendance per Strategy......................................................................... 79

Appendix I. Tutoring Partner Self-Assessment of Best Practices 2015-16............................................. 81

Appendix J: Cleaning Procedures and Sampling Decisions.................................................................. 88

Appendix K. Assessment Outcome Data Figures................................................................................ 89

Appendix L. Assessment Data Figures, Subgroup Analyses................................................................. 95
5 

Executive Summary

Background

The Tutoring Partnership was a network of 28 community-based programs that provide intentional 

academic interventions through tutoring to students in Saint Paul and Minneapolis. The common goal 

across these programs was improving academic outcomes for students. The Tutoring Partnership, 

operated by the Saint Paul Public Schools Foundation, supported this common goal by helping improve 

program quality through provision of research-based professional development, technical assistance, and 

tutor training. In 2013, the Saint Paul Schools Foundation was awarded a Social Innovation Fund (SIF) 

sub grant from Greater Twin Cities United Way to examine the implementation of the Tutoring 

Partnership and student outcomes in tutoring programs receiving various levels of capacity building 

services from the Tutoring Partnership. 

In the first two program years of the SIF grant (2013-2014 and 2014-2015 school years), the Saint Paul 

Public Schools Foundation implemented the Tutoring Partnership model through two strategic 

approaches. Strategy A, the Tutoring Partnership model that pre-dates the SIF grant, included

professional development, technical assistance and tutor training for programs in Saint Paul. Strategy B, 

the enhanced Tutoring Partnership model, included increased intensity and depth of support and 

supplemental funding for four additional organizations in Saint Paul. In the third and final SIF grant year 

(2015-2016 school year), the Tutoring Partnership expanded to serve Minneapolis locations of the 

programs receiving enhanced Tutoring Partnership services in Strategy B. For this report, these 

Minneapolis-based locations receiving enhanced Strategy B services only in Year 3 are called Strategy B3. 

In the middle of the third SIF year, the Saint Paul Public Schools Foundation announced that it was 

dissolving as an organization in August 2016, therefore the Tutoring Partnership would no longer in 

operation after Year 3. The mid-year announcement was accompanied by staff layoffs, and changes in 

plans to meet more ambitious service delivery goals in the 2015-2016 school year. The implications of the 

program closure are discussed throughout the report.

This Executive Summary summarizes findings from the implementation and outcomes study of the 

Tutoring Partnership in the three years funded through SIF. 

Description of Study

The study builds on previous research about the Tutoring Partnership by examining both student-level 

and program-level outcomes. In addition to learning more about the effect of the Tutoring Partnership, 

this study asked a series of questions about how the Partnership was implemented in the SIF years. 

Implementation Evaluation Questions 

To better understand the implementation of the Tutoring Partnership, the evaluation posed four primary 

implementation evaluation questions. Data to answer three of the four questions were collected in all 

three years of the evaluation. Additional data were collected in Year 3 to address questions about 

program and tutor perceptions of the Tutoring Partnership. 
6 

• Years 1-3: What are program and tutor characteristics? 

• Years 1-3: To what extent and how did the tutoring programs in Strategies A, B, and B3 

participate in the services offered by the Tutoring Partnership (e.g. professional development, 

technical assistance, and tutor training)? 

• Year 3 only: How do programs perceive the effectiveness of services provided to them through 

the Tutoring Partnership? What is their level of satisfaction with the services? How do programs 

perceive their growth? 

• Years 1-3: To what extent do programs achieve Best Practices for Tutoring Programs as rated by 

programs on the Best Practices self-assessment?

Outcomes Evaluation Questions 

The outcome of the Tutoring Partnership was measured both by changes in program quality and by the 

reading achievement of tutored students. Because the Tutoring Partnership was especially concerned 

about the reading achievement of disadvantaged students, the evaluation explored data by those 

subgroups of students. Due to limited sample size, the evaluation question about changes in program 

quality is exploratory. 

Confirmatory Outcome Questions 

• Years 1-3: To what extent do the three capacity building Strategies (Strategies A, B, and B3) 

affect reading growth for students?

• Years 1-3: To what extent do the three capacity building Strategies (Strategies A, B, and B3) 

affect reading proficiency for students?

Exploratory Outcome Questions

• Years 1-3: Low-income students receiving tutoring: 

o To what extent does tutoring have a differing effect on reading growth and proficiency 

for low-income students, compared to students not from low-income families?

• Years 1-3: Racial/ethnic subgroups receiving tutoring: 

o To what extent does tutoring have a differing effect on reading growth and proficiency 

for specific racial and ethnic groups (i.e., American Indian, Asian American, Latino, 

African American, and Caucasian)? 

• Years 1-3: Program quality: 

o To what extent does participation in the Tutoring Partnership improve program quality as 

measured by Program Quality Assessment over time? Does this growth differ by 

Strategy? Does this growth differ based on the number of years receiving the 

intervention? 

Measures 

The implementation study used a mixed method approach to answer the implementation evaluation 

questions. These included surveys of both tutors and tutoring program staff, as well as interviews of 

Tutoring Partnership staff and tutoring program staff. Administrative records about participation in 

Tutoring Partnership services were also included in the implementation study. 

The outcome study used school district assessments administered by Saint Paul Public Schools and 

Minneapolis Public Schools for tutored students in grades kindergarten through 5th grade. Finally, the 

Program Quality Assessment was used to measure program quality across all Strategies. Table 1 

summarizes the outcome study measures. 
7 

Table 1: Outcome Study Measures, Student-level and Program-level (2013-2016)

Saint Paul Strategies A & B Minneapolis Strategy B3

Student-level measures Grade(s) Grades(s)

 MPS Total Literacy fall-to-spring gain Not administered K 

 Mondo Text Level fall-to-spring gain K-5 Not administered

 MAP fall-to-spring gain Not administered 1-5 

 MCA spring scale score 3-5 3-5 

 MCA proficiency 3-5 3-5 

Program-level measures Years Years 

 Program Quality Assessment (PQA) 1-3 1-3 

Note: All analyses compare all tutored students and students in different subgroups when sample sizes allow (students eligible for 

free/reduced-price lunch; White, Black, Latino, Asian, and Native American students). 

Analysis 

Interviews and surveys were coded and analyzed descriptively. Where sample size allowed, t-tests were 

used to detect significant differences between groups. T-tests were also used to examine differences 

between domains and the total score on the Program Quality Assessment. Student-level assessments 

were analyzed using multivariate regression and a difference-in-difference approach. This approach 

compared the reading growth and proficiency of tutored students in the year they were tutored to the 

year before and after they received tutoring services. 

Findings

Implementation Findings 

• Programs served students in schools and community-based settings. Programs varied widely in 

program size, number of staff, and number of tutors. 

• Tutors in all Strategies had high levels of education, were mostly White, and predominately English 

speakers. 

• Strategy B/B3 programs had higher rates of participation in key Tutoring Partnership services such as 

professional development workshops, technical assistance, and tutor training. Requiring participation 

in these services did have an association with higher levels of participation when comparing rates to 

Strategy A, who were not required to participate. 

• Overall, program staff felt the support they received from the Tutoring Partnership was beneficial. 

Programs found the tutor trainings to be most helpful; however, participation in tutor training was 

relatively low. 

• Programs across all Strategies rated themselves highly as using Best Practices for Tutoring Programs. 

Overall Student Findings 

• Tutored students in Strategy A had significantly higher growth than the comparison group in all five 

Mondo subtests (Letter Recognition, Letter Sound, Print Concepts, Phonemic Awareness, and Word 

Knowledge) in Years 1 and 2. Strategy A tutored students also showed significant gains over the 

comparison students in three subtests in Year 3 (Letter Recognition, Letter Sound, and Print 

Concepts). 

• Tutored students in Strategy B had significantly higher growth in all five Mondo subscales in Year 2. 

There was not significant growth on any Mondo subscales for Strategy B programs in Years 1 and 3. 
8 

• Tutored kindergarten students in Strategy B3 had significantly higher growth in Year 1 (a year when 

their programs were not participating in the Tutoring Partnership), and all tutored students in 

Strategy B3 programs had greater growth on the MCA in Year 3 (the only Tutoring Partnership year). 

 

Low-Income Students

• Low-income and higher income students tutored in Strategies A and B saw significant growth in 

reading skills in Years 1-3 combined. 

• Low-income tutored students in Strategy B3 had significantly lower gains than comparison students 

on the MAP assessment. 

 

Race/Ethnicity Subgroups 

• Tutored students from all racial/ethnic backgrounds in Strategy A made significantly greater gains on 

the five Mondo subscales. 

• Significant gains were also found for most subgroups tutored students in Strategy B on Letter 

Recognition Letter Sound, Print Concepts, Phonemic Awareness (Asian, Black, Latino, and White), 

and Word Knowledge (Asian, Black, and Latino). 

• Black students tutored in Strategy B3 programs made significantly greater gains on Total Literacy. 

• Latino and White students in the comparison group for Strategy B3 made significant gains over 

tutored students on MAP. 

Program Quality 

• Programs in Strategy A saw a decline in the quality of Academic Climate/Skill Building from Year 1 to 

Year 3. 

• Strategy B3 programs had a significant increase in the Supportive Environment domain, with the 

highest score achieved in Year 3.

Conclusions

• Overall, tutoring programs were satisfied with the Tutoring Partnership services, and 

found them to be helpful to their programs. Programs found tutor training to be the most 

helpful service, however, it was not utilized by most tutors. For initiatives like the Tutoring 

Partnership, engaging tutors who may not be familiar with their services may be a difficult 

challenge. Initiatives and programs offering tutor training may need to provide additional 

incentives to entice tutors to training opportunities, make trainings highly practical and hands-on, 

and provide training at a variety of times and locations to boost training participation. 

• Requiring participation in Tutoring Partnership services increased uptake. Strategy B 

and Strategy B3 programs had requirements around their participation in Tutoring Partnership 

services. Thus, they had higher levels of participation in PD workshops and technical assistance 

services. Programs in Strategy A could opt-in to services, and tended to have lower levels of 

participation. Initiatives like the Tutoring Partnership should consider the requirements of 

participating and the readiness to change when they are working with programs on quality 

improvement. 

• Tutor and student demographics are not aligned. The Tutoring Partnership had a primary 

goal of closing the achievement gap that exists between low-income and high-income students, 

and between white students and students of color. While tutors were highly-educated, their 

language and racial/ethnic backgrounds did not match students they were working with who 
9 

were primarily students of color, and more than 40% were English Language Learners. Efforts by 

the Tutoring Partnership to increase the cultural competency of tutoring programs and their 

tutors were just getting underway during the SIF grant. Tutoring programs should try moving 

forward to continue building cultural competency, awareness of racial biases, and relationship 

building skills as a key aspect of their training. 

• Overall, tutored students are making greater gains in the year they are tutored. These 

gains are present for low-income and higher-income students, and for students from different 

racial/ethnic backgrounds. Limitations in the sample sizes and sample demographics made it 

challenging to state if one Tutoring Partnership Strategy was more effective than another at 

producing these gains. 

• Programs maintained moderate levels of quality. Based on PQA scores, programs in all 

three Strategies had moderate levels of quality in the three years of the SIF funding. It was 

difficult to determine changes in program quality because of limited sample sizes; further study 

into the associations of program quality to student outcomes would benefit the tutoring field. 
10

Introduction

The Tutoring Partnership

The Tutoring Partnership was a network of 28 community-based programs that provide intentional 

academic interventions through tutoring to students in Saint Paul and Minneapolis. The common goal 

across these programs was improving academic outcomes for students. The Tutoring Partnership, 

operated by the Saint Paul Public Schools Foundation, supported this common goal by helping improve 

program quality through provision of research-based professional development, technical assistance, and 

tutor training. In 2013, the Saint Paul Schools Foundation was awarded a Social Innovation Fund (SIF) 

sub grant from Greater Twin Cities United Way to examine the implementation of the Tutoring 

Partnership and student outcomes in tutoring programs receiving various levels of capacity building 

services from the Tutoring Partnership. 

In the first two program years of the SIF grant (2013-2014 and 2014-2015 school years), the Saint Paul 

Public Schools Foundation implemented the Tutoring Partnership model through two strategic 

approaches. Strategy A, the Tutoring Partnership model that pre-dates the SIF grant, included

professional development, technical assistance and tutor training for programs in Saint Paul. Strategy B, 

the enhanced Tutoring Partnership model, included increased intensity and depth of support and 

supplemental funding for four additional organizations in Saint Paul. In the third and final SIF grant year 

(2015-2016 school year), the Tutoring Partnership expanded to serve Minneapolis locations of the 

programs receiving enhanced Tutoring Partnership services in Strategy B. For this report, these 

Minneapolis-based locations receiving enhanced Strategy B services only in Year 3 are called Strategy B3. 

In the middle of the third SIF year, the Saint Paul Public Schools Foundation announced that it was 

dissolving as an organization in August 2016, therefore the Tutoring Partnership would no longer in 

operation after Year 3. The mid-year announcement was accompanied by staff layoffs, and changes in 

plans to meet more ambitious service delivery goals in the 2015-2016 school year. The implications of the 

program closure are discussed throughout the report.

This final implementation and outcome report summarizes analyses that explore how the Tutoring 

Partnership was implemented over the three years of SIF funding, the changes in program quality 

associated with participation in the Tutoring Partnership, and the reading growth and proficiency of 

students enrolled in participating tutoring programs. 

Tutoring as a Strategy to Address the Achievement Gap

The educational achievement gap in Minnesota is one of the largest in the country (Vanneman et al., 

2009). Saint Paul Public Schools student achievement data show that the majority of students are not 

meeting the minimum state educational standards in math and reading (Saint Paul Public Schools Data 

Center, 2013). Disparities in achievement are evident for students of color and students from low-income 

households; across the district, the population of students requiring extra support is growing. 

Tutoring may be an effective way to help raise student achievement and close the achievement gap

(Bixby et al., 2011). Based on a request for more coordination among community programs in the Saint 
11

Paul Public Schools, the Tutoring Partnership began in 2007 as a small network of seven tutoring 

programs. The goal of the Tutoring Partnership was to create a coordinated a city-wide effort to increase 

the number of well-trained tutors and improve the quality of tutoring programs in Saint Paul. Prior to the 

Tutoring Partnership, there were no standards or quality indicators for tutoring in Saint Paul Public 

Schools. Programs operated without mutually agreed upon best practices or a common measurement 

system. Many tutoring programs did not have a process for continuous quality improvement or 

evaluation. Therefore, there was limited data showing that associations of tutoring with desired 

outcomes. From 2007 to 2012, the Tutoring Partnership added more tutoring programs to the network 

and provided standards and research-based best practices toward which programs could work. In the 

2012-2013 school year, the Tutoring Partnership instituted four requirements for programs to become 

part of the Partnership. Programs needed to agree to: 

• Conduct criminal background checks on those working with minors

• Provide tutor orientation and ongoing training

• Participate in the study of tutoring by submitting student rosters and receiving info about their 

program

• Conduct a self- assessment of their program each year

In the 2013-2014 school year, with the new SIF award, the Tutoring Partnership further refined their 

service and sought to improve program quality through professional development, tutor training and 

technical assistance. Through SIF, the goal of the Tutoring Partnership was to ensure students who are 

more at-risk for academic challenges have access to higher quality programs and subsequently better 

academic outcomes.

Prior Research

A goal of SIF funding is to expand the portfolio of evidence-based solutions to key community challenges

(Corporation for National and Community Service, 2016). Prior to the SIF evaluation, the Tutoring 

Partnership had an annual evaluation focused on student outcomes in reading and math. These 

evaluations used methods such as ordinary least squares regression and hierarchical linear modeling to 

compare the growth and proficiency of students tutored in a Tutoring Partnership program to students 

who were not tutored in a Tutoring Partnership program. Findings in favor of the Tutoring Partnership 

tutored students were found annually, though findings for various assessments were not consistent over 

time. 

There were strengths and limitations to this evaluation approach. A strength of the approach was that 

tutored students could be compared to demographically similar students who did not receive tutoring. 

This type of annual evaluation gave a picture of how student outcomes were changing for cohorts of 

tutored students. A challenge of the annual evaluations was that selection bias was a threat to validity. 

Each tutoring program operated with their own selection criteria for students, and it was difficult to 

estimate or correct for selection bias effects in the annual evaluations. While the evaluation used 

propensity score matching to create a matched comparison group, this matching approach was still 

limited by the information available on characteristics related to selection. An additional challenge was 

that evaluators did not have longitudinal data on students to examine the longer-term growth that may 

be experienced because of tutoring. Finally, limited information about the implementation of the Tutoring 

Partnership and tutoring programs also gave an incomplete understanding of student results.
12

With the more intensive services offered to programs in Strategy B and B3 through the SIF grant, this

evaluation focused on reading growth and proficiency of students attending programs receiving varying 

levels of Tutoring Partnership support, as well as growth in tutoring program quality. The evaluation 

design built upon the previous research on the Tutoring Partnership, while making methodological 

improvements to overcome some of the challenges experienced in the previous evaluations. Prior

research on the Tutoring Partnership was a good starting point for building the evidence of the impact of 

the Tutoring Partnership. SIF funding helps to better understand the implementation of the Tutoring 

Partnership and its role in closing the achievement gap. 

Theory of Change

With its collaborative partnership with local public school districts, the Tutoring Partnership sought to 

accelerate student achievement and close academic achievement gaps by increasing the quality of 

tutoring programs and tutors in Saint Paul and Minneapolis. The Partnership posited that program quality 

directly affect student outcomes. Research showed that students who participated in high-quality youth 

programs had higher academic achievement including higher standardized test scores, better school 

attendance, higher graduation rates, increased engagement in learning, and better attitudes toward 

school (Little et al., 2008; Vandell et al., 2007; Eccles & Gootman, 2002; Roth et al., 1998). Because of 

the direct link between program quality and student outcomes, the Tutoring Partnership provided 

intentional, data-driven professional development, tutor training and technical assistance grounded in 

Best Practices for Tutoring Programs (Bixby et al., 2011). Through this focus on quality and best 

practices, tutoring programs sought to improve tutoring practices and positively affect student 

achievement. 

Core Beliefs 

BELIEF #1: Collective impact model creates common goals across programs. 

The model was based on the collective impact approach, where individual programs joined a formal 

partnership with common goals and shared measurement systems. It provided a uniform evaluation 

system for all tutoring programs. In addition, the Tutoring Partnership served as a learning community 

that determined and shared what worked to increase student achievement. 

BELIEF #2: Data should drive decisions about capacity building services to scale best 

practices. 

The Tutoring Partnership provided data-driven professional development, tutor training and technical 

assistance to help programs improve. This support was rooted in the Saint Paul Public Schools 

Foundation’s best practice framework and aimed to improve the quality of tutoring interventions. By 

increasing program quality, the Tutoring Partnership sought to increase student outcomes.

BELIEF #3: We need to commit to improving cultural competency and addressing racism. 

The Tutoring Partnership realized that educational disparities seen in Saint Paul and Minneapolis will 

continue to persist if there were not improvements in people’s capacity to work effectively across 

differences. The Tutoring Partnership was committed to improving cultural proficiency both at the 

collaborative and program level. Cultural competency was a top priority for the Tutoring Partnership. It 

was the second of the eight Best Practices for Tutoring Programs. The Tutoring Partnership provided 

professional development and tutor training around cultural competency and facilitated discussions about 

racism in its collaborative meetings. 
13

Building from its core beliefs, the Tutoring Partnership structured three services to build the quality and 

capacity of tutoring programs: professional development workshops for tutoring program staff, technical 

assistance for tutoring programs, and tutor training for tutors working directly with students. Tutoring 

programs in Strategy A opted-in to these services, and the Tutoring Partnership set a target dosage goal 

for each type of support (see Table 2). Tutoring programs in Strategy B and later B3 were required to 

participate in services. Staff were required to participate in professional development workshops, and 

program staff were charged with making sure that tutors attended at least two tutor trainings. The 

Tutoring Partnership set more intensive dosage goals for Strategy B programs, believing that the receipt 

of more services would accelerate their program growth. 

Table 2: Targeted Service Hours for Tutoring Partnership Years 1-3 

Year 1 Year 2 Year 3 

Strategy A 16 programs 18 programs 17 programs

Professional Development

Across All Programs 20 20 40

Per Program Hours 1.25 1.11 2.35

Technical Assistance 

Across All Programs 70 70 100

Per Program Hours 4.38 3.89 5.89

Tutor Training

Across All Programs 50 50 55

Per Program Hours 3.16 2.77 3.25

Strategy B/B3 4 programs 4 programs 4 programs

Professional Development

Across All Programs 30 30 50

Per Program Hours 7.5 7.5 12.5

Technical Assistance 

Across All Programs 60 60 150

Per Program Hours 15 15 37.5

Tutor Training

Across All Programs 50 50 60

Per Program Hours 12.5 12.5 15

Note: SPPSF increased the number of professional development, technical assistance, and tutor 

training hours in Year 3 to further accelerate growth in program quality and student outcomes. 

In addition, programs in Strategy B/B3 received supplemental funds in in each year from the Tutoring 

Partnership. These funds were intended to be used to increase staff time, purchase materials, or 

purchase other supports that would help the program meet its quality improvement goals. Tutoring 

programs created annual budgets and worked with Tutoring Partnership staff to determine where 

additional funds would be beneficial. Funding amounts ranged from $10,000 - $25,000 per program per 

year, depending type of size and scope of changes proposed by the program.

Figure 1 is the theory of change for the Tutoring Partnership. Across the three SIF years, the core beliefs 

of the program informed in the intervention Strategies and services. These services were intended to 

improve the quality of the tutoring programs, and in turn, improve students’ reading skills. 
14

Figure 1: Tutoring Partnership Theory of Change
15

Evaluation Overview

Implementation and Outcomes 

This evaluation had two primary goals. The first was to document and describe the implementation of the 

Tutoring Partnership in the three years of the SIF grant. We examined the selection of programs, up-take 

of services, implementation successes and challenges, and perceptions of effectiveness of the services 

provided by the Tutoring Partnership. The implementation evaluation was conducted in partnership with 

the Center for Applied Research and Educational Improvement (CAREI) at the University of Minnesota. 

CAREI staff collected and analyzed implementation in Years 1-2 of the study, and staff at Child Trends 

finalized data collection and analysis in Year 3. Child Trends received previously collected implementation 

data and reports from CAREI to include in this final report. 

The second evaluation goal was to compare the student and program level outcomes associated with 

participation in the Tutoring Partnership. Program quality was measured each year for programs in each 

Strategy (A, B, and B3). To explore changes to program quality between Strategies overtime, we used 

descriptive statistics to compare frequencies and means for each subscale and the overall score. For 

student level outcomes, we relied on student literacy assessment data collected from local school 

districts. Using a difference-in-difference design, we compared reading growth and proficiency of tutored 

students Kindergarten through Grade 5 who attended programs in Strategy A, B, or B3. The hypothesis 

tested in the evaluation was that that the more intense, individualized support of Strategy B and B3 

would result in a greater increase in program quality and consequently, greater acceleration of student 

achievement in literacy. Due to limited sample size, this evaluation did not examine the student math 

outcomes or the outcomes of tutored students in Grades 6-12. Growth in math and growth for secondary 

students are areas that would benefit from future research. 

Level of Evidence 

In SIF Year 1, the evaluation proposed to build on previous evidence found from the Tutoring Partnership

by using a hierarchical linear modeling approach to compare tutored to non-tutored student matched 

using propensity score matching. Analysis conducted from Year 1 student data found that treatment and 

comparison groups were still not equivalent in terms of race/ethnicity composition (Child Trends, 2015). 

Further, the Tutoring Partnership had plans to expand implementation of the initiative for programs 

serving students in Minneapolis Public Schools (the prior no-treatment comparison group), thus limiting 

the opportunity to compare to non-tutored students in Minneapolis. 

Taking these limitations into account, using the current evaluation design, we will estimate the effect of 

the Tutoring Partnership’s capacity-building efforts on both program and student outcomes. For the 

program-level effects, we use ANOVA to compare the growth on tutoring program quality measures 

administered throughout the three years of the SIF project. For student-level outcomes, we will use a 

difference-in-difference approach. In the difference-in-difference model, we will compare students’ 

growth on assessments in the year they were tutored to their growth the year before and the year after 

they were tutored (where data are available). With these methods, our goal is to present a preliminary 

level of evidence for the Tutoring Partnership model. 
16

Implementation Evaluation Questions

To better understand the implementation of the Tutoring Partnership, the evaluation posed four primary 

implementation questions. Data to answer the first two questions were collected in all three years of the 

evaluation. Additional data were collected in Year 3 to address questions of program and tutor 

perceptions of the Tutoring Partnership. 

1. Years 1-3: What are program and tutor characteristics? 

2. Years 1-3: To what extent and how did the tutoring programs in Strategies A, B, and B3 

participate in the services offered by the Tutoring Partnership (e.g. professional development, 

technical assistance, and tutor training)? 

3. Year 3 only: How do programs perceive the effectiveness of services provided to them through 

the Tutoring Partnership? What is their level of satisfaction with the services? How do programs 

perceive their growth? 

4. Years 1-3: To what extent do programs achieve Best Practices for Tutoring Programs as rated by 

programs on the Best Practices self-assessment?

Outcome Evaluation Questions

The outcome of the Tutoring Partnership was measured both by changes in program quality and by the 

reading achievement of tutored students. Because the Tutoring Partnership was especially concerned 

about the reading achievement of disadvantaged students, the evaluation explored data by those 

subgroups of students. Due to limited sample size, the evaluation question about changes in program 

quality is exploratory. 

Confirmatory Outcome Questions 

1. Years 1-3: To what extent do the three capacity building Strategies (Strategies A, B, and B3) 

affect reading growth for students?

2. Years 1-3: To what extent do the three capacity building Strategies (Strategies A, B, and B3) 

affect reading proficiency for students?

Exploratory Outcome Questions 

1. Years 1-3: Low-income students receiving tutoring: 

a. To what extent does tutoring have a differing effect on reading growth and proficiency 

for low-income students, compared to students not from low-income families?

2. Years 1-3: Racial/ethnic subgroups receiving tutoring: 

a. To what extent does tutoring have a differing effect on reading growth and proficiency 

for specific racial and ethnic groups (i.e., American Indian, Asian American, Latino, 

African American, and Caucasian)? 

3. Years 1-3: Program quality: 

a. To what extent does participation in the Tutoring Partnership improve program quality as 

measured by Program Quality Assessment over time? Does this growth differ by 

Strategy? Does this growth differ based on the number of years receiving the 

intervention? 
17

Implementation Study

Implementation Study Design 

The goal of the implementation study was to understand the selection of programs, up-take of services, 

implementation successes and challenges, and perceptions of effectiveness of the services provided by 

the Tutoring Partnership. The study was also interested in documenting how the Tutoring Partnership 

services changed overtime as this could affect the quality of the tutoring programs and the student 

outcomes. For example, an expansion of the Tutoring Partnership from Saint Paul to Minneapolis in SIF 

Year 3 (2015-2016) presented an opportunity to understand the outcomes of programs in students who 

had received only one year of Tutoring Partnership support. 

Implementation Measures and Analysis

The evaluation used five data sources to answer the questions surrounding the implementation of the 

Tutoring Partnership: program documentation, Tutoring Partnership staff interviews, tutoring program 

staff interviews, a tutoring program staff survey, and a tutor survey. 

Program Documentation (Years 1-3) 

In each year, the Tutoring Partnership maintained detailed records of each tutoring program’s 

participation in capacity building services. These records consisted of a) Professional development 

attendance records, b) Technical assistance records, c) Tutor training attendance records, and d) 

program self-assessment on Best Practices in Tutoring Programs. Prior to merging and analyzing these 

records, Child Trends conducted a series of interviews with the tutoring program staff to gather details 

about the structure of each record. During these interviews, evaluators and program staff also discussed 

any changes to record keeping protocols over the three years of the SIF grant, which assisted the 

evaluators in cleaning records and merging files prior to analysis. 

We reviewed Tutoring Partnership program records and documentation for each of the three years 

funded through SIF. Where data were available, we compared records within and across years. For 

example, we could determine which program staff had attended each professional development 

workshop, and developed a staff involvement metric of how many staff attended more than one 

workshop. Similar procedures were used to gauge attendance at tutor trainings. We coded technical 

assistance records to identify the type of technical assistance (email, phone call, in-person visit), and 

content of the technical assistance (evaluation support, program development, program quality support, 

partnership development, and resource development). Finally, the Tutoring Partnership collected program 

self-assessments of Best Practices in Tutoring Programs. These self-assessments were analyzed using 

descriptive statistics and compared across years. 

Tutoring Partnership Staff Interviews (Year 3 only) 

In the third and final year of SIF for the Tutoring Partnership, evaluators interviewed four key staff from 

the Tutoring Partnership. Staff were interviewed individually, with the interview protocol lasting about 

between 60-120 minutes. Interview constructs primarily focused on staff descriptions of the key 
18

components of the Tutoring Partnership, changes to the program overtime, strengths and limitations of 

the services offered, successes and challenges to implementation, and changes they would like to see 

moving forward (see Appendix A). Interviews were coded for themes in each area. 

Tutoring Program Staff Interviews (Strategy B/B3, Year 3 only) 

To complete the in-depth case studies of the four programs receiving enhanced services through Strategy 

B or B3 of the Tutoring Partnership, Child Trends staff conducted interviews with one key staff member 

at each of the four tutoring programs (ACES, CommonBond Communities, PPL, and Volunteers of 

America). The interview lasted approximately 60 minutes, and consisted of topics such as the program’s 

priorities in the last year, perceptions of the Tutoring Partnership, supervision and training of tutors, 

expansion to Minneapolis-based locations, reasons to take-up or opt-out of Tutoring Partnership services, 

and future needs (Appendix B). We coded interviews for main themes in each area. 

Tutor Survey (Year 3 only) 

Tutors participating in programs responded to an online survey in the spring of the third and final year of 

SIF. The survey asked tutors about their history as tutors, their demographics, attendance at tutor 

training offered by the Tutoring Partnership, perceptions of tutor training, and their strengths and 

challenges as a tutor (see Appendix C). Completion of the survey took approximately 15 minutes, and 

respondents’ names were entered in a raffle for a $25 gift card as an incentive to complete the survey. 

A total of 669 tutors across all 20 tutoring programs in Year 3 responded to the survey, for a response 

rate of 45 percent. The survey analysis included both complete and partially complete responses. We 

analyzed the survey using descriptive statistics including frequency and mean for each survey question. 

The survey contained two new scales developed to measure tutors’ perceptions of their strengths, and 

the challenges they encountered in their role. Finally, we compared frequencies and means for tutors 

based on their program participation in Tutoring Partnership Strategies (A, B, and B3). 

Tutoring Program Staff Survey (Year 3 only) 

One key staff from each of the tutoring programs (ex. a program director or coordinator, but not 

someone in a tutoring role) was asked to participate in a 30-minute survey that asked about the 

program’s structure, perceptions and beliefs about quality improvement, participation in the Tutoring 

Partnership services, and changes made because of participating in the Tutoring Partnership (see 

Appendix D). All 20 key staff in Year 3 completed a survey for a 100 percent response rate. We analyzed 

this survey using descriptive statistics and compared responses of programs in Strategy A, B, and B3. 

Tutoring Partnership’s Best Practices Self-Assessment 

The Best Practices self-assessment was developed by the Tutoring Partnership to measure overall 

program functioning regarding their implementation of the Tutoring Partnership’s Eight Best Practices: (1) 

Organizational Management, (2) Cultural Proficiency, (3) Student Recruitment and Management, (4) 

Tutor Recruitment, (5) Tutor Training, (6) Tutoring Intervention, (7) Engagement, and (8) Evaluation 

(Bixby et al., 2011). The director or manager of a program typically completed this survey at the end of 

school year. In the survey, respondents rated their level of agreement with statements relating how well 

they implemented each Best Practice (1-Strongly Disagree to 5- Strongly Agree). 
19

Implementation Findings

What are program and tutor characteristics? 

Program Characteristics

The implementation study used program documentation and a program staff survey to gather 

information about the characteristics of programs participating in the Tutoring Partnership. Across Years 

1-3, 28 programs participated in the Tutoring Partnership. Table 3 lists the participating tutoring 

programs, the program location, and their years participating in the Tutoring Partnership. 

Table 3: Tutoring programs participating in the Tutoring Partnership 2013-2016

Organization Name Program Name Strategy Year 1 Year 2 Year 3 

Central Community Services Afterschool @ Central A X X X 

Central Community Services Middle School Math Group A X X 

Cherokee Park United Church Building Blocks Tutorial A X X 

East Side Learning Center East Side Learning Center A X X X 

Hmong American Partnership Hmong/

Karen(ni) Youth Pride (HYKP) A X X X 

Interfaith Action of Greater St. Paul Project SPIRIT A X X X 

Keystone Community Services Hmong Youth & Family 

Program A X X X 

Keystone Community Services West 7th Community Kids A X X X 

LDA of Minnesota Learning Connections A X X 

Mid-Continent Oceanographic 

Institute 

Mid-Continent Oceanographic 

Institute A X X X 

Neighborhood House Kids Connect A X X 

Reading Partners Reading Partners A X 

Saint Paul Public Libraries Homework Center A X X X 

Saint Paul Public Libraries Reading Together A X X X 

ServeMN MN Math Corps A X X X 

ServeMN MN Reading Corps A X X X 

Simpson Housing Services Passage Community Tutoring 

Program A X 

The Sanneh Foundation Dreamine Program A X X X 

Twin Cities Housing Development 

Corporation 

STEM Learning Initiative A X X X 

YWCA of St. Paul YW Reads A X X 

Athletes Committed to Educating 

Students 

ACES, Saint Paul B X X X 

CommonBond Communities CommonBond, Saint Paul B X X X 

Project for Pride in Living PPL, Saint Paul B X X X 

Volunteers of America AARP 

Experience Corps

Experience Corps, Saint Paul B X X X 

Athletes Committed to Educating 

Students 

ACES, Minneapolis B3 X 

CommonBond Communities CommonBond, Minneapolis B3 X 

Project for Pride in Living PPL, Minneapolis B3 X 

Volunteers of America AARP 

Experience Corps 

Experience Corps, Minneapolis B3 X 

Source: Tutoring Partnership program documentation records, 2013-2016
20

Based on responses to the program staff survey, over a third (42%) of programs served children in 

community-based locations (see Figure 2) in Year 3. About a quarter (26%) of programs tutored students 

both in-school and community-based locations and less than a quarter (21%) of programs offered

tutoring exclusively in the school. 

Figure 2. Program service location 

21%

42%

26%

11%

In school only

Community-based only

Both in-school and communitybased

Other

Source: Child Trends survey of Tutoring Program staff, spring 2016

As shown in Table 4, tutoring programs employed five full-time employees and six part-time employees 

on average. Additionally, Strategy B/B3 program had a higher average number of tutors per program 

than Strategy A programs. 

Table 4. Average number of staff per position and Strategy

All Strategy A Strategy B/B3

Average N Range Average N Range Average N Range

Number full-time 5 0 - 25 6 1 - 25 6 3 - 8 

Number part-time 6 0 - 16 5 0 - 16 9 1 - 15

Tutors (paid or volunteer) 96 4 - 375 86 4 - 375 114 87 - 160

Source: Child Trends survey of Tutoring Program staff, Spring 2016

In addition, 47% of programs paid less than half of their tutors while about a quarter (26%) of programs 

paid none of their tutors. About three quarters (74%) of programs have less than a quarter of tutors with 

a teaching license. Most programs reported that their tutors had an average caseload of one to five 

students.
21

Tutor Characteristics

Based on results from the Year 3 Tutor Survey, across each Strategy, 75% or more of tutors were White 

or Caucasian while all other race/ethnicities were represented in 15% or less of tutors per Strategy (see 

Table 5). In addition, Strategy A had a larger Asian or Pacific Islander American population than in 

Strategy B and Strategy B3. 

Table 5. Tutor race/ethnicity per strategy

Strategy A Strategy B Strategy B3

Percentage n Percentage n Percentage n 

American Indian or Alaska Native 1% 5 * * * * 

Asian or Pacific Islander American 11% 48 * * * * 

Hispanic or Latino 6% 25 * * * * 

Black or African American 11% 48 15% 10 11% 5 

White or Caucasian 78% 343 75% 50 79% 37

Multiracial 4% 15 * * * * 

Other * * * * * * 

Source: Child Trends Tutor Survey, Spring 2016. *Indicates number of respondents was less than five and not 

reported to protect confidentiality of the respondent. 

 

Figure 3 shows the highest level of education achieved for each Strategy. Over a third (37%) of Strategy 

B3 tutors had a graduate degree, while only 28% of Strategy B tutors and 19% of Strategy A tutors had 

the same degree. Strategy A tutors also had a larger portion of tutors with some college than tutors in 

Strategy B3. Differences between Strategy B and B3 tutors are notable, because these were the same 

programs but serve students in different cities (either Saint Paul or Minneapolis). 

Figure 3. Tutor's highest level of education per Strategy

3% 4%

18%

4%

0%

33%

9%

19%

9%

1% 1%

15%

7%

0%

36%

3%

28%

7%

0% 2%

7% 4%

0%

35%

9%

37%

7%

Some High

School

High School

Diploma or

GED

Some

college, but

no degree

Associate

degree

Associate

degree in

Education or

related field

Bachelor's

Degree

Bachelor's

Degree in

Education or

related field

Graduate

Degree

Graduate

Degree in

Education or

related field

Strategy A Strategy B Strategy B3

Source: Child Trends Tutor Survey, Spring 2016. 

Over a quarter (29%) of tutors in Strategy A were between 18 to 24 years old (see Table 6). Strategy B3 

and Strategy B had more tutors between the ages of 65 to 74 years old (30% and 26%, respectively) 
22

than Strategy A. In addition, there was a higher portion of tutors who were 35 to 44 years old in Strategy 

B3 than in the other two Strategies. 

Table 6. Age of tutors per strategy

Strategy A Strategy B Strategy B3

Percentage N Percentage N Percentage N 

under 17 years old 2% 10 * * * * 

18-24 years old 29% 126 21% 14 13% 6 

25-34 years old 18% 77 14% 9 20% 9 

35-44 years old 5% 24 * * 11% 5 

45-54 years old 5% 20 11% 7 * * 

55-64 years old 18% 80 20% 13 20% 9 

65-74 years old 18% 81 26% 17 30% 14

75 years old or older 5% 20 * * * * 

Source: Child Trends Tutor Survey, Spring 2016 *Indicates number of respondents was less than five 

and not reported to protect confidentiality of the respondent.

Nearly all tutors spoke English at home (see Table 7). Strategy A tutors had larger portions of Spanish 

and Hmong speakers while Strategy B and B3 had very few tutors from non-English speaking homes. 

Table 7. Tutor's language(s) spoken at home 

Strategy A Strategy B Strategy B3

Percentage n Percentage n Percentage n 

English 97% 424 100% 67 100% 46

Spanish 5% 20 0% 0 * * 

Hmong 4% 18 0% 0 * * 

Somali 1% 4 * * * * 

Vietnamese * * 0% 0 0% 0 

Arabic 1% 5 0% 0 0% 0 

Karen 1% 5 0% 0 0% 0 

Other 2% 10 * * 0% 0 

Source: Child Trends Tutor Survey, Spring 2016 *Indicates number of respondents was less than five 

and not reported to protect confidentiality of the respondent.

Summary of Program and Tutor Characteristics

Programs participating in the tutoring partnership varied in terms of the number of staff and number of 

tutors. Programs served students primarily in school-based settings or in community-based settings like a 

community center, library, or in public housing settings. Tutors across all Strategies tended to have high 

levels of education, identified as White, and are English speakers. Demographic similarities and 

differences between the population of tutors and students are notable. The majority of students were 

students of color, and half are English Language Learners (see p. 33). While these racial/ethnic and 

language differences did not necessarily mean that these tutors were not an appropriate match for these 

tutored students. Recent literature suggests that a match between the racial/ethnic group of a teacher 

and student has a positive effect on teacher expectations (Gershenson, Bolt, & Papageorge, 2016), and 
23

student achievement (Villegas and Lucas, 2004). More research is needed about the effects of same-race 

tutors on student academic and social-emotional outcomes. However, given the similarities between 

tutoring and classroom instruction, one could conclude that having a more diverse population of tutors 

would serve students’ academic outcomes. 

To what extent and how did the Tutoring Partners in Strategies A, B, and B3 

participate in professional development, technical assistance, and tutor 

training? 

Professional Development

The Tutoring Partnership offered staff at tutoring programs a variety of different Professional 

Development (PD) Workshops. Initially, the Tutoring Partnership used a mix of internal and external 

trainers; however, they later developed their own PD workshops to replace external trainers who 

Tutoring Partners reported as offering inconsistent workshops that were not related to tutoring. External 

trainers that received higher levels of satisfaction were asked to continue. 

Along with providing general Tutor Partnership meetings such as annual fall kick-offs, luncheons, and end 

of the year celebrations, tutoring programs were offered PD workshops relating to the Program Quality 

Assessment (PQA), reporting and assessment, program evaluations, math instruction, and organizational 

management. Appendix E has a detailed list of workshops offered throughout the years of this evaluation. 

Participation by one key staff member in workshops related to the PQA were required; however, all other 

PD workshops were optional. 

Table 8 shows the total number of PD workshop participants in each Strategy per year. Across all 

Strategy A Programs, an average of 81 participants attended PD workshops, ranging from 59 to 95

participants. Across Strategy B/B3 Programs, an average of 35 participants attended workshops with a

range of 23 to 44 participants per year. It should be noted that because there were more programs in 

Strategy A, total attendance was higher than for Strategy B/B3 programs. However, when comparing the 

average number of attendees per program in each Strategy, program staff in Strategy B/B3 programs 

attended PD workshops more often in each year. 

Table 8. Overall Professional Development Workshop attendance per Strategy and Year

Year 1 Year 2 Year 3

Strategy A

Total attendees 89 59 95

Avg attendees per program

Range of avg attendance

2 

1 - 5 

1 

0 - 2 

1 

0 - 2 

Strategy B/B3

Total attendees 39 23 44

Avg attendees per program

Range of avg attendance

3 

0 - 16

2 

0 - 5 

2 

0 - 5 

Source: Tutoring Partnership PD Workshop Attendance Records 2013-2016

PD workshop attendance fluctuated by topic and year of the evaluation (see Appendix F for more details). 

Staff in Strategy B/B3 programs were required to attend at least two PD workshops, though they could 

select which PD workshops they could attend. Workshop attendance was encouraged, but not required 

for programs in Strategy A. To gauge commitment to professional development, we searched attendance 

records for staff members that attended more than one PD workshop in each year. About half or more of 
24

participants across Strategies and years attended more than one workshop (see Figure 4). In Year 2, 

87% of Strategy B/B3 and 81% of Strategy A participants attended multiple trainings. Eighty-two percent 

of Strategy A staff participated in more than one workshop while only 57 percent of Strategy B/B3 staff 

attended more than one workshop in year three. 

Figure 4. Percentage of tutoring program staff who attended more than one PD workshop 

45%

81% 82%

62%

87%

57%

Year 1 Year 2 Year 3

Strategy A Strategy B/B3

Source: Tutoring Partnership PD Workshop Records, 2013-2016

INSIGHTS FROM INTERVIEWS

Child Trends interviewed Tutoring Partnership staff about the strengths and limitations of the PD workshops. 

Tutoring Partnership staff believed the PD workshops delivered critical content, and created necessary time 

and space for programs to reflect and share ideas together. They also reported that they always tried to 

use best practices for working with adult learners, often having interactive sessions with facilitated 

discussions rather than lecture-style presentations. Tutoring Partnership staff recognized, however, that 

attending a PD workshop without follow-up on-site coaching has limitations. It was frequently the case 

where limited resources made that type of follow-up difficult to achieve with all PD workshop attendees. 

Tutoring Partnership Staff Interviews, Summer 2016

Technical Assistance

While the Tutoring Partnership set goals for the number of hours of technical assistance it would provide, 

most of the technical assistance was provided “on demand” when programs requested support on 

particular topics. If a tutoring program contacted the Tutoring Partnership, they would receive 

individualized support, delivered in-person, by phone, or by email. The technical assistance content 

offered covered seven categories:

1. Resource Development – helping partners secure and/or manage resources like national service, 

funding, materials; 
25

2. Partnership Development – helping partners build relationships with external entities, helping

partners connect with other partners;

3. Evaluation – support around logic models, database management, student/program assessment,

measuring outcomes/progress, surveys, meeting evaluation requirements;

4. Youth Program Quality Inventory (YPQI) 1 – giving programs their data, coaching, improvement

plan support;

5. Program Observation – formal or non-formal observation of training, tutoring, etc.

6. Program Development – support with meeting partnership requirements, organizational

management, program development (e.g., tutor training, student recruitment, staffing, retention,

curriculum review), program quality (outside of YPQI); and

7. Other – computer support, professional coaching.

Table 9 shows the technical assistance (TA) hours and averages used per Strategy and year. Overall, 

programs received an average of 7 to 31 hours of support each year, averaging about 60 minutes per TA 

session. In all three years, Strategy B/B3 programs used more technical assistant hours than Strategy A 

programs. The average length per session was also higher for Strategy B/B3 programs in each year (see 

Appendix G for more details). 

Table 9. Technical assistance hours received by Strategy and Year

Year 1 Year 2 Year 32

Average 

hours per 

program

Average 

minutes per 

session

Average 

hours per 

program

Average 

minutes per 

session

Average 

hours per 

program

Average

minutes per 

session

Strategy A 7.6 46 7.7 57 7.7 65

Strategy B/B3 30.8 64 26.2 70 15.0 62

Source: Tutoring Partnership Technical Assistance Records, 2013-2016

1 The YPQI is a national program quality intervention model created by the Weikart Center for Youth Program 

Quality. The Foundation used elements of the YPQI that were specific to their intervention. 

2 Tutoring Partnership staff had to reprioritize staff time towards closing out the program starting February 2016 in 

Year 3. This may explain the dip in TA hours for Strategy B/B3 programs. While programs may have been interested 

in more TA support, staff were not as available as in previous months and years. 

INSIGHTS FROM INTERVIEWS

When asked about the strengths of Technical Assistance, Tutoring Partnership staff emphasized the 

potential of technical assistance to support behavior change in tutoring program staff. By seeking out 

additional resources, tutoring programs could access the latest in best practices, and build internal capacity 

in areas of evaluation, community partnerships, and program development. With respect to limitations of 

TA, Tutoring Partnership staff acknowledged that the “opt-in” approach to TA was challenging. Programs 

were requesting support only if they believed it was needed. This resulted in situations where programs who 

needed the most support in an area didn’t ask for it. It may also be possible that higher quality programs 

are more skilled at determining areas for improvement, and therefore benefiting more than a lower quality 

program. Finally, programs with low capacity may not have the time to engage in technical assistance and 

implement the change it requires. 

Tutoring Partnership Staff Interviews, Summer 2016
26

Tutor Trainings 

Along with providing subject-specific tutor trainings related to literacy and math, the Tutoring Partnership

also offered a variety of skill-based trainings about tutoring students such as behavior management,

active learning, and intercultural competency. As part of the agreement to become a Tutoring Partner, all

programs – regardless of Strategy – pledged that all tutors would receive both an onboarding orientation 

and additional training on literacy, math, or skill-based tutor training. Some programs provided this

training themselves, while others relied on the Tutoring Partnership provided trainings to support the

ongoing training needs for their tutors.

Rates of participation were calculated by looking at the number of tutors who participated in Tutoring 

Partnership tutor trainings per Strategy and year over the total number of tutors within each Strategy by 

year. As shown in Table 10, participation in Tutoring Partnership tutor training increased in each Strategy 

across the years. Nearly a third of Strategy B tutors in Year 1 attended literacy tutor trainings. The most 

attended literacy trainings included Literacy Skills for Early Readers, Comprehension Lessons for Early 

Readers, and ABC’s of Reading (see Appendix H). Additionally, skill-based tutor trainings had the highest 

rates of participation throughout all Strategies and years. In Year 3, many Strategy B/B3 tutors attended 

multiple skill-based tutor trainings, which accounts for the high participation rate. The Tutoring Basics 

training, which the Tutoring Partnership encouraged for all new tutors, was the most highly attended 

skill-based training. 

Table10. Number of attendees and percent attendance at Tutoring Partnership tutor training 

by Strategy and Year

Year 1 Year 2 Year 3 

Strategy A

(n=731)

Strategy B/B3

(n=179)

Strategy A

(n=926)

Strategy B/B3

(n=183)

Strategy A

(n=850)

Strategy B/B3

(n=183)

Literacy Tutor

Training

4% 

(n=30)

29% 

(n=52)

10% 

(n=93)

23% 

(n=43)

10%

(n=89)

20% 

(n=36) 

Math Tutor 

Training

2%

(n=17)

5% 

(n=9)

6% 

(n=54)

23% 

(n=42)

2% 

(n=20)

33% 

(n=61) 

Skill-Based Tutor 

Training

9% 

(n=47)

22% 

(n=39)

24% 

(n=219)

58% 

(n=106)

33% 

(n=284)

139% 

(n=254) 

Total Attendees 8%

(n=58)

31%

(n=55) 

22%

(n=203) 

68%

(n=124) 

35%

(n=300) 

181%

(n=331)

Source: Tutoring Partnership Tutoring Training Attendance Records, 2013-2016. 

Given that tutors were expected to engage in ongoing training, we compared tutor training attendance 

records to identify which tutors attended more than one training offered through the Tutoring Partnership 

each year. Across all years, Strategy B/B3 tutors were much more likely to attend more than one tutor 

trainings than Strategy A tutors (see Figure 5). 
27 

Figure 5 . Percent attendance at more than one tutor training by Year and Strategy

2%

10%

5%

15%

22%

37%

Year 1 Year 2 Year 3

Strategy A Strategy B/B3

Source: Child Trends Tutor Training Attendance Records, Spring 2016

INSIGHTS FROM INTERVIEWS

When asked about the strengths and limitations of Tutor Training, Tutoring Partnership staff identified the 

research-based and practical strategies as a strength of their tutor training. The trainings were geared 

towards skills and tactics that tutors could implement and practice by role playing. A limitation mentioned 

by Tutoring Partnership staff was that tutors worked with a variety of different students across multiple sites 

and grade levels, which required tutors to know certain approaches or have different skills to address the 

needs of their students. The Tutoring Partnership could not cover all of this in their trainings. Tutoring 

Partnership staff relied on tutoring programs to train their tutors about topics specific to their program. 

Programs were also responsible for tracking attendance for program-offered training; program staff often 

did not have the systems or staff capacity to track this regularly, nor did they report that attendance to the 

Tutoring Partnership. 

Tutoring Partnership Staff Interviews, Summer 2016

Summary of Tutoring Program Participation in Services

Program staff and tutors in Strategy B/B3 programs participated in services offered through the Tutoring 

Partnership at higher rates than staff and tutors in Strategy A. This was true across PD workshops, 

technical assistance, and tutor training. 

Program staff across all tutoring programs participated in a variety of PD workshops in each SIF year. 

Staff from Strategy B/B3 programs had higher average rates of attendance than staff from Strategy A 

programs. Strategy B/B3 programs also higher rates of staff that attended more than one workshop in 

Year 1 and 2. As mentioned in the introduction, the Tutoring Partnership announced the closure of their 

program in the winter of the final SIF year. Some staff at the Tutoring Partnership who had been 

supporting PD workshops were laid off, and remaining staff reported that capacity to offer all the planned 

workshops for the spring was diminished. (Anonymous, personal communication, July 2016). This decline 

in participation in Year 3 may be an artifact of Strategy A program staff attending workshops in the fall, 
28

while Strategy B/B3 program staff may have planned on attending more in the spring, which were not 

offered. 

Strategy B/B3 programs took up more hours of technical assistance than Strategy A program in each 

year. As was discussed in the Theory of Change technical assistance was opt-in for Strategy A programs, 

and a requirement for Strategy B/B3 programs. While we do not have measures of quality for the 

technical assistance provided, the model of requiring at least some minimum level (15 hours) of technical 

assistance was an effective method to increasing the amount of assistance provided to Strategy B 

programs. The amount of technical assistance sought out by Strategy A programs remained consistent 

(around 7.5 hours) each year. Strategy B/B3 programs had the highest rates of technical assistance 

dosage in Year 1 (about 31 hours), fewer hours in Year 2 (26 hours), and the fewest hours in Year 3 (15 

hours). This decline in technical assistance provision could be explained by the fact that programs may 

need more intensive support during the first year of a quality improvement effort. It is also notable that 

as the Tutoring Partnership announced its closure in Year 3, it is possible that the intensity of technical 

assistance was not available to Strategy B/B3 programs to the same extent in the final year. Strategy A 

programs did not exhibit that same trend, however. These programs were not given incentives nor 

required to participate in technical assistance, so further study is needed to better understand opt-in

patterns in technical assistance. 

Finally, tutors in Strategy B/B3 programs had higher rates of attendance at multiple trainings when 

comparing attendance of tutors in Strategy A. As noted above, attendance rates at multiple trainings was 

low, especially for Strategy A tutors. Tutoring programs offered trainings specific to their tutors, so we 

cannot assume that tutors were not receiving ongoing training. However, given that limited staff capacity 

to deliver tutor training was a reason for offering trainings through the Tutoring Partnership, it may be 

the case that few tutors received training beyond an orientation and basic training, 

How do programs perceive the effectiveness of services provided to them 

through the Tutoring Partnership? What is their level of satisfaction with the 

services? How do programs perceive their growth? 

Child Trends surveyed 20 directors and/or managers at each tutoring program to understand their 

perceptions of Tutoring Partnership primary services (professional development, technical assistance, and 

tutor training), and other secondary services like evaluation support, communication with school districts, 

and connections with other organizations. When asked to rank the Tutoring Partnership services in order 

of their effectiveness in making changes to their program, respondents most frequently ranked tutor 

trainings as being the most effective (see Table 11). A quarter of Strategy B/B3 programs also ranked 

technical assistance, evaluation support, and Tutoring Partnership meetings as the number one service 

offered.
29

Table 11. Ranking of Tutoring Partnership services by strategy

All Strategies Strategy A Strategy B/B3

Percent #1 rank Percent #1 rank Percent #1 rank

Tutor trainings 32% 33% 25%

Professional development workshops 16% 20% 0%

Technical assistance 16% 13% 25%

Evaluation support 16% 13% 25%

Volunteer recruitment 5% 7% 0%

Communication with school districts 5% 7% 0%

Tutoring Partnership meetings 5% 0% 25%

Continuous Quality Improvement 5% 6% 0%

Connections with organizations 0% 0% 0%

Source: Child Trends Program Staff Survey, Spring 2016

 

Moreover, almost all respondents (84%) would strongly recommend tutor trainings to other organizations 

followed by continuous quality improvement (68%), connections with organizations (61%), professional 

development (58%), and evaluation support (53%) services (see Figure 6). Overall, over half of 

respondents would strongly recommend or somewhat recommend all Tutoring Partnership services 

(ranging from 61% to 100%).

Figure 6. Tutoring Partners willing to recommend services to other tutoring organizations

84%

68% 61% 58% 53%

42%

33% 32% 26%

11%

26% 33% 42%

16% 42%

28%

68%

37%

16%

5%

22% 16%

5% 5% 6%

16% 11% 17% 21%

Tutor trainings Continuous

Quality

Improvement

Connections

with

organizations

Professional

development

Evaluation

support

Technical

assistance

Communication

with school

districts

Tutoring

Partnership

meetings

Volunteer

recruitment

Strongly recommend Somewhat recommend Somewhat do not recommend Strongly do not recommend N/A

Figure 7 shows how survey respondents rated statements related to the Tutoring Partnership’s 

organizational mission. Most respondents (84%) viewed ongoing staff training and coaching, as well as 

orientation, training, and development of tutors as being extremely important. Over half of respondents 

(ranging from 53% to 68%) also rated using data for quality improvement and establishing processes to 

connect with families, the community, schools, and teachers as being extremely important. Less than 
30

INSIGHTS FROM INTERVIEWS

Child Trends also interviewed one staff member from each of the four Strategy B/B3 programs 

receiving enhanced Tutoring Partnership services to gain a deeper understanding of their 

experiences. When asked to describe the reasons why they would recommend the Tutoring 

Partnership to other tutoring programs, staff at these programs listed services that they found 

beneficial including tutor trainings, PD workshops around data and quality, networking, and 

technical assistance. Program staff said that the trainings and workshops were of high quality and 

free. They also reported that technical assistance helped them think about how to use data to 

improve their overall program quality...

“Technical support made us thoughtful about the way we are looking at our data. It’s 

influenced how we think about and make changes to our program.”

In addition, one staff member from all programs were surveyed in the Tutoring Program Staff 

Survey. Over half of program staff survey respondents (58%) reported that their program 

continuously communicates with other tutoring organizations. When asked to describe this 

communication, respondents primarily listed the other organizations they work or collaborate with. 

Respondents made general comments noting that they collaborate.

Strategy B/B3 Program Staff Interviews, Summer 2016

half of respondents (42%) perceived adopting standards and aligning assessment tools as being 

extremely important to their overall functioning. 

Figure 7. Tutoring Partners rating of Tutoring Partnership’s organizational mission

42%

53%

63%

68%

84%

84%

37%

37%

32%

21%

16%

11%

11%

5%

11%

5%

11%

5%

5%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Adopting standards and aligning assessment tools

Establishing processes for connecting with schools

and teachers

Establishing processes for connecting with families

and the community

Using data for quality improvement

Supporting the orientation, training, and

development of tutors

Providing and participating in ongoing training and

coaching to staff members

Extremely important Moderately important Slightly important Not at all important
31

As shown in Figure 8, over half (65%) of survey respondents strongly agreed that their program is of 

higher quality because they participated in the Tutoring Partnership. Additionally, 60% of respondents 

strongly agreed that they made changes to their program because of joining the Tutoring Partnership. 

While all respondents generally agree that the Tutoring Partnership’s primary purpose is to help support 

programs achieve their quality goals, only about a third of respondents strongly agreed with that 

statement. Similarly, all respondents agreed that their experiences with the Tutoring Partnership have

been what they expected, however, only 35% of respondents strongly agreed with that statement. 

Figure 8. Tutoring Partners perceptions of the Tutoring Partnership

35%

35%

60%

65%

50%

60%

30%

30%

0% 20% 40% 60% 80% 100%

My experience with the Tutoring Partnership has been

what I expected.

The primary purpose of the Tutoring Partnership is to

support programs by helping them achieve program

quality goals.

We have made changes to our program as a result of

joining the Tutoring Partnership.

I believe my program is of higher quality because we

participated in the Tutoring Partnership.

Strongly Agree Agree Disagree/Strongly disagree

Child Trends asked respondents what changes they made in their program that provided the biggest 

benefit for the children served. Over half (53%) of respondents discussed how changing their tutoring 

strategies has impacted children. Thirty percent of these respondents reported that they adopted more 

standardized curriculum and lesson plans. Another 18% talked about their implementation of reflection 

strategies when working with students. Additionally, nearly a quarter (24%) of respondents stated that 

expanding their services and organizational capacity was the most beneficial change to the children. 

Survey respondents also reported aspects of their program that were the most difficult to change. 

Twenty-one percent stated that changing tutoring content was challenging for their program. Other 

difficult changes mentioned included evaluation, assessment and observation (16%), connecting with 

other schools (11%), managing tutor quality and retention (11%), connecting with families (11%), and 

training attendance of tutors (11%). 

Additional comments left by survey respondents were generally about how “great” and “valuable” the 

Tutoring Partnership has been to their program. All Strategy B/B3 programs generally described the 

Tutoring Partnership as being instrumental to their program’s success: 

“We have a better sense of what quality means from the Tutoring Partnership.”

“The Tutoring Partnership has helped our program so much and I am confident that we will be 

able to continue to grow.” 
32

INSIGHTS FROM INTERVIEWS

Child Trends asked staff from the four Strategy B/B3 programs about their most challenging 

change. Half of these programs discussed topics related to intervention quality such as creating a 

standardized curriculum and planning lessons to provide consistent and high quality services to 

students. Another program discussed the difficulty in initially convincing their program to use data 

and why it would be valuable. 

Strategy B/B3 program staff were also asked what changes or new strategies may be needed to 

support the continuation and improvement of their programs. Half of programs mentioned the 

need to line up high quality trainings for their staff and tutors without overburdening their current 

staff. In addition, one program staff member mentioned that the Tutoring Partnership helped them 

evaluate and hone the quality of their program. This respondent said that they will work to stay 

informed to make sure they keep elevating their best practices. When asked to describe the 

resources that are necessary for their programs, almost all program staff mentioned funding needs 

to attract and keep high quality staff. They also reported their need for high quality tutor trainings 

and professional development workshops. Lastly, one program’s director stated that they need to 

make time for continuous quality improvement efforts. 

Strategy B/B3 Program Staff Interviews, Summer 2016

To what extent does the program achieve Best Practices for Tutoring Programs 

as rated by programs on the Best Practices self-assessment? 

The director or manager of a program typically completed the Best Practices self-assessment on behalf of 

his/her program at the end of school year. In the survey, respondents rated their level of agreement with 

statements relating how well they implemented each Best Practice (1-Strongly Disagree to 5- Strongly 

Agree). 

As shown in Table 12, respondents consistently rated themselves as meeting each item of the Best 

Practices. The overall average per Strategy and across years ranged from 3.89 to 4.17. Cultural 

Proficiency items were the lowest rated with a total average of 3.70. Items in this Best Practice included 

prioritizing cultural proficiency, hiring staff who are culturally competent, and providing initial cultural 

competency training and ongoing training to support cultural proficiency (see Appendix I). There were no 

significant differences between Strategy A and B/B3 programs on Best Practices ratings. 

Table 12. Best Practice self-assessment scores by strategy and year 

Year 1 Year 2 Year 3 Average 

Score Across 

All 

Strategies 

and Years 

Strategy A Strategy 

B 

Strategy A Strategy 

B 

Strategy A Strategy 

B/B3

Organizational 

Management 4.21 4.17 4.08 3.88 4.22 4.38 4.16

Cultural Proficiency 3.65 3.75 3.50 3.75 3.97 3.56 3.70

Student Recruitment 

and Management 4.16 4.30 3.87 3.90 4.20 3.95 4.06

Tutor Recruitment 3.91 4.33 3.87 4.50 4.07 4.42 4.18

Tutor Training 3.90 4.38 4.02 4.50 3.38 4.75 4.16
33

 Year 1 Year 2 Year 3 Average 

Score Across 

All 

Strategies 

and Years 

Strategy A Strategy 

B 

Strategy A Strategy 

B 

Strategy A Strategy 

B/B3

Tutor Intervention 4.01 4.00 3.89 4.05 4.19 4.35 4.08

Engagement 4.13 3.75 3.87 3.81 4.15 3.95 3.94

Evaluation 4.08 3.94 4.02 4.13 4.08 4.00 4.04

Overall Average 4.01 4.08 3.89 4.07 4.03 4.17 4.04

Source: Tutoring Program self-assessments on Best Practices for Tutoring Programs 2013-2016. 

Summary of Best Practices Achievement

Programs in all Strategies tended to rank themselves as implementing the Best Practices for Tutoring 

Programs in all three years. There is a benefit to asking programs to self-report this information as it 

provides a useful window into the way programs perceive their strengths and weaknesses. However, like 

many self-report tools, the Best Practices assessment may be prone to social desirability-bias where the 

respondent may rate themselves in a favorable way. Another limitation of this assessment is that because 

programs rated themselves highly in Year 1, the scale may have a ceiling effect where programs cannot 

indicate their improvement. Finally, it is possible that the scale accurately reflects the programs use of 

best practices, but as program staff knowledge about best practices increased overtime, they may have 

rated themselves more critically in later years. 
34

INSIGHTS FROM INTERVIEWS

In interviews with Tutoring Partnership staff, Child Trends asked about the successes and challenges of the 

Tutoring Partnership during the SIF project. The growth of the Tutoring Partnership was noted as a major 

success by staff. They discussed how the Tutoring Partnership came about as a way to connect the 

community and systems to work at the child-level. Staff were proud that the Tutoring Partnership 

embraced the idea that collectively they can work together to impact the community. 

Staff also discussed their work with the Strategy B/B3 programs as a success. They noted that these 

programs were all motivated to improve in some way, which allowed the Tutoring Partnership to easily work 

with them. Respondents viewed these programs as making major progress and honing their 

implementation as high-quality tutoring programs. 

Staff also discussed the value of getting programs to buy -in to the Academic Skill Building Tool, which is a 

sub-scale on the Program Quality Assessment developed by the Tutoring Partnership. One staff stated that 

this helped move the conversation around skill building (e.g. what are you offering to students and what do 

you do to increase the skills of your students). By having the common language in the tool used by all 

programs, this made it easier for the Tutoring Partnership to help programs. 

Tutoring Partnership staff talked about program-level and internal challenges. Staff recognized that 

programs are experiencing constant challenges such as staff turnover, low tutoring dosage, and lack of staff 

capacity and funding. They understood that these issues can be very challenging when trying to invoke 

change. Keeping in mind these program-level challenges, Tutoring Partnership staff aimed to keep 

assignments and suggestions very manageable, which a respondent stated was never easy. Other 

challenges to the Tutoring Partnership were internal to their organization. These included challenges about 

exactly when to expand their own staff, and sometimes competing with tutoring programs for funds from 

local funders. 

Finally, Child Trends also asked respondents Tutoring Partnership staff to reflect on what they perceived to 

be the most important service that the Tutoring Partnership provided to improve program quality. Half of 

Most Tutoring Partnership staff stated that technical assistance is the most important service for long-term 

program quality improvements. They reported that technical assistance is most impactful when it is provided 

by someone who has a good relationship with the program and understands their needs. 

Tutoring Partnership Staff Interviews, Summer 2016

Discussion of Implementation Findings

The Tutoring Partnership was an initiative that sought to improve the quality of tutoring programs in 

Saint Paul and Minneapolis. With SIF funding from 2013-2016, the Tutoring Partnership provided 

capacity-building services 28 tutoring organizations using two strategic approaches. The majority of 

programs (24 in Strategy A) received the typical Tutoring Partnership services – these pre-dated SIF 

funding. An additional four programs (Strategy B, Years 1-3 and Strategy B3, Year 3 only) received more 

intensive services through the Tutoring Partnership. The implementation study focused on describing the 
35

characteristics of participating programs and tutors, participation in the initiative’s services, perceptions of 

participants and staff, and the use of best practices in tutoring programs. Key findings include: 

• Programs served students in schools and community-based settings. Programs varied widely in 

program size, number of staff, and number of tutors. 3

• Tutors in all Strategies had high levels of education, were mostly White, and predominately English 

speakers. 

• Strategy B/B3 programs had higher rates of participation in key services such as professional 

development workshops, technical assistance, and tutor training. Requiring participation in these 

services did have an association with higher levels of participation when comparing rates to Strategy 

A, who were not required to participate. 

• Overall, programs felt the support they received from the Tutoring Partnership was beneficial. 

Programs found the tutor trainings to be most helpful; however, participation in tutor training was 

relatively low. 

• Programs across all Strategies rated themselves highly as using Best Practices for Tutoring Programs. 

The Tutoring Partnership achieved several implementation successes, including balancing the needs of 

individual programs while providing services that are useful across the network of programs. Based on 

the overall satisfaction of tutoring programs, this unique aspect of the Tutoring Partnership was 

implemented successfully. They also successfully engaged the four Strategy B/B3 programs in a more 

robust set of capacity-building services by requiring some levels of participation. Tutoring Partnership 

staff reported the level of engagement from these four programs to be high, and they perceived strong 

growth in the quality of these programs. 

The partnership faced several implementation challenges. Participation in the services required programs 

to opt-in to what they saw was needed. Even when participation was required (as with Strategy B/B3 

programs), engagement was not easy to track or enforce. Even if staff were attending workshops, 

seeking technical assistance, and tutors attended trainings, it is not clear how often this led to changes in 

on-the-ground tutoring practices. 

Quality improvement efforts often use a tool to assess a program’s readiness and willingness to 

participate in the quality improvement process. While this was not used in the Tutoring Partnership, 

programs like it may benefit from assessing organizational capacity and staff readiness and ability to 

change. Readiness and ability to change can be measured before the quality improvement initiative 

begins and throughout to ensure that the services continue to be responsive. 

 

3 Based on previous implementation reports on the Tutoring Partnership, programs also varied widely in dosage of 

tutored delivered to students (Michlin, Schultz, and Harty, 2015). 
36

Outcomes Study

Study Design

The outcomes study of the Tutoring Partnership explored the associations of participation in the program 

with both tutoring program-level outcomes and student-level outcomes. The study examined 

confirmatory questions about the growth and proficiency in reading skills for students tutored in Strategy 

A, B, or B3. Exploratory questions included the growth and proficiency in reading skills for subgroups of 

students (i.e. low-income vs. higher income; racial/ethnic subgroups), and the change in tutoring 

program quality overtime. 

Student Reading Skills 

Since its inception, the Tutoring Partnership has had a primary focus on improving literacy skills. Later, 

math and social-emotional skills were also included in professional development workshops and tutor 

training offered through the Tutoring Partnership. However, due to the limited number of programs that 

explicitly focused on math or social-emotional skills, this evaluation is limited to the growth and 

proficiency in reading. The study uses a pooled sample of students tutored over the Years 1-3 of the SIF 

grant period as the primary sample. To estimate the effect of tutoring on student reading growth, we 

used a difference-in-difference approach to compare the growth on a reading assessment in the year a 

student was tutored to the growth in non-tutored years. Regression-adjusted rates of reading proficiency 

were compared in each year for tutored and comparison students to determine the association of tutoring 

with being a proficient reader. 

Tutoring Program Quality 

To measure the quality of tutoring programs, a sample of programs were observed by Tutoring 

Partnership staff using the Program Quality Assessment (PQA) in the fall and spring of each year. Given 

the more intensive uptake in services, we hypothesize that programs in Strategy B/B3 will have greater 

gains in PQA scores relative to the programs receiving less intensive service in Strategy A. Due to limited 

sample size, evaluation questions about tutoring program quality are considered exploratory in this study. 

Level of Evidence 

In the original Subgrantee Evaluation Plan (SEP), Child Trends proposed a quasi-experimental propensity 

score matching approach to match tutored students to non-tutored students within each Strategy 4, and 

comparing effect sizes across Strategies. The goal of the original SEP was to establish a moderate level of 

evidence for the Tutoring Partnership by comparing the growth and proficiency in reading by using the 

matched comparison group. This approach would have allowed for stronger internal and external validity 

than previous research on the Tutoring Partnership. Child Trends used the propensity score matching 

approach in Year 1, and delivered the Year 1 Interim Impact Report to CNCS. The report noted that the 

study encountered several challenges with this methodology. The primary challenge was that the while 

 

4 Strategy A (Tutoring Partnership as usual in Saint Paul), Strategy B (enhanced Tutoring Partnership in Saint Paul) 

and then-called Strategy C (no Tutoring Partnership support in Minneapolis). 
37

students could be well-matched within Strategy, students across Strategies were not well-matched. This 

matching challenge between Strategies reflects the population differences between Saint Paul and 

Minneapolis. Programs in Strategy A and B were in Saint Paul, and have similar populations, while 

Strategy B3 programs were located in Minneapolis with different populations. The matching challenges 

limited our understanding about the growth and proficiency across the different treatment groups. A 

secondary challenge was that the non-treated group of programs in Minneapolis began participating in 

the Tutoring Partnership in Year 3, thus becoming a treated group for the last year of the project. 

In summer 2016, Child Trends submitted a revised SEP to address the challenges of the previous 

approach, and is now seeking a preliminary level of evidence. The new methodology described here will 

provide good internal validity about the associations between participation in the Tutoring Partnership 

and outcomes observed. However, due to the uniqueness of the student population represented in this 

study, our findings will have limited external validity to urban school districts with similar demographics. 

Sample Description

From Years 1-3, tutoring programs participating in the Tutoring Partnership provided tutoring services to 

21,168 students preK – 12th grade. Given the majority of students were tutored in elementary school, the 

focus for this study was on reading outcomes for students Kindergarten through 5th grade. Programs in 

Strategy A served most of the students in the sample, with just 11% of students served in Strategy B 

programs, and an additional 8% served in Strategy B3 programs. Tutored students came from diverse 

language and racial/ethnic backgrounds. Nearly all tutored students were from low-income families, and 

were eligible for free or reduced priced lunch. Table 13 provides the number and percent of students 

tutored in each Strategy for key demographic factors.

Table 13. Demographics of tutored students by Strategy (K through Grade 5)

 All 

Tutored 

Students 

Year One Year Two Year Three 

Strategy 

A 

Strategy 

B 

Strategy 

B3

Strategy 

A 

Strategy 

B 

Strategy 

B3

Strategy 

A 

Strategy 

B 

Strategy 

B3

All Students 

(N)

5907 1733 171 222 1418 204 210 1428 251 270

English 

Language 

Learners 

42%

(n=2502) 

42%

(n=727) 

46% 

(n=79)

35%

(n=78) 

44%

(n=620) 

39%

(n=79) 

43%

(n=91)

46%

(n=663) 

31% 

(n=79)

32%

(n=86)

Free/Reduce

d Lunch 

85%

(n=5025) 

84%

(n=1448) 

88%

(n=151) 

75% 

(n=168)

86%

(n=1220) 

90%

(n=183) 

91%

(n=192) 

85%

(n=1212) 

82%

(n=206) 

91% 

(n=245)

Special 

Education

7% 

(n=417) 

5%

(n=88) 

11%

(n=18) 

9%

(n=20) 

6%

(n=81) 

12%

(n=24) 

14%

(n=29)

6%

(n=81) 

18%

(n=44) 

12%

(n=32)

Racial/Ethnic Groups

White 12%

(n=693) 

12%

(n=206) 

8%

(n=13) 

11%

(n=25) 

13%

(n=184) 

14% 

(n=29)

8%

(n=17)

11%

(n=161) 

11%

(n=30)

10%

(n=28) 

Black 43%

(n=2521) 

38%

(n=651) 

49%

(n=83) 

70%

(n=155) 

42%

(n=594) 

38%

(n=78)

58%

(n=122) 

41%

(n=545)

50%

(n=133)

59%

(n=160)

Latino 14%

(n=808) 

15%

(n=263) 

7%

(n=12)

14%

(n=31) 

13%

(n=180) 

10%

(n=21) 

26%

(n=54)

11%

(n=163) 

9%

(n=23)

23%

(n=61)

Asian 29%

(n=1708) 

33%

(n=580) 

36% 

(n=61)

4%

(n=8)

29%

(n=416) 

35%

(n=72) 

5%

(n=10)

34%

(n=484) 

25%

(n=66)

4%

(n=11)
38

 All 

Tutored 

Students 

Year One Year Two Year Three 

Strategy 

A 

Strategy 

B 

Strategy 

B3

Strategy 

A 

Strategy 

B 

Strategy 

B3

Strategy 

A 

Strategy 

B 

Strategy 

B3

Native 

American 

3%

(n=153)

2% 

(n=33) 

1%

(n=2)

1%

(n=3)

3% 

(n=44) 

2%

(n=4)

3%

(n=7)

2%

(n=35) 

6%

(n=15)

4%

(n=10)

Source: Tutoring program rosters and school district demographic records, 2013-2016

Across Strategies, tutoring took place primarily in grades Kindergarten through 3rd grade; however, about 

11% of tutored students were tutored in 4th or 5th grade. The grade in which tutoring occurred varied by 

Strategy, where Strategy B programs had higher levels of tutoring at 4th and 5th grade than the other 

Strategies. Table 14 presents the number and percent of tutored student per Strategy Kindergarten 

through 5th grade. Overall samples for Strategy B and B3 programs are relatively small, and may limit the 

ability to analyze subgroups of students within these Strategies. 

Table 14. Percentage and number of students tutored by strategy

 All 

Tutored

Students 

Year One Year Two Year Three

Strategy 

A 

Strategy 

B 

Strategy 

B3

Strategy 

A 

Strategy 

B 

Strategy 

B3

Strategy 

A 

Strategy 

B 

Strategy 

B3

Kindergarten 23% 

(n=1362)

27% 

(n=457)

9%

(n=16)

30%

(n=67)

20%

(n=282)

14%

(n=29)

25%

(n=52)

26% 

(n=367)

10% 

(n=25)

18%

(n=49)

Grade 1 23% 

(n=1381)

28% 

(n=488)

23%

(n=40)

26%

(n=57)

26%

(n=374)

14%

(n=28)

14%

(n=30)

21% 

(n=301)

12% 

(n=30)

12%

(n=33)

Grade 2 21% 

(n=1225)

23% 

(n=401)

26% 

(n=44)

15%

(n=33)

23% 

(n=321)

24%

(n=49)

19%

(n=40)

18% 

(n=262)

13% 

(n=33)

16%

(n=42)

Grade 3 20% 

(n=1174)

22% 

(n=377)

8% 

(n=13)

19%

(n=42)

26%

(n=368)

8%

(n=16)

13%

(n=28)

21% 

(n=295)

6% 

(n=14)

8%

(n=21)

Grade 4 8% 

(n=490)

<1%

(n=9)

19%

(n=33)

7%

(n=16)

4%

(n=52)

36% 

(n=73)

23%

(n=49)

9% 

(n=126)

24% 

(n=61)

26%

(n=71)

Grade 5 5% 

(n=294)

<1%

(n=2)

15% 

(n=25)

3%

(n=7)

1%

(n=21)

4%

(n=9)

5%

(n=11)

5% 

(n=77)

35% 

(n=88)

20%

(n=54)

Total 5908 1734 171 222 1418 204 210 1428 251 270

Source: Tutoring program rosters and school district demographic records, 2013-2016

Outcome Measures

Program Quality Measures 

The Program Quality Assessment (PQA) was administered to programs across the years of the evaluation

by certified observers. Depending on the ages of the students served at the tutoring program, the Youth 

Program Quality Assessment (YPAQ) or the School Age Program Quality Assessment (SAPQA) was 

administered. Both PQAs are composed of five domains: (1) Safe Environment, (2) Supportive 

Environment, (3) Interaction, (4) Engagement, and (5) Academic Skill Building. Each domain contains a 

set of domain-specific items per scale. Observers rated each item on a scale of one to five: 1-never 

happens for no children, 3-sometimes happens for a few children, and 5-always happens for all children. 
39

The tool was created and published by the David P. Weikart Center for Youth Program Quality, a joint 

venture of the Forum for Youth Investment and the HighScope Educational Research Foundation. The 

Weikart Center has conducted a validation study where the PQA was found to be a reliable measure of 

key quality domains in multiple types of youth programs (Smith and Hohmann, 2005). 

The Tutoring Partnership administered a new version of the PQA assessment in Years 2 and 3. In the 

new PQA, the old Academic Climate domain was replaced with Academic Skill Building. The standards for 

the new Academic Skill Building domain were higher than the standards for the Academic Climate, and 

harder for programs to achieve. 

Student Level Measures 

In each SIF year, tutoring programs submitted rosters and dosage records for each participating student 

served during the program year. Each program was asked to provide as much information as possible for 

every tutored student, including student name, ID, birthdate, gender, school name, name of tutoring 

program, and the total hours of literacy tutoring and math tutoring each student received during the 

school year. Collated student rosters were submitted by Child Trends to the Research, Evaluation and 

Assessment (REA) department of the Saint Paul and Minneapolis school districts participating in the 

evaluation. 5 Through district data sharing agreements, Child Trends was granted access to student test 

score data and demographic information. Saint Paul Public Schools (SPPS) and Minneapolis Public Schools 

(MPS) shared demographic data on all tutored students, including their grade, ethnicity/race, gender, 

eligibility for free and reduced price lunch, English language learner status, special education status, 

school number, and attendance rates. Five different assessment tools (administered by Saint Paul Public 

Schools and/or Minneapolis Public Schools) were available to examine the effect of tutoring on student 

outcomes. These assessments include the Mondo Bookshop Assessment, Minneapolis Kindergarten 

Literacy Assessment, Measures of Academic Progress (MAP), and the Minnesota Comprehensive

Assessments MCA-III (see Table 15 for details of analyses by outcome and grades).

• Mondo Bookshop Assessment (Saint Paul Public Schools only) is administered to kindergarteners 

through fifth graders in Saint Paul Public Schools. The Mondo measures aspects of reading such as 

Text Level (overall reading ability), Print Concepts (e.g., understanding how books work), LetterSound Correspondence (i.e., ability to read nonsense words), Word Knowledge (i.e., ability to read a 

list of sight words), and Oral Language (measures of receptive vocabulary). Classroom teachers 

administer the Mondo to students twice annually, in either the fall or winter as well as in the spring.6 

• Literacy Measures (Minneapolis Public Schools only) are administered in Minneapolis Public Schools to 

kindergarteners in the fall and spring. Kindergartners are assessed on Language, Phonemic 

Awareness, Alphabetic Principles, and Total Literacy. In the current analysis, gains from fall to spring 

on Total Literacy scores are compared between groups. 

 

5 Some tutored students attended charter schools rather than a traditional public school. Due to limited resources, 

this evaluation could obtain student-level assessment data only from public school districts. 

6 The Mondo subtest are arranged by stages of reading development, not by grade level. In Saint Paul Public 

Schools, once a student reaches the ceiling for the subtest, the teacher is not required to assess the student on that 

subtest again for that school year. 
40

• The MAP is administered in the fall and spring to certain students in Minneapolis Public Schools. It 

assesses math skills in addition to reading skills. In alignment with the Tutoring Partnership’s 

targeted goals, the current analyses examine MAP Reading scores for first through fifth grade 

Minneapolis students. 

• The MCA-III assesses reading skills, is aligned with the common core, and is administered in the 

spring by both Saint Paul Public Schools and Minneapolis Public Schools. Scores on the MCA-III are 

used to determine proficiency in reading, and are also used to compare students to one another on 

reading skills. Proficiency is a dichotomous (i.e., yes/no) variable in the analysis that indicates 

whether a student has met grade-specific state standards. MCA-III analyses were conducted for third 

through fifth graders.

Table 15: Analyses conducted by outcome and grades (2013-2016)

Saint Paul Strategies A & B

Grade(s)

Minneapolis Strategy B3

Grades(s) 

 MPS Total Literacy fall-to-spring gain Not administered K 

 Mondo Text Level fall-to-spring gain K-5 Not administered

 MAP fall-to-spring gain Not administered 1-5 

 MCA spring scale score 3-5 3-5 

 MCA proficiency 3-5 3-5 

Note: All analyses compare all tutored students and students in different subgroups when sample sizes allow (students eligible for 

free/reduced-price lunch; White, Black, Latino, Asian, and Native American students). 

Outcome Analysis

Identifying Tutored Students 

To identify students who were tutored in each Strategy, tutoring programs supplied student rosters with 

student name, ID number, grade level, school name, school district, and tutoring dosage. Students K-5th

grade were combined across years and shared with Minneapolis Public Schools and Saint Paul Public 

Schools. Research staff at the school districts matched student records using a student records database 

to find the demographic and assessment scores for each student in Years 1-3. In Saint Paul, records 

unmatched by algorithm were looked up by hand. 7 Each district was asked to also provide demographic 

and assessment information from the year prior to the SIF grant (2012-2013) to establish a growth trend 

for students tutored in Year 1. Students were sometimes tutored in only one grade, and sometimes more 

than one grade. Demographic and assessment information was provided in the tutored and non-tutored 

years of the SIF project. 

 

 

7 Overall, 67% of students included on Saint Paul tutoring program rosters were matched with district data; 89% of 

students included on Minneapolis tutoring program rosters were matched with district data. That match rate in Saint 

Paul includes 2,316 students that were found using a manual look up process. Based on student rosters, we believe 

most unmatched students attended charter schools outside of Minneapolis Public Schools and Saint Paul Public 

Schools. 
41

Data Cleaning and Missing Data 

We took several steps to clean and examine data before analyses. First, all variables were checked for 

correct variable status (ordinal, numeric, string), and we created dummy variables from categorical 

demographic variables to use in regression models. Second, we ran frequencies on all variables to 

determine the extent of missing data. Cases without at least one demographic characteristic and one 

assessment score were dropped from the analysis sample (n=3417). Third, in instances where 

assessment scores were missing from a given time period, values were imputed with the score from the 

closest available time period. For example, if a first grader was missing a fall assessment score, their 

spring score from kindergarten would be used as the fall baseline for first grade. A dummy variable was 

created to indicate cases where an imputed assessment score was used and this dummy variable was a 

control variable in regression models. No other data were imputed, therefore, when demographic data 

were missing, the case was excluded from the model. Finally, the scale of the Total Literacy assessment 

used for Minneapolis kindergarteners changed after Year 1. To account for this, Total Literacy scores for 

each year were standardized and are presented as change in standard deviation units. Appendix J 

provides additional details about the cleaning procedures. For all data cleaning and analysis, Child Trends 

used the SAS® statistical software package. 

Difference-in-Difference 

The confirmatory research question asks about reading growth during the tutored years. Because our 

study does not have a pool of students who were never tutored, tutored students are both the treatment 

and comparison groups in the analyses. In the year(s) tutored, the students are referred to as Tutored 

Group; in the SIF years when they did not receive tutoring, the students are the Comparison Group. For 

example, (see Figure 9), a tutored student in Year 1 would be in the tutored group, and in Years 2 and 3 

when they were not tutored, they would be in the comparison group in each year. 

Figure 9: A Tutored Student in the Sample

To understand the gains from fall to spring in reading skills, we use a difference-in-difference approach

(Lechner, 2010; Somers et al., 2013; Card and Krueger, 1994) to compare the gains in the year tutored

to the gains in year or years where the student was not tutored. A difference-in-difference model 
42

assesses the effects on the outcome by measuring change from a pre-intervention baseline to a postintervention outcome. Because the model subtracts the difference at the baseline period, the model 

essentially adjusts for any differences between groups found at the baseline assessment. If the gain from 

fall to spring is greater for the group experiencing the intervention, the intervention is said to have an 

effect. 

An assumption of the difference-in-difference approach is that the intervention group and comparison 

group, absent of intervention, would have the same rate of growth on the outcome (Somers et al., 2013). 

Because our comparison group is comprised of the same students who received tutoring at some point in 

the study, we mitigated the selection bias that often occurs when comparing treated to non-treated 

cases. The characteristics for being selected into tutoring were the same for the both groups. Also, our 

comparison group is comprised of students’ growth in either the year before or the year after tutoring. To 

ensure the comparison group had face validity, we used descriptive statistics and t-tests to examine the 

difference between the growth in the year before and the year after tutoring. No significant differences 

were found. Any acceleration in growth that might have been observed in the tutored year did not appear 

to carry over to following years. Therefore, we have confidence that the comparison group of students in 

years before and after tutoring is a valid comparison group for this study. 

For each assessment (Mondo subscales, Total Literacy, MAP, and MCA), the difference-in-difference 

model compared the fall to spring gains in the Tutored Group to the Comparison Group. Fall and spring 

scores were adjusted using a multivariate regression which controlled for student-level demographics 

including: race/ethnicity, gender, special education status, English Language Learner status, and 

free/reduced price lunch status. The equation below shows the estimation model:

 

 

Equation 1: Difference-in-Difference Estimation

(Tutored T2 – T1) – (Comparison T2 - T1) = Difference-in-Difference Estimate

Where T1 = fall, T2 = spring

Control variables: Male, Black, Latino, Asian, Native American, Special Education, English 

Language Learner, Free/Reduced Price Lunch recipient

Analyzing Program Quality 

Child Trends collected the domain and total scores for each program observed using the PQA assessment 

in Years 1-3. Data was checked for outliers and missing values. No program was missing a domain or 

total scores, and all data points were within the expected range. No observations were dropped from the 

analysis. We conducted t-tests to check for significant differences in each domain and total score for each 

year. As mentioned in the Findings section below, because there were only four programs in Strategy B 

and B3, readers should interpret the significant differences with caution. They may have statistical

significance, but may not be well-representative of the program quality of the tutoring programs. 
43

Outcome Findings: Student Proficiency and Growth

Confirmatory Outcome Questions 

To what extent do the three capacity building Strategies (Strategies A, B, and B3) 

affect reading growth for students?

 

For each strategy, Child Trends compared the assessment scores of tutored students to a non-tutored, 

comparison group before (fall scores) and after tutoring took place (spring scores) for each year of this 

evaluation. The primary assessment for students in Strategy A and B (students in Saint Paul Public 

Schools) was the Mondo Assessment. The Mondo Assessment contains five subscales: Letter Recognition, 

Letter Sound, Print Concepts, Phonemic Awareness, and Word Knowledge. The primary assessments for 

students in Strategy B3 were MAP and MCA. Each of these two assessments have one overall score that 

represents the students’ reading ability. The difference between the fall to spring was calculated for 

tutored and comparison students, and was adjusted with multivariate regression to control for 

demographic factors. Effect sizes were then calculated to reflect the difference (in standard deviation 

units) between the average gains experienced by the tutored students and the average gains 

experienced comparison group. Throughout this section, positive effect sizes indicate that the tutored 

group achieved larger gains than the comparison group. When there are negative effect sizes, that 

indicates that the comparison group had larger gains (fall to spring) than the tutored group. Each effect 

size is also shows with the statistical significance of that effect. 

Summary of Growth in Reading Gains by Strategy

The tutored group in Strategy A in Saint Paul made significant gains over the comparison group in all five 

Mondo subscales, with effect sizes 8 ranging from 0.17 (letter sound, in Year 1) to 1.12 (word knowledge, 

year 2). Tutored students in Strategy A also made significant gains on three of five subscales (letter 

recognition, letter sound, and print concepts) in Year 3, though effect sizes only ranged from 0.05 to 0.23

(see Appendix K for full results).

Students in Strategy B had significant gains on all Mondo subscales, though only in Year 2. Effect sizes 

ranged from 0.62 for letter recognition to 1.42 for word knowledge. 

Kindergarteners in Strategy B3 had significant gains in Year 1 in Total Literacy, though Years 1 and 2 

were prior to receiving the capacity-building services from the Tutoring Partnership. Table 16 summarizes 

the significant findings by assessment for each Strategy and year. 

 

8 Cohen’s D (d) statistic is used to determine effect size of tutoring. An effect size of 0.2 is considered small, 0.5 is 

considered moderate, and 0.8 is considered large (Cohen, 1988). Other rigorous evaluations of tutoring programs 

have typical found small effect sizes, often around 0.1 (Jacob, Smith, Willard, & Rifkin, 2014). Therefore, we 

anticipated small effect sizes for this evaluation. 
44

Table 16: Effect Sizes and Significance Levels for Tutored Students by Strategy by Year

Year 1 Year 2 Year 3 

A B B3 A B B3 A B B3

Mondo 

Letter 

Recognition

0.22 *** 0.13 NS - 0.41 *** 0.62 *** - 0.07 ** 0.13 NS - 

Mondo 

Letter Sound 0.17 ** 0.05 NS - 0.87 *** 0.97 *** - 0.23 *** 0.10 NS - 

Mondo Print 

Concepts 0.30 *** 0.05 NS - 0.61 *** 0.79 *** - 0.05 ** 0.05 NS - 

Mondo 

Phonemic 

Awareness

0.61 *** 0.04 NS - 1.05 *** 1.16 *** - 0.06 NS -0.04 NS - 

Mondo Word 

Knowledge 0.72 *** 0.26 NS - 1.12 *** 1.43 *** - 0.05 NS 0.08 NS - 

MCA

- - 0.01 NS - - 0.16 NS - - 0.54 * 

MAP

- - -0.02 NS - - -0.21 NS - - -0.03 NS

Total 

Literacy - - 0.48 * - - -0.26 NS - - -0.06 NS

Notes: (*) indicates differences are statistically significant to at least p < .05, (**) p<.01, and (***) p<.001. (–) indicates assessment was not available for the 

Strategy in that year. (NS) indicates non-significant gains.

 

To what extent do the three capacity building Strategies (Strategies A, B, and B3) 

affect reading proficiency for students?

For this study, reading proficiency is defined as scoring 50 or above on the MCA-III, the state’s 

standardized reading assessment. The MCA-III is administered each spring beginning in third grade, and 

thus our sample analyzes rates of proficiency (regression adjusted for demographic factors) for third 

through fifth graders. There were insufficient comparison group scores in Year 2 for Strategy A and B 

programs to run the analysis; however, comparing proficiency rates across Strategies shows that tutored 

students had higher rates of proficiency than the non-tutored comparison group in Year 1 (p>.05) and 

Year 3 (p>.001). For reading proficiency, no significant differences were found between tutored students 

and comparison students for Strategy B3 in any year. 
45

Figure 10. Adjusted Percent Proficiency by Strategy and Year

Note: (*) indicates differences are statistically significant to at least p < .05, (**) p<.01, and (***) p<.0001. MCA Proficiency data was too small 

to analyze in Year 2 for Strategies A and B. Data Source: District Research, Evaluation and Assessment Department.

Exploratory Outcome Questions 

To what extent does tutoring have a differing effect on reading growth and 

proficiency for low-income students, compared to students not from low-income 

families?

Because the Tutoring Partnership focused on closing the achievement gap for low-income students, we 

explored the reading growth for tutored and comparison students based on family income level. Lowincome status was determined by whether a student qualified for free or reduced price lunch. For these 

subgroup analyses, the analysis pools data across Years 1-3 to allow for sufficient sample sizes. 

Summary of Reading Gains for Low-income Students 

Nearly all (85%) students receiving tutoring from programs participating in the Tutoring Partnership were 

low-income. This exploratory question sought to learn more about the associations of tutoring with 

reading growth for low-income and the smaller portion of the sample of higher income students. Overall, 

both low-income and higher income students in Strategy A and B experienced significant gains on Mondo 

subscales, except for the higher income students in Strategy B on letter sound. Effect sizes were in the 

moderate to large range from 0.32 to 1.14. 

Low-income and higher income students in Strategy A experienced similar gains on print concepts (0.54, 

0.53), while low-income students had larger gains on letter recognition (0.41, 0.35). Higher income 

students had larger gains on letter sound (0.85, 0.75), phonemic awareness (1.13, 1.00) and word 

knowledge (1.14, 0.98). 

19%

22%

8%

22% 23%

10%

17%

26%

14% 14%

15%

9%

12%

21% 21%

12%

Strategy A Strategy B Strategy B3 Strategy A Strategy B Strategy B3 Strategy A Strategy B Strategy B3

Year 1 Year 2 Year 3

Tutoring Group Comparison Group ***

***

*
46

For students in Strategy B, low-income and higher income students had similarly sized gains on letter 

recognition (0.51, 0.50), phonemic awareness (0.81, 0.81), and word knowledge (1.07, 1.10). Lowincome students had larger gains on print concepts (0.49, 0.38). 

In Strategy B3, low-income tutored students had significantly lower gains than low-income comparison

students on the MAP assessment (effect size -0.23). There were not significant findings on the MCA or 

Total Literacy scale for kindergarteners. A full set of figures for these subgroup analyses are available in 

Appendix L. 

Table 17: Summary of Effect Sizes for Tutored Students by Low-income Status, Years 1-3 

Combined

Strategy A Strategy B Strategy B3

Non-FRL FRL Non-FRL FRL Non-FRL FRL 

Mondo Letter 

Recognition

0.35 *** 0.41 *** 0.50 *** 0.51 *** - - 

Mondo Letter 

Sound

0.85 *** 0.75 *** 0.32 NS 0.46 *** - - 

Mondo Print 

Concepts

0.53 *** 0.54 *** 0.38 ** 0.49 *** - - 

Mondo 

Phonemic 

Awareness

1.13 *** 1.00 *** 0.81 ** 0.81 *** - - 

Mondo Word 

Knowledge

1.14 *** 0.98 *** 1.10 *** 1.07 *** - - 

MCA - - - - -0.91 NS 0.21 NS

MAP - - - - -0.13 NS -0.23 **

Total Literacy - - - - 0.89 NS 0.20 NS

Notes: (*) indicates differences are statistically significant to at least p < .05, (**) p<.01, and (***) p<.0001. (–) 

indicates assessment was not available for the strategy in that year. (NS) indicates non-significant gains.

To what extent does tutoring have a differing effect on reading growth and 

proficiency for specific racial and ethnic groups (i.e., American Indian, Asian 

American, Latino, African American, and White)? 

A key goal of the Tutoring Partnership was to reduce the achievement gap seen between white students 

and students of color. In this set of analyses, we compare the gains in reading skills among racial/ethnic 

groups. As with the subgroup analyses by income status, these analyses also pool tutored and 

comparison students from Years 1-3 to create sufficient sample size (see Table 18). 

Summary of Reading Growth by Race/Ethnicity Groups

Students across all race/ethnicity groups in Strategy A made significant gains on all five Mondo subscales, 

with effect sizes ranging from 0.37 to 1.23. Interestingly, there were not consistent levels of growth 

across all subgroups nor did one race/ethnicity group have the highest or lowest gains when comparing 

subtests. For example, Native American students had the highest growth in letter recognition (0.47), and 

phonemic awareness (1.23) while Latino students had the highest growth in letter sound (0.88). 
47

In Strategy B, tutored students in all race/ethnicity subgroups made significant gains on all five Mondo 

subscales. The exception was White tutored students in Strategy B, who did not make significant gains 

over the comparison group on letter sound. Effect sizes across groups and subscales ranged from 0.30 to 

1.16. In Strategy B, Asian tutored students had the highest gains on four of the five Mondo subscales. 

Differences between race/ethnicity subgroups were also larger in Strategy B than in Strategy A. For 

example, on letter sound, Asian students had twice the gains of Black students (0.74, 0.37). In Strategy 

A, the gains between Asian students and Black students were more similar (0.43, 0.38). 

Findings for students in Strategy B3 were mixed in terms of significant growth in reading. Black students 

made significant gains on the Total Literacy scale administered in kindergarten (effect size 0.39). 

However, on the MAP assessment, Latino and White comparison students made grader gains than the 

tutored students of the same race/ethnicity (-0.75, -0.64). A full set of figures for these subgroup 

analyses are available in Appendix L. 

 
48

Table 18: Summary of Effect Sizes and Significance Levels by Race/Ethnicity Groups, Years 1-3 Combined

Years 1-3 Combined

Strategy A Strategy B Strategy B3

Asian Black Latino Native White Asian Black Latino White Asian Black Latino White

Mondo 

Letter 

Recognition

0.43 *** 0.38 *** 0.43 *** 0.47 ** 0.37 *** 0.74 *** 0.37 *** 0.41 ** 0.25 NS - - - - 

Mondo 

Letter 

Sound 

0.75 *** 0.72 *** 0.88 *** 0.79 *** 0.79 *** 0.56 *** 0.34 ** 0.42 ** 0.44 ** - - - - 

Mondo 

Print 

Concepts

0.59 *** 0.51 *** 0.59 *** 0.53 *** 0.46 *** 0.65 *** 0.37 *** 0.47 *** 0.30 ** - - - - 

Mondo 

Phonemic 

Awareness 

1.03 *** 0.95 *** 0.95 *** 1.23 *** 1.10 *** 1.03 *** 0.64 *** 0.63 ** 0.51 ** - - - - 

Mondo 

Word 

Knowledge

1.00 *** 0.99 *** 0.99 *** 0.94 *** 1.02 *** 1.03 *** 1.07 *** 1.16 *** 1.11 *** - - - - 

MCA - - - - - - - - - 0.05 NS 0.05 NS 0.29 NS 0.59 NS

MAP - - - - - - - - - 0.14 NS 0.05 NS -0.75 *** -0.64 * 

Total 

Literacy - - - - - - - - - -0.15 NS 0.39 * 0.33 NS 0.11 NS

Notes: (*) indicates differences are statistically significant to at least p < .05, (**) p<.01, and (***) p<.0001. (–) indicates assessment was not available for the strategy in that year. (NS) 

indicates non-significant gains. S in Strategies B and B3 are too small to analyze.

 

 
49

To what extent does participation in the Tutoring Partnership improve 

program quality as measured by Program Quality Assessment over time? Does 

this growth differ by Strategy? 

The PQA is composed of five domains and within each domain, there are multiple items that observers

rate on a scale of one to five: 1-never happens for no children, 3-sometimes happens for a few children, 

and 5-always happens for all children. To calculate a domain scores, ratings from individual items within 

the domain are averaged. Strategy A programs received consistent Tutoring Partnership services across 

all years of this evaluation. Figure 11 shows the PQA scores for Strategy A programs per domain and 

across the years of this evaluation. In Year 3, Strategy A programs scored significantly lower than in Year 

1 on the Academic Skills Building domain. As mentioned, this difference may be attributed to the changes 

in the PQA.

Figure 11. Strategy A PQA Scores per domain and year

4.74

4.11

3.33

2.50

3.43 3.62

4.68

4.03

3.33

2.35

2.93

3.46

4.67

4.07

3.28

2.24

2.73+

3.40

Safe Environment Supportive

Environment

Interaction Engagement Academic

Climate/Skill

Building

Total Scale

Year 1

(9)

Year 2

(16)

Year 3

 (15)

Source: Program Quality Assessments, 2013-2016

While Strategy B programs received enhanced Tutoring Partnership services throughout the evaluation, 

there were no significant differences in any of the PQA domains (see Figure 12). 
50

Figure 12. Strategy B PQA Scores per domain and year

4.74

3.76

3.30

2.78 2.80

3.49

4.65

4.15

3.65

2.71 2.53

3.54

4.88

3.88

2.37 2.16

2.66

3.19

Safe Environment Supportive

Environment

Interaction Engagement Academic

Climate/Skill

Building

Total Scale

Year 1

(5)

Year 2

(4)

Year 3

(2)

Source: Program Quality Assessments, 2013-2016 

Strategy B3 programs received no Tutoring Partnership services in Years 1 and 2. However, Strategy B3 

programs began receiving enhanced Tutoring Partnership services in Year 3. As shown in the Supportive 

Environment domain in Figure 13, Strategy B3 program scored significantly higher in Year 3 than in Years 

1 and 2. 

Figure 13. Strategy B3 PQA Scores per domain and year

4.64

3.42+

3.08

2.12 2.08

3.09

4.57

3.84+

3.24

2.40 2.35

3.28

4.70

4.11

3.53

2.91 2.88

3.63

Safe Environment Supportive

Environment

Interaction Engagement Academic

Climate/Skill

Building

Total Scale

Year 1

(5)

Year 2

(4)

Year 3

(2)

Source: Program Quality Assessments, 2013-2016
51

Summary of Program Quality Growth 

Tutoring programs were observed at the site level in each Strategy to determine year-by-year growth on 

the Program Quality Assessment. Based on the analysis of domain and total scale scores, programs in 

Strategy A maintain moderate-level quality throughout the SIF years. The analysis found a significant 

decline in the Academic Climate/Skills Building domain between Year 3 and Year 1, however, it is 

unknown if the site quality declined or it was challenging for the sites to meet the more rigorous items in 

the revised domain. Strategy B programs also maintained an overall moderate level of quality in each 

year. Programs in Strategy B3 saw a significant increase in the Supportive Environment domain overtime. 

While this is a promising finding given these programs had just one year of Tutoring Partnership services, 

the low number of programs observed (n=5, 4, 2) in each year means readers should interpret these 

results with caution. 

Discussion of Outcome Findings

The Tutoring Partnership sought to improve tutoring program quality and positively affect student 

reading growth and proficiency. The outcome study examined confirmatory and exploratory questions 

about student growth in reading as well as an exploratory question about the change in tutoring program 

quality overtime. Significant findings include: 

Overall 

• Tutored students in Strategy A had significantly higher growth than the comparison group in all five 

Mondo subtests (Letter Recognition, Letter Sound, Print Concepts, Phonemic Awareness, and Word 

Knowledge) in Years 1 and 2. Strategy A tutored students also saw significant gains over the 

comparison students in three subtests in Year 3 (Letter Recognition, Letter Sound, and Print 

Concepts). Effect sizes ranged from 0.05 to 1.12. 

• Tutored students in Strategy B had significantly higher growth in all five Mondo subtests in Year 2. 

Effect sizes for these students ranged from 0.62 to 1.43, representing larger growth than students in 

Strategy A. This was as expected since tutoring programs in Strategy B receive more intensive 

supports from the Tutoring Partnership. 

• Tutored kindergarten students in Strategy B3 had significantly higher growth in Year 1 on the Total 

Literacy scale (a year when their programs were not participating in the Tutoring Partnership), and 

tutored students in Strategy B3 programs had greater growth on the MCA in Year 3 (a Tutoring 

Partnership year). The effect sizes were moderate for both significant findings (0.48, 0.54). 

 

Low-Income Students

• Low-income and higher income students tutored in Strategies A and B saw significant growth in 

reading skills. The magnitude of the effect sizes were different by Strategy, but there was not a 

consistent pattern of higher income students or low-income students making the greatest gains 

across assessments. 

• Low-income comparison group students in Strategy B3 had significantly greater gains than tutored 

students on the MAP assessment. Though the effect is in the small-moderate range (-0.23), this was 

an unexpected finding. 

 

Race/Ethnicity Subgroups 

• Tutored students from all racial/ethnic backgrounds in Strategy A made significantly greater gains on 

the five Mondo subscales. Effect sizes ranged from moderate (0.37) to large (1.23). 
52

• Significant gains were also found for most subgroups tutored students in Strategy B on Letter 

Recognition Letter Sound, Print Concepts, Phonemic Awareness (Asian, Black, Latino, and White), 

and Word Knowledge (Asian, Black, and Latino). Effect sizes for Strategy B were similar in magnitude 

to Strategy A, ranging from 0.37 to 1.16. 

• Black students tutored in Strategy B3 programs made significantly greater gains on Total Literacy. 

This was the only race/ethnicity subgroup to have a significant gain on this assessment. The effect 

size was moderate (0.39). 

• Latino and White students in the comparison group for Strategy B3 made significant gains over 

tutored students on MAP (effect sizes, -0.75, -0.64). This was an unexpected result and the effect 

sizes are moderate. 

Program Quality 

• Programs in Strategy A saw a decline in the quality of Academic Climate/Skill Building from Year 1 to 

Year 3. 

• Strategy B3 programs had a significant increase in the Supportive Environment domain, with the 

highest score achieved in Year 3.

Overall, students enrolled in tutoring made greater gains in the year tutored than in years when they 

were not tutored. From this analysis, the presence of significantly greater gains did not vary by the 

student’s family income or their racial/ethnic background. During the three years of the Tutoring 

Partnership funded through SIF, tutoring programs maintained a mid-range level of quality that did not 

substantially change overtime. 

Conclusions and Limitations

Conclusions 

Overall, tutoring programs were satisfied with the Tutoring Partnership services, and found 

them to be helpful to their programs. Programs found tutor training to be the most helpful service, 

however, it was not utilized by most tutors. For initiatives like the Tutoring Partnership, engaging tutors 

who may not be familiar with their services may be a difficult challenge. Initiatives and programs offering 

tutor training may need to provide additional incentives to entice tutors to training opportunities, make 

trainings highly practical and hands-on, and provide training at a variety of times and locations to boost 

training participation. 

Requiring participation in Tutoring Partnership services increased uptake. Strategy B and 

Strategy B3 programs had requirements around their participation in Tutoring Partnership services. Thus, 

they had higher levels of participation in PD workshops and technical assistance services. Programs in 

Strategy A could opt-in to services, and tended to have lower levels of participation. Initiatives like the 

Tutoring Partnership may consider the requirements of participating and the readiness to change when 

they are working with programs on quality improvement. 
53

Tutor and student demographics are not aligned. The Tutoring Partnership has a primary goal of 

closing the achievement gap that exists between low-income and high-income students, and between 

white students and students of color. While tutors were highly-educated, their language and racial/ethnic 

backgrounds did not match students they were working with who are primarily students of color, and 

more than 40% are English Language Learners. Efforts by the Tutoring Partnership to increase the 

cultural competency of tutoring programs and their tutors were just getting underway during the SIF 

grant. Tutoring programs should make an effort moving forward to continue building cultural 

competency, awareness of racial biases, and relationship building skills a key aspect of their training. 

Overall, tutored students are making greater gains in the year they are tutored. These gains 

are present for low-income and higher-income students, and for students from different racial/ethnic 

backgrounds. Limitations in the sample sizes and sample demographics make it challenging to state if 

one Tutoring Partnership Strategy is more effective than another at producing these gains. 

Programs maintained moderate levels of quality. Based on PQA scores, programs in all three 

Strategies had moderate levels of quality in the three years of the SIF funding. It was difficult to 

determine changes in program quality because of limited sample sizes; further study into the associations 

of program quality to student outcomes will be of great benefit to the tutoring field. 

Limitations 

While the implementation study had few limitations, the outcome study grappled with several challenges. 

Firstly, samples of students across samples were not equivalent in size. Findings tended to be strong for 

students in Strategy A. While some positive trends were seen for students in Strategy B and B3, limited 

significant findings were due at least in part to insufficient power. In both Strategies B and C, particular 

subgroups could not be analyzed due to small sample sizes. Thus, the analyses for Strategies B and B3 

should be interpreted as preliminary. Further study with larger sample sizes would greatly help our 

understanding of the change in student outcomes by Strategy. 

Secondly, student samples between Strategy groups were not equivalent in terms of student 

demographics. In the Interim Impact Report for this study, we reported that the previous method of 

using propensity score matching to match students within and across Strategies was unsuccessful. This 

was mostly due to the population differences from Saint Paul to Minneapolis. Even when comparing the 

demographics of students within the same city (Saint Paul) for Strategies A and B (see Table 13), there 

were differences the racial/ethnic backgrounds of the programs in each Strategy. While we were able to 

see significant growth in the year students were tutored, we have limited ability to detect differences in 

student outcomes that can bet attributed to a given Strategy. 

Lastly, analysis is limited to the select academic outcomes assessed by the districts. No common 

measures were administered to students in kindergarten through fifth grade across district, except for the 

MCA. However, there was insufficient MCA data in Saint Paul in Year 2 to analyze growth in that 

measure. Moreover, some of the measures used in the analysis are curriculum-based instructional tools 

that were not designed for research purposes. Greater comparability and variety in measures across 

districts would strengthen external validity (i.e., allow findings to be generalized more broadly). Including 

measures for different domains of student outcomes, such as social-emotional, behavioral, and/or 

interpersonal outcomes, would provide a more comprehensive picture of the role of tutoring in students’ 

lives. 
54

References

Bixby, K.E., Gordon, E.E., Gozali-Lee, E., Gray Akyea, S. & Nippolt, P.L. (2011). Best Practices for 

Tutoring Programs: A Guide to Quality. Saint Paul, MN: Saint Paul Public Schools Foundation. 

Card, D., and Kreuger, A. (1994). Minimum wages and unemployment: A case study of the fast-food 

industry in New Jersey and Pennsylvania. American Economic Review, 84, 4, 772-793. 

Corporation for National and Community Service. (2016). Social Innovation Fund. Retrieved from: 

http://www.nationalservice.gov/programs/social-innovation-fund. 

Eccles, J. and Gootman, J.A. (Eds.) (2002). Community Programs to Promote Youth Development.

Washington, DC: National Academies Press.

Jacob, R. T., Smith, T. J., Willard, J. A., Rifkin, R. E. (2014). Reading Partners: The implementation and 

effectiveness of a one-on-one tutoring program delivered by community volunteers. New York 

City, NY: MRDC.

Lechner, M. (2011). The estimation of causal effects by difference-in-difference models. Foundation and 

Trends ® in Econometrics, 4, 3, 165-224. 

Little, P.M.D., Wimer C., & Weiss H.B. (2008). After School Programs in the 21st Century: Their Potential 

and What It Takes to Achieve It. Harvard Family Research Project Issues and Opportunities in 

Out-of-School Time Evaluation, 10. 

Michlin, M., Schultz, A., Harty, C. (2015). Saint Paul Public Schools Foundation Tutoring Partnership Social 

Innovation Fund Year One Implementation Evaluation Report (2013-2014). Saint Paul, MN: 

Center for Applied Research and Education Improvement. 

Roth, J., Brooks-Gunn, J., Murray, L., & Foster, W. (1998). Promoting healthy adolescents: Synthesis of 

youth development program evaluations. Journal of Research on Adolescence, 8,424–460.

Saint Paul Public Schools Data Center. (2013). Strong Schools, Strong Communities Monitoring and 

Reporting System: VisionCards. Saint Paul, MN: Department of Research, Evaluation, and 

Assessment.
55

Smith, C., Hohmann, C. (2005). High/Scope YPQA Technical Report: Full Findings From the YPQA 

Validation Study (pp. 5 – 6).

Soli, M., Beckett, A., Hirilall, A., Li, W., & Tout, K. (2015). Tutoring Partnership Impact Evaluation: 

Interim Report. Minneapolis, MN: Child Trends. 

Somers, M., Zhu, P., Jacob, R., and Bloom, H. (2013). The Validity and Precision of the Comparative 

Interrupted Time Series Design and the Difference-in-Difference Design in Educational 

Evaluation. New York: MDRC. 

Vandell, D.L., Reisner, E.R., & Pierce, K.M. (2007). Outcomes Linked to High-Quality Afterschool 

Programs: Longitudinal Findings from the Study of Promising Afterschool Programs. Washington, 

DC: Policy Studies Associates, Inc.

Vanneman, A., Hamilton, L., Baldwin Anderson, J., and Rahman, T. (2009). Achievement Gaps: How 

Black and White Students in Public Schools Perform in Mathematics and Reading on the National 

Assessment of Educational Progress, (NCES 2009-455).
56

Appendix A. Tutoring Partnership 

Staff Interview Protocol

We will start with a few questions about your role at the Tutoring Partnership:

Could you describe your roles and responsibilities related to the Tutoring Partnership? How long have you 

been involved with the program? Have you had other roles than the one you currently hold? 

Great, the next few questions are to highlight your history and work with programs across the years.

• Can you describe the history of how tutoring programs were invited to participate in the Tutoring 

Partnership? How do programs formally “JOIN” the partnership?

• Probe: Did this change with the SIF grant? How have you seen the contract and funding for the four 

SIF subcontractor sites working? What was the intent behind this structure? 

• Probe: How did you decide which services would be “enhanced” for strategy B vs. business as usual 

for strategy A?

You delivery a wide variety of services to programs and we’d be interested in hearing more information 

about each. To facilitate that, we have a list of questions that we’d like to ask you about each service. 

Does that sound ok? 

1. How did you 

decide to

make it 

required or 

optional?

2. How did 

you 

determine 

dosage 

goals?

3. Can you describe 

your efforts to 

maintain fidelity and 

consistency in 

delivering the various 

components? Are 

these efforts 

effective?

4. Strengths 

of TP 

approach

5. Limitations of 

TP approach

a Professional 

Development 

services

b Technical 

assistance

d Tutor trainings

e Communication 

with school 

districts

f Connections 

with 

organizations 

and Tutoring 

Partnership 

meetings

g CQI services

• Probe: Are there services that you wanted to add or make required at some point? 
57

• Do you ever see programs opting out of services that you believe would benefit them? What was 

your approach to that?

We’d like to talk about strategies and goals for the Tutoring Partnership and how they have changed 

overtime. 

• What would you say have been the top priorities of the Tutoring Partnership this year?

• How were this year’s priorities different than in years 1 or 2 of SIF? 

We are nearing the end of the interview. I have some final questions for you about the overall 

perceptions of Tutoring Partnership over the past three years. 

• What have been the biggest successes of the program?

• What have been the key challenges for the Tutoring Partnership program?

• Have you seen differences in the four contractor programs participation as a result of the SIF 

enhanced partnership? 

• If you had to select the most important service that the Tutoring Partnership provides to improve 

program quality, what would it be? Most important service to improve student outcomes? 

• I would like to close by asking if you have any other information to add that you think is important 

for the evaluation of the Tutoring Partnership?
58

Appendix B. Tutoring Program Staff 

Interview

Interview Introduction: 

Hello, my name is _______ and I’m here with ______. We’re calling from Child Trends because we 

scheduled a phone interview for the purpose of the Tutoring Partnership SIF Evaluation. Are you still 

able to do the interview with me now? (if not, ask if there’s a better time that works for them).

[Consent]

Thank you so much for taking the time to participate in this interview about the Tutoring Partnership. As 

a reminder, these interviews are a part of the SIF Evaluation funded by the Greater Twin Cities United 

Way. Thank you for recently completing the online survey, your responses were very helpful. The 

purpose of this interview is to ask you a few more in-depth questions about your program in relation to 

the Tutoring Partnership. In this interview, we will ask you a series of questions about your perceptions 

of the program and your experiences with program services and perceptions of Tutoring Partnership. 

Please let me know if you are unfamiliar with a topic and we can move on to the next question. We will 

not link your name directly with information in any reports or research papers. Participation in the study 

is voluntary and refusal to participate will not affect your participation or standing in the Tutoring 

Partnership or with funders. At any time throughout the interview if you would rather not answer a 

question, let me know and we will move onto the next one. This interview will take about 45 minutes to 

complete. I will be asking you a series of questions and my colleague will type your responses as we go 

along. By continuing with the interview, you are consenting to participate in this study.

Do you have any questions before we begin?

[To be completed by interviewer]

Subject ID: ____

Date: Interviewer: _______ Note taker: _______

[Interview]

Respondent-Level Questions

1. Could you confirm your title at your organization?

2. How long have you been involved in your program?

3. Could you describe your roles and responsibilities related to your work with the Tutoring 

Partnership? 

Program-Level Questions

4. What have been the top priorities for your program this year? 

a. Probe: How do you set priorities for your program? What sources of information help you 

understand where you need to focus? How many sources of information do you need to 

report to?

We’d like to ask you a few questions to expand your responses to a few questions in the online survey.

5. In your opinion, what has been the role/purpose of the Tutoring Partnership for your program?
59

6. You indicated that you would recommend the Tutoring Partnership to other tutoring programs. 

What aspects of the TP make you likely to recommend it? Were there particular services/supports 

you would recommend? Why?

7. What has been the role/purpose of training at your program? 

a. Probe: why do you believe training is important?

8. You indicated that social/emotional learning is important to your organizational mission. Do you 

use a certain social/emotional curriculum? Do you require any social/emotional trainings, such as 

Behavior Management? 

a. Probe: What specific social/emotional skill does your program focus on? [Skills include 

self-awareness, self-management, social awareness, relationship skills, and responsible 

decision making] 

9. You reported that tutors are supervised through observations, can you describe how this process 

more about this process. How often are observations completed? How will results of the 

observation be communicated? What kind of information is reported?

a. Probe: Does your program provide training for site level staff on how to conduct 

observations? 

Minneapolis Site Questions

10. What has it been like to expand the TP services to your Minneapolis sites?

11. What types of changes have they made in Minneapolis?

12. How would you describe the challenges faced in Minneapolis? Are the challenges similar to those 

experiences in Saint Paul Sites? 

Perceptions of TP

 

13. Can you tell us more about how your program became involved with the Tutoring Partnership? 

a. Probes: When did you join? How were you invited to participate? Did you make a formal 

commitment? Do you formally recommit to the Partnership each year? 

The Tutoring Partnership provides a variety of services to programs and we would like to ask you a few 

questions about your program’s participation in services offered.

14. Some TP services are optional to programs and some are required. Can you describe how your 

program decided which Tutoring Partnership services that you would participate in within the last 

year? What has been the role/purpose of each TP service on your program within the last year? 

• Professional development (e.g., Strategic planning workshop, Cross Cultural Conflict 

workshop, Applying a Growth Mindset Workshop)

• Technical assistance (e.g., onsite discussions about program changes, training plan support, 

improvement plan support, program consultation and coaching, requests for resources and 

research)

• Volunteer recruitment

• Tutor trainings (e.g., Tutoring Basics, ABCs of Reading, Adolescent Literacy, Everyday Math, 

Math in the Middle Grades)

• Communication with school districts

• Connections with organizations
60

• Tutoring Partnership meetings (e.g., September kickoff, midyear networking lunch, year-end 

celebration)

• Continuous Quality Improvement (e.g., PQA external observations, Planning with Data, YPQA 

Basics)

• Evaluation support (e.g., AIMSWeb and FAST support, access to district data, Sprockets 

database support, data portfolios)

15. Have there been services offered that you chose not to participate in? Why? Were there any 

barriers to your participation?

16. [Note: only ask if interviewee has worked there for more than one year] How has your 

participation in the Tutoring Partnership services changed in the past 3 years? 

17. What do you believe is the most important TP service provided to improve organizational capacity 

at your program? What is less important? [Organizational capacity is the ability for an 

organization to run smoothly. To do all the things that are important to their mission]

18. Thinking about all the changes you have made in your program during the Tutoring Partnership, 

what changes do you think were the easiest to make? What about the most challenging to make?

19. What changes or new strategies may be needed to support the continuation and improvement of 

your program?

20. What resources do you think is absolutely necessary for your program? [If money, how would 

you use it?]

We are nearing the end of the interview. I have some final questions for you about the Tutoring 

Partnership.

21. To what degree do you feel the goals of the Tutoring Partnership align with your program?

22. Probe: Are there aspects of the Tutoring Partnership that made your work more difficult or 

challenging? 

23. Do you plan to grow the number of students served or your site locations? If so, what exactly do 

you plan on doing and when? If not, why not? What does growth mean for your program? 

24. I would like to close by asking if you have any other information to add that you think is 

important for the evaluation of the Tutoring Partnership?

Those are all the questions I have for you! Is there anything else you’d like to add or that you think I 

may have missed? 
61

Appendix C. Tutor Survey

Your tutoring program participates in a Twin Cities tutoring collaborative called the Tutoring

Partnership. It is a group of 20 youth programs that all provide academic-skill building to students in the 

Twin Cities. The Tutoring Partnership provides training, coaching and support to these programs and 

their tutors in order to increase the quality of tutoring in our community.

This part of the survey is composed of three sections:

1. Tutoring information

2. Training participation and perceptions

3. Strengths and challenges

At the end of the survey, you will have the opportunity to enter a drawing for a $25 gift card as a token 

of appreciation for your time.

If you have any questions, you may contact the evaluation manager, Mallory Warner-Richter, at 

mwarnerrichter@childtrends.org or 612-331-2223 ext. 18.

Section 1: Tutoring Information

1. Please select your tutoring program

[Dropdown list of choices]

2. How many students did you regularly tutor this year?

1-5 students

6-10 students

11-15 students

More than 15 students

3. Please select all grades that you tutored this year:

Pre-Kindergarten 

Kindergarten

1 

2 

3 

4 

5 

6 

7 

8 

9 

10

11

12

4. What are the primary academic subjects you focused on this year?

Literacy

Math

Science 

Other (please specify)
62

5. What are the non-academic skills you focused on this year?

Study habits

Self-regulation

Building student self-confidence

Organization

Being persistent when faced with a challenge

Increasing ability to focus on work

N/A - I did not work on these skills with my student(s).

Other (please specify)

6. How long have you tutored at your current program?

Less than 1 year

1 to 5 years

6 to 10 years

10 to 20 years

More than 20 years

7. About how many years of tutoring experience do you have?

Less than 1 year

1 to 5 years

6 to 10 years

11 to 20 years

More than 20 years

8. About how many years have your been working or volunteering in the education field?

Less than 1 year

1 to 5 years

6 to 10 years

11 to 20 years

More than 20 years

9. Are you currently or have you ever been a licensed teacher?

Yes

No

10. Are you a paid tutor?

Yes

No

Section 2: Training Participation and Perceptions

To access descriptions of Tutor Partnership trainings, please copy and paste the following

link: http://sppsfoundation.org/volunteer/tutor-training

11. Please indicate which tutor trainings you attended in 2015-16 school year. If you did not participate in 

any trainings, please choose "N/A."

Advance Active Learning

Advance Planning & Reflection

Foundations of Intercultural Development (eLearning course)

Tutor Basics

Understanding Survival Based Youth

Behavior Management Tutor Training

Tutoring English Language Learners
63

ABCs of Reading Tutor Training (Grades K-4)

Literacy Skills for Early Readers (Grades Pre-K-2)

Comprehension Lesson for Early Readers (Grades K-4)

Middle Grades Matter: Adolescent Literacy Tutoring Training (Grades 5-8)

Elementary Math 2.0 Tutor Training (Grades K-5)

Math in the Middle Grades Tutor Training (Grades 6-8)

Math in the Middle Grades 2.0 Tutor Training (Grades 6-8)

N/A

Other (please specify)

12. How effective was the training in helping your abilities in these areas? If a description does not apply, 

please select "N/A."

Very

ineffective

Somewhat 

ineffective

Somewhat 

effective

Very

effective

N/A

My comfort level interacting with other 

program staff, teachers, students, and 

families of diverse backgrounds.

My ability to effectively tutor students.

My comfort level managing students' 

problematic behaviors during tutoring 

sessions.

My intentional interactions with tutored 

students.

My ability to effectively tutor English 

Language Learners.

My commitment to education.

13. Please indicate how strongly you agree or disagree with the following statements.

Strongly 

disagree

Disagree Agree Strongly 

agree

Tutoring has been a rewarding experience for me.

I feel confident in my ability to implement researchbased tutoring strategies.

I believe I have helped students academically.

Individualizing tutoring strategies based on the 

student's need is challenging.

I find it difficult to align my tutoring intervention 

with district curriculum.

I know how to create a lesson plan for a tutoring 

session.

14. Please select the following statement that best describes the changes you've made in your work as a 

tutor since participating in trainings offered by your program this year.

I have not made any changes.

I've made minor changes.

I've made some changes.

I've made major changes.

15. How often did you apply something you learned from a tutor training to your work with students?

Never

Not often (once or twice)

Somewhat often (three or four times)

Very often (monthly or more)
64

16. How often did you use various supports that were available to help you change your work with

students?

Never Not often 

(once or 

twice) 

Somewhat 

often (three or 

four times)

Very often 

(monthly or 

more)

N/A- This 

support was not 

offered to me

Materials from the 

training

Advice from 

program staff

Advice from other 

tutors

My own experience 

in

education

Other (please specify)

17. What was the hardest to change? Please briefly describe why.

Section 3: Strengths and Challenges

18. What are your top three strengths as a tutor?

I keep my student on task.

I am culturally aware.

I can identify, create, and apply learning strategies to help students learn and reflect.

I meet students where they are (academically, emotionally).

I implement strategies such as question asking and reflection activities to successfully increase 

reading comprehension.

I’m good at building relationships.

I scaffold with students to build and develop their higher order thinking.

I show up to tutoring on a consistent, regular schedule.

I can successfully address key literacy skills for early readers (e.g. letter names, letter sounds, 

blending, etc.)

I explain things in different ways so the student understands.

I can manage behaviors using a strengths-based approach.

I bring humor and playfulness to tutoring sessions.

I address the needs of multilingual students.

I’m a good role model.

Other (please specify)

19. Please select all challenges that apply to your experiences during tutoring sessions.

Student is not motivated.

Student unwilling to do the work.

I feel upset by a student's behavior.

Student is often absent.

I don't know how to address a student's challenges.

Student struggles to stay on task.

Student struggles with new concepts.

Tutoring topics do not align with classroom lessons.

I don't have the skills to work with a particular student.

Student gets anxious.

Limited time during the school day.

Other (please specify)
65

20. What additional tutoring supports do you believe would be beneficial in the future?

More formal feedback/guidance from supervisor (e.g. results from formal observation)

Peer learning opportunities

More training dates/times

Additional non-academic trainings (e.g. social/emotional development or cultural sensitivity)

More planning/preparing time

More/new materials (e.g. books, white boards, markers)

Other (please specify)

21. Are there any other comments you would like to add about the support or your experience as a tutor?

Section 4: Demographics

22. Which of the following best describes your racial/ethnic group? (Please select all that apply)

Black or African American

White or Caucasian

Asian or Pacific Islander

Hispanic or Latino

American Indian or Alaska Native

Two or more races

Other (please describe)

23. What is the highest level of education you have attained?

Some High School

High School Diploma or GED

Some college, but no degree

Associate degree

Associate degree in Education or related field

Bachelor’s Degree

Bachelor’s Degree in Education or related field

Graduate Degree

Graduate Degree in Education or related field

24. What is your age?

Under 17 years old

18-24 years old

25-34 years old

35-44 years old

45-54 years old

55-64 years old

65-74 years old

75 years old or older

25. What language(s) do you speak at home most often with family members? (Please select all that 

apply)

English

Spanish

Hmong

Somali

Vietnamese

Lao

Arabic

Other (please specify)
66

Appendix D. Tutoring Program Staff 

Survey

Thank you for taking the time to complete this survey about your program. As part of the Social 

Innovation Fund grant evaluation, Greater Twin Cities United Way has partnered with Child Trends, a 

non-profit research organization, to conduct an independent evaluation of the Tutoring Partnership. As 

part of this evaluation, we are asking one key staff member from each program to respond on behalf of 

your program. The purpose of this survey is to collect systematic information about your tutoring 

programs during the 2015-16 school year and gain more detailed knowledge about how tutoring is 

working.

This survey has three parts:

1. Implementation of your model

2. Participation in the Tutoring Partnership

3. Experiences and perceptions of the Tutoring Partnership

This survey will take approximately 30 minutes to complete, and participation is voluntary. If you have 

any questions please feel free to contact Mallory Warner-Richter, Senior Research Analyst at Child Trends 

via email (mwarnerrichter@childtrends.org) or by phone (612) 331-2223 ex. 18.

Section 1: Program Information 

1. Please select your program: [dropdown menu of programs]

2. In 2-3 sentences, please describe your organization's mission.

3. Please select the type of services provided by your program to tutored students (select all that apply).

Literacy tutoring

Math tutoring

Other tutoring (e.g., science, writing, foreign languages)

Homework help

Enrichment activities (e.g., physical education, dance, art)

Social/emotional learning

Parent education

Housing services

College prep/access services

Mentoring

Other (please specify)

4. To what degree are the services offered by your program important to your organization's mission?

Very 

important 

Important Somewhat 

important

Not 

important

Service not

offered

Literacy tutoring

Math tutoring

Other tutoring (e.g., science, writing, 

foreign

language)

Homework help
67

Very 

important 

Important Somewhat 

important

Not 

important

Service not

offered

Enrichment activities (i.e. physical 

education,

dance, art, etc)

Social/emotional learning

Parent education

Housing services

College prep/access services

Mentoring

Other

5. Please indicate your program's service location(s).

In-school only

Community-based only

Both in-school and community based

Other (please specify)

6. Please indicate the number of program staff and volunteer tutors at your program.

Full-Time program staff (not including tutors): _______________________

Part-Time program staff (not including tutors): _________________________

Tutors (paid or volunteer): ___________________________

7. Are your tutors paid or volunteers?

All tutors are paid.

More than half of tutors are paid.

Less than half of tutors are paid.

None of the tutors are paid.

8. What portion of your tutors are licensed teachers?

(i.e., have ever or currently hold a teaching license issued by the state)

None of our tutors are licensed.

Between 1-25% are licensed.

Between 26-50% are licensed

Between 51-75% are licensed

Between 75-100% are licensed

9. Please indicate the average caseload for your tutors per year.

1-5 students per tutor

6-10 students per tutor

11-20 students per tutor

More than 20 students per tutor

10. Select all characteristics your organization considers when selecting tutors.

Current teaching license

Level of education

Past experience

Availability

Cultural similarities with students served

Student in education field

Passion for the mission

Interest in working directly with children

Other (please specify)
68

11. What is the minimum level of education needed to be a tutor at your organization?

Some high school

High School Diploma or GED

Some college

Associates Degree

Bachelors Degree

Graduate Degree

Section 2: Program Model

12. Please rank the supports/resources that you believe is most helpful for tutors. If support/resources 

isn't offered, please select N/A.

Orientation (i.e., orientation to the site, program rules, responsibilities, etc.) N/A

Initial training (i.e., training on tutoring content, process, strategies, etc.) N/A

Ongoing trainings (i.e., trainings that occur throughout the program year) N/A

On-site tutor support N/A

Tutor manual/handbook N/A

Regular check-in meetings with program staff N/A

Observations of tutors to provide feedback and support N/A

Tutor recognition (i.e., awards, leadership roles, etc.) N/A

13. Please rank the supports/resources that you believe is results in change in tutor practice. If

support/resources isn't offered, please select N/A.

Orientation (i.e., orientation to the site, program rules, responsibilities, etc.) N/A

Initial training (i.e., training on tutoring content, process, strategies, etc.) N/A

Ongoing trainings (i.e., trainings that occur throughout the program year) N/A

On-site tutor support N/A

Tutor manual/handbook N/A

Regular check-in meetings with program staff N/A

Observations of tutors to provide feedback and support N/A

Tutor recognition (i.e., awards, leadership roles, etc.) N/A

14. Indicate whether the following training topics are required, optional or not provided by your program.

Literacy interventions/strategies

Math interventions/strategies

Behavior management

Youth development

Cultural sensitivity

Social-emotional learning

Student assessment training

Other training (please specify if it is required or optional)

15. How are tutors supervised? (select all that apply)

Regular one-to-one meetings

Regular small group meetings (2-5 tutors)

Regular large group meetings (more than 5 tutors)

Observations

Other (please specify)
69

16. How frequently are the following group sizes are used to tutor your students?

Never Almost 

Never

Occasionally Almost 

every time

Every 

time

One-on-one (1 tutor: 1 student)

Small group tutoring (1 tutor: 2-5 

students)

Medium group tutoring (1 tutor: 6-9 

students)

Large group tutoring (1 tutor: 10 or 

more students)

17. What is the average duration of time between the following events (please indicate number of days 

or weeks)

Duration between student referral and enrollment

Duration between enrollment and tutoring services

Duration between tutoring sessions (i.e., time elapsed between session 1 and session 2)

18. Are students ever placed on a wait list?

Yes

No

Name of curriculum/curricula:

19. Does your program use curriculum? If so, please include the name(s) in the comment box.

No, we do not use a curriculum.

Yes, we create our own curriculum.

Yes, we purchase a curriculum.

20. Please enter your program's start and end date for the 2015-2016 school year.

Start Date: _________________________

End Date: _________________________

21. How frequently does your program monitor student progress using student assessments or data

systems?

Daily

Weekly

Monthly

Quarterly

Twice per year

Once per year

N/A - Our program does not use progress monitoring or data systems.

Please describe your progress monitoring tools or data systems (ex. tool name, who administers 

the tool)
70

Section 3: Dosage Targets

22. Please select the average number of minutes spent with students per tutoring session.

10 minutes or less

11-30 minutes

31-60 minutes

61-90 minutes

91-120 minutes

Greater than 120 minutes

Other (please specify)

23. Please briefly describe the challenges, if any, you encountered when trying to meet dosage targets.

Section 4: Tutoring Partnership Participation

24. Please rank the following Tutoring Partnership services in order of their effectiveness in making

changes to your program? If services was not used, please select N/A.

A. Professional development (e.g., Strategic planning workshop, Cross Cultural Conflict 

workshop, Applying a Growth Mindset Workshop)

B. Technical assistance (e.g., onsite discussions about program changes, training plan support,

improvement plan support, program consultation and coaching, requests for resources and 

research)

C. Volunteer recruitment

D. Tutor trainings (e.g., Tutoring Basics, ABCs of Reading, Adolescent Literacy, Everyday Math, 

Math in the Middle Grades)

E. Communication with school districts 

F. Connections with organizations 

G. Tutoring Partnership meetings (e.g., September kickoff, midyear networking lunch, year-end

celebration)

H. Continuous Quality Improvement (e.g., PQA external observations, Planning with Data, YPQA 

Basics) 

I. Evaluation support (e.g., AIMSWeb and FAST support, access to district data, Sprockets 

database support, data portfolios)

25. Please indicate your willingness to recommend the following Tutoring Partnership services to other 

tutoring organizations? If services were not used, please select N/A.

Strongly do not 

recommend

Somewhat do 

not recommend

Somewhat 

recommend

Strongly 

recommend

N/A

Professional development 

(e.g., Strategic planning 

workshop, Cross

Cultural Conflict workshop, 

Applying a Growth Mindset 

Workshop)

Technical assistance (e.g., 

onsite discussions about 

program changes,

training plan support, 

improvement plan support, 

program consultation

and coaching, requests for 

resources and research)
71

Strongly do not 

recommend

Somewhat do 

not recommend

Somewhat 

recommend

Strongly 

recommend

N/A

Volunteer recruitment

Tutor trainings (e.g., 

Tutoring Basics, ABCs of 

Reading, Adolescent

Literacy, Everyday Math, 

Math in the Middle Grades)

Communication with school 

districts

Connections with 

organizations

Tutoring Partnership 

meetings (e.g., September 

kickoff, midyear

networking lunch, year-end 

celebration)

Continuous Quality 

Improvement (e.g., PQA 

external observations,

Planning with Data, YPQA 

Basics)

Evaluation support (e.g., 

AIMSWeb and FAST 

support, access to

district data, Sprockets 

database support, data 

portfolios)

26. Does your program collaborate and/or continuously communicate with other tutoring organizations? 

If so, please describe below.

Yes

No

Please briefly describe your collaboration and/or communication with other tutoring 

organizations.

27. Is your program a SIF contractor program?

Yes

No

Section 5: SIF Contractor Experiences (Only SIF Contractor Programs)

28. As a SIF contractor, your program receives additional funding as part of the Tutoring Partnership. 

Please describe how you have used your funds in the 2015-2016 school year.

29. What specific changes have you made as a result of these additional funds?
72

Section 6: Perceptions

30. Please rate the level of importance for the following list of activities.

Not at all 

important

Slightly 

important

Moderately 

important

Extremely 

important

Using data for quality improvement

Adopting standards and aligning 

assessment tools

Supporting the orientation, training, 

and development of tutors

Providing and participating in ongoing 

training and coaching to staff 

members

Establishing processes for connecting 

with families and the community

Establishing processes for connecting 

with schools and teachers

31. Please indicate how strongly you agree or disagree with the following statements

Strongly 

disagree

Disagree Agree Strongly 

agree

I believe my program is of higher quality because we 

participated in the

Tutoring Partnership.

My experience with the Tutoring Partnership has 

been what I expected.

I believe that continuous quality improvement 

activities are important for my

program to successfully tutor students.

I would recommend that other programs join the 

Tutoring Partnership.

Collaboration and communication among tutoring 

organizations is beneficial to

all participating programs.

The primary purpose of the Tutoring Partnership is 

to support programs by

helping them achieve program quality goals.

We have made changes to our program as a result 

of joining the Tutoring

Partnership.

32. In the 2015-16 school year, what was one change your program made that you believe had the 

biggest benefit to children?

33. In the 2015-16 school year, what aspect of your program was the most difficult to change and why?

34. Please describe any plans to sustain changes your program has made during your time in the 

Tutoring Partnership.

35. Are there any other comments you would like to add about the Tutoring Partnership

or your experience in the partnership?
73

Appendix E. Professional 

Development Workshops by SIF 

Years Offered

Year 1 Year 2 Year 3

Tutoring Partnership Meetings

Tutoring Partnership Celebration x x x 

Tutoring Partnership Annual Fall Kick-Off x x x 

Tutoring Partnership Luncheon x x 

SIF Contractor Meetings

SIF Contractor: Creating a Development Tutor Training Plan x 

SIF Convening x 

PQA Workshops

YPQA Basics for Self-Assessment x x x 

Best Practices in ASB x 

ASB PQA Webinar x x 

Mini PQA Basics x 

Mini PQA Basics for Self-Assessment x 

PQA Self-Assessment Brown Bag x 

SEL Workshop: Applying Growth Mindset x 

Planning and Reflection x 

Reporting and Assessment Workshops

AIMSweb Reporting x x 

AIMSweb Student Assessment x 

FAST Training x 

FAST Assessment x 

Program Evaluation Workshops

Program Evaluation as a Tool x 

Program Evaluation Brown Bag x 

Math Workshops

Math Summit x 

Math Networking Meeting x 

Organizational Management Workshops

Program Level

Planning with Data x x x 

Developing a Logic Model x 

Cultural Competency for Management x 

Theory of Change x 

Microsoft Excel Training x 

Strategic Planning for Nonprofits x 

Cross Cultural Conflict x 

Tutor Level

Volunteer Management Brown Bag x x x 

Tutor as Mentors x 

Cultural Competency for Direct Service Staff x 

Quality Coaching x 
74

Year 1 Year 2 Year 3

Training with Intention x 

Student Level

Youthprise Convening x 

Tutoring English Language Learners x 

Family Level

Family Engagement x 

Parent Academy x 

 
75

Appendix F. Professional 

Development Attendance per 

Strategy 

 Year One Year Two Year Three

Strategy A Strategy B/B3 Strategy A Strategy B/B3 Strategy A Strategy B/B3

Total % Total % Total % Total % Total % Total % 

Tutoring 

Partnership 

Meetings 

45 51% 14 36% 28 47% 18 78% 68 72% 31 70%

Tutoring 

Partnership 

Celebration 

28 31% 9 23% 20 34% 9 39% 18 19% 10 23%

Tutoring 

Partnership 

Annual Fall 

Kick-Off 

17 19% 5 13% N/A N/A N/A N/A 31 33% 13 30%

Tutoring 

Partnership 

Luncheon

N/A N/A N/A N/A 8 14% 9 39% 19 20% 8 18%

SIF Contractor 

Meetings 

N/A N/A 11 28% N/A N/A N/A N/A N/A N/A 11 25%

SIF Contractor: 

Creating a 

Development 

Tutor Training 

Plan

N/A N/A 11 28% N/A N/A N/A N/A N/A N/A N/A N/A

SIF Convening N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 11 25%

PQA 

Workshops

35 39% 18 46% 77 131% 17 74% 74 78% 54 123%

YPQA Basics for 

SelfAssessment

32 36% 17 44% 39 66% 3 13% 12 13% 10 23%

Planning with 

Data 

16 18% 18 46% 36 61% 16 70% 13 14% 8 18%

Best Practices 

in ASB

3 3% 1 3% N/A N/A N/A N/A N/A N/A N/A N/A

ASB PQA 

Webinar

N/A N/A N/A N/A 38 64% 14 61% 32 34% 17 39%

Mini PQA Basics N/A N/A N/A N/A N/A N/A N/A N/A 4 4% 0 0%

Mini PQA Basics 

for SelfAssessment

N/A N/A N/A N/A N/A N/A N/A N/A 4 4% 10 23%

PQA SelfAssessment 

Brown Bag

N/A N/A N/A N/A N/A N/A N/A N/A 16 17% 10 23%
76

 Year One Year Two Year Three

Strategy A Strategy B/B3 Strategy A Strategy B/B3 Strategy A Strategy B/B3

Total % Total % Total % Total % Total % Total % 

SEL Workshop: 

Applying 

Growth 

Mindset

N/A N/A N/A N/A N/A N/A N/A N/A 4 4% 6 14%

Planning and 

Reflection

N/A N/A N/A N/A N/A N/A N/A N/A 2 2% 1 2%

Reporting and 

Assessment 

Workshops

23 26% 39 100% 17 29% 1 4% 30 32% 10 23%

AIMSweb 

Reporting

9 10% 8 21% 17 29% 1 4% N/A N/A N/A N/A

AIMSweb 

Student 

Assessment

14 16% 31 79% N/A N/A N/A N/A N/A N/A N/A N/A

FAST Training N/A N/A N/A N/A N/A N/A N/A N/A 15 16% 10 23%

FAST 

Assessment

N/A N/A N/A N/A N/A N/A N/A N/A 15 16% 0 0%

Program 

Evaluation 

Workshops

N/A N/A N/A N/A 15 25% 12 52% N/A N/A N/A N/A 

Program 

Evaluation as a 

Tool 

N/A N/A N/A N/A 13 22% 9 39% N/A N/A N/A N/A

Program 

Evaluation 

Brown Bag

N/A N/A N/A N/A 2 3% 3 13% N/A N/A N/A N/A

Math 

Workshops

N/A N/A N/A N/A 8 14% 5 22% 4 4% 6 14%

Math Summit N/A N/A N/A N/A 8 14% 5 22% N/A N/A N/A N/A

Math 

Networking 

Meeting

N/A N/A N/A N/A N/A N/A N/A N/A 4 4% 6 14%

Organizational 

Management 

Workshops

90 101% 52 133% 89 151% 63 274% 33 35% 34 77%

Program Level 32 36% 27 69% 60 102% 23 100% 29 31% 32 73%

Developing a 

Logic Model

1 1% 0 0% N/A N/A N/A N/A N/A N/A N/A N/A

Cultural 

Competency for 

Management 

15 17% 9 23% N/A N/A N/A N/A N/A N/A N/A N/A

Theory of 

Change 

N/A N/A N/A N/A 16 27% 4 17% N/A N/A N/A N/A

Microsoft Excel

Training

N/A N/A N/A N/A 8 14% 3 13% N/A N/A N/A N/A

Strategic 

Planning for 

Nonprofits 

N/A N/A N/A N/A N/A N/A N/A N/A 3 3% 8 18%
77

 Year One Year Two Year Three

Strategy A Strategy B/B3 Strategy A Strategy B/B3 Strategy A Strategy B/B3

Total % Total % Total % Total % Total % Total % 

Cross Cultural 

Conflict 

N/A N/A N/A N/A N/A N/A N/A N/A 13 14% 16 36%

Tutor Level 45 51% 21 54% 29 49% 40 174% 4 4% 2 5%

Volunteer 

Management 

Brown Bag

8 9% 5 13% 14 24% 9 39% 4 4% 2 5%

Tutor as 

Mentors

10 11% 12 31% N/A N/A N/A N/A N/A N/A N/A N/A

Cultural 

Competency for 

Direct Service 

Staff 

27 30% 4 10% N/A N/A N/A N/A N/A N/A N/A N/A

Quality 

Coaching 

N/A N/A N/A N/A 15 25% 13 57% N/A N/A N/A N/A

Training with 

Intention 

N/A N/A N/A N/A 0 0% 18 78% N/A N/A N/A N/A

Student Level 13 15% 4 10% 0 0% 0 0% 0 0% 0 0%

Youthprise 

Convening 

9 10% 4 10% N/A N/A N/A N/A N/A N/A N/A N/A

Tutoring English 

Language 

Learners

4 4% 0 0% N/A N/A N/A N/A N/A N/A N/A N/A

Family Level 14 16% 11 28% 0 0% 0 0% 0 0% 0 0%

Family 

Engagement

10 11% 5 13% N/A N/A N/A N/A N/A N/A N/A N/A

Parent 

Academy

4 4% 6 15% N/A N/A N/A N/A N/A N/A N/A N/A

Total 89 100% 39 100% 59 100% 23 100% 95 100% 44 100%

Participants 

that attended 

more than one 

workshop

40 45% 24 62% 48 81% 20 87% 78 82% 25 57%
78

Appendix G. Technical Assistance 

Usage per Strategy

 Year One Year Two Year Three

Strategy A 

(n=141 sessions) 

Strategy B/B3

(n=116 sessions) 

Strategy A 

(n=161 sessions) 

Strategy B/B3

(n=90 sessions) 

Strategy A 

(n=140 sessions) 

Strategy B/B3

(n=58 sessions) 

Total

(hours)

Average 

(min) 

Total

(hours)

Average 

(min) 

Total

(hours)

Average 

(min) 

Total

(hours)

Average 

(min) 

Total

(hours)

Average 

(min) 

Total

(hours)

Average 

(min) 

Resource 

Development

25.4 42.4 54.7 71.3 10.2 32.11 6.3 38.00 10.8 37.94 3.5 30.00

Partnership 

Development

7.0 38.2 4.9 42.1 9.3 46.67 7.0 52.50 15.9 63.67 3.3 48.75

Evaluation 

20.8 31.1 14.0 46.7 32.4 55.57 18.8 80.36 14.4 39.31 7.3 48.33

YPQI

21.5 71.7 18.7 93.3 70.0 70.0 45.1 77.29 72.9 89.29 28.7 90.53

Program 

Observation 

1.8 35.0 1.5 90.0 N/A N/A N/A N/A 5.0 150.0 0.0 0.0 

Program 

Development

29.6 59.2 27.8 59.5 32.3 55.43 27.5 71.74 31.2 55.14 17.3 54.74

Other

1.0 20.0 1.8 27.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Total

107.0 45.5 123.3 63.8 154.3 57.5 104.7 69.8 151.2 64.8 60.0 62.1

Source: Tutoring Partnership Technical Assistance Records 2013-2016. The n-value represents the total number of 

TA sessions per strategy and year. 
79

Appendix H. Tutor Training 

Attendance per Strategy

 

Year 1 Year 2 Year 3

Strategy A Strategy B/B3 Strategy A Strategy B/B3 Strategy A Strategy B/B3

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Literacy Tutor 

Training

7

3

1 

30 4% 

1

7

9 

52 29% 

9

2

6 

93 10% 

1

8

3 

43 23% 

8

5

0 

89 10% 

1

8

3 

36 20% 

ABCs of Reading 

(Grades K-5) 10 1% 4 2% 33 4% 15 8% 35 4% 12 7% 

Literacy Skills 

for Early 

Readers (Grades 

Prek-2) 

6 1% 32 18% 29 3% 18 10% 19 2% 16 9% 

Comprehension 

Lessons for Early 

Readers (Grades 

Prek-4) 

5 1% 14 8% 25 3% 9 5% 22 3% 5 3% 

Adolescent 

Literacy Training 9 1% 2 1% 

Middle Grades 

Matter: 

Adolescent 

Literacy Tutor 

Training 

 6 1% 1 1% 13 2% 3 2% 

Math Tutor 

Training 17 2% 9 5% 54 6% 42 23% 20 2% 61 33% 

Math in the 

Middle Grades 

(Grades 6-8) 

8 1% 5 3% 27 3% 21 11% 5 1% 18 10% 

Everyday Math 

Tutor Training 9 1% 4 2% 19 2% 21 11% 

Math in the 

Middle Grades 

2.0 (Grades 6-8) 

 8 1% 0 0% 13 2% 3 2% 

Elementary 

Math (Grades K5) 

 0 0% 34 19% 

Elementary 

Math 2.0 

(Grades K-5) 

 2 0% 6 3% 

General Tutor 

Training 47 6% 39 22% 219 24% 106 58% 284 33% 254 139% 

Tutoring Basics 

(All grades) 22 3% 28 16% 149 16% 43 23% 83 10% 99 54% 

Understanding 

Survival-Based 

Youth (All 

grades)

10 1% 5 3% 12 1% 29 16% 8 1% 10 5% 
80

 

Year 1 Year 2 Year 3

Strategy A Strategy B/B3 Strategy A Strategy B/B3 Strategy A Strategy B/B3

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Total Tutors

n % 

Behavior 

Management 

(All grades)

11 2% 5 3% 36 4% 28 15% 12 1% 84 46% 

Quality Toolkit: 

Active Learning/ 

Advanced Active 

Learning

3 0% 1 1% 8 1% 20 11% 

Quality Toolkit: 

Reframing 

Conflict

1 0% 0 0% 

Tutoring English 

Language 

Learners (All 

grades)

 22 2% 6 3% 31 4% 4 2% 

Advanced 

Planning and 

Reflection

 9 1% 0 0% 

eLearning 

Course: 

Foundation of 

Intercultural 

Competency 

(ongoing 

course)

 133 16% 37 20% 

Total 58 8% 55 31% 203 22% 124 68% 300 35% 331 181% 

Participants that 

attended more 

than one 

training

14 2% 27 15% 88 10% 40 22% 39 5% 67 37% 

 
81

Appendix I. Tutoring Partner SelfAssessment of Best Practices 2015-16

1. Tutoring Partner Self-Assessment of Best Practices

The purpose of this survey is to give you an opportunity to look more deeply into how your program 

implements best practices. We encourage you to have a conversation with your team (if applicable) 

about your responses to provide a richer reflection and a more accurate assessment.

The results for your individual programs will remain confidential; the Foundation will use aggregate 

results to evaluate the Tutoring Partnership and its services to partners.

Please complete this survey reflecting back on the 2015-2016 programming year. See the Best Practices 

Guide for more information on the specific indicators if needed.

1. Please list your name and the name of your program

Program name:

Staff name (staff member

completing the self-assessment):

Position title:

2. Best Practice: Organizational Management

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program's mission statement clearly communicates what the program aspires to 

accomplish.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program is aligned with the supporting organization’s mission.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our programmatic activities are aligned with the organization's strategic plan.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:
82

7. Our tutoring program has a yearly project-specific work plan that accurately reflects program 

goals, activities and responsibilities.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

9. Our organization provides management and staff with opportunities for professional and skill 

development, as well as performance appraisals.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

10. Is this indicator important to you?

Yes No

Comments:

11. Our organization supports the development of management skills for * program leadership.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

12. Is this indicator important to you?

Yes No

Comments:

3. Best Practice: Cultural Proficiency

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program prioritizes cultural proficiency to effectively meet the diverse needs of all 

students.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program prioritizes selecting staff and tutors who are culturally competent.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our tutoring program provides initial cultural competency training.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:
83

7. Our tutoring program provides ongoing training to support cultural proficiency.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

4. Best Practice: Student Recruitment and Management

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program has a clearly defined target group.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program implements a plan to recruit student participants.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our tutoring program has a standard process for enrolling or registering new students.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:

7. Our tutoring program promotes high student attendance and participation throughout the 

year.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

9. Our tutoring program has a student retention plan, including specific goals for participation in 

tutoring

program from year to year.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

10. Is this indicator important to you?

Yes No

Comments:
84

5. Best Practice: Tutor Recruitment and Management

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program implements a tutor recruitment plan.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program establishes a tutor screening policy which includes background and 

reference checks.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our tutoring program selects tutors who are appropriate for the student target groups.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:

7. Our tutoring program has a designated staff member who provides support, guidance and 

feedback to tutors.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

9. Our tutoring program has a tutor retention plan, including specific goals for long-term 

involvement.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

10. Is this indicator important to you?

Yes No

Comments:

11. Our program holds tutor appreciation or recognition events.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

12. Is this indicator important to you?

Yes No

Comments:
85

6. Best Practice: Tutor Training

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program prioritizes tutor training by implementing a comprehensive training plan.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program requires an initial program orientation for every tutor.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our tutoring program requires initial training for every tutor.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:

7. Our tutoring program provides ongoing training and professional development opportunities 

for tutors.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

7. Tutoring Intervention

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Student participants attend tutoring frequently and consistently with a minimum of 90 minutes 

per week.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring interventions are tailored to individual student needs and progress.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:
86

5. Our tutoring program provides a lesson plan or outline for each tutoring session.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:

7. Our program's tutoring interventions are aligned with school district curriculum.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

9. Our tutoring program implements low student-tutor ratios to foster positive relationships.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

10. Is this indicator important to you?

Yes No

Comments:

8. Best Practice: Engagement with Families, Schools, and Communities

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program is committed to partnering with families, schools, and community.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program communicates and engages regularly with families.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our tutoring program communicates and engages regularly with schools.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:
87

7. Our tutoring program communicates and engages regularly with community partners.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

8. Is this indicator important to you?

Yes No

Comments:

9. Best Practice: Evaluation

Tutoring Partner Self-Assessment of Best Practices 2015-2016

1. Our tutoring program promotes a culture of evaluation.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

2. Is this indicator important to you?

Yes No

Comments:

3. Our tutoring program uses evaluation results to continually improve the quality and 

effectiveness of its tutoring. 

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

4. Is this indicator important to you?

Yes No

Comments:

5. Our tutoring program has a logic model that aligns program activities with expected outcomes.

Strongly Disagree Disagree Neutral Agree Strongly Agree

Comments:

6. Is this indicator important to you?

Yes No

Comments:

7. Our tutoring program uses an evaluation plan that clearly outlines how it measures student 

outcomes.

Strongly Disagree Disagree Neutral Agree Strongly Agree 

Comments:

8. Is this indicator important to you?

Yes No

Comments:
88

Appendix J: Cleaning Procedures 

and Sampling Decisions

The figures below describe how the sample size changed throughout the cleaning procedures. For 

Strategy A and B programs located in Saint Paul, 16,211 cases from the student rosters were sent to 

Saint Paul Public Schools research staff. After the matching process, 10,813 cases came back with 

sufficient demographic and assessment data for analysis. Cases were dropped if the grade indicated they 

were preschool students (9,816 remained), and finally, cases were dropped if the grade was between 6-

12th grade. The final sample for Strategy A and B was 7,247 students. 

Figure 14. Flow of Strategy A and Strategy B Tutored Students (Saint Paul)

16,211

Tutored students 

sent to SPPS

10,813

Tutored Students 

matched in SPPS with 

sufficient 

demographic and 

assessment data

9,816

(Dropped records if 

grade was Preschool 

Screening (PS))

Final Sample

7,247 

(Dropped records if 

grade was greater than 

or equal to 6)

A roster of 4,957 cases was sent to Minneapolis Public Schools research staff. After conducting the 

matching procedure, 4,396 cases were returned to Child Trends. Of those 979 had sufficient demographic 

and assessment data for analysis. Ten cases were dropped because they were duplicated records of the 

same student, and students in grades 6-12th grade were also dropped. The final sample for Strategy B3 

in Minneapolis was 811 students. 

Figure 15. Flow of Strategy B3 Tutored Students (Minneapolis)

4,957

Tutored 

students sent to 

MPS

4,396

Tutored 

Students 

matched in MPS

979

Tutored students 

with sufficient 

demographic and 

assessment data

969

(Dropped 

duplicated 

records)

Final Sample

811

(Dropped records 

if grade was 

greater than or 

equal to 6)
89

Appendix K. Assessment Outcome

Data Figures

Figure 16. Strategy A Before and After Tutoring Comparison of Mondo Phonemic Awareness 

Scores per year (Scale 0-52)

25

32

45

32

40 41

52

20

27

31

45 46

n=845 n=108 n=571 n=575 n=675 n=433 n=528 n=1193 n=661 n=1069 n=332 n=539

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=845 Tutoring Group Year 1 Before n=108 Comparison Group

***

***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 17. Strategy A Before and After Tutoring Comparison of Mondo Word Knowledge 

Scores per year (Scale 0-20)

9

11

16

10

17

15

20

6

9

11

16 17

n=1148 n=136 n=929 n=763 n=947 n=716 n=783 n=1380 n=823 n=1352 n=608 n=808

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=1148 Tutoring Group Year 1 Before n=136 Comparison Group

***

***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
90

Figure 18. Strategy A Before and After Tutoring Comparison of Mondo Letter Recognition Fall 

to Spring Gains by year (Scale 0-52)

42 44

52

44

48

50 52

37

41

44

52 52

n=873 n=144 n=610 n=635 n=783 n= 416 n=654 n=1193 n=699 n=1146 n=328 n=510

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=873 Tutoring Group Year 1 Before n=144 Comparison Group

***

**

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 19. Strategy A Before and After Tutoring Comparison of Mondo Letter-Sound 

Correspondence Fall to Spring Gains by year (Scale 0-55)

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
91

Figure 20. Strategy A Before and After Tutoring Comparison of Mondo Print Concepts Fall to 

Spring Gains by year (Scale 0-18)

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 21. Strategy B Before and After Tutoring Comparison of Mondo Phonemic Awareness 

Scores per year (Scale 0-52)

30 32

42 43

30

41 43

20

30 31

44 46

n=57 n=108 n=47 n=575 n=85 n=433 n=78 n=1193 n=69 n=1069 n=51 n=539

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=57 Year 1 Before n=108

***

Tutoring Group Comparison Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
92

Figure 22. Strategy B Before and After Tutoring Comparison of Mondo Word Knowledge 

Scores per year (Scale 0-20)

9

11

16 16

14 15

20

6

9

11

16 17

n=83 n=136 n=68 n=763 n=97 n=716 n=96 n=1380 n=84 n=1352 n=66 n=808

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=83 Year 1 Before n=136

***

Tutoring Group Comparison Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**)

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 23. Strategy B Before and After Tutoring Comparison of Mondo Letter Recognition Fall 

to Spring Gains by year (Scale 0-52)

40

44

51 49

38

50 51

37

40

44

52 52

n=72 n=114 n=49 n=635 n=95 n=416 n=73 n=1193 n=75 n=1146 n=40 n=510

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=72 Year 1 Before n=114

***

Tutoring Group Comparison Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
93

Figure 24. Strategy B Before and After Tutoring Comparison of Mondo Letter-Sound 

Correspondence Fall to Spring Gains by year (Scale 0-55)

23

25

37 38

23

39

33

21

25 26

40

38

n=109 n=170 n=114 n=857 n=132 n=1021 n=130 n=2032 n=162 n=1856 n=121 n=1131

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=109 Year 1 Before n=170

***

Tutoring Group Comparison Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 25. Strategy B Before and After Tutoring Comparison of Mondo Print Concepts Fall to 

Spring Gains by year (Scale 0-18)

12 13

16 16

12

16

17

10

12

13

17 17

n=84 n=142 n=57 n=762 n=94 n=711 n=92 n=1384 n=83 n=1338 n=57 n=743

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=84 Year 1 Before n=142

***

Tutoring Group Comparison Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
94

Figure 26. Strategy B3 Before and After Tutoring Comparison of MAP Reading Assessment

Fall to Spring Gains by year 

163

161

168

167

163

161

169

171

175

169

179

173

n=152 n=225 n=204 n=299 n=133 n=370 n=170 n=379 n=204 n=345 n=128 n=202

Before After Before After Before After

Year 1 Year 2 Year 3

Year 1 Before n=152 Tutoring Group Year 1 Before n=225 Comparison Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
95

Appendix L. Assessment Data 

Figures, Subgroup Analyses

Figure 27. Strategy A Before and After Tutoring Comparison of Mondo Letter Recognition Fall 

to Spring Gains by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-52)

41

51 50

44

37

50 48

42

n=249 n=86 n=120 n=192 n=1629 n=818 n=995 n=1306

Before After Before After

Non-FRL FRL

Non-FRL Before n=249 Non-FRL Before n=86 Comparison Group

*** ***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 28. Strategy A Before and After Comparison of Mondo Letter-Sound Correspondence 

Fall to Spring Gains by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-55) 

Non-FRL

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department. 

22

39 39

32

19

34 35

29

Before n=325 n=141 n=199 n=280 n=2518 n=1507 n=1881 n=2225

Before After Before After

FRL

Non-FRL Before Before n=325 Non-FRL Before n=141 Comparison Group

***

***

Tutoring Group
96

Figure 29. Strategy A Before and After Tutoring Comparison of Mondo Print Concepts Fall to 

Spring Gains by Free/Reduced Price Lunch Status, Years 1-3 combined, (Scale 0-18)

11

16

15

14

10

15 15

13

n=310 n=129 n=196 n=255 n=2024 n=1148 n=1504 n=1672

Before After Before After

Non-FRL FRL

Non-FRL Before n=310 Non-FRL Before n=129 Comparison Group

*** ***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 30. Strategy A Before and After Tutoring Comparison of Mondo Phonemic Awareness 

19

43

40

29

19

42

39

30

n=217 n=67 n=94 n=175 n=1572 n=752 n=945 n=1291

Before After Before After

Non-FRL FRL

Non-FRL Before n=217 Non-FRL Before n=67 Comparison Group

*** ***

Tutoring Group

Scores by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-52)

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
97

Figure 31. Strategy A Before and After Tutoring Comparison of Mondo Word Knowledge 

Scores by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-20)

6

16

14

11

6

15

13

11

n=318 n=132 n=213 n=284 n=2030 n=1151 n=1540 n=1706

Before After Before After

Non-FRL FRL

FRL After n=1540 FRL After n=1706 Comparison Group

***

***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 32. Strategy B Before and After Tutoring Comparison of Mondo Letter Recognition 

Fall to Spring Gains by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-52)

35

48 46

39

33

48 47

40

n=25 n=17 n=15 n=34 n=201 n=168 n=132 n=264

Before After Before After

Non-FRL FRL

Non-FRL Before n=25 Non-FRL Before n=17 Comparison Group

*** ***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
98

Figure 33. Strategy B Before and After Comparison of Mondo Letter-Sound Correspondence 

Fall to Spring Gains by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-55)

20

34

32

38

16

30 30 31

n=42 n=33 n=41 n=76 n=333 n=307 n=294 n=547

Before After Before After

Non-FRL FRL

Non-FRL Before n=42 Non-FRL Before n=33 Comparison Group

***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 34. Strategy B Before and After Tutoring Comparison of Mondo Print Concepts Fall to 

Spring Gains by Free/Reduced Price Lunch Status, Years 1-3 combined, (Scale 0-18)

11

15 15

14

10

15 14

13

n=28 n=20 n=20 n=72 n=215 n=212 n=170 n=463

Before After Before After

Non-FRL FRL

Non-FRL Before n=28 Non-FRL Before n=20 Comparison Group

** ***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
99

Figure 35. Strategy B Before and After Tutoring Comparison of Mondo Phonemic Awareness 

Scores by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-52)

23

38 36

26

22

38 36

28

n=23 n=16 n=17 n=35 n=173 n=166 n=144 n=319

Before After Before After

Non-FRL FRL

Non-FRL Before n=23 Non-FRL Before n=16 Comparison Group

** ***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**)

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 36. Strategy B Before and After Tutoring Comparison of Mondo Word Knowledge 

Scores by Free/Reduced Price Lunch Status, Year 1-3 combined, (Scale 0-20)

7

15 15

11

6

13

12

9

n=31 n=24 n=25 n=84 n=216 n=217 n=188 n=499

Before After Before After

Non-FRL FRL

Non-FRL Before n=31 Non-FRL Before n=24 Comparison Group

***

***

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
100

Figure 37. Strategy B3 Before and After Tutoring Comparison of MAP Reading Assessment by 

Free/Reduced Price Lunch Status, Year 1-3 combined

157

166

169

181

165

163

169

171

n=19 n=27 n=29 n=18 n=452 n=554 n=455 n=503

Before After Before After

Non-FRL FRL

Non-FRL Before n=19 Non-FRL Before n=27 Comparison Group

**

Tutoring Group

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 38. Strategy A Before and After Tutoring Comparison of Mondo Letter Recognition 

Scores by Race/Ethnicity, Years 1-3 Combined, Scale 0-52

37

49

37

50

36

49

39

51

35

48 47

42

49

43

49

42

49

39

47

40

n=716

 n=344

n=607n=291n=240n=128n=266n=115

n=49n=26

n=442

 n=604

n=339n=476n=152n=178n=150n=201

n=32n=39

Black Asian Latino White Native Black Asian Latino White Native

Before After

Before Black n=716 Before Black n=344 Comparison Group

*** ***

Tutoring Group

*** *** **

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
101

Figure 39. Strategy A Before and After Tutoring Comparison of Mondo Letter-Sound 

Correspondence Scores by Race/Ethnicity, Years 1-3 Combined, (Scale 0-55)

20

35

18

34

20

35

20

37

19

35 35

29

34

28

37

27

37

32

34

27

n=1099

n=654n=965n=548n=371n=230n=331n=166

n=77n=50

n=859n=976n=675n=842n=270n=312n=214n=303

n=62n=72

Black Asian Latino White Native Black Asian Latino White Native

Before After

Before Black n=1099 Before Black n=654 Comparison Group

***

***

Tutoring Group

*** *** ***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 40. Strategy A Before and After Comparison of Mondo Print Concepts Scores by

Race/Ethnicity, Years 1-3 Combined, (Scale 0-18)

10

15

9

15

10

16

11

16

10

15 15

13

15

13

15

13

15

14

15

13

n=898n=487n=767n=439n=283n=165n=327n=149

n=59n=37

n=675n=772n=571n=639n=202n=210n=208n=258

n=44n=48

Black Asian Latino White Native Black Asian Latino White Native

Before After

Before Black n=898 Before Black n=487 Comparison Group

*** ***

Tutoring Group

*** *** ***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.
102

Figure 41. Strategy A Before and After Tutoring Comparison of Mondo Phonemic Awareness 

Scores by Race/Ethnicity, Years 1-3 Combined, (Scale 0-52)

20

42

17

41

19

41

21

45

21

46

39

31

38

29

39

31

42

32

46

32

n=662n=309n=650n=306n=216n=102n=220

n=80n=41n=22

n=402n=581n=388n=539n=121n=151n=102n=162

n=26n=33

Black Asian Latino White Native Black Asian Latino White Native

Before Black n=662 Before Black n=309 Comparison Group

***

***

Tutoring Group

*** ***

***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 42. Strategy A Before and After Comparison of Mondo Print Concepts Scores by

Race/Ethnicity, Years 1-3 Combined, (Scale 0-18)

10

14

7

15

13

15

14

15

14

13 13 12

17

13

18

15

n=105n=83 n=79 n=96 n=26 n=30 n=30 n=19 n=77n=215 n=68n=167 n=26 n=70 n=18 n=74

Black Asian Latino White Black Asian Latino White

Before After

Before Black n=105 Before Black n=83 Comparison Group

***

Tutoring Group

***

*** **

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Native American sample was too small to analyze. Data Source: District Research, Evaluation and Assessment 

Department.
103

Figure 43. Strategy A Before and After Tutoring Comparison of Mondo Word Knowledge 

Scores by Race/Ethnicity, Year 1-3 combined, (Scale 0-20)

6

14

6

15

5

14

6

16

6

14

13

10

13

11

13

10

14

11

13

11

n=891n=483n=771n=440n=292n=166n=334n=156

n=60n=38

n=687n=771n=581n=661n=215n=221n=225n=286

n=45n=51

Black Asian Latino White Native Black Asian Latino White Native

Before After

Before Black n=891 Before Black n=483 Comparison Group

*** ***

Tutoring Group

*** *** ***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Data Source: District Research, Evaluation and Assessment Department.

Figure 44. Strategy B Before and After Tutoring Comparison of Mondo Letter Recognition 

Scores by Race/Ethnicity, Years 1-3 Combined, Scale 0-52

36

47

25

49

43

49

44

47 48

42 43

35

52

42

52

44

n=100

n=66n=76n=76n=27n=24n=22n=16n=58

n=132

n=51n=84n=22n=47n=15n=34

Black Asian Latino White Black Asian Latino White

Before After

Before Black n=100 Before Black n=66 Comparison Group

***

Tutoring Group

***

**

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Native American sample was too small to analyze. Data Source: District Research, Evaluation and Assessment 

Department.
104

Figure 45. Strategy B Before and After Tutoring Comparison of Mondo Phonemic Awareness 

Scores by Race/Ethnicity, Year 1-3 combined, (Scale 0-52)

25

37

12

37 38 40

34

39 37

30 29

23

50

33

46

36

n=79 n=61 n=68 n=82 n=24 n=21 n=23 n=15 n=65n=151 n=57n=130 n=22 n=39 n=16 n=32

Black Asian Latino White Black Asian Latino White

Before After

Before Black n=79 Before Black n=61 Comparison Group

***

Tutoring Group

***

**

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Native sample was too small to analyze. Data Source: District Research, Evaluation and Assessment Department.

Figure 46. Strategy B Before and After Tutoring Comparison of Mondo Letter-Sound 

Correspondence Scores by Race/Ethnicity, Years 1-3 Combined, (Scale 0-55)

20

31

9

29

20

29

16

34

32

33

24

28

34

31 30

35

n=164n=138n=123n=122

n=40n=39n=37n=35

n=159n=258

n=94

n=184

n=39n=82n=35n=83

Black Asian Latino White Black Asian Latino White

Before After

Before Black n=164 Before Black n=138 Comparison Group

***

Tutoring Group

 

** 

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Native American sample was too small to analyze. Data Source: District Research, Evaluation and Assessment 

Department.
105

Figure 47. Strategy B Before and After Tutoring Comparison of Mondo Word Knowledge 

Scores by Race/Ethnicity, Year 1-3 combined, (Scale 0-20)

6

13

3

13

11

14

10

15

13

9 9

8

18

9

17

11

n=107

n=84n=80n=99n=27n=32n=30n=23n=89

n=228

n=75

n=184

n=27n=78n=21n=83

Black Asian Latino White Black Asian Latino White

Before After

Before Black n=107 Before Black n=84 Comparison Group

***

Tutoring Group

***

*** ***

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Native sample was too small to analyze. Data Source: District Research, Evaluation and Assessment Department.

Figure 48. Strategy B3 Before and After Tutoring Comparison of MAP Reading Scores by 

Race/Ethnicity, Year 1-3 combined

162

163

166

170 171

161

167

166

170

158

167 168

173

175

170

174

168

178

170

n=299n=347

n=21n=26n=94

n=130

n=42n=59n=15n=19

n=329n=316

n=22n=21n=76

n=111

n=46n=52n=21

Black Asian Latino White Native Black Asian Latino White Native

Before After

Before Black n=299 Tutoring Group Before Black n=347 Comparison Group

*** *

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. After tutoring Native American sample size was too small to analyze. Data Source: District Research, Evaluation and 

Assessment Department.
106

Figure 49. Strategy B3 Before and After Tutoring Comparison of Total Literacy by 

Race/Ethnicity, Year 1-3 combined

-0.10

0.22

-0.54

-0.32

-0.08

0.35

-0.05 -0.12

-0.31

-0.43

0.04

n=139 n=132 n=31 n=31 n=31 n=22 n=76 n=67 n=22 n=23 n=21

Black Latino White Black Latino White

Before After

Before Black n=139 Tutoring Group Before Black n=132 Comparison Group

*

Note: Values are predicted means after adjusting for covariates. (*) indicates differences are statistically significant to at least p < .05, (**) 

p<.01, and (***) p<.0001. Asian, Native American, and after tutoring White samples were too small to analyze. Data Source: District Research, 

Evaluation and Assessment Department.