SANKOFA READING PROGRAM 

YEAR 4 (FINAL) IMPLEMENTATION AND IMPACT 

EVALUATION REPORT 

2016-2017 

Prepared by: 

Chris Desjardins, PhD 

Danielle Dupuis, PhD 

Jason Johnson, MEd
Center for Applied Research and Educational Improvement, University of Minnesota i 

Research, Development and Engagement to Improve Education

Acknowledgements 

The authors wish to acknowledge the leadership and staff at the Network for the Development of 

Children of African Descent (NdCAD) and the Saint Paul Promise Neighborhood (SPPN) 

involved with the Sankofa Reading Program. Thank you to the Department of Research, 

Evaluation, and Assessment at Saint Paul Public Schools and Saint Paul City School for their 

support. Thank you to Dr. Priscilla Gibson, Professor at the University of Minnesota School of 

Social Work for her collaboration on the qualitative study as part of the evaluation. Finally, 

thanks to the children and parents who participated in the Sankofa program and this evaluation.

The Social Innovation Fund (SIF) was a program that received funding from 2010 to 2016 from 

the Corporation for National and Community Service, a federal agency that engages millions of 

Americans in service through its AmeriCorps, Senior Corps, and Volunteer Generation Fund 

programs, and leads the nation’s volunteer and service efforts. Using public and private 

resources to find and grow community-based nonprofits with evidence of results, SIF 

intermediaries received funding to award subgrants that focus on overcoming challenges in 

economic opportunity, healthy futures, and youth development. Although CNCS made its last 

SIF intermediary awards in fiscal year 2016, SIF intermediaries will continue to administer their 

subgrant programs until their federal funding is exhausted. 

This report is based upon work supported by the Corporation for National and Community 

Service, Generation Next and Greater Twin Cities United Way through the Social Innovation 

Fund. Opinions or points of view expressed in this document are those of the authors and do not 

necessarily reflect the official position of, or a position that is endorsed by, the Corporation, 

Generation Next, United Way or the Social Innovation Fund Program. 

How to Cite this Report 

Desjardins, C. D., Dupuis, D., & Johnson, J. (2018). Sankofa Reading Program year 4 (final) 

implementation and impact evaluation report. St. Paul, MN: University of Minnesota, College of 

Education and Human Development, Center for Applied Research and Educational 

Improvement. 

Contact Information

Chris Desjardins, PhD 

Research Associate 

Center for Applied Research and Educational Improvement 

College of Education and Human Development 

University of Minnesota 

1954 Buford Avenue, Suite 425 

Saint Paul, MN 55108 

Email: cdesjard@umn.edu
Center for Applied Research and Educational Improvement, University of Minnesota ii

Table of Contents

Executive Summary ........................................................................................................................ 1

Background ................................................................................................................................. 1

Evaluation Design Overview ...................................................................................................... 1

Prior Research ............................................................................................................................. 1

Targeted Level of Evidence ........................................................................................................ 2

Evaluation Findings .................................................................................................................... 2

Introduction ..................................................................................................................................... 7

Program Description ................................................................................................................... 7

The Sankofa Logic Model .......................................................................................................... 9

Focus of the Evaluation ................................................................................................................ 13

Research Questions ................................................................................................................... 13

Impact Evaluation ..................................................................................................................... 13

Implementation Evaluation ....................................................................................................... 14

Methods......................................................................................................................................... 16

Impact Evaluation ..................................................................................................................... 16

Implementation Evaluation ....................................................................................................... 22

Results ........................................................................................................................................... 24

Impact Evaluation for Years 3 and 4 ........................................................................................ 24

Implementation Evaluation for Year 4 ..................................................................................... 36

Summary of the Findings .............................................................................................................. 47

Summary of the Impact Evaluation for Years 3 and 4.............................................................. 47

Summary of the Implementation Evaluation for Year 4 ........................................................... 49

Interpretations and Limitations ..................................................................................................... 51

Conclusions and Recommendations ............................................................................................. 55

Impact Evaluation for Years 3 and 4 ........................................................................................ 55

Implementation Evaluation for Year 4 ..................................................................................... 56

 
Center for Applied Research and Educational Improvement, University of Minnesota iii

List of Tables 

Table 1. Description of the Direct Link between Sankofa Reading Program Elements and 

Student Achievement and Supporting Research .............................................................11 

Table 2. Comparison of Sankofa Students’ Pre- and Post-Program DRA II Scores with SPPS 

Soft Target Reading Goals for Years 1 through 4 ..........................................................25

Table 3. Number of Sankofa Students by Pre- and Post-Program SPPS Soft Target Outcomes for

Years 1 through 4............................................................................................................26

Table 4. Summary information about the ITS design by measure ...............................................29

Table 5. Means and Standard Deviations of MCA-Reading Proficiency and Scale Scores of 

Sankofa and Comparison Group Students......................................................................32

Table 6. Qualifications of Sankofa Tutors ....................................................................................37

Table 7. Summary of observation checklists across the observed cohorts. ..................................39

Table 8. Enrollment and Completion Rates by Cohort Location..................................................40 

Table 9. Elementary Schools of Sankofa Participants for Years 1 through 4...............................43

Table 10. Demographics of Sankofa Participants in all years .......................................................45

List of Figures 

Figure 1. Map of the Saint Paul Promise Neighborhood in Saint Paul, Minnesota ........................8

Figure 2. Sankofa symbol ...............................................................................................................9

Figure 3. Sankofa logic model ......................................................................................................10

Figure 4. Potential effects of Sankofa on reading ability..............................................................21

Figure 5. Daily school attendance against weeks since the start of Sankofa. ...............................28

Figure 6. Percent of data missing by weeks since the start of Sankofa by assessment. ...............29 

Figure 7. Distribution of MCA scale scores for Sankofa participants in Years 3 and 4...............33

List of Appendices 

Appendix A. Assessments. ...........................................................................................................60

Appendix B. Propensity Score Matching for Years 3 and 4.........................................................64

Appendix C. Soft-Target Goals for DRA .....................................................................................67

Appendix D. Interrupted Time Series Regression Tables. ...........................................................68

Appendix E. Parent Survey ...........................................................................................................75 

Appendix F. Example Lesson Plan ...............................................................................................77

Appendix G. Sankofa Observation Checklist ...............................................................................78
Center for Applied Research and Educational Improvement, University of Minnesota iv

About this Report 

This evaluation report is a Final Report for the Sankofa Reading Program, and is 

intended to fulfill the SIF requirements to determine at least a moderate level of 

evidence for funded projects. It includes implementation and impact studies focused 

on students who participated in the program in 2016-2017. 
Center for Applied Research and Educational Improvement, University of Minnesota 1 

Executive Summary 

Background 

In September 2012, the Corporation for National & Community Service (CNCS), a federal 

agency, through its Social Innovation Fund (SIF), awarded the Greater Twin Cities United Way 

(GTCUW) and the Twin Cities STRIVE Alliance (now called Generation Next) funding to serve 

as a grant making intermediary for projects to address youth development, specifically, closing 

the achievement gap in the Twin Cities area of Minneapolis and Saint Paul. Six organizations 

were selected as subgrantees, and the Center for Applied Research and Educational Improvement 

(CAREI) at the University of Minnesota was contracted to provide external evaluation services. 

Subsequently, the Saint Paul Promise Neighborhood (SPPN), a community initiative sponsored 

by the Amherst H. Wilder Foundation, was awarded subgrantee funding from GTCUW and 

Generation Next to partner with NdCAD to implement the Sankofa reading program at its 

partner schools. 

The Sankofa reading program is a culturally-based literacy tutoring program for children of 

African descent provided by the Network for the Development of Children of African Descent 

(NdCAD), a non-profit family education center in Saint Paul, Minnesota. The program consists 

of intensive after-school tutoring that includes strong reading and cultural components. The 

specific goals are to increase reading skills, increase independent guided reading levels, and 

improve academic performance. SPPN partnered with NdCAD to deliver Sankofa to low 

performing African and African-descended readers that resided in its geographical area and that 

were enrolled in kindergarten through third grade from 2013-2017. 

Evaluation Design Overview 

This evaluation included an implementation evaluation and an impact evaluation of preliminary 

and moderate evidence. Rigorous assessment of implementation fidelity was carried out to 

ensure that the core components of Sankofa took place. The preliminary evidence evaluation 

utilized a one-group pre-post design, and descriptive analyses were carried out on pre- and postprogram performance of participants on a standardized reading assessment. The designs for 

moderate evidence utilized an interrupted time series (ITS) design and matched comparison to 

assess the effect of Sankofa on reading ability using progress monitoring assessments developed 

by FastBridge Learning and a statewide standardized reading assessment. 

Prior Research 

Educational researchers who have investigated the impact of culturally relevant pedagogy have 

hypothesized that when children form a strong cultural identity and understand their 

community’s cultural heritage, values, and contributions, they become intrinsically motivated to 

succeed (e.g., Barnes, 1991; Grills, Cooke, Douglas, Subica, Villanueva, & Hudson, 2015; 

Ladson-Billings, 1995). A basic belief underlying NdCAD programming is that reading is a 
Center for Applied Research and Educational Improvement, University of Minnesota 2 

pragmatic, navigational tool for learning about one’s roots and where one is going.

Consequently, an overarching goal of the Sankofa program is to help children of African descent 

develop and build literacy skills within the context of their cultural heritage, thus giving them a 

firm foundation for academic achievement and life-long learning. 

The links between Sankofa program elements and increased student achievement are supported 

by the findings of numerous research studies on successful learning in urban schools. Critical 

factors identified in previous research include careful monitoring of student progress, frequent 

communication with parents, a strong academic focus, a learning environment that is attractive 

and safe, dedicated and caring teachers, extensive scaffolding, an emphasis on understanding, 

and intentional efforts to motivate students (e.g., Pressley, Raphael, Gallagher, & DiBella, 2004). 

Similarly, investigations of effective primary-grade reading instruction in low-income schools 

have identified school factors and teacher factors that have a positive impact on the reading 

achievement of K – 3 students (e.g., Taylor, Pearson, Clark & Walpole, 2000). School factors 

included strong links to parents, systematic assessment of student progress, and a high level of 

school communication and collaboration. Teacher factors included small-group instruction, 

independent reading, student on-task behavior, home communication, and instructions for 

applying phonics to everyday reading. 

In 2010, NdCAD worked with GrayHall, independent evaluation consultants, to conduct an 

evaluation of the Sankofa reading program (Hall & Gray, 2010). Students’ reading levels were 

assessed pre- and post-intervention using the Developmental Reading Assessment: Second 

Edition (DRA II). The evaluation found that 75% of the 333 students who completed Sankofa in 

the period 2004 through 2010 increased their reading levels, with half of the students improved 

by two or more levels. 

Targeted Level of Evidence 

A previous evaluation of Sankofa provided preliminary evidence of program impact. The 

existing evaluation of the program (Hall & Gray, 2010) used a one-group pre-post design and 

found that program participants improved in reading performance as measured by level changes 

on the DRA II. The evaluation reported here was designed to obtain a moderate level of evidence 

of program impact employing a matched-comparison and ITS design. Strong evidence was not 

targeted because randomized controlled trials or other rigorous experimental designs were not 

feasible at this time, given the resources available for the evaluation, and the time frame of the 

grant. 

Evaluation Findings 

Implementation Evaluation Questions for Year 4. The following implementation evaluation 

questions were addressed in Year 4, school year 2016 - 2017: 

1. Is the Sankofa program being implemented as intended? 

2. Is the Sankofa program delivery consistent with the intended program structure? 

3. Has the Sankofa program achieved its intended implementation output and goals? 
Center for Applied Research and Educational Improvement, University of Minnesota 3 

4. What program services are received by the comparison groups? 

5. What are reasons why participants do not complete Sankofa? 

6. What is the average number of follow-ups (by NdCAD tutors and/or SPPN School 

Partnership Coordinators) before participants discontinue the program? 

Implementation Findings for Year 4. The target population of the Sankofa reading program 

was made up of kindergarten through third-grade African or children of African-descent enrolled 

in elementary schools or living in the SPPN boundaries with low reading scores based on the 

Measures of Academic Progress (Northwest Evaluation Association, 2018) or Mondo Bookshop 

Assessment. The implementation evaluation found that all of the student participants lived in the 

SPPN, identified as African American, and 88% of student participants were enrolled in an 

elementary school located in the SPPN. 

For Year 4 of the Sankofa program, the participation goal was an enrollment of 144 K – 3

students. The enrollment of SIF-supported students was 143, which was 99% of the target. 

Across the four years of the evaulation, 311 participants enrolled in Sankofa.

The program completion goal was for 80% of student participants to complete the program in 

Year 4. The actual completion rate of 83% was above this goal. This is a large increase from 

Year 3, where the completion rate was only 37%. The program structure presented in the 

Sankofa logic model (see Figure 3) states that the program offers each cohort 40.5 hours of 

tutoring. The implementation evaluation found that actual programming ranged from 39 to 40.5 

hours with individual student dosage ranging from 9.1 to 35.6 hours. In Year 4, no student 

received the full 40.5 hours of dosage. The student-tutor ratio ranged from 3:1 to 8:1 (with a goal 

of 5:1). 

The Sankofa reading program was designed to have a strong emphasis on African culture. The 

implementation analysis on program fidelity found that all Sankofa tutors were African or of 

African descent and all the lesson plans included African cultural teachings, rituals, and 

activities. 

An examination of the tutors’ lesson plans found that all plans included content specified in the 

logic model. More specifically, in addition to culturally related activities, all plans included 

academic activities, cognitive skill building, individual and small group tutoring, and group 

dialogue and interactions. This was further corroborated by observations made by the NdCAD 

staff. 

The parent engagement goal for Year 4 was for 100% of Sankofa students’ parents to receive 

weekly communications. This goal was completely met. One-thousand, three-hundred and five 

notes were sent to parents of children attending cohorts in Year 4. This resulted in an average of 

nine notes per Sankofa participant. This is right on the target of one note per child per week or a 

total of nine communications during Sankofa. Therefore, parent communication appeared to be 

implemented with a high degree of fidelity. 
Center for Applied Research and Educational Improvement, University of Minnesota 4 

Based on the implementation evaluation, the following recommendations are provided: 

1. Consider developing new, or utilizing a diversity of, strategies to keep attendance high. 

Because no participant received the full dosage of 40.5 hours, this likely reduced the 

impact of Sankofa on its participants. If the full dosage of 40.5 hours is not necessary, 

then modifying the logic model is recommended. 

2. Given the range of reading abilities of participants entering Sankofa, additional tutor 

trainings that focus on working with readers with varied skills might be beneficial. 

3. As the number of participants that Sankofa serves that are homeless or highly mobile 

increases, it might be useful to develop a tutoring training module focused on supporting 

students facing housing mobility, as they may have special reading, social, emotional, 

and behavioral needs. 

Impact Evaluation Questions for Years 3 and 4. The impact evaluation for Years 3 and 4, 

school years 2015 – 2016 and 2016 – 2017, were separated into confirmatory and exploratory 

impact questions. The following confirmatory questions were addressed in Years 3 and 4: 

1. Do children who participate in Sankofa exhibit increased reading performance as 

identified by pre-post program scores on the Developmental Reading Assessment 

(DRA)? 

2. Do children who participate in Sankofa exhibit greater levels of social emotional 

development such as confidence in reading after the nine-week program? 

3. Do children who participate in Sankofa exhibit greater rates of literacy growth after 

completing Sankofa, compared with their growth before program participation, according 

to FastBridge Learning reading measures? 

The following exploratory questions were addressed in Years 3 and 4: 

1. How do children who complete Sankofa perform on third-grade reading proficiency 

measures? 

2. Do children who complete Sankofa perform better on third-grade reading proficiency 

measures than similar children in Saint Paul Public Schools? 

3. Do African-centered and African-theme texts change participants’ attitudes about 

reading? 

4. How does having (African/African American) tutors change parent involvement in 

improving child’s literacy?
Center for Applied Research and Educational Improvement, University of Minnesota 5 

The confirmatory evaluation included a pre/post design without a comparison group using the 

DRA II and an ITS design using measures developed by FastBridge Learning, where students 

were measured weekly for six weeks prior to Sankofa and six weeks immediately following 

Sankofa. The DRA II design was capable of providing preliminary evidence, while the ITS was 

capable of providing a moderate level of evidence (the evidence targeted by the SIF) provided 

internal validity was high. In addition, a parent survey was administered to provide preliminary 

evidence about changes in Sankofa participants’ behaviors.

Impact Findings for Years 3 and 4. 

The confirmatory impact evaluation for Years 3 and 4 found the following: 

 Evidence of an effect of Sankofa using a pre/post design with the DRA II as the outcome 

o 90% of Sankofa participants increased by at least one reading level (preliminary 

level of evidence design). 

 Parents reported an increase in positive behavior or attitude in their children 

o 80% of parents reported that their children had stronger connections to their 

family and community since participating in Sankofa.

 For the ITS design, evidence of an effect of Sankofa based on FastBridge Learning’s

Decodable Words assessment such that the expected growth for Sankofa participants was 

0.978 words read correctly per minute more per week than prior to Sankofa (moderate 

level of evidence design). However, this effect was not significant after correcting for 

multiple comparisons and it was not corroborated by the pooled multiple imputation 

analysis. 

 No evidence of an effect of Sankofa using an ITS design with FastBridge Learning’s 

Letter Words and Sight Words assessments (moderate level of evidence design). 

The exploratory impact evaluation included a matched comparison with the MCA reading as the 

outcome and a qualitative study. Unlike the confirmatory evaluation questions, these questions 

were intended to inform research not policy and there were no direct links between these 

questions and the Sankofa logic model. 

The exploratory impact evaluation for Years 3 and 4 found the following: 

 On the MCA, 27% of Sankofa participants partially met MCA proficiency and 5% met 

MCA proficiency. Sankofa participants ranged from the 1st to the 68th percentile on the 

MCA scale score. 

 A statistically significant effect on the MCA found that Sankofa participants were 

expected to score 7.25 points lower on the MCA than the non-Sankofa Saint Paul Public 

School (SPPS) comparison students. 

o Given that it was unclear that Sankofa and comparison students were balanced on 

a valid prior achievement measure, there may be limited statistical validity to this 

finding.

o This finding was not corroborated by the unmatched analysis. 
Center for Applied Research and Educational Improvement, University of Minnesota 6 

The qualitative study results showed that parents’ and staff’s perceptions were positive with 

regards to the African-centered approach used by NdCAD; that parents found it meaningful to 

read about Africans and one’s culture; and that parents expressed a range of opinions regarding 

the importance of reading African-centered text relative to texts on other races. Finally, parents 

expressed mixed views regarding who the best tutor for their child was. 

Based on the impact evaluation for Years 3 and 4, the following recommendations are 

provided: 

1. Continue to address logistical barriers that hinder data collection that make the use of 

an ITS design difficult. The ITS approach was welcomed by NdCAD and SPPN staff 

and the ITS design, when combined with robust data collection and a valid outcome 

measure (such as the FBL measures), represents the best opportunity to observe a 

moderate-level of evidence. 

2. NdCAD staff may consider using the FBL measures in lieu of the DRA II to track 

progress before, during, and/or after Sankofa. While this approach would not 

necessarily need to be as involved as the ITS design utilized for the SIF, it would 

allow NdCAD to collect data that goes beyond the DRA II and would allow them to 

measure growth. Using the FBL measures during Sankofa could allow NdCAD to 

understand for which participants Sankofa is having the strongest effects and which 

participants continue to struggle. This, when coupled with individualized learning 

plans, could make the impact of Sankofa larger. 

3. If the ITS proves to be too logistically difficult to implement, future considerations 

could involve regression discontinuity, where if prospective participants fall below a 

certain reading threshold they are assigned to Sankofa and if they are above it they 

are not. However, this would involve working closely with SPPS and Saint Paul City 

School to ensure that this approach would be feasible and that the outcome measure 

used to estimate the treatment effect is appropriate and available. 

4. The qualitative study provided the following recommendations for Sankofa regarding 

the two exploratory questions: 

a. Allow Sankofa staff to conduct all aspects of recruiting, screening, and enrolling 

children into Sankofa. Parents may be getting mixed information about it. 

b. Continue using the African-centered approach. Parents appreciate that their 

children were learning about their race, history, and culture. The African-centered 

approach is working well for the majority of the parents. 

c. Consider allowing tutors to make home visits. The benefits may result from tutors 

having a built-in contact with a parent(s). 

d. Conduct an annual evaluation of all aspects of Sankofa and develop a method to 

share the findings with parents. 

e. Develop a strategy for former students/graduates of Sankofa and their parents to 

remain connected to the program. 
Center for Applied Research and Educational Improvement, University of Minnesota 7 

Introduction 

Program Description 

In September 2012, the Corporation for National & Community Service (CNCS), a federal 

agency, through its Social Innovation Fund (SIF), awarded the Greater Twin Cities United Way 

(GTCUW) and the Twin Cities STRIVE Alliance (now called Generation Next) funding to serve 

as a grant making intermediary for projects to address youth development, specifically, closing 

the achievement gap in the Twin Cities area of Minneapolis and Saint Paul. Six organizations 

were selected as subgrantees, and the Center for Applied Research and Educational Improvement 

(CAREI) at the University of Minnesota was contracted to provide external evaluation services. 

Subsequently, the Saint Paul Promise Neighborhood (SPPN), a community initiative sponsored 

by the Amherst H. Wilder Foundation, was awarded subgrantee funding from GTCUW and 

Generation Next to partner with NdCAD to implement the Sankofa reading program at its 

partner schools. 

Sankofa is a culturally based literacy tutoring program for children of African descent provided 

by NdCAD, a non-profit family education center in Saint Paul, Minnesota. The center offers a 

variety of literacy and cultural enrichment programs for children, youth, and adults residing in 

the Twin Cities metropolitan area (Figure 1). Through the SIF award, SPPN partnered with 

NdCAD to deliver the Sankofa Reading Program. This evaluation report focuses on 

implementation and impact information from the third and fourth years of the SIF grant. The 

subgrant period reported here covers school years 2013-20171but focused on school years 2015 - 

2017.

 

Figure 1. Map of the Saint Paul Promise Neighborhood in Saint Paul, Minnesota. 

1 For the interrupted time series design, data were also included for school year 2017 – 2018. 
Center for Applied Research and Educational Improvement, University of Minnesota 8 

African culture is at the core of NdCAD’s mission: “We exist to strengthen the cultural 

connections within communities of African descent that promote, sustain, and enhance the 

healthy development of our children.” 

The Sankofa reading program provides intensive and short-term after-school tutoring that 

includes strong reading and cultural components. The programming for the SIF targeted African 

or students of African-descent with the lowest reading scores, using the Measures of Academic 

Progress (Northwest Evaluation Association, 2018) or Mondo Bookshop Assessment, that were 

enrolled in elementary schools or lived in the SPPN boundaries. The specific goals were to 

increase reading skills, increase independent guided reading levels, and improve academic 

performance. 

Educational researchers who have investigated culturally relevant pedagogy have hypothesized 

that when children form a strong cultural identity and understand their community’s cultural 

heritage, values, and contributions, they become intrinsically motivated to succeed (e.g., Barnes, 

1991; Grills, Cooke, Douglas, Subica, Villanueva, & Hudson, 2015; Ladson-Billings, 1995). A 

basic belief underlying NdCAD programming is that reading is a pragmatic, navigational tool for 

learning about one’s roots and where one is going. Consequently, an overarching goal of the 

Sankofa program is to help children of African descent develop and build literacy skills within 

the context of their cultural heritage, thus giving them a firm foundation for academic 

achievement and life-long learning. The name, Sankofa, was selected for NdCAD’s reading 

intervention because of its significance in African culture. Sankofa is an Adinkra symbol of the 

Akan people of West Africa. It symbolizes the importance of African people going back to their 

roots in order to move forward. Sankofa is represented by a mythical bird that flies forward 

while looking back with an egg (symbolizing the future) in its mouth (see Figure 2). 

Figure 2. Sankofa symbol.

A unique feature of the Sankofa reading program is the integration of culturally responsive and 

culturally specific practices to holistically address students’ learning needs and to engage the 

students in their own learning. To this end, the program employs a variety of cultural learning 

strategies to help students make critical connections between what they know about their cultural 

selves, their cognitive and skill development, and their academic achievement. For example, the 

opening and closing rituals of each reading intervention session incorporate reciting the seven 
Center for Applied Research and Educational Improvement, University of Minnesota 9 

principles of Nguzo Saba. Culturally based call and response affirmations are used to help 

students become more present to African intellectual traditions and ancestors. Reading materials 

feature Africans or African Americans. All Sankofa tutors are from the cultural community of 

the students. These and other culturally based program practices help students see important 

relationships between themselves as learners and as persons of African descent. 

The Sankofa reading intervention is scheduled for a nine-week period with three 90-minute 

sessions per week, providing a total of 40.5 hours of tutoring. The tutoring includes a 

combination of individual and small group lessons. The student-to-tutor ratio is 5:1. The program 

has a pool of tutors who have extensive experience working with children and families from the 

primary cultural community of the students served and who receive training in the 

implementation requirements of Sankofa. The tutoring program utilizes a combination of 

methods that are carried out within five interrelated program design elements. The design 

elements are: a) assessment of independent guided reading levels, alphabet knowledge, and 

reading skills across the five dimensions of reading; b) diagnosis of skill gaps and instructional 

needs, and development of individual learning plans; c) use of culturally congruent leveled texts 

from a wide genre; d) guided reading lessons in a reading workshop model; and e) integration of 

culturally responsive ritual, routines, and practices to intrinsically motivate students to read and 

learn. 

The Sankofa Logic Model 

The logic model for Sankofa is displayed in Figure 3. The logic model includes resources/inputs, 

activities/operations, outputs/goals, short-term outcomes, and long-term impacts. Table 1 

presents a description of the direct link between program elements and increased student 

achievement, including research that supports program elements. 

Resources/inputs. The resources/inputs include the following program characteristics: 

Culturally relevant pedagogy and critical inquiry/investigation, a balanced literacy approach, an 

academic focus, high expectations, positive youth/adult connections, highly qualified/trained 

tutors who reflect the students’ cultural background, a low student-to-tutor ratio, and a program 

aligned with the organization’s mission. Resources/inputs also include the approach of 

countering self-doubt, deconstructing myths, and parental/community engagement. Instructional 

methods utilized in Sankofa are consistent with the mission and include counter-narratives and 

exposure to African knowledge, history, heritage, and literature.

 

Activities/operations. The activities/operations component of the logic model describes 

the program content, delivery, and structure. Program content and delivery are comprised of 

cultural teachings, rituals, and activities; academic activities; cognitive skill building; Africancentered, leveled texts; instructional links with school day/standards; individual and small group 

tutoring; group dialogue, interactions, storytelling, and free books. Program structure consists of 

40.5 hours of tutoring, three 90-minute tutoring sessions per week for 9 weeks, individual and 

small group instruction, pre/post assessment, individual learning, and a student-to-tutor ratio of 

5:1. 
Center for Applied Research and Educational Improvement, University of Minnesota 10

Outputs/goals. Outputs/goals are established for student participation and student 

performance. The participation goal for Years 1 and 2 were for 120 kindergarten to third-grade 

students to enroll in Sankofa with an 80% or greater completion rate. The student performance 

goal was for at least 80% of participating students to improve their program assessment scores. 

For Year 3 the participation goal was for 90 students to enroll in Sankofa with an 80% or greater 

completion rate. For Year 4 the participation goal was for 144 students to enroll in Sankofa with 

an 80% or greater completion rate. Of these 144 students, the goal was to have 87 students 

participate in the interrupted time series (ITS) design that was implemented in Year 4. Across the 

four years, 311 participants enrolled in Sankofa. 

Academic/cognitive short-term outcomes. The academic/cognitive short-term 

outcomes address two broad areas, academic/cognitive and social/cultural/emotional. 

Academic/cognitive outcomes concern reading levels, reading skills, homework completion, 

study skills, independent reading, classroom behavior, school attendance, attitude toward 

reading. Social/cultural/emotional outcomes concern confidence in reading and learning, peer 

interactions, problem-solving, knowledge and respect for culture and heritage, and connections 

with family and community. 

Long-term outcomes. The long-term outcomes are outcomes that the program is 

designed to impact throughout the students’ education and beyond. The long-term 

academic/cognitive outcomes are higher grades and test scores, on-time promotion, return to 

regular track for special education students, high school graduation, postsecondary plans, and 

increased parent involvement in schools. The social/cultural/emotional long-term outcomes are 

increased self-efficacy and confidence in learning, increased critical thinking and problemsolving, increased knowledge of self and respect for one’s culture and heritage, stronger 

connections with family and community, and higher aspirations and goal setting. 
Center for Applied Research and Educational Improvement, University of Minnesota 10

Figure 3. Sankofa logic model.
Center for Applied Research and Educational Improvement, University of Minnesota 11

Table 1 

Description of the Direct Link between Sankofa Reading Program Elements and Student 

Achievement and Supporting Research. 

Key 

Program Elements Link to Student Achievement Research Literature 

Reading skill development, 

assessment, and curriculum: 

● Phonogram cards 

● Phonics activities 

● Culturally congruent leveled 

texts 

● Comprehension strategies 

● Self-selected free books for 

building a home library 

● Diagnosis of skill gaps 

informs instruction 

● Individualized learning plans 

are developed and learning 

goals set 

● Mastery and use of automatic 

sound/letter correspondence 

with 70 phonograms 

● Practice phoneme 

manipulation 

● Practice word formation and 

independent reading 

● Progress is monitored 

● Key components of a 

successful reading program 

include phonological 

awareness, phonics, cognitive 

strategies, and considerable 

independent reading (McEwan, 

2002)

● Reading assessment and 

instruction work together to 

benefit students (Afterbath, 

2012) 

● Student interest impacts 

reading comprehension (Asher, 

1979)

Culturally relevant pedagogy: 

● Culturally specific learning 

strategies 

● Cultural rituals/routines 

● Culturally specific texts 

● Counter-narratives 

● Tutors reflective of students’ 

cultural community 

● Tutors and parents work 

together as community 

educators to develop youth 

● Provides real world contextual 

learning, application, and 

relevance 

● Builds self-efficacy and 

motivation as well as prosocial personal and cultural 

identity to combat self-doubt 

● Community and cultural 

connections support 

educational success 

● Tutors and parents support 

student learning and 

communicate high 

expectations 

● A sociocultural approach 

supports literacy develop- mint 

and learning (Barnes, 1991; 

Compton-Lilly, 2009; Harbor, 

2012; Hassett, 2008; Lozenski 

& Ford, 2014; Matsuda, 1995; 

Richardson & Eccles, 2007) 

Instructional method links with 

school day/standards: 

● Guided reading/reading 

workshop 

● Balanced literacy approach 

● Instructional scaffolding 

● Learn and apply seven 

comprehension strategies 

● Scaffolding promotes 

independent, self-directed 

reading and comprehension 

● Demands are matched to 

student level and expectations 

are clear 

● Scaffolding instruction 

supports learning (Lauer, 

Akiba, Wilderson, Apthorp, 

Snow, & Martin-Glenn, 2006) 

● A balanced approach that 

includes leveled books and 

leveled phonics and trained 

instructors is effective for 

struggling readers (Morris, 

2015) 

Frequent and regular tutoring 

sessions 

● Three 90-minute sessions per 

week for 9 weeks (40.5 hours 

of instruction) 

● Length of a reading 

intervention is positively 

related to reading outcomes 

(Elbaum, Vaughn, Hughes, & 

Moody, 2000) 
Center for Applied Research and Educational Improvement, University of Minnesota 12

The links between program elements and increased student achievement are supported by the 

findings of numerous research studies on successful learning in urban schools. One such study 

was carried out by Pressley, Raphael, Gallagher, and DiBella (2004) who conducted a qualitative 

investigation of a high achieving K-12 school serving urban, African American students. 

Pressley and his colleagues identified several critical factors that contributed to the school’s 

effectiveness. These factors included careful monitoring of student progress, frequent 

communication with parents, a strong academic focus, a learning environment that is attractive 

and safe, dedicated and caring teachers, extensive scaffolding, an emphasis on understanding, 

and intentional efforts to motivate students. 

Investigations of effective primary-grade reading instruction have identified similar factors that 

are essential for high levels of reading achievement in low-income schools. Taylor, Pearson, 

Clark, and Walpole (2000) found several school factors and several teacher factors that had a 

positive impact on the reading achievement of K through 3 students. School factors included 

strong links to parents, systematic assessment of student progress, and a high level of school 

communication and collaboration. Teacher factors included small-group instruction, independent 

reading, student on-task behavior, home communication, and instructions for applying phonics to 

everyday reading. 

The culturally relevant pedagogical elements are supported by the work of Gloria LadsonBillings who has investigated culturally responsive instruction and successful learning of African 

American students (Ladson-Billings, 1995, 2000, 2011, 2012). Ladson-Billings’ theory of 

culturally relevant pedagogy emphasizes the importance of “. . . a teacher who believes deeply 

in the intellectual capability of the student and his or her own efficacious abilities” (LadsonBillings, 2012, p. 118). Accordingly, the Sankofa model includes student outcomes such as 

increased reading performance, improved classroom behavior and participation, and enhanced 

self-confidence. 

In 2010, NdCAD worked with GrayHall, independent evaluation consultants, to conduct an 

evaluation of the literacy outcomes of students participating in Sankofa (Hall & Gray, 2010). 

Students’ reading levels were assessed pre- and post-intervention using the Developmental 

Reading Assessment (DRA II). Over the study period, 38 Sankofa cohorts were held at 28 

locations in Minneapolis and Saint Paul; students from 31 schools (15 in St. Paul and 16 in 

Minneapolis) participated. The evaluation found that 75% of the 333 students who completed 

Sankofa in the period 2004 through 2010 increased their reading levels, with half of the students 

improving by two or more levels. 
Center for Applied Research and Educational Improvement, University of Minnesota 13

Focus of the Evaluation 

This report presents an evaluation of NdCAD’s Sankofa program implemented in the SPPN from 

spring 2013 through fall 2017, with a particular focus on the third and fourth years, school years 

2015 – 2016 and 2016 – 2017, of the SIF. 

Research Questions

This evaluation was designed to answer questions about Sankofa’s impact and implementation, 

based on the logic model presented in Figure 3. For both the impact and implementation 

evaluation, confirmatory and exploratory questions were co-developed with the SPPN and 

NdCAD staff. 

Impact Evaluation

Confirmatory. The confirmatory questions were the primary research questions of our 

evaluation. They were developed based on the logic model and they were intended to inform 

policy (Corporation for National & Community Service, 2013). Based on initial power analyses, 

we were statistically powered to answer them and they contained fewer validity concerns. There 

were four confirmatory impact questions proposed in the original and revised SEPs. They were: 

1. Do children who participate in Sankofa exhibit increased reading performance as 

identified by pre-post program scores on the Developmental Reading Assessment 

(DRA)? 

2. Do children who participate in Sankofa exhibit greater levels of social emotional 

development such as confidence in reading after the nine-week program? 

3. Do children who participate in Sankofa exhibit greater reading proficiency as identified 

by Mondo Bookshop Assessments after the nine-week program than similar children who 

do not participate in any literacy development program or similar children who 

participate in a non-culturally-specific literacy development program? 

4. Do children who participate in Sankofa exhibit greater rates of literacy growth after 

completing Sankofa, compared with their growth before program participation, according 

to FastBridge Learning reading measures? 

Question 1 was examined each year; question 3 was addressed only in Years 1 and 2; question 4 

was addressed in Year 4; and finally, question 2 was addressed in Years 1, 2, and Year 4. When 

the SEP was revised and approved on March 3, 2017, a new quasi-experimental design was 

implemented. This design was used in place of the matched comparison that used the Mondo and 

proposed using an interrupted time series (ITS) design with reading measures developed by 

FastBridge Learning (FBL) (see Methods section). Question 3 was only partially addressed as a 

critical, original partner in the evaluation was unable to share information about the specific 

reading interventions students were receiving as well as students’ reading assessment data.

Therefore, a non-culturally-specific literacy comparison group could not be created. 
Center for Applied Research and Educational Improvement, University of Minnesota 14

Exploratory. The exploratory questions were not intended to inform policy but instead 

future research (Corporation for National & Community Service, 2013). They addressed longerterm outcomes that were implied by the logic model but that lacked any strong a priori evidence. 

These questions had less statistical certainty and contained many more validity concerns. These 

questions could result in future evidence that could inform practice or research but not 

necessarily (see Interpretations and Limitations section). There were five exploratory impact 

questions proposed in the original and revised SEPs. They were: 

1. How do children who complete Sankofa perform on third-grade reading proficiency 

measures? 

2. Do children who complete Sankofa perform better on third-grade reading proficiency 

measures than similar children who do not participate in any literacy development 

program or similar children who participate in a non-culturally-specific literacy 

development program? 

3. Do children who complete Sankofa perform better on third-grade reading proficiency 

measures than similar children in St. Paul Public Schools? 

4. Do African-centered and African-theme texts change participants’ attitudes about 

reading? 

5. How does having (African/African American) tutors change parent involvement in 

improving child’s literacy?

Questions 1 and 3 were examined for all four years; questions 4 and 5 were addressed only in 

Year 4; and question 2 was never addressed. Questions 4 and 5 were added after the SEP was 

revised and could only be examined in Year 4. Question 2 was never addressed as a critical, 

original partner in the evaluation was unable to share information about specific reading 

interventions students were receiving and students’ reading assessment data. Therefore, we were 

unable to develop comparable control groups. 

Implementation Evaluation 

There were six implementation evaluation questions proposed in the original and revised SEPs. 

They were: 

1. Is the Sankofa program being implemented as intended? Does program content include 

the following elements? 

a. Cultural teachings, rituals, and activities 

b. Academic activities 

c. Cognitive skill building 

d. African-centered, leveled texts 

e. Instructional links with school day/standards (e.g., readers workshop, scaffold 

instruction) 

f. Individual and small group tutoring 

g. Group dialogue, interactions, storytelling, and free books 
Center for Applied Research and Educational Improvement, University of Minnesota 15

2. Is the Sankofa program delivery consistent with the intended program structure? 

a. 40.5 total hours of tutoring per student 

b. Three 90-minute tutoring sessions per week for a period of 9 weeks 

c. Individual and small group instruction 

d. Pre and post reading outcome assessments 

e. Individual learning plans 

f. 5:1 student/tutor ratio 

3. Has the Sankofa program achieved its intended implementation output and goals? 

a. How many students participated in Sankofa? 

b. What percent of enrolled students completed the Sankofa program? 

c. What percent of parents received weekly communications? 

4. What program services are received by the comparison groups? 

a. What is the content of the tutoring program received by the comparison groups? 

b. How are program services delivered to the comparison groups? 

c. What is the duration of program services received by the comparison groups? 

5. What are reasons why participants do not complete Sankofa? 

6. What is the average number of follow-ups (by NdCAD tutors and/or SPPN School 

Partnership Coordinators) before participants discontinue the program? 

Questions 5 and 6 were added after the SEP was revised and were examined only in Year 4. 

Question 4 was not examined because comparable reading comparison groups could not be 

created. 
Center for Applied Research and Educational Improvement, University of Minnesota 16

Methods 

Impact Evaluation 

Several research designs were utilized to answer the eight impact questions examined during the 

evaluation. The questions, data sources, and data analytic plans associated with each design are 

presented by design. 

Pretest-posttest design without a comparison. This design was used to answer 

confirmatory question 1, “Do children who participate in Sankofa exhibit increased reading 

performance as identified by pre-post program scores on the Developmental Reading 

Assessment (DRA)?”, and is capable of providing a preliminary level of evidence. The 

assessment associated with this design was the Development Reading Assessment: Second 

Edition (DRA II).2 The DRA II has been administered to Sankofa participants as a pre-post 

assessment for several years and, consequently, Sankofa staff have been able to monitor the 

effectiveness of the Sankofa reading intervention in a consistent manner over time. Additionally, 

prior preliminary evidence based on the DRA II for the Sankofa program exists (Hall & Gray, 

2010). 

The DRA II is an individually administered standardized reading test which measures five 

dimensions of reading: phonemic awareness, phonics/decoding, comprehension, fluency, and 

vocabulary (DRA II K-8 Technical Manual, 2011). It is used to identify a child’s reading level, 

accuracy, fluency, and comprehension. Preliminary evidence outcomes also included the Saint 

Paul Public School (SPPS) soft target reading goals and scores on a reading attitude instrument 

developed by McKenna and Kear (1990). The soft target reading goals were to be interpreted as 

goals for expected reading levels for grades K through 6 within a designated time period of the 

school year. 

DRA II performance, attainment of soft target reading goals, and reading attitude outcomes were 

considered preliminary evidence because reading outcome measures were only obtained from 

Sankofa participants and not from a comparison group of similar students who did not participate 

in the Sankofa program. The DRA II data came from a database maintained by the SPPN. 

The preliminary evidence measures were: 

● DRA II reading levels 

● Attainment of SPPS soft target reading goals 

The DRA II data were analyzed descriptively. In Years 1 & 2, 99 Sankofa participants in K (n = 

26), grade 1 (n = 36), grade 2 (n = 22), and grade 3 (n = 15) provided data. For Years 3 & 4, 202 

Sankofa participants in K (n = 36), grade 1 (n = 56), grade 2 (n = 48), and grade 3 (n = 62) 

provided data. Eleven students (grade K: 2; grade 1: 3; grade 2: 1; and grade 3: 4) were missing 

pre-Sankofa DRA data (approximately 5.5% of the students) and 56 students (grade K: 6; grade 

 

2

 Information about all the assessments is available in Appendix A. 
Center for Applied Research and Educational Improvement, University of Minnesota 17

1: 15; grade 2: 13; and grade 3: 22) were missing post-Sankofa DRA data (approximately 27% of 

the students).

Parent survey. To answer confirmatory question 2, “Do children who participate in 

Sankofa exhibit greater levels of social emotional development such as confidence in reading 

after the nine-week program?”, a survey was administered to parents/relatives/guardians of 

Sankofa participants. At the end of the Sankofa program, family and friends of the Sankofa 

participants were invited to attend the graduation ceremony that takes place on the last day of 

each cohort. The celebration includes a shared meal and the awarding of certificates and other 

commendations to the Sankofa students. Near the conclusion of the event, all adult members of 

the Sankofa students’ families in attendance are invited to complete a survey about the Sankofa 

program. The “parent survey” completed by a parent, a grandparent, other relative, and/or a 

guardian attending the graduation ceremony, provides an opportunity for the respondent to 

provide feedback to the Sankofa staff and provided preliminary evidence about changes in child 

behaviors observed since attending Sankofa.

Matched comparison. This design was used to answer confirmatory question 3, “Do 

children who participate in Sankofa exhibit greater reading proficiency as identified by Mondo 

Bookshop Assessments after the nine-week program than similar children who do not participate 

in any literacy development program or similar children who participate in a non-culturallyspecific literacy development program?”, and exploratory question 3, “Do children who 

complete Sankofa perform better on third-grade reading proficiency measures than similar 

children in Saint Paul Public Schools?”, and is capable of providing a moderate level of 

evidence. The assessments associated with this design were the Mondo Bookshop Assessments 

and the MCA III - Reading. 

 

Mondo Bookshop Assessments. The Mondo Bookshop Assessments measures aspects of 

reading such as Text Level (overall reading ability), Print Concepts (understanding how books 

work), Letter-Sound Correspondence (ability to read nonsense words), and Oral Language 

(productive vocabulary). The Mondo tests are intended to be used for curriculum development 

rather than as a summative reading achievement measure. However, the only reading assessment 

that was administered district-wide to SPPS children in grades K – 3 was the Mondo and it was 

the only extant assessment available that would allow a comparison between Sankofa 

participants and non-participants. 

The data file provided by SPPS included scores on three Mondo subtests: Oral Language, 

Benchmark Text Level/Comprehension, and Letter/Sound Correspondence. These subtests were 

individually administered by the teachers to students in grades K – 3 at SPPS only to provide 

literacy assessments at the beginning, middle, and end of a school year. All three administrations 

were not required for all students, and all subtests were not required. SPPS guidelines stipulate 

which students are to be given which subtests based on their grade level and their previous 

performance. To minimize the time spent testing, the SPPS guidelines explicitly state that 

teachers should only administer the subtests that are required. Therefore, many students in the 

SPPS data file had incomplete data on the Mondo subtests because they did not require the 

assessment. Students were more likely to have beginning- and end-of-school-year scores than 

mid-year scores, and beginning- versus end-of-school-year was considered to be the best 
Center for Applied Research and Educational Improvement, University of Minnesota 18

comparison for capturing the pre- to post-program change in Sankofa participants’ reading 

performance. Therefore, beginning- and end-of-year scores were included in the moderate 

evidence analyses. Students who participated in Sankofa were compared to a group of nonSankofa SPPS students selected by means of propensity score matching (see subsection on 

Propensity Score Matching). Students from Saint Paul City School were excluded from these 

analysis. 

Oral Language is a test of a student’s receptive language. The test is made up of 15 sentences 

that present different syntactical/grammatical structures of oral English. The teacher reads each 

sentence aloud and the student is asked to repeat the sentence verbatim as read by the teacher. 

One point is awarded for each sentence repeated correctly in every detail. 

Benchmark Text Level/Comprehension is a test that determines a student’s instructional text 

level, or more specifically, the level at which a student is able to read and comprehend with 90-

94% accuracy. After the student’s reading level is recorded, comprehension of nonfiction and 

fiction texts is assessed utilizing retell/recount procedures. Text Level scores are reported as 

letters of the alphabet, ranging from A to W. Scores of A and B are in Mondo’s early emergent 

stage, C and D are in the emergent stage, and so on. A student who is a non-reader is given a 

score of zero. 

In Years 1 and 2, analysis of covariance models were fit for both subtests separately. The 

dependent variable, the difference between beginning-of-year and end-of-year score, was 

regressed onto condition (0: comparison, 1: Sankofa) controlling for beginning-of-year score. 

Mondo data were available for 34 Sankofa participants and 34 comparison students in Years 1 

and 2. 

The Mondo was not used in Years 3 and 4 as a result of the revised SEP. The Mondo was 

removed because 1) we did not believe it would be possible to obtain moderate evidence using 

that design, 2) our use of the Mondo as an outcome was not a validated use for the instrument, 

and 3) there was no definitive source to determine whether a non-Sankofa student was attending 

another reading or cultural program and therefore we could not create appropriate comparison 

groups. 

Minnesota Comprehensive Assessment - Reading. The MCA are the state of 

Minnesota’s tests that meet the requirements of the Federal Elementary and Secondary Education 

Act. The tests that were implemented during the evaluation were in their third revision and are 

titled the MCA-III. The MCA-III tests were administered only in the spring. Scores on the MCAIII were used to determine proficiency in reading and math and are also used to compare students 

to one another on reading and math skills. Proficiency is a dichotomous (i.e., yes/no) variable 

that indicates whether a student has met grade-specific state standards. For the MCA-III reading 

test analyses, third-grade students who participated in Sankofa were compared to a group of nonSankofa SPPS students selected by means of propensity score matching (see subsection on 

Propensity Score Matching). Students in kindergarten through second grade do not take the 

MCA and were not included in the analysis nor were students from Saint Paul City School. 
Center for Applied Research and Educational Improvement, University of Minnesota 19

The MCA data were analyzed with independent t-tests in Years 1 and 2 and were presented in 

the in Years 1 and 2 interim report. For Years 3 and 4, we used a mixed effects model and 

regressed the scaled MCA score onto condition (0: comparison, 1: Sankofa) and controlled for 

sex (because of lack of balance) and academic school year as fixed effects and school as a 

random effect. For Years 1 and 2, data were available on 9 Sankofa participants and 9 

comparison students. For Years 3 and 4, data were available on 37 Sankofa participants (60.6% 

of Sankofa students served in Years 3 and 4) and 253 comparison students. 

Descriptive analysis using MCA data were also conducted to answer exploratory question 1, 

“How do children who complete Sankofa perform on third-grade reading proficiency 

measures?”.

Propensity score matching. For Years 1 and 2, propensity score matching was done using 

nearest neighbors without replacement. The propensity score model for Years 1 and 2 included 

the following variables:

● School 

● Sex 

● Race/Ethnicity 

● Free-/reduced-price lunch status 

● Special education status 

● English language learner status 

and these variables were regressed onto whether a student was in Sankofa or not. The original, 

approved SEP stated that a pre-intervention reading score would be included as a covariate. 

However, SPPS staff recommended not including the Mondo, the only potential district-wide 

pre-intervention measure, because of the way that the Mondo was administered to students (see 

above) and because the number of administrations are determined by student grade-level and 

prior performance. In the revised SEP, we attempted to remove this pre-intervention measure, 

however, external evaluators requested a pre-intervention reading measure and the Mondo was 

included in the Years 3 and 4 propensity score models. 

For Years 3 and 4, we restricted the comparison pool to only students who identified as African 

or African American. For Years 3 and 4, the following variables were included: 

● Sex 

● English language learner status 

● Free-/reduced-price lunch status 

● Special education status 

● Continuous enrollment at the same school 

● Pre-test reading level as measured by the Mondo 

and these variables were again regressed onto whether a student was in Sankofa. For Years 3 & 

4, we used radius matching with replacement (Deheji & Wahba, 2002). Radius matching is a 

form of caliper matching (Caliendo & Kopeinig, 2005), which defines an acceptable maximum 

propensity score distance (caliper) and uses all individuals within the caliper as comparisons. We 
Center for Applied Research and Educational Improvement, University of Minnesota 20

set our radius to .05 and subsequently all comparison students that were within .05 of a Sankofa 

participant were retained for the analysis. All Sankofa students with MCA data were matched to 

the 253 comparison students using this method. 

Information about balance and the distribution of the propensity scores for Years 3 and 4 is

presented in Appendix B. 

 

Interrupted time series. This design was used to answer confirmatory question 4, “Do 

children who participate in Sankofa exhibit greater rates of literacy growth after completing 

Sankofa, compared with their growth before program participation, according to FastBridge 

Learning reading measures?”, and is capable of providing a moderate level of evidence. The 

assessments associated with this design were developed by FBL. For this evaluation, we used the 

Letters Sounds, Sight Words, Decodable Words, earlyReading, and CBMreading assessments. 

The revised SEP indicated we would also use the Letter Names assessment, however, after 

discussion with NdCAD staff, we decided to not administer this assessment but instead to just 

administer Letter Sounds to participants in grade K and 1.

The ITS design was only implemented to participants in Year 43because of the timing of the 

SEP revision. The design required that NdCAD staff assess Sankofa participants weekly during 

the 6 weeks prior to Sankofa and 6 weeks immediately after Sankofa. However, because of the 

complexity of implementing this design (see Results section), no Sankofa participant had 

complete data. 

To analyze the ITS data, a longitudinal linear mixed effects model was used (Pinheiro & Bates, 

2000; Fitzmaurice, Laird, & Ware, 2004; Diggle, Heagerty, Liang, & Zeger, 2002). 

Mathematically, the model shown on the right graph in Figure 4 can be expressed as: 

Eq. 1 

which states that student i’s observed reading score at time j (Yij) is a function of the mean 

intercept effect and a random deviation (β1 and b1i) and a mean slope effect and a random 

deviation (β2 and b2i

4

) and a dummy variable, Xi, that takes on the value of 0 pre-Sankofa and 1 

post-Sankofa. In this equation, β3 and β4 represents the estimated causal effect of Sankofa with β3

representing the intercept/level effect and β4 representing the slope effect. Omitted from 

Equation 1 but included in our models was grade and cohort, our proxy for school, which were 

included as covariates in the level-2 equation for the intercept (i.e., the β1i equation). 

 

3 However, some participants served in Year 5, school year 2017 – 2018, were included in the 

ITS to increase the sample size. 

4 For some models, this variance could not be estimated as it was effectively zero. Therefore, it 

was removed from the models. See Appendix D for details. 
Center for Applied Research and Educational Improvement, University of Minnesota 21

Weeks

Words Read Correctly Per Minute

20

25

30

35

40

0 5 10

Weeks

Words Read Correctly Per Minute

20

30

40

50

0 5 10

Figure 4. Potential effects of Sankofa on reading ability. The graph on the left corresponds to an 

intercept/level effect (most likely) and the graph on the right corresponds to a slope and intercept 

effect. Not pictured is a slope only effect. The solid blue line corresponds to growth in reading 

prior to Sankofa and the dotted blue line corresponds to expected growth if there is no effect of 

Sankofa. The black line corresponds to expected growth if there is an effect of Sankofa. The 

dotted black line corresponds to unmeasured growth during Sankofa. 

We first fit a reduced version of the model presented in Equation 1 where the slope effect, β4, 

was not present. We tested the following null hypothesis, which would correspond to an 

intercept, or level-effect, of Sankofa (the relationship pictured on the left-side of Figure 4). 

We then tested for a slope effect of Sankofa. 

The slope effect was believed to be an unlikely effect of the Sankofa program but it was 

examined. The p-values for the linear mixed effects models were calculated using Satterthwaite’s 

method for approximating the degrees of freedom for t-tests (Kuznetsova, Brockhoff, & 

Christensen, 2017) and if an effect was found to be significant we corrected for multiple 

comparisons. 

Students’ school attendance served as a nonequivalent dependent variable. A nonequivalent 

dependent variable is a variable that should not be affected by the Sankofa intervention but 
Center for Applied Research and Educational Improvement, University of Minnesota 22

would respond in a similar way as the primary dependent variable to a relevant validity threat 

(Shadish, Cook, & Campbell, 2002). While attendance data does not represent an ideal 

nonequivalent dependent variable, as it could potentially be affected by Sankofa, it allowed us to 

assess whether the attendance pattern remained constant before and after Sankofa. If attendance 

remained constant, but reading improved, this increases the likelihood that the effect was 

attributable to Sankofa and not an outside factor (such as a general increase in aptitude because 

of an increase in attending school). 

Qualitative study. A qualitative study was conducted from June through November, 

2017 to address the impact exploratory questions 4 and 5 that were added in Years 3 and 4 of the 

SIF. CAREI collaborated with University of Minnesota School of Social Work faculty member 

Priscilla Gibson, NdCAD, and SPPN staff to conduct a parent focus group (n = 6) and parent 

phone interviews (n = 4). The questions for parents included topics such as their participation in 

the Parent Power Program (a NdCAD parent training program to help parents improve reading in 

their children). All parents who participated in the study received a $20 Target gift card. All data 

were collected in English with the exception of the focus group, which was conducted in Somali.

Of the 10 parents who participated in the study, nine were mothers. The parents’ households 

consisted of a total of 25 children with one to five children per household. The number of children 

per household who attended the Sankofa reading program were one child (in eight of the 

households), two children (in one household) and three children (in one household). All parents 

indicated that their children completed Sankofa and three reported they had children attending other 

tutoring programs. 

A focus group was also conducted with Sankofa staff (n = 9) to explore to what extent having 

African/African American tutors affects parental engagement in improving their child’s literacy. 

Of the nine staff members who participated, four were tutors. All had received training on 

content such as phonogram cards, the instruction approach, student assessments, culture, lesson 

planning, history, and the meaning and foundation of Sankofa. 

All focus groups and interviews were audio-taped and transcribed. Data analysis included multiple 

reviews of both the transcripts and audio tapes. Through repeated readings of the transcriptions and 

listening to tapes, codes were developed and merged using open and selective coding methods. 

Implementation Evaluation 

Fidelity of implementation data were either obtained electronically from a database maintained 

by the SPPN or via hard copies of records provided to SPPN by NdCAD. The fidelity of 

implementation measures included: 

1. Number of student participants 

2. Location of Sankofa cohorts 

3. Dates of Sankofa cohorts 

4. Attendance of student participants 

5. Number of hours of Sankofa programming received by student participants 

6. Number of books given away to student participants 
Center for Applied Research and Educational Improvement, University of Minnesota 23

7. Qualifications of tutors 

8. Content of lesson plans 

9. Observations of tutoring sessions 

10. Communications sent to participants’ parents

11. Responses to a parent survey 

These data sources were analyzed descriptively in Years 1 through 4, with no statistical testing 

performed. 
Center for Applied Research and Educational Improvement, University of Minnesota 24

Results 

The results are presented by evaluation question, first for the impact evaluation followed by the 

implementation evaluation. 

Impact Evaluation for Years 3 and 4 

Confirmatory question 1: Do children who participate in Sankofa exhibit increased 

reading performance as identified by pre-post program scores on the Developmental 

Reading Assessment (DRA)? In Years 3 and 4, DRA II pre- and post-data were available on 

146 Sankofa participants. Data were available on 30, 41, 35, and 40 participants in grades K 

through 3. Of the 146 participants, 90% of the participants increased their DRA II performance 

by at least one reading level. The student performance goal stated in the logic model (Figure 3) 

was that 80% of the students would improve their program assessment scores. Therefore, the 

actual improvement rate of 90% exceeded this goal. This was similar to what was found for 

Years 1 and 2 (96% improvement rate). Ninety percent, 88%, 91%, and 93% of participants in 

grades K through 3 improved by at least one level during Sankofa. On average, participants in 

grades 2 and 3 increased the most levels (2.17 and 2.15, respectively) followed by grade 1 (2.07) 

and grade K (1.43). 

The Sankofa participants’ performance on the DRA II was compared to the soft target gradelevel reading goals established by SPPS (see Appendix C) by comparing the participants’ preand post-program DRA II scores to the targets for the time points associated with each 

participant’s DRA II administrations. The Sankofa participants’ pre- and post-program 

performance was categorized as below, meets, or exceeds the SPPS target for their grade level. 

The results are displayed in Table 2. For all grade levels across all the years, including 

kindergarteners, the post-program results show that the percent below target decreased. Table 2 

also shows that for Years 3 and 4, that kindergarteners were the best readers, based on SPPS 

targets, at the start of the Sankofa program relative to the other grades and that more than half of 

the participants in grades 2 and 3 were struggling readers at the start of Sankofa. Because of this, 

the largest effect (i.e., the greatest decrease in % below from pre- to post-Sankofa) was 

associated with grade 3. 

While 90% of the student participants increased by at least one reading level from pre- to postprogram, only 11 participants (8%) who were below target based on their pre-Sankofa DRA II 

scores were at meets or exceeds target after Sankofa (see Table 3). Therefore, it is informative to 

examine the progress that was made by students in the below-below group (i.e., the group that 

was below the SPPS target both pre- and post-Sankofa). When the performance of the belowbelow group (n = 58) was analyzed separately, it was found that the reading level increases for 

this group ranged from 0 to 6 levels, with an average of 1.74. The most common increase was 1 

or 2 levels, attained by 78% of the students in this group. Forty-one percent attained an increase 

of 2 levels, and 10% attained an increase of 3 levels. Therefore, even though these students did 

make progress during the Sankofa intervention, it was not enough to reach grade-level 

expectations based on SPPS soft targets.
Center for Applied Research and Educational Improvement, University of Minnesota 25

Table 2 

Comparison of Sankofa Students’ Pre- and Post-Program DRA II Scores with SPPS Soft Target 

Reading Goals for Years 1 through 4.

Year Grade N Pre/Post Below Meets Exceeds 

1 - 2

K 26 Pre 8% 46% 46% 

Post 4% 8% 88% 

1 36 Pre 61% 6% 33% 

Post 56% 3% 42% 

2 22 Pre 59% 0% 41% 

Post 55% 0% 45% 

3 15 Pre 53% 7% 40% 

Post 47% 0% 53% 

Total 99 Pre 45% 15% 39% 

Post 40% 3% 57% 

3 - 4

K 36 Pre 11% 58% 25% 

Post 8% 14% 61% 

1 56 Pre 50% 20% 25% 

Post 38% 16% 20% 

2 48 Pre 56% 10% 31% 

Post 44% 4% 25% 

3 62 Pre 53% 13% 27% 

Post 32% 6% 26% 

Total 202 Pre 46% 22% 27% 

Post 32% 10% 30% 

The other group that had a fairly large number of participants, was the exceeds-exceeds group (n 

= 37). The increases for the exceeds-exceeds group ranged from 0 to 6 levels, with an average of 

2.65 levels. The most commonly occurring increases for the exceeds-exceeds group were 1 level 

(24%), 2 levels (22%), and 3 levels (27%). As originally noted in the Years 1 and 2 report it 

appears that students in the exceeds-exceeds group were able to make more progress during 

Sankofa than students in the below-below group, but not by a large margin. 
Center for Applied Research and Educational Improvement, University of Minnesota 26

Table 3 

Number of Sankofa Students by Pre- and Post-Program SPPS Soft Target Outcomes for Years 1 

through 4.

Year Target Outcomes 

Category 

Grade Across 

Grades 

Mean Increase in Levels 

K 1 2 3

1 - 2 Below-Below 1 19 12 7 39 1.77 

Below-Meets 0 1 0 0 1 - 

Below-Exceeds 1 2 1 1 5 1.10 

Meets-Below 0 1 0 0 1 - 

Meets-Meets 2 0 0 0 2 4.20 

Meets-Exceeds 10 1 0 1 12 2.33 

Exceeds-Below 0 0 0 0 0 - 

Exceeds-Meets 0 0 0 0 0 - 

Exceeds-Exceeds 12 12 9 6 39 2.03 

3 - 4 Below-Below 2 17 19 20 58 1.74 

Below-Meets 2 0 0 4 6 2.17 

Below-Exceeds 0 2 2 1 5 4.60 

Meets-Below 1 2 1 0 4 1.00 

Meets-Meets 2 6 1 0 9 0.78 

Meets-Exceeds 14 2 1 2 19 1.95 

Exceeds-Meets 1 3 1 0 5 1.00 

Exceeds-Below 0 2 1 0 3 0.67 

Exceeds-Exceeds 8 7 9 13 37 2.65 

Confirmatory question 2: Do children who participate in Sankofa exhibit greater 

levels of social emotional development such as confidence in reading after the nine-week 

program? For Years 3 and 4, a total of 10 individuals in two of the Sankofa cohorts completed 

the parent survey. There were no parent surveys for the other 7 cohorts. If we assume that each 

survey corresponds to a single participant, then the response rate was 7%. However, given that 

the surveys did not require identifying the Sankofa participant, there could be multiple surveys 

for a single student. Additional caution should be exercised in generalizing the results since 

opinions of the adults who attended the ceremony may be different from those who were not in 

attendance and those who took the time to fill out this voluntary survey may differ from those 

who chose not to. For example, adults attending the ceremony and taking time to respond to the 

survey may have been more positive about the Sankofa reading program. 

The parent survey contained a total of ten questions. Five of the questions were open-ended and 

five were forced-choice. The items asked the respondents about changes they noticed in their 

child related to reading and school performance, the parts of the program they found most 

helpful, their suggestions for improving the program, whether or not they would recommend the 

program to others, and any additional comments they would like to make. A copy of the survey 
Center for Applied Research and Educational Improvement, University of Minnesota 27

instrument is provided in Appendix E. The respondents’ open-ended comments were examined 

and organized according to topic; a summary is provided in the following paragraphs. 

Increases in positive behavior or attitude. The first question on the survey asked if 

respondents had seen an increase in 11 behaviors and attitudes as a result of participating in the 

Sankofa reading program. Respondents could select yes, no, or don’t know/haven’t observed. 

Across the nine respondents who answered this question, participants rated seeing an increase in 

63% to 100% of the 11 behaviors and attitudes, with an average of 88%. 

Stronger connections to family and community. Respondents were then asked if their 

child had formed stronger connections to family and community since participating in Sankofa. 

All but two respondents said their child had formed strong connections to family since 

participating in Sankofa, and one of the two who did not report stronger connections noted that 

their child had always had a strong family connection. Similarly, eight of the 10 respondents said 

their child had formed a stronger connection to their community since participating in Sankofa. 

 

Other changes. When asked about other changes in their children, several respondents 

noted increases in excitement about reading and writing, and school more generally. In addition, 

one respondent noted that their child had become a better and more respectful communicator 

since participating in Sankofa. 

Program components. Respondents were then asked three questions about seven specific 

program components and any other component they found helpful, as well as what Sankofa 

could do to improve. Regarding the seven specific program components, 50% reported that 

instruction on phonemic awareness was helpful, 75% reported that instruction on the seven 

comprehension strategies was helpful, 75% reported that learning about African heritage was 

helpful, 88% reported that learning about positive learning environments was helpful, 88% 

reported that Sankofa staff having high expectations of their child was helpful, 25% reported that 

the low student-teacher ratio was helpful, and 50% reported that the weekly communication from 

the tutors was helpful. Other program components respondents found helpful included: 

connection with other students, having something to do after school, excitement for learning, and 

the connection to community and family. 

Recommending the program to others and satisfaction. All respondents (100%) indicated that 

that they would recommend the program to family or friends. The final item of the survey asked 

the parents to rate their overall satisfaction with the Sankofa program using a 5-point Likert 

scale. Of the nine respondents who provided a rating, 100% selected satisfied or very satisfied.

Confirmatory question 3: Do children who participate in Sankofa exhibit greater 

reading proficiency as identified by Mondo Bookshop Assessments after the nine-week 

program than similar children who do not participate in any literacy development program 

or similar children who participate in a non-culturally-specific literacy development 

program? As indicated in the Methods section, this evaluation question could only be partially 

addressed because a non-culturally-specific literacy comparison group could not be created and 

was removed in the revised SEP. For Years 1 and 2, when this question was partially addressed, 

there was no evidence of an effect of the Sankofa program based on the Mondo Bookshop 
Center for Applied Research and Educational Improvement, University of Minnesota 28

Assessments. Specifically, there was no significant difference between Sankofa participants and 

comparison students on Oral Language (F(1, 65) = 1.46, p = .231, Cohen’s d = 0.25) or the 

Benchmark Text Level/Comprehension (F(1, 65) = 1.10, p = .754, Cohen’s d = -0.02) subtests 

after controlling for beginning-of-year score on these assessments, respectively. Of the 60 

possible Sankofa participants, data were only available for 34 participants. In other words, 43% 

of eligible Sankofa participants did not have Mondo available. The non-significant small-tomedium effect size for Oral Language suggests that the analysis may have been underpowered 

and unable to detect a program effect. 

Confirmatory question 4: Do children who participate in Sankofa exhibit greater 

rates of literacy growth after completing Sankofa, compared with their growth before 

program participation, according to FastBridge Learning reading measures? Prior to 

performing the ITS analysis, we examined the relationship between daily school attendance (the 

nonequivalent dependent variable) before and after Sankofa. If attendance remained constant, 

then this provides validity evidence that any significant Sankofa effect measured by the FBL 

assessments cannot be explained by an increase in attending school. 

Figure 5. Daily school attendance against weeks since the start of Sankofa. The points are daily, 

observed attendance records for a participant, the line is a locally weight scatterplot smoother, 

and the gray rectangle corresponds to time when participants were attending Sankofa. 

A plot of attendance for participants included in the ITS analysis is shown in Figure 5. The black 

line corresponds to a non-parametric smoother, which smooths over data and more heavily 

weighs the observations that are closer in time to come up with an estimate of the slope of the 

line at a particular point in time and can be used to detect non-linear growth and subsequently 

visually assess change in attendance over time. Figure 5 shows that attendance appears to be 

non-linear and that it increases after completing Sankofa. This figure indicates that it will be 
Center for Applied Research and Educational Improvement, University of Minnesota 29

impossible to ascertain if a significant effect is from an increase in attendance at school or 

Sankofa. However, it should be noted that attendance was expected to be affected by Sankofa 

(see the logic model in Figure 3) and the utility of attendance as a nonequivalent dependent 

variable is questionable as the observed increase in attendance may also represent a positive 

effect of Sankofa. 

Initial observations indicated that the CBMReading and earlyReading assessments were too 

difficult for Sankofa participants. Because of this it was decided to not collect post-Sankofa data 

on these measures and to stop administering these assessments. Therefore, for the ITS analyses 

presented below, we used only the Decodable Words, Letter Sounds, and Sight Words 

assessments and performed our analyses controlling for grade and cohort (as indicated in the 

Methods section above). 

We anticipated having data on 87 Sankofa participants for the ITS design but ended up with data 

on 92 Sankofa participants, exceeding this goal. A total of 13, 28, 21, and 30 Sankofa 

participants in grades K, 1, 2, and 3 provided FBL assessment data. 

Table 4 

Summary information about the ITS design by measure 

Assessment Time Grade Participants 

with any data 

Average number of assessments 

per participant 

Decodable Word 

Pre 

2 17 1.882

3 24 3.625

Post 

2 14 3.714

3 23 3.174

Letter Sounds 

Pre 

K 11 2.545

1 24 3.500

Post 

K 13 3.077

1 18 3.833

Sight Word 

Pre 

2 17 1.882

3 25 3.520

Post 

2 14 3.714

3 23 3.130

Table 4 provides summary information about the ITS design by assessment. Three general 

observations about the implementation of the ITS design can be made from this table. First, the 

number of participants with pre-Sankofa data by grade was greater than the number of 

participants providing data post-Sankofa. The one exception was for Letter Sounds and grade K. 

This implies participant attrition in the design. Second, the average number of assessments taken 

by the participants was greater post-Sankofa than pre-Sankofa. Third, the average number of 
Center for Applied Research and Educational Improvement, University of Minnesota 30

assessments per participant, both pre- or post-Sankofa, was around three, suggesting that, on 

average, participants were missing about 50% of their data. While no participant had complete 

data, i.e., six assessments pre-Sankofa and six assessments post-Sankofa, there were several 

participants with five assessments pre-Sankofa and ten participants with six assessments postSankofa. However, none of the participants with five assessments pre-Sankofa had six 

assessments post-Sankofa. 

The percent of missing data, assuming that each of the 92 participants could provide data at each 

assessment window, is shown in Figure 6. Figure 6 shows that the percent of missing data, at any 

assessment window, ranged from just above 20% to just over 80%, and that the closer the 

assessment window was to the start of Sankofa, the lower the percent of missing data. Part of the 

reason for this missing data was that SPPN was still recruiting participants within six weeks of 

the start of Sankofa and those participants could not have had six weeks of pre-Sankofa 

assessments. 

Weeks since the start of Sankofa Percent of data missing

0

20

40

60

80

100

−5 0 5 10 15

●

●

●

● ●

●

●

●

●

●

●

●

Decodable Word

−5 0 5 10 15

●

●

●

●

●

●

●

●

●

●

●

●

Letter Sounds

−5 0 5 10 15

●

●

●

● ●

●

●

●

●

●

●

●

Sight Word

Figure 6. Percent of data missing by weeks since the start of Sankofa by assessment. The dotted 

line corresponds to the assessments pre-Sankofa and the solid line corresponds to the 

assessments post-Sankofa. 

To address the extensive missing data, the analysis was conducted two ways. First, only 

participants with at least one pre-Sankofa and one post-Sankofa assessment were included in the 

analyses. This analysis used listwise deletion and removed all participants that did not have preand post-Sankofa data. Second, multiple imputation was performed using the pan package (Zhao
Center for Applied Research and Educational Improvement, University of Minnesota 31

& Schafer, 2016) in R. The only variables available for multiple imputation were the cohort 

identifier, the participant identifier, the participant grade, the participant’ other scores on the 

assessment(s), and time since the start of Sankofa. In order to reduce the between-imputation 

variability, we specified 100 imputations. We then applied Rubin’s rules to pool the findings 

across the imputations (Rubin, 1974). However, it should be noted that variables that were 

included in the imputation model were driven by availability and not variables that we believed 

were correlated with the assessment score or missingness and even after an extremely long burnin time of 1,000,000 it was not clear if stationarity was actually achieved. 

The Letter Sounds assessment was administered to Sankofa participants in grades K and 1. The 

target in the revised SEP was for 11 participants in grade K and with a power of 0.8, we 

anticipated being able to detect a level effect of 18 letter sounds read correctly per minute 

(LSRCM) and a slope effect of 2.2 LSRCM per week. Overall, 25 participants in grade K and 1 

contributed Letter Sounds data for the analysis. The level effect was estimated at 5.66 LSRCM 

and was not significant (t = 1.034, df = 129.582, p = .303).5 The slope effect was estimated at 

0.023 and was not significant (t = .027, df = 131.308, p = .979). The results from the pooled 

multiple imputation analysis that included all 41 participants in grades K and 1 that had any 

Letter Sounds data also concluded no significant level (estimate = 4.409, t = 0.958, df = 777.733, 

p = .338) or slope effect (estimate = 0.078, t = 0.129, df = 496.660, p = .898)6. 

The Sight Words assessment was administered to Sankofa participants in grades 2 and 3. The 

target in the revised SEP was for 76 participants in grades 2 and 3 and with a power of 0.8, we 

anticipated being able to detect a level effect of 7 words read correctly per minute (WRCM) and 

a slope effect of 0.8 WRCM per week. Overall, 28 participants in grade 2 and 3 contributed Sight 

Words data for the analysis. The level effect was estimated at 0.785 WRCM and was not 

significant (t = 0.079, df = 135.358, p =.938). The slope effect was estimated at 1.198 and was 

not significant (t = 0.213, df = 134.336, p = .423). The results from the pooled multiple 

imputation analysis that included all 51 participants in grades 2 and 3 that had any Sight Words 

data also concluded no significant level (estimate = -2.363, t = -0.327, df = 918.208, p = .744) or 

slope effect (estimate = -0.384, t = -0.423, df = 753.557, p = .672).

The Decodable Words assessment was administered to Sankofa participants in grades 2 and 3. 

The target in the revised SEP for Decodable Words was the same as for Sight Words. Overall, 27 

participants in grade 2 and 3 contributed Decodable Words data. The level effect was estimated 

at 0.056 WRCM, which was not significant (t = 0.022, df = 127.236, p = .982). However, there 

was evidence of a significant slope effect of Sankofa on WRCM (t = 2.453, df = 132.192, p = 

.015). The slope effect was estimated at 0.978. However, after correcting for multiple 

comparisons7, using either a Bonferroni’s correction or the Benjamini-Hochberg method 

(Benjamini & Hochberg, 1995), the slope effect was no longer significant. The results from the 

pooled multiple imputation analysis that included all 51 participants in grades 2 and 3 that had 

any Decodable Words data concluded no significant level (estimate = 0.193, t = 1.426, df = 

 

5 Parameter tables for all ITS models are shown in Appendix D. 

6 See appendix D for full results from the multiple imputation analysis. 

7 To correct for an inflated probability of observing an effect by chance when there is no effect 

(i.e. Type I error). 
Center for Applied Research and Educational Improvement, University of Minnesota 32

381.727, p = .155) or slope effect (estimate = 0.165, t = 0.209, df = 465.186, p = .834). Because 

the slope effect was not corroborated by the pooled multiple imputation analysis and did not 

persist after adjusting for multiple comparisons, this effect was not interpreted further. 

In summary, there were no significant effects on either level or slope for the Letter Sounds, Sight 

Words, or Decodable Words assessments and while the slope effect for Decodable Words was 

significant, after correcting for multiple comparison the effect was no longer significant 

(suggesting the size of the effect could have been observed by chance). The slope effect for 

Decodable Words was also found to not be significant based on the pooled multiple imputation 

analysis. 

 

Exploratory question 1: How do children who complete Sankofa perform on thirdgrade reading proficiency measures? MCA-Reading data were available for 37 Sankofa 

participants in grade 3 for Years 3 and 4. Descriptive information for Sankofa participants and 

their matched comparisons are shown in Table 4. The data shows that 5% of Sankofa participants 

met reading proficiency in grade 3 and 27% partially met reading proficiency. In contrast, for the 

matched comparison (n = 253), 40% partially met MCA proficiency, 21% met MCA proficiency, 

and 2% exceeded MCA proficiency. In addition, the average MCA scale score for Sankofa 

participants (323.5) was lower than for the comparison group (333.4). However, there was a 

wide range of MCA scale scores for Sankofa participants (Figure 7) and the scale scores ranged 

from 301 (corresponding to the 1st percentile) to 362 (corresponding to the 68th percentile). 

Exploratory question 2: Do children who complete Sankofa perform better on thirdgrade reading proficiency measures than similar children who do not participate in any 

literacy development program or similar children who participate in a non-culturallyspecific literacy development program? Not addressed during the evaluation.

Table 4 

Means and Standard Deviations of MCA-Reading Proficiency and Scale Scores of Sankofa and 

Comparison Group Students. 

Group % partially meets 

MCA proficiency8

% meets MCA 

proficiency 

% exceeds MCA 

proficiency 

Average MCA 

scale score 

Sankofa 27% 5% 0% 323.5

Comparison 40% 21% 2% 333.4

 

8 Scale scores corresponding to levels of proficiencies are available on the Minnesota 

Department of Education website. 
Center for Applied Research and Educational Improvement, University of Minnesota 33

MCA Scale Score

Number of Sankofa Participants

0

2

4

6

8

10

300 320 340 360

Figure 7. Distribution of MCA scale scores for Sankofa participants in Years 3 and 4. 

Exploratory question 3: Do children who complete Sankofa perform better on 

third-grade reading proficiency measures than similar children in Saint Paul Public 

Schools? For the combined Year 3 and 4 analysis, the results of the matched-comparison 

analysis indicated a significant effect of Sankofa on third-grade participants reading 

achievement, as measured by the MCA-Reading, but in favor of the comparison students after 

controlling for sex, year, and school (see Table 4). The estimated average treatment effect of the 

treated was -7.25 scale scores with SE = 3.60, t = -2.014, p = .045 (see Appendix B). However, 

this effect was not observed when comparing Sankofa students to all the potential SPPS students 

(unmatched comparison) using a mixed effects regression that controlled for the covariates used 

in PSM as well as the school (t = 0.98, df = 358.49, p = .327). In Years 1 and 2, there was no 

difference between Sankofa participant and comparison students on this measure.

Exploratory question 4: Do African-centered and African-theme texts change 

participants’ attitudes about reading? Both parents’ and staff’s perceptions were positive with 

regard to using an African-centered approach to reading and to Sankofa itself. Findings from 

parents and Sankofa staff are reported in the sections below. 

Parents’ perceptions. When sharing their thoughts after hearing the phrase, “reading is 

important for children of African-descent,” parents related to an African identity such as 

knowing about their culture and ancestors, knowledge about history and knowing about their 

backgrounds. One parent connected the importance of reading African-centered text to having 

positive role models. The parent said, 

That's good because they don't really have any positive Black people on television 

or they don't really see too many Black children books.
Center for Applied Research and Educational Improvement, University of Minnesota 34

Another parent related it to a developmental stage and competing with the racial majority group 

by saying, 

They (children) know that they are children and can learn like Whites.

While concurring with the importance of reading African-centered text, one parent offered the 

following caveat. The parent said, 

It’s important but it doesn’t cure everything. It doesn’t tell very much. 

All 10 parents in the study reported that reading about Africans was very meaningful to their 

children. Two parents connected the reading program to their children developing a positive 

racial identity. They said, “being happy with his race” and “it is good for them to know where 

they come from.” 

Another parent believed that reading about Africans helps children dispel negative images of 

their culture. One parent said, “Before, they (children) didn’t know; some thought that they 

(Africans) were savages.” 

When speaking about the work of Sankofa tutors, one parent said, “...and it sounds like they 

(tutors) made it very fun. She (parent’s child) comes home with all of these different pictures and 

books and packets and posters and things like that. They definitely made it fun but that’s what 

the kids need.” 

Parents appreciated that their children were “gaining knowledge.” They also appreciated the 

“value of children learning about their culture,” and “being exposed to new knowledge.”

In responding to the question about whether there are differences between reading Africancentered text and text about other races, parents had a range of perceptions with no majority 

agreement. Some parents noted the differences as “very meaningful” and allowed his/her child to 

“see people like yourself.” Another parent described the difference as building on knowledge by 

saying, “If they know where they come from, they can learn about other cultures.”

In contrast, another parent was definite that there was no difference and said, “It’s about gaining 

knowledge.” Yet, others had various views. One parent agreed that there was no difference in 

reading African-centered text but conceded that it may be meaningful to her daughter. She said, 

Well maybe (for her, daughter) but she's so little. So, we kinda don't really try to 

separate her. I'm not about everybody's different colors and all that kind of stuff. 

We do talk about all of them, not just our own, and equality and everyone is the 

same inside. I try not to isolate her too much at this point.

 

Overall, parents reported positive changes in their children’s abilities since attending the Sankofa

program. Some changes that they mentioned included: positive attitude, reading better, writing 
Center for Applied Research and Educational Improvement, University of Minnesota 35

better, improvement in math and pronunciations. Most parents identified children’s reading as 

the area that had the most gains. 

Parents’ lessons learned were in the form of knowledge gained by their children. They expressed 

pleasure in the realization that their children “were exposed to reading about culture,” “loved 

Black movies and history,” “identified as African American,” and “talking about what he read.” 

One parent reported that Sankofa supported her/his daughter’s reality by saying, 

Oh I didn't know what the program (Sankofa) was or anything behind it. It was 

just explained to me as a program to help the kids enjoy reading. It wasn't called 

out that they were gonna be focusing on culture. I'm used to the schools having 

their version, if you will, of history so that's why I said it was refreshing. Then she 

was reading some other stuff that was more, I guess, real for lack of a better 

word. 

Staff members’ perceptions. All nine Sankofa staff members in the focus group firmly 

believed that an African-centered approach was the best tool for teaching reading to children of 

African descent. When entering the program, children were observed to have a “lack of 

awareness about their culture and race” and to “say that they are not connected to Africa.” 

According to staff, after these same children were in Sankofa for a period of time, their “selfconfidence increased,” “they were more interested in content about their culture,” and there 

was “excitement” about the content. 

Tutors observed that children responded “positively to the texts used in the program.” They also 

reported that children “liked looking at illustrations” in books that included people who looked 

like them and that the children could “visualize” themselves in the stories. Tutors also described 

how the program promotes children’s self-knowledge by using (a) “opening and closing 

rituals,” (b) “modeling language and belongingness by the tutors calling each other, brother 

and/or sister,” (c) “wearing traditional African clothing,” (d) “using call and response,” and 

(c) “applying the Seven Principles of Kwanza.” 

Exploratory question 5: How does having (African/African American) tutors 

change parent involvement in improving child’s literacy? 

Parents’ perceptions. Parents had mixed views about their own engagement in their 

children’s learning. Mostly, parents characterized themselves as having moderate to no

(engagement) regarding their own level of engagement in the Sankofa reading program. 

However, one parent was very interested and wanted to, “learn more about what they're doing 

and sitting down listening to what they're doing.” 

Most parents (n = 6) had interacted with their children’s tutor. They communicated with the 

tutors through email, phone or text. Similarly, parents had different perceptions about who was 

the best tutor for their children. Some felt that tutors belonging to the same racial group as the 

students increased relatedness, while others said that it played no role in their engagement with 

their children’s learning. One parent identified himself/herself as the best tutor for her/his child 
Center for Applied Research and Educational Improvement, University of Minnesota 36

by replying, “me” in response to the question. Other parents noted that it depended on the 

person’s “knowledge of the topic.”

Initially, parents shared no suggestions on how Sankofa could improve their children’s reading. 

All said that the program is doing a “good job.” When pressed with, “there is always room for 

improvement,” some parents suggested more emphasis on “phonetic awareness,” “involving 

parents,” and “listening and writing skills.” 

Some parents provided other comments about Sankofa. One parent thought the initial start of 

the Sankofa was confusing. The parent said, 

They (school personnel) thought they had the program but they didn't have 

transportation so parents had to go and get the kids. The kids didn't get on the 

school bus to go home. So, you know it was more of a technical or administrative 

problem not a program content problem.

Another parent commented on Sankofa’s use of group-based instead of individual-based 

learning. The parent said, 

Certain kids do need that because their reading skills ain't up to par like 

everybody else’s. So, for them to really understand what's going on, I think some 

kids need one-on-one. They (Sankofa) could probably just take a couple minutes 

for some one-on-one schooling. 

Staff members’ perceptions. Sankofa staff had many suggestions to increase parent 

engagement such as, “making calls to parents as part of our routine,” “involving tutors in home 

visits,” being more vocal about the concept of “it takes a village,” “connecting all aspects” of 

Sankofa for parents, and “being clear about their relationship with children.” They reportedly 

believe that another benefit of the program is forming close relationships with their students and 

treating them as if they were fictive kin (blood related). 

Staff members’ recommended strategies for improving Sankofa included, using the “knowledge 

and wisdom we have gained to build on the African-centered approach.” “developing and using 

in games.” and using more materials that “relate to modern day Africa.” 

Implementation Evaluation for Year 4 

Sankofa tutors. In order to be selected as a Sankofa tutor, an individual must have, at a 

minimum, experience as a tutor or working with children in the community. During Year 4, 

school year 2016 – 2017, 19 individuals served as tutors in the Sankofa reading program. 

Information about the tutors’ qualifications is provided in Table 6 (p. 37). All tutors were either 

African or African American. Everyone had completed at least some college, one had a master’s 

degree, eight had a bachelor’s degree, and three had associate degrees. One tutor had a 

Minnesota teaching license in elementary education. This is similar to what was found for Years 

1 through 3. 
Center for Applied Research and Educational Improvement, University of Minnesota 37

Table 6 

Qualifications of Sankofa Tutors. 

African/African 

American 

Minnesota 

Teacher 

License 

Highest Level 

of Education 

Attained

Field(s) of Study 

Yes Yes (K-8) Masters 

Literature, African, American 

studies communication, 

language arts 

Yes No Bachelors BA Film and Media Studies 

Yes No Associate 

Education, Art, 

Communications, Pursuing 

Bachelors in Education 

Yes No Some College Some College Criminal 

Justice 

Yes No Some College Currently In process 

Yes No Some College Currently In process 

Yes No Some College Currently In process 

Yes No Some College Some College Criminal 

Justice 

Yes No Bachelors BA Professional Education 

Teaching 

Yes No Associate A.A.S. Paralegal Studies 

Yes No Bachelors BA Biochemistry 

Yes No Bachelors BA Psychology and Ethnic 

Studies 

Yes No Bachelors BA Psychology 

Yes No Bachelors BA Fine Arts 

Yes No High School N/A 

All tutors completed two or three trainings conducted annually by NdCAD staff on 

implementation of the Sankofa reading intervention. In-house training is given on implementing 

reading strategies as well as on culturally responsive strategies included in the Sankofa program. 

The required and additional trainings are listed below. 

Required Trainings

● Organization overview/mission/purpose 

● Literacy and identity connections and use of culturally specific leveled texts 

● Triangulation of Sankofa curriculum-instruction-assessment 

● Balanced literacy approach and implementing the Readers Workshop 

● Conducting reading assessments and writing individual learning plans 

● Developing daily activity plans and instructional resources/tools 

● Teaching and modeling comprehension strategies 

● Teaching phonemic awareness using the Spalding Phonogram Cards 

● Strategies for implementing read-alouds and independent guided reading 

● Cultural rituals and routines of the Sankofa reading program 
Center for Applied Research and Educational Improvement, University of Minnesota 38

● Strategies for building student self-regulation and behavior management 

● Program policies, procedures, and expectations 

 Additional Trainings 

● Understanding and building cultural resiliency in students 

● Multiple intelligences and learning styles 

● Understanding mental health needs/issues of students/families 

● Understanding the program logic model 

● Understanding learning resistance and student engagement strategies 

● Parent engagement strategies 

Tutors were selected for each team based on how well the individuals complement one another 

regarding their strengths in different learning styles such as visualizing and hands-on activities. 

The team of tutors for each Sankofa cohort met weekly to develop and discuss lesson plans, and 

the team debriefs after each session. When a special approach seemed needed for an individual 

child or group of children, the team held additional strategy meetings to devise an appropriate 

plan. 

Question 1: Is the Sankofa program being implemented as intended? Does program 

content include the following elements? To answer Question 1, a form was created by Sankofa 

staff for tutors to record their plans for each lesson. An example lesson plan form is included in 

the Appendix F. The evaluators examined 274 completed lesson plans for Year 4. In addition, 

NdCAD staff observed three or four sessions each for seven cohorts for a total of 23 

observations. During these observations, NdCAD used a checklist to ensure that the sessions 

were implemented as intended. Observation data by cohort were summarized across the cohorts 

and are presented in Table 7. These data are included in Appendix G. 

Cultural teachings, rituals, and activities. One hundred percent of the lesson plans 

examined in Year 4 had opening and closing rituals. This was consistent across all four years of 

the SIF. Ninety-six percent of the sessions were observed to be using cultural routines and rituals 

and upon entry to the Sankofa session, the NdCAD observer was greeted 100% with Hotep. 

Academic activities and cognitive skill building. One hundred percent of the lesson plans 

examined in Year 4 had whole group instruction on comprehension strategies and reflection and 

review questions. This was consistent across all four years of the SIF. One hundred percent of 

the observed sessions included learning objectives for the session.

African-centered, leveled texts. One hundred percent of the lesson plans examined in 

Year 4 used African-centered texts (including Grandpa’s face, the Hickory Chair, and A is for 

Africa). This was consistent across all four years of the SIF. In addition, 78% of the sessions that 

were observed had culturally specific, leveled texts on display. 

Instructional links with school day/standards (e.g., readers workshop, scaffold 

instruction). The use of African-centered, leveled texts or instructional links with school 

day/standards could not be determined by examining the lesson plans because the lesson plan 

form did not contain sections to note these items. However, in sessions that were observed by the 
Center for Applied Research and Educational Improvement, University of Minnesota 39

evaluator in Years 1 and 2, all texts were African centered, and scaffolding was noted with 

respect to previewing the text, presenting key vocabulary, reading the text section by section, and 

discussing each section. 

Table 7 

Summary of observation checklists across the observed cohorts. 

Number of sessions 

demonstrating item 

Percent of sessions 

demonstrating item 

1. Does it look like planned program is going on? 22 96% 

2. Is the Daily schedule posted clearly for all to 

see? 

22 96% 

3. Does the daily plan list the learning objectives 

for the session, e.g. Specific reading skills, 

strategies, activities and supplies? 

23 100% 

4. Are culturally rich program materials displayed 

in a planned and organized manner? 

22 96% 

5. Are there culturally specific leveled books 

displayed. 

18 78% 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or 

child/child? 

23 100% 

7. Was I greeted when I entered the classroom? 

Who greeted me? Was I greeted with Hotep? 

23 100% 

8. Are you observing cultural routines and 

rituals? 

22 96% 

9. Are connections to child’s home and 

community being reinforced (through verbal, 

visual, etc....)?

22 96% 

10. Are the separate group activities going on? 23 100% 

11. Is it clear what type of activities are taking 

place in each section of the room? 

23 100% 

12. Is the sign in/attendance sheet in the same 

place every session? 

23 100% 

13. Is the snack in the same place every session? 23 100% 

Individual and small group tutoring. One hundred percent of the lesson plans examined 

in Year 4 had time allocated to small group tutoring and independent silent reading. This was 

consistent across all four years of the SIF. 

Group dialogue, interactions, storytelling, and free books. One hundred percent of the 

sessions were observed to have rich and meaningful dialog among the group as well as group 

work. 

To encourage reading at home, the Sankofa reading program gives away books free of charge to 

participating students to help them build their home libraries. The books that are gifted to 
Center for Applied Research and Educational Improvement, University of Minnesota 40

students are new or gently used books donated by the community. On the last session of each 

week, all Sankofa students who are in attendance that day receive three books. Each student is 

able to select books from a large collection. Based on records kept by NdCAD, a total of 6,525 

books were given to the students in Year 4. On average, each child received approximately 48 

books (or approximately 8 books per week per child). A total of 11,697 books were given away 

during the SIF. 

Question 2: Is the Sankofa program delivery consistent with the intended program 

structure? 

40.5 total hours of tutoring per student and three 90-minute tutoring sessions per week 

for a period of 9 weeks. It is the intention of the Sankofa reading program to schedule each 

cohort for a nine-week period with three 90-minute sessions per week, totaling 27 sessions and 

40.5 program hours per cohort. In Year 4, the number of sessions ranged from 26 to 27 with a 

mean of 26 and the total number of sessions was 241 for Year 4. The number of program hours 

per cohort ranged from 39 to 40.5 with a mean of 40.2 (see Table 8). 

Table 8 

Enrollment and Completion Rates by Cohort Location. 

Cohort 

Number 

Cohort Dates Location Enrollment Cohort 

Completion 

Rate 

Number 

of 

Sankofa 

Sessions 

Program 

Hours 

1 5/20/13 –

7/22/13 

Wilder 

Center 

9 89% 27 40.5

2 10/7/13 –

12/18/13 

St. Paul 

City School 

17 82% 32 40.5

3 1/13/14 –

4/14/14 

Maxfield 

Magnet 

Elementary 

9 89% 27 40.5

4 2/3/14 –

4/15/14 

St. Paul 

City School 

7 100% 28 40.5

5 5/27/14 –

7/23/14 

NdCAD 5 60% 26 39.0

6 10/13/14 –

12/17/14 

Jackson 

Preparatory 

Magnet 

14 100% 25 37.5

7 10/20/14 –

1/8/15 

St. Paul 

City School 

13 85% 27 40.5

8 11/3/14 –

1/14/15 

NdCAD 6 100% 24 36.0

9 1/26/15 –

3/25/15 

Jackson 

Preparatory 

Magnet 

18 83% 26 39.0
Center for Applied Research and Educational Improvement, University of Minnesota 41

Table 8 

Enrollment and Completion Rates by Cohort Location (Continued). 

Cohort 

Number 

Cohort Dates Location Enrollment Cohort 

Completion 

Rate 

Number 

of 

Sankofa 

Sessions 

Program 

Hours 

10 2/11/15 –

4/22/15 

NdCAD 

(Maxfield 

Magnet 

Elementary) 

14 100% 27 40.5

11 6/30/15 –

8/26/15 NdCAD 5

a

60% 26 39

12 10/27/15 –

02/02/16 

Saint Paul 

City School 12 8% 27 40.5

13 11/10/15 –

1/27/16 

Benjamin E 

Mays 11 27% 27 40.5

14 2/22/16 –

4/27/16 NdCAD 15 53% 27 40.5

15 2/29/16 –

5/4/16 

Jackson 

Elementary 16 44% 27 40.5

16 10/10/16-

12/14/16 

Jackson 

Elementary 

16 56% 27 40.5

17 10/10/16-

12/7/16 

NdCAD 18 83% 27 40.5

18 10/11/16-

12/21/16 

B.E.Mays 

Elementary 

25 76% 27 40.5

19 10/11/16-

12/21/16 

St. Paul 

City School 

18 67% 27 40.5

20 2/14/17-

4/26/17 

St. Paul 

City School 

6 100% 27 40.5

21 2/21/17-

5/30/17 

B.E.Mays 

Elementary 

20 95% 26 39.0

22 4/10/17-

5/24/17 

Maxfield 

Elementary 

13 92% 27 40.5

23 4/10/17-

6/10/17 

Jackson 

Elementary 

8 88% 26 39.0

24 6/29/17-

8/24/17 

Liberty 

Plaza 

19 89% 27 40.5

Dosage, hours of programming received by each participant, was estimated by multiplying each 

participant’s attendance rate by the number of program hours for their respective cohort. The 

average dosage ranged from 9.1 hours to 35.6 hours across cohorts. These results indicate that no 

student received the full dosage of programming indicated in Sankofa’s logic model (40.5 

hours). 
Center for Applied Research and Educational Improvement, University of Minnesota 42

Individual and small group instruction. Evaluator observations in Years 1 and 2 

observed both individual and small group instruction. No observations by the evaluators was 

performed in Years 3 and 4 because of the high fidelity observed in Years 1 and 2. In addition, 

100% of the sessions that NdCAD observed had separate group activities taking place.

Pre and post assessment. For all cohorts in Year 4 (as well as Years 1 through 3 of the 

SIF), NdCAD administered the DRA II to students at the beginning and end of the Sankofa 

program. The results from this assessment were presented in the impact evaluation results section 

above. In addition, in Year 4 for cohorts participating in the ITS, FBL data were collected on 

students 6 prior to weeks and 6 weeks after Sankofa (though no participant had complete data).

Individual learning plans. No data on individual learning plans was collected by the 

evaluator or provided by NdCAD or SPPN during any year of the SIF. However, all Sankofa 

tutors were required to undergo training to write individual learning plans and 96% of the 

NdCAD observed sessions appeared to be using a planned program but it was unclear if this 

extended to individual learning plans or was just for the group. 

5:1 student/tutor ratio. The Sankofa tutors work in teams made up of one lead tutor and 

the number of assistant tutors needed to have a student:teacher ratio of 5:1. The student:teacher 

ratio observed during Year 4 ranged from 3:1 to 8:1. In Years 1 and 2, this ratio was observed to 

be 5:1 and in Year 3, it ranged from 5:1 to 6:1. This suggests that across the years, there was 

high fidelity with more variability observed in Year 4. 

Question 3: Has the Sankofa program achieved its intended implementation output 

and goals? Throughout the SIF funding, recruitment efforts targeted children of African descent 

who had low reading scores according to the Mondo assessments, which were administered at 

the schools, and went to school in the SPPN or lived in the SPPN. The administrators (school 

principals) at each SPPN partner school have access to all school data and shared student data 

with SPPN in order to identify students for recruitment purposes. SPPN School Partnership 

Coordinators at Benjamin E. Mays Elementary, Jackson Elementary, and Maxfield Elementary 

assisted in identifying students who would be most likely to benefit from participation in 

Sankofa. School administrators and staff assisted in identifying students at Saint Paul City 

School (SPPN did not have a school partnership coordinator at Saint Paul City School yet during 

the SIF grant). School administrators then prepared a list of students they felt would benefit from 

the Sankofa reading program. The parents of students on the list were then contacted by SPPN 

School Partnership Coordinators regarding enrollment in the Sankofa reading program. 

How many students participated in Sankofa? Between October 2016 and August 2017, 

Sankofa offered its reading program to nine cohorts of children enrolled in kindergarten through 

third grade. The total number of SIF-supported participants across the nine cohorts was 143 in 

Year (see Table 9). The original target for Year 4 was to serve 60 students in four cohorts. Six 

additional cohorts were added to Year 4 at the time the SEP was amended to accommodate the 

newly implemented ITS design. The revised Year 4 target was to serve 144 students in ten 

cohorts. Students in the tenth cohort (Cohort 25) were not included in this report because 

programming for Cohort 25 did not end until late 2017. 
Center for Applied Research and Educational Improvement, University of Minnesota 43

Table 9 

Elementary Schools of Sankofa Participants for Years 1 through 4. 

School School District or Charter Number of 

Participants 

Percent of 

Participants 

Years 1 and 2 

Schools located in the Saint Paul Promise Neighborhood: 

 Jackson Elementary St. Paul Public Schools 36 32% 

 Maxfield Elementary St. Paul Public Schools 32 29% 

 St. Paul City School Charter 34 30% 

Schools not located in the Saint Paul Promise Neighborhood: 

 Galtier Elementary St. Paul Public Schools 1 1% 

 J.J. Hill Montessori Magnet St. Paul Public Schools 1 1% 

 L’Etoile du Nord French Imm St. Paul Public Schools 1 1% 

 Mississippi Creative Arts St. Paul Public Schools 2 2% 

 Obama Elementary St. Paul Public Schools 2 2% 

 Wellstone Elementary St. Paul Public Schools 1 1% 

 Sheridan Arts Magnet Minneapolis Public Schools 1 1% 

 Urban Academy Charter Charter 1 1% 

Year 3 

School located in the Saint Paul Promise Neighborhood: 

 Benjamin E. Mays Elementary St. Paul Public Schools 11 19% 

 Jackson Elementary St. Paul Public Schools 20 34% 

 Maxfield Elementary St. Paul Public Schools 15 25% 

 Saint Paul City School Charter 12 20% 

School not located in the Saint Paul Promise Neighborhood: 

 Bruce F Vento Elementary St. Paul Public Schools 1 2% 

Year 4 

School located in the Saint Paul Promise Neighborhood: 

 Benjamin E. Mays Elementary St. Paul Public Schools 46 32% 

 Jackson Elementary St. Paul Public Schools 24 17% 

 Maxfield Elementary St. Paul Public Schools 31 22% 

 Saint Paul City School Charter 24 17% 

School not located in the Saint Paul Promise Neighborhood: 

 Four Seasons Elementary St. Paul Public Schools 2 1% 

 Expo/Harriet Elementary St. Paul Public Schools 1 1% 

 Obama Elementary St. Paul Public Schools 1 1% 

 Galtier Elementary St. Paul Public Schools 1 1% 

 Capital Hill GT Magnet St. Paul Public Schools 1 1% 

 Bancroft Elementary Minneapolis Public Schools 2 1% 

 Higher Ground Academy Charter 3 2% 

 Universal Academy Charter 2 1% 

 Twin Cities International Elem. Charter 1 1% 

 Dugsi Academy Charter 1 1% 

Total 311 

Nonetheless, the percent of students that were served was 100% of the original Year 4 target and 

99% of the revised Year 4 target. In Year 4, SPPN offered parents gift cards for enrolling their 
Center for Applied Research and Educational Improvement, University of Minnesota 44

students in Sankofa and built a stronger relationship with the new Maxfield Elementary 

principal, which helped with enrollment in Year 4, which was high relative to Years 1 through 3. 

A total of 311 SIF-supported participants were served over the course of Years 1 through 4. 

In Year 4, the 124 SIF-supported participants in Cohorts 16 through 23 came from one of the 

four different elementary schools located in the Saint Paul Promise Neighborhood (Benjamin E. 

Mays Elementary, Jackson Elementary, Maxfield Elementary, and Saint Paul City School). 

Benjamin E. Mays Elementary, Jackson Elementary, and Maxfield Elementary are in the SPPS 

district, and Saint Paul City School is a charter school. The 19 SIF-supported participants in 

Cohort 24 came from one of 11 different elementary schools, only one of which was located in 

the Saint Paul Promise Neighborhood (Benjamin E. Mays Elementary). The remaining 10 

schools include five schools that were part of the SPPS district, one school that was part of the 

Minneapolis Public School district, and four charter schools located in Minneapolis or Saint 

Paul. School enrollment data were not available for four participants in Cohort 24. Table 9 

includes further information about the schools the participants attended in Years 1 through 4. 

Demographic data for Sankofa participants in Cohorts 16 through 24 were provided to the 

evaluators from NdCAD. Demographic characteristics are summarized in Table 10 (p. 45). More 

boys (55%) than girls (44%) participated in Sankofa in Year 4 (similar to Years 1 through 3). 

With respect to grade level, third graders made up the largest group, followed by first graders. 

There were relatively fewer kindergarten and second grade students. In Years 1 through 3, first 

graders were the largest group (for Year 3, second graders were tied as the largest group with 

first graders). All the students identified as African American in Years 3 and 4. 

The Sankofa program was offered at six different locations during the time period October 2016 

to August 2017 and during the time of the SIF (see Table 8). Four of the locations were 

elementary schools in the SPPN (Benjamin E. Mays Elementary, Jackson Elementary, Maxfield 

Elementary, and Saint Paul City School). The other locations were at the NdCAD facility on 

Fairview Avenue N. in Saint Paul (cohort 17) and at Liberty Plaza, an affordable housing 

development in Saint Paul (Cohort 24). All students in Cohort 17 were from Maxfield 

Elementary and the students in Cohort 24 were from eleven different schools (see above for 

more details about the schools Cohort 24 were enrolled in). 

What percent of enrolled students completed the Sankofa program? The Sankofa 

program was designed to accommodate a maximum of 10 or 15 students in each cohort, 

depending on the location. The number of SIF-supported students per cohort ranged from 6 to 25 

for Year 4 (see Table 9). In Table 9, we see that 83% of participants completed Sankofa, which 

was above the target of 80%, but that the completion rate ranged from 56% to 100% across the 

cohorts. Across all the years, a total of 216 Sankofa participants (70%) completed the Sankofa 

program. While this falls short of the target of 80%, it is likely because of Year 3, where the 

completion rate was 37% and was attributed to the increase levels of student mobility and 

homelessness of served participants. 
Center for Applied Research and Educational Improvement, University of Minnesota 45

Table 10 

Demographics of Sankofa Participants in all years.

Demographic Characteristic n % 

Years 1-2 

Gender (n = 82) 

 Female 35 43% 

 Male 47 57% 

Grade Level (n = 100) 

 K 26 26% 

 1 36 36% 

 2 22 22% 

 3 16 16% 

Ethnicity (n = 96) 

 African 92 96% 

 White 1 1% 

 Native American 2 2% 

 Mexican 1 1% 

Year 3 

Gender (n = 59) 

 Female 24 41% 

 Male 35 59% 

Grade Level (n = 58) 

 K 7 12% 

 1 18 31% 

 2 18 31% 

 3 15 26% 

Ethnicity (n = 59) 

African American 59 100% 

Years 4 

Gender (N = 143) 

 Female 64 45% 

 Male 79 55% 

Grade Level (N = 143) 

 K 28 20% 

 1 39 27% 

 2 30 21% 

 3 46 32% 

Ethnicity (N = 143) 

African American 143 100% 

What percent of parents received weekly communications? Communication with 

parents is a very important part of the Sankofa program. To ensure that 100% of Sankofa 

students’ parents receive weekly communications, the lead tutor prepares a hand-written note for 

each child that is given to the child on the last session of each week to take home to his or her 
Center for Applied Research and Educational Improvement, University of Minnesota 46

parents. The lead tutor includes information about the strategies that were covered that week, the 

strengths exhibited by the child, and what the parents might work on at home with the child. The 

content of messages to the parents comes primarily from the observation notebooks that tutors 

take to every session. When observing and interacting with the students, the tutors especially 

look for increases in engagement, increases in confidence, and greater enjoyment of learning to 

read. In special cases where engagement levels remain low, the team along with a program 

administrator will phone the parent to discuss the situation and identify appropriate strategies. 

The tutors also connected with parents if they pick up their children after the sessions. The 

parents are invited to come into the building to talk to the tutors about their child. This is seen as 

an opportunity for the tutors to acknowledge the important role played by the parents and to 

encourage their active participation in helping their child acquire reading skills. 

One-thousand, three-hundred and five notes were sent to parents of children attending Cohorts 

16 through 24. This results in an average of nine notes per student. This is right on the target of 

one note per child per week or a total of nine communications during Sankofa. Therefore, parent 

communication appeared to be implemented with a high degree of fidelity and was similar across 

the four years of SIF funding. 

Question 4: What program services are received by the comparison groups? Not 

addressed during the evaluation (see Methods section). 

Question 5: What are reasons why participants do not complete Sankofa? Several 

reasons were provided to SPPN staff as to why students did not complete Sankofa. This included 

that the child was homeless (n = 2), that the child had enrolled in another program (n = 2), that 

language was a barrier (n = 1), that transportation was a barrier (n = 2), and that they left the 

school or state (n = 2). For one of the participants where transportation was a barrier, SPPN 

successfully addressed this and the child was able to complete the Sankofa program. One 

participant that left a cohort re-enrolled in later Sankofa cohort. Additionally, many of the 

parents/guardians of the participants who left the program did not provide a reason. 

Question 6: What is the average number of follow-ups (by NdCAD tutors and/or 

SPPN School Partnership Coordinators) before participants discontinue the program? The 

average number of follow-ups made by SPPN or NdCAD staff was 0.826 where both the median 

and mode number of follow-ups were 1, respectively. The number of follows-up ranged from 0 

to 2. 
Center for Applied Research and Educational Improvement, University of Minnesota 47

Summary of the Findings

Summary of the Impact Evaluation for Years 3 and 4 

Confirmatory question 1: Do children who participate in Sankofa exhibit increased reading 

performance as identified by pre-post program scores on the Developmental Reading 

Assessment (DRA)? 

 146 Sankofa participants in Years 3 and 4 provided DRA II data. 

 90% increased their DRA II performance by at least one reading level surpassing the goal 

of 80% in the logic model. 

 This improvement rate was similar to Years 1 and 2 (96% improvement rate) 

 For all grades, the percent of participants below SPPS soft target decreased from pre to 

post. 

Confirmatory question 2: Do children who participate in Sankofa exhibit greater levels of 

social emotional development such as confidence in reading after the nine-week program? 

 The majority of parents noted an increase in positive behavior or attitude in their 

children’s behavior. 

 80% of parents reported their children had strong connections to their family and 

community.

 Parents noted excitement about reading and writing, and school more generally. 

Confirmatory question 3: Do children who participate in Sankofa exhibit greater reading 

proficiency as identified by Mondo Bookshop Assessments after the nine-week program 

than similar children who do not participate in any literacy development program or 

similar children who participate in a non-culturally-specific literacy development 

program? 

 Only partially addressed in Years 1 and 2 and not addressed in Years 3 and 4 as it was 

removed from the revised SEP. 

 Compared to a SPPS derived matched-comparison group in Years 1 and 2, there was no 

evidence of an effect of Sankofa as measured by the Oral Language and Benchmark Text 

Level/Comprehension. 

 For the Oral Language assessment, a small-to-medium effect size for Oral Language was 

found suggesting the analysis might have been underpowered (n = 68). 

Confirmatory question 4: Do children who participate in Sankofa exhibit greater rates of 

literacy growth after completing Sankofa, compared with their growth before program 

participation, according to FastBridge Learning reading measures? 

 No evidence of an effect of Sankofa using an ITS design with FBL’s Letter Words, Sight 

Words, and Decodable Words assessments (moderate level of evidence design). Initially 

a significant slope effect was found for Decodable Words but this effect was no longer 
Center for Applied Research and Educational Improvement, University of Minnesota 48

significant after correcting for multiple comparisons and was not corroborated by the 

multiple imputation pooled analysis. 

 No participants provided complete pre-Sankofa and post-Sankofa data on these 

assessments. 

 The non-equivalent dependent variable, school attendance, showed an increase after 

Sankofa (Figure 5).

Exploratory question 1: How do children who complete Sankofa perform on third-grade 

reading proficiency measures? 

 MCA data were available on 39 Sankofa participants. 

 27% of these Sankofa participants partially met MCA proficiency and 5% met MCA 

proficiency. 

 The average MCA scale score for Sankofa participants was 323.5 (which lies between the 

11th and 12th percentile). 

 Sankofa participants ranged from the 1st to the 68th percentile on the MCA scale score. 

Exploratory question 2: Do children who complete Sankofa perform better on third-grade 

reading proficiency measures than similar children who do not participate in any literacy 

development program or similar children who participate in a non-culturally-specific 

literacy development program? 

 Was not addressed during the evaluation. 

Exploratory question 3: Do children who complete Sankofa perform better on third-grade 

reading proficiency measures than similar children in Saint Paul Public Schools? 

 In Years 1 and 2, no difference was found, whereas in Years 3 and 4, a significant 

difference was found. 

 The MCA-Reading scale score analysis found that Sankofa participants scored 

significantly lower than their matched comparisons from SPPS. 

 On average, Sankofa participants were expected to score 7.25 points lower on the MCA 

than the non-Sankofa SPPS comparison. 

 This finding was not observed in the unmatched analysis. 

Exploratory question 4: Do African-centered and African-theme texts change participants’ 

attitudes about reading? 

 Both parents’ and staff’s perceptions were positive with regards to using an Africancentered approach to reading and to Sankofa itself. 

 Parents felt it was meaningful that their children were reading about Africans. 

 Parents had a range of perceptions, with no majority agreement, regarding whether there 

are differences between reading African-centered text and text about other races. 

 Parents were pleased that their children were reading about their culture. 
Center for Applied Research and Educational Improvement, University of Minnesota 49

 Sankofa staff members in the focus group firmly believed that an African-centered 

approach was the best tool for teaching reading to children of African descent. 

Exploratory question 5: How does having (African/African American) tutors change 

parent involvement in improving child’s literacy? 

 Parents had mixed views about their own engagement in their children’s learning 

and who was the best tutor for their child(ren).

 Parents described their engagement from moderate to none.

Summary of the Implementation Evaluation for Year 4 

Question 1: Is the Sankofa program being implemented as intended? Does program 

content include the following elements? 

 The Sankofa program continues to be implemented as intended. 

 Examination of lesson plans and observation checklists showed evidence of 

o Cultural teachings, rituals, and activities being used. 

o Academic activities and cognitive skill building. 

o The use of African-centered, leveled texts. 

o Individual and small group tutoring 

o Group dialogue and providing free books to Sankofa participants 

 While not directly observed in Year 4, previous evaluator observations in Years 1 and 2 

found evidence of scaffolding. 

 The implementation of Sankofa tracks closely the logic model presented in Figure 3. 

Question 2: Is the Sankofa program delivery consistent with the intended program 

structure? 

 The Sankofa program continues to be delivered consistently with its intended program 

structure. 

 In Year 4, the number of sessions range from 15 to 27 with a mean of 24. The number of 

program hours per session ranged from 22.5 to 40.5 with a mean of 36.7 and the dosage 

ranged from 9.1 to 35.6 hours. 

 Pre- and post-Sankofa DRA II assessments were administered to most Sankofa 

participants and many Sankofa participants participated in the ITS study. 

 During Year 4, the student to teacher ratio ranged from 3:1 to 8:1. 

Question 3: Has the Sankofa program achieved its intended implementation output and 

goals? 

 In Year 4, Sankofa was delivered to 143 students. The original target for Year 4 was 60, 

which was later revised to 147 to accommodate the ITS study. 

 Cohorts were held at 6 different locations in the SPPN. 
Center for Applied Research and Educational Improvement, University of Minnesota 50

 One hundred percent of participants identified as African American and there were 

slightly more male than female participants. 

 The completion rate in Year 4 was 83%, which was above the target of 80%. This was 

also well above the completion rate of 37% in Year 3. 

 One-thousand, three-hundred and five notes were sent to parents of children attending 

cohorts 16 through 24. This results in an average of 9 notes per student, which implies 

that parent communication was implemented with a high degree of fidelity and was 

similar across the four years of SIF funding. 

Question 4: What program services are received by the comparison groups? 

 Was not addressed during the evaluation due to unavailable data. 

Question 5: What are reasons why participants do not complete Sankofa? 

 Homelessness, enrolling in other programs, leaving the school or state, and barriers 

associated with language and transportation were identified as reasons participants did 

not complete Sankofa.

 For one of the participants where transportation was a barrier, SPPN was able to 

successfully address this and the participant completed Sankofa.

Question 6: What is the average number of follow-ups (by NdCAD tutors and/or SPPN 

School Partnership Coordinators) before participants discontinue the program? 

 The average number of follow-ups that was made by SPPN or NdCAD staff was 0.826 

where both the median and mode number of follow-ups were 1, respectively. 

 The number of follows-up ranged from 0 to 2. 
Center for Applied Research and Educational Improvement, University of Minnesota 51

Interpretations and Limitations 

The impact evaluation found a significant effect of Sankofa on the DRA II (preliminary level of 

evidence design). The preliminary evidence design, however, lacked a comparison group, either 

a matched-comparison or comparing participants with themselves over time, and suffered from 

limited internal validity as the program effect could have been caused by external factors or just 

represent normative growth. 

The impact evaluation found a significant effect of Sankofa on growth in reading for the 

Decodable Words assessment using an ITS design (moderate level of evidence design) but not 

for the Letter Sounds or Sight Word assessments. However, this effect for Decodable Words was 

not significant after correcting for multiple comparisons (suggesting this finding could have been 

obtained by chance) and it was not corroborated by the multiple imputation pooled analysis. For 

the exploratory matched-comparison design, there were no effects of Sankofa on the MCAReading. 

The purpose of this section is to provide some insight into the interpretations and limitations of 

these findings. 

Regarding program implementation, was the program implement as intended? Based on 

the logic model presented in Figure 3, the answer is largely yes. Sankofa has been consistently 

shown to be implemented during the SIF period with high fidelity (often exceeding their goals). 

Across the four years, the Sankofa program has utilized cultural teachings and rituals with 

academic activities. They have used African-centered level texts, instructional links, and 

scaffolding. The program has incorporated group dialogue, provided opportunities for child-tochild and child-to-tutor interactions, storytelling, and NdCAD has given an immense, and 

extremely laudably, number of books to their participants. Staff have administered the DRA II to 

participants entering and completing Sankofa and while it was not directly observed by the 

evaluators, the tutors were required to undergoing training in writing individual learning plans, 

which they then likely tailored to their participants’ needs. 

The largest variation in the Sankofa program has been around the actual dosage that participants 

have received versus the intended dosage (intended to be 40.5 hours), and the variation in the 

student to tutor ratio (intended to be 5:1). Of the two, the variation in the student to tutor ratio 

has been much less. However, each of these deviations, particularly received dosage, could 

accumulate to hinder one’s ability to detect program effects across all of the evaluation designs. 

If full dosage is critical for receiving the Sankofa program, then deviation from this should 

dramatically reduce the program’s impact.

Sankofa serves a very heterogeneous reading population and it is increasingly serving 

participants with greater needs, because of homelessness and heightened mobility. Therefore, 

there might need to be some alterations in how the program is delivered to effectively cater to 

these diverse learning needs and there may need to be new strategies to bolster attendance, and 

subsequently ensure that participants receive the full programming. 
Center for Applied Research and Educational Improvement, University of Minnesota 52

Regarding the research design, was it implemented as intended? Given our targeted level of 

evidence (moderate) and the use of quasi-experimental designs, our concerns with evaluating this 

question surrounds validity, specifically internal validity. Validity ultimately refers to the extent 

to which our findings are trustworthy, and internal validity is concerned with the extent that our 

research design was properly implemented to support the nature of our claims 

For the SIF evaluation, Sankofa was expected to target the lowest performing readers. However, 

during the SIF, Sankofa served students with a wide range of reading abilities (e.g., see the MCA 

and DRA II analyses) and it is possible that serving students with such a myriad of abilities could 

cloud the evaluation’s ability to detect an effect. However, as was noted earlier, SPPN was 

recruiting low performing students based on the Mondo. This implies disagreement about a 

student’s reading abilities on these assessments. Outside of the SIF, Sankofa serves all students 

and there is nothing in their logic model that requires Sankofa participants to be low performing 

readers. Therefore, it is unclear the extent to which this is a validity threat outside of the 

evaluation. 

In this evaluation, we used two quasi-experimental designs, a matched-comparison and an ITS 

design. Initially, a matched-comparison was used in Years 1 and 2 that compared the 

performance of SPPS students to Sankofa participants on the Mondo Bookshop Assessment. As 

noted at the time of the revised SEP, the use of the Mondo Bookshop Assessment as a 

summative assessment is not a valid use and instead this assessment was intended to be used for 

curriculum-development. Therefore, we removed the Mondo Bookshop Assessment in the 

revised SEP. In addition, we were unable to create the counterfactual comparison groups that we 

originally proposed because the data were not collected by SPPS or from a program partner. 

Throughout the SIF, we compared the performance of 3rd graders in SPPS to 3rd grade 

participants in Sankofa on the MCA-Reading. In Years 1 and 2, we found no effect, while in 

Years 3 and 4, we found a significant, negative effect of Sankofa. How valid is this finding? 

First, this effect was exploratory in nature. There was nothing in the logic model to directly link 

Sankofa participation to MCA-Reading scores. Therefore, it would be an improper use of this 

analysis to evaluate the program based on this effect. Second, for a matched-comparison to 

provide an unbiased estimate of a treatment effect, we need variables in our PSM that are related 

to the treatment (Sankofa) or the outcome measure (MCA-Reading) and preferably both. The 

variables included in our model were not derived by theory, but instead by their availability. In 

our PSM, the variables that were included were the variables that SPPS collects data on and not 

necessarily variables related to participation in Sankofa (or a similar reading or cultural 

intervention) or achievement. The more these variables related to the treatment and/or the 

outcome, the less bias there should be in our estimate of the treatment effect. For the Mondo 

Bookshop Assessment, we were unable to include a prior measure of achievement and for the 

MCA, we used the Mondo Bookshop Assessment as our prior measure. However, as noted 

earlier, this assessment was administered largely on an as-needed, formative basis, so students in 

SPPS have varied amounts of data available for this assessment and varied opportunities to 

provide this data. 
Center for Applied Research and Educational Improvement, University of Minnesota 53

For the matched comparison analyses, we included only students in SPPS and not in Saint Paul 

City School as there was no common prior achievement measure across the schools. In an ideal 

design, all students at SPPS and Saint Paul City School would have been given a common 

reading assessment prior to Sankofa and this would have been used as our prior achievement 

variable in our PSM model. However, a common assessment was not used at the two locations 

making it possible that the Sankofa and comparison students had significantly different reading 

abilities that were not adjusted for during our analysis and the exclusion of the Saint Paul City 

School students would have reduced the generalizability of any significant findings. 

Finally, it is unclear if the MCA is a malleable enough outcome for a program with a duration of 

9 weeks to affect. If the Sankofa program had a meaningful “bump” in reading that occurred 

during the program and persisted for only a short while afterward, then the MCA might not be 

able to detect a difference as it is administered in the spring and many Sankofa participants could 

have completed Sankofa almost a year earlier and this effect may have disappeared. 

For these reasons, as well as the exploratory nature of this impact question, any significant 

findings based on the MCA may be dubious and should be interpreted with caution. 

Because of our shortcomings with a matched-comparison design, in the final year of the SIF, an 

ITS was proposed and implemented. The ITS used FBL’s assessments, which are intended to be 

used for progress monitoring. Unlike the Mondo Book Assessment and the MCA, the use of the 

FBL measures for assessing the impact of a 9-week reading program is a clearly valid and 

intended use. However, there were no significant effects found in the ITS study after correcting 

for multiple comparisons. 

Why did the ITS fail to find a significant treatment effect for the measures? Most 

importantly, the implementation of the ITS was logistically difficult. It was difficult to identify 

prospective participants far enough in advance to assess them six weeks prior to Sankofa and 

then to track them long enough to assess them six weeks after Sankofa. Because of this no 

participant had complete data (i.e., six assessments both pre- and post-Sankofa). Assessing 

participants six times over six weeks put a burden on the participant, the assessor, and the 

participant’s family because special arrangements were needed just for their child to be assessed 

for a minute (the length of time it takes to administer the FBL assessments) because SPPS did 

not allow NdCAD to collect data in their schools. Finally, because the ITS was implemented in 

the final year, little time was available to work out these logistical hurdles and combat them 

before going “live.” 

The extent to which the missing data were ignorable for both the ITS and the matchedcomparisons has important consequences. If participants were missing data on these measures 

just because they were low on the FBL assessments, Mondo Bookshop Assessment, or the MCA 

then this introduces bias into our estimates. Given the extent of the missingness for all our 

measures, the importance of this cannot be understated. 

Arguably, missing data was the single largest internal validity threat. While we used multiple 

imputation to attempt to address missingness, it could be argued, similar to the matched 

comparison analysis, that because the variables included in the multiple imputation model were 
Center for Applied Research and Educational Improvement, University of Minnesota 54

driven by availability and not by theory that this analysis had limited statistical validity. In 

addition, regardless of the burn-in period used for the multiple imputation model, the chains 

failed to reach stationarity suggesting that the model may not have converged. 

While the ITS failed to find a significant treatment effect for all the measures and was hindered 

with missingness, it still represented the best opportunity for obtaining moderate-evidence. Why? 

First, only the FBL assessment used in the ITS has been shown to be valid for progress 

monitoring over a short-time frame. Second, the ITS allowed us to skirt the issue of creating a 

matched-comparison counterfactual group by comparing Sankofa participants to themselves. 

Third, with complete data, it is probable, though not guaranteed, that we would have detected an 

effect of Sankofa for Letter Sounds, Sight Words, and Decodable Words. 

Finally, the utility of daily school attendance as a nonequivalent dependent variable is 

questionable. This variable is one of the short-term outcomes in the logic model that Sankofa 

was expected to improve, which did happen (Figure 3). However, because it is an outcome for 

the program, it is not considered nonequivalent. In this evaluation, we did not separately measure 

a nonequivalent dependent variable and instead relied on SPPS and Saint Paul City School to 

provide that data. However, there was no guarantee that the data provided would have been an 

appropriate nonequivalent dependent variable. The design could be made stronger by using a 

nonequivalent dependent variable that is not already an outcome expected to be affected by 

Sankofa. Instead, an appropriate variable would relate to an overall increase in the level of 

student achievement (such as mathematics achievement or stable housing) but not expected to be 

affected by participating in Sankofa. 

At this point, a significant effect of Sankofa on reading proficiency was found only for the DRA 

II (a preliminary level of evidence design). Based on this evaluation, our recommendation is to 

continue developing strategies to ensure high attendance and full dosage during Sankofa, and to 

continue to work at developing an evaluation framework that can validly measure the impact of 

Sankofa, and is practically feasible to implement (i.e., that minimizes missing data and 

subsequently maintains high internal validity). The framework needs to also include access to 

data that can be collected by the program or the evaluation and does not depend on the schools. 
Center for Applied Research and Educational Improvement, University of Minnesota 55

Conclusions and Recommendations 

Impact Evaluation for Years 3 and 4 

The confirmatory impact evaluation for Years 3 and 4 found the following: 

 Evidence of an effect of Sankofa using a pre/post design with the DRA II as the outcome.

o 90% of Sankofa participants increased by at least one reading level (preliminary 

level of evidence design). 

 Parents reported an increase in positive behavior or attitude in their children 

o 80% of parents reported that their children had stronger connections to their 

family and community since participating in Sankofa.

 No evidence of an effect of Sankofa using an ITS design with FBL’s Letter Words, Sight 

Words, and Decodable Words assessments (moderate level of evidence design). 

o Initially a significant slope effect was found for Decodable Words but this effect 

was no longer significant after correcting for multiple comparisons and was not 

corroborated by the multiple imputation pooled analysis. 

 The ITS design had substantial missing data. 

The exploratory impact evaluation for Years 3 and 4 found the following: 

 On the MCA, 27% of Sankofa participants partially met MCA proficiency and 5% met 

MCA proficiency. Sankofa participants ranged from the 1st to the 68th percentile on the 

MCA scale score. 

 A statistically significant effect on the MCA found that Sankofa participants were 

expected to score 7.25 points lower on the MCA than the non-Sankofa SPPS comparison. 

Given that it is unclear that Sankofa and comparison students were balanced on a valid 

prior achievement measure, there is limited statistical validity to this finding. 

 This effect was not observed in the unmatched analysis. 

The qualitative study results showed that parents’ and staff’s perceptions were positive with 

regards to the African-centered approach used by NdCAD; that parents found it meaningful to 

read about Africans and one’s culture; and that parents expressed a range of opinions regarding 

the importance of reading African-centered text relative to texts on other races. Finally, parents 

expressed mixed views regarding who the best tutor for their child was. 

Based on the impact evaluation for Years 3 and 4, the following recommendations are provided: 

1. Continue to address logistical barriers that hinder data collection that make the use of 

an ITS design difficult. The ITS approach was welcomed by NdCAD and SPPN staff 

and the ITS design, when combined with robust data collection and a valid outcome 

measure (such as the FBL measures), represents the best opportunity to observe a 

moderate-level of evidence. 

2. NdCAD staff may consider using FBL measures in lieu of the DRA II to track 

progress before, during, and/or after Sankofa. While this approach would not 
Center for Applied Research and Educational Improvement, University of Minnesota 56

necessarily need to be as involved as the ITS design utilized for the SIF, it would 

allow NdCAD to collect data that goes beyond the DRA II and would allow them to 

measure growth. Using the FBL measures during Sankofa could allow NdCAD to 

understand for which participants Sankofa is having the strongest effects and which 

participants continue to struggle. This, when coupled with individualized learning 

plans, could make the impact of Sankofa larger. 

3. If the ITS proves to be too logistically difficult to implement, future considerations 

could involve regression discontinuity where if prospective participants fall below a 

certain reading threshold they are assigned to Sankofa and if they are above it they 

are not. However, this would involve working closely with SPPS to ensure that this 

approach would be feasible and that the outcome measure used to estimate the 

treatment effect is appropriate. 

4. The qualitative study provided the following recommendations for Sankofa regarding 

the two exploratory questions: 

a. Allow Sankofa staff to conduct all aspects of recruiting, screening and enrolling 

children into Sankofa. Parents may be getting mixed information about it. 

b. Continue using the African-centered approach. Parents appreciate that their 

children were learning about their race, history and culture. The African-centered 

approach is working well for the majority of the parents. 

c. Consider allowing tutors to make home visits. The benefits may result from tutors 

having a built-in contact with parent. 

d. Conduct an annual evaluation of all aspects of Sankofa and develop a method to 

share the findings with parents. 

e. Develop a strategy for former students/graduates of Sankofa and their parents to 

remain connected to the program. 

Implementation Evaluation for Year 4 

The Sankofa program continues to be implemented with high fidelity. The program content and 

delivery closely resemble that proposed in the logic model. The biggest hurdles for 

implementation appeared to be around the number of hours that participants are actually 

receiving. 

Based on the implementation evaluation, the following recommendations are provided: 

1. Consider developing new, or utilizing a diversity of, strategies to keep attendance 

high. Because no participant received the full dosage of 40.5 hours, this likely 

reduced the impact of Sankofa on its participants. If the full dosage of 40.5 hours is 

not necessary, then modifying the logic model is recommended. 

2. Given the range of reading abilities of participants entering Sankofa, additional tutor 

trainings that focus on working with readers with varied abilities might be beneficial. 
Center for Applied Research and Educational Improvement, University of Minnesota 57

3. As the number of participants that Sankofa serves that are homeless or highly mobile 

increases, it might be useful to develop a tutoring training module focused on 

working with these children as they may have special reading, social, emotional, and 

behavioral needs. 
Center for Applied Research and Educational Improvement, University of Minnesota 58

References 

Barnes, E. J. (1991). The Black community as the source of positive self-concept for Black 

children: A theoretical perspective. In R. Jones (Ed.), Black psychology (pp. 667-692). 

Berkeley, CA: Cobb and Henry. 

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: a practical and 

powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B 

(Methodological), 289–300. 

Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of 

propensity score matching. Journal of Economic Surveys, 22(1), 31–72. 

Corporation for National & Community Service. (2013, May). Social innovation fund: Content 

requirements for subgrantee evaluation plans. 

Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental 

causal studies. Review of Economics and Statistics, 84(1), 151–161. 

Diggle, P., Heagerty, P.J., Liang, K.-Y., & Zeger, S. L. (2002). Analysis of longitudinal data. 

Oxford University Press. 

DRA2 K-8 Technical Manual (2nd ed.). 2011. Boston, MA: Pearson. 

Fitzmaurice, G. M., Laird, N. M., & Ware, J. H. (2004). Applied longitudinal analysis. John 

Wiley & Sons. 

Grills, C., Cooke, D., Douglas, J., Subica, A., Villanueva, S., & Hudson, B. (2015). Culture, 

racial socialization, and positive African American youth development. Journal of Black 

Psychology, 1-31, Retrieved from DOI: 10.1177/0095798415578004 

Hall, N., & Gray, K. (2010). NdCAD’s Sankofa Reading Tutorial Program and “Parent 

Power” Literacy & Advocacy Training Workshops: Evaluation report. St. Paul, MN: 

GrayHall LLP. 

Hill, P. W., & Jaggar, A. M. (2003). An Evaluation Report on the Impact of Mondo Publishing’s 

Building Essential Literacy (BEL) Design and Bookshop Reading Program 1998–1999 

and 1999–2000 School Years. Unpublished Manuscript. Retrieved from 

http://mon.cuesta.com/Pages/articles/Hill.Jaggar.Supovitz.pdf

Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. B. (2017). lmerTest Package: Tests in 

Linear Mixed Effects Models. Journal of Statistical Software, 82(13), 1–26. 

https://doi.org/10.18637/jss.v082.i13

Ladson-Billings, G. (1995). Toward a theory of culturally relevant pedagogy. American 

Educational Research Journal, 32(3), 465-491. 
Center for Applied Research and Educational Improvement, University of Minnesota 59

Ladson-Billings, G. (2000). Fighting for our lives. Journal of Teacher Education, 53(1), 206. 

Ladson-Billings, G. (2011). Boyz to men? Teaching to restore Black boys’ childhood. Race 

Ethnicity and Education, 14(1), 7-15. 

Ladson-Billings, G. (2012). Through a glass darkly: The persistence of race in education 

research and scholarship. Educational Researcher, 41(4),115-120. 

Minnesota Department of Education. (2009, December). Technical Manual for Minnesota’s Title 

I and Title III Assessments. Roseville, MN: Minnesota Department of Education. 

National Center on Intensive Intervention. (2017, January 16). Academic Progress Monitoring 

Tool Charts. http://www.intensiveintervention.org/resources/tools-charts. 

Northwest Evaluation Association (2018). Measures of Academic Progress. 

https://www.nwea.org.

Pinheiro, J., & Bates, D. (2000). Mixed-effects models in S and S-PLUS. Springer Science & 

Business Media. R. 

Pressley, M., Raphael, L., Gallagher, J. D., & DiBella, J. (2004). Providence-St. Mel School: 

How a school that works for African American students works. Journal of Educational 

Psychology, 96(2), 216-235. 

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized 

studies. Journal of Educational Psychology, 66(5), 688. 

Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental 

designs for generalized causal inference. Houghton, Mifflin. 

Supovitz, J. (2003). External Review of the Evaluation of Mondo Publishing’s Building 

Essential Literacy (BEL) Design and Bookshop Reading Program. Unpublished 

Manuscript.

Taylor, B.M., Pearson, P.D., Clark, K., & Walpole, S. (2000). Effective schools and 

accomplished teachers: Lessons about primary-grade reading instruction in low-income 

schools. The Elementary School Journal, 101(2), 121-165.

Zhao, J. H., & Schafer, J. L. (2016). pan: Multiple imputation for multivariate panel or 

 clustered data. R package version 1.4. 
Center for Applied Research and Educational Improvement, University of Minnesota 60

Appendix A 

Assessments

Developmental Reading Assessment: Second Edition 

Reliability. The DRA II technical manual states that “triangulation of the multiple forms 

of reliability analyses that were conducted shows that the DRA II is a reliable measure in that it 

produces stable, consistent results over time, different raters, and different samples of work or 

content. Specifically, it demonstrates moderate to high internal consistency reliability, parallel 

equivalence, test-retest reliability, and inter-rater reliability. While the results demonstrate that 

the DRA II has relatively little measurement error associated with content, time, and rater, it is 

important that examiners follow the administration and scoring guidelines provided in the DRA 

II Kits. Furthermore, it is highly recommended that examiners participate in professional 

development from Pearson professionals in order to further minimize any measurement error.”

Validity. The DRA II technical manual states that “it is imperative that a test is valid in 

order for the results to be accurately applied and interpreted. The findings presented on contentrelated validity, criterion-related validity and construct validity provide support for the validity 

of the DRA II. Specifically, the data show that the DRA II subtests measure those constructs it 

was designed to measure – oral fluency and reading comprehension. Elaborating on this, results 

show that oral fluency and reading comprehension, as measured by the DRA II, represent unique 

dimensions of reading. However, results also show that these two subtests are correlated with 

one another at a moderate level – as would be expected since comprehension and fluency are 

both related facets of reading which are highly correlated to the total reading score. Additionally, 

results indicate that fluency and reading comprehension measures are developmental in nature, 

as demonstrated by the strong correlations with age, and that accuracy is influenced, as 

predicted, by reading more challenging texts. In sum, the results presented indicate that the DRA 

II is a valid measure that can accurately measure students’ oral reading fluency and 

comprehension level.” 

The technical manual can be download at 

http://assets.pearsonschool.com/asset_mgr/current/20139/DRA II_Technical_Manual_2012.pdf. 
Center for Applied Research and Educational Improvement, University of Minnesota 61

Mondo Bookshop Assessments 

Psychometric information. Because the assessments were designed for curriculum 

development, Mondo Education Publishing has not conducted reliability or validity testing. 

However, one study that examined the effectiveness of the curriculum did calculate the 

assessments’ reliability and validity within the specific research study. In the study, assessments 

in the control schools were administered by independent evaluators, while assessments in 

treatment schools were administered by the student’s teacher. Each year, a random sample of 

students in treatment schools was re-tested by independent evaluators to determine the reliability 

and validity of the teachers’ assessments (Supovitz, 2003). Interrater reliability was at 0.8 or 

above for all measures except Concepts About Print. According to the study, “The mean scores 

of classroom teachers and independent assessors are also very comparable, with differences 

largely accounted for by the fact that retesting typically took place some three weeks after the 

initial testing, explaining the slightly higher scores of the independent assessors (Hill & Jaggar, 

2003).”
Center for Applied Research and Educational Improvement, University of Minnesota 62

Minnesota Comprehensive Assessments 

Psychometric information. The MCAs have gone through the federal review process 

and the U.S. Department of Education has approved the use of the MCA’s in Minnesota. The 

reliability for the MCA-II (MCA-III data are not yet available) ranges from .89 to .92 depending 

on the grade. The Minnesota Department of Education (MDE) uses a number of methods to 

document validity. Content validity is achieved through an item creation and selection process 

and through the bookmarking process used by MDE to set the scores. In addition, MDE uses a 

model fitting and scaling process to assess unidimensionality. Because it is a criterion-referenced 

assessment, the MCA’s are lacking concurrent or predictive validity studies. They do have 

classification accuracy values that range from 72.9 to 75.0 (Minnesota Department of Education, 

2009).
Center for Applied Research and Educational Improvement, University of Minnesota 63

FastBridge Learning. LLC (FASTTM) Assessments

Description. The Formative Assessment System for Teachers (FASTTM) is a suite of 

cloud-based technology-enhanced assessments with a built-in database and automated reports 

(fastbridge.org). The tools are used in more than 30 states and 8,800,000 test administrations 

were conducted in the 2015-16 academic year alone. FASTTM was initially developed with 

funding from IES and OSEP. It is a system of assessments and online services that offers a 

rigorously-developed, highly efficient, instructionally relevant, easy-to-implement, and userfriendly solution to gather and process data to guide instruction for K–12 students by providing 

universal screening and progress monitoring designed to screen, diagnose, monitor and inform 

instruction. The suite of assessments includes earlyReading, CBMreading, AUTOreading, 

aReading, COMPefficiency, as well as many math, and behavioral measures. An independent 

survey of teachers (n = 2689 responses) by the Iowa Department of Education indicated that 86% 

of educators believe FAST “may” or “will definitely support increased student achievement” 

(Greg Felderman, Iowa Department of Education, personal communication, January 2015).

Psychometric information. For the evaluation, we used letters sounds with K. 

CBMreading and sight words with grades 1 – 3, decodable words with grades 2 - 3. 

Psychometric and use information for these measures are listed on the academic progress 

monitoring section of the National Center on Intensive Intervention’s (NCII) website (NCII, 

2017). These measures have all been shown to have adequate reliability of performance level 

score (median alternate form reliabilities ranging from 0.74 – 0.94 for CBMreading; 0.89 for 

letters sounds; 0.98 for decodable words; and 0.94 for sight words), reliability of slope (median 

reliabilities of 0.70, 0.83, and 0.70 for grades 1 – 3 for CBMreading; 0.59 for letter sounds; 0.75 

for decodable words; and 0.90 for sight words) and validity of performance level (NCII, 2017). 

Please see NCII’s website for more specific validity evidence broken down by the FastBridge 

measure and grade. 
Center for Applied Research and Educational Improvement, University of Minnesota 64

Appendix B 

Propensity Score Matching for Years 3 and 4 

Propensity Score

Percent of Total

0

20

40

60

80

0.0 0.1 0.2 0.3 0.4

Comparison − Year 4 Comparison − Year 3

Sankofa − Year 4

0.0 0.1 0.2 0.3 0.4

0

20

40

60

80

Sankofa − Year 3

Figure B1. Distribution of propensity scored for the matched comparison for Years 3 and 4. 
Center for Applied Research and Educational Improvement, University of Minnesota 65

Table B1 

Baseline Equivalency of Covariates Used in Propensity Score Matching for Years 3 and 4. 

Covariates Standardized Mean 

Differences Before Matching 

Standardized Mean 

Differences After Matching 

Year 3 

Propensity Score 1.0001 0.0347 

Black - - 

Grade Level - - 

LEP flag No 0.5477 0.0000 

LEP flag Yes 0.5477 0.0000 

Same School 0.7304 0.0120 

Female 0.0475 0.24751 

Male 0.0475 0.24751 

Mondo 0.7206 0.10071 

Year 4 

Propensity Score 0.7014 0.0105 

Black - - 

Grade - - 

LEP flag No 0.0379 0.0215 

LEP flag Yes 0.0379 0.0215 

Same School 0.1438 0.0148 

Female 0.1113 0.07541 

Male 0.1113 0.07541 

Mondo 0.6317 0.0259 

1 Following Rosenbaum (2010), standardized differences are based on a pooled standard 

deviation with equal weighting given to the standard deviations of the treatment and comparison 

groups before matching. The standardized mean differences in for sex (i.e., Female and Male) 

exceed the suggested 0.05 threshold but are still below 0.25, which best practices suggests is 

acceptable if we control for them in any outcome analysis. 
Center for Applied Research and Educational Improvement, University of Minnesota 66

Table B2. 

Regression Results of MCA-Reading Scale Scores on Sankofa Participation in Years 3 and 4 

Variables Estimate Standard 

Error 

t-value df p-value 

Intercept 339.23 2.81 120.55 28.30 <.001

Sankofa -7.25 3.60 -2.01 284.19 0.045

Male -3.47 2.31 -1.50 279.25 0.134

Year 4 -3.20 2.40 -1.33 284.47 0.183

Var(School) 45.33 

Table B3. 

Regression Results of MCA-Reading Scale Scores on Sankofa Participation in Years 3 and 4 for 

Full Sample 

Variables Estimate Standard 

Error 

t-value df p-value 

Intercept 305.11 3.19 95.78 145.01 <.001

Sankofa 2.61 2.66 0.98 358.49 0.327

Male -3.10 1.54 -2.01 357.76 0.045

LEP 4.25 2.21 1.92 365.66 0.055

Same School -0.16 2.14 -0.07 328.31 0.942

Mondo 2.65 0.16 16.84 364.01 <.001

Year 4 -2.47 1.57 -1.57 363.56 0.117

Var(School) 32.21 
Center for Applied Research and Educational Improvement, University of Minnesota 67

Appendix C 

Soft-Target Goals for DRA II 

The reading level expectations listed below are to be interpreted as “soft-target” goals for 

expected reading levels within designated time period of the September to June school year. 

These reading level expectations were provided by Saint Paul Public Schools for the 2009-10 

school year. 

(Source: Jan Magrane, Program Manager PreK-6, SPPS District, as presented in N. Hall and K 

Gray, July 2010, NdCAD’s Sankofa Reading Tutorial Program and “Parent Power” Literacy & 

Advocacy Training Workshops: Evaluation Report) 

Grade September November March June 

DRA2 GRL DRA2 GRL DRA2 GRL DRA2 GRL 

Kindergarten 

---------- 

---------

- 

---------- 

---------

- 

1 A 2 B 

Grade 1 2 B 7 E 12 G 16 I 

Grade 2 16 I 18 J 24 L 28 M 

Grade 3 28 M 30 N 34 O 38 P 

Grade 4 38 P 38 P 40 QRS 40 QRS 

Grade 5 40 QRS 40 QRS 50 TU 50 TU

Grade 6 50 TU 50 TU 60 V 60 V 
Center for Applied Research and Educational Improvement, University of Minnesota 68

Appendix D 

Interrupted Time Series Regression Tables 

In Tables D1 – D6, Intercept refers to β1; Time refers to β2, which is time since the start of 

Sankofa in days; Sankofa refers to the level effect, β3; and Time:Sankofa refers to the slope 

effect, β4 in Equation. The grades (Kindergarten, Grade 3, and Grade 4) were dummy-coded 

control variables that were included but not explicitly stated in Equation 1. RI refers to the 

random intercepts and RS, when included, refers to the random slopes. Finally, Residual refers to 

the residual variance. 
Center for Applied Research and Educational Improvement, University of Minnesota 69

Table D1 

Parameter table for Letter Sounds for the level-effect from the listwise deletion analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 41.158 6.108 6.739 17.848 < .001 

Time 0.972 0.443 2.195 115.262 .03

Sankofa 5.658 5.475 1.034 129.582 .303

Kindergarten -24.56 7.203 -3.41 16.871 .003

Cohort 21 -0.861 7.884 -0.109 15.674 .914

Cohort 22 10.82 8.091 1.337 17.913 .198

Cohort 23 5.938 9.89 0.60 16.701 .556

Cohort 24 -10.553 8.409 -1.255 17.21 .226

Cohort 25 -21.954 10.334 -2.124 15.777 .050

Cohort 26 -27.378 13.334 -2.053 15.659 .057

Var(RI) 157.758 

Var(RS) 0.015

Cov(RI, RS) -0.691 

Residual 60.627

Table D2 

Parameter table for Letter Sounds for the slope-effect from the listwise deletion analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 41.132 6.179 6.657 18.602 < .001

Time 0.957 0.707 1.354 132.166 .178

Sankofa 5.582 6.172 0.904 133.545 .367

Kindergarten -24.555 7.204 -3.409 16.866 .003

Cohort 21 -0.873 7.895 -0.111 15.723 .913

Cohort 22 10.825 8.096 1.337 17.906 .198

Cohort 23 5.943 9.892 0.601 16.7 .556

Cohort 24 -10.537 8.452 -1.247 17.548 .229

Cohort 25 -21.958 10.338 -2.124 15.806 .05

Cohort 26 -27.379 13.337 -2.053 15.66 .057

Time:Sankofa 0.023 0.872 0.027 131.308 .979

Var(RI) 157.997 

Var(RS) 0.015

Cov(RI, RS) -0.697 

Residual 61.076
Center for Applied Research and Educational Improvement, University of Minnesota 70

Table D3 

Parameter table for Letter Sounds for the level-effect from the pooled multiple imputation 

analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 24.333 7.649 3.181 4324.656 .001

Time 1.181 0.337 3.508 575.416 < .001

Sankofa 4.409 4.600 0.958 777.733 .338

Grade 1 13.983 6.286 2.224 2610.967 .026

Cohort 21 -3.572 8.552 -0.418 5671.659 .676

Cohort 22 0.265 8.106 0.033 5494.044 .974

Cohort 23 1.381 11.609 0.119 23681.799 .905

Cohort 24 -15.038 8.380 -1.795 5082.234 .073

Cohort 25 -12.79 12.441 -1.028 4137.261 .304

Cohort 26 1.042 10.11 0.103 863.317 .918

Var(RI) 234.956 

Var(RS) 0.634

Cov(RI, RS) -5.664 

Residual 72.983 

Table D4 

Parameter table for Letter Sounds for the slope-effect from the pooled multiple imputation 

analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 24.196 7.678 3.151 4568.128 .002

Time 1.142 0.464 2.464 479.435 .014

Sankofa 4.058 5.31 0.764 719.857 .445

Grade 1 13.983 6.286 2.224 2610.972 .026

Cohort 21 -3.572 8.552 -0.418 5671.672 .676

Cohort 22 0.265 8.106 0.033 5494.050 .974

Cohort 23 1.381 11.609 0.119 23681.807 .905

Cohort 24 -15.038 8.38 -1.795 5082.261 .073

Cohort 25 -12.79 12.441 -1.028 4137.261 .304

Cohort 26 1.042 10.11 0.103 863.316 .918

Time:Sankofa 0.078 0.606 0.129 496.660 .898

Var(RI) 234.955 

Var(RS) 0.634

Cov(RI, RS) -5.664 

Residual 73.015 
Center for Applied Research and Educational Improvement, University of Minnesota 71

Table D5 

Parameter table for Sight Words for the level-effect from the listwise deletion analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 49.975 33.542 1.490 19.873 .152

Time 0.820 0.707 1.160 135.434 .248

Sankofa 0.785 9.987 0.079 135.358 .937

Grade 3 46.689 19.511 2.393 20.255 .027

Cohort 21 -26.233 39.35 -0.667 19.884 .513

Cohort 22 -49.582 51.313 -0.966 19.961 .345

Cohort 23 -45.583 45.515 -1.002 20.007 .329

Cohort 24 -46.205 38.518 -1.200 20.019 .244

Cohort 25 -31.337 40.842 -0.767 19.882 .452

Cohort 26 -7.061 36.756 -0.192 19.900 .850

Var(RI) 1079.798 

Residual 280.917

Table D6 

Parameter table for Sight Words for the slope-effect from the listwise deletion analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 48.981 33.577 1.459 19.928 .160

Time -0.066 1.310 -0.050 134.35 .960

Sankofa -1.819 10.512 -0.173 134.381 .863

Grade 3 46.633 19.518 2.389 20.256 .027

Cohort 21 -26.968 39.375 -0.685 19.907 .501

Cohort 22 -50.684 51.35 -0.987 19.99 .335

Cohort 23 -46.728 45.554 -1.026 20.047 .317

Cohort 24 -45.525 38.542 -1.181 20.039 .251

Cohort 25 -32.182 40.87 -0.787 19.909 .440

Cohort 26 -7.933 36.785 -0.216 19.935 .831

Time:Sankofa 1.198 0.213 1.490 134.336 .423

Var(RI) 1080.51 

Residual 281.625 
Center for Applied Research and Educational Improvement, University of Minnesota 72

Table D7 

Parameter table for Sight Words for the level-effect from the pooled multiple imputation 

analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 51.592 24.025 2.147 13285.252 .032

Time 0.802 0.483 1.662 579.768 .097

Sankofa -2.363 7.238 -0.327 918.208 .744

Grade 3 28.086 7.955 3.531 5172.853 < .001

Cohort 21 -13.52 25.658 -0.527 14845.346 .598

Cohort 22 -23.51 26.231 -0.896 12292.034 .37

Cohort 23 -28.322 26.466 -1.07 18933.062 .285

Cohort 24 -19.766 25.37 -0.779 16073.355 .436

Cohort 25 -21.373 26.006 -0.822 13313.136 .411

Cohort 26 -5.305 25.733 -0.206 11697.662 .837

Var(RI) 516.560 

Var(RS) 0.272

Cov(RI, RS) 0.452

Residual 234.676 

Table D8 

Parameter table for Sight Words for the slope-effect from the pooled multiple imputation 

analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 52.264 24.066 2.172 13167.553 .030

Time 0.995 0.677 1.47 578.192 .142

Sankofa -0.634 8.258 -0.077 922.069 .939

Grade 3 28.086 7.955 3.531 5172.885 < .001

Cohort 21 -13.52 25.658 -0.527 14845.376 .598

Cohort 22 -23.51 26.231 -0.896 12292.049 .370

Cohort 23 -28.322 26.466 -1.07 18933.143 .285

Cohort 24 -19.766 25.37 -0.779 16073.439 .436

Cohort 25 -21.373 26.006 -0.822 13313.139 .411

Cohort 26 -5.305 25.733 -0.206 11697.690 .837

Time:Sankofa -0.384 0.908 -0.423 753.557 .672

Var(RI) 516.551 

Var(RS) 0.272

Cov(RI, RS) 0.452

Residual 234.750 
Center for Applied Research and Educational Improvement, University of Minnesota 73

Table D9 

Parameter table for Decodable Words for the level-effect from the listwise deletion analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 23.080 16.603 1.39 18.498 .181

Time 0.354 0.206 1.718 111.974 .088

Sankofa 0.056 2.527 0.022 127.236 .982

Grade 3 19.079 9.912 1.925 19.501 .069

Cohort 21 -14.788 19.516 -0.758 18.407 .458

Cohort 22 -33.025 25.809 -1.28 19.585 .216

Cohort 23 -21.799 22.693 -0.961 18.934 .349

Cohort 24 -13.278 19.13 -0.694 18.692 .496

Cohort 25 -3.122 20.127 -0.155 18.019 .878

Cohort 26 5.309 18.053 0.294 17.904 .772

Var(RI) 366.95

Var(RS) 0.005

Cov(RI, RS) -0.702 

Residual 13.572

Table D10 

Parameter table for Decodable Words for the slope-effect from the listwise deletion analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 21.708 16.59 1.309 18.475 .207

Time -0.311 0.337 -0.923 132.109 .358

Sankofa -2.86 2.766 -1.034 132.388 .303

Grade 3 18.989 9.902 1.918 19.473 .070

Cohort 21 -14.093 19.509 -0.722 18.417 .479

Cohort 22 -30.277 25.816 -1.173 19.63 .255

Cohort 23 -18.989 22.71 -0.836 19.024 .413

Cohort 24 -11.15 19.137 -0.583 18.757 .567

Cohort 25 -4.222 20.12 -0.21 18.032 .836

Cohort 26 4.246 18.05 0.235 17.932 .817

Time:Sankofa 0.978 0.399 2.453 132.192 .015

Var(RI) 351.502 

Var(RS) 0.004

Cov(RI, RS) -0.579 

Residual 13.567
Center for Applied Research and Educational Improvement, University of Minnesota 74

Table D11 

Parameter table for Decodable Words for the level-effect from the pooled multiple imputation 

analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 21.059 18.783 1.121 10729.922 .262

Time 0.609 0.427 1.426 381.727 .155

Sankofa 0.193 6.207 0.031 567.036 .975

Grade 3 22.581 6.584 3.43 1721.356 .001

Cohort 21 -9.538 20.193 -0.472 9143.931 .637

Cohort 22 -12.677 20.569 -0.616 8899.393 .538

Cohort 23 -18.328 20.814 -0.881 11279.789 .379

Cohort 24 -10.168 19.943 -0.51 10123.685 .610

Cohort 25 -11.456 20.679 -0.554 6163.314 .580

Cohort 26 0.174 20.931 0.008 3397.933 .993

Var(RI) 321.102 

Var(RS) 0.287

Cov(RI, RS) -0.537 

Residual 149.570 

Table D12 

Parameter table for Decodable Words for the slope-effect from the pooled multiple imputation 

analysis. 

Parameter Estimate Standard Error t-value df p-value 

Intercept 20.77 18.831 1.103 10368.144 .270

Time 0.526 0.588 0.895 397.56 .371

Sankofa -0.55 6.946 -0.079 636.642 .937

Grade 3 22.581 6.584 3.43 1721.357 .001

Cohort 21 -9.538 20.193 -0.472 9143.956 .637

Cohort 22 -12.677 20.569 -0.616 8899.422 .538

Cohort 23 -18.328 20.814 -0.881 11279.823 .379

Cohort 24 -10.168 19.944 -0.51 10123.708 .610

Cohort 25 -11.456 20.679 -0.554 6163.33 .580

Cohort 26 0.174 20.931 0.008 3397.941 .993

Time:Sankofa 0.165 0.789 0.209 465.186 .834

Var(RI) 321.100 

Var(RS) 0.287

Cov(RI, RS) -0.536 

Residual 149.593 
Center for Applied Research and Educational Improvement, University of Minnesota 75

Appendix E 

Parent Survey

NdCAD SANKOFA READING PROGRAM 

PARENT EVALUATION 

Your Name: _________________________________Date:________________________________ 

Child’s Nam______________________________________________________________________

Are you participating in Parent Power? _______ Yes

_______ No

Directions: 

If you have more than one child in the Sankofa Reading Program, please fill out the following evaluation 

for each child. We would like to know about each individual child’s progress after participating in the 

Sankofa Reading Program. 

1. Have you noticed an increase in your child’s behavior or attitude since participating in the Sankofa

Reading Program in the following areas? Please write an X in the appropriate column.

Behavior/Attitude Yes No

Don’t 

Know/Haven’t 

Observed 

Sounding out words 

Reading without being told to (independent reading) 

Continuing to try even though reading may be difficult 

Reading to others 

Reading for fun 

Confidence in reading 

Enjoyment of school 

Homework completion 

Knowledge of and respect for African culture and heritage 

African pride 

Positive interactions with other children 

2. Does your child have a stronger connection to your family as a result of having

participated in Sankofa?

_______ Yes 

_______ No 

3. Does your child have a stronger connection to their community as a result of

having participated in Sankofa?

_______ Yes 

_______ No 
Center for Applied Research and Educational Improvement, University of Minnesota 76

4. Use the space below to describe any other changes you have noticed in your child. 

5. Were the following components of Sankofa helpful for your child? Please write an X in the 

appropriate column. 

Program Component Yes No Not Sure 

Instruction on phonemic awareness 

Instruction on 7 comprehension strategies 

Learning about African heritage 

Positive learning environment 

High expectations from Sankofa staff 

Low student/tutor ratio 

Receiving a weekly note from the tutor 

6. Please list any other parts of the Sankofa that you found helpful for your child. 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

7. What could we do to improve Sankofa for future students? 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

8. Would you recommend Sankofa to family members or friends? Please circle your response. 

 Yes No Not Sure

9. Please rate your overall satisfaction with Sankofa. Please circle one. 

Very Dissatisfied Dissatisfied Neutral/Not Sure Satisfied Very Satisfied

 Please tell us why you gave the Sankofa this rating. 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

10. Please use the space below to write any other comments/suggestions you have about Sankofa. 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 

___________________________________________________________________________________ 
Center for Applied Research and Educational Improvement, University of Minnesota 77
Center for Applied Research and Educational Improvement, University of Minnesota 78

Appendix F 

Example Lesson Plan
Center for Applied Research and Educational Improvement, University of Minnesota 79

Appendix G 

Sankofa Observation Checklist

 Table G1. 

Observation checklist for Cohort: 16, Group: Maxfield 

Checklist Item Date 

10/17/16 11/02/16 11/22/16 

1. Does it look like planned program is going on? X X X 

2. Is the daily schedule posted clearly for all to see? X X X 

3. Does the daily plan list the learning objectives for 

the session, e.g. Specific reading skills, strategies, 

activities and supplies? 

X X X 

4. Are culturally rich program materials displayed in 

a planned and organized manner? X X X 

5. Are there culturally specific leveled books 

displayed. 

No 

display 

X 

Two books in 

basket 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or 

child/child? 

X X 

X 

Lead tutor had great 

connection with a 

couple 

7. Was I greeted when I entered the classroom? Who 

greeted me? Was I greeted with Hotep? 

X Only 

lead 

tutors 

X 

Tutors and 

students 

X 

Student distracted 

everyone in room. 

8. Are you observing cultural routines and rituals? X X X 

9. Are connections to child’s home and community 

being reinforced (through verbal, visual, etc....)? X X X 

10. Are the separate group activities going on? X 

X 

Silent 

reading, 

helping with 

letter 

games/sounds 

X 

11. Is it clear what type of activities are taking place 

in each section of the room? X X X 

12. Is the sign in/attendance sheet in the same place 

every session? X X X 

13. Is the snack in the same place every session? X X X 

Tutor Responsibilities: 10/17/16 - Main focus was teaching students their phonemic sounds because 

most of the students were non-readers.11/2/16 – Lead tutor not participating in rituals setting up lesson 

groups. 11/22/16 – All fully participated in rituals. 
Center for Applied Research and Educational Improvement, University of Minnesota 80

Table G2. 

Observation checklist for Cohort: 17, Group: Jackson.

Checklist Item Date 

10/18/16 11/16/16 12/6/16 

1. Does it look like planned program is going on? X X X 

2. Is the Daily schedule posted clearly for all to 

see? X X X 

3. Does the daily plan list the learning objectives 

for the session, e.g. Specific reading skills, 

strategies, activities and supplies? 

X X X 

4. Are culturally rich program materials 

displayed in a planned and organized manner? 

X 

On small window sill X X 

5. Are there culturally specific leveled books 

displayed. X 

X 

Two books in 

basket 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or 

child/child? 

X 

Tutors well engaged 

with students 

X 

X 

Lead tutor 

had great 

connectio

n with a 

couple 

7. Was I greeted when I entered the classroom? 

Who greeted me? Was I greeted with Hotep? 

X 

After prompted by 

tutors 

X 

Tutors and students 

X 

Student 

distracted 

everyone 

in room. 

8. Are you observing cultural routines and 

rituals? 

X 

Very good X X 

9. Are connections to child’s home and 

community being reinforced (through verbal, 

visual, etc....)?

X 

Especially during 

comp time strategy 

X X 

10. Are the separate group activities going on? X 

X 

Silent reading, 

helping with letter 

games/sounds 

X 

11. Is it clear what type of activities are taking 

place in each section of the room? X X X 

12. Is the sign in/attendance sheet in the same 

place every session? 

X 

Assist. Tutor takes 

attendance 

X X 

13. Is the snack in the same place every session? 

X 

Set up before kids 

arrived 

X X 

 Tutor Responsibilities: N/A 
Center for Applied Research and Educational Improvement, University of Minnesota 81

Table G3. 

Observation checklist for Cohort: 18, Group: Ben E. Mays

Checklist Item Date 

10/26/16 11/15/16 12/14/16 

1. Does it look like planned program is going on? X X X 

2. Is the Daily schedule posted clearly for all to see? X X X 

3. Does the daily plan list the learning objectives for the 

session, e.g. Specific reading skills, strategies, activities and 

supplies? 

X X X 

4. Are culturally rich program materials displayed in a 

planned and organized manner? X X X 

5. Are there culturally specific leveled books displayed. X X 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or child/child? X X X 

7. Was I greeted when I entered the classroom? Who greeted 

me? Was I greeted with Hotep? X X X 

8. Are you observing cultural routines and rituals? X X X 

9. Are connections to child’s home and community being 

reinforced (through verbal, visual, etc....)? X X X 

10. Are the separate group activities going on? X X X 

11. Is it clear what type of activities are taking place in each 

section of the room? X X X 

12. Is the sign in/attendance sheet in the same place every 

session? X X X 

13. Is the snack in the same place every session? X X X 

Tutor Responsibilities: 10/26/16 – Lead tutor lead whole group reading a modeled comprehension strategy. Lead tutor 

took attendance and managed snack time. 11/15/16 – Lead tutor reinforced cultural practices and made connections to 

children’s homes during whole group reading time. Assistant tutors managed transitions and supported children’s skills 

development. 12/14/16 – Lead tutor reviewed lessons learned from previous weeks. Culturally rich materials were set up 

throughout the space. Assistant tutors redirected children to stay on track and modeled good behavior. 
Center for Applied Research and Educational Improvement, University of Minnesota 82

aTable G4 

Observation checklist for Cohort: 20, Group: Saint Paul City School

Checklist Item 

Date 

2/27/17 3/8/17 3/22/17 4/12/17 

1. Does it look like planned program is going on? X 

X 

Lesson well 

thought and 

implemented 

X X 

2. Is the Daily schedule posted clearly for all to 

see? X X X X 

3. Does the daily plan list the learning objectives 

for the session, e.g. Specific reading skills, 

strategies, activities and supplies? 

X X 

X 

All 

learning, 

reading 

strategies 

completed 

X 

4. Are culturally rich program materials displayed 

in a planned and organized manner? X X X X 

5. Are there culturally specific leveled books 

displayed. 

Cultural books 

were present 

but not 

displayed 

X 

Books are 

displayed in 

same spot 

X 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or 

child/child? 

X X 

X 

Student dis. 

reading 

strategies/ 

learning 

obj. 

themselves. 

X 

7. Was I greeted when I entered the classroom? 

Who greeted me? Was I greeted with Hotep? X X X X 

8. Are you observing cultural routines and rituals? 

X 

Students are 

shy about 

participating 

X 

Students are 

leading call and 

response and 

Nguzo Saba 

X X 
Center for Applied Research and Educational Improvement, University of Minnesota 83

Table G4 

Observation checklist for Cohort: 20, Group: Saint Paul City School (Continued)

9. Are connections to child’s home and 

community being reinforced (through verbal, 

visual, etc....)?

X X 

X 

Tutor does 

terrific job 

connecting 

values & 

principles to 

students’ home 

and school lives. 

X 

10. Are the separate group activities going on? X X X X 

11. Is it clear what type of activities are taking 

place in each section of the room? X X X X 

12. Is the sign in/attendance sheet in the same 

place every session? X X X X 

13. Is the snack in the same place every session? X X X X 

Tutor Responsibilities: 3/22/17 – Overall wonderful job! 4/12/17 – Every student is excited about 

coming to the program and are even promoting the program to their peers! 
Center for Applied Research and Educational Improvement, University of Minnesota 84

Table G5 

Observation checklist for Cohort: 21, Group: Ben E. Mays 

Checklist Item Date 

3/30/17 3/8/17 2/28/17 4/20/17 

1. Does it look like planned program is 

going on? 

X 

Tutors are 

implementing 

lesson plans 

effectively 

X Tutors lessons 

seem 

uncoordinated 

X 

2. Is the Daily schedule posted clearly for 

all to see? X X Daily schedule not posted X 

3. Does the daily plan list the learning 

objectives for the session, e.g. Specific 

reading skills, strategies, activities and 

supplies? 

X X 

X Learning ob. 

Displayed but not 

implemented/time 

management 

X 

4. Are culturally rich program materials 

displayed in a planned and organized 

manner? 

X X 

No program 

materials are 

displayed 

X 

5. Are there culturally specific leveled 

books displayed. X X 

No cultural 

leveled books are 

displayed 

X 

6. Did you observe rich and meaningful 

dialog among/within the group e.g. 

adult/child and/or child/child? 

X 

Students are using 

comprehend. 

Strategy when 

discussing life at 

home & comm. 

X X X 

7. Was I greeted when I entered the 

classroom? Who greeted me? Was I 

greeted with Hotep? 

X X 

I was not greeted 

with Hotep even 

after prompt 

X 

8. Are you observing cultural routines and 

rituals? X 

X 

Students are 

more engaged 

in routines and 

rituals 

Students did not 

participate in 

rituals 

X 

9. Are connections to child’s home and 

community being reinforced (through 

verbal, visual, etc....)?

X X X X 

10. Are the separate group activities going 

on? X X X X 
Center for Applied Research and Educational Improvement, University of Minnesota 85

Table G5 

Observation checklist for Cohort: 21, Group: Ben E. Mays (Continued)

11. Is it clear what type of activities are taking 

place in each section of the room? 

X 

Students with 

similar reading 

levels are grouped 

together 

X X X 

12. Is the sign in/attendance sheet in the same 

place every session? X X X X 

13. Is the snack in the same place every 

session? X 

X Designated 

Sankofa area 

for snack 

X Snack in 

cafeteria at 

different 

tables 

X 

Tutor Responsibilities: N/A 
Center for Applied Research and Educational Improvement, University of Minnesota 86

Table G6 

Observation checklist for Cohort: 22, Group: Maxfield

Checklist Item Date 

4/11/17 5/16/17 5/22/17 

1. Does it look like planned program is going on? X X X 

2. Is the Daily schedule posted clearly for all to 

see?

X 

On wall near 

door 

X X 

3. Does the daily plan list the learning objectives 

for the session, e.g. Specific reading skills, 

strategies, activities and supplies? 

X 

Yes X X 

4. Are culturally rich program materials displayed 

in a planned and organized manner? 

X 

On a small table X X 

5. Are there culturally specific leveled books 

displayed. X X X 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or 

child/child? 

X 

Yes awesome 

group of men 

tutors 

X 

Many quest., 

much 

conversation, 

many giggles 

X 

Wonderful 

conversations 

7. Was I greeted when I entered the classroom? 

Who greeted me? Was I greeted with Hotep? 

Only staff 

greeted me 

X 

Staff prompted 

students 

X 

All greeted me 

8. Are you observing cultural routines and rituals? X X X 

9. Are connections to child’s home and community 

being reinforced (through verbal, visual, etc....)? Not Yet X X 

10. Are the separate group activities going on? X X X 

11. Is it clear what type of activities are taking 

place in each section of the room? X X X 

12. Is the sign in/attendance sheet in the same 

place every session? 

X 

Yes near door X X 

13. Is the snack in the same place every session? X X X 

Tutor Responsibilities: 4/11/17 – Bring new pack of comprehension strategies. 5/16/17 – Bring more 

prizes and free books. 
Center for Applied Research and Educational Improvement, University of Minnesota 87

Table G7 

Observation checklist for Cohort: 23, Group: Jackson

Checklist Item Date 

4/12/17 5/17/17 5/31/17 

1. Does it look like planned program is going on? X X X 

2. Is the Daily schedule posted clearly for all to 

see?

X 

On white board X X 

3. Does the daily plan list the learning objectives 

for the session, e.g. Specific reading skills, 

strategies, activities and supplies? 

X X X 

4. Are culturally rich program materials displayed 

in a planned and organized manner? 

X 

Window sill X X 

5. Are there culturally specific leveled books 

displayed. 

X 

On long table 

near white board 

X X 

6. Did you observe rich and meaningful dialog 

among/within the group e.g. adult/child and/or 

child/child? 

X X X 

7. Was I greeted when I entered the classroom? 

Who greeted me? Was I greeted with Hotep? 

X 

Prompted by 

tutor 

X 

Yes, even by 

name 

X 

8. Are you observing cultural routines and rituals? X X X 

9. Are connections to child’s home and community 

being reinforced (through verbal, visual, etc....)? Not Yet 

X 

Many 

connects made 

by students 

X 

10. Are the separate group activities going on? 

X 

Two small 

groups 

X 

Three small 

groups 

X 

11. Is it clear what type of activities are taking 

place in each section of the room? 

X X X 

12. Is the sign in/attendance sheet in the same place 

every session? 

X 

Table near white 

board 

X X 

13. Is the snack in the same place every session? X 

Cafeteria X X 

Tutor Responsibilities: 4/12/17 – Bring phonemic book to tutor for lesson planning. 

 5/31/17 – Assessment materials.