Pre-School-U: Implementation Evaluation 

Findings

A Program by Detroit Public Television

August 31, 2014

Prepared by the Michigan Public Health Institute
1 

Introduction

In 2012, the United Way for Southeastern Michigan (UW SEM) was awarded a grant from the Social 

Innovation Fund (SIF). UWSEM distributed these funds to 11 sub-grantee organizations, all of which 

aimed to improve school readiness among children, ages zero to five within 10 regions in metropolitan 

Detroit. Detroit Public Television (DPTV) is a sub-grantee of UWSEM, implementing Pre-School-U: A 

Program for Parents and Caregivers (PSU). In collaboration with the Interfaith Leadership Council of 

Metropolitan Detroit (IFLC), DPTV endeavored to address the relationship between poverty, 

parenting/caregiving, and children’s academic outcomes through the PSU program. DPTV aimed to 

recruit 900 (450 in each of the comparison and treatment groups) low-income parents/caregivers for

study participation over the five year implementation period. Broadly, PSU integrates media and a 

facilitated, educational curriculum for parents and caregivers with the aim of improving children’s

language, cognitive, socio-emotional, and communication skills. 

At the inception of this project, PSU had not been evaluated for its implementation processes, whether 

its underlying theoretical assumptions are valid, or for its ability to impact participants. Therefore, DPTV 

partnered with the Michigan Public Health Institute (MPHI) to prepare a sub-grantee evaluation plan 

(SEP). The study design outlined in the SEP focuses on implementation and formative evaluation prior to 

evaluating impacts. Specifically, over the five year project period, year 2 was designated as a pilot year, 

with only process evaluation activities taking place and impact evaluation commencing in subsequent 

years of the project.

The following report covers the period spanning September 1, 2013 to August 31, 2014 (year 2), and 

provides an overview of program progress and implementation evaluation findings to date. Findings 

described provide insight into aspects of study context as well as implementation dimensions such as 

fidelity and program dosage. Evaluation results are also key to describing specific phases of program 

implementation, detecting barriers and key success factors, and identifying lessons learned and 

recommendations for the future. Data sources for the implementation evaluation included 

observations, key informant interviews, attendance logs, and surveys. While the following narrative 

provides a comprehensive overview of these methods, information gathered through the evaluation 

was provided to DPTV and collaborators on an ongoing basis to inform continuous process monitoring 

and program improvement. 

Background

PSU aims to provide caregivers and parents with the knowledge and skills they need to best prepare 

their children for school. As initially conceived, the intervention consists of six hours of programming

(i.e. 6 sessions) with each session focusing on a specific theme. The sessions are carried out by trained 

peer facilitators from the local community. In each session, caregivers watch a series of video vignettes 

followed by facilitated discussion. Caregivers are also given an opportunity to practice skills and role play 

in a safe and open environment. Participants are encouraged to provide praise and support for one 

another during the sessions. 
2 

Pre-School-U integrates elements of Social Learning Theory (SLT) in order to bring about behavioral 

change. Pre-School-U focuses on constructs central to SLT including observational learning, outcome 

expectancies, and self-efficacy. SLT also suggests that skill building and self-efficacy can be enhanced by 

the presence of supportive peers. Thus, Pre-School-U emphasizes the importance of skilled facilitation

and participants having the opportunity to give and receive social support. 

One of the unique aspects of PSU is that the program is implemented in faith-based settings, which gives 

the program potential to reach parents and caregivers that are disconnected from traditional 

educational settings. To execute this aspect of the program, one organization, the Interfaith Leadership 

Council (IFLC), is responsible for identifying member religious organizations to implement PSU. The IFLC 

congregation coordinator works with each congregation to identify and provide ongoing support to 

volunteers who are responsible for facilitating PSU at their respective sites. DPTV also provides an initial 

orientation and training session to all PSU lay facilitators. In addition, DPTV oversees implementation 

and provides support to the congregation coordinator. 

The logic model, presented in Appendix A, further outlines the mechanisms by which PSU will address 

the problem of young children’s significant under-preparedness for school. 

Evaluation Approach

Because PSU had not previously been evaluated for its processes or ability to produce outcomes, the 

SEP details a multi-pronged approach. The evaluation has two distinct purposes: 1) to refine and 

describe the PSU curriculum for caregivers and 2) to evaluate its outcomes. To achieve these goals, 

Implementation and formative evaluation activities focus on exploring program fidelity, including 

adherence (e.g. content, dose, coverage) and moderators of adherence, such as intervention 

complexity, implementation facilitation, quality of delivery, participant responsiveness, recruitment, and 

context. Formative evaluation also strives to identify essential program components as well as outputs 

and possible covariates. 

Implementation Evaluation

The PSU program design gives facilitators a great deal of flexibility in how the curriculum is carried out; 

as such, it was expected that some adaptations would be made at the ground level. Therefore, the 

evaluation during the first year of programming (year 2) focused primarily on intervention development. 

Specifically, evaluation activities were designed to gauge fidelity of implementation. Evaluation of 

program fidelity plays an important role in intervention development as it includes systematic efforts to 

review, critique, and revise the theoretical underpinnings and technical ingredients of intervention 

models.1 According to the framework for implementation fidelity developed by Carroll and colleagues,2

program fidelity can be best understood as adherence, i.e., how closely those responsible for delivering 

an intervention actually adhere to the intervention as outlined by its designers. Adherence includes 

 

1 Kazdin, A. E. (1994). Psychotherapy for children and adolescents. 

2 Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for 

implementation fidelity. Implementation Science, 2(40), 1-9. 
3 

several subcategories such as content, frequency, and duration and coverage. The level of adherence 

may be moderated by several variables such as the intervention complexity, facilitation strategies, 

quality of delivery, and participant responsiveness.2 This framework suggests that the assessment of 

fidelity of implementation should also involve the identification of components that are essential for the 

intervention to produce the intended outcomes. Similar to what others have done,3 recruitment and 

context will be assessed as additional moderators. 

The evaluators employed a qualitative case study design in which observation of the program in 

operation and interviews formed the principle means of data collection. Trained observers from the 

evaluation team spent time observing program implementation over the course of one year. To 

supplement observations, program facilitators also participated in key informant interviews. Other 

information, such as participant attendance and facilitator satisfaction with training provided, was

collected via attendance logs and surveys. 

The evaluation team initially adopted a troubleshooting strategy with the goal of providing timely 

feedback to program developers4

. Systematic data collection was followed by rapid analysis for key 

themes, such as those reflecting elements of the PSU model, its theoretical basis, and prescribed 

program procedures. Other themes included program successes and barriers. Results and 

recommendations were provided to program stakeholders on an ongoing basis in order to equip them 

to remedy problems and refine programming. 

After the first iteration of PSU, the program was significantly modified. The evaluation team had 

planned to conduct additional qualitative data collection in order to provide an analysis of the program 

once it reached stability. This second phase of the evaluation did not occur as data collection was 

reduced after the sub-grantee announced that it would be leaving the SIF. Therefore, the findings 

included in the current report are limited to the results of the rapid data collection and feedback which 

were provided to stakeholders as part of the troubleshooting strategy. 

Impact Evaluation

Following implementation evaluation, the SEP outlined impact evaluation was planned to commence in 

year 3. Impact evaluation was to utilize a quasi-experimental design to evaluate PSU in terms of its 

anticipated program impacts (i.e. kindergarten readiness) as well as those specified in the program logic 

model (see Appendix A). To maximize the ability to make causal inferences about PSU and observed 

outcomes, the evaluation design was to include pre-post testing, matched comparison groups, and 

triangulation of data sources. Sampling and group assignment would be done at the congregation-level. 

To build a moderate level of evidence supporting PSU by year 5, this evaluation planned to utilize a 

between-group quasi-experimental design to evaluate anticipated program impacts as specified by the 

 

3 Hasson, H. (2010). Systematic evaluation of implementation fidelity of complex interventions in health and social 

care. Implement Sci, 5(1), 67. 4 Chen, H. T. (Ed.). (2005). Practical program evaluation: Assessing and improving planning, implementation, and 

effectiveness. Sage.
4 

logic model. These impacts relate to the program’s ability to increase caregivers’ knowledge, selfefficacy, social support, stress, and positive interactions which promote school readiness. 

Since it is not feasible to randomly assign sites to serve as control groups, a quasi-experimental design 

would have offered the strongest analyses for building a moderate level of evidence for PSU

effectiveness. To carry out this plan, PSU would incorporate sites on a rolling basis (e.g. new sites begin 

PSU). As seen in Table 1, 20 new congregations would have be added each year with 10 serving as the 

intervention group and 10 serving as the comparison group. 

Table 1. Impact Evaluation Sampling Schedule

Congregation Year 3 Year 4 Year 5

1-10 Treatment

11-20 Comparison

21-30 Treatment

31-40 Comparison

41-50 Treatment

51-60 Comparison

Data Collection and Measurement

For implementation evaluation occurring in year 2, evaluators utilized a qualitative case study design in 

which observation of the program in operation and interviews formed the principle means of data 

collection. Case study methods sought to provide a holistic picture of the implementation of PSU. Most 

importantly, the evaluators used case study methods as a formative evaluation approach whereby 

findings from observations and key informant interviews were rapidly fed back to programmers so that 

information could be used for program improvement. 

Observations. Trained observers from the evaluation team spent time observing the program 

implementation over the course of one year. Observations provided an opportunity to 

systematically describe events, behaviors, and other characteristics occurring during PSU 

implementation within the social setting in which it occurred. Further, the observations focused 

on the extent that facilitators were prepared to implement the program with respect to 

strategies consistent with Pre-School-U’s theoretical underpinnings. These facilitation strategies 

include providing participants with opportunities to practice skills, conducting role plays, 

facilitating problem-solving, and encouraging mutual support. Observation also helped inform 

the development of questions for subsequent key informant interviews. Two observations took 

place in the Fall and early winter of 2013 during the first cycle of PSU implementation at two 

different congregations within the SIF regions. Two additional observations occurred during the 

Spring of 2014 at one preschool location, outside of the SIF regions, to help inform ongoing 

program development.

Key Informant Interviews. Key informant interviews with program facilitators occurred following 

PSU implementation. Interview questions were open-ended and focused on program fidelity, 

adaptation, and outcomes/impacts. Interviews tended to last about 45 minutes and were
5 

conducted via phone. Evaluators received IRB permission to waive documentation of consent; 

therefore, participants provided verbal consent. Evaluators conducted three interviews with 

program facilitators at each initial implementation site in late December, 2013 and early 

January, 2014. The program facilitator for PSU at the preschool location also participated in 

informal, ongoing discussion with evaluators after implementing several of the PSU sessions. 

This provided an opportunity for, both, evaluators and program facilitators to maintain an 

ongoing exchange of information with one another.

Attendance Logs. Attendance logs were maintained for each session of PSU to assess participant 

engagement, program dosage, and the general composition of program participants. 

Intake Forms. Intake forms provided demographic information such as age, gender, and race. In 

addition, participants were asked to indicate their caregiver role (i.e. parent, grandparent, foster 

parent, etc.)

Meeting Notes and Quarterly Progress Reports. Notes from program staff meetings and subgrantee quarterly progress reports were collected and used to understand implementation 

barriers at an administrative level. 
6 

Evaluation Findings

Timeline of Program Implementation

To provide the reader with a sense of context, a timeline of program implementation is provided. In 

preparation for PSU implementation, facilitator training occurred on September 23, 2013 and included 

seven congregation facilitators, representing four congregations. 

Following training, PSU implementation began in the Fall of 2013 at three religious congregations 

located within the SIF target zip codes. Implementation sites included: the Muslim Center of Detroit, 

Christ the King Catholic Church, and the Palmer Park Ward of the Church of Jesus Christ of Latter Day 

Saints. Two out of the three congregations completed the 6 planned PSU sessions (1 hour each) by the 

end of January 2014. One congregation, however, only implemented 3 sessions within this timeframe 

due to significant challenges with recruitment and retention.

Once the first round of implementation was completed, implementation evaluation findings were 

reviewed and used to inform program modifications during January and February of 2014. The modified 

3-session PSU curriculum was then piloted at PerrinVille preschool located in Livonia, Michigan. 

Although this site is not located in the SIF target regions, it afforded the opportunity to further pilot and 

refine programming without influencing the number of participants who could participate in PSU during 

the impact evaluation. 

In April, 2014, DPTV began discussions to leave the SIF portfolio. The decision to shift from previous 

plans to focusing on working to sustain the program after federal funding ends was made. To pilot a 

fee-for-service model, implementation was extended to Head Start centers. In preparation for this 2nd

cycle of PSU implementation within SIF regions, facilitator trainings were conducted and included 25 

facilitators from Wayne Metro Head Start centers. The trainees included 4 facilitators from 2 SIFfunded sites. The 2nd training for congregations was held on April 29th and included 7 congregation 

facilitators from three congregations. 

Following these trainings, 2 Head Start centers and 4 congregations located within SIF-funded zip codes 

implemented the revised 3-session PSU curriculum. 

PSU Facilitator Training

The intent of the facilitator training was to prepare facilitators for implementation of PSU. Evaluators 

observed the training provided to seven congregation facilitators. In addition, the facilitators were asked 

to complete a satisfaction survey after the training completion. Overall, the facilitators’ evaluation of 

the training was favorable. The majority felt the training was taught at the right level (100%), that the 

length of the training was appropriate (86%), and that the training was “good” (57%) or “excellent” 

(43%). As a result of the training, 14% strongly agreed and 71% agreed that they felt confident in their 

ability to facilitate PSU sessions. Results from the evaluator’s observations indicated that more 

preparation in the area of facilitation may be needed. Specifically, the evaluators suggested that greater 

emphasis be placed on training the facilitators to provide problem-solving activities and skill building 

opportunities during their implementation of PSU. In addition, little attention was paid to recruiting 
7 

hard-to-reach populations during the trainings. The majority of recruitment strategies which were 

discussed relied on in-tact networks of parents/caregivers within the congregation setting. The 

evaluators expressed concern those parents/caregivers most in need of PSU would not be reached 

unless more targeted recruitment strategies were adopted. These suggestions were provided to the 

congregation coordinator who provided ongoing support to each of implementation sites. 

Following the first cycle of PSU implementation, facilitators also provided feedback on the trainings 

during key informant interviews. During the interviews, facilitators were asked how they felt the training 

did or did not prepare them for implementation of the program. All interview participants reported that

the training was informative, easy to understand and useful for implementing PSU. Interview 

participants felt that the materials were clearly explained and facilitators were satisfied with the training 

overall. After attending the training, however, all three participants reported feeling as though they did 

not understand fully the nature and extent of PSU programing. This was a significant barrier when trying 

to describe the program to prospective participants and portray the program in a convincing way. 

Additional training and/or materials may have been helpful to facilitators in marketing more effectively 

and generating excitement around the program. All three facilitators reported to work or had worked in 

educational settings and felt that more extensive training may be needed for those without this 

professional background. In particular, facilitators noted the great importance of understanding how to 

incorporate interactive activities throughout the program and noted additional ideas for handson/physical activities should be described during training. 

After the initial round of implementation, and further program piloting at PerrinVille preschool during 

the winter months, a second round of trainings were conducted with 28 trainees from Head Start 

Centers and 7 trainees from congregations. Training participants were asked to complete a survey 

consisting of the following domains: 1) Satisfaction with training 2) Understanding of the Pre-School-U 

Curriculum, 3) Perceived benefit of program 4) Self-efficacy to implement Pre-School-U, and 5) 

Anticipated Barriers. On average, the results suggest that Head Start participants were generally 

satisfied with all aspects of the training. Participants were particularly satisfied with the presenter’s 

preparedness and ability to effectively communicate, as well as the level of instruction.

Most participants either agreed or strongly agreed to understanding the goals of Pre-School-U after 

completing the training. Similarly, the majority agreed or strongly agreed that the program will be of 

benefit to the parents and caregivers within their communities. The majority of participants reported to 

be somewhat or very confident that they could successfully implement the program at their sites. 

Participants felt strongest in their abilities to facilitate workshops in an interactive fashion (imparting 

knowledge, leading discussion, and facilitating skill-building activities), but were slightly less confident in 

the area of recruiting parents and caregivers.

Two themes emerged regarding anticipated barriers to implementation. The first theme related to 

recruitment and retention with six comments falling into this category. Comments expressed concerns 

in getting parents and caregivers to enroll and maintain participation throughout the duration of the 

program. Cultural appropriateness and language barriers constituted a second theme which included 

three comments. These participants expressed a need for materials to be translated into Arabic and 

related difficulties associated with having to deliver the program in two languages simultaneously.
8 

Observations confirmed that the training session was well-organized, clearly presented, and highly 

interactive. The trainees appeared to be very engaged and the facilitators were proactive in helping 

trainees problem-solve around implementation concerns unique to their diverse settings. 

Recruitment & Retention

Following training, facilitators were charged with recruiting parents and caregivers for participation in 

PSU. In total, 88 parents and caregivers were recruited into PSU. Participant characteristics are 

displayed in table 1. The vast majority of participants were female (84%). PSU attracted diverse age 

groups with 33% of participants in their twenties, 25% were ages 30-39, and 33% aged 40 and above. 

The average participant age was 39. The racial composition was largely Black, which represented 58% of 

parents and caregivers. The majority of participants indicated to be a parent (72%) or a grandparent 

(16%) of a pre-school aged child. 

Table 1 

PSU Participant Characteristics (N=88)

Variable N % 

Gender

Male 14 84%

Female 74 84%

Age (Mean=39)

20-29 29 33%

30-39 22 25%

40-49 13 15%

50-59 2 2%

60 and over 14 16%

Missing/Refused 8 9%

Race

Black 51 58%

White 14 16%

Hispanic 6 7%

Native American 1 1%

Other 12 14%

Missing/Refused 4 5%

Relationship to Child

Parent 63 72%

Grandparent 14 16%

Aunt 3 4%

Other 3 4%

Missing/Refused 5 6%

Location 

Congregation 57 65%

Head Start 31 35%
9 

The evaluators monitored recruitment and retention closely in order to determine the extent that the 

program will be able to recruit and retain the number of participants required for future impact 

evaluation. In order for the planned impact evaluation to achieve its analytic goals, each 

implementation of PSU would need to recruit and retain 15 parents/caregivers and to retain 80% of 

participants in all of the offered sessions. Table 2 summarizes recruitment and retention results for the 

3 sites implementing PSU during the fall and the 6 sites, which implemented the refined program during 

the spring months. During the fall cycle, the 3 sites implementing PSU recruited an average of 8 

participants into the program. The mean number of sessions attended (dose received) was 1.3 of 6 

planned sessions (dose administered). Looking at dose-received another way, the average participant 

received 26% of the program. No participants during the fall cycle received all 6 sessions. During the 2nd

cycle of implementation with the 3-session format, both recruitment and retention improved. The 

average number of recruited participants increased from 7.7 to 10.8 and the dose received increased 

from 26% to 70% of the available programming. However, only 42% of parents/caregivers were 

retained in all workshop sessions. 

Table 2

Recruitment & Retention by Implementation Cycle

Cycle # of Sites

Avg. # 

Participants 

per Site

Avg. # of 

Sessions 

Attended

Mean % of 

Sessions 

Attended

% Attending 

All Sessions 

Fall (6 Workshop Format) 3 7.7 1.3 21% 0

Spring (3 Workshop Format) 6 10.8 2.1 70% 42%

Table 3 displays the number of participants recruited, the average number of workshops attended per 

participant, the mean percent of workshops attended per participant, and the percent of participants

attending all workshops. The most successful implementation with respect to retention and recruitment 

was the Church of Latter Day Saints (LDS) in Palmer Park in their 2nd round of PSU. This site recruited 16 

parents/caregivers with the average participant attending 2.6 of the 6 workshops which were offered. 

Of the 16 participants, 69% attended all three workshops. None of the implementation sites were able 

to achieve the target recruitment and retention rate. 
10

Table 3 

Recruitment & Retention by Implementation Site

Site

# of 

Participants 

Recruited

Avg. # of 

Sessions 

Attended

Mean % of 

Sessions 

Attended

% Attending 

All Sessions

Fall Cycle (6 Session Format)

LDS Palmer Park 11 1.6 27% 0%

Christ the King Church 6 1.0 17% 0%

Muslim Center 6 1.0 17% 0%

Spring Cycle (3 Session Format)

LDS Palmer Park 16 2.6 85% 69%

Wayne Metro Head Start Hamtramck 16 2.3 77% 50%

Wayne Metro Head Start Courtland 15 1.1 38% 0%

St. Johns Presbyterian Church 9 2.6 85% 67%

Calvary Presbyterian 6 2.3 78% 33%

Christ the King Church Round 2 3 1.0 33% 0%

As these data indicate, recruitment and retention posed significant problems during the first cycle of 

implementation as no sites were able to recruit and retain the number of participants which would be 

required for impact evaluation. To inform program improvement, facilitators were asked to describe 

their recruitment efforts, including what they found to be successful and areas for improvement. 

Facilitators reported using multiple methods for recruitment such as announcements during religious 

functions and on Facebook, flyers, one-on-one interactions, etc. Despite the multiple tactics employed, 

facilitators found recruitment and retention to be extremely difficult and the biggest barriers to program 

implementation.

Complicating recruitment efforts was the fact that program logistics were confirmed too close to the 

actual PSU start date (i.e. participants and facilitators had little time to plan). Further, facilitators felt 

that materials, ideas, and time for marketing the program were insufficient for effectively 

communicating the nature and purpose of PSU to prospective participants.

As facilitators marketed the program, it became apparent that calling PSU a “parenting program” was 

intimidating or a turn-off to prospective participants. Facilitators felt participating in a “parenting 

program” implied that something may be wrong with participants’ current caregiving skills. Additionally, 

facilitators felt that six sessions may have been too many and deterred participants. Program incentives 

also may not have been sufficient for attracting participants. 

Facilitators felt recruitment efforts were not successful for reaching those individuals in greatest need 

for a program like PSU; therefore, PSU was not believed to be reaching its intended audience.

Facilitators reported that those who did attend were already competent in their caregiving skills and, 

thus, became disinterested with program content quickly. Participants wanted PSU to add something 

additional to their existing skill set; facilitators felt the program was not designed to appeal to those at a 

higher level of skill. 
11

Some of those who agreed to participate during recruitment did not show up at the start of 

implementation. Facilitators reported these individuals expressed concerns regarding childcare and 

discomfort with bringing their children to PSU sessions. While transportation could have been arranged 

for participants, one facilitator noted this program feature was not well-communicated or understood 

among participants. 

Analysis of notes from program staff meetings and communications between DPTV and the evaluators 

indicated that many of the aforementioned challenges were attributed to a lack of capacity on the part 

of the IFLC. Indeed, the IFLC was responsible for employing and supervising the congregation 

coordinator whose job it was to support each site in their implementation of PSU. From DPTV’s 

perspective, based upon the coordinator’s interview and professional experience, this individual was 

well equipped to address these issues but did not provide the level of assistance which was needed. The 

congregation coordinator then resigned before the first cycle of implementation was complete, which 

further limited IFLC’s ability to support PSU’s implementation. 

Exploration of these challenges helped to inform strategies for improving recruitment and retention 

after the first cycle of PSU. Suggested improvements included:

• Marketing and reframing PSU as something other than a “parenting program”; 

• Creating additional promotional materials, e.g. pamphlets, cards, testimonials from past 

participants; 

• Delivering PSU content over fewer sessions and exploring different delivery times and spacing of 

sessions (e.g. 1x/month over a semester) or even as an after school program; 

• Offering the PSU at the beginning of the Fall, when people are preparing for school, or in the 

Spring/Summer, when the days are longer; 

• Recruiting caregivers through a Head Start in the neighborhood;

• Targeting additional populations (e.g. college students studying early childhood education or 

high school seniors would increase participation); and

• Offering incentives and/or toolkits for participants to practice learnings more easily at home.

• Strengthening the partnership between DPTV and IFLC 

During program refinement which occurred before the second cycle, program developers incorporated 

much feedback generated from formative evaluation which may have contributed to the improved rates 

of recruitment and retention. During this cycle, qualitative data collection did not occur since DPTV 

announced that they would be departing as a SIF sub-grantee. 

Program Components and Format

Observations and facilitator interviews also captured strengths and areas for improvement with respect 

to the program format, curriculum, and delivery. Observations and facilitator interviews suggested 

adequate adherence to the PSU curriculum as all adhered to the following prescribed format:

• Review former week’s lesson (if applicable);

o Review how participants used what they learned;
12

o Group problem-solving around challenges experienced;

• Introduce new topics to be covered in this week’s session;

• Alternate between showing video vignettes using the iPad (and CD rom) and facilitating 

discussion with program participants;

• Provide practical, hands-on examples of activities discussed in the videos throughout the 

session; 

o Participants come up with ideas for activities;

o Participants practice activities (on their own);

o Participant practice activities (with their children).

Several factors were successful for engaging participants during the program sessions. The most 

commonly cited and observed strategy for engaging participants was using interactive activities 

throughout the sessions. Facilitators expressed a strong desire to receive more guidance around and 

ideas for supplementing the curriculum with more activities that were both hands-on and physical. 

Facilitators wanted more interactive activities so they could not just show participants the program 

content, but have them learn by doing it. Hands-on activities were also important as icebreakers, in 

getting people more comfortable to be participating, and as a fun element of the session.

For those groups who had children present, involving the children in the conversation and doing 

interactive activities was successful in getting participants to engage more fully. Sharing something 

funny or breaking the ice was important for getting participants to feel more comfortable. Facilitators 

drew on their own anecdotal experience as parents and/or grandparents. One facilitator reported asking 

parents/caregivers about how they were cared for as a child was also successful in getting people to 

participate, especially if a mother and grandmother attended the sessions together. Facilitators used the 

environment to play games and challenge participants to create new mechanisms for teaching their 

child (e.g. eye spy). 

Facilitators found co-facilitators to be an important way to administer the program as one person was 

able to walk around and further engage participants while the other was teaching. Having a co-facilitator 

also provided additional examples/anecdotal evidence to share during the program. Having a cofacilitator from the community was felt to be especially important for one facilitator who was of a 

different demographic composition and community than the participants. 

One facilitator noted that participants responded very well to the informal time after class and hung 

around to chat and share. The facilitator felt participants may have been nervous before and during the 

session, but noted participants were more likely to share once they were warmed up and felt less 

“conspicuous.” Engaging with the participants by sitting with them rather than standing, increased 

participants’ comfort and willingness to participate. Facilitators supplemented the curriculum with 

several additional activities (e.g. using the refreshments to build things or create shapes) as well as 

materials (e.g. welcome packet, program notebook, etc.).

Important components of the Pre-School-U program were the video vignettes. Facilitators were trained 

on the iPad to display the vignettes and felt the training was clear and informative. The iPads themselves 

were generally reported as being easy to use, although some found them more useful than others. One 
13

facilitator felt the iPad was an unnecessary tool. Facilitators reported showing all the vignettes as it was 

unclear as to whether they were to select only certain videos to show or were required to show all. 

Intervention participants represented a diverse group and, while no specific comments were made by 

participants, the facilitators felt the vignettes were culturally appropriate and relevant. Some reported 

challenges facilitators expressed with respect to showing the vignettes included redundancy with videos 

and the ability to show the vignettes more easily in a different order as some videos from the different 

sections seemed related to others. 

One facilitator did not like the fact that the facilitator questions appeared on the screen after the video 

and it would be nice to have a DVD with just the vignettes and to use the iPad only as a teaching tool for 

the facilitators. Some technical issues revolved around the HDMI cord being too short for one facilitator 

and another reported having to bring in her own TV to the church every week, as the church did not 

have one that was compatible with the HDMI cord. One facilitator reported having difficulties with the 

vignettes presented in session 2 skipping and another felt the vignette where the child and father are 

playing a game with his wallet was inappropriate.

There were some barriers with respect to participant engagement. Intervention participants seemed 

nervous during the refreshment period and formal instruction time; after the session, however, they 

spoke informally for a long time after each session. Facilitators thought more informal time might be 

helpful. 

As participants were most engaged while doing hands-on activities, the program curriculum should 

involve a great deal more interactive activities to engage participants and break the ice. To increase 

participation, it is important for facilitators to ask challenging questions to stimulate critical thinking. 

Participants also appeared to agree with everything the facilitators said and reported they were already 

doing the things the facilitators described; this may have been an indication that participants did not 

feel comfortable sharing honestly about their caregiving practices.

During the initial round of implementation, having a different group at each session and inconsistent 

participation limited participants’ willingness to participate and share and continues the theme of 

feeling comfortable as a group. Facilitators also reported that those most in need of the intervention 

generally were not present. As the facilitators prepared for the sessions some became overwhelmed 

with the amount of responsibility and felt more time to prepare for and market the program would have 

been beneficial. Further, have more “team” members planning and implementing Pre-School-U may 

help distribute the responsibilities more. Consistent participation during the second iteration of 

implementation seemed to improve participants comfort level with one another and their willingness to 

share and provide social support.
14

Lessons Learned & Program Modifications

Evaluation findings were shared with DPTV and collaborators on an ongoing basis to inform 

improvements to PSU. Over the first year of implementation, the following modifications were made to 

strengthen programming:

• Recruitment and retention of participants was PSU’s biggest challenge. To address the 

challenge of recruitment and retention, the curriculum was revised so that it can be presented 

in a shorter amount of time (e.g. 3 hours of programming). The curriculum content was 

reordered and additional activities added. DPTV believes that these curriculum changes will 

better draw people into the conversations. In addition, a new congregations coordinator who 

has extensive background in participant recruitment and retention was been hired to better 

assist the congregations in this area. The congregation coordinator instituted one-on-one 

meetings with each congregation which focused on developing stronger recruitment and 

retention plans. In addition, program facilitators were encouraged to send text messages to 

participants for the purpose of engaging them in PSU between sessions. The IFLC also developed 

a recruitment flyer which could be customized by each congregation and assisted with printing 

and distribution. It was hoped that these efforts will enable the project to meet its recruitment 

and retention goals. 

• The observation found that the training was lacking in facilitation strategies that included, 

providing participants with opportunities to practice skills, conducting role plays, facilitating 

problem-solving, and encouraging mutual support. In response, additional training materials 

were developed and provided to facilitators. In addition, the program coordinator provided oneon-one training to each facilitator to ensure they were comfortable with employing these 

strategies during implementation. 

• Interviews with program facilitators revealed that they felt overwhelmed with all elements of 

the project including recruitment, retention, facilitation, workshop logistics, and program 

evaluation. In response, DPTV created additional training materials including an 

implementation manual. In order to reduce the burden of program facilitators, DPTV suggests 

that congregations identify several individuals to fill the various roles associated with program 

implementation. The new implementation manual includes descriptions of these roles and also 

provides recruitment strategies. 

• DPTV piloted the revised program (revisions described above) at a Head Start in Livonia during 

the month of March. While concerns persist around program dosage and theory with the 

revised curriculum, changes have been made to strengthen the training and implementation 

manual to include more problem-solving activities, role-plays, and opportunities to practice new 

skills and receive feedback. 
Appendix A. PSU Logic Model

Situation:

Young children in 

the ten target 

regions are 

significantly 

underprepared 

for school

Goal:

To address the 

relationship 

between 

poverty, 

caregiving, and 

children’s 

academic 

outcomes 

through a faith-

based caregiver 

educational 

curriculum, “Pre-

School-U.”

Inputs:

Funding

Program staff

Program 

materials (e.g. 

DVD vignettes)

Other service 

providers and 

partner 

organizations

Low-income 

caregivers caring 

for children ages 

0-5 

Paraprofessional 

staff

Churches and 

interfaith 

leadership 

support

Expertise

Activities:

Provide training 

to program 

facilitators

Provide 6 hours 

of programming 

to caregivers

Provide 

information (e.g. 

early 

development) via 

video vignettes

Model skills via 

video vignettes

Behavioral 

practice and/or 

role play

Group support, 

discussion, & 

problem solving

Encourage & 

support 

caregivers

Provide a safe 

and trusted 

learning 

environment

Outputs:

# of facilitators 

trained

# of caregivers 

recruited into the 

program

# of facilitated 

groups attended

Length of groups 

attended

# of families 

connected to 

networks of 

social support

# of DVDs 

distributed/

program referrals 

made by 

participants

Primary 

Outcomes: (caregiver-level) 

Improved knowledge of 

and expectations 

for healthy child 

development 

among 

caregivers

Caregivers know and understand 

5 domains of 

school readiness

Improved confidence and 

understanding of 

caregiving skills 

and decisions

Improved social support among 

caregivers

Secondary 

Outcomes: (caregiver-level) 

Reduction in 

stress related to 

caregiving

Increased 

frequency of 

reading to 

children

Increased positive 

interactions with 

child

Impacts: (child-level) 

Improved School 

Readiness: • Cognitive 

development

• Language 

development

• Social & 

emotional 

development

• Physical health 

and motor 

development

• Approaches to 

learning

Decreased child 

abuse/ neglect