ICF Transcription 

RAW_Youth Interventions That Work - Webinar

1

Katy: Good afternoon, thank you so much for joining us 

today. My name is Katy Hussey-Sloniker and I work at 

the Office of Research and Evaluation at AmeriCorps. 

Many of you today might be asking yourself, AmeriCorps? 

First, let me share the news that in October of this 

year, our agency rebranded ourselves from the 

Corporation for National and Community Service to 

AmeriCorps.

Every year our agency invests in individuals and 

organizations making a difference in communities across 

America. We continue our belief that volunteers in 

service to others represents the best qualities of 

America. We believe that AmeriCorps, the name, 

represents those ideals and if you haven't heard of the 

change, I’m happy to be sharing this exciting news with 

you. 

Second, our federal agency knows that the past nine 

months, COVID has impacted all of our lives in ways we 

could not have imagined one year ago. We hope that you, 

your family, friends and communities remain healthy and 

are staying safe. At the AmeriCorps Office of Research 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

2

and Evaluation, our objectives are to support our 

agency’s mission by building knowledge on civic 

engagement, volunteering and national service. We 

conduct in-house research, but also fund research 

through competitive grants to researchers, scholars and 

[unintelligible] at institutions of higher education 

and support research and evaluation of our programs and 

grantees of which the social innovation fund is one. 

We started this webinar series as one way we share our 

ongoing research and findings. We like to think just 

because we need to be physically distant, it doesn’t 

mean we have to be socially distant. Our agency’s 

mission of working with communities and volunteers to 

strengthen community outcomes couldn’t be more critical 

than it is at this very moment. 

Vibrant communities are based on strong community 

engagement, social capital and participation. And as an 

office devoted to building knowledge around these 

topics, we’re always looking to highlight cross-hitting 

research from a variety of disciplines that broadens 

how we define or impact engagement across the United 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

3

States. Before we begin, we’d like to cover a few 

housekeeping items before we connect. 

Janelle: Thank you all for your patience as we get started 

with Adobe Connect. This webinar will be recorded and 

posted online following the presentation. There is no 

dial in phone line. All audio is broadcast over the 

internet using your computer speakers. All participants 

will be in listen only mode until the question and 

answer session following the presentations at which 

time you can ask a question using your computer’s 

microphone by selecting the raised hand feature from 

the menu above. 

You can also ask questions at any time during the 

presentation by using the Q and A or chat box below. As 

mentioned earlier, this webinar is being recorded and 

if you have any questions or experience technical 

difficulties, please let us know using the Q and A or 

chat boxes below. I believe that takes care of our 

housekeeping items. Back to you, Katy.

Katy: Thanks, Janelle. Our webinar today is a showcase of 

the final social innovation fund cohort awarded in 

August of 2016. I have had the pleasure of working with 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

4

each of the SIF grantee teams, as we like to call them, 

for almost their entire award performance period. The 

grantees under this final funded opportunity from our 

agency all have two things in common. One, they are 

members of one of the six federal tiered evidence 

programs with SIF, and two, they all have a focused 

area of youth development. 

As you will hear, their programming of designs and 

outcomes impact identified are all different. Even each 

of their geographic interventions delivery strategies 

vary. I’m excited for each of you to hear what they 

have accomplished from Friends of the Children scaling 

a national model, to North Carolina Partnership for 

Children working on state-based model, to United Way 

for Southeastern Michigan implementing a regional 

model. And, finally, to United Way of Central Indiana 

executing a neighborhood model. 

Now I would like to introduce Dr. Lily Zandniapour, 

Research and Evaluation Manager for The AmeriCorps 

Office of Research and Evaluation, to briefly provide 

remarks on the social innovation front. From Lily, 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

5

we’ll proceed to the first presentation and conclude 

with open discussion from Q and A. We expect for the 

presentations to last for a total of 55 to 60 minutes. 

During the webinar, please feel free to type in 

questions or provide comments in the chat box and all 

speakers will respond during the Q and A time period. 

For now, I would like to introduce to you Dr. Lily 

Zandniapour.

Lily Zandniapour: Thank you so much, Katy, and good 

afternoon, everyone. As Katy said, I’ll be brief and 

just want to provide you with a bit of background on 

the social innovation fund. The social innovation fund 

was approved and administered by AmeriCorps that 

received congressional funding from 2010 to 2016 and, 

as Katy mentioned, is one of six tiered evidence 

initiatives that was designed and implemented at the 

federal level during President Obama’s administration. 

In the decade of system implementation, a total of 59 

grants were awarded, for a total federal investment of 

$382 million dollars in promising programs in three 

focus areas of youth development, economic opportunity, 

and health. The majority of these grants were five year 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

6

awards. SIF program had a two-tiered model where funds 

were awarded to an openly-competitive process to 

intermediary grant makers, who would then hold open 

competitions of their own and make awards to subgrantees implementing selected intervention models with 

evidence that these work. 

SIF congressional appropriation was leveraged in a

public-private funding structure that required the 

grantees and the sub-grantees to match one to one 

through private funding their federal share received. 

In all, the SIF portfolio resulted in the investment of 

a little over a billion dollars in communities across 

the country. Again, the full grantees presenting today 

are the final selected cohort of those 59 grantees. 

Next slide, please. Creating the public-private 

partnership for funding was only one aspect of SIF 

statutes and grantee participation. The key objectives 

of the initiative were to, first of all, invest in 

promising interventions that address social and 

community challenges and grow their impact. Secondly, 

to invest in evaluation and capacity building in order 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

7

to support the development and use of rigorous evidence 

to measure the effectiveness of each funded 

intervention and inform decision making. And third, to 

work with grantees to scale and sustain the efforts of 

their public-private partnerships. 

Over the decade of implementation, the SIF program has 

proved successful in strengthening and sustaining the 

capacity of our funded grantee partners for conducting 

rigorous evaluation. The way we’d like to say it is 

this moves the evidence needle around social service 

programming so that we can make what works work for 

more people. 

When SIF was put through a third party, independent, 

for truly a wide robust impact evaluation at the 

national level, the findings revealed that the funding 

and support of each cohort of grantees and their subgrantees spurred high quality local evaluations that 

are building knowledge and a body of evidence across 

their supported program models to inform practice in 

their communities, their region, and the nation. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

8

Today the 2016 SIF cohorts of grantees will be sharing 

their findings and lessons learned to inform and build 

knowledge and to outline their next steps to sustain 

their work. Katy and I would like to congratulate each 

of them on their achievement in computing their 

evaluations and now disseminating their results. 

We are excited to have partnered on this project, and 

look forward to both publishing their final report on 

the AmeriCorps evidence exchange in the coming months. 

Let me now introduce our first presentation, the 

Friends of the Children. Take it away, please. Thank 

you.

Susan: Thank you, Lily. I’m Susan Walsh, I’m the Director 

of Research and Strategic Impact at Friends of the 

Children. And I just would love to take a minute to 

thank Katy, our Program Officer, for all the great work 

with us over the last four years. And I’d also like to 

introduce Dr. Shelley Osborn, from ICF. Shelley is the 

principal investigator of our SIF evaluation and just 

has been an amazing partner on this project. Next 

slide, please. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

9

So, who are we at Friends of the Children? We had hoped 

to show you a video, but we had some technical issues. 

So we’re going to place the video link in the Chat for 

you to view after the webinar if you’d like to learn 

more about us. Friends of the Children is a national 

non-profit that began services in Portland, Oregon over 

27 years ago. 

At Friends of the Children, we select and invite youth 

and their families to participate in a paid 

professional mentoring program from the time that 

children are 4 to 6 years old, through high school 

graduation. Twelve and a half years for every child, no 

matter what. All of our children have unique talents, 

interests, and dreams and they also face significant

systemic barriers to achieving their goals. Next slide, 

please. 

To identify and select our children, we partner closely 

with schools, child welfare agencies, and community 

agencies such as early intervention programs and such. 

Together with our partners, we conduct a rigorous risk 

and protective factor assessment, and also make 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

10

arrangements to meet the children for observations, all 

of which helps us understand which children are in high 

need of additional family supports and advocacies. Next 

slide, please. 

So, what makes up our program? The core of our program 

are professional mentors that we call Friends. Our 

Friends are trained, salaried, professional mentors and 

each Friend carries a caseload of eight children. 

Friends work four hours a week on average, with and for 

each youth and family and they represent the 

communities within which our families live. 

On average, our Friends’ tenure with the organization 

is about five years, so each child is partnered with 

approximately two to three friends over the course of 

our twelve and a half year program. Our work is 

individualized and intentional working in the child and 

family’s home, school, and community. And one of our 

main premises for our organization over the course of 

the last 27 years has been evaluation - evaluation, 

measurement, and improvement. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

11

We use data internally in a networked ETO system, 

Efforts to Outcome system, so that Friends 

intentionally plan their work in relation to annually 

set roadmap goals that they set with their children 

together each year, the children and caregivers. And 

for the last 25 years-plus, we’ve been committed to 

external evaluation. 

Third party evaluation of our long-term outcomes show 

that 83 percent of our youths receive a high school 

diploma or GED; over 92 percent enroll in postsecondary education or enter the workforce; 93 percent 

successfully wait to parent until after their teen 

years; and 93 percent also remained free of criminal 

justice involvement. Next slide, please. 

So, when we were developing our SIF evaluation plan, it 

was heavily influenced by our existing external 

evaluation when we were making decisions about the 

design. And most significantly, we have an ongoing RCT, 

Randomized Control Trial, evaluation. This evaluation 

started in 2007 with NIH funding at four of our sites. 

And we have findings right now through the early years 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

12

of the study, up until the children average the eleven 

year goals. 

And you can see here on the slide, our findings were

mainly in the area which we would expect, in the area 

of social/emotional development in the children, as 

well as protective factors of parents’ perceptions of 

their children’s behavior. We are now just received 

funding to finish the study with the help of NIH and so

we’ll be looking at end point and two year follow-up 

analyses. 

So when we were going into SIF, because of this 

enormous investment, you’ll see in the slides that 

follow, we decided to focus our SIF evaluation plan on 

an implementation study of the fidelity with which we 

scale our program. This is our third - our first thirdparty implementation study. We also incorporated data 

analyses strategies to evaluate whether our SIF sites 

were on track with these social/emotional learning 

outcomes since all of the children that were involved 

with the SIF grant were in their early years, were 

young as well, elementary school. And we also wanted to 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

13

understand whether we were continuing to have a 

positive influence on parental professions as well. 

Next slide, please? 

Our primary goal of SIF funding was to learn how to 

build the appropriate capacity as a network to scale 

our program model with fidelity around the country. We 

want every child that would benefit from a friend to be 

able to join our program and experience positive 

outcomes. So we started in these sites that you see her 

on the slide. 

So in the beginning of, pre-SIF, we were in seven 

locations. And then if you want to advance the slide? 

And with SIF, we expanded to 14 sites and that’s the 

yellow dots here. So we expanded services by starting 

five new programs in new locations, and also expanding 

on the existing sites that we had in Seattle and 

Boston. Next slide, please? So now I’d like to take a 

minute and turn everything over to Shelley to talk more 

about our SIF evaluation design.

Shelley: All right, Susan, thank you so much. So again, I’m 

Shelley Osborn, principle investigator for the 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

14

evaluation that we did for Friends of the Children for 

SIF. So ICF conducted a four year evaluation of the 

Friends of the Children program, so starting in 2016 

through 2020. 

Our implementation evaluation used a mixed methods 

design that drew from program data, primarily in each 

of those efforts to outcomes that Susan mentioned, the 

systematic data that Friends of the Children collects. 

Information we obtain from monthly calls with the site 

staff, so that included executive directors, program 

directors, including Friend and Friend supervisors and 

then direct observations that we did at a couple of 

sites.

We have planned more site evaluations, which of course 

COVID put a kibosh on that. So our research questions 

for the implementation set to be centered on program 

models fidelity, as Susan mentioned. So you know, the 

specified number of hours that Friends spend with their 

youth, with caregivers, with the school staff, as well 

as how the professional development training was 

implemented. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

15

The child welfare study involved administrative data 

from state agencies and answered questions specific to 

child welfare outcomes such as the length of stay in 

foster care, the number of placements and the time to 

permanency. And that was conducted in two site. The 

school study also used administrative data looking at 

attendance and behavior, which was primarily measured 

by just disciplinary incidents, and then academic 

performance which was measured by MAP and/or reading 

test scores. 

And then our caregiver survey was designed to answer 

research questions related to improvements in parents’ 

perceptions of gains in their child’s socioemotional 

learning and their own parenting efficacy. And that 

survey was conducted in 2019 and 2020. So that’s just a 

quick overview, and then I’m going to turn it back over 

to Susan for the next slide.

Susan: So right now we are still finalizing our review of 

ICF’s final evaluation report and data analyses. And so 

we really wanted to focus our time mostly on what we 

learned over the course of the four years in 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

16

implementing this evaluation. And like I said, the 

implementation study was really important and 

groundbreaking for us at Friends of the Children. We 

learned that the process of doing an implementation 

study was so incredibly valuable. 

Our ICF team has been true partners. They shared 

findings with us along the course of the study, which 

we’ve been using strategically to inform program 

improvements as we scale. And mainly some of the 

findings we found that we were actually able to really 

operationalize is that we needed to strengthen our 

training support and supervisor our program staff. And 

what this led to was us investing in a new quality 

improvement system building initiative. 

We went out and in the middle of SIF and we sought 

funding to be able to create a QIF system. And also 

invest in a learning management system so that our 

delivery to our staff across the country was able to be 

adaptable to the variety of diverse adult learning 

styles. And this is something that’s benefitting our 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

17

entire network, not just our SIF sites, and positioning 

us well for future expansion and quality. 

Part of this process was that ICF, what we learned from 

our implementation study, was the need to really focus 

on the voice of our Friends, the voice of our internal 

program, ground level program staff, to inform the 

process and create systems for that that live on and 

are continuous. So we’ve instituted a Friend Advisory 

Board and a network-wide program committee that helps 

inform, review and develop our quality improvement 

system building priorities. 

We also learned that SIF was an incredibly effective 

mode for building national infrastructure for 

expansion. So through the implementation study, some of 

the findings along the way have really led for us to be 

encouraged to grow our finance and our HR team, and to 

model the SIF matching fund approach with private 

investors. So we’ve been successful doing that, which 

is really, really great. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

18

And then the administrative data analyses, making sure 

the juice is worth the squeeze. We could have an entire 

webinar about what we learned about developing data use 

agreements with schools and child welfare and it is a 

very complicated process. 

We’ve learned that with the school data that 

social/emotional learning is really our sweet spot in 

the early years of the program. And so that the 

enormous investment in school data, admin data in 

elementary school is probably not worth the squeeze. 

But we are realizing with child welfare data analyses, 

we did have some positive findings in the area of 

reductions of lengths of stay in foster care. So we’re 

excited to learn more about that. Again, we’re still 

unpacking the results. 

And finally, with caregiver voice - you know, one of 

our SIF sites piloted a new two generation approach to 

our program model where our program teams do more 

intentional work with families, as well as children 

with the goal to advance not only child outcomes, but 

also family stability outcomes. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

19

So over the course of this SIF grant term, our 

caregiver survey became a more primary focus of our 

evaluation. And again, ICF was wonderful adapting to 

our priorities, especially during COVID when our work 

with families became even more important and the 

logistics of fielding a caregiver survey became even 

more complicated. So we really have learned a lot 

through that process. I’m going to kick it back to 

Shelley to share a little bit about those findings of 

the caregiver survey. 

Shelley: Okay, so next slide please? And I know we just 

have a minute left, so I’ll go over this really 

quickly. So we conducted the survey twice in 2019 and 

2020 with all seven sites. And again, when we talk 

about sort of the socioemotional skill development, 

we’re talking about things like self-management and 

self-regulation, self-awareness. 

And we had parents and caregivers sort of answer a 

series of questions from not at all true to very true, 

thinking about before their child had a Friend and then 

in the past two months before the survey. And we saw 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

20

gains in all of the areas, even when we controlled for 

several factors. The second group of findings, we 

really wanted to not only figure out from caregivers 

what they thought about their child’s growth and 

development, but for themselves. 

And so we saw improvements, again, self-reported 

improvements for a cross-sectional survey, but improved 

problem solving within the family, ability to handle 

sort of parenting demands and then those important 

connections that Friends were able to help caregivers 

make. So educational assistance, like books for their 

home, tutoring, library access, cultural or art 

activities, fitness or sports activities. At least 40 

percent of the caregivers in both years of the survey 

reported that Friends helped them connect to those 

areas. And I’ll turn it back to Susan. Susan, are you 

there?

Susan: Sorry, I was on mute. I know we’re out of time, but 

thank you very much. The SIF slide shows where we were 

with this. If you advanced one more slide, you’ll see 

where we’re headed. We’re hoping to get to 25 sites by 

2025. And our SIF evaluation, particularly the 2Gen 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

21

work, is where all of these new sites want to do their 

work. So really, we would not have been where we are 

with our 2Gen innovation without this SIF evaluation. 

So thank you so much. I will now pass on to the North 

Carolina Partnership for Children. 

Stephanie S: Thank you so much and hello, everyone. My name 

is Stephanie Sidberry with the North Carolina 

Partnership for Children. And I’m going to talk to you 

about the lessons learned and the evaluation from our 

project, Shape NC. We are housed at the North Carolina 

Partnership for Children, which is the state level 

organization that administers Smart Start. And Smart 

Start was created in 1993 as an innovative solution to 

a problem. 

Children were coming to school unprepared to learn. So 

policymakers recognized this and established Smart 

Start as a public/private partnership. We have a 

network of independent private organization called 

local partnerships, it’s about 75 of them, that work 

across all 100 counties throughout the state. Next 

slide, please? So before I begin, I’d like to give 

kudos to the rest of our team. I don't know if they’re 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

22

able to join or not, we had conflicting calls this 

morning, this afternoon. 

But Courtney Latta-Sosebey and Angela Lewis are our 

implementation coaches and these two very smart, 

dedicated early childhood professionals, I want to 

thank publicly for all of their hard work over the last 

three years. Now missing from this is our evaluator, 

who was with NCPC for most of this project, Dr. Kim 

McCum Storton [phonetic]. She moved on to another 

organization, last year, but I still want to 

acknowledge her because a lot of what I’m going to be 

talking about today was a result of her work. Also want 

to thank Katy for all of your guidance and support 

throughout this project. Next slide, please? 

So back in 2010, the Blue Cross and Blue Shield North 

Carolina Foundation came to NCPC with this idea to 

utilize our network of 75 local partnerships to address 

early childhood obesity and so Shape was created. And 

so since then, we’ve had two prior funding phases, one 

from 2011 to 2013 and then phase two was 2014 to 2016. 

Next slide? 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

23

So in 2016, NCPC was lucky enough to be awarded funding 

from back then SIF to conduct a rigorous evaluation to 

determine if the Shape model would continue to produce 

outcomes on early childhood body mass index and health. 

So NCPC had conducted evaluations in our two previous 

phases and both found that BMI, body mass index, for 

children where centers implemented this full Shape 

model, realized more significant decreases in child BMI 

than children that were in centers that only 

implemented portions of the program. 

So this is the model that we used for this phase of 

Shape. And so we had three main bubbles. We had our 

expert partners that consisted of Be Active Kids, which 

is a health program for children ages birth to five 

that provides resources and education and training on 

the benefits of physical activity of childhood. 

The Natural Learning Initiative, which is housed at NC 

State’s College of Design, which works with childcare 

centers to transform existing traditional playgrounds 

into natural play and early learning environments. We 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

24

don't call them playgrounds anymore once we transform 

them - they become outdoor learning environments. Go 

NAPSAAC, which is the nutrition and physical activity 

self-assessment for childcare which is an evidencebased online assessment tool housed at UNC Chapel Hill 

that childcares can use to assess their current best 

practice implementation in various content area, such 

as infant and child physical activity, infant feeding 

and breastfeeding, child nutrition, oral health and 

such. So those are our three core partners. 

Over about midway through this phase, we did add Farm 

to Early Care and Education as one of our partners, but 

they weren’t a factor in our evaluation. Next up is our 

learning spaces. So the way we disseminated knowledge 

down to the locals was through peer to peer, in person 

learning collaboratives, technical assistance to 

childcare centers, coaching and providing online 

resources and conferences. And then the next bubble, we 

sub-granted funds to several local partnerships. We 

actually funded three local partnerships and one 

community-based organization. And then these 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

25

organizations in turn passed money down to childcare 

centers. Next slide? 

So the evaluation was set up as a quasi-experimental 

design, delay treatment design, meaning that childcare 

centers were not randomly assigned to treatment and 

comparison groups, but they were matched variables 

related to their readiness to implement the program 

such as how long they had been in operation, their 

quality rating, staff stability, director, owner 

stability. 

We also took into consideration whether the center 

accepted subsidies, which we used as a proxy for 

poverty, to ensure we were reaching the children most 

in need in each county. So the evaluation assigned the 

centers to either a treatment group or a comparison 

group. Centers and treatment groups immediately began 

working with our core partners in receiving training 

and technical assistance from the Shape team. 

So as you can see, data was collected in the fall and 

spring of year one and fall and the spring of year two. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

26

And I apologize, because there’s a typo on that slide -

“year two” should be 2018 to 2019. Now our study was 

originally designed to be three years, but due to COVID 

in spring of this year, we decided to cancel data 

collection because we couldn’t get into the centers and 

it was just not the safe thing to do. So we decided to 

just focus on two years’ worth of data for the 

evaluation. Next slide? 

So the research questions we were examining were what 

was the impact of Shape on the number of health and 

nutrition best practices implemented in the childcare 

centers. What is the impact of Shape on physical 

activity in children and what is the impact of Shape on 

the percentage of participating children at a healthy 

weight as measured by their body mass index compared to 

the children in our comparison site? Next slide. 

So what we found was overall the evaluation determined 

that Shape produced large and significant improvements 

in opportunities for indoor and outdoor physical 

activity among children participation at the childcare 

sites. The evaluation also determined that programs 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

27

from our three partners, Be Active Kids, Go NAPSAAC and 

NLI, the Natural Learning Initiative, are powerful and 

effective interventions that bring opportunities for 

children to engage in physical activity, both indoors 

and outdoors. 

So specifically, for research question number one, what 

we found was there was no significant impact on Shape 

on the number of healthy nutrition policies and 

practices, best practices, after one or two years of 

implementation. Now there was a bit of a caveat with 

that. 

The evaluators, we had come to a decision with the 

evaluators to use the Yale’s Rudd Center for Food 

Policy and Obesity survey as a validated tool to assess 

children, policy and practices. Of the 46 physical 

activity policies and practices they examined, 23 were 

already in childcare licensing regulations, meaning 

that our centers were really supposed to adhering to 

them anyway. So that would explain why we didn’t see 

any kind of significant impacts on Shape health and 

nutrition policies practices. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

28

But as far as research question number two which was 

the impact of Shape on physical activity in children, 

we did see that after two years children at all 20 

treatment sites engaged in indoor free play which was 

an increase from one site at baseline. So that was 

statistically significant. And then after two years the 

number of sites in which children engaged in structured 

indoor play increased from eight sites to 17 sites, and 

that was also statistically significant. Next slide? 

Continuing with outcomes from our second research 

question. After two years of treatment, the number of 

centers in which children participated in outdoor 

structured play increased from 4 to 17. And then for 

research question three, which was the impact of Shape 

on body mass index, we found that after two years the 

percentage of children in the healthy weight category 

decreased significantly, unfortunately, from 73.8 

percent at baseline to 65.7 percent. While the 

percentage of children in the obese category increased 

significantly from 9.5 percent at baseline to 15.2 

percent at baseline. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

29

Now although we definitely did not want to see obesity 

increase, I can say that more than two-thirds of the 

children in both treatment and comparison groups were 

at a healthy weight category at baseline. And most of 

those children in both groups stayed at a healthy

weight after one and two years of treatment, so that it 

is a good thing. So where are we going next? Next 

slide?

So what we’re going to do with our model is we’re going 

to pull out this little piece on Coaching for Technical 

Assistance. We know that the Shape model significantly 

increased physical activity. We also learned throughout 

this project how important it was to have ongoing 

technical assistance and coaching as a means to embed 

the Shape model into childcare centers. 

Now, Shape as a program is sunsetting in 2021, but we 

wanted to continue implementing these best practices 

within centers across the state. So we decided the best 

way to do that is to focus on the coaching piece. So 

we’re going to leverage the skill and knowledge base of 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

30

our current implementation coaches, Angela and 

Courtney, as well as the Shape-trained TAs across our 

local partnership network. And so we’re going to scale 

TA provision across this network to all of the other 

local partnerships. 

And we realize we have about 150 TAs across the state, 

and so we’re going by this time hoping that by

providing them technical assistance and training, we 

can continue to move the needle on integrating best 

practices around healthier eating, increasing physical 

activity, and outdoor play and learning. We will also 

support coaching towards new and enhanced policies, 

collaborations, [unintelligible] that influence a wide 

range of child and family outcomes. 

We know that our Shape TAs are highly trained and 

competent, so we want to duplicate that across our 

network. And what we’re hoping is that this will lead 

to significantly expanded capacity of the local TAs 

across our network, to support childcare centers in 

their critical roles in promoting positive outcomes for 

children. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

31

Now this will be, unfortunately, an unfunded project. I 

mean, we will have funding at the state level, but the 

technical assistance and coaching that we will provide 

to the network of TAs will include ways to engage 

communities in raising money. That’s another lesson 

that we learned with this project is that it’s so 

important to engage the community and get them and 

parents involved. 

And you just never know what kind of skill and 

expertise you have within your community to help build 

out, continue building out outdoor learning 

environments, and to keep funding coming so that these 

centers can start embedding these best practices for 

healthier eating and active play. So with that, next 

slide. Thank you very much for your time, and I will 

turn it over to the United Way of Southeast Michigan.

Ty: Thank you. I’m really excited to be here. My name is Ty 

Partridge and I’m a professor at Wayne State 

University. I’ve been working on projects with the 

United Way since about 2004, and I have to say this is 

particularly meaningful for me in that SIF has been a 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

32

daily part of my world since 2012, I believe. So it’s 

going to be really weird to not have SIF stuff going on 

in my day to day life coming up at the end of this 

project. 

I also want to acknowledge Chris Trentacosta, who is a 

professor at Wayne State University who’s been working 

with us on this project, and is really the program 

expert. And Jeff Miles who many of you might know is 

leading the SIF project and early childhood initiatives 

for United Way and Randi Burlew who is the lead 

evaluator from Philliber Research that’s working with 

us. So, the next slide, please. 

This project has been a tremendous undertaking and I 

just want to acknowledge all of the people that have 

participated in this. Obviously we could not do this 

without the support and encouragement and knowledge of 

the AmeriCorps crew and all of the fantastic work that 

Katy and Lily and everyone has given us over the years. 

There’s a huge team at United Way. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

33

And this intervention, it’s a single intervention, a 

single program, but we’re actually implementing it 

through five different agencies: Oakland Family 

Services, Care of Southeastern Michigan, Leaps and 

Bounds Family Services, the National Kidney Foundation 

of Michigan, and Access who have all been tremendous 

partners in this project. All right, next slide, 

please. 

So just a little bit about the G.O.A.L.S program. This 

program is really designed to serve low income families 

with children between the ages of 0 and 5, living in 

the Metro Detroit area. We really tried to have a much 

more regional approach. The G.O.A.L.S program is built 

around the family checkup model, which is an evidencebased intervention that uses motivational interviewing 

to use sort of enhanced case management services to 

help children - or families with young children - to 

procure resources and engage in different services that 

they need. 

And our approach to this was to take their original 

family checkup model and expand on it by using a multi-
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

34

agency delivery approach, but also pairing it with the 

tremendous referral capacity of United Way and the 211 

program. And so we think of this as a really intensive, 

but light-touch case management, home visiting type of 

intervention, but that has scalability to a regional 

and larger context without the same resource cost of 

something like traditional home visiting. And I’ll talk 

a little bit more about how it works in a little bit. 

All right, the next slide. 

So again, this is a single intervention, and it’s being 

implemented through five different agencies and the 

idea that we had was really to create a regional 

network. And so these agencies, in a way, almost form 

sort of a meta-agency where we had regular learning 

community sessions, we were sharing resources, they’re 

sharing oftentimes referring to each other or sharing 

training materials or training resources. And so it 

really did operate like this much broader community 

intervention model that transcended different agencies. 

And that was one of the things that we think was really 

unique about our approach here, was extending and 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

35

really creating a collaborative network across the 

community. All right, next slide. So a little bit off 

the timeline. United Way of Southeastern Michigan was 

fortunate enough to have been awarded a Social 

Innovation Fund grant in 2012, I believe. And we 

learned a tremendous amount, often the hard way. 

And one of the things that we learned was to make sure 

that we had a planning year and a pilot process to help 

us kind of get up to speed. So we really started the 

piloting of this program after a year of intensive 

planning. In 2017 we did a pilot and took our learnings 

from that and really started the programming in full on 

February of 2018, continued to enroll families through 

December of 2019 and then our data collection ended 

just this past August. All right, next slide. 

So a little bit about the families that we enrolled. We 

had about just under 1,000, we had 992 families 

participate in this program. We’re evenly split between 

an intervention group and a control group. The sample 

was about half were married. We had a really diverse 

sample, 47 percent of the sample was African American, 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

36

or Black, 24 percent were Middle Eastern or North 

African or Arabic descent and then 23 percent white. 

And we also had a lot of language diversity, if you’ll 

notice. 24 percent of our sample, their primary 

language was Arabic. And education was pretty evenly 

distributed from less than high school through some 

college, although we did have some college graduates as 

well, and advanced degrees. Next slide. 

We also wanted to highlight some of the health factors, 

particularly in the context of COVID, because we’re 

going to take a look at some COVID impacts in a little 

bit. We found that 86 percent of our sample was on 

Medicaid. Almost all had insurance, but 17 percent had 

unmet, uninsured medical need. And really important in 

terms of the COVID risk and how this is going to impact 

their sample over the next year or so, that 36 percent 

of them had chronic health conditions that put them at 

higher risk for COVID complications. All right, next 

slide? 

So, as I said, this was primarily an implementation 

study. We really wanted to take a known empirically 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

37

validated intervention, but implement it in a different 

way and in a larger scale. But we also did include a 

randomized control trial, where we have a comparison 

group and an intervention group. We have a broad range 

of assessments, that I’ll talk about in just a second, 

that we give to all families. 

And then basically what happens is that if you’re in 

the comparison group, you’re given a list of resources, 

and basically referred to the services of the agency 

that you were enrolled in the program through. And it’s 

kind of a standard of care. The intervention group was 

given the family checkup assessments, and there’s a 

three session process where the family’s checkup 

specialist will meet with the family, build a level of 

rapport, and then provide really intensive feedback to 

the family. And then using a motivational interviewing 

approach, work with the family to set goals and refer 

them into specific pathways across our entire network. 

And then we follow each cohort up at six months and 

twelve months following the initial assessment. So, the 

next slide. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

38

The assessment as I mentioned is really broad. We 

looked at things like basic need, that includes things 

like housing stability, income security, food security. 

We looked at some family vulnerabilities domains like 

mental health, substance use, legal involvement. 

Positive family supports like life skills, adequate 

family social support, community involvement, those 

sorts of things. 

We looked at both adult education needs and child 

education needs. We looked at child development 

milestones, primarily through the ASQ, the Ages 

Questionnaire, as well as some parenting skills and 

parenting knowledge and the quality of the parent child 

relationship, at least in the family checkup model, 

through the PICCOLO Assessment. The majority of these 

assessments really come from the Arizona SelfSufficiency Matrix, and a structured interview that we 

developed to fill out the Arizona Self-Sufficiency 

Matrix. All right, next slide. 

So, the way the family checkup model works is you give 

that comprehensive set of assessments, and then we had 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

39

scoring criteria that really puts you into categories 

of green, yellow, and red. And if you were in the green 

category, that was really a strength for your program 

or your family. If you were in the yellow range, that 

was something that could need some intervention, but 

wasn’t as severe. And if it was in the red range, well 

this is something that should really be brought to the 

family’s attention and try to work with them to develop 

some goals around that. And then we did set some goals 

and we used this goal feedback sheet just to kind of 

work with the family to set those goals. All right, 

next slide. 

Just an example of one particular family. They had a 

goal of trying to get the child to eat more foods. We 

enrolled them in a FEAST program, which is a food 

intervention, a food-based intervention with the United 

Way. And then when we followed up with them at six 

months, the child had started eating more foods. And so 

you can see that we kind of look at different specific 

questions, or different specific needs of a family, and 

then helping develop referral pathways for them and 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

40

then following up on those to see how they worked. Next 

slide, please. 

This just gives you a little bit of an idea of the

kinds of goals that were being set, and the kinds of 

referrals that we’re making. Education and parenting 

skills were really the big things that folks were 

looking for, but we also had a range of other basic 

needs, like housing, food stability, mobility or 

transportation was a big issue, as well as some 

healthcare coverage and then on the right we have the 

different pathways that we referred people to. And 

these were all specific intervention pathways that we 

developed as a part of this program. All right next 

slide. 

For those of you that when you get the slides we’ve got 

some links to some of our pathways. Parents as 

Teachers, for example, was a big one. Ready4K, which is 

a text-based intervention that provides specific skills 

and resources for parents to help get their children 

ready for kindergarten, that are tailored to that 

particular child. We have healthcare access through a 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

41

virtual chat program. The National Kidney Foundation of 

Michigan has a diabetes prevention program that’s been 

really helpful. And then again I mentioned the FEAST

program really working with families to help really 

develop a better attitude around and better strategies 

around health and nutrition that’s been very effective. 

Next slide. 

So, looking at some of the outcomes that we’ve looked 

at, we used a logistic progression approach to look at 

how participants changed in terms of their relative 

risk in each of these domains compared to the 

intervention group and really across the board had 

pretty substantial, significant improvement for the 

family checkup model families relative to the control 

families, in a wide range of domains. And it’s 

important to remember that the intervention or the 

control group was actually - they were still receiving 

interventions, it was just the point of the standard of 

care. 

So this is above and beyond what the typical standard 

of care would be to families reaching out to the 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

42

agencies for supports. All right, next slide. Then as 

we all know, COVID hit and this did impact the way we 

delivered services. We’ve learned a lot about how to do 

this kind of case management remotely. And I just 

wanted to talk a little bit about some of the - we 

retrospectively went back and asked families that had 

participated how COVID has impacted them. And then we 

also looked at families who were enrolled and receiving 

services across the pre-COVID to post-COVID periods and 

how that has impacted them in their outcomes. So, go 

ahead to the next slide. 

Based on our retroactive surveys, what we found is that 

of our families we got 380 respondents that we had data 

on at this point. Four percent of them had been COVID 

positive themselves. I think most striking is that 40 

percent of the families in our sample had an extended 

family member or close friend that was COVID positive 

and about 18 percent of them had a family member or 

close friend die from COVID. And that’s really just 

gives you an idea about the impact that it’s having on 

the lives of families in this region. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

43

We had really good compliance in our sample. 94 percent 

wore a mask when they went outside. But this is a group 

that doesn’t have a lot of choice about staying at home 

and they had to leave home typically 3.5 to 4 days a 

week and primarily for work or getting groceries. Did

have a pretty significant impact, particularly on 

income and food access, as well as increasing stress. 

And then finally, the next slide. 

We did find in our study where we were looking at 

families who were already in the program before COVID 

hit and then had outcomes after COVID, that it really 

negatively impacted them in terms of inadequate 

healthcare, inadequate employment, decreased family 

connections, being disconnected from the community, 

parenting challenges and mental health challenges. And 

this was across the board, whether they were in the 

intervention group or in the control group. 

But if you go to the next slide, we did find that being 

a part of the family checkup model, and having this 

strong rapport already developed with an agency and 

having those outreach connections pretty well 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

44

established, did lead to a rebound effect. While it hit 

everybody really hard initially, the families that were 

in the family checkup model program were between 1.8 

and 3.5 times less likely to show an increased risk as 

the COVID pandemic progressed. So once we got past 

about April 30th, they started to really rebound in a 

way that the control group did not and we thought that 

was really powerful. 

So, I really appreciate the time that everyone has 

given us today, and I’m happy to share these findings 

with you and a little bit about our program and I 

welcome any questions at the end. And now I will hand 

it off to United Way of Central Indiana.

Whitney: Thanks, Ty, from one United Way to another. I 

appreciate you passing that along. For those who don’t 

know me, my name is Whitney and I’m a Senior Program 

Officer with United Way, working on the Great Families 

2020 program. And you might see as we go along some 

parallels actually between our program and United Way 

of Southeast Michigan’s program. They were a strong 

collaborator and kind of resource for us in the 

beginning as we were shaping this program. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

45

So, today we’re just really going to walk through 

programmatically high-level, the Great Families 2020 

program, and specifically some tools, data and 

evaluation tools that we use, and some lessons learned 

at the end. Next slide, please. So, as I mentioned, I’m 

Whitney, and I also have the pleasure of being joined 

in the presentation with me later on, members of our 

strategic information team from United Way of Central 

Indiana, Denise Luster and Stephanie Fritz. And 

additionally, representation from our third party 

evaluator, PPI and Polis, Breanca Merritt, who will 

talk through some, like I said, the more specific data 

tools that we use in this program. 

I also want to shout out some additional program staff 

that I see that are on the call joining today. So, 

Candace Gary and Dionna L. Marshall, they’re pivotal to 

our programmatic work of Great Families. So thanks to 

them for joining on the call today. Next slide, please?

So I want to start this time as I mentioned earlier on 

thanks to the [unintelligible] and her work, but I want 

to take a brief moment to talk through this because 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

46

it’s really integral of Great Families 2020, and it’s 

essentially the philosophical framework in which we use 

to develop our programming. So you’ll see essentially 

the goal here is to support whole families. So, taking 

child-focused work and merging it with a parent-focused 

work with the ultimate goal of serving the whole 

family. Next slide, please. 

So, here you can see there’s six integral cause to 

Great Families, or 2Gen rather, that we used to embed 

into Great Families 2020 programming. So the early 

child education piece is kind of where we focused our 

work in 2Gen because we were already funding and 

supporting that work in the space, but it could also 

include K through 12 programming as well. 

But for us, we had already funded very strongly early 

childcare work. And that’s kind of the focus that we 

took in 2Gen, so the lesson learned as we’re expanding 

from this work through Great Families, broadening that 

to include just larger early education work. And then 

additionally some of the adult’s component. So that 

would include post-secondary and employment pathways, 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

47

getting credentials into high wage earning jobs, and 

then increasing economic aspects. 

So for us, in the program the way we did that was 

through what’s called the Center for Working Families 

model that’s embedded in community centers and as a 

coaching and enrollment model to serve some of those 

outcomes there. And then health and well-being and 

social capital, which tend to be kind of tertiary to 

2Gen work, but it’s really integral to the whole 

component of it. 

Health and well-being, as we know, supporting like 

whole social/emotional work with families and then 

social capital which is often called kind of the secret 

sauce. But essentially it’s building connections and 

kind of your network of sorts using cohort and 

different peer to peer building strategies. 

So, we really used this model to create Great Families 

2020, and helped some of our sub-grantees learn this 

through this funding source. Next slide, please. So for 

our program explicitly, we also did sub-grants. So we 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

48

ultimately funded eight sub-grantees in five target 

area throughout Marion County, which is where 

Indianapolis is located. So we had a competitive grant 

process where we had two rounds and awarded four and 

four sub-grantees in each of those rounds and 

ultimately were able to give out three and half years 

of funding to these sub-grantees to build and implement 

2Gen strategies at their own organizations. 

They applied, it was competitive, we reviewed. And we 

really left it up to the sub-grantees to kind of 

dictate how they were going to bridge and implement 

some of this 2Gen programming at their own sites 

knowing that they know their programs best. They were 

already offering some level of this component, but this 

funding really allowed us to come alongside them and 

support them into the program and use this funding to 

help them kind of grasp some of the 2Gen concepts and 

theories. 

And for us, as United Way, it was a way to try on a 

2Gen hat. It was the first time we ourselves kind of 

brought together our early childcare work and our adult 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

49

work. So it was definitely a pilot in that regard and 

we could have a whole other presentation on lessons 

learned from that. But I will say for us it was unique 

in that we let sub-grantees have autonomy of how they 

wanted to implement the funding and utilize it at their 

sites. So you can see above the sites themselves are 

listed. So the blue boxes are what would traditionally 

be like a community center, and in the gray kind of 

varies. 

So if you look at the Near Westside partnership, 

there’s two sub-grantees there, but the MCCOY really 

focused on social capital support. They were a child 

advocate system level of provider. And then on the Near 

Eastside, you have Englewood and East Tenth who are a 

traditional childcare center partnering with a 

community center. So there’s a lot of variation in 

terms of how that works. 

But ultimately for us, it was a recruitment strategy 

and an enrollment into Great Families to be a Great 

Families site. So the only stipulations really that we 

implemented was that a child had to be enrolled in a 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

50

high quality - and for us, the way we define that is 

pass the quality 3 or 4, it’s our ranking system here. 

So that child 0 to 5 enrolled in high quality and then 

an adult or guardian in that unit being enrolled in at 

least one center for working family service. So that 

could be financial support, educational coaching, 

mentorship. 

And so then the sites enroll them into the program and 

then ultimately sat down and did what we call a family 

success plan that looked really similar to what Ty kind 

of talked about, that family engagement plan. And then 

they would kind of lay out a goal for that family in 

terms of outcomes and what they wanted to see. And then 

so for us, we could come alongside sub-grantees and 

offer additional technical assistance, connect them 

with mental health, social capital services and 

programmings while they were enrolling families into 

the program. Next slide, please? 

So essentially theory of change, long-term for us. This 

would be the ideal situation for a family coming out of 

the Great Families program, that their child would be, 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

51

enter kindergarten ready to learn. Families would be 

financially stable. Children and parents would be 

physically and mentally healthy. Families will have 

formal and informal support networks to lay that social 

capital piece. And then parents or guardians, adults, 

will acquire 21st century financials and job tracking, 

job skills, for high wage jobs. 

So I will pass it over to Denise who is going to talk 

to some of the specifics of how we measured the 

highlighted boxes there, so the kindergarten piece and 

the physical and mental health piece. We’ll talk 

through some of the tools in which how our program 

tracks some of that. So next slide, and with that, I’ll 

give it over to Denise. 

Denise: Thank you, Whitney. As Whitney mentioned earlier, 

my name is Denise Luster. I’m the Vice President of 

Research here at United Way of Central Indiana. The 

evaluation tool that we use was ISTAR/KR. It’s the only 

tool at this level from the state of Indiana. It stands 

for the Indiana Standards Tool for Alternate Reporting 

of Kindergarten Readiness. The tool assesses 

kindergarten readiness and overall child development. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

52

It is free and it’s administered either via web or 

paper, paper copy. 

I think people are still using paper copies for things, 

but we found over the years that many childcare centers 

prefer to use sometimes paper copies because they’re 

able to control it and administer is at their own 

accord in terms of time. It’s used from infancy to 

first grade to assess a child’s developmental skills in 

five domains - physical, personal care, socio-emotional 

skills, English language arts and math. 

It is age-based and benchmarked, and what that means is 

it’s to assess a child’s development on a total of 30 

different indicators listed by and observing age in 

months as compared to that child’s age in months. For 

example, if a child is 47 months old and has a personal 

score of 52 months, the findings would suggest that the 

child is above this age range and likely kindergarten 

ready. Next slide? 

So the next one, we’re going to talk about the 

evaluation barriers in data collection in our data 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

53

sharing agreements. Like I stated initially, our goal 

was to have state ISTAR/KR data and combine that with 

our Great Families 2020 youth PII data, personally 

identifiable data, to come get to our early childhood 

education, or ECE outcomes. 

The one thing - we started down the road several years 

ago when we first received the grant from CNCS trying 

to get a data sharing agreement with the State of 

Indiana Department of Education. So if anyone out there 

has ever had to get a data sharing agreement with any 

of their Departments of Education in their state, they 

know how hard that can be. 

So this was about a two year process in which we have 

to go back and forth with the State of Indiana 

Department of Education to finally get the data sharing 

agreement in place, which was a big deal for us in 

terms of this project and others. Once we got that data 

sharing agreement, got that data sharing agreement, we 

were able to assess 335 youth records that we submitted 

to the Department of Education, but we only had, only 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

54

22 records match which was at a 6.6 match rate, which 

was very low. 

Some of the barriers to some of that is, one, if you 

don't have an STN number, which is like almost like the 

Holy Grail of numbers which is a student identification 

number or a testing number, it’s hard to do that level 

of matching. Because we’re relying on the organizations

and the parents to give us a certain PII record to that 

matching test. 

And some of the ways we were able to collect data, the 

data that we asked the parents to provide to us, was 

date of birth, address, those things. You’d think those 

would be sufficient enough. Believe it or not, there’s 

a great sample of two Denise Lusters to be born on the 

same day and they have a similar address. So we ran 

into that. 

And then we also ran into, when you have a data sharing 

agreement, you’re still at the will of the Department 

of Education once we deliver that data over to them to 

turn that data back over to us. So those are some of 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

55

the issues we ran into. I guess the greater finding in 

this is through these data sharing agreements, it would 

be nice to get the STN number, but through FERPA 

regulations, that is almost impossible to do. They just 

don't provide those because you could get access to 

almost - when you get that STN number, it’s not just 

testing, you have access to pretty much all of the 

information on that child and their parents. And I 

think I am turning this over to Stephanie then. Next 

slide? 

Stephanie F: So continuing on to the evaluation barriers 

that we experienced, we actually then in 2019 the 

ISTAR/KR was actually decommissioned and replaced with 

the Kindergarten Readiness Indicators. And then along 

with this, this was decided by the State Board of 

Education, and it was the requirement for many high 

quality ECE providers that our Great Families 2020 

youth were enrolled in. 

So some of the issues we experienced with the 

Kindergarten Readiness Indicators assessment was, first 

of all, ISTAR/KR was web and paper based; however, the 

KRI was only web based. So the Department of Education 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

56

provided a one year waiver for early childcare 

providers that had no computer or internet to delay the 

implementation of the KRI. So that put us towards the 

end of our evaluation when many of these providers just 

started entering data. 

Additionally, the KRI is done as an annual assessment, 

so the ISTAR/KR was actually administered throughout 

the year so that way they could see the child where 

they started at the year, and then where they ended the 

year. So the KRI was provided only once a year, so we 

were unable to see growth in youth. And finally, the 

KRI is a population level assessment. So that means 

that the data is actually only tracked at the smallest 

level, it’s actually county level. It’s all aggregated. 

It’s not an individual assessment on youth. 

So that did not allow us to, again, track individual 

child’s growth which again was a barrier to our 

evaluation. I am going to move this forward to Breanca 

Merritt, because we were able to pivot and look at some 

youth outcomes, so Breanca is going to be able to speak 

to what outcomes we were able to track. Next slide? 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

57

Breanca: So thank you, Stephanie. So as alluded to, we did 

have some outcomes, even though we had an adult 

component to the program as well. These findings are 

talking about the youth outcomes. So one of our primary 

outcomes was just looking at general attendance. And 

what we found was that 70 percent of youth who were 

enrolled were enrolled in high quality childcare on 

average 156 days, which is pretty good when we think 

about the range of children in there from 0 to 6, and 

the childcare inclusive of pre-K as well. 

Moving to our ISTAR/KR indicators, Stephanie kind of 

walked through and Denise, some of the barriers with 

data collection and matching at that level with 

childcare agencies as they relate to youth. So what we 

- even though we had not a lot of folks who matched, 

not a lot of youth who matched, we were able to 

identify 22 youth who did, were able to be cross 

checked with those Department of Education records. 

So at the time of program exit, we found that 32 

percent, about a third, of all children had mastered 

all functional performance threads. We knew they 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

58

mastered, again, about five different areas by the end 

of their time in early learning, and 95 percent had 

mastered at least half. Initially 18 percent mastered 

all math and English language arts threads and 55 

percent had mastered at least half. 

So again, thinking about the range of kids of those 22 

youth that we were able to match with, those that we 

were able to get results for, did have pretty positive 

outcomes. Related to just youth outcomes as well, a big 

component, like other presenters have mentioned, are 

parenting outcomes as well. 

So we did use an assessment tool to understand 

protective factors and how parents engage with their 

children. And so what we found is that parents reported 

their own perception of child development and parenting 

improved significantly over the course of enrollment. 

We saw that over time across multiple time periods and 

so from the first baseline assessment to follow-up, 

second follow-up and at the conclusion of the program. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

59

So it’s important because not only did we see those 

changes over time, we saw it consistently across 

different sites as well. 

So with regards to the neighborhood they were in and 

depending on where they received services, importantly, 

we saw these trends across regardless of where children 

were attending school. And so with that, I’ll turn it 

back over to Whitney to wrap everything up. Next slide, 

please. 

Whitney: Thanks, Breanca. I know we are at time, so I’m not 

going to read this to you all here, but you can see 

kind of the things that we talked through in terms of 

data sharing, partnerships. And again, just making 

sure, like aware of the policy changes. So like the 

ISTAR/KR was a huge change that impacted greatly our 

evaluation outcomes. And one of the things we did have 

I want to highlight, in the beginning of the program 

was an advisory group that we could kind of bounce 

ideas and collaborate, but that went away once we 

created the program. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

60

So one of our big lessons learned was to continue that 

advisory group throughout the course of the program. 

But I know we’re at time, so I just want to say thank 

you all for listening to our presentation. We are here 

and available offline if you have any additional 

questions about that. And thanks again to Katy and team 

for the opportunity to present and your support along 

the way of the grant. So with that, I will turn it back 

to Katy.

Katy: So thank you very much, Whitney. Before we open the 

line up for Q and A, I’d like to share a few final 

closing remarks. I’ll be quick about it. First, I’d 

like to thank those who presented today. Each grantee 

showcase reinforce the SIF focus of identifying, 

validating, growing promising approaches through 

evaluation designs assessed for quality and rigor, 

building capacity for local communities and leveraging 

funds through public/private partnerships. 

As a final cohort of the Social Innovation Fund 

investment, their work, like their predecessors, not 

only developed innovative programming in local 

communities, but also created learning communities and 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

61

pockets of sub-grantees and local stakeholders yearning 

to understand what works best for their citizens. 

Second, it should be noted that there were many Social 

Innovation Funds, leadership, staff and colleagues over 

the last ten years who we owe a debt of gratitude. 

Those that developed the policies, the standard 

operating procedures and provided the day to day 

administrative and technical support to provide 

management and oversight of this federal investment. 

Our investment in evaluation and legacy around 

knowledge networks and products will continue to 

contribute through the AmeriCorps Evidence Exchange. 

Included in the Evidence Exchange repository are 108 

grantee designed intervention implementation and impact 

reports. In December, four more from these four 

grantees today will be added as the final cohort. Next 

slide? 

If you’d like to reach out directly to those who 

presented today, we’ve included email contact 

information for each of the grantee award leads, and 

for the Office of Research and Evaluation team co-
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

62

hosting the event today. Andrea, I’ll turn it over to 

you to begin the question and answer session. 

Andrea: Okay, well thank you so much. This has been a lot 

of great information and what an undertaking in terms 

of all of these four projects. So I know we only have 

about ten minutes, so I would urge you to type in any 

questions that you have. We will send these questions 

over to the presenters. So even if we don't get to all 

of them today, you will get responses as long as you 

include your email, if you don't mind. 

Okay, so you can ask a question by typing it into the Q 

and A box. Or, you could also just use the microphone 

on your computer and if you select that, the raised 

hand feature, we will give you microphone rights. So 

I’ve been looking at some of the comments and some of 

the, also some of the themes that have been coming. And 

I just really want to start off by asking if each of 

you could just, if another organization was going to 

start an intervention for youth in their community, 

what would be the first couple steps that you would 

take? I mean, you all mentioned a number of different 

aspects. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

63

This was an undertaking in terms of bringing evaluation 

from the beginning. But it also, you know, a number of 

you talked about partnerships, issues around data 

sharing and planning. So what would be the first couple 

of steps that you would recommend to any other 

organization taking this on? And I’ll start with 

Friends of the Children. 

Susan: Yes, that’s a great, great question. When we are, 

when a community is interested in potentially 

replicating Friends of the Children, we do look very 

early into the available partnerships and the 

interested partnerships in the community. I think one 

of our biggest lessons learned through scaling is that 

it could be that, of course, there’s children in every 

community who needs Friends of the Children, but they 

have to - every community needs to want us to be there. 

And it’s really important that we go and listen to 

community partners, participate in their activities and 

learn about where the gaps are in the community that 

potentially we could fill. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

64

And we generally, the catalyst for that is identifying 

a champion. And so when we have a community that’s 

interested in working with us, we spend a lot of time 

one-on-one talking to people, to identify someone in 

the community who is willing to do that work locally in 

partnership with our national office to explore the 

partnerships and explore the potential catalytic 

funding opportunities. 

Andrea: Thank you. North Carolina Partnership for Children?

Stephanie: I would concur that partnerships are very 

important. All of our partners are experts in their 

fields. And we could not have gotten this far in our 

project without their knowledge and skill and buy-in, 

but also I want to say, community engagement is also 

very important. 

When building out these outdoor learning environments 

for children, it was just so vitally important to get 

the community involved, get donations of supplies and 

materials. Several of the centers found out that 

parents had specific skillsets in carpentry and 

whatever and so they were able to donate time and/or 

services to these centers. So it’s just so important, 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

65

especially when you’re talking about maintaining these 

outdoor learning environments after the funding is 

gone, to really build the strong engagement piece with 

the community in order to sustain this work long-term. 

Andrea: Thank you, Stephanie. And the United Way for 

Southeastern Michigan?

Ty: I would echo the sentiments that have been expressed. I 

would also say, was reminded of a quote that Iri Isco 

[phonetic], a famous community psychologist once said 

that you have to begin before the beginning. And I 

think the fact that the level of community partnerships 

that were a part of our project have been ongoing for a 

really long time really helped. The fact that we spent 

that first year really collaboratively building this 

program. 

We had an idea, but with the different agencies 

involved, evaluators, content expert, the United Way 

staff really collaboratively building this project 

together gives a lot of shared ownership and brings 

everybody’s perspective and expertise into the fold. 

And I think along with that, is that we had tremendous 

communication, just ongoing communication in multiple 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

66

directions. And I think that that was really the key. 

We really did build a community among the partners and 

agencies. 

Andrea: Thank you. And then finally, the United Way of 

Central Indiana?

Whitney: Sure. I would echo what others have said, so 

partnerships and consistent communication. And I think 

for us that consistent communication was key. We had a 

lot of turnover, both on our end and in sub-grantee and 

in the beginning. And I think just that initial trust 

building and moving folks along and really coming 

alongside both, and I think giving credit where credit 

is due. Like organizations are going to know their 

programming best and I think for us coming in and being 

that consistent cheerleader and communicator. 

And particularly when you’re looking at philosophical 

frameworks, like 2Gen that may be really hard for folks 

to grasp in the beginning. And that is some of the real 

work of structuring programs and shifting them. It’s 

definitely a larger shift. And so I think for us having 

that consistent communication and learning, and 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

67

learning together even as we were figuring out what 

2Gen looks like for us was crucial. 

And then now we can move some of that 2Gen programming 

into a fund, the Family Opportunity fund, that we have 

really gleaned and highlighted 2Gen in that. So I would 

say NC understanding partnership and consistent 

communication throughout it. 

Andrea: Thank you. And I know it’s almost 3:30, but I do 

want to ask another question of all of you. Now part of 

what we do here, of course, is building evidence around 

our program, but also looking at how we match national 

service in our programs to effective programs. And now 

that you have all done this evaluation, you know what’s 

working with your own programs, how would you see 

possibly bringing in national service, whether it’s the 

AmeriCorps programs or the Senior Corps program to work 

with your interventions? And I’ll start with - I’ll do 

it backwards now, let’s start with United Way of 

Central Indiana. 

Whitney: Actually, I’m going to have Denise answer that 

one, from the evaluation standpoint. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

68

Katy: Denise, I don't know if you want to go first, since 

you’re representing United Way directly. 

Stephanie F: I can step in - this is Stephanie. So when it 

comes to the national service, there are many of our 

sub-grantees actually incorporate and will hire 

AmeriCorps members to implement some of their 

programming or some of their evaluations. And so we do 

look at organizations such as that to provide those 

services to provide learning services for AmeriCorps 

members, and to take care of their grants and their 

placements in order to help their ongoing programming. 

Andrea: Thanks, anyone else? 

Stephanie S: This is Stephanie. I think long-term it would 

be a great opportunity for some of those members to get 

engaged in any of the communities around our state that 

are previous or current Shape centers. I mean, there’s 

loads of opportunity there to help not only with the 

outdoor learning environments, but indoors as well, 

volunteering or working with the children in any 

capacity would be more than welcome, and I think a 

great addition to this project. 

Andrea: All right, thank you. And lastly, there are, again, 

we have one minute left, but you all mentioned some of 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

69

the hurdles that you faced with COVID and the different 

things that you had to do in order to compensate both 

for, it sounds like your evaluation, but also your 

service delivery. So any lessons learned on that end? 

Again, if you were talking to another organization 

trying to do this work. 

Stephanie S: Sure, this is Stephanie again. One of the 

things that we did was we went completely online with 

pretty much everything. So our in person learning 

collaboratives switched to virtual. We used Zoom. And 

what we have found is that we can’t have the long 

learning collaboratives like we used to. In the past 

they would be around six to seven hours. 

So of course, when we’re talking about virtual 

platform, we’ve had to shrink that considerably. So 

we’ve had to get more creative about making sure the 

information that we’re relaying to our sub-grantees and 

our childcare centers is really on point. But I think 

that we have been able to make it work and that’s 

really going to be our kind of operating model at least 

going forward in the short-term future. 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

70

That any training, coaching, any opportunities for 

training and coaching will be virtual from now on. It’s 

just a matter of making sure you present information in 

a way that’s engaging, that can keep people’s attention 

and it’s still a valuable learning opportunity, even 

though it’s much shorter than what they would have 

gotten in person. 

Andrea: Great, thank you. So I’ll take one more response, 

and then I’ll end. Anyone else? 

Ty: This is Ty Partridge from United Way in Southeast 

Michigan. And I think the fact that we had 

intentionally tried to develop this as a technology 

based approach to begin with really helped and really 

leveraged a lot of text messaging which is more 

resource friendly to a lot of families who might have, 

might not have access to wireless internet all the 

time. And really being sensitive to access to 

technology for a lot of families and trying to 

incorporate multiple methods, whether it’s through the 

phone, through text, as well as virtual platforms. 

Andrea: Okay, well thank you. And thank you again for this 

wonderful work that all of you have done. As Katy said, 

it’s been such wonderful to watch this grow. So I want 
ICF Transcription 

RAW_Youth Interventions That Work - Webinar

71

to be, of course, respectful of everyone’s time. So I 

want to thank you, again, to our speakers and our 

audience today for a great discussion. We will be 

sending out a post-webinar survey, so please let us 

know your thoughts on this webinar, especially any 

ideas for future webinars. And as you all know, we will 

be posting this in about a month. So have a great rest 

of your day. 

[End of File]