Evaluation of the 

Connected Youth 

Initiative: Final Report 

Trevor Fronius 

Sarah Guckenburg 

Darius Taylor 

Hannah Sutherland 

Anthony Petrosino 

2020 
© 2020 WestEd. All rights reserved. 

WestEd is a nonpartisan, nonprofit research, development, and service agency that works with education and 

other communities throughout the United States and abroad to promote excellence, achieve equity, and improve 

learning for children, youth, and adults. WestEd has more than a dozen offices nationwide, from Massachusetts, 

Vermont, Georgia, and Washington, DC, to Arizona and California, with headquarters in San Francisco. 
– i –

Contents 

Executive Summary 1

Program Overview 1

Prior Research 2

Evaluation Design 2

Measures, Instruments, and Analytic Approach 3

Research Questions 4

Findings 5

Key Updates 8

Next Steps 9

Introduction 10

Background 10

Overview of Current Study 15

Program Theory, Logic Model, and Outcomes of Interest 16

Research Questions 22

Contribution of the Study 24

Implementation Evaluation 25

Study Design 25

Implementation Findings 30

Lessons Learned 52

Changes to the Subgrantee Evaluation Plan 55

Impact Study 56

Design 56

Matched Group Comparison of Participant-Level Estimates 56

Study Participants and Sample Flow 72
– ii –

Measures 76

Data Collection Activities 82

Analysis and Results 85

Conclusion 101

Summary of Implementation Study Findings 101

Summary of Impact Study Findings 102

Lessons Learned 105

Recommendations 108

Next Steps 109

Other Aspects of Study Logistics & Feasibility 110

Human Subjects Protection 110

Timeline 110

Evaluator/Subgrantee Roles and Involvement 111

LIST OF TABLES 

Table 1. Formal Titles of SIF-funded CYI Collaboratives and Subgrantee Organizations 12

Table 2. Implementation Data Collection Method, Source, and Timeline 28

Table 3. Data Quality Self-Assessment: Weekly Time Spent on Data Related Activities 32

Table 4. Examples of Technical Assistance Provided to CYI Sites 33

Table 5. Contextual Factors by Collaborative 35

Table 6. Grant and Match Funding by Collaborative, as noted in their SIF CYI Applications 37

Table 7. Combination of Services Received by CYI Participants 42

Table 8. Examples of CYI Participant Success 46

Table 9. Baseline Statistical Equivalence Between Groups Before and After Matching 60

Table 10. Definitions of Treatment and Comparison Groups within the CYI Study 73

Table 11. Sample Flow of Study Participants 73

Table 12. Factor Loadings by Well-being scale 77

Table 13. Factor Loadings by Hope & Executive Functioning Scale 78

Table 14. Well-being Models by Survey Period 86

Table 15. Housing & Financial Stability Models by Survey Period 87

Table 16. Transportation & Employment Models by Survey Period 87
– iii –

Table 17. Social Support Models by Survey Period 88

Table 18. Health Access Models by Survey Period 89

Table 19. Hope Models by Survey Period 89

Table 20. Executive Functioning Models by Survey Period 90

Table 21. Housing Models by Survey Period 91

Table 22. Employment Models by Survey Period 91

Table 23. Education Models by Survey Period 92

Table 24. Health Models by Survey Period 94

Table 25. Transportation Models by Survey Period 95

Table 26. Financial Models by Survey Period 96

Table 27. Permanency Models by Survey Period 96

Table 28. Summary of Outcome Models 97

LIST OF FIGURES 

Figure 1. Map of Connected Youth Initiative SIF Sites 13

Figure 2. The Connected Youth Initiative Logic Model 21

Figure 3. Changes in Implementation Readiness Scores (Across All Collaboratives) 31

Figure 4. Implementation Timeline 38

Figure 5. Percent of CYI Participants Receiving Services (N=741) 41

Figure 6. Number of CYI Participants Receiving Services by Site 42

Figure 7. Distribution of Propensity Scores and Area of Common Support 71
– 1 –

Executive Summary

This report represents the culmination of the implementation and impact study conducted on the 

Connected Youth Initiative (CYI). The CYI was led by a Social Innovation Fund (SIF)1 grantee, Nebraska 

Children and Families Foundation (Nebraska Children), which served as the intermediary for six 

collaboratives represented by five subgrantees2 across Nebraska: (1) Hall County Community 

Collaborative, (2) Blue Valley Action Partnership, (3) Central Plains Center for Services, (4) Norfolk Area 

United Way, and (5) Fremont Area United Way. The six collaboratives3 were:

• Hall County Community Collaborative • Sandhills Collaborative

• Southeast NE Older Youth Collaborative • Norfolk Family Coalition

• Families 1st Partnership • Fremont Family Coalition

Nebraska Children is a member of the 2015 SIF cohort. WestEd served as the lead external evaluation 

partner and contracted with the Nebraska Center for Justice Research at the University of Nebraska and 

its subcontracted partner, Category One Consulting.

Program Overview

CYI serves young people, ages 14–24, who are or have been in the Nebraska foster care system, have 

had contact with child protective services and/or with the juvenile justice system (but are not on 

probation), and/or are homeless or near homeless. CYI uses a collective impact approach to create 

systems change in rural Nebraska communities by serving young people through the following essential 

elements: Youth Leadership, Opportunity PassportTM Program, support service funding, voluntary case 

1 The Social Innovation Fund (SIF) was a program that received funding from 2010 to 2016 from the Corporation for National 

and Community Service, a federal agency that engages millions of Americans in service through its AmeriCorps, Senior Corps, 

and Volunteer Generation Fund programs, and leads the nation’s volunteer and service efforts. Using public and private 

resources to find and grow community-based nonprofits with evidence of results, SIF intermediaries received funding to 

award subgrants that focus on overcoming challenges in economic opportunity, healthy futures, and youth development. 

Although CNCS made its last SIF intermediary awards in fiscal year 2016, SIF intermediaries will continue to administer their 

subgrant programs until their federal funding is exhausted.

2

In addition to the subgrantees listed here, The Connection Homeless Shelter, Inc., served as a subgrantee in one site during the 

first year of the project. 

3 Collaborative names reflect the names as of the submission of this report. 
– 2 –

management (coaching), and system support. Ultimately, the objective of CYI is to realize positive 

outcomes for participating young people in domains of education, employment, permanence, housing, 

health, transportation, and economic stability.

Prior Research

The CYI model and the evaluation study’s design are rooted in prior research on CYI’s predecessor, 

Project Everlast, that has proven successful in the urban areas of Lincoln and Omaha. The Project 

Everlast model was rated as reaching a “preliminary evidence level” by the Corporation for National and 

Community Service (CNCS), as it demonstrated improved outcomes using a one-group modified time 

series design, drawing on pre-/post-testing data from 2010 through 2016 on 1,217 young people, ages 

16–24. Data collected over the course of initial evaluation (2010–2016) found gains among Project 

Everlast participants in outcomes related to employment, education, housing, and transportation. 

However, the initial evaluation of Project Everlast lacked a control group to compare differences in 

outcomes, an internal validity threat addressed in the current matched comparison study.

Evaluation Design

The current study of CYI, Project Everlast’s successor, features a quasi-experimental design with a 

comparison group. Such a design intends to move the evidence rating from a preliminary evidence level,

as for the evaluation of Project Everlast, to a moderate evidence level. This design and evidence level 

were targeted because CYI was not at a stage of implementation amenable to experimental design and 

because the design is considered acceptable evidence by many “best practice lists” and “evidence-based 

registries.” Specifically, the study was designed to compare the outcomes of young people enrolled in 

CYI with the outcomes of similar young people who were offered the program but either declined to 

participate or received only minimal support that would not likely result in measurable outcomes 

associated with CYI.

4 WestEd used a propensity score matching procedure to reduce potential selection 

bias in drawing a comparison sample of young people. The evaluation design called for an ambitious 

target sample of 200 total study participants across each of the six communities (also referred to as 

collaboratives or sites) for a total sample of 600 treatment and 600 comparison participants. The study 

4 The comparison group was defined based on feedback from implementation staff in order to identify an acceptable level of 

exposure while still defining a participant as a comparison study subject.
– 3 –

faced recruitment and retention challenges that were exacerbated by a reduction in SIF’s funding, which 

truncated the overall evaluation period. The final baseline analytic sample fell considerably short of the 

initial target number, with a total of 418 participants across the six communities (328 treatment and 90 

comparison participants) but was considered statistically robust enough to examine overall impacts. 

Follow-up data were collected in two different follow-up periods.5 The exposure period leading to the 

first follow-up survey averaged just over 7.5 months. 

Measures, Instruments, and Analytic Approach

The impact evaluation was conducted on participant-level outcomes in which the individual is the unit of 

analysis. The primary outcome of interest was defined as overall well-being and was operationalized 

using measures from a survey developed by Nebraska Children in collaboration with the WestEd 

evaluation team.6 The study conducted a series of measurement analyses to create and provide initial 

validation for a well-being index suitable for the CYI population. Participant surveys were the source 

data for the computed Well-Being Scale (and subscales) and other measures of interest that contributed

to the impact study. There were several data collection activities that informed the findings in this final 

report. The core evaluation and implementation activities were documented using the following data 

collection tools:

• Common Referral Form

• Transitional Service Survey (i.e., CYI youth survey) 

•

Implementation data, including:

− Support Service Fund Tracking Form

− Coaching Form

− Youth Gathering and Youth Leadership Participation Forms

− Opportunity Passport Database 

5 The number of participants who contributed data in both follow-up periods was notably low. The last follow-up impact estimate 

should thus be interpreted with caution and considered exploratory in nature. The last follow up is reported in the full report; 

however, it is omitted from the executive summary. 

6 The Transitional Service Survey is an adaptation of the Opportunity Passport Program survey and has been further refined for 

the CYI SIF evaluation. 
– 4 –

The impact study utilized a series of multivariate linear, logistic, and ordinal logistic regression models 

that were adjusted for baseline differences in the outcome measure and other covariates to examine 

the outcomes specified in the research questions, which are described in the next section. Additionally, 

Full Information Maximum Likelihood (FIML) models were constructed and examined for certain 

outcomes of interest. 

In addition to the impact study, an implementation study was also conducted for two purposes. The first 

was to provide feedback to guide ongoing system development and refinement by Nebraska Children. 

Second, the implementation evaluation provided evidence on what the CYI system at the collaborative 

level is, how young people are experiencing the system, how well it is being implemented across CYI 

communities, and the variation of implementation across communities.

Research Questions

CYI’s theory of change and anticipated outcomes informed the following research questions. Each 

question is categorized by whether it applies to impact or implementation. Overall, the study was

guided by one primary impact question, three secondary impact questions, and seven implementation 

questions.

Impact

1. [Primary RQ] To what extent does CYI improve overall well-being for the young people

who participate, compared to similar young people who do not participate?

2. To what extent does CYI improve access to stable housing for the young people who 

participate, compared to similar young people who do not participate?

3. To what extent does CYI improve access to stable employment and education for the 

young people who participate, compared to similar young people who do not participate?

4. To what extent does CYI improve other domains of well-being for the young people who 

participate, compared to similar young people who do not participate?

Implementation

5. How prepared are the communities to implement CYI, and what is their capacity around 

data collection?
– 5 –

6. What contextual factors about the community, partners, and region influence the 

implementation of CYI? 

7. What are the variations in how CYI is implemented in communities (e.g., services offered, 

services accepted, participation)? 

8. What challenges are the subgrantees facing in the implementation of the CYI model? 

9. What positive effects are subgrantees seeing? 

10. To what extent, if any, has the focus on collective impact increased the sustainability and 

capacity of the collaboratives? 

Findings 

Although CYI implementation did not lead to statistical impacts on the overall well-being of participants 

relative to the comparison group, impact estimates among participants at the first follow-up period 

suggest that CYI involvement was statistically associated with positive outcomes related to: 

1. safe and stable living situation, 

1. financial stability, 

2. perceived hope, and 

3. emergency care utilization. 

The impact study conducted on CYI involved a well-designed quasi-experimental study that utilized 

propensity score matching procedures to form the study groups. Several implementation challenges 

reduced the confidence in which claims can be asserted on the impact of CYI across participants. These 

challenges included recruitment into CYI, baseline equivalence between groups, and retention for 

follow-up data collection among both treatment and comparison groups. Attrition bias and the lack of 

baseline equivalence between groups for age and multiple outcomes diminished the internal validity of 

the study, and thus all findings should be viewed with caution. 

Beyond impact estimates, there are also many qualitative examples of success of young people who 

engaged with CYI. Young people were able to purchase cars through an asset purchase, which allowed 

them to get a job for the first time, find housing and employment in their local community, and access 
– 6 –

health services and supports to address their immediate challenges (housing, health care, parenting 

help, job training) and long-term challenges (reducing debt, planning for further education or training, 

job skills). CYI also helped educate the collaboratives and broader community about this population, an 

older and yet unconnected youth population living in these counties and in need of support. 

Lessons Learned

The study also highlighted several lessons related to the implementation of CYI, and the design and 

approach to evaluation. 

Implementation

1. In rural communities, there are fewer people available to serve multiple roles, including 

grant compliance, managing implementation, and data and evaluation tasks. 

2. Transportation is a barrier for young people in rural communities. 

3. It was difficult for communities to balance implementation and grant compliance from the 

beginning, which, in turn, led to delays in both. 

4. For some collaboratives, shifting their focus to this population was difficult, as their 

original focus was on early childhood or other populations. 

5. The needs of young people evolve over time and programs need to be persistent in 

understanding the challenges young people face. Earning the trust of young people so 

they will engage with a program may take time. 

Evaluation

6. CYI is a complex, community-driven initiative. The nature of the initiative and the youth 

who engage with CYI is such that recruitment and participation are not easy to predict and 

plan for in advance. 

7. While contamination in a complex community initiative like CYI is likely unavoidable, 

stronger recruitment and documentation practices may have improved the evaluation 

team’s ability to correctly attribute exactly what treatment each person received.

8. The rural nature of the CYI SIF communities is such that recruitment was saturated within 

the community “hubs,” and reaching young people in more isolated areas proved a 

challenge across sites. More thorough vetting of sample estimates, beyond a commitment 
– 7 –

by sites to reach target numbers, would have established more realistic estimations for 

the study. 

9. Alternative approaches to data collection in order to transfer the burden from program 

sites to the evaluation team may need to be considered. More robust online systems may 

have improved the timeliness, quality, and consistency of implementation data. 

10. The evaluation team attempted to capture and incorporate formative feedback into the 

outcome study through calls and sites visits with Nebraska Children and implementation 

staff; however, a deeper focus in this area with program providers early on and 

throughout the implementation period may have further refined the quality of the

outcome measures. 

Recommendations

The lessons above and many other lessons learned, highlighted in the full report, inform the following 

recommendations:

Implementation

1. Assess capacity and commitment of partners to ensure continuous collaboration and 

implementation.

2. Identify gaps in capacity among partners and provide targeted trainings to increase the 

collaboratives’ capacity in those areas.

3. Increase the collaboratives’ capacity to use data to answer implementation questions and 

promote continuous quality improvement. 

4. Include young people, if not already included, in the collaboratives and in leadership roles 

within CYI.

5. Operationalize each component of CYI and develop a theory of change or a logic model for 

each component, and identify what the expected outcome is for each component. 

6. Consider how some components may be adapted to overcome the barriers of rural 

communities. 

7. Utilize current CYI participants to inform and contribute to the development of CYI and 

the recruitment and retention of participants.
– 8 –

8. Establish ongoing practices to sustain collaborative communication and partnerships. 

Evaluation

9. Provide sites with technology to facilitate mobile recruitment and program engagement. 

Video chat and mobile databases, for example, may extend program reach and the overall 

sample.

10. Transfer survey data collection to dedicated personnel with experience in communitybased survey data collection. Dedicated staff, in coordination with sites, may increase the 

overall ability to contact young people, log communication, and provide follow up for 

nonrespondents.

11. Continue to identify additional incentives for youth to participate in follow-up surveys. 

12. Conduct more frequent calls to maintain regular contact for follow up data collection.

13. Create online tools that are mobile-friendly and available for use offline to support 

ongoing collection of implementation data. 

14. Refine the data collection instruments to ensure that measures are responsive to rapidly

changing experiences for young people in a transitional period. 

15. Continue to explore ways of measuring well-being that are more appropriate and effective 

for the CYI population..

Key Updates

The implementation and evaluation of CYI was not without challenges related to the logistics of the 

study; many of these are discussed in the prior section and elsewhere in the full report. In addition, the 

CYI implementation and evaluation timelines were impacted by unforeseen challenges. The evaluation 

period was cut short by more than a year due to program cuts within CNCS and the Social Innovation 

Fund. The evaluation was originally planned to be carried out through July 2020; however, data 

collection was truncated, with the last survey collected in April 2019. This change meant a shorter 

recruitment and follow-up data collection period, which compounded challenges with recruitment and 

led to a smaller sample size than originally planned. Although loss of federal funds impacted the overall 

study budget, Nebraska Children supplemented the considerably reduced evaluation budget to maintain 

the described study design for the duration of the reported evaluation period.
– 9 –

Next Steps

CYI carried on with its implementation after this report was written, until the end of the contract period:

March 31, 2020. During this time, Nebraska Children worked with each site and its CYI workgroup to 

define what CYI will look like moving forward. Next steps will vary by site, depending on the specific 

needs, focus, and additional resources to support youth who might be served through CYI. Nebraska 

Children is actively reflecting on the CYI experience, including the SIF evaluation, and how the initiative 

fits within its broader efforts to engage and support older youth populations across the state.
– 10 –

Introduction

WestEd researchers, in collaboration with the Social Innovation Fund (SIF) grantee, Nebraska Children 

and Families Foundation (Nebraska Children), prepared this final evaluation report summarizing a study 

of the Connected Youth Initiative (CYI). This report on the implementation and outcomes of CYI in six 

rural communities funded by SIF is meant to inform stakeholders within and outside Nebraska who are 

supporting reengagement strategies for older young people (those aged 14 through 24). The report also 

describes implications of this study to inform future efforts working with young people who could be 

considered “unconnected” (see next paragraph for a definition). The report starts with an overview of 

CYI, including prior evidence on the model; then discusses the current study, including its design, key 

questions, methods, analyses, and findings; and concludes with implications and lessons learned. 

Background

CYI aims to address a critical social problem: Thousands of young people in rural Nebraska are 

unconnected from a positive life course due to child welfare, juvenile justice system involvement, or 

homelessness, and are lacking the services and supports they need to make successful transitions to 

adulthood. In this study, unconnected young people are defined as being aged 14 through 24 and not 

connected to a positive life course because they are experiencing one or more of the following:

•

currently being, or having been, in the Nebraska foster care system

• having had contact with child protective services

• having had juvenile justice system involvement (but not currently on probation)

• having experienced homelessness or near homelessness

These are young people who have, and continue to face, adverse circumstances that put them at 

reduced odds for experiencing a positive and healthy adult life. For example, young people who are 

unconnected from education and employment “have a disproportionate share of problems as they age, 

including chronic unemployment, poverty, mental health disorders, criminal behaviors, incarceration, 
– 11 –

poor health, and early mortality.”

7 Furthermore, in a study on the economic burden of unconnected 

youth,

8

researchers found that each unconnected youth costs taxpayers an average of just over $13,000 

per year and about $250,000 over their lifetime.9It is in the best interest of both the young person and 

taxpayers to address this issue. Young people who are unconnected need intensive support and a 

network of stakeholders in the community to assist them in their transition to independent adulthood. 

These are the young people at the core of, and targeted by, CYI services. Their outcomes are the critical 

focus of this evaluation study. 

CYI uses a collective impact approach to create systems change in rural Nebraska communities in order 

to better serve the needs of unconnected young people. It builds on a model (Project Everlast) that has 

proven successful in the urban areas of Lincoln and Omaha, Nebraska. The objective of CYI is to realize 

positive outcomes for participating young people in domains of education, employment, permanence, 

housing, health, transportation, and economic stability. CYI is currently implemented by six rural 

collaboratives funded by SIF (Table 1). 

7 Mendelson, T., Mmari, K., Blum, R. W., Catalano, R. F., & Brindis, C. D. (2018). Opportunity youth: Insights and opportunities for a 

public health approach to reengage disconnected teenagers and young adults. Public Health Reports, 133(1_suppl), 54S–64S.

8 Unconnected youth are more commonly referred to as “disconnected” or “opportunity youth” and defined as youth, ages 16–24, 

who are out of work and school. The outcomes and statistics cited within this section reflect this definition, which differs slightly 

from the CYI eligibility criteria.

9 Belfield, C. R., Levin, H. M., & Rosen, R. (2012). The economic value of opportunity youth. Corporation for National and 

Community Service.
– 12 –

Table 1. Names of Connected Youth Initiative Collaboratives and Subgrantee Organizations 

Name of Collaborative as of the 

Time of This Report 

Name of Collaborative as of 

Implementation of SIF CYI 

Name of Subgrantee 

Organization 

Hall County Community 

Collaborative

Hall County Community 

Collaborative 

Years 1–3: Hall County 

Community Collaborative 

Year 4: Central Plains Center for 

Services 

Southeast NE Older Youth 

Collaborative

Blue Valley Community Action 

Partnership 

Blue Valley Community Action 

Partnership 

Families 1st Partnership Families 1st Partnership Year 1: The Connection 

Homeless Shelter, Inc 

Years 2–4: Central Plains Center 

for Services 

Sandhills Collaborative Through the Eyes of the Child 

Coalition 

Central Plains Center for 

Services 

Norfolk Family Coalition Norfolk Family Coalition Norfolk Area United Way 

Fremont Family Coalition Fremont Family Coalition Fremont Area United Way 

Each community collaborative selected for CYI implementation is comprised of multiple local 

municipalities and counties. These collaboratives collectively represent over 40 of the 93 counties in 

Nebraska. Subgrantees within each collaborative that proposed to serve low-income, philanthropically 

underserved areas were prioritized in the subgrantee selection process. Although they vary in terms of 

location and context (e.g., politics, industry), all collaboratives serve rural communities with 

demographics generally reflective of the state outside its major urban hubs of Omaha and Lincoln 

(Figure 1). 
– 13 –

Figure 1. Map of Connected Youth Initiative Sites Funded by the Social Innovation Fund 

Note: The formal names of collaboratives have changed since the production of this map. See Table 1 for details. 

Nebraska Children and the SIF collaboratives (subgrantees) all adopted the CYI Logic Model (see Figure 2 

later in this report) based on the Project Everlast intervention theory of change. The logic model was 

designed to inform programming that improves the well-being of targeted young people within a 

community infrastructure built on collective impact strategies10 and a prevention-focused system of 

care. CYI addresses a critical social problem: Thousands of young people (ages 14-24) in rural Nebraska 

are unconnected from a positive life course due to child welfare, juvenile justice system involvement, or 

homelessness, and are lacking the services and supports they need to make successful transitions to 

adulthood. 

This report presents findings from an implementation and impact study of CYI over the course of three 

years (April 2016 through April 2019), using several methods and tools. The primary method to examine 

impact was a quasi-experimental study that utilized statistical matching methods to compare outcomes 

10 Collective Impact is a formal framework for collaboration that Is grounded in five core conditions: a common agenda; shared 

measurement; mutually reinforcing activities; continuous communication; and backbone support. 
– 14 –

for the program participants to a sample of youth eligible for CYI who did not take up services through 

CYI or received only minimal exposure to the treatment services.

11

The matched comparison study used self-report survey data, collected at several points from treatment 

and comparison individuals, to examine the impact of CYI on overall well-being in terms of:

• Permanence

• Hope and executive functioning 

• Economic stability

• Education

• Employment

• Housing

• Transportation

• Physical and mental health

Self-report data were collected from the young people at baseline during their enrollment period, and 

twice annually (each April and October) during their period of study enrollment. These data were then 

used to construct the primary analytic sample for the impact evaluation. As described in the sections to 

follow, the study design is rooted in prior research on CYI’s predecessor, Project Everlast, which

informed both the implementation of CYI and how the initiative was evaluated. 

Overview of Prior Research on Project Everlast

The Project Everlast model was rated as reaching a “preliminary evidence level” by the Corporation for 

National and Community Service (CNCS), as it demonstrated improved outcomes using a one-group 

modified time series design, drawing on pre-/post-testing data from 2010–2016 on 1,217 young people, 

aged 16–24. 

11 As described later, in the Study Participants and Sample Flow section, researchers realized over the course of the evaluation that 

almost all young people who come into any contact with CYI experience some form of CYI practices. In response, WestEd and 

Nebraska Children established a more stringent criteria to define treatment, based on implementation staff feedback related to 

participant engagement. For example, coaches specified that participants require a minimum of two contacts to establish 

engagement. 
– 15 –

Young people receiving any type of Project Everlast service were asked to complete a “Transitional 

Services Survey” twice annually, in April and October.

12 This survey was also accompanied by a 

“Common Referral Form” and collected at the time of enrollment into CYI. The Common Referral Form is 

designed to be completed by any young person entering the CYI service array across the state of 

Nebraska. This form and the Transitional Services Survey itself were developed collaboratively by a 

representative group of Nebraska stakeholders. Using the form and survey allowed for the collection of 

true baseline data from young people before they received any services. The first survey that a young 

person took was used to establish a baseline, with subsequent surveys allowing for the tracking of 

changes in outcomes. Data collected over the course of initial evaluation (2010–2016) found gains 

among Project Everlast participants in outcomes related to employment, education, housing, and 

transportation. However, the evaluation of Project Everlast lacked a control group to compare 

differences in outcomes, an internal validity threat addressed in the current matched comparison study.

Overview of Current Study 

The current study of CYI features a quasi-experimental design with a comparison group. Such a design 

intends to move the evidence rating from a preliminary evidence level, as was the case with the 

evaluation of Project Everlast, to a moderate evidence level. Quasi-experimental study results are also 

considered acceptable evidence by many “best practice lists” and “evidence-based registries.”

Specifically, the study was designed to measure outcomes for young people enrolled in CYI against 

outcomes of similar young people who were offered the program but either declined to participate in 

program services or received only minimal support that would not likely result in measurable outcomes 

that are associated with CYI.

13

Nebraska Children selected a diverse portfolio of six subgrantees to implement CYI using this approach. 

Proposed participant-level and system-level measures, which are detailed below, captured outcomes 

12 In October 2015, Project Everlast staff began using the Transitional Services Survey in place of the Opportunity PassportTM 

Survey, which had been used since inception in 2010. The topical focus of this Transitional Services Survey is nearly identical to 

that of the Opportunity PassportTM Survey; the difference is that the survey was shortened and additional questions were added 

to assess “hope” and “self-regulation.” The Opportunity Passport Survey is still fielded; however, it is only fielded with the 

specific set of young people who are participants in Opportunity Passport.

13 The comparison group was defined based on feedback from implementation staff in order to identify an acceptable level of 

exposure while still defining a participant as a comparison study subject.
– 16 –

across seven well-being domains and the functioning of the collaboratives’ systems supports. The set of 

outcomes specifically addresses the Social Innovation Fund’s Youth Development Issue Area.

Like its predecessor, Project Everlast, CYI also aims to bring young people together with service 

providers, funders, and decision-makers to create supportive communities committed to improving 

outcomes for young people, aged 14–24, with foster care, juvenile justice, and/or homelessness 

experiences. Also like Project Everlast, CYI is designed to build strong collaborations and infrastructure 

necessary for community ownership of well-being and the realization of improved outcomes for young 

people. 

Program Theory, Logic Model, and Outcomes of Interest

The Project Everlast model theory of change, which CYI is based on, is supported by the aforementioned 

study of effectiveness (at the preliminary evidence level) and involves the promotion of evidence-based 

programming, multilevel systemic change, and the collaborations to develop strategic partnerships. 

Nebraska Children was a statewide early adopter of the general capacity-building technical assistance 

activities needed to build a multilevel Prevention Support System in communities and states.14 As the 

implementation of interventions is an iterative rather than a linear process, so too are the development 

and planning of intervention implementation, particularly those that are complex, involving multiple 

components, and spanning across organizational systems and contexts (e.g., cultural, demographic, 

geographic). The first process in the iterative development of, and planning for, complex system 

interventions is the: 1) identification of evidence currently available through extant research and data; 

2) development and refinement of an intervention change theory; 3) initial modeling of the intervention 

processes and outcomes, and 4) implementation of broad-based strategies, encompassing stakeholder 

engagement, input, and modification or adaptation during each of these steps. Nebraska Children, 

WestEd, and SIF subgrantees collaborated at the start of the project to further refine Project Everlast’s 

intervention theory of change to improve the well-being of targeted young people as well as the 

community infrastructure supporting their successful transition to adulthood. In addition to highlighting 

the anticipated changes in outcomes for young people, the theory of change underscores that this 

14Brodowski, M. L., Counts, J. M., Gillam, R. J., Baker, L., Collins, V. S., Winkle, E., Skala, J., Stokes, K., Gomez, R., & Redmon, J. 

(2013). Translating evidence-based policy to practice: A multi-level partnership using the interactive systems framework. Alliance 

for Children and Families.
– 17 –

process will also effectively engage stakeholders across systems to better identify and more 

appropriately address needs and positively impact the life trajectory of the target population. 

A goal for Nebraska Children is to facilitate a broad-based, multisector transformation of Nebraska’s 

ability to mitigate negative life outcomes for one of its most vulnerable population groups. Through this 

process, communities and the state are anticipated to realize long-term outcomes by investing earlier in 

the life trajectory of the target population, avoiding higher-end costs of chronic unemployment, 

homelessness, poor health, and welfare system and justice system involvement. Project Everlast has 

demonstrated its ability in Nebraska’s urban areas to improve outcomes for young people in the areas 

of educational attainment, including postsecondary education; employment; health care access and 

utilization (physical, dental, and mental); establishing permanent connections to lifelong supportive 

relationships for general well-being and support in times of crisis; eliminating homelessness upon exiting 

the foster care system and beyond; accumulating savings and assets; and having access to 

transportation to attend to demands of school, work, and daily life. Project Everlast achieved these 

positive outcomes by engaging with multiple stakeholders involved with young people currently in, and 

exiting, the child welfare system. Project Everlast engaged these stakeholders in a systematic planning 

process to develop interventions that: 1) increase the understanding of risk and protective factors for

young people in the foster care system and the relationship between those factors and outcomes; 2) are

grounded in Nebraska-specific data and the experiences of young people and service providers; and 3) 

build a Nebraska-specific practice model from the Jim Casey Youth Opportunities Initiative Practice 

Pathways Tool, the USHHS Service Array community assessment and planning model, and the 

President’s Office of Management and Budget’s Program Assessment Rating Tool (PART). CYI’s theory of 

change, which is based on the theory of change developed through Project Everlast, is operationalized 

as a logic model, which is summarized in Figure 2 at the end of this section.

Nebraska has a diverse range of geographic and cultural contexts that can differently impact youth and 

young adults. The diversity within the state also impacts the structure and practices of support systems 

and the infrastructure capacity and practices in place differently across the state. One of the major 

geographic and cultural contrasts in Nebraska is along the rural-urban continuum. Of Nebraska’s 93 

counties, 32 have fewer than 5 persons per square mile, and Nebraska’s largest metropolitan area has 

over 1,500 persons per square mile. Given this diversity, the theory of change builds from the premise 

that a single intervention model may not be universally appropriate across the state. The Project 

Everlast model requires more rigorous evaluation in a statewide context — particularly within rural 
– 18 –

settings — to determine if it is effective in a different setting and so that differences between urban and 

rural implementation of the practice model can be documented. Accordingly, the CYI evaluation study

documented in this report focuses on rural areas. 

As depicted in the CYI logic model (Figure 2), the intervention has a number of inputs and resources 

provided by Nebraska Children and communities. These inputs include funding from Nebraska Children 

to six regional collaboratives that engage in community-based regional planning efforts; these efforts 

focus on improving outcomes for young people. Similar efforts were previously occurring through

Nebraska Children’s Community Well-Being (CWB) work, which identified, reviewed, and adapted 

various intervention models over time and in different community contexts (with significant community 

input and support). In each of the CYI collaboratives, CYI builds off this earlier work, identifying solutions 

to address improved outcomes for young people and making sure that interventions are in place and 

meant to complement each other in serving the needs of young people.

Another input from the collaboratives is their engaging of stakeholders across political and geographic 

levels in a cross-sector implementation process that is driven by data. This process includes local data 

reporting, the role of what is known as a “Central Navigator” in each collaborative, and the knowledge 

of the collaborative partners and stakeholders in human resources and other services.

The activities that encompass CYI include a variety of possible services and programs that are targeted 

to the specific needs of each young person being served. These services and programs include a menu of 

options targeting social well-being, such as case management, leadership training, and community 

service opportunities.

Overall, five core activities were implemented across all six subgrantee communities as reflected in the 

logic model: Youth Leadership, Opportunity PassportTM Program, support service funding, voluntary case 

management (coaching), and system support. Young people participated in services and programs 

according to their individual needs. Output data were tracked for each core activity, including the 

number of young people who were offered and ended up participating in various programs and services. 

System support data were gathered formatively by WestEd through interviews and site visits. 

While variation across communities was anticipated, there was sufficient implementation consistency as 

defined by adherence to the core activities, enabling WestEd to evaluate all six communities as a whole.

Each core activity is supported by a set of guiding strategies and primary components formalized into 

program documents by Nebraska Children. Additionally, more specific fidelity measures within each 
– 19 –

core activity were identified collaboratively by WestEd and Nebraska Children (e.g., minimum number of 

months engaged in coaching during enrollment).

System support is a core part of the CYI logic model and acts as a mechanism to establish consistency 

and model fidelity across subgrantee communities. For example, trainings regarding model components 

were provided to each community using the same set of training materials and trainers. While 

communities may have been trained separately on a given component, they all participated in the same 

training curriculum within a short timeframe (typically one month). 

Nevertheless, a necessary aspect of the CYI model is that each subgrantee can modify the 

implementation of its core activities to best fit its own structure and community resources. Thus, some 

variation in delivery mechanisms is not only acceptable, but also expected. As an example, WestEd 

researchers observed variations in the number of agencies that provided voluntary case management 

(coaching) and the extent to which a backbone organization provided direct or indirect services (e.g.,

created contracts with other agencies). Despite observing some variation, researchers do not believe the 

differences to be significant enough to prevent a single evaluation of all communities.

15 The variations 

captured through ongoing interviews with collaborative members and site visits are discussed in the 

Summary of Implementation Study Findings section. The services provided to young people are meant 

to align with the anticipated outcomes of CYI, which fall into the following domains:

• Permanence

• Hope and executive functioning 

• Economic stability

• Education

• Employment

• Housing

• Transportation

• Physical and mental health

15 Statistical controls discussed later in the report were introduced to adjust for the potential clustering of variance within 

individual sites. 
– 20 –

If a young person is identified as having a need in one of these domains and participates in services to 

address that need, then it is anticipated that the young person will experience improved outcomes in 

that identified need area. For example, if a young person is enrolled in the Opportunity PassportTM

Program to address financial well-being, anticipated short-term outcomes of CYI participation are for the 

participant to have an established bank account and have all monthly expenses met. The long-term 

impact of CYI, across all young people who participate in services, is expected to be that the young 

people targeted by services are able to not only to survive but to thrive.

In addition to the young person’s individual experience with CYI, and the services and outcomes CYI is 

designed to achieve, there is a systems support piece to the initiative that impacts communities and 

how they work together. The systems support section of the logic model shows that training and 

support to communities on CYI, including collective impact, Youth Leadership, and other services, will 

allow communities to increase their own knowledge and skills to address the needs of these young 

people — which is intended to ultimately lead to improved communication, identification of service 

gaps, and a greater likelihood of sustainability.
– 21 –

Figure 2. The Connected Youth Initiative Logic Model
– 22 –

Research Questions

CYI’s theory of change and anticipated outcomes identified informed the following research questions. 

Due to the length of the evaluation period, the questions were targeted at short- and medium-term 

outcomes. Each question is categorized by whether it applies to impact or implementation analysis. 

Overall, the study was guided by one primary impact question, three secondary impact questions, and 

seven implementation questions.

Primary Research Question

The primary research question for the study is guided by the theory that young people who participate 

in CYI will experience improvements across a variety of well-being domains, depending on their needs

and program experiences. 

1. To what extent does CYI improve overall well-being for the young people who participate,

compared to similar young people who do not participate?

To address this question, WestEd researchers created a composite variable, or index, that represents

measures of each domain of well-being referenced in the outcomes section of the CYI logic model

(Figure 2). The well-being index was based on a standardized score index using participant-level selfreport measures across individual domains of well-being. The index was constructed and refined using 

baseline data over the course of the evaluation period. (Additional details on the index’s construction 

are discussed in the Measures section later in this report.) The study examined the change in this index 

from baseline to the first and last follow-up periods,

16 with reference to observed change in the 

comparison group.17

Secondary Research Questions

The secondary research questions18 examine the impact of CYI on specific individual domains of wellbeing that are considered core objectives of the initiative. These outcomes are based on similar prior 

16 The majority of participants were offered at least two surveys; however, due to the need to recruit participants through 

December 2018, a small percentage of participant were only exposed to one follow-up period, in April 2019.

17 The follow-up period for each young person varied because follow-up data were collected in both April and October each year. 

18 The original subgrantee evaluation plan included an additional exploratory question related to the economic impact of CYI. Due 

to the SIF funding cuts that impacted the overall evaluation budget and length of the evaluation, WestEd researchers dropped 

this question from the study.
– 23 –

work conducted by Nebraska Children in rural and urban communities. These questions are considered 

exploratory and meant to examine whether the initiative is most impactful in certain domains of wellbeing for young people. 

2. To what extent does CYI improve access to stable housing for the young people who 

participate, compared to similar young people who do not participate?

3. To what extent does CYI improve access to stable employment and education for the 

young people who participate, compared to similar young people who do not participate? 

4. To what extent does CYI improve other domains of well-being for the young people who 

participate, compared to similar young people who do not participate? 

Implementation Questions 

The implementation questions are meant to provide some context as to why young people in CYI 

experience certain outcomes, and how conditions vary across the collaboratives. 

Capacity and Community Context 

5. How prepared are the communities to implement CYI, and what is their capacity around 

data collection? 

6. What contextual factors about the community, partners, and region influence the 

implementation of CYI? 

Implementation of Community Youth Initiative 

7. What are the variations in how CYI is implemented in communities (e.g., services offered, 

services accepted, participation)? 

8. What challenges are the subgrantees facing in the implementation of the CYI model? 

9. What positive effects are subgrantees seeing? 

Collective Impact and Sustainability 

10. To what extent, if any, has the focus on collective impact increased the sustainability and 

capacity of the collaboratives? 
– 24 –

Contribution of the Study

The current study improves on the design used in earlier preliminary evidence with an outcome 

evaluation that was designed to meet standards for a “moderate evidence level.”

19 The study also 

expands beyond urban areas to examine the implementation of CYI in six rural collaboratives in 

Nebraska. The moderate evidence level is reached by implementing a quasi-experimental matched 

comparison study that examines the impact of the program across outcomes. WestEd researchers used 

a propensity score matching procedure (described in more detail in the Impact Study section) to reduce 

potential selection bias in drawing a comparison sample of young people who were targeted for CYI and

completed the intake form but otherwise did not participate in programming, as well as young people

who received some services that were deemed minimal, based on the program model.

Additionally, the development of the well-being index, a composite of several domains of well-being, is a 

unique contribution to the field. There are many ways to capture domains of well-being, and this study 

presented an opportunity to construct and test an index that utilizes participant-level measures for the 

population of interest, which should be of considerable interest to other stakeholders who seek to 

measure change in young people involved in similar interventions.

19 As discussed later in the report, the study was designed to meet a moderate evidence level rating; however, attrition and 

differential attrition were high by evidence standards and thus affected the study’s ability to meet such an evidence rating.
– 25 –

Implementation Evaluation 

The implementation study served two purposes. The first was to provide feedback to guide ongoing 

system development and refinement by Nebraska Children. Second, the implementation evaluation 

provides evidence on what the Connected Youth Initiative system at the collaborative level is, how 

young people are experiencing the system, how well it is being implemented across CYI communities 

(also referred to as collaboratives or sites), and the variation of implementation across communities. 

The data from the implementation evaluation also sheds light on any observed outcomes. Specifically, 

implementation data were collected on capacity and community context, implementation of CYI, data 

use, and collective impact and sustainability. The following sections describe the methods and findings

of the implementation study. 

Study Design

The implementation study used a mixed-method approach that included qualitative and quantitative 

data collection and analysis. A wide range of methods were used to capture data on individual 

collaboratives’ approaches to implementing CYI and on participants’ experiences. Interviews, site visits,

and collaborative-level and participant-level data were included to document the successes, challenges, 

and experiences of communities with CYI. Table 2 at the end of this section provides a summary of all 

the data collected for the implementation evaluation, as well as the data sources and timeline.

Interviews and Site Visits 

The primary process to collect community-level implementation data from each CYI site was to assign an 

experienced researcher to serve as a liaison to each collaborative. Liaisons then carefully worked with 

their assigned sites to learn how CYI was being implemented within the community context of each 

collaborative. The method to collect data was mainly through interviews and site visits.

Interviews and site visits were conducted to document how the project was being implemented, what 

challenges the collaboratives faced, and whether the context (e.g., politics) was also changing and 
– 26 –

impacting the program. Regular monthly20 site-level interviews were conducted using a structured 

interview protocol, with questions addressing the history of the collaborative and the community,

21 the 

collaborative’s structure, the implementation process for CYI, successes and challenges to 

implementation, training needs, and changes in community context. Site visits to each of the 

collaboratives were conducted twice throughout the study to collect data on the community 

collaborative meetings and processes, as well as to provide technical assistance on data use, drawing on

the implementation data collected to date. Interviews and site visits were used to gather data on 

implementation efforts. At the end of the study period, final interviews were conducted with a larger 

group of stakeholders from each collaborative to gather reflective data on implementation, system-level 

changes, and sustainability. A final interview was conducted with Nebraska Children’s leadership to 

document final reflections about the successes, challenges, capacity, and sustainability of CYI in the rural 

sites and lessons learned.

Collaborative-Level Participation Data

To collect information on how sites delivered services, the number of young people who were offered 

and access the different services within CYI, and site variations to the program model, the evaluation 

team accessed implementation data collected at the site level.

22 These data were collected from 

community service partners by a person in the Central Navigator role at each site. The Central Navigator, 

or a designated staff person, tracked each young person starting at enrollment, and recorded the 

services each young person was referred to and their level of participation in the CYI components in the 

project database. Data collected on participation in the Opportunity PassportTM Program, such as 

attendance at financial literacy education sessions, the establishment of a bank account, amount saved, 

amount matched, and any asset purchases, were collected in the Opportunity Passport database.

20 Monthly calls were rescheduled to be every two months (bimonthly) part way through the evaluation period to save on costs 

due to a reduction in the evaluation budget.

21 Thisset of questions was asked in only the first set of interviews.

22 Data collection activities and tools are described in additional detail in the Impact Study section to follow. 
– 27 –

Collaborative Implementation Readiness and Capacity Data

As part of the initial implementation evaluation, collaboratives were asked to complete two selfassessments; one in September of 2016 and again in fall 2017.23 The two self-assessment tools are 

explained in more detail in the following paragraphs.

Implementation Readiness Self-Assessment Tool

The purpose of the implementation readiness self-assessment tool was to assess the degree to which 

partners were ready to implement CYI with quality. This tool was first used in September 2016 and was 

slightly updated so it would be more appropriate for use again in Year 2. The tool included 44 questions 

in Year 1 and 43 questions in Year 2, and assessed six main areas, including general capacity, intent, 

consistency, delivery, quality assurance and improvement, and motivation and willingness (each area is 

further described at the end of this paragraph). Collaboratives responded to each question using a 5-

point scale with higher scores indicating a greater level of readiness for implementing CYI (i.e., 1 = no 

partners have begun this; 2 = some partners are starting to work on this; 3 = all partners are working on 

this; 4 = all partners have almost completed this; 5 = all partners have accomplished this.) 

• General Capacity – Assessed whether their partners had the infrastructure, resources, and 

experience to implement their CYI work effectively (6 questions). 

•

Intent – Assessed whether their partners demonstrated the skills, knowledge, and 

experience to implement CYI with a high degree of quality (4 questions).

• Consistency – Assessed whether the initiative was operationalized to ensure consistency

and coherence, and whether all processes and procedures were documented (7 questions). 

• Delivery – Assessed whether staff were appropriately prepared and supported through 

training and supervision to deliver CYI as designed (5 questions).

• Quality Assurance and Improvement – Assessed whether the implementation and results of 

CYI, including participation, successful completion, and outcomes, were documented and 

used to inform continuous improvement decisions (12 questions). 

• Motivation and Willingness – Assessed the vision, incentives, risk management, mission, 

culture, innovation, and support related to CYI (9 questions).

23 The fall 2017 self-assessment data collection took place between September and October 2017.
– 28 –

Data Quality Self-Assessment Tool

The purpose of the data quality self-assessment tool was to learn more about the data each 

collaborative collected or planned to collect as part of CYI. The tool was first used in September 2016 

and was heavily updated to fit the collaboratives’ Year 2 data activities. The updated tool included 10 

questions assessing perceptions of the Youth Database among site staff, their opinions about data 

collection processes or systems, their perceptions about forms from direct service providers, and if they 

had any requests for technical assistance. The tool included a combination of open-ended and Likertscale matrices. The following list provides a brief description of each of the categories:

• Youth Database – Assessed opinions about the database, database capabilities, and level of 

understanding of database requirements (3 questions).

• Data System Perceptions – Assessed data system perceptions, data management 

workloads, and process improvement ideas (3 questions).

• Forms and Direct Service Providers – Assessed the extent to which collaboratives believed

forms were completed and obtained by direct service providers (2 questions).

• Technical Assistance – Assessed helpfulness of trainings and technical assistance and asked 

for any other additional information (2 questions).

Table 2. Implementation Data Collection Method, Source, and Timeline

Implementation Data 

Collection Method

Who Provided the Data Dates

Site Visits All site Central Navigators, site 

coordinators, and other relevant site 

personnel

October 2016

June 2018

Implementation 

Readiness SelfAssessment

Central Navigators, with input from 

other relevant site personnel

September 2016

September 2017

Data Quality 

Assessment

Central Navigators, with input from 

other relevant site personnel

September 2016

September 2017

Regular Community 

Check-In Calls

Central Navigators and site coordinators Held monthly from February 

2017 to June 2017; held 

bimonthly from August 2017 to 

December 2018; held monthly in 

January and February 2019
– 29 –

Implementation Data 

Collection Method 

Who Provided the Data Dates 

Final Stakeholder 

Interviews 

Central Navigators, site coordinators, 

and relevant site personnel identified by 

Nebraska Children 

March – April 2019 

Nebraska Children 

Interviews and Calls 

External evaluation team (WestEd) and 

CYI implementation team (Nebraska 

Children) 

Full team calls held monthly, 

June 2016 – July 2019 

Youth 

Implementation 

Database, Including 

CYI Component Forms 

and Participant 

Surveys 

Completed by CYI participants, Central 

Navigators, other service providers; data 

entered by Central Navigators 

Ongoing data entry; data 

extracted for analysis in May 

2019

Collective Impact 

Survey 

CYI implementation staff and 

collaborative partners 

Fall 2017 and fall 2018 

Opportunity 

PassportTM Program 

Data System 

Data completed/entered by Opportunity 

Passport service provider as various 

program milestones are achieved 

Ongoing data entry; data 

extracted for analysis in May 

2019

CYI Grantee 

Applications 

Filled out by site personnel Applications submitted to 

Nebraska Children in January 

2016; all applications reviewed 

for data purposes by evaluation 

team in May 2016 

CYI Programmatic 

Documents and 

Materials 

Gathered from site personnel and 

Nebraska Children 

Documents received in spring 

2016

Analysis Method 

WestEd analyzed qualitative data using Excel to summarize and code for recurring themes that spoke to 

the implementation questions. Researchers created multiple spreadsheets with data from the bimonthly 

CYI evaluation check-in calls, the final interview conducted with CYI sites, and system-level interviews 

conducted with a broader group of CYI stakeholders in each community. The evaluation questions were 

used to identify themes within each data source. Those themes were organized by each evaluation 

question and are summarized in the Implementation Findings section. In some places, quotes are used 

to illustrate a theme or provide an example. Only quotes from sources that are not identifiable are used. 

For the quantitative implementation data, specifically the program participation data, univariate analysis 

was performed to examine the distribution of selected variables from the Youth Database and 
– 30 –

Opportunity PassportTM Program Data System to describe the type and intensity of services received by 

CYI participants by site. The sample that was used for the implementation data consisted of 741 CYI 

participants who received services between April 2016 and March 2019. This sample is larger than the 

analytic sample24 used for the outcome study and provides a broader picture of CYI implementation.

Implementation Findings

CYI was implemented across six collaboratives in rural Nebraska. These collaboratives vary in geography, 

need and capacity. The implementation evaluation findings are organized by the evaluation questions 

and are meant to provide context as to why youth in CYI experienced certain outcomes and how 

conditions varied across the collaboratives. 

How Prepared Are the Communities to Implement CYI, and What Is 

Their Capacity Around Data Collection? 

To understand how prepared communities were to implement CYI, the results of two data selfassessment tools were used, supplemented by interview data and documentation of technical 

assistance provided by Nebraska Children to CYI sites throughout the grant. Overall, sites scored 

themselves as being strong (i.e., high in the scaled response) on implementation readiness at baseline, 

but there was also an increase across all domains between fall 2016 and fall 2017 (Figure 3). Capacity 

around data and data collection varied among sites, but also increased over the course of the evaluation

(Table 3). Technical assistance was offered throughout the grant, with the focus on grant compliance, 

especially in the beginning, as well as implementation and evaluation. Each of the types of technical 

assistance offered is described in more detail in Table 4.

Motivation and Implementation Readiness

Overall, sites mentioned that goals for CYI were to foster collaboration and pool resources across the 

organizations in their community who are serving young people in this age group (ages 14–24). Sites also 

highlighted that the communities involved in CYI have traditionally been philanthropically underserved 

due to their rural geography. Motivation and willingness to achieve such goals were reported as strong

across most sites on the Implementation Readiness Self-Assessment Tool at baseline. All scores fell 

24 The analytic sample as described in the Impact Study section to follow is comprised of any youth who participated in at least one 

Transitional Service Survey. Participation in the survey was not a requirement for enrollment in CYI and therefore the 

implementation data represent a larger sample of youth. 
– 31 –

between 2 and 4, or between some partners are starting to work on and all partners have almost 

completed this. These scores could be due to sites having varying levels of prior system engagement and 

collaboration prior to the start of CYI. 

The implementation readiness scores increased across all domains between fall 2016 and fall 2017. 

Figure 3 shows the “consistency” score saw the largest change and “general capacity” score saw the 

smallest change. Consistency of implementation included items about whether the initiative was 

operationalized to ensure consistency and coherence, and whether all processes and procedures were 

documented by the partners. General capacity included items about whether site partners had the 

infrastructure, resources, and experience to implement their CYI work effectively. 

Figure 3. Changes in Implementation Readiness Scores (Across All Collaboratives) 

*Level of Readiness: 1 = no partners have begun this; 2 = some partners are starting to work on this; 3 = all partners are working on this; 4 = 

all partners have almost completed this; 5 = all partners have accomplished this.

Data Capacity 

Specific data capacity, including data collection, varied from site to site. In fall 2016, the evaluation team 

administered the first iteration of the Data Quality Self-Assessment tool. In this version of the tool, sites 

were asked about the current state of their data, their perceptions of their data system, and data 

reporting capacity. In fall 2017, the evaluation team made substantial revisions to the tool to align with 

the actual tasks and activities the sites were facing in Year 2, as noted earlier. In Year 1, sites were asked 

to report how long they anticipated entering data for CYI, and in Year 2, they were asked how long they 

had spent entering data for CYI. Overall, the sites reported in Year 2 that the current time spent 
– 32 –

reporting data each week met or exceeded what they had expected (Table 3). The Data Quality SelfAssessment was not conducted after fall 2017, but, according to information that the researchers 

received during the regular community check-in calls with the evaluation team, the sites had a smoother 

time with the data entry once they had become more accustomed to the data-related activities. 

Table 3. Data Quality Self-Assessment: Weekly Time Spent on Data-Related Activities 

CYI Site

Anticipated CYI Data 

Entry (Year 1)

Current Data Entry

(Year 2)

Southeast Nebraska Older 

Youth Collaborative 3–5 hours 5 or more hours

Sandhills Collaborative 3–5 hours 5 or more hours

Families 1st Partnership 3–5 hours 5 or more hours

Fremont Family Coalition 1–3 hours 1–3 hours

Hall County Community 

Collaborative 3–5 hours 5 or more hours

Norfolk Family Coalition 1–3 hours 3–5 hours

Ongoing Technical Assistance

Regular meetings and technical assistance (TA) provided by Nebraska Children to CYI sites provided 

another opportunity to gauge how prepared the communities were to implement CYI and how their 

level of preparation changed over time. These TA offerings fell into three categories: compliance, 

implementation, and evaluation (see examples of the TA offered in each these categories in Table 4). 

Results and reflections from these TA meetings and events were shared with the evaluation team in the 

team’s monthly meetings with Nebraska Children. The purpose of these meetings was to provide 

ongoing feedback from the evaluation team to Nebraska Children, as well as to hear updates from 

Nebraska Children on site-level CYI implementation. 

In addition, Nebraska Children staff were asked to reflect on the capacity of, as well as how prepared,

CYI sites were to implement the initiative. Nebraska Children staff agreed that much of the upfront focus 

of their TA to sites, out of necessity, was on grant compliance but that many sites improved and built 
– 33 –

their capacity in this area. However, they noted that in some cases new capacity was specific to CYI 

grant requirements and was not necessarily transferrable to general grant development or 

management. In other cases, sites did change some of their internal practices in how they managed staff 

and accounting. TA was also offered on the CYI components throughout the grant. Several TA events 

were focused specifically on Youth Leadership, which may have been necessary because both the sites 

and Nebraska Children noted it as one of the more challenging components to define and implement.

Table 4. Examples of Technical Assistance Provided to CYI Sites

Technical Assistance Focus Area Examples of Technical Assistance Frequency

Compliance Contract monitoring Yearly

SIF Budget and Budget Narrative

SIF Marketing

Grant Onboarding

Policies and procedure review

Finance and compliance

Implementation Youth Recruitment As needed

Youth Eligibility

Youth Leadership mini trainings/training modules

Sustainability

Opportunity PassportTM Monthly

Central Navigation

Statewide Youth Leadership practice group Quarterly 

Youth ThriveTM strategy meetings/calls

Evaluation Output tracking forms Yearly

Data tracking As needed

Evaluation calls with WestEd Monthly

What Contextual Factors About the Community, Partners, and Region 

Influence the Implementation of CYI?

Collaborative Structure

Each collaborative involved in CYI has a slightly different structure, including partners and collaborators, 

history, geographic service area, and funding. A more detailed explanation of these structures is 

available in Tables 5 and 6.
– 34 –

Table 5 summarizes the following features of each collaborative: partners, counties involved, history 

(existence prior to CYI, year established, original focus), meeting frequency, unique features, and 

presence of the Community Response system.

25 The following points summarize the similarities and 

differences between the collaboratives:

•

In each collaborative, the main partner agency that applied for the grant has been in 

existence since at least 2013.

• Southeast Nebraska Older Youth Collaborative has been in operation for the longest 

amount of time (since 1966), and the Norfolk Family Coalition has been in operation for 

the shortest amount of time (since 2013).

• The number of counties served by a CYI collaborative ranges from 1 (Fremont Family 

Coalition, Families 1st Partnership) to 13 (Hall County Community Collaborative).

• Prior to the start of CYI, the collaboratives were focused on youth, families, or early 

childhood. The Sandhills Collaborative was specifically focused on services for youth 

entering the court system. 

• The collaboratives met on a monthly or bimonthly (every two months) basis.

• Each site had a slightly unique approach to CYI. For example, Southeast Nebraska Older 

Youth Collaborative split the Central Navigator position into two roles because the 

geography served warranted another position, while the Norfolk Family Coalition formed 

with collective impact in mind. 

25 Community Response is a system of supports and services for children and families to prevent their unnecessary entry into the 

child welfare system and/or other high-end systems of care.
– 35 –

Table 5. Contextual Factors, by Collaborative 

Context/ Feature Southeast Nebraska 

Older Youth 

Collaborative 

Sandhills Collaborative Families 1st

Partnership 

Fremont Family 

Coalition 

Hall County Community 

Collaborative (H3C) 

Norfolk Family 

Coalition 

Major partners Blue Valley Community 

Action Partnership 

(BVCA) & Southeast 

Nebraska Community 

Action Partnership 

(SENCA) 

Through the Eyes of the 

Child – 8th Judicial 

District Team & Central 

Plains Center for Services 

Families First 

Partnership 

with 501 C3 

pass-through 

sponsorship by 

Mid-Nebraska 

Community 

Foundation 

Fremont Family 

Coalition & 

Fremont Area 

United Way 

Hall County Community 

Collaborative; Region 3 

Behavioral Health & 

Heartland CASA 

Norfolk Area 

United Way & 

the Social 

Innovation 

SubCommittee 

Number of 

counties involved 

in CYI 

9 6 1 1 13 4 

Collaborative 

existed prior to 

CYI 

Yes Yes Yes Yes Yes Yes 

How long had 

collaborative or 

subgrantee been 

in place prior to 

CYI 

BVCA Est. 1966 Central Plains Est. 1995 

Families First 

Partnership 

Est. 2011 

Fremont Family 

Coalition Est. 

2012

H3C Est. 2005

Norfolk Family 

Coalition Est. 

2013

Original focus of 

the collaborative 

Services for families and 

youth 

Services for youth 

entering court system 

Services for 

youth to 

become selfreliant 

Early childhood 

Prevention system 

change; collective 

impact 

Early/young 

children 

Meeting 

frequency Monthly Monthly Monthly Monthly Bimonthly Bimonthly 

Unique feature Central navigation was 

divided into 2 roles Coalition led by a judge Led by health department 

Already had a 

community 

prevention 

system for older 

youth in place 

Did not continue after 

2019

Formed to 

have a 

“collective 

impact”

Community 

Response prior to 

CYI 

No No Yes Yes Yes Yes 
– 36–

Funding

The funding for each collaborative is another important feature of how each site approached 

implementation of CYI. Each collaborative applied for between $100,000 and $150,000 from the SIF CYI 

grant. Due to the structure of the grant, the collaboratives were each required to come up with 

matching funds that were not from other federal grants. Though some collaboratives’ applications were 

not explicit about their matching funds, almost all collaboratives outlined some kind of match in their 

budget appendix. 

Table 6. Grant and Matching Funding, by Collaborative, as Noted in Their SIF CYI Applications

Collaborative Average SIF 

Subaward

Per Year

Average Match 

Requirement 

Per Year

Examples of Matching Funding Sources

Southeast Nebraska 

Older Youth 

Collaborative

$150,000 $80,994

• Nebraska Children & local foundations

• Jefferson County (Crime Commission)

• City of Fairbury

• Blue Valley Community Action

Partnership

• Southeast Nebraska Community Action

Sandhills 

Collaborative $100,000 $63,318

• Nebraska Children & local Foundations

• Central Plains Center for Services

• Centris Federal Credit Union

Families 1st

Partnership $150,000 $25,448

• Nebraska Children & local foundations

• Centris Federal Credit Union

Fremont Family 

Coalition $150,000 $56,216

• Nebraska Children & local foundations

• Fremont Area United Way

Hall County 

Collaborative $150,000 $96,618

• Nebraska Children & local foundations

• Hall County Juvenile Justice Services

• Centris Federal Credit Union

Norfolk Family 

Coalition $150,000 $72,250

• Nebraska Children & local foundations

• Bank of the West

• Norfolk Area United Way

• Madison County (Crime Commission)

• St. John’s Benefaction Fund
– 37 –

What Are the Variations in How CYI is Implemented in Communities 

(E.g., Services Offered, Services Accepted, Participation)?

Implementation Timeline 

Each site implemented the different components of CYI at its own pace, according to the needs of its 

communities and its capacity to ramp up implementation. Figure 4 shows the timeline of 

implementation of the different CYI components among the sites. Every site began implementing the 

coaching component between July 2016 and February 2017. Similarly, each site had a Central Navigator

role established by sometime between April 2016 and February 2017. The Support Services Fund was 

implemented starting in August 2016 and was in all communities by January 2017. The Youth Leadership 

component was first implemented in August 2016 and was in all communities by October 2017. The 

Opportunity PassportTM (OP) component took the longest to implement. It began in April 2016 and was 

in all communities by November 2017. 

Figure 4. Implementation Timeline 
– 38 –

Implementation Perceptions 

CYI communities and their evaluation liaison held monthly or bimonthly calls throughout the grant to 

discuss the implementation of CYI, each of the CYI components, the evaluation, and the successes and 

challenges faced in the communities. After analyzing the interview notes from these calls, the themes 

that emerged were related to recruitment and challenges with implementing specific CYI components. 

Although recruitment was a constant challenge for sites, recruitment did increase as CYI components 

were implemented. Challenges with offering OP and Youth leadership were attributed to the rural

nature of the communities and to difficulties with transportation. Sites generally agreed that support

services funds and coaching were successful components in in CYI implementation. 

Based on an examination of rollout of the CYI components across sites, and feedback from staff working 

at each site, recruitment into CYI increased as components of CYI began to be implemented in each 

community, especially when OP was one of the components that had been set up in each community. 

For example, some communities saw waitlists for OP soon after the program was offered. Some sites 

also experienced delays in offering OP, which were mainly attributed to the initial challenges with 

securing financial partners in banking institutions or auto dealerships. Other sites had to resort to 

offering individual, one-on-one OP classes instead of a group class because of the geographical spread in 

these rural counties and related difficulties that young people had with transportation and coming to a 

common location for the class. Another community had to cancel some scheduled classes due to lack of 

interest and attendance. Despite these challenges, OP seemed to be a popular program with young 

people across all CYI communities. 

The Youth Leadership component of CYI was both popular and successful in some sites and a challenge 

to implement in other sites. Three sites reported success early on with implementing the Youth 

Leadership component. Sites reported attracting a core group who attended regularly, and one site even

had young people eventually age out of the program, requiring them to recruit new participants for 

their Youth Leadership program. In sites that found this component to be a challenge, the challenge was 

both in defining what a Youth Leadership event might be and in finding a person or organization to 

implement Youth Leadership. There were challenges with trying to embed Youth Leadership for CYI 

within existing leadership programs for young people. This was a challenge because CYI brought in a mix 

of populations. For example, a school-based leadership program did not fulfill the opportunity for young 

people to become involved with and connected to the larger community.
– 39–

The implementation of the support services fund and coaching were considered by sites to be a success 

throughout the grant period. Some sites reported that they enrolled young people to capacity in 

coaching. The challenges with coaching were mainly around the forms and documentation of coaching 

activities that were necessary for grant compliance and for tracking services received by young people 

through CYI. The support services fund was implemented around the same time period and 

implemented smoothly in all sites. Some site staff talked about what an important component this was 

for engaging young people, although sites emphasized the importance of describing the funds as part of 

the services to young people and not just as a way to pay a bill. One site talked about the need to put a 

$500 cap on the support services fund, with exceptions as needed, to help with budgeting and planning 

purposes. The challenge with the support services fund was that it was an expensive component of CYI.

Many sites hope to sustain the fund but recognize the challenge of doing so, due to the expense.

Participation 

The Youth Database tracked coaching, support services funding, and Youth Leadership activities, and a 

separate program-specific database tracked Opportunity PassportTM. For each of the components, 

subcategories about the specific service or hours and dollars spent were also tracked for each 

participant. The data presented in this section reflect services received through March 2019.

26 Across all 

the CYI communities, 741 young people received some type of service, with many receiving more than 

one. Coaching was received by 401 participants, the category with the highest number of participants 

receiving services, followed by Youth Leadership, Opportunity PassportTM, and the support services fund 

(Figure 5). 

26 The following figures represents data from June 2016 through March 2019; however, the dates vary individually by collaborative. 

See Figure 4 for a timeline of implementation for each CYI component for each collaborative.
– 40 –

Figure 5. Percentage of CYI Participants Receiving Services (N=741) 

In terms of percentage of participant engagement in CYI components, coaching was utilized by 54.1

percent of all participants accessing services, whereas the other three components are each utilized by 

about 25.0 percent. When the services are examined by site, coaching has the greatest number of 

service recipients in all sites except Norfolk and Fremont (Figure 6). Table 7 presents the variation in 

how young people experienced single or combinations of CYI services. The coaching service only was 

used by 21.5 percent of young people, followed by support services only (6.2%), Youth Leadership only 

(5.1%), and OP only (1.5%). The combination of two CYI services utilized by the largest percent of young 

people was coaching and OP (7.7%), followed by coaching and Youth Leadership (4.5%). Young people 

who experienced three CYI services were most served by coaching, OP, and Youth Leadership (3.9%). 

Finally, 5.7 percent of young people experienced the four CYI services (coaching, support services, OP, 

and Youth Leadership).

 
– 42 –

Figure 6. Number of CYI Participants Receiving Services, by Site 

Table 7. Combination of Services That CYI Participants Received 

Combination of Services Received Count Percentage 

No services recorded 220 29.7 

Only coaching 159 21.5 

Coaching & OP 57 7.7 

Support services only 46 6.2 

Coaching, support services, OP, & Youth Leadership 42 5.7 

Youth Leadership only 38 5.1 

Coaching & support services 37 5 

Coaching & Youth Leadership 33 4.5 

Coaching, OP, & Youth Leadership 29 3.9 

Coaching, support services, & OP 22 3 

Coaching, support services, & Youth Leadership 22 3 

OP & Youth Leadership 12 1.6 

OP only 11 1.5 

Youth Leadership & support services 6 0.8 

OP & support services 4 0.5 

Support services, OPP, & Youth Leadership 3 0.4 

Total 741 100
– 42–

Key findings about youth participation in each of the components of CYI are summarized in the following 

list. 

• Coaching. Sandhills Collaborative had the highest average number of hours per participant

(43 hours); Norfolk had the lowest average number of hours per participant (8.5 hours).

The most common type of coaching indicated across sites was financial coaching.

• Support Services Fund. The average amount of support services funding spent per CYI

participant ranged from $579.39 in Families 1st Partnership to $975.25 in Fremont Family

Coalition. Housing was by far the most often used category for the support services fund.

• Youth Leadership. The number of Youth Leadership events held across the sites ranged

from Sandhills Collaborative having 13 events to Families 1st Partnership having 102

events. The subcategory with the most Youth Leadership events was activity programing

(84), followed by leadership opportunity and youth leadership curriculum (78) and service

learning (62).

• Opportunity PassportTM. The OP data included in this study are from up through March

2019. Up to that point, 175 participants across the CYI communities had participated in OP.

Of those, 36 participants had made an asset purchase through the program (ranging from 1

in Norfolk Family Coalition to 10 in each of three sites: Hall County Collaborative, Fremont

Family Coalition, and Southeast Nebraska Older Youth Collaborative). Youth could make

more than one asset purchase, which accounts for some of the differences between

communities. CYI participants continue to engage with OP, and some have made additional

progress toward asset purchases since these data were collected.

What Challenges Are the Subgrantees Facing in the Implementation of 

the CYI Model?

The challenges facing implementation can be categorized into two overall themes: youth- and 

component-level challenges, and collaborative-level challenges. Youth- and component-level challenges 

include specific challenges youth are facing in their lives as well as challenges around recruitment and 

implementation of two CYI components: Youth Leadership and OP. Collaborative-level challenges 

include issues around grant compliance and partnerships.
– 43–

Youth- and Component-Level Challenges

The CYI communities faced several challenges with the implementation of the various components, and 

youth experienced challenges that impacted their engagement in CYI. Many of these challenges remain, 

although some have evolved over time and some have been overcome. All the CYI communities were

able to implement each component of the initiative with varying degrees of success. The areas with the 

most challenges included recruitment, implementing OP, and implementing Youth Leadership. Related 

to the implementation of the CYI components are challenges that youth are facing in the CYI 

communities. Some of these challenges were described as surprises to the CYI sites and impacted their 

approach to and experience with CYI recruitment and implementation.

Surprises in Engaging Population. Communities were asked by the evaluation team in their final 

interviews what surprised them about engaging with this population throughout the grant period. The 

responses fell into a few categories: young people’s debt, difficulty recruiting eligible young people, 

health needs of young people, and how the rural nature of the communities impacted engagement of 

young people. A few sites also mentioned being surprised that even though they had the resources and 

CYI, young people were not coming to them. They realized that there was a lack of trust and that many 

of these young people are used to surviving on their own.

Recruitment. In the monthly interviews with the evaluation team, every CYI site mentioned at some 

point that recruitment was slow and presented a challenge. The suspected reasons for lower 

recruitment, suggested in site staff interviews, included that staff felt they had already reached most of 

the young people needing services in their communities. In rural communities, staff at some CYI sites 

felt they had reached a saturation point and had exhausted some efforts, locations, and partners, where 

most referrals had come from. Other sites experienced seasonal changes with recruitment, with staff 

suggesting that the summer is typically a difficult time to recruit and enroll young people. 

Components of CYI. Two components mentioned as being challenging in implementation included OP

and Youth Leadership. The challenges mentioned about OP included finding banks and financial 

institutions to partner with for the program, which resulted in implementation delays. The influx of new 

potential participants or, in some cases, holding centralized classes for young people in a rural 

community were also challenges related to OP. The challenges with Youth Leadership included engaging 

only a small group of young people, and similar rural challenges as encountered with OP, as well as 

challenges of defining what was considered a Youth Leadership event.
– 44–

Collaborative-Level Challenges

Members of each of the CYI communities were asked to reflect on their experience and the evolution of 

the collaborative over the course of the grant. They were also asked about specific challenges that 

emerged in the collaborative relative to partner engagement and grants management and compliance.

Partner Engagement. Interviewees from many CYI communities talked about how the collaboratives’

membership and engagement of members ebbed and flowed throughout the grant, depending on their 

own organization or professional priorities. Collaboratives also faced challenges related to defining their 

focus, logistical changes, and engaging specific sectors of the community in the collaborative, such as 

stakeholders from probation and education sectors.

Grant Compliance. All six CYI communities said that the matching funds requirement was a challenge. 

Some said there was confusing information about both the match requirements and eligibility for 

funding in the beginning of the grant. Over time, there was not as much confusion but more concern (or 

stress) about meeting the match requirement. A few of the sites had not previously been part of a

federal grant, so the auditing and tracking process was new to them. Other grant requirements also 

posed challenges to some communities. For example, one site struggled to determine what could be 

spent with support service funds. Additionally, the exclusion of youth who were on probation was 

another challenge discussed across sites. The paperwork (e.g., coaching forms) was another challenge 

specifically mentioned by one community as “a source of frustration.”

What Positive Effects Are Subgrantees Seeing?

The CYI communities were asked about the positive effects they were seeing both with the young 

people they serve and with the CYI collaboratives they are part of. There were numerous examples of 

young people getting jobs, increasing their financial literacy, saving for a car to improve access to jobs 

and services, and finding reliable housing. There were also examples of collaborative successes, such as 

how new partnerships were engaged and sustained over the period of the grant, and many said these 

partnerships will continue long after the grant ends. Awareness was raised about a population of young 

people living in the community that were not connected and now have a range of services available 

from businesses, banks, service providers, schools, and others who have been trained and understand 

the need for positive youth development. The themes that emerged around positive effects of CYI were 

discussed as they relate to participant-level and collaborative-level effects. 
– 45–

Participant-Level Success

There were many stories about young people who had engaged in CYI and gradually started to see 

successes as the result of participating in one or more of the components of CYI. Table 8 provides brief 

examples of individual or community success stories mentioned during the final evaluation interviews. 

Table 8. Examples of CYI Participant Success

CYI Site Participant Success Examples

Southeast 

Nebraska Older 

Youth 

Collaborative

Opportunity PassportTM Program successes including car purchases made by young 

people who took part in the program. 

Sandhills 

Collaborative

A young person involved in CYI went on to open a back account and was also able to 

take out a loan. This young person also found support in living with other CYI youth. 

Fremont Family 

Coalition

Four or five youth went through the career placement program with Metro, a program 

that provided job training and placement.

Norfolk Family 

Coalition

Opportunity PassportTM Program successes including car purchases made by young 

people who took part in the program. Other successes were the formation of 

partnerships in the collaborative and the engagement of youth as evident by the 

increase in attendance over the years at the CYI Christmas party. 

Families 1st

Partnership

A young person who had struggled was now enrolled in community college, was living 

in sober housing, and went on to become employed. Opportunity PassportTM Program 

asset purchases were also given as examples of success for youth in CYI.

Note: Hall County Community Collaborative did not participate in evaluation interviews in 2019 and therefore is not included in this table.

In addition to the individual anecdotes, other success themes emerged across the CYI community. 

Recruitment and retention were described by some sites as being successful, even though these aspects 

were challenging for others. CYI communities observed an increase in recruitment of young people into 

CYI as more components of CYI were made available in their community. For example, as OP and Youth 

Leadership were holding classes and events, communities perceived recruitment to increase. Another 

observation offered by one community was that the recruitment into CYI improved during the “survey 

months” because there was an intentional effort made to get in touch with each CYI participant in those 

months. (Every April and October, all participants were asked to complete a survey related to their 

perceived well-being across several dimensions.) Another success noted by at least one community was 

that over time they saw young people reconnecting with CYI after having once been involved in CYI in 
– 46 –

some way earlier but then not being in touch for a while. This reconnecting demonstrated to the 

community that these youths were comfortable with the connection that had been established earlier.

Interviewees offered examples about how young people were getting more involved in the community 

and events that involved or were developed specifically to serve young people. Examples included:

• Youth Summits that involved CYI young people in the planning stages

• Representation of young people on community councils

• Health care packages developed by Youth Leadership participants to provide information

on how to access healthcare, specifically for those who qualify for Medicaid

Other successes pointed to how CYI organized access to services for young people, such as:

•

“One Stop Wednesdays” in one site, in which agencies come together and make staff

available to meet with young people

• Services to expectant parents, intersecting with CYI through new grants and funding

Collaborative-Level Success

Interviewees were asked to reflect on what positive effects they have seen in the collaborative and in 

the community as a result of CYI. They offered examples related to communication and the formation of 

new partnerships, encouraging engagement of core CYI collaborative members, and observations of 

how the community is better serving CYI young people.

Community Partnerships

When asked about what the collaboratives have excelled in, many interviewees talked about improved 

community partnerships. Some named specific components of CYI, such as coaching and OP, or positive 

youth development in general. Others talked about the experience of the partnerships formed as part of 

the collaborative in terms of learning more about what different agencies in the community do and how 

they can serve the CYI population. Other examples that were offered included how members have 

observed a lack of competition among agencies, how there has been a core group of partners who have 

come together for CYI, and that the communication among these partners to serve young people has 

been positive. One site specifically mentioned the partnership with banking businesses on financial 

literacy and OP, noting that the information they received through CYI was new to them, and having the 

businesses at the table “has been significant.”
– 47–

Engagement of some collaborative members has been a challenge for sites. However, the sites did see 

results in engagement when employing certain strategies. One such strategy was to rotate facilitation 

and responsibilities of meetings across different members. The goal of this strategy is to make all 

members feel equal and to share in roles of managing the meetings and not letting it fall to only a few 

people. One community said that the presence of the Community Response system has helped with CYI 

collaborative engagement and that the service array meeting that was held for Community Response

was an opportunity for partners to talk more about CYI. There was also the observation that those who 

come to regular meetings remained engaged in CYI and were invested. Some sites said they saw 

improvement in both regular attendance at meetings and relationships among members over time. One 

interviewee commented that meetings are more collaborative now than had been the case with more of 

a report-out style that had been done in the beginning of the grant period.

Positive Changes in the Community

Interviewees were asked how the community has changed to support young people since the beginning 

of CYI. Although not all answers were positive, and a few said they had not seen as much change as they 

would like, many responses did indicate success and positive change. One theme across communities 

was the newfound awareness of the population served by CYI. At least three communities said that the 

development of an awareness that unconnected young people are living in the community was a 

positive change. One person went on to add that there was also recognition of a “deeper issue” or 

“mind shift change” in the community: an appreciation about what situations these young people face 

and how the community can come together to address those situations. One example that was offered 

was that partners have come together for CYI, resulting in a change in practice. The police department 

and schools began to shift their approach to how they addressed this population and searched out 

referrals for services for young people. Employers and landlords were also noted as examples of how 

the community has come together “to give them a chance.” Several communities mentioned that there 

has been a change in how people may have judged young people in the past and that now there is less 

negative talk about them. Finally, the availability of resources and services for young people to be 

successful in the community was noted as a positive change for the CYI communities. 
– 48–

To What Extent, If Any, Has the Focus on Collective Impact Increased 

the Sustainability and Capacity of the Collaboratives? 

CYI communities were asked to reflect on how the collaborative changed over time, specifically changes 

in the communication, data sharing, and member engagement. For each of these, interviewees reflected 

on both the positive and the challenging aspects of the collaborative.

Communication

Collaborative members described how the communication improved over time. Many members across 

the communities recognized that in the early days of CYI, the collaboratives spent the majority of their 

time learning and understanding the CYI concept, theory of change, and individual components. 

Eventually less time was spent on these topics and more time was spent on implementation. The initial

lack of understanding or clarity about CYI did influence communication among collaborative members. 

For example, interviewees talked about how people initially came to collaborative meetings “to listen”

and learn about CYI, also described as “learning all together” about CYI. Interviewees described these 

early meetings as consisting of people only talking about “what they knew versus looking at things 

collaboratively” or their area of expertise. As one site noted, in the beginning there was a tendency to 

“only talk about frustration around what to do with these kids” and feeling discouraged about how to 

connect young people with the resources they need. 

Interviewees noted that over time there was a change in communication among collaborative members. 

Most sites said that communication improved. The improvement was attributed to a better 

understanding of CYI, sharing the facilitator role in the collaborative, forming a subgroup specific to CYI, 

keeping the same people engaged and coming to meetings over time, and feeling less siloed and more 

open with their communication. According to the CYI collaborative members interviewed, results of this 

improved communication are evident in the content and structure of the meetings. Examples included: 

the meetings having more meaning with key decisions that need to be made, meetings described as 

“more productive,” the content moving from the frustration of connecting young people to focusing on 

success stories, teamwork and strategies to fill gaps in services, and the collaborative nature of serving 

the same population across multiple agencies.

Interviewees also discussed communication challenges over time within the CYI collaboratives. One 

example was that securing certification as a 501C3 took up a lot of the collaborative’s time and focus, 

leaving little time for CYI. Other challenges related to communication were around the duration of the 
– 49–

grant period for CYI and members noting that discussions around sustainability often did not “go 

anywhere.” Interviewees also noted that once components of CYI were up and running, it was difficult 

to re-engage with the collaborative despite the improved engagement by participants.

Data Sharing

Interviewees were asked about how data sharing among collaborative members changed over time. 

Overall, members interviewed said that data sharing improved over time, but mainly within the context 

of the CYI Youth Database creation and evolution. Members talked less about sharing their own 

organizational data directly with each other or at meetings. Interviewees also acknowledged that 

members could feel overwhelmed by data or that engaging with data during meetings was challenging 

for any group, especially taking the next step to use the data to inform decision-making. Others said 

there was not much change in data sharing, that members were kept up to date on the CYI data, and the 

data were distributed at meetings. Others described examples of data sharing being useful for “telling 

where we need to focus our attention ... and [where we ] can really drill down into helping this 

population,” as well as seeing where the funds are being used and identifying other partners who may 

be able to help. Finally, one other member said the data were used as a reflective tool, specifically 

referring to the CYI data dashboards.

Member Engagement

CYI members were asked about how collaborative member engagement had changed over time. All 

those interviewed said engagement had either remained steady, with some ebbs and flows, or improved 

over time. Interviewees said that some of the members’ engagement had changed depending on their 

work responsibilities or when certain parts of CYI related more to their role versus others’ roles. Others 

said engagement grew over time as their services and other initiatives (such as Community Response) 

were also brought into the community to serve both the youth and their families. Another person noted 

that the engagement increased for those who were more invested, but engagement decreased over 

time for those who were not as interested in CYI. Some of the challenges noted for member 

engagement included the development of the 501C3 in one community, and the fact that many people 

are overcommitted and find it difficult to engage in CYI, beyond providing support as needed. For those 

who said engagement improved, they noted that meetings are perceived as more collaborative now,

relative to the typical “summary and report out” meeting that would take place earlier on. Another 
– 50 –

person noted that engagement had been steady, and that they have welcomed new partners recently 

who also engage with the same population as CYI.

Measuring Collective Impact Among Collaboratives

Over the course of the CYI evaluation, University of Omaha’s Juvenile Justice Institute conducted surveys 

in fall 2017 and fall 2018 across all collaboratives to measure the perceived strength of the 

collaboratives relative to the five core domains of collective impact: common agenda, shared 

measurement, mutually reinforcing activities, continuous communication, and backbone support. The 

response rates were considerably low and thus precluded the use of the data for modeling purposes. 

Further, the population identified for the survey varied considerably between survey administrations,

with nearly 35 percent fewer potential respondents in 2018 than in 2017. The findings do, however, 

illuminate some key factors about the collaboratives related to collective impact. The average number of 

people that sit on a CYI collaborative is 13. The breadth of agency representation among survey 

respondents varied widely across collaboratives, ranging from 2 to 10 agencies responding on behalf of 

a given collaborative. The results of the survey, while taken with an abundance of caution due to the low 

response rates and change in sampling frame, suggest the following:

• For the most part, collaboratives with low response rates self-reported higher levels of

collective impact.

• Differences in self-reported collective impact were negligible across survey years, with all

but “continuous communication” showing slight gains.

• Consistent with prior research, the domains of collective impact were positively correlated

with one another, and particularly between the backbone agency and its primary functions

— common agenda and continuous communication.

The results from the survey may not represent the full collaborative, but qualitative findings through 

interviews with coordinators do not indicate that nonrespondents would have changed the survey 

results. Nevertheless, while collaboratives indicated a united effort to serve the needs of young people, 

the survey was unable to measure the degree to which collective impact increased the capacity or 

sustainability of the collaboratives in these efforts. The lack of responses, however, may serve as an 

implicit indicator that the collaboratives have not yet achieved a high level of collective impact.
– 51 –

CYI Priority Components With Greatest Impact for Young People

Members of each CYI site were asked what components are having the greatest impact for young 

people. The answers varied across sites, with many interviewees offering several components instead of 

only one. Some also indicated nearly all of the CYI components. The top mentioned component was OP.

Some said OP is “life changing” and is about “having connections and resources...and learning to trust 

and have a relationship and have a plan.” Interviewees also noted how the coaching element combined 

with OP can create trust and provide support to young people in so many areas of their lives. The 

second most often mentioned component was coaching and was often described in combination with 

other components, such as OP, the services support funds, Youth Leadership, and the Central Navigator. 

One person said, “Central navigation has been the best for the overall community, to youth and families 

and anyone seeking services.” Finally, one interviewee noted that the training on collective impact was 

also a component that had the greatest impact for young people. The reason given was that in the past 

agencies had been used to running programs in silos, and after the multiple trainings on collective 

impact there was a group of people that “got it” and remained engaged in CYI, with its members being 

“the most engaging and collaborative.”

Sustainability 

Interviewees at each site were asked to reflect on which components of CYI would be the easiest to 

sustain, the hardest to sustain, and the most important to sustain. As mentioned in the preceding 

section, many of the CYI components were identified multiple times in each of these categories. The 

components identified as the easiest to sustain include OP, coaching, Youth Leadership, the Central 

Navigator role, and support services fund. The three components said to be the hardest to sustain were 

OP, Youth Leadership, and support services fund (also noted as the easiest to sustain). When asked what 

is most important to sustain, some said “all of it” or that it was hard to choose, and others said coaching, 

the Central Navigator, and support services fund.

Lessons Learned

The implementation of CYI experienced both successes and challenges. Both experiences occurred in all 

six collaboratives and for each of the CYI components. There are many anecdotes of success about

young people who engaged with CYI, whether through OP, coaching, support services funds, or other 

aspects of CYI. Young people were able to purchase cars through an asset purchase which allowed them 
– 52–

to get a job for the first time; young people were able to find housing and employment in their local 

community; coaching services enabled young people to access health services and support to address 

their immediate needs (housing, health care, parenting help, job training) and longer-term challenges 

(reducing debt, planning for further education or training, acquiring job skills). CYI also provided a 

platform to educate the collaboratives and broader community about this population, an unconnected 

population living in these counties that was in need of support. CYI resulted in new partnerships with 

financial institutions, employers, and others who had not typically been engaged with the collaboratives 

in the past.

There were also challenges implementing CYI. This population was not the original focus of many of the 

collaboratives, pivoting into this new area was difficult for them. The rural nature of the collaboratives 

also offered challenges to both the young people and the collaborative members. Attendance at 

meetings, engaging with activities, and transportation proved to be difficult for all involved in CYI. There 

were many ways these challenges were addressed, including more one-on-one OP classes or having a 

coach meet young people where they lived. For the collaboratives, some have started to move to a 

more virtual meeting space to make sure all can attend and not have to drive far. Recruitment into CYI 

was also a consistent challenge throughout the evaluation period across the sites. In some cases, 

building trust and engaging young people was challenging; in other cases, the sites felt they had reached 

most of the eligible young people in their community and thus had reached a level of saturation. CYI 

components, such as OP, took longer to establish in some communities than expected. Youth Leadership 

remains a difficult component to define and implement, with a number of different models attempted 

(consultant-led, existing Youth Leadership groups, collaborative-led).

The experience of CYI appears to have had an impact on the communities where it has operated, the 

people involved, and Nebraska Children, which oversaw this initiative. Several lessons have been 

learned around grant management, capacity building, and specific project components that will apply to 

future iterations of the CYI model:

• Balance implementation and grant compliance from the start. SIF grant compliance and

building capacity for grant management were large areas of focus for Nebraska Children

and the communities at the start of the grant period. This focus slowed the initial

implementation of the model and recruitment of participants.
– 53 –

•

In rural communities, there are fewer people to serve a multitude of roles. Nebraska

Children realized that major CYI components (grant compliance, accounting,

implementation, evaluation, etc.) were falling on the same few people in each

collaborative. Managing the multiple roles was a challenge, with each requiring different

skill sets and dedicated time to learn and execute. This sentiment was also discussed by the

collaboratives during their final interviews with the evaluation team. Interviewees

indicated that the majority of work to keep CYI going fell to only a few key staff members.

• Components aren’t always easily transferrable from other non-SIF communities. Youth

Leadership was one component of CYI that Nebraska Children staff did not feel reached its

full potential. Youth Leadership was not easily defined and therefore did not transition well

to CYI communities. Although there are other communities in Nebraska implementing

Youth Leadership successfully with Nebraska Children’s guidance, this component was not

one that sites in this study implemented consistently.

• Older youth are still a secondary priority. Nebraska Children staff reflected on the

experience of the collaboratives which had been focused on a different population prior to

CYI, such as much younger children. These collaboratives found it more difficult to shift

their focus to the needs of older youth.

• Understanding the needs of older youth evolved over time. Related to the prior lesson,

for communities that had not focused on older youth in the past, they were not familiar

with CYI participants’ needs. For instance, some were surprised by the amount of debt that

some young people had. Others did not anticipate the amount of time needed to build

trust and engagement with this population. Interviewees reflected on the fact that many of

these young people were used to fending for themselves, so engaging with them and

gaining their trust took time.

• Transportation can be a major barrier in rural communities. Every community expressed

that transportation was a major challenge to implementation. Since the communities are

rural and there is a lack of public transportation, most young people did not have the

transportation to attend events or classes.
– 54 –

Changes to the Subgrantee Evaluation Plan 

Overall there were no major changes to the planned implementation evaluation outlined in the 

Subgrantee Evaluation Plan. The evaluation approach, data collection, and analysis went largely to plan 

and yielded the data that were expected. The only changes to note were adjustments in the timing of 

two data collection activities: interviews with the collaboratives, and site visits. Instead of conducting 

the planned interviews twice a year with the collaboratives, the evaluation team conducted more 

frequent and ongoing interviews with a core set of staff from each site. These interviews varied in how 

often they were conducted, but on average were conducted once a month and then once every other 

month in the final year of the evaluation. This regular contact allowed the evaluation liaison for each site 

to build a rapport with the staff and address any data collection or implementation questions. Any 

questions the evaluation team could not address were then passed on to Nebraska Children staff to 

follow up on. Instead of annual site visits (which would have resulted in four visits per site), two site 

visits were conducted throughout the evaluation period. The two planned site visits varied in the 

purpose and data collection. The first visit focused on collecting formative data through observations 

and interviews. The second visit focused on data use and capacity-building; the evaluation team 

presented data to collaborative staff and engaged them in technical assistance on data use to build their 

capacity to use data for real-time planning and monitoring. 
– 55–

Impact Study

Design 

The impact study design was a matched comparison group design that examines outcomes among CYI

participants relative to outcomes among similar young people designated as a comparison group. The 

comparison group consisted of people who were eligible for the program and completed an initial 

Common Referral Form but otherwise did not take up services, as well as people who took up only a 

minimal amount of the CYI model’s services. The primary impact evaluation analysis was conducted on 

participant-level outcomes in which the individual was the unit of analysis. 

Matched Group Comparison of Participant-Level 

Estimates

The evaluation of CYI used a quasi-experimental research design to examine participant-level impacts of 

the program that was guided by the primary and exploratory research questions. Specifically, the study 

employed a propensity score matched group comparison study design. This approach required the 

following general steps:

1. Obtained participant-level self-report data from subgrantees, collected using the Common

Referral Form and Transitional Services Survey at intake.

2. Merged data from the six subgrantees into a master data file. Cleaned and assessed the

file for missing data. Reconciled any missing data with other subgrantee data, as possible.

3. Applied matching procedures, using the self-report data for young people to construct the

comparison group.

4. Ran univariate and bivariate estimates to examine unadjusted differences between

groups.

5. Produced primary and exploratory impact estimates between the treatment and

comparison groups.
– 56 –

Matching Procedure

The matching procedure is a critical factor in this quasi-experimental evaluation study. Short of a true 

experimental study, the matched comparison allows the evaluation team to control for observable 

differences between the treatment and comparison groups. There are several matching procedures that 

are considered reasonable approaches to control for observable differences between groups. Each 

attempts to control for selection bias of participants in the program. Any matched comparison design 

relies on the concept of “unconfoundedness,” or that by adjusting for observable characteristics, 

selection bias will be removed from the impact estimates.

27 The approach selected for the current 

evaluation is called a propensity score matching (PSM) comparison. This method involves creating a 

propensity score for each young person in the treatment (CYI uptake) and comparison (no CYI uptake) 

groups based on certain characteristics collected across the entire sample. A propensity score is a 

statistical process that estimates the likelihood that any one young person in the sample would have the 

necessary characteristics to be selected and enrolled in the program. The goal is to match a comparison 

young person to a treatment group young person as closely as possible on the propensity score. This 

matching should, in theory, reduce observable differences between the treatment and comparison 

groups, permitting more reliable estimates of the treatment (participation in CYI) effect. The following 

describes the propensity score matching procedures used in this study.

1. Merged all subgrantee data and reconciled issues with missing or nonvalid data entries.

The resulting file included all possible observable characteristics of young people that

were used to identify and enroll eligible participants.

2. Produced the propensity scores using the variables reflecting baseline characteristics of

young people. The scores were produced by using a logit model to estimate likelihood of

enrollment into the program, controlling for characteristics. Omitting important variables

from this model may produce biases in the impact estimates due to imprecision in the

propensity estimates28 (thus, the estimates err on the side of inclusivity).

3. The estimates were then used to produce a propensity score for each young person. The

variables included at baseline in the PSM model are:

27 Caliendo, M., & Kopeinig, S. (2008). Some practical guidance for the implementation of propensity score matching. Journal of 

economic surveys, 22(1), 31–72.

28 Caliendo & Kopeinig, 2008.
– 57 –

a. Indicator for prevalence of experience of:

i. Foster care or other alternative parenting situations

ii. Probation

iii. Homelessness

b. Count of:

i. Current services needed within the CYI domains of well-being

ii. Current services received

iii. Public assistance supports

c. Age

d. Percentage male

e. Race/ethnicity

All indicators were derived from the Common Referral Form. There was very little missing 

data on the Common Referral Form (less than 3%), and missing data were imputed using 

survey data for items if such data were available (e.g., age, gender) through other data 

sources.

29

4. The next step was to match treatment and comparison groups using a matching algorithm.

While the original design called for radius matching, ultimately nearest-neighbor matching

was used as the treatment and comparison participant had very similar baseline

characteristics. Specifically, nearest-neighbor matching with replacement was used to

account for the uneven balance between numbers of participants in treatment and

comparison groups. The treatment-to-comparison ratio in the sample is 4:1; thus,

individual comparison subjects were matched to multiple treatment subjects in order to

retain the maximum sample. A caliper of 0.2 was set to ensure matching occurred only

among subjects with very similar scores. Finally, an area of common support (see Figure 7)

was established and the analytic sample was limited to only those within the area of

29 Ideally, the matching procedures would have included covariates used to determine eligibility as well as baseline measures of 

the outcomes for the project. Unfortunately, there were considerable amounts of data within-item missing data on the baseline 

surveys, which limited the ability to utilize those variables for matching.
– 58–

common support. Less than 2 percent of participants fell out of this area (all were 

treatment participants), again indicating a good match between groups. 

5. Finally, once the matched pairs were formed, the characteristics of these groups were

examined to ensure groups were similar. The only statistical difference among matching

covariates that persisted between groups following the matching procedures was age. The

statistical difference is presented further in the Analysis and Results section. To control for

this difference, age was included as an additional coefficient in the impact models.

The results of the matching procedures can be seen in statistical terms in Table 9 and in visual terms in 

Figure 7. While, baseline equivalence between the treatment and comparison group is statistically 

similar across all matching covariates other than age, there were multiple baseline measures of 

outcomes that remain statistically different between groups following matching procedures. 

Furthermore, these differences are greater than 0.25 standard deviations for the following covariates: 

age, school enrollment, savings account status, sufficient savings for expenses (Table 9). The statistical 

adjustments and implications for baseline statistical differences are further discussed in the Analysis and 

Results section and the Conclusion section of this report.
– 59 –

Table 9. Baseline Statistical Equivalence Between Groups After Matching

Comparison 

(N=90)

SD Treatment 

(N=328)

SD Raw Difference Pooled SD Standardized 

Difference

Significant Outcome Measure

Age 19.54 2.249 18.62 2.486 0.93 2.370 0.39 **

Male 0.37 0.485 0.45 0.498 -0.08 0.491 -0.17

Race/Ethnicity

White 0.71 0.456 0.68 0.466 0.03 0.461 0.06

Black or African American - - - - - - -

Asian - - - - - - -

AIAN - - - - - - -

NHPI - - - - - - -

Multiple Races 0.08 0.280 0.08 0.272 0.00 0.276 0.01

Hispanic or Latino 0.10 0.297 0.08 0.272 0.02 0.285 0.06

Alternative Parent/Guardianship Experiences0.78 0.418 0.78 0.415 0.00 0.417 -0.01

Probation 0.38 0.488 0.30 0.457 0.08 0.473 0.17

Homelessness 0.33 0.474 0.30 0.460 0.03 0.467 0.07

Current Needs Count (Max=9) 2.50 1.900 2.50 1.770 0.00 1.836 0.00

Current Services (Max=10) 1.12 1.560 1.33 1.660 -0.20 1.611 -0.13

Current Public Assistance 

(Max= 5)

0.92 1.250 0.95 1.000 -0.03 1.132 -0.03

Hope Score 5.85 1.330 6.13 1.450 -0.28 1.391 -0.20 ** X

Stable Living 0.76 0.432 0.78 0.412 -0.03 0.422 -0.06 X

Safe and Stable Living 0.73 0.446 0.77 0.421 -0.04 0.434 -0.09 X

Enrolled (in school) 0.44 0.499 0.68 0.485 -0.24 0.492 -0.48 ** X

Emergency Room Visits 1.33 3.200 0.87 1.770 0.46 2.586 0.18 X

Transportation Access to 

Job/School

0.86 0.352 0.92 0.280 -0.06 0.318 -0.18 X

Money to Cover Expenses 0.35 0.479 0.63 0.485 -0.28 0.482 -0.58 ** X

Savings 0.21 0.412 0.36 0.479 -0.14 0.447 -0.32 ** X

* <.05 ** <.01 ***<.001
– 60 –

Visually, the two groups appear similar in terms of the distribution of propensity scores derived from the 

enrollment data. Figure 7 provides a visual depiction of the propensity scores among treatment and 

comparison individuals for those within and outside of the area of common support. While there is 

some variation in the distributions, overall the distributions follow a similar pattern between groups.

Figure 7. Distribution of Propensity Scores and Area of Common Support

The terms “on support” and “off support” represent those participants who were included in the matched sample following the propensity 

matching procedures. There were six treatment participants who fell off support (i.e., not in match sample) due to lack of a similar 

participant in the comparison group based on their propensity scores.

The statistical and visual analyses suggest that the final matching model produced two groups that are

similar in terms of observed characteristics associated with enrollment in CYI, but lack equivalence for 

age and multiple outcomes of interest. While age and baseline measures of the outcomes are included 

in the models to reduce the potential issue of selection bias improving internal validity, the difference 

between groups falls outside the range defined as acceptable for CNCS evidence standards. Implications 

for these differences are discussed in the Conclusion section of this report. 
– 61 –

Study Participants and Sample Flow

The sample for the CYI evaluation study is derived from young people eligible for the program across six 

collaboratives serving more than 40 counties in the state of Nebraska. Again, the eligibility criteria for 

young people to enter the program are unconnected young people aged 14–24 who find themselves 

disconnected from a positive life course because they:

• Are currently or have been in the Nebraska foster care system,

• Have had contact with child protective services,

• Have had juvenile justice system involvement (including diversion or young adults

transitioning out of Probation), and/or

• Have experienced homelessness or near-homelessness.

Due to the nature of the evaluation, any young person eligible for the program and identified by the 

collaborative was invited to participate and comprises the sample from which the treatment and 

comparison groups were constructed. Of the people who were invited, not all took up services; and 

among those who did, the extent to which individuals engaged in programming varied considerably (see 

Implementation Evaluation section earlier in this report). This variation in program engagement was 

leveraged to construct the comparison group. The study was originally designed to compare CYI 

participants to a similar matched group of young people who did not take CYI services; however, in 

practice the variation in program engagement led to a more inclusive eligibility criteria for the 

comparison group. Prior to examining participant outcomes, and after consultation with Nebraska 

Children and implementation staff, including partners who provide coaching, the research team

determined that true treatment should be defined as more than two months or two contacts with 

coaching staff. The researchers also determined that access to one-time support service funding (i.e., 

emergency funds) should not be considered true engagement with CYI and should therefore not be 

considered as a treatment eligibility criterion. Thus, the newly constructed eligibility criteria for the 

matched comparison study are as defined in Table 10.
– 62–

Table 10. Definitions of Treatment and Comparison Groups Within the CYI Study

Group

Support Service 

Funds (# of times)

AND/ 

OR

Coaching

(# of sessions 

or months)

AND/ 

OR

Youth 

Leadership

AND/ 

OR

Opportunity 

Passport

Treatment

2 or more 2 or more 1 or more 1 or more

Comparison

Fewer than 2 Fewer than 2 None None

Any combination of the service array described above would result in assignment to the group labeled in the left column. 

The study flow of participants derived for the treatment and comparison groups is described in Table 11.

The impact study only includes those young people who completed at least one transitional services 

survey at any time during their involvement in CYI. Thus, the sample described in the Implementation 

Evaluation section earlier in this report varies from the sample described in the following sections, which 

focus on the CYI impact study.

Table 11. Sample Flow of Study Participants

Study Time Period Number of People 

Included

Number of People 

Not Included

Notes

Enrollment 960 0

All participants who completed 

enrollment materials

Consented 724 236

219 young people either actively did not 

consent to share data for purposes of 

the evaluation or their consent 

materials contained discrepancies for 

which the default decision was to 

remove data from the study.

Additionally, 17 young people engaged 

in services across multiple sites. These 

young people are reported as duplicates 

in the implementation study but treated 

as one unit in the impact study sample. 

Baseline Survey 418 306

The removed young people enrolled in 

CYI and consented to share data but did 

not complete a survey at any time 

during enrollment. 

Assignment to Study 

Group 328 90

The removed young people enrolled but 

did not meet the definition of eligible 

treatment and thus represent the 

comparison group for the study.

Treatment Allocation 328 0

Given the observational nature of the 

study, all young people assigned to 
– 63 –

Study Time Period Number of People 

Included 

Number of People 

Not Included 

Notes 

treatment took up some level of 

treatment services that met the 

definition of treatment allocation. 

First Follow up 

294 (246 

treatment and 48 

comparison) 

124

These young people completed a 

baseline survey but not a follow-up 

survey. 

Second Follow up 

166 (148 

Treatment and 18 

Comparison) 

128

These young people completed one 

follow-up survey but did not complete a 

second follow-up survey. 

Primary Analytic Sample 256 38

Among the 294 young people in the first 

follow-up sample, 256 had data that 

allowed for analysis in the primary 

analytic sample. 

Consent Process 

The consent process was altered partway through the evaluation period due to significant challenges in 

acquiring consent from members of the CYI target population. Consent was difficult to acquire due to 

participants’ transiency, relationships to parents and guardians, and the nature of when and where 

program staff interacted with participants. Interaction often happened in the community and at times 

when the program staff did not have access to study materials, including consent materials. The process 

was revised to allow for a waiver of consent for all enrolled participants who did not have an 

opportunity to consent to the data sharing component of the study, as indicated by no documentation 

of a consent form in the young person’s file.30 Of the 960 young people who enrolled in CYI across the 

six SIF sites, 236 participants either opted to not share their data for the evaluation or may have been 

presented the opportunity to assent but did not sign the form or did not complete it entirely. Thus, the 

enrolled sample for whom data were accessible to the evaluation team totaled 724 participants. 

Attrition 

Of the 714 participants who consented to provide data, only 57.0 percent (N=418) completed a baseline 

survey and were considered eligible to be included in the analytic sample. Furthermore, of this baseline 

analytic sample, just 70 percent completed at least one additional follow-up survey after three months 

or longer following the baseline survey. Unfortunately, the differential attrition was considerable. The 

attrition for the treatment sample at first follow up was 25.0 percent, whereas the attrition rate for the 

30 The waiver of consent was reviewed and approved by WestEd’s Institutional Review Board in Year 3 (2018) of the project. 
– 64 –

comparison sample was 46.7 percent, resulting in a differential attrition rate of 21.7 percent at the first 

follow-up period. Due to high overall and differential attrition in the study, both the treatment and 

comparison groups were examined to determine whether the characteristics of the young people who 

were retained at follow up were statistically different from those who were lost to attrition. 

A paired t-test was conducted to compare baseline characteristics and measures of the outcomes for 

both treatment and comparison groups. The average age for the treatment group among both retained 

and those who were lost to attrition are statistically similar; however, among treatment participants, the 

percentage of male participants as well as the baseline measures of each of the outcomes are 

statistically different between these groups (retained vs. those who were lost to attrition) among 

treatment participants. Within the comparison group, both demographics and baseline measures of 

outcomes are statistically equivalent among those retained compared to those who were lost to 

attrition from the study. Statistical methods were used to retain the full sample by drawing on 

information available for each participant within the outcome models. Specifically, rather than impute 

the data, Full Information Maximum Likelihood (FIML) models were used to retain the sample, based on 

available data. While the FIML models do retain the full sample for estimates at follow up, the statistical 

differences in the retained study group, particularly the treatment group, are a notable limitation in the 

study.

Changes to the Subgrantee Evaluation Plan

The changes to the Subgrantee Evaluation Plan (SEP) are largely a result of the attrition experienced at 

each stage of the data collection process. Although the baseline survey was designed to be administered 

as part of the enrollment process, a large proportion of CYI participants did not complete the survey or 

completed a survey more than 60 days after their date of enrollment. Further, many participants did not 

complete a follow-up survey or completed a survey very close in proximity to the baseline survey (i.e., 

less than 90 days following baseline). These issues reduced the overall analytic sample at each time 

period. In addition, the SEP proposed an examination of system-level impacts related to the Collective 

Impact model that CYI was situated within at each of the communities. While system changes are 

measured and reported as part of the implementation study, the impact study did not include any 

system-level outcomes. Sites were asked to complete a survey to assess system changes across discrete 

dimensions of Collective Impact, but the response rate was extremely low, rendering the data unusable 

for measuring impact or even providing a descriptive understanding of system-level change.
– 65 –

Measures

The primary impact evaluation was conducted on participant-level outcomes in which the individual is 

the unit of analysis. The primary outcome is defined in the research questions (listed earlier in the 

report) as overall well-being and was operationalized using measures from a survey developed by 

Nebraska Children in collaboration with the WestEd evaluation team.

31 A participant survey (entitled 

Transitional Services Survey) was the source of data for the computed Well-Being Scale (and subscales) 

and other measures of interest that contribute to the CYI impact study. 

Well-Being Scale

The primary measure of interest for the impact investigation was a Well-Being Scale that was developed 

by WestEd researchers with the intention of capturing a holistic cross-section of well-being for CYI 

participants. The scale was composed of items from the Transitional Services Survey (TSS) that focus on 

participants’ finances, education, housing, employment, health, transportation, and permanence. The 

first step in developing this scale was to use item mapping32 to identify items that represented these 

subconstructs. After mapping, all identified items were used to conduct exploratory and confirmatory 

factor analyses. These analyses produced four correlated factors, or categories, that served as 

subconstructs, or components of the overall well-being scale. The relationship between a survey item 

and an overall construct is called its factor loading. Factor loadings are used to determine overall 

constructs as well as the relative strength of relationship between individual items and constructs that 

are surfaced in the analysis. Through this process, four constructs were revealed. The following labels 

were given to the factors based on the items that loaded to the factors, supplemented by contextual 

conversations with Nebraska Children personnel: (1) Health Access, (2) Social Support, (3) Housing and 

Financial Stability, and (4) Transportation and Employment. 

Reliability statistics were computed using each survey item underlying each factor across the three 

survey periods (Table 12). Cronbach’s coefficient alpha was used to assess reliability. The closer the 

coefficient alpha is to 1, the more reliable a scale is considered. In statistical terms, the closer the alpha 

is to 1, the greater covariance exists amongst the items that make up the scale. A “good” measure of 

31 The Transitional Service Survey is an adaptation of the Opportunity Passport Program survey and has been further refined for

the CYI SIF evaluation. 

32 Item mapping involved identifying current survey items and mapping them against the domains of well-being defined within the 

CYI logic model.
– 66 –

reliability has an alpha that exceeds 0.65. Alpha statistics of less than 0.5 are considered less reliable. 

The alpha statistics should generally improve over time because participants have already had 

experience taking the survey during a previous data collection period. Reductions in alpha statistics over 

time could be a result of smaller sample sizes. Also, factors that have fewer items associated with them

generally tend to be less reliable, and factors with more items tend to be more reliable. An explanation 

for the lower alpha scores in this study could also be related to the number of items that were “crossloaded” that make up the scale. These are items that could have been placed with other factors but, due 

to the strength of the correlation and based on discussions with program staff, a decision was made to 

place these items with a particular factor rather than other possible factors.

Table 12. Factor Loadings, by Well-Being Scale

Factor Number

of items

Number

of cross 

loadings

Baseline 

alpha

Follow-up 

1

alpha

Last 

follow-up

alpha

Health Access 5 1 .70 .68 .64

Social Support 3 0 .82 .83 .85

Housing and Financial Stability 8 1 .67 .63 .64

Transportation and Employment 4 3 .52 .49 .50

The four factors of Health Access, Social Support, Housing and Financial Stability, and Transportation 

and Employment were standardized and combined to create the overall Well-Being Scale. Each factor is 

considered an index, or total, of individual standardized survey items that contribute to the factor. 

Reliability statistics were computed for the overall Well-Being Scale. The coefficient alphas for the 

overall measure were 0.67 at baseline, 0.63 at follow-up 1, and 0.56 at last follow-up. In addition, to test 

content and internal structure validation analyses, the evaluation team conducted response process 

validation tests through in-person focus groups with program staff at each of the project sites. Staff 

reaffirmed the reliability of the individual items contributing to each of the factors constructed for the 

study. Furthermore, test content and face validity tests were conducted on individual items from the 

survey instrument from which the TSS was derived. While these tests predate the SIF project, the study 

team was confident in the validity of the instrument specific to the study’s sample population.
– 67 –

Hope and Executive Functioning

In addition to the constructed Well-Being Scale, the Transitional Services Survey included items to assess 

changes in the young persons’ perceptions of hope and executive functioning. These measures are 

drawn from validated instruments: (1) the Child/Adolescent Wellness Scale (CAWS)33 and (2) the Adult

Hope Scale (AHS)34. The CAWS and AHS demonstrated strong internal reliability, with coefficient alpha 

levels between 0.75 and 0.84 with similar populations, respectively. The measures listed in Table 13

show similar measures of alpha, consistent with the earlier validation findings. 

Table 13. Factor Loadings, by Hope and Executive Functioning Scale

Factor # of Items # of cross 

loadings

Baseline alpha Follow-up 1

alpha

Last Follow-Up

alpha

Hope 6 0 .88 .91 .89

Executive Functioning 10 2 .89 .90 .93

As mentioned previously, the overall Well-Being Scale includes aspects of financial, education, housing, 

employment, health, transportation, and permanence statuses. WestEd, in collaboration with Nebraska 

Children, thought it necessary to explore individual items that contribute to these subconstructs of the 

Well-Being Scale. The following sections detail measures that were used to operationalize each 

subconstruct and that were examined as individual outcomes. 

Housing 

Four different measures of housing status were investigated from three TSS item responses. The first 

item, Safe Living, asked participants, “Do you feel safe in your current living situation?” to which 

participants respond “yes” or “no.” The second item, Stable Living, asked participants, “Do you feel that 

your current living situation is stable?” to which participants responded “yes” or “no.” The third item of 

interest asked participants, “What is your current living situation?” to which participants were free to 

select one choice from 14 different living situations. This item was recoded to a variable labeled 

Unstable Living, which categorized all participants who selected “couch surfing,” “homeless,” or “other”

33 Copeland, E. P., Nelson, R. B., & Traughber, M. C. (2010). Wellness dimensions relate to happiness in children and

adolescents. Advances in School Mental Health Promotion, 3(4), 25-37.

34 Snyder, C. R., Harris, C., Anderson, J. R., Holleran, S. A., Irving, L. M., & Sigmon, S. T. (2007). Adult Hope Scale (AHS). Journal of 

Personality and social psychology, 60, 570-585.
– 68 –

as “unstable living.” All other participants who selected living situations such as “with parents,” “group 

home,” “school dorm,” and so on, were categorized as “more stable living.” The final measure for 

housing was a combination of the first two items labeled Safe and Stable Living. Participants who 

selected “yes” to both the safe living and stable living items were categorized as “safe and stable living.”

All other participants who answered both items were categorized as “not safe and stable living.”

Employment

Five different measures of employment status were investigated from four TSS item responses. The first 

item of interest asked participants their current employment status, to which participants selected one 

response from among: “not employed and not trying to get a job,” “not employed, but trying to get a 

job,” or “employed.” This item was recoded to a dichotomous variable labeled Stable Employment, 

which combined the “not employed” categories into “unemployed” and retained the “employed”

category. The second item, Employment Length, asked participants the length of time they have been 

with their current employer. This was a 6-category variable ordered in three-month intervals (up to 2 

years) recoded into a 3-category variable: “unemployed”, “employed < 1 year”, and “employed ≥ 1 year.”

The third and fourth items of interest were self-reported Weekly Hours (“average number of hours you 

work per week”) and Hourly Wage (“dollar amount earned hourly”). The fifth measure is referenced as 

Employed or Enrolled in conjunction with the following section on education.

Education

Two different measures of education status were investigated from two TSS item responses. The first 

item of interest (labeled Educational Enrollment) asked participants their current educational 

enrollment status to which participants selected either “not enrolled” or “enrolled.” The second 

educational measure was a combination of Stable Employment and Educational Enrollment and was 

labeled Employed or Enrolled. Participants who selected “employed” or “enrolled” for responses to 

these two items, respectively, were categorized as “employed or enrolled.” All other participants who 

answered both items were categorized as “not employed and enrolled.” This measure is a response to 

the transitional age of CYI participants in which employment or education status alone is not a true 

measure of positive change for all participants. 
– 69 –

Health 

Five different measures of health status were investigated from seven TSS item responses. The first 

health item asked participants to select their insurance status from nine response options. The nine 

response options were recategorized into two options: “insured” and “uninsured” (labeled Insurance 

Status). The next item, Annual Physical, was a recategorization of responses to the question, When did 

you have your last physical exam..? Participants who responded “Less than 1 year” or “1 to 2 years”

were categorized as “yes” for receiving an annual physical. All other participants who answered “more 

than two years ago” and “never” were categorized as “no.” The third measure, ER Visits, was a 

continuous item that asked participants “How many times have you visited the ER room in the past 6 

months?” which was recategorized into “None,” “Once,” “More than once.” The final two measures,

Needed Care in Past and Unmet Needs Now, were both ordinal composite measures. The first was a 

“check all that apply” response to the item “Has there been a time over the past 6 months where you 

thought you should get medical care, dental care, or care from a mental health professional, but you did 

not or weren’t able to?” Response options of “No,” “Yes, ...medical...,” “Yes, ...dental...,” or “Yes,

...mental...” were combined into an ordinal measure ranging from 0 (“No” or none selected) to 3 

(depending on how many needs were selected). The Unmet Needs Now measure was a combination of 

three items that asks respondents “Do you have any unmet Physical or medical needs right now?” “Do 

you have any unmet dental needs right now?” and “Do you have any unmet mental needs right now?” As 

with the previous measure, response options to these three questions were combined into an ordinal 

measure ranging from 0 (all “No” selected) to 3 (depending on how many needs were selected).

Transportation 

Four different measures of transportation status were investigated from four dichotomous TSS item

responses. All four items had response options of “yes” or “no.” These items asked: “Do you have access 

to the transportation you need to get to school or work?” (labeled Transportation access for 

school/work); “Do you have access to transportation you need for things like therapy, medical 

appointments, supportive services, etc.?” (labeled Transportation access for health); “Is the 

transportation you use reliable and consistent?” (labeled Reliable Transportation); and if old enough, 

“Do you have a driver’s license?” (labeled Driver’s License). 
– 70 –

Finances 

Three different measures of financial status were investigated from three dichotomous TSS item

responses. All three items had response options of “yes” or “no.” These items asked: “In the past month, 

did you have enough money to cover your expenses?” (labeled Enough money to cover monthly 

expenses); “Right now, do you have a bank account?” (labeled Bank account); and “Do you currently 

have any savings?” (labeled Any savings?).

Permanency

Three different measures of permanency status were investigated from three ordinal TSS item

responses. All three items had response options of: “Enough people you can count on,” “Too few people 

you can count on,” or “No one you can count on.” These items asked: “Do you have enough people to 

count on when you need someone to give you good advice about a crisis?” (labeled Support for advice 

about a crisis); “Do you have enough people to count on when you need someone to loan you money in 

an emergency?” (labeled Support in a financial emergency); and “Do you have enough people to count 

on when you need someone to give you good advice about your job or school?” (labeled Support for 

advice about job/school). 

In addition to the outcomes examined, several other measures were included in each model to control 

for observable factors that may influence the outcome. These are referred to as covariates and are 

described further in the following section.

Covariates

There were measures included in the analytic models that were developed from demographic variables 

and implementation data. First, treatment status was developed from the propensity score matching 

mentioned earlier. Age at baseline was found to be statistically significant across treatment and 

comparison groups so this was included in our models. Baseline measure of the specified outcome for 

each model. Program dosage was computed as a count of services received from the implementation 

data that indicated which services participants received from CYI (coaching, OP, support services 

funding, or leadership events). Support Services funding was computed as the total amount of support 
– 71 –

service funding received.35 Finally, Days Elapsed was a computation of the days that have elapsed from 

baseline to follow-up. 

Data Collection Activities

There were several data collection activities used to inform the CYI impact study. The core evaluation 

and implementation activities were documented using the following data collection tools:

• Common Referral Form

• Transitional Service Survey (i.e., CYI youth survey)

•

Implementation data, including:

− Support Service Fund Tracking Form

− Coaching Form

− Youth Gathering and Youth Leadership Participation Forms

− Opportunity Passport Database

The Common Referral Form was collected by CYI coordinators, Central Navigators, or other service 

providers, ideally at the time of enrollment into CYI. The Common Referral Form includes contact 

information, demographics, current needs, current supports, systems involvement, and other basic 

information. The Common Referral Form is used to help triage and commence an individualized service 

plan for young people. As noted previously, this form was the source of data for the matching 

procedures in the impact study. All participants included in the evaluation were required to have a 

Common Referral Form.

The Transitional Service Survey was administered for the purposes of the evaluation once around the 

time of enrollment,

36 and then twice each year, in April and October, for the duration of the study. All 

CYI participants were eligible to complete a survey twice per year for the duration of their enrollment in 

35 Given the considerable variation in the CYI experience among participants, treatment dosage was included to control for 

correlation between dosage and outcomes. 

36 There was considerable variation in the time when participants completed a baseline TSS and their date of enrollment. For some 

participants, the TSS baseline date preceded the date of enrollment, likely due to misreported dates on the Common Referral 

Form. For many others, the TSS followed the Common Referral Form enrollment date by a varying number of days. As described 

in the Analysis and Results section to follow, days elapsed was used a basis for creating the analytic sample.
– 72 –

the program regardless of activity. Participants received a $10 incentive for each completed survey.

Additionally, Nebraska Children introduced a raffle in 2018 that randomly selected 10 survey 

participants to receive $100 incentives. Participants were able to complete the survey either online or 

by paper. For baseline, participants were provided a link or a paper copy of the survey by their assigned 

coach or another service provider, the CYI Central Navigator, or the CYI coordinator. For follow-up 

surveys, participants were contacted by phone, email, mailing, or multiple methods. The outreach was 

completed by program staff, such as those described above, as the evaluation team did not have access 

to identifiable information. WestEd and Nebraska Children collaborated to provide technical assistance 

and training to all sites via webinars and email on effective strategies for retaining participants for 

survey participation. A survey retention tracking form was also created, but it is unclear whether sites 

used the form in their practices. 

The timing of the survey relative to the timing of the intervention was unique to each individual. For the 

first follow-up survey, the sample was limited to participants with at least 90 days elapsed from the time 

of the baseline survey to allow for some exposure to the treatment services. Some participants were 

exposed for a considerably longer period because participants were not required to complete surveys at 

each follow up in order to be retained in the analytic sample. The average days elapsed from baseline to 

the first follow up was 228.72 days. The average days elapsed were very similar for both treatment 

(229.05 days) and comparison (227.10 days) participants. The data collection process was the same for 

both the treatment and comparison participants. 

Implementation data were collected by CYI program staff at each site for any young person who 

enrolled in CYI and accessed its services. The implementation data were organized according to the core 

components of CYI: coaching, Youth Leadership, financial literacy training, and financial support. The 

quality and reliability of the data varied by component and site. The support services funds and coaching 

data were recorded for all individuals each time these services were accessed. The data for coaching 

included the number of hours coached and related domain of well-being (e.g., employment, education, 

housing). The support service funds data include the total amount of funds provided and the related 

domain of well-being. These data also included transactional information to track the providers or 

community stakeholders (e.g., auto shop, landlords) who received the funds. The data for both support 

service funds and coaching were aggregated monthly and provided to WestEd on a regular basis. There 

were cases in which coaching data were misreported or not reported for the SIF evaluation. It is not 

possible to determine the extent to which this missed reporting occurred, but WestEd and Nebraska 
– 73 –

Children took several steps to mitigate data quality concerns. Nebraska Children provided extensive 

training and technical assistance on data collection and WestEd provided targeted guidance as part of 

regular community check-in calls. In addition, to supplement the coaching data reported by sites, 

Nebraska Children retrieved additional coaching data from Central Plains Center for Services. Central 

Plains provides coaching across the state, including in several of the SIF sites. The data were used to 

determine length of coaching services and integrated into the analytic data file for young people who 

did not have a coaching record. Overall, approximately 9 percent of the sample had a PALS (i.e., Central 

Plains) coaching record but no record in the Youth Database. If the PALS record was active for greater 

than 60 days (based on treatment eligibility criteria described in Table 8), the coaching status was 

imputed to be valid for the individual. 

The Youth Leadership data were collected each time a CYI site held a gathering event for young people 

or individuals participated in a Youth Leadership activity. The data included an attendance list, date, and 

category of event. The Youth Leadership data varied considerably across sites, and, as noted previously, 

the reliability of these data were not strong relative to other implementation data, due to inconsistent 

definitions of what sites considered Youth Leadership activities as well as the detail provided in the 

youth gathering forms. The final data source was the OP database file. All CYI participants who 

participated in OP were recorded in the OP database. The data were collected by the national 

organization, Child Trends, which oversees the data collection and monitoring of OP. The OP database is 

extensive and generally of good quality, given that it is a core requirement of OP implementation sites. 

The data used for the SIF evaluation included the number of financial literacy courses completed, the 

amount of money saved as part of the program, and whether an asset was purchased as a result of OP 

participation. The data were provided to WestEd one time, in spring 2019, to capture all CYI participants

who may have been exposed to OP over the course of the evaluation period. 

The coaching, support service funds, and Youth Leadership data were reported to Nebraska Children 

monthly by sites. The data were audited by Nebraska Children staff routinely, and outliers and 

discrepancies were remedied as they were found. The most common issue was missing or inaccurate 

data, such as implementation data that preceded enrollment start date for young people. In this case,

the issue was typically due to an incorrect date entry. 
– 74 –

Changes to the Subgrantee Evaluation Plan

The data collection activities for the CYI evaluation generally align with those proposed in the SEP. The 

only notable change was in securing administrative data. Although preliminary conversations were 

initiated with several state agencies and progress was made toward data sharing agreements, ultimately 

the only data obtained for consideration in the CYI evaluation were the Homeless Management 

Information System data. However, these data were not included in any analyses because fewer than 10 

CYI participants were identified in this system. 

Analysis and Results

The quantitative data were analyzed to ascertain the impact that CYI had on the lives of participants. 

Some analyses have already been mentioned thus far, including: the propensity score analysis which 

guided the identification of the treatment and comparison groups; baseline equivalency to investigate 

the statistically significant differences that were present across groups on variables used for propensity 

score matching; and exploratory and confirmatory factor analysis which aided in the development of 

the Well-Being Scale (and subscales). 

The following sections highlight the adjusted differences as a result of respective regression models. 

Raw changes between survey periods were also explored. Raw differences in continuous variables were 

examined using two sample t-tests and categorical variables were examined using the chi-square test of 

homogeneity. Linear, logistic, and ordinary logistic regression models were used to compute adjusted 

differences in measures of interest at the first follow-up period and last follow-up across treatment and 

comparison groups. Given the volume of tables at each follow-up period, the report limits the narrative 

discussion and tables to only covariates of interest (i.e., outcomes). All models included participants who 

had at least 3 months elapse between baseline and follow up, and the models controlled for the 

specified outcome measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow up, and clustered site standard errors. Adjusted 

differences of measures of interest across treatment and comparison groups that produced p-values

below 0.1 were further explored using Full Information Maximum Likelihood (FIML) modelling.

37

37 FIML is a technique used to retain the full sample by drawing on the values available within each unit (participant) to estimate 

the population parameters using the value that maximizes the likelihood function. Thus, data are not imputed but the data in 

hand are used to establish the parameters for the full sample. The full models, including all covariates for each outcome at each 

time period, are available upon request from this report’s primary author.
– 75 –

Well-Being

The Well-Being Scale standardized scores ranged from about -2 to +1 standardized units. At baseline,

there was a statistically significant difference across study groups (diff: 0.204; p<.05). The comparison 

group (-0.20) had a lower standardized well-being score on average, compared to the treatment group

(0.01). 

The adjusted difference in well-being score between treatment and comparison groups was computed 

using a linear regression model. At follow-up 1, the treatment group had 0.045 more standardized units 

of well-being than the comparison group (p=0.613; Cohen’s D=0.127). At last follow-up, the direction of 

findings was reversed, and the comparison group had 0.029 more standardized units of well-being than 

the treatment group (p=0.403, Cohen’s D=0.073).

38 No FIML models were computed for well-being,

given its statistical nonsignificance and relatively weak statistical effect sizes. 

Table 14. Well-Being Models, by Survey Period

Housing and Financial Stability

The Housing and Financial Stability (HFS) sub-score was one of four factors of the Well-Being Scale. HFS

ranged from about -2 to +1 standardized units. At baseline, there was a statistically significant difference 

across study groups (diff: 0.198; p<.05). The comparison group (-0.21) had a lower HFS score on average,

compared to the treatment group (-0.01). 

The adjusted difference in HFS score between treatment and comparison groups was computed using a 

linear regression model. At follow-up 1, the treatment group had 0.108 more standardized units of HSF 

than the comparison group (p=0.291; Cohen’s D=0.227). At last follow-up, the treatment group had 

0.071 more standardized units of HSF than the comparison group (p=0.112, Cohen’s D=0.164). No FIML 

38 As noted later in the limitations, the sample at last follow up was considerably reduced and biased toward the treatment group; 

thus, calling into question whether it is representative of the sample examined at first follow up. 

Model n Treatment Estimate S.E. P-value Effect Size Type Effect Size

Well-Being Score at 

Follow-Up 1

256 0.045 0.065 0.613 Cohen’s D 0.127

Well-Being Score at Last 

Follow-Up

165 -0.029 0.033 0.403 Cohen’s D -0.073

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models controlled 

for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, days elapsed 

from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey period.
– 76 –

models were computed for HFS, given its statistical nonsignificance, although the effect size suggests a 

small, but non-minimal statistical effect. 

Table 15. Housing and Financial Stability Models, by Survey Period

Transportation and Employment

The Transportation and Employment (T&E) sub-score was the second of four factors of the Well-Being 

Scale. T&E ranged from about -2 to +1 standardized units. At baseline there was a non–statistically 

significant difference across study groups (diff: 0.06; p=0.478). The comparison group (-0.181) had a 

lower standardized T&E score on average, compared to the treatment group (-0.125). 

The adjusted difference in T&E score between treatment and comparison groups was computed using a 

linear regression model. At follow-up 1, the treatment group had 0.123 more standardized units of T&E 

than the comparison group (p=0.316; Cohen’s D=0.187). At last follow-up, the direction of findings was

reversed and the comparison group had 0.262 more standardized units of HSF than the treatment group 

(p=0.175, Cohen’s D=-0.565). No FIML models were computed for T&E, given its statistical 

nonsignificance and small statistical effect sizes at first follow up. 

Table 16. Transportation and Employment Models, by Survey Period

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Housing and Financial 

Stability at Follow-Up 1

255 0.108 0.063 0.291 Cohen’s D 0.227

Housing and Financial

Stability at Last Follow-Up

165 0.071 0.112 0.547 Cohen’s D 0.164

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 

for that survey period.

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Transportation and 

Employment at Follow-Up 1

256 0.123 0.079 0.316 Cohen’s D 0.187

Transportation and 

Employment at Last FollowUp

165 -0.262 0.170 0.175 Cohen’s D -0.565

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have 

p<.05 for that survey period
– 877 –

Social Support

The Social Support (SS) sub-score was the third of four factors of the Well-Being Scale. SS ranged from 

about -3.2 to +1 standardized units. At baseline there was not a statistically significant difference across 

study groups (diff: 0.03; p=0.7355). The comparison group (-0.109) had a lower standardized SS score on 

average, compared to the treatment group (-0.073). 

The adjusted difference in SS score between treatment and comparison groups was computed using a 

linear regression model. At follow-up 1, the comparison group had 0.200 more standardized units of SS 

than the treatment group (p=0.290; Cohen’s D=-0.208). At last follow-up, the direction of findings was

reversed and the treatment group had 0.227 more standardized units of SS than the comparison group 

(p=0.107, Cohen’s D=0.231). No FIML models were computed for SS, given its statistical nonsignificance 

and relatively weak statistical effect sizes.

Table 17. Social Support Models, by Survey Period

Health Access

The Health Access (HA) sub-score was the last of the four factors of the Well-Being Scale. HA ranged 

from about -3 to +1 standardized units. At baseline there was not a statistically significant difference 

across study groups (diff: 0.14; p=0.112). The comparison group (-0.156) had a lower standardized HA

score on average, compared to the treatment group (-0.017). 

The adjusted difference in HA score between treatment and comparison groups was computed using a 

linear regression model. At follow-up 1, the treatment group had 0.192 more standardized units of HA 

than the comparison group (p=0.149; Cohen’s D=-0.298). At last follow-up, the comparison group had 

0.091 more standardized units of HA than the treatment group (p=0.230, Cohen’s D=-0.154). No FIML 

models were computed for HA, given its statistical nonsignificance and despite the small statistical effect 

at first follow up. 

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Social Support at FollowUp 1

240 -0.200 0.170 0.290 Cohen’s D -0.208

Social Support at Last 

Follow-Up

157 0.227 0.120 0.107 Cohen’s D 0.231

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models controlled 

for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, days 

elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey period.
– 78 –

Table 18. Health Access Models, by Survey Period

Hope

The hope scale ranged from 6 to 48 raw score units. At baseline there was a statistically significant 

difference across study groups (diff: 2.88; p=0.03). The comparison group (34.65) had a lower hope 

score on average, compared to the treatment group (37.54). 

The adjusted difference in hope scores between treatment and comparison groups was computed using 

a linear regression model. At follow-up 1, the treatment group had 2.43 more hope score points than 

the comparison group (p=0.026; Cohen’s D=0.413). At last follow-up, the comparison group had 1.38 

more hope score points than the treatment group (p=0.190, Cohen’s D=-0.336). Since the follow-up 1 

model had a p-value less than 0.10, an FIML model was computed. At follow-up 1, the treatment group 

had 1.934 more hope score points than the comparison group (p=0.04; Cohen’s D=0.112) on average.

Table 19. Hope Models, by Survey Period

Executive Functioning

The executive functioning scale ranged from 8 to 40 raw score units. At baseline, there was a statistically 

significant difference across study groups (diff: 2.10; p=0.03). The comparison group (26.70) had a lower 

executive functioning score on average, compared to the treatment group (28.80). 

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Health Access at Follow-Up 1 254 0.192 0.152 0.149 Cohen’s D 0.298

Health Access at Last Follow-Up 164 -0.091 0.068 0.230 Cohen’s D -0.154

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 

for that survey period.

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Hope Score at Follow-Up 1 256 2.430 1.227 0.026 Cohen’s D 0.413

Hope Score at Last Follow-Up 165 -1.381 0.933 0.190 Cohen’s D -0.336

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, 

days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey 

period
– 79 –

The adjusted difference in executive functioning score between treatment and comparison groups was 

computed using a linear regression model. At follow-up 1, the treatment group had 0.44 more executive 

functioning score points than the comparison group (p=0.780; Cohen’s D=0.065). At last follow-up, the 

treatment group had 0.43 more hope score points than the comparison group (p=0.6880, Cohen’s D=-

0.067). No FIML models were computed for executive functioning, given its statistical nonsignificance 

and relatively weak statistical effect sizes.

Table 20. Executive Functioning Models, by Survey Period

Housing

At baseline there was no statistically significant differences in the proportions of participants that 

reported safe living conditions, stable living conditions, unstable living situations, and the combination 

of safe and stable living conditions across study groups.

The adjusted housing measures to explore differences across study groups were computed using logistic 

regression models. Table 21 displays follow-up 1 and last follow-up estimates for outcomes of interest 

specified in the SIF subgrantee evaluation plan. The treatment group had 3.35 times the odds of stable 

living conditions, compared to the comparison group (p<.05). The treatment group also had 3.8 times 

the odds of safe and stable living conditions, compared to the comparison group (p<.05). Though the 

differences were statistically nonsignificant, the treatment group leaned towards an increased odds of 

unstable living conditions at follow-up 1 compared to the comparison group, then more stable living 

conditions at last follow up. Additionally, at last follow up, the treatment group retained the increased 

odds of stable living and safe and stable living conditions on average (though statistically nonsignificant). 

Since the follow-up 1 models for stable living and safe and stable living had p-values less than 0.10, FIML 

models were computed. The results are consistent with the non-FIML models showing the treatment 

group had 3 times the odds of stable living conditions, compared to the comparison group (p<.001). 

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Executive Functioning Score at 

Follow-Up 1

256 0.441 1.512 0.780 Cohen’s D 0.065

Executive Functioning Score at 

Last Follow-Up

165 0.425 1.008 0.688 Cohen’s D 0.067

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 

for that survey period.
– 80 –

Also, the treatment group had 3.5 times the odds of safe and stable living conditions, compared to the 

comparison group (p<.001)

Table 21. Housing Models, by Survey Period

Employment

At baseline there was no statistically significant difference across treatment groups in the proportion of 

participants that reported stable employment and length of time with current employer. 

The adjusted employment measures to explore differences across study groups were computed using 

logistic, ordinal logistic, and linear regression models. Table 22 displays follow-up 1 and last follow-up 

estimates for outcomes of interest specified in the SIF evaluation plan. None of the measures was 

statistically significant. There were notable trends in directionality. The comparison group had slightly 

increased odds of stable employment, compared to the treatment group which increased at last followup. Average weekly working hours and hourly wage suggest virtually no difference in the employment 

experiences of the treatment and comparison groups. No FIML models were computed for employment 

measures, given the statistical nonsignificance and relatively weak statistical effect sizes.

Table 22. Employment Models, by Survey Period

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Stable Living at Follow-Up 1 254 1.209 0.243 0.000 Odds Ratio 3.351

Safe and Stable Living at Follow-Up 

1

254 1.335 0.289 0.000 Odds Ratio 3.800

Unstable living conditions at 

Follow-Up 1

253 0.226 1.094 0.836 Odds Ratio 1.253

Stable Living at Last Follow-Up 165 0.097 0.554 0.861 Odds Ratio 1.102

Safe and Stable Living at Last 

Follow-Up

165 0.794 0.716 0.268 Odds Ratio 2.212

Unstable living conditions at Last 

Follow-Up

153 -1.160 1.416 0.412 Odds Ratio 0.313

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models controlled 

for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, days elapsed 

from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey period.

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Stable Employment at Follow-Up 1 252 -0.017 0.450 0.970 Odds Ratio 0.983
– 81 –

Education

At baseline, there was a statistically significant difference across groups in the proportion of participants 

that reported being enrolled in school, with treatment participants reporting a higher rate of enrollment 

(.68) relative to the comparison group (.44). Employed or Enrolled participants showed no statistically 

significant difference across all three survey periods.

The adjusted educational measures to explore differences across study groups were computed using 

logistic regression models. Table 23 displays follow-up 1 and last follow-up estimates for outcomes of 

interest specified in the SIF evaluation plan. Again, educational enrollment resulted in statistically 

significant estimates at first and last follow-up. The comparison group had 75 percent and 81 percent

increased odds of being enrolled in some educational program, compared to the treatment group at 

first and last follow-up, respectively. Since the first and last follow-up models for educational enrollment

had p-values less than 0.10, FIML models were computed. The results are consistent with the non-FIML 

models showing the comparison group had 73 percent (p=.007) and 82 percent (p=.071) increased odds 

of being enrolled in an education program, compared to the treatment group at first and last follow-up, 

respectively. 

Table 23. Education Models, by Survey Period

Employment Length at Follow-Up 1 254 0.242 0.471 0.607 Odds Ratio 1.274

Weekly Working Hours at FollowUp 1246 -0.863 1.839 0.656 Cohen’s D -0.044

Hourly Wage at Follow-Up 1 242 0.176 0.754 0.823 Cohen’s D 0.034

Stable Employment at Last FollowUp163 -0.218 0.731 0.766 Odds Ratio 0.804

Employment Length at Last FollowUp163 -0.085 0.703 0.904 Odds Ratio 0.919

Weekly Working Hours at Last 

Follow-Up

159 0.876 6.464 0.897 Cohen’s D 0.054

Hourly Wage at Last Follow-Up 153 0.802 1.557 0.625 Cohen’s D 0.148

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models controlled 

for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, days elapsed 

from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey period

Model n

Treatment 

Estimate S.E. P-value Effect Size Type

Effect 

Size

Enrolled (in School) at Follow-Up 1 252 -1.387 0.514 0.007 Odds Ratio 0.250

Employed or Enrolled Participants

at Follow-Up 1

253 -1.183 0.882 0.180 Odds Ratio 0.306
– 82 –

Health

There were no statistical differences at baseline across treatment groups for the health-related 

outcomes of interest.

The adjusted health measures to explore differences across study groups were computed using logistic 

and ordinal logistic regression models. Table 24 displays follow-up 1 and last follow-up estimates for 

outcomes of interest specified in the SIF evaluation plan. The models for emergency room visits were 

statistically significant (p<.05). The treatment group had 57 percent and 71 percent decreased odds of 

more than one emergency room visit in the past 6 months, compared to the comparison group at first 

and last follow-up. Since the first and last follow-up models for emergency room visits had p-values less 

than 0.10, FIML models were computed. The results are consistent with the non-FIML models showing 

the treatment group had 44 percent (p=.04) and 68 percent (p=.05) decreased odds of more than one 

emergency room visit in the past 6 months, compared to the comparison group at first and last followup.

Enrolled (in School) at Last FollowUp158 -1.674 0.945 0.077 Odds Ratio 0.187

Employed or Enrolled Participants

at Last Follow-Up

162 0.027 0.852 0.974 Odds Ratio 1.028

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, 

days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey 

period.
– 83 –

Table 24. Health Models, by Survey Period

Transportation

At baseline, there were no statistically significant differences in transportation variables across 

treatment groups. 

The adjusted transportation measures to explore differences across study groups were computed using 

logistic regression models. Table 25 displays follow-up 1 and last follow-up estimates for outcomes of 

interest specified in the SIF evaluation plan. The model for transportation access to school or work was 

statistically significant (p<.05). The treatment group had 4.24 times increased odds of having 

transportation access for school or work, compared to the comparison group at first follow-up. Since the 

follow-up models for transportation access to school or work had a p-value less than 0.10, an FIML 

model was computed. The result is consistent with the non-FIML models showing the treatment group 

had 5.4 (p=.002) times increased odds of having transportation access for school or work, compared to 

the comparison group at first follow-up; however, this finding does not hold up during the last follow-up 

period.

Model n

Treatment 

Estimate S.E. Pvalue

Effect Size 

Type

Effect 

Size

Insurance at Follow-Up 1 230 0.544 0.954 0.569 Odds Ratio 0.654

Unmet Needs Right Now at Follow-Up 

1

241 -1.034 0.714 0.148 Odds Ratio 0.356

Physical Exam within the past 1–2 years

at Follow-Up 1

202 -0.144 1.043 0.890 Odds Ratio 0.866

Needed care in the past 6 months at 

Follow-Up 1

256 -0.452 0.488 0.354 Odds Ratio 0.636

E.R. Visits at Follow-Up 1 256 -0.843 0.302 0.005 Odds Ratio 0.430

Insurance at Last Follow-Up 146 -0.070 0.745 0.925 Odds Ratio 0.932

Unmet Needs Right Now at Last 

Follow-Up

157 -1.290 1.444 0.372 Odds Ratio 0.275

Physical Exam within the past 1–2 years

at Last Follow-Up

129 0.575 0.688 0.403 Odds Ratio 1.778

Needed care in the past 6 months at 

Last Follow-Up

165 0.231 1.139 0.839 Odds Ratio 1.260

E.R. Visits at Last Follow-Up 165 -1.228 0.589 0.037 Odds Ratio 0.293

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have 

p<.05 for that survey period
– 84 –

Table 25. Transportation Models by Survey Period

Finances

At baseline, two of the three financial measures were statistically significant across treatment groups. 

CYI participants were more likely to report having a savings account (0.36) and enough money to cover 

monthly expenses (0.63), relative to the comparison group (0.21 and 0.36, respectively). 

The adjusted financial measures to explore differences across study groups were computed using 

logistic regression models. Table 26 displays follow-up 1 and last follow-up estimates for outcomes of 

interest specified in the SIF evaluation plan. The model for enough money to cover expenses in the past 

month was statistically significant at first and last follow-ups. The treatment group had a 1.8 times and 

4.2 times increased odds of having enough money to cover monthly expenses in the last month, 

compared to the comparison group at first and last follow-up, respectively. The model for participants 

who had any savings was significant at follow-up 1. The treatment group had 4.0 times increased odds

of having any savings, compared to the comparison group at first follow-up. Since the follow-up models 

for enough money to cover expenses and having any savings had p-value less than 0.10, FIML models 

were computed. The results suggested no statistically difference exists for enough money to cover 

expenses between the treatment and comparison groups at first and last follow-ups (ORs: 1.4; p>.05). 

The FIML model for having any savings at follow-up 1 was consistent with the non-FIML model (OR: 2.0; 

p=.02).

Model n

Treatment 

Estimate S.E. Pvalue

Effect Size 

Type

Effect 

Size

Transportation access to job/school at 

Follow-Up 1

227 1.445 0.312 0.000 Odds Ratio 4.240

Transportation access for health at 

Follow-Up 1

226 1.299 0.861 0.132 Odds Ratio 3.666

Access to reliable transportation at 

Follow-Up 1

234 0.958 0.655 0.144 Odds Ratio 2.607

Has Driver’s License at Follow-Up 1 228 -0.466 0.699 0.505 Odds Ratio 0.628

Transportation access to job/school at 

Last Follow-Up

143 0.851 1.863 0.648 Odds Ratio 2.343

Transportation access for health at Last 

Follow-Up

145 0.694 1.322 0.600 Odds Ratio 2.002

Access to reliable transportation at Last 

Follow-Up

156 -0.581 0.718 0.419 Odds Ratio 0.559

Has Driver’s License at Last Follow-Up 150 -1.149 0.714 0.108 Odds Ratio 0.317

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at 

enrollment, days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have 

p<.05 for that survey period
– 85 –

Table 26. Financial Models by Survey Period

Permanency

There were no statistically significant differences in responses to three permanency measures across 

study groups (p<.05) at baseline. 

The adjusted permanency measures to explore differences across study groups were computed using 

logistic regression models. Table 27 displays follow-up 1 and last follow-up estimates for outcomes of 

interest specified in the SIF evaluation plan. None of the measures was statistically significant. There 

were notable trends in directionality. The comparison group had slightly increased odds of having access 

to support at first follow-up, compared with the treatment group. At last follow-up, the treatment group 

showed higher odds of having access to support. No FIML models were computed for permanency

measures, given the statistical nonsignificance and relatively weak statistical effect sizes.

Table 27. Permanency Models, by Survey Period

Model n

Treatment 

Estimate S.E. Pvalue

Effect Size 

Type

Effect 

Size

Enough Money to Cover Expenses at 

Follow-Up 1

187 0.590 0.218 0.007 Odds Ratio 1.804

Has Bank Account at Follow-Up 1 187 0.531 0.456 0.244 Odds Ratio 1.701

Any Savings? at Follow-Up 1 187 1.390 0.609 0.022 Odds Ratio 4.017

Enough Money to Cover Expenses at Last 

Follow-Up

114 1.426 0.582 0.014 Odds Ratio 4.162

Has Bank Account at Last Follow-Up 114 -0.764 0.775 0.324 Odds Ratio 0.466

Any Savings? at Last Follow-Up 114 0.981 0.796 0.218 Odds Ratio 2.666

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total support services funds received, age at enrollment, 

days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that survey 

period

Model n

Treatment 

Estimate S.E. Pvalue

Effect Size 

Type

Effect 

Size

Support for advice about a crisis at 

Follow-Up 1

237 -0.300 0.798 0.707 Odds Ratio 0.741

Support in a financial emergency at 

Follow-Up 1

210 -0.349 0.314 0.266 Odds Ratio 0.706

Support for advice about job/school at 

Follow-Up 1

232 -0.287 0.962 0.765 Odds Ratio 0.750

Support for advice about a crisis at Last 

Follow-Up

153 0.308 0.811 0.705 Odds Ratio 1.360

Support in a financial emergency at 

Last Follow-Up

137 0.438 0.377 0.245 Odds Ratio 1.550
– 86 –

The outcomes presented in the impact models are visualized in Table 28. The table visualizes statistical 

significance and effect sizes of reach outcome as square figures. As described in the legend, green 

figures represent positive effects and red figures represent negative effects. Shaded figures represent 

statistically significant findings. There are three sizes of squares to denote small, medium, and large 

effects. The effect size type notes whether effects are calculates as Cohen’s d (d) or Odds Ratio (O.R.). 

Legend for Table 28 

Positive 

Small Effect 

Not significant 

Negative 

Small Effect 

Not significant 

Positive 

Small Effect 

Significant 

Negative 

Small Effect 

Significant 

Positive 

Medium Effect 

Not significant 

Negative 

Medium Effect 

Not significant 

Positive 

Medium Effect 

Significant 

Negative 

Medium Effect 

Significant 

Positive 

Large Effect 

Not significant 

Negative 

Large Effect 

Not significant 

Positive 

Large Effect 

Significant 

Negative 

Large Effect 

Significant 

Table 28. Summary of Outcome Models 

Domain Outcome First 

Follow 

Up

Last 

Follow 

Up

Effect 

Size 

Type 

Well-being Overall Well-being d 

Support for advice about job/school at 

Last Follow-Up

151 0.858 0.708 0.225 Odds Ratio 2.359 

Notes: All models included participants who had >= 3 months elapse between baseline and follow-up 1. All models 

controlled for the specified measure at baseline, program dosage, total flex funds received, age at enrollment, 

days elapsed from baseline to follow-up 1, and clustered site standard errors. Bolded statistics have p<.05 for that 

survey period.
– 87 –

Domain Outcome First 

Follow 

Up

Last 

Follow 

Up

Effect 

Size 

Type 

Well-being Housing and Financial Stability d

Well-being Transportation and Employment d

Well-being Social Support d

Well-being Health Access d

Well-being Hope d

Well-being Executive Functioning d

Housing Stable Living O.R. 

Housing Safe and Stable Living O.R. 

Employment Stable Employment O.R. 

Employment Employment Length O.R. 

Employment Weekly Working Hours d

Employment Hourly Wage d 

Education Enrolled O.R.

Opportunity Employed OR Enrolled O.R. 

Health Insurance O.R. 

Health Unmet Needs Right Now O.R. 

Health Physical Exam within the past 1-2 

years 

O.R. 

Health Needed care in the past 6 months O.R. 

Health E.R. Visits O.R. 

Transportation Transportation access to 

job/school 

O.R. 
– 88 –

Domain Outcome First 

Follow 

Up

Last 

Follow 

Up

Effect 

Size 

Type 

Transportation Transportation access for health O.R. 

Transportation Access to reliable transportation O.R. 

Transportation Has Driver’s License O.R. 

Financial Enough Money to Cover Expenses O.R. 

Financial Has Bank Account O.R. 

Financial Any Savings O.R. 

Permanency Support for advice about a crisis O.R. 

Permanency Support in a financial emergency O.R. 

Permanency Support for advice about 

job/school 

O.R. 

Reflection on Impact Study and Findings 

The primary and exploratory outcomes described in the prior sections indicate both design and 

methodological challenges within the impact study. Most notable were the challenges with overall and 

differential attrition as well as lack of baseline equivalence for age and multiple outcomes of interest. 

However, the findings also indicate areas of promise and strength for program staff. Despite the internal 

validity issues as well as lack of change to overall well-being, there were improvements across multiple 

domains among participants. These findings, as well as lessons learned, are further discussed in the 

following sections. 

Changes to the Subgrantee Evaluation Plan 

There were changes to the analyses as previously described in the subgrantee evaluation plan. Most 

notable is the additional secondary or exploratory outcomes that were analyzed. The changes were 

generated following a series of discussions between WestEd and Nebraska Children about the WellBeing Scale and its role as the principal measure of program impact. Those discussions concluded that 

even though the Well-Being Scale it is an important contribution as a conceptual construct, there may 

be only a small number of underlying survey items that aligned to the CYI model as practiced in the field. 
– 89 –

Therefore, the potential exists for either underestimation or spurious findings from relying solely on the 

Well-Being Scale. To address this concern, WestEd constructed a number of additional analyses focused 

on individual survey items as the outcomes, and examined those outcomes for change at follow up. 
– 90 –

Conclusion

The implementation and evaluation of CYI surfaced many successes and also many challenges for the 

young people, implementation staff, and partners involved. The sections to follow summarize the key 

findings from the study as well as its limitations and lessons learned that are critical for other 

communities seeking to implement complex collaborative initiatives for unconnected young people.

Summary of Implementation Study Findings

The implementation of CYI offers key findings and lessons learned from the experience of the six 

collaboratives and the Foundation who oversaw this complex initiative. There are many anecdotes of 

success among young people who engaged with CYI, whether it be through Opportunity PassportTM, 

coaching, support services funds or other aspects of CYI. Young people were able to acquire cars 

through an asset purchase which allowed them to get a job for the first time, young people were able to 

find housing and employment in their local community, coaching services enabled young people access 

to health services and support to address their immediate (housing, health care, parenting help, job 

training) and long term challenges (reducing debt, planning for further education or training, job skills). 

CYI also provided a platform to educate the collaboratives and broader community about this 

population, an older and yet unconnected population living in these counties and in need of support. 

This resulted in new partnerships with financial institutions, employers, and others who had not typically 

been engaged with the collaboratives in the past.

The results of the outcome study are reflected in some of the implementation challenges. 

Implementation of CYI components took longer than expected in these rural communities. Related to 

this was the challenge communities faced recruiting young people into CYI. There were many factors 

that contributed to this including the rural geography of the communities and lack of transportation of 

young people to engage with CYI components such as OP or coaching. The delay in establishing CYI in 

each community also resulted in fewer participants. The early stages of the grant were largely focused 

on grant compliance and learning about CYI at the collaborative level. Once CYI components were 

implemented, recruitment improved, but this still resulted in a small sample which impacted the 

precision of the outcome study. Although the implementation study documented success stories of 
– 91 –

young people turning their lives around and the community shifting their approach and view of this 

population and forming new partnerships to better serve unconnected young people, ultimately 

additional time and sample are needed to fully understand CYI and its impact. 

The implementation study findings also suggest that there was variation in how the collaboratives 

implemented some CYI components. Youth Leadership is an example of one of the components that was 

implemented using a variety of models and approaches and this evolved over the grant period. To better 

understand how young people experienced each component of CYI, outside of Opportunity PassportTM, 

more data on fidelity, exposure, and quality of program delivery would be needed. In addition, common 

data definitions would also need to be established to ensure that each site was capturing and recording 

the same data on each component. The sites increased their capacity around data collection over the 

course of the grant, but it was not clear if all sites had a common understanding and definition of each 

data element that comprised CYI engagement. 

There were no major changes to the planned implementation evaluation outlined in the SEP. The 

evaluation approach, data collection, and analysis were consistent with the SEP with the exception of 

changes in the timing of two data collection activities, interviews with the collaboratives and site visits.

More frequent interviews were conducted using the research liaison model so that each site had a 

chance to talk monthly or bimonthly with an evaluation team member. There were two site visits 

conducted over the life of the 4-year project in lieu of annual site visits.

Summary of Impact Study Findings

The impact study conducted of the CYI initiative involved a quasi-experimental study that utilized 

propensity score matching procedures to form the study groups. This design is considered among the 

stronger quasi-experimental designs regarding internal validity if the matched sample demonstrates 

between group equivalence at baseline. The design, although short of a true experimental study, still 

should have produced strong causal evidence related to program impact. The results from the matching 

procedures described above yielded statistically equivalent groups across nearly all measures related to 

enrollment; however, age and multiple baseline measures of outcomes were not statistically equivalent 

at baseline and fell outside the acceptable range of 0.25 standard deviations. In an effort to reduce 

selection bias, all models controlled for age and the pretest on the outcome measure. Despite these 

statistical adjustments, the many implementation challenges, and specifically the diminished internal 
– 92 –

validity due to baseline equivalence and attrition bias, reduced the confidence in which causal claims 

could be asserted about the impact of CYI. These limitations among others are discussed in the Lessons 

Learned section, to follow.

Despite the challenges incurred over the course of the evaluation period, the impact study indicates that 

the implementation of CYI resulted in some positive impacts among those served through the initiative. 

The estimates among participants at the first follow up period suggest that CYI had a statistically 

significant and meaningful impact across several outcomes. Notably, perceptions by CYI participants of 

their living situation as “safe and stable” improved relative to the comparison group at the first follow 

up period. CYI participants also realized statistically significant improvement in their odds of having a 

savings account and enough money to cover monthly bills at the first follow up relative to the 

comparison group. Given the lynch pin for CYI is to create safe, stable, and supportive environments for 

young people as they transition to independent adult living, improving the financial stability and living 

environment of treatment participants is evidence of success for CYI. Another positive impact for CYI 

participants included improving their perceptions of hope relative to the comparison group at first 

follow up as well. There was also a statistically significant reduction in ER visits among for treatment 

group participants relative to those in the comparison group. Generally, in cases in which statistically 

significant impacts between groups were not observed, it was due to CYI and comparison participants 

both improving from baseline to the first follow up period. Again, these findings are met with caution 

given the threats to internal validity in the study. Most notably, the study is unable to demonstrate 

baseline equivalence between groups for age and multiple significant outcomes. 

Furthermore, as mentioned earlier, retention was a challenge, and the number of participants who 

contributed data at more than one follow up period was notably low. Because of this, the last follow up 

impact estimate should be interpreted with caution and considered exploratory in nature. The findings 

reveal, however, that most of the positive results realized during the first follow up were indeed 

sustained through the last follow up period. However, due to retention issues, statistical power was 

diminished, and external validity weakened. This resulted in substantially less ability to detect 

statistically significant differences between groups and causally attribute any difference to CYI. A 

reduction in emergency room visits and prevalence of having enough savings to cover bills were the only 

statistical improvements for CYI participants at the last follow up period. 

As described above, the quasi-experimental study was designed to maintain strong internal validity and 

was consistent with what was proposed in the SEP as it relates to the statistical matching procedures; 
– 93 –

however, the study suffered from challenges that threaten the internal validity relative to baseline

equivalence and attrition. Aside from baseline equivalence, overall and differential attrition were high 

(30 percent and 21 percent, respectively). The potential attrition bias does limit the confidence in 

making causal inferences about the impact of CYI; however, the impacts that were observed do align to 

the model’s theory of change. Further, it limits the study’s ability to generalize these findings beyond 

that of the study sample as it is not clear whether response bias is due to greater engagement or greater 

need among those who were retained at follow up. 

Overall, the study design did not indicate a statistically significant difference between groups on the 

primary outcome of interest; the Well-Being Scale. Among secondary outcomes, the study indicates 

moderate evidence of impact for outcomes related to safe and successful living, financial savings, ER 

visits, and self-reported “hope”. 

Despite the internal validity issues related to baseline equivalence and attrition bias, external SIF 

reviewers determine the evaluation met a “Moderate” evidence rating for purposes of the CNCS 

Evidence Exchange. It is important to note that some of the findings are promising and provide a 

roadmap for future implementation and research teams who seek to implement and evaluate CYI or 

related initiatives. Specifically, this evidence suggests individual items that are foundational to the 

program, rather than a more holistic constructed outcome such as the Well-Being Scale, were a more 

meaningful and measurable gauge of change. And as described below, engaging the type of 

“unconnected” young people who are eligible for a program like CYI, in both the program and an 

accompanying evaluation (e.g., completing surveys) faces significant challenges. Nevertheless, the 

impact study does yield some findings that are positive and aligned with the program’s theory of change 

and the stakeholders’ experiences on the ground. The findings also provide an understanding of the 

dimensions of well-being in which additional energy and investment may prove most fruitful. For 

example, more investment in both housing supports and establishing strong relationships with financial 

institutions to support financial goals could further increase CYI’s impact on young people. Further, the 

study surfaced the complexities of tracking and retaining potential participants for both implementation 

and evaluation. These and other lessons learned as well as limitations are discussed in the following 

section.
– 94 –

Lessons Learned

The implementation and evaluation of CYI garnered many successes, challenges, and lessons learned 

over the course of the SIF evaluation period. The following section highlights several of those key 

reflections and provides guidance in the form of next steps for CYI and other similar programs and any 

respective studies. 

The experience of CYI has had an impact on the communities where it has operated, the people 

involved, and the Foundation who oversaw this initiative. To that end, several lessons have been learned 

around grant management, capacity building, and specific project components that will apply to future 

iterations of this important model. 

Firstly, much was learned about understanding the way that rural geography impacts CYI 

implementation that were not experienced with the urban-focused Project Everlast. These lessons 

include that there are fewer people to serve in multiple roles including grant compliance, managing 

implementation, and data and evaluation tasks. The other lesson related to geography is that 

transportation can be even more of a barrier for young people in rural communities. This impacted both 

young people and CYI partners and resulted in challenges around engagement, recruitment, and 

implementation. An additional set of lessons relates to implementation, specifically balancing 

implementation and grant compliance from the beginning to ensure there are not delays or setbacks in 

either area. The other implementation lesson is that components are not always easily transferrable 

between communities and may need more support tailored within local context to be implemented with 

success. The final set of implementation lessons relates to the CYI population. For some collaboratives it 

was difficult to shift their focus to this older young person population when their original focus was on 

early childhood or other populations. The other lesson related to the CYI population is that the needs of 

young people evolve over time and programs also need to be persistent in understanding the challenges 

young people face. It may also take time to earn the trust of this older population so that they will 

engage with a program. 

The study also highlighted several other lessons that relate to the design and approach to the 

evaluation. 

First, CYI is a complex community-driven initiative that presented challenges to the design and 

implementation of a rigorous evaluation study. The nature of the initiative and the young people who 

engage with CYI are such that recruitment and participation is not easy to predict and plan for in 
– 95 –

advance. The initiative is highly individualized and tailored to the specific needs of young people who 

take up services. This leads to high variability in the level and type of engagement among CYI 

participants. Further, the implementation data revealed that even youth people in the comparison 

group were often exposed to some level of treatment. While contamination in a complex community 

initiative like CYI is likely unavoidable, stronger recruitment and documentation practices may have 

improved the evaluation team’s ability to correctly attribute exactly what treatment each person 

received. Regardless of accurate and reliable practices, the rural nature of the CYI SIF communities is 

such that recruitment was saturated within the community “hubs” but reaching young people in more 

isolated areas proved a challenge across sites. A lesson learned is that more thorough vetting of sample 

estimates, beyond a commitment by sites to reach target numbers, would have established more 

realistic estimations for the study. 

Evaluation activities also required considerable time investment for program staff in each of the six 

sites. The primary burden included data collection and entry for enrollment and implementation data, 

and retention during semi-annual survey periods. Nebraska Children and WestEd attempted to be 

responsive to these burdens by providing additional technical assistance, training, and materials to assist 

with data collection. Nevertheless, the time burden persisted and there was notable variability in the 

quality of data across sites as well as the sites’ ability to retain study participants at follow up survey 

periods. A lesson learned is that alternative approaches that transfer burden from program sites to the 

evaluation team may need to be considered. More robust online systems may have improved the 

timeliness, quality and, consistency of implementation data. Further, WestEd, if granted access to 

participant identifiers, may have served a more active role in the retention process drawing on 

established and practiced guidelines for survey retention (e.g., contacting participants at various time 

intervals about their survey response). While it would have come at a cost to the evaluation budget, it 

may have reduced the overall and differential attrition for the study. 

The complex and individualized nature of CYI led to an impact study built around a more holistic 

measure of well-being. The intention of the Well-Being Scale was to allow for and incorporate how 

multiple implementation experiences might lead to varied outcomes. The measurement analysis process 

yielded latent constructs that suggested overlap between the individual measures of well-being. While 

conceptually important, it became evident that CYI may primarily address only a select number of

outcomes across participants, such as connecting young people with housing support and resources and

providing financial training to improve financial independence. While these are included items within 
– 96 –

the Well-Being Scale, these effects are watered down by other outcomes captured by the Scale that may 

be less likely to be realized through CYI. In other words, CYI could have impact on one or more measures 

of well-being if measured separately, but it may show no difference on a large aggregate measure such

as the Well-Being Scale. To be responsive to this concern, the study was adjusted to include several 

additional exploratory outcomes. A lesson learned is that the evaluation should work closely with 

program providers early on and throughout the implementation period to align and refine outcomes. 

These outcomes should not just be based on the proposed logic model but should be influenced by how 

implementation played out in practice. Further, items should be considered that provide a more 

nuanced and gradient understanding of change. 

Finally, we also learned that while the CYI SIF evaluation was not without challenges, the evaluation 

partnership was co-developed and co-owned by WestEd and Nebraska Children, and this made for a 

meaningful and supportive experience for sites. Given the rural nature of the intervention, and the 

geographic spread and size of Nebraska, regular in person communication with each of the sites was not 

possible. Instead, WestEd met virtually with site staff on a regular basis, incorporated several 

opportunities for group and individualized technical assistance, and conducted site visits to document 

and authenticate the site-level experiences and role within the evaluation. The collaboration between 

WestEd, Nebraska Children, and the local sites led to an evaluation that was responsive and collected 

deep contextual data to better inform the outcomes observed in the impact study. A lesson learned is 

that prioritizing the collaborative spirit of the evaluation process sustained motivation and engagement 

across partners despite the challenges carrying out the study. This and the other lessons learned 

described above are transformed into specific recommendations below. 
– 97 –

Recommendations

Domain Recommendations

Capacity, Readiness, 

and Community 

Context

• Assess capacity and commitment of partners to ensure continuous

collaboration and implementation

• Identify gaps in capacity among partners and provide target trainings to

increase the collaboratives’ capacity in those areas

• Increase collaboratives’ capacity to use data to answer implementation

questions and promote continuous quality improvement. Also use data to

confirm or negate anecdotal evidence so that adjustments can be made

based on the experience of all participants.

• Include young people, if not already included, in the collaboratives and in

leadership roles within CYI

Implementation of 

CYI

• Operationalize each component of CYI, develop a separate theory of

change or logic model for the component and what the expected

outcome will be. Include in this process definitions of all activities, events,

and categories of service to ensure consistency of implementation over

time and across sites.

• Consider how some components may be adapted to overcome the

barriers of rural communities. For example, OP was originally a designed

as a group intervention but was changed to be more of individual model

to accommodate these challenges. What other adaptations should be

made to CYI components to reach all eligible young people in rural and

often geographically distant communities?

• How can current CYI participants inform and contribute to the

recruitment, retention, and development of CYI (beyond participating in

the collaboratives)?

• Specific to Youth Leadership, define the types of events, activities, and

one-on-one meetings that are considered Youth Leadership. How does

coaching fit into Youth Leadership? Consider the database and tracking

implications when making these decisions so that Youth Leadership can

be more clearly explained and modeled for other rural communities.

• Specific to OP, how can the scope and reach of OP expand to include

more young people?

Sustainability • Communication and partnerships were considered a successful part of

what communities want to sustain. How can these relationships

continue? Should there be facilitated planning sessions for communities

after the grant ends? For example, OP was often referred to by

interviewees that it was the component with the greatest impact, but it

also reached the fewest young people.

• Are there components of CYI that are more important to sustain than

others? When communities were asked this question (importance,

easiest, hardest), there was not a clear pattern.

• The collaborative successes, from interviewees, gave examples of shifts in

mindsets, and partnering with nontraditional entities (banks, employers).

How can the scope and scale of these partnerships be sustained over

time?
– 98 –

Domain Recommendations 

Recruitment & 

Retention 

• Provide sites with technology to facilitate mobile recruitment and

program engagement. Video chat and mobile databases, for example,

may extend program reach and overall sample.

• Transfer survey data collection to dedicated personnel with experience in

community-based survey data collection. Dedicated staff, in coordination

with sites, may increase overall ability to contact, log communication, and

provide thorough follow up for nonrespondents.

• Continue to identify additional incentives for young people to participate

in follow up surveys.

• Conduct more frequent retention calls during in between survey

administrations to maintain regular contact.

Data Collection & 

Measurement 

• Create online tools to support ongoing implementation data collection

that are mobile friendly and available for use offline.

• Refine the data collection instruments to ensure measures are responsive

to rapid change for young people in a transitional period. Further explore

ways to operationalize constructs of well-being in a way that is reflective

of the CYI population and the touch points of the program.

Next Steps 

CYI will carry on with its implementation over the next several months until the end of the contract 

period of March 31, 2020. During this time, Nebraska Children will work with each site and its CYI 

workgroup to define what CYI will look like moving forward. It will vary by site depending on the specific 

needs, focus, and additional resources to support young people who might be served through CYI. 

Nebraska Children is actively reflecting on the CYI experience, including the SIF evaluation, and how the 

initiative fits within its broader efforts to engage and support older youth populations across the state. 
– 99 –

Other Aspects of Study Logistics 

and Feasibility

The implementation and evaluation of the Connected Youth Initiative (CYI) was not without challenges 

related to the logistics of the study; many of these are discussed above. The following section highlights

specific challenges related to the administrative logistics of running the project as well as how the 

programmatic and evaluation teams sought to overcome them. 

Human Subjects Protection

Oversight of the research pertaining to the protection of human subjects as part of CYI was provided by 

WestEd’s Institutional Review Board (IRB) housed within the agency’s Office of Research Integrity. The 

initial study design called for an active consent process for young people of age and a parental consent 

with an active assent process for minors. Early in the evaluation period it was determined that the 

approved consent process was a barrier to recruitment, data collection, and data sharing for the 

purposes of the SIF study. The enrollment process often took place at unpredictable times and places 

due to the nature of how young people came into contact with CYI stakeholders. This led to enrollment 

taking place without the appropriate consent materials in hand. Additionally, there were several 

instances in which seeking parental consent would potentially put the young people at undue risk due to 

toxic relationships with family members. In other instances, underage youth were entirely estranged 

from parents or other guardians, and so consent was not possible. Through careful guidance and 

documentation, WestEd and Nebraska Children developed and received approval for a waiver of 

consent process that applied prospectively and retrospectively in select circumstances. While 

recruitment and attrition remained a challenge in the study, the adjustment to implement a waiver of 

consent process did improve the team’s ability to maximize who could be considered for the study. 

Timeline

The CYI implementation and evaluation timelines were impacted in multiple ways. The implementation 

timeline (see Figure 4, p.35) varied considerably by site. It took 19 months for all core components of CYI 
– 100 –

to be implemented across all six sites. The delayed and varied implementation timeline impacted sites’

abilities to both recruit enough participants and offer a full array of services, which may have, in turn, 

affected participants’ outcomes. Further, the evaluation period was cut short by more than a year due 

to program cuts within the Corporation for National and Community Service and the Social Innovation 

Fund. The evaluation was originally planned to be carried out through July 2020; however, data 

collection was truncated with the last survey collected in April 2019. This led to a shorter recruitment 

and follow up data collection period; this compounded challenges with recruitment leading to a smaller 

sample size than originally planned. The loss of federal funds did impact the overall study budget; 

however, Nebraska Children supplemented the considerably reduced evaluation budget to maintain the 

described study design for the duration of the reported evaluation period. 

Evaluator/Subgrantee Roles and Involvement

There were no major changes to the evaluator proposed and selected for the CYI SIF Evaluation. WestEd 

was the primary external evaluation partner and received support from the Nebraska Center for Justice 

Research as well as their subcontracted partner, Category One Consulting from June 2016 through 

November 2017. There were changes with subgrantee sites. Most notable were the changes to the 

organizations and/or stakeholders that comprise the community collaboratives overseeing CYI. Over the 

course of the evaluation period, multiple sites saw turnover in program staff, collaborative leadership, 

or among the organizations responsible for the administration of the subaward (i.e., the backbone 

agency). These changes impacted sites in different ways, and this was most visible in the challenged 

sites that were balancing programming implementation with grant administration as discussed in the 

earlier Implementation Evaluation section of this report.