Formative Evaluation of Family Finding 

FINAL REPORT 

August, 2016 

Karin Malm, MS, Senior Program Area Director 

Sarah Catherine Williams, MSW, Research Scientist 

Kristina Rosinsky, MPP, Senior Research Analyst
Table of Contents Page 

i 

1 

Executive Summary 

I. Introduction 

A. Overview of previous evaluation findings 1 

B. Description of services 2 

C. Evaluation design 3

II. Program Implementation 6 

A. Description of the model 6 

B. Referral process 9 

C. Description of children served 12 

D. Program fidelity 16 

E. Fidelity by model component 17 

F. Timing and fidelity 22 

G. Implementation drivers 23

III. Program Outcomes 25 

A. Permanency and placement outcomes 25 

B. Child-related outcomes 31 

C. Fidelity and outcomes 32 

D. Summary of findings 36 

IV. Discussion and Recommendations 38 

A. Referrals 38 

B. Implementation supports 39

C. Program fidelity 40 

D. Outcomes 41

E. Framework for rigorous evaluation 41 

References 

Appendices 

Appendix A: Data Sources 

Appendix B: Fidelity of Implementation 

Appendix C: Method for Calculating Component Fidelity 

Appendix D: Discharge Reasons 

Appendix E: Associations between Programs and Outcomes after Accounting for 

Time/Case Status and Child/Case Characteristics 

Appendix F: Bivariate Fidelity Analyses 

Appendix G: Multivariate Fidelity Analyses 

Appendix H: Variable Definitions 

Appendix I: Permanency Readiness Measure 

Appendix J: Child and Adolescent Needs and Strengths Assessment
i 

Executive Summary 

This report presents findings from an evaluation of enhancements made by the Children’s Home Society 

of North Carolina (CHS) to Family Finding (FF), a relative search and engagement model, designed to 

identify and engage family members to provide support and permanent homes for children in foster 

care. Prior to CHS’s work to enhance the model and expand services across the state, Child Trends 

conducted a rigorous impact and comprehensive implementation evaluation of FF services in nine North 

Carolina counties. That evaluation yielded some evidence consistent with practitioners’ and program 

developers’ expectations about how FF works, but also some evidence to the contrary. FF specialists did 

succeed in identifying and engaging relatives and other kin; however children who received FF services 

were no more likely than children assigned to the control group to experience positive placement 

outcomes. 

In addition, several evaluations of FF (including the previous one in North Carolina) found full 

implementation of the model to be challenging for provider agencies. While the program developer’s 

description of the model includes six components or steps,1 evaluations found only the first four 

components were operational and fully implemented. Unfortunately, the final two components—

evaluation and follow-up supports—which relied on collaboration between the worker conducting FF

services and the case-carrying social worker, were never clearly articulated during trainings and thus, 

unable to be fully implemented. These challenges suggest the need for more research to determine 

whether and how model fidelity can be attained, and whether consistent implementation with fidelity 

would result in positive outcomes. It is on this foundation that CHS developed an enhanced FF model, 

and together with Child Trends, developed measures of fidelity, and collaborated on this evaluation. 

Description of services 

In response to the earlier findings, CHS revised the six-step FF model by offering three increasingly 

intensive tiers of services (See Figure). Each tier is targeted to specific populations, to ensure that 

children receive the specific array of services most likely to 

most efficient manner possible. Tier 1 is a diligent search 

service designed only to notify relatives of children new to 

out-of-home care. Tier 2 is similar in scope to the original 

FF model’s first four components (see figure), while Tier 3 

includes the first four steps and two new steps developed 

by CHS, designed to engage relatives and fictive kin who 

can provide legal permanence to youth through adoption, 

guardianship, or transfer of custody. In addition to the FF 

services, CHS offered their child-specific adoption 

recruitment (CSR) services to counties, based on the 

Wendy’s Wonderful Kids model. 

support their permanency outcomes in the 

 

1

 The six steps in the model include discovery, engagement, planning, decision-making, evaluation, and follow-up 

supports (National Institute for Permanent Family Connectedness, n.d.).
ii

Evaluation design

Given the substantial revisions to the model, and as a precursor to a second rigorous evaluation, Child 

Trends and CHS (together with funding partners) proposed a formative evaluation to assess the overall 

referral process (of children to FF) and the degree to which the different tiers of FF were implemented 

with fidelity. As part of the evaluation Child Trends would also identify program challenges and new 

ways to overcome the challenges. In addition, we tracked outcomes for the children served to compare 

them with outcomes experienced by children served during the previous evaluation, and to inform the 

design of a future impact evaluation. Outcomes examined included: achievement of legal permanency; 

experiencing a placement step down; experiencing a move from a non-relative to a relative home; 

obtaining commitments to legal or relational permanency; the number of family connections 

discovered; a child’s knowledge of their family history; a child’s overall permanency readiness; and 

general child well-being. 

Children served 

Between January 1, 2014 and February 19, 2016, 375 children received FF services and 113 children 

received CSR services. As expected, some counties referred more cases than others. Early on, CHS 

struggled to meet the referral goals in each county and DSS staff reported several reasons for not 

referring children, including not believing there was a need for the services and few children meeting 

the eligibility criteria. CHS began intensive marketing and outreach efforts in July 2014, which were 

successful in increasing the number of referrals over time. By the end of the evaluation period, CHS was 

receiving appropriate referrals on a consistent basis and had developed a waiting list. 

As expected, there were differences in the characteristics between children served by each program. 

Children served by Tier 3 and CSR were older, more likely to have impairments, have been in care 

longer, and had more prior placements than children served by Tier 1 and Tier 2, characteristics that 

may make them less likely to experience positive outcomes. Further, children served during the previous

evaluation period have characteristics that may make them less likely than children served by Tier 3 and 

CSR to experience positive outcomes. 

Key findings 

Our examination of program implementation and fidelity and child outcomes found: 

 CHS implemented the new tiered FF program in a purposeful and supportive manner. CHS leaders 

supported the program by providing guidance and providing comprehensive training to FF staff. 

Department of Social Service (DSS) administrators and staff genuinely appreciated the trainings 

provided by CHS and requested additional trainings due to high turnover. In addition, CHS’s 

reputation for providing quality services and the high degree of trust between DSS and CHS 

leadership provided a foundation for strong collaboration. 

 Earlier model components (i.e., those implemented previously under the original FF model) were 

implemented with overall greater fidelity than the later components (i.e., the newly added FF 

activities, such as child and family preparation). This finding is not surprising given that many of the 

FF specialists and supervisors had several years’ experience conducting the former model of FF. 
iii

However, FF specialists were able to implement the newer components of the model for some 

cases, demonstrating it is possible to implement all components of the model. 

 Overall, compared with children served during the previous evaluation, children served by CHS 

during this study period (receiving Tiers 2 and 3) experienced better permanency and placement 

outcomes. However, the findings should be interpreted with considerable caution. While we 

controlled for child and case characteristics to the extent possible, there could be a number of 

untested reasons for the slightly better outcomes experienced by children during this study. 

 In general, when FF services were implemented with greater fidelity, children experienced better 

outcomes. This finding, too, should be interpreted with caution as it represents an association 

between fidelity and outcomes, not a causal relationship. 

Overall, the findings suggest CHS’ efforts to modify and enhance the original FF model and expand FF

services across the state were successful, and thus likely warrant further fidelity testing and additional, 

more rigorous evaluation. The methods developed to measure fidelity, and the continuous assessment 

of findings, provide a strong foundation upon which CHS can continue to improve the model and further

expand its services. The detailed information collected as a part of the evaluation on the types of 

children served by each tier of FF and CSR provides CHS with important data for modifying their 

outreach efforts. This information is also important as it tells us that additional, more targeted outreach 

efforts may be necessary in order to obtain an adequate sample for a rigorous impact study. 

Recommendations 

It is important to remember that implementation research finds it takes programs two to four years to 

reach full model fidelity. While CHS has implemented some form of FF for more than five years, their 

experience with the enhanced model spans just two years. Not only did CHS modify and expand the 

model activities, they greatly expanded the size of their staff and service area. Prior to undergoing a 

subsequent rigorous evaluation, it would benefit CHS (and the field) to be assured their FF practice is 

being implemented with full fidelity. We therefore offer the following recommendations prior to 

conducting a second rigorous evaluation: 

 Increase support among stakeholders including continued training for DSS workers and other 

stakeholders, and consideration of co-location of FF specialists in DSS offices. Target trainings to all 

levels of staff to ensure widespread buy-in and have FF specialists discuss their role and goals with 

DSS workers.

 More training for FF specialists so fidelity of the newer model stages—child and family preparation 

and post-decision making meeting—can improve. 

 Continue to monitor fidelity and outcomes to determine whether fidelity can be improved and/or 

sustained. 

 Expand efforts to strategically assign cases to maintain manageable caseloads including staggering 

referrals to FF specialists and assessing the difficulty of each case. 

 Conduct additional small-scale analyses prior to another rigorous evaluation including testing more 

rigorous fidelity measures, and understanding how the services translate into outcomes. 
1 

I. Introduction 

The current study examines the enhancement and expansion of the Family Finding (FF) services 

provided by the Children’s Home Society of North Carolina (CHS). This section presents a brief overview 

of findings from previous evaluations of FF, a description of North Carolina’s Permanency Innovation 

Initiative, and the design for this evaluation. 

A. Overview of previous evaluation findings 

Prior to CHS’ expansion of their FF services across the state, Child Trends conducted a rigorous impact 

evaluation of the services in nine North Carolina counties. The previous evaluation yielded some 

evidence consistent with practitioners’ and program developers’ expectations about how FF works, but 

also some evidence to the contrary. For the most part, the intervention served the intended population: 

in general, the study population was disconnected from their family members, though perhaps to a 

lesser degree than agency staff presumed. Family Finding specialists (FF specialists) did succeed in 

identifying and engaging relatives and other kin; however, children who received FF services were no 

more likely than were control group children to experience a step down in their placement during the 

study period. In addition, no impacts were found on child welfare permanency and safety outcomes 

examined. We did find potential positive impacts on contact with relatives. In addition, the program may 

have improved safety outcomes and placement stability in a subgroup of counties. It is important to 

remember that due to the huge array of factors affecting human behavior, achieving sizeable impacts 

through social service interventions such as FF is difficult and rare. 

As has been the case in other sites, full implementation of the FF model as originally developed faced 

challenges. Following the completion of the North Carolina evaluation and the hosting of a Family 

Finding Forum,2 Child Trends reviewed results from 13 recent evaluations (including North Carolina) 

completed at the same time or shortly after the North Carolina evaluation. Overall, the evidence 

available was not sufficient to conclude that FF improves youth outcomes above and beyond existing, 

traditional services. At the same time, the evidence was not sufficient to conclude that FF does not 

improve outcomes. We identified three hypotheses regarding the lack of consistently positive impacts: 

1) FF may not have been completely and consistently implemented, 2) study parameters may not have 

been sufficient to detect impacts, and 3) assumptions regarding how intervention activities and outputs 

will result in outcomes may be flawed. 

While the program developer’s description of the model includes six components or steps,3

 evaluations 

found only the first four components were operational and fully implemented. In sites with a specialized 

worker design (i.e., FF coordinator), the last two components—evaluation and follow-up supports—rely 

heavily on collaboration between the specialized worker and the case-carrying social worker. After 

family meetings are conducted and plans are developed, responsibility for maintaining the child-family 

 

2

 In February 2014, with funding from The Duke Endowment, Child Trends hosted a “Family Finding Forum” during which 

25 participants discussed the recent findings from evaluations and identified lessons learned from across the studies. 

3

 The six steps in the model include discovery, engagement, planning, decision-making, evaluation, and follow-up 

supports (National Institute for Permanent Family Connectedness, n.d.).
2 

connections that were initiated and moving ahead with permanency plans shifts to the child’s social 

worker. In other words, the FF worker is responsible for the first four components, and the child’s social 

worker is responsible for the last two components. But the model’s last two components were never 

clearly articulated during trainings, so neither the FF staff nor the social workers who were expected to 

follow through on these activities understood well how these components were to be implemented. 

These challenges suggested more research was needed to determine whether and how model fidelity 

can be attained, and whether consistent implementation with fidelity would result in positive outcomes. 

It is on this foundation that CHS developed an enhanced model and measures of fidelity, and 

collaborated with Child Trends on this formative evaluation. 

B. Description of services 

In 2013, North Carolina’s legislature authorized funding for the Permanency Innovation Initiative (PII) 

Fund, which reimburses CHS for three strategies—Family Finding (following an enhanced model as 

described below), Child Specific Adoption Recruitment Services (following the Wendy’s Wonderful Kids 

Model), and Permanency Training Services.4 The funding allowed for the expansion of services so that 

by July 1, 2016, services would be available to all counties in the state. 

In response to the earlier evaluation’s findings, CHS revised the six-step FF model by offering three 

increasingly intensive tiers of services: Tier 1 includes the first two components of the original model, 

Tier 2 includes the first four components, and Tier 3 includes the full array of services while substantially 

enhancing the final two steps. While CHS drew inspiration from the original model, the new components 

differ in scope and intent, and include additional activities that: prepare the youth and family for 

permanency; finalize a permanent placement through adoption, or transfer of guardianship or custody; 

and provide ongoing monitoring and support. The different tiers are targeted to specific populations of 

children, to ensure that the children receive the specific array of services most likely to support 

permanency outcomes in the most efficient manner possible. In summary, CHS’s new model, the subject 

of this evaluation, includes: 

 Tier 1: A 1- to 2-month diligent search service designed to notify relatives of children new to 

out-of-home care including the first two components of the original FF model; 

 Tier 2: A 3- to 4-month family engagement service designed to build a social support system for 

children and youth in out-of-home care, including the first four components of the original FF

model; and 

 Tier 3: A 12-month permanency service, including the first four components of the original 

model and two new components developed by CHS, designed to engage relatives and kin who 

can provide legal permanence to youth through adoption, guardianship, or transfer of custody. 

In addition to the enhancement of the model’s steps, or components, and more directed targeting of 

the services, CHS planned to eventually expand their services across the state to all 100 counties. As part 

 

4

The training services were delivered by CHS to assess the readiness of county departments of social services to

implement the permanency strategies. 
3 

of the model enhancements, CHS, in collaboration with Child Trends, developed a fidelity tool to assess 

the program’s adherence to the model. The tool defines the critical program components and can be

used to assess the progress of implementation across counties, cases, and individual staff. The tool was 

not designed as a performance management scale, though the information gathered can inform plans 

for supervision and professional development. The fidelity tool identifies strengths and challenges with 

implementation in order to help refine the program model and program activities. In addition to the 

fidelity tools, CHS modified their case management database, the ECHO system, to include FF elements 

and the data on fidelity, and administered surveys to assess the connectedness, well-being, and 

permanency readiness of youth.5 

C. Evaluation design

Given the substantial revisions to the model, and as a precursor to a second rigorous evaluation, Child 

Trends and CHS (together with funding partners) proposed a formative evaluation. The evaluation was 

designed to assess the overall referral process to FF, and the degree to which the different tiers of FF were 

implemented with fidelity (with particular emphasis on fully-enhanced Tier 3 services model), identify 

challenges that would call for further modifications, and identify new ways to overcome challenges. In 

addition, to understand whether and how Tier 3 affects children’s outcomes, we assessed the degree to 

which individual model components were being implemented with fidelity. This will inform future 

findings regarding the intervention’s effectiveness. 

While a formative evaluation was essential, some type of outcome tracking was also necessary to inform 

future research and program development. Examining child permanency outcomes for the Tier 3 group 

will increase the accuracy of any power analyses conducted to inform the design of a second rigorous 

evaluation. Additionally, if only a small number of children receiving Tier 3 services achieve permanency, 

CHS may need to consider alternative approaches for enhancing permanency. 

Research questions. The following research questions guided the study:

 To what degree did CHS implement each component of the full, CHS Tier 3 service model? What 

were the challenges to and facilitators of implementation? 

 How do the Tier 2 and Tier 3 services compare, and, were there variations in implementation 

fidelity and outcomes experienced? 

 To what extent were the children receiving each of the tiers of service, and Child Specific 

Adoption Recruitment (CSR) appropriate for those services? 

In addition, we explored the following research question: 

 Did children served by the FF tiers and CSR reach permanency? What types of permanency (i.e., 

adoption, guardianship) were most frequent? Do child characteristics mediate associations of 

service type with outcomes? 

 

5 Unfortunately, few surveys of youth to assess “connectedness” were completed; we are unable to include this in our 

evaluation. 
4

Data collection and analysis. To answer these research questions, we tracked fidelity to the FF model, 

examined program operations and context, and tracked permanency and well-being outcomes for 

children served by CHS through all three tiers and CSR.

 Program fidelity. Several types of data informed our examination of the model. Child Trends 

conducted interviews and focus groups with CHS staff to gain in-depth understanding of how 

each component of the model was implemented, as well as a description of the training and

coaching provided. In addition to the qualitative field work, Child Trends worked with CHS to 

monitor use of the fidelity tool and to track fidelity results. Child Trends also worked closely with

CHS on modifications to the ECHO system to ensure that all six components of the Tier 3 model 

were operationalized to ensure regular collection of case-level data. Child Trends obtained 

periodic ECHO data extracts and examined the extent to which children and families served were

receiving each model component.

 Program operations and context. We conducted interviews and focus groups with staff from 

county DSS offices in order to understand how the services are implemented within the broader

context of local child welfare systems. We travelled to each of the three regions and used a mix 

of in-person and telephone interviews and focus groups to collect information about local

programmatic context and the challenges and facilitators specific to each community. We also 

used quantitative data from ECHO and the state administrative data system to inform program

operations and context and to compare the demographic and case history characteristics of 

youth receiving different tiers of service. See Appendix A for information about the source of 

each measure.

 Outcomes. We tracked outcomes in two areas—permanency/placement and child wellbeing—using data from the ECHO and state administrative data systems. We assessed 

outcomes separately for each of the service populations (see Appendix A). Some outcomes 

were measured through surveys administered by CHS to capture youth well-being and level of 

permanency readiness (e.g., Child and Adolescents Needs and Strengths Assessment and the 

Permanency Readiness Measure). We conducted descriptive analyses summarizing the 

percentage of children experiencing positive permanency and well-being outcomes. Given the 

evaluation design, we did not have pre-determined hypotheses about whether one population

subgroup would experience better outcomes than another subgroup, other than what we could 

expect due to differences in eligibility criteria.

6

For instance, differences in outcomes observed

across populations may be due to systematic differences in the populations served—not just in 

terms of case goals, but in other case history and demographic characteristics—rather than to

differences in the effectiveness of the interventions (or lack of effectiveness). 

 

6 We did, however, expect to see some differences in the types of permanency outcomes achieved across subgroups 

because having had parental rights terminated is a prerequisite for referral to CSR services. Therefore, we expected to 

see a higher share of children receiving this service adopted than among the subgroup of youth not referred for CSR

services, as many of them likely had goals other than adoption.
5

We expected to see variation in outcomes achieved among children that receive Tier 3 services. This

variation may be due not only to variation in the case history and demographic characteristics within

the Tier 3 group, but also to variation in the implementation of each Tier 3 component. Therefore, in

addition to reporting on the demographic and case history characteristics of the Tier 3 group, we also

reported on the variation with which each component was implemented across Tier 3 children.

There are limitations regarding what can be learned from the outcome tracking. Because of the lack of 

comparison groups, we have limited capacity to make inferences about whether the intervention directly

caused any observed outcomes. Eligible children in participating counties were referred to specific 

service types (i.e., Tier 1, Tier 2, etc.) based on their needs, so findings should be interpreted with 

caution.
6 

II. Program Implementation 

In this section we describe the enhanced array of services, the referral process, and the children served, 

including whether children referred to and served by FF were similar to children targeted by CHS. We 

also discuss the degree to which the services were delivered as intended (i.e., fidelity of 

implementation). 

A. Description of the model 

In the following section, we describe each component of the enhanced FF model. As shown in the figure 

below, Tier 1 services comprise the discovery stage, Tier 2 services comprise the discovery through 

planning stages, and Tier 3 services comprise the full model. 

Discovery. The goal of discovery is to identify as many

family members and other adults connected to the 

child as possible. FF specialists review the child’s case

file; interview the child (if appropriate) in addition to 

family members and other supportive adults; and 

conduct searches via Internet websites (e.g., 

Ancestry.com or Facebook) or request a search 

through Seneca.7 Successful discovery is achieved 

when the FF specialist has identified at least 40 

connections or they determine that have uncovered 

extensive knowledge about the family. Another 

important discovery activity is mobility mapping (a 

visual representation of the child’s life), which helps t

well as who they would or would not like to reconnec

he child identify who/what family is to them as 

t with. 

Engagement. The goal of engagement is to enlist the support of as many family members and 

connections interested and willing to be a support to the child as possible. FF specialists use 

individualized engagement strategies (e.g., in-person interviews, phone conversations, and written 

letters and e-mails) with the intent of identifying a group of family connections who are appropriate and 

willing to participate in a meeting to plan how to keep the child safely connected to loving family 

members. This is also the point at which Stories from the Heart videos are filmed, when appropriate.8

The videos provide children the opportunity to share their thoughts and feelings about what they want 

for themselves, as well as what they need from their family. While often shared during family meetings 

in the later stages of the model, the videos are also useful in individual engagement attempts. During 

 

7

FF specialists send a search request to Seneca Family of Agencies. Seneca has a designated staff person experienced in 

intensive relative searches. After a request is made and search completed, Seneca returns a list of potential connections 

and contact information. 

8

 Stories from the Heart videos are not filmed with every child; some children do not want to be filmed and some DSS 

agencies will not allow them to be filmed. 
7 

engagement, FF specialists begin preparing family connections to assist the social worker with planning 

and decision-making for the child’s relational and legal permanence. 

Assessments. FF specialists are expected to complete a series of assessments to inform their work with 

the child and provide data for the evaluation. Within the first few days of a referral, the child’s social 

worker completes an initial assessment providing information on the child’s background, case goals, 

family members, and medical information. Next, after the initial visit with the child, the FF specialist 

completes the Child and Adolescent Needs and Strengths Assessment (CANS); this is done again at case 

closure. The CANS is an observational tool that identifies needs and strengths to consider when working 

with a child. If the child has a relative who commits to permanency through the FF process, the FF 

specialist completes a separate section of the CANS that identifies the caregivers’ needs and strengths. 

The FF specialists also complete the Permanency Readiness Measure on a quarterly basis, which 

measures the child’s preparedness for permanency. 

Child and family preparation. One of CHS’s enhancements to the model is the implementation of a 

variety of activities undertaken to prepare the child and family for permanency. The preparation 

activities are either child-focused or family-focused. Child-focused activities help the child deal with 

his/her own emotional issues, whereas the family-focused activities help prepare families to be a 

support to the child. 

Child preparation activities occur on at least a monthly basis throughout the FF case. In addition to 

explaining the FF process to the children, FF specialists help children overcome any emotional barriers 

they may have that could impede them from achieving permanency. Several FF specialists used activities 

from the 3-5-7 Model©,9 as well as self-developed hands-on activities to help children deal with the 

grief and loss associated with being removed from their families of origin. For example, one FF specialist 

had a deck of index cards with different questions (e.g., tell me about a happy memory from your 

childhood) for the child to answer. For older youth, one FF specialist helped the youth create a 

soundtrack of their life with songs meaningful to the youth. Other preparation strategies included 

creating Lifebooks with the children,10 doing arts and crafts projects, playing basketball or throwing a 

baseball—anything to get the children comfortable enough to open up to the FF specialist about their 

feelings. 

FF specialists also complete preparation activities with family members who are interested in playing a 

role in the child’s life (either as an emotional support or a placement resource). These activities can 

occur at any time during the FF case, but for the most part take place after the decision-making 

meetings (described in further detail below). Some activities focus on helping family members and 

children develop a healthy relationship (e.g., regular visitation, Skype calls for families who live out of 

state), where others focus on readying families to make a full commitment to the child (e.g., 

 

9

 The 3-5-7 Model © is used by caseworkers to help children reconcile losses, rebuild relationships, and enhance 

belonging to a family (Henry, 2007). The FF specialists did not implement the 3-5-7 Model in its entirety or with fidelity, 

but used it as a guide in their work with children. 

10 Lifebooks are tools to help children document their memories prior to and during foster care (Child Welfare 

Information Gateway, n.d.). 
8

understanding what will happen once a child is placed in their home, offering financial assistance). This 

often includes individualized training—unique in the child welfare field—tailored to each family’s needs, 

and covers topics such as behavior management or understanding the impact of trauma. Certified 

parent educators from CHS also offer other standardized or evidence-based trainings, such as MAPP, 

Deciding Together, and Triple P (Positive Parenting Program).

Planning. The goal of this model component is to plan for the successful future of the child via “blended 

perspectives” meetings with parents, family members, and others important to the child. FF specialists 

bring the identified connections together to learn more about the child (e.g., history and 

strengths/needs), and well as about the child’s life-long need for support and affection. The FF specialist

leads the team in creating a connect-o-gram (which is a visual depiction of the child’s level of 

connectedness), as well as in the development of an unmet needs statement for the child, which helps 

the team understand the areas in which the child could use support. The team then discusses potential 

permanent placement arrangements, as well as the role connections can play in supporting the child. 

They also identify challenges to achieving such plans and create solutions to combat them. Equal value is 

placed on the need for a permanent placement for the child and creating an enduring network of 

support. 

Decision-making. The goal of decision-making is for the family, in collaboration with the DSS social 

worker and other professional team members, to make decisions about the child’s legal and relational 

permanency. At the decision-making meeting, the team reviews the child’s mobility map (created during 

the discovery phase) as well as the connect-o-gram and unmet needs statement (created during the 

blended perspectives meeting), and identifies a family leader. The goal for the FF specialist is to leave 

the meeting with at least five family connections committed to being an emotional support and at least 

one plan for legal permanence. 

Post-decision making meeting support. In previous iterations of the model, the FF specialist transitioned 

off of the case, with the responsibility for following through on the developed plans falling to the social 

worker. This approach proved to be unsuccessful in the earlier evaluation of FF. Under the enhanced 

Tier 3 model, in cases with identified permanent plans, the FF specialist stays involved until those plans 

come to fruition (e.g., custody is divested, child is placed with relatives, etc.), which can be 12 to 18 

months. The goal of this enhanced version is to support the family through the rest of the process, and 

help the child make a successful transition into a permanent home. At this point, the FF case is 

considered “inactive,” but FF specialists reported this term to be inaccurate, and that the end of the 

decision-making phase is “really the beginning” of their work with families. 

As mentioned above, this is the point in which family preparation activities begin in earnest, and child 

preparation activities continue on a monthly basis. In addition to preparation activities, FF specialists 

keep the prospective permanent family involved and engaged in the process through phone calls, visits, 

and emails. They also help complete home studies and background checks, attend Child and Family 

Team Meetings and court hearings, and assist with the Interstate Compact on the Placement of Children 

process for children with plans to be placed with out-of-state relatives. For cases moving toward 
9 

adoption, the FF specialists make a referral to the CHS adoption specialist, who can further assist the 

family. As one DSS supervisor put it, “CHS is managing the family and DSS is managing the process.”

Team member relationships. While not an explicit component or phase of the model, an important 

aspect of the enhanced model is the emphasis placed on relationship-building and communication 

between the FF specialist and the DSS social worker. FF specialists are expected to make contact with 

the social worker within 24 hours of being assigned a case to schedule an initial meeting with the social 

worker, to take place within three days of case assignment. The purpose of this initial meeting is to 

gather information on the child, review the case file, and fill out necessary consent forms (e.g., to film 

the Stories from the Heart video). This meeting is sometimes combined with the collaborative team 

meeting, which is supposed to happen within two weeks of the case opening. Each collaborative team 

meeting is different, but the goal is to get all professional team members (e.g., social worker, guardian 

ad litem, therapist, etc.) on board with the process and “on the same page.” FF specialists also 

participated in monthly DSS meetings, such as Child and Family Team meetings or treatment team 

meetings, and would sometimes extend the time of such meetings to have the collaborative team 

meeting. However, some FF specialists reported not always being invited to or made aware of these 

meetings. 

FF specialists also maintain weekly contact (usually by email) with the social worker as well as provide a 

final report at the time of FF case closure. The weekly emails provide updates on FF activity and 

progress. Some counties reported only receiving monthly—not weekly—updates from the FF specialist, 

and reported some FF specialists were more consistent than others. The frequency of these updates 

tapers off during later phases of the model, when the frequency of activities naturally slows down. Most 

social workers indicated that they appreciate the weekly contact, where others reported it was too 

frequent and would rather have had updates only when there was something to report. Some social 

workers reported that the final report (which lists all contacts discovered for the child, what happened 

at the FF meetings, and the FF specialist’s case notes) was helpful to inform the court about diligent 

efforts to locate family members. 

B. Referral process 

DSS social workers make the decision to refer children for services with their supervisors, often during 

regular case staffings or other professional team meetings (e.g., Permanency Planning Action Teams, 

Permanency Round Tables). Due to the limited number of referrals allotted to each county, some 

counties prioritized the children most in need of services. Social workers in one county reported that the

local judge was ordering FF for some cases, and a guardian ad litem from another county mentioned 

that advocates were able to recommend that a child be referred for FF, but that it was up to the DSS 

social worker to make the referral. 

DSS social workers send the completed referral form to the CHS supervisor, either via fax or email. To 

aide in the decision on which tier of services is appropriate for the child, the two-page form lists the 

criteria for each service tier. The social worker indicates to which tier the child is being referred and fills 

in the demographic and case information on the child. Social workers described the form as self-
10

explanatory and easy to use. One county reported some technological issues within DSS that delayed 

the process, but otherwise, reported no difficulties with the referral process. 

Upon receipt of the form, the CHS supervisor reviews and approves the referral, and then assigns it to a 

FF specialist and creates a record for the child in CHS’s case management system (ECHO). This is usually 

done within a week of receiving the referral, but two counties reported delayed responses from CHS. If a 

large number of referrals come in at one time, CHS supervisors will stagger the start of services so they 

do not overload the FF specialist with many new cases at once. The number of referrals in each county 

ebbed and flowed, and there was a wait list for services from time to time. 

Between January 1, 2014 and February 19, 2016, a total of 478 children were referred for FF services. At 

the end of data collection, there were 42 cases on a wait list, pending staff assignment. Of the remaining 

referrals, 25 cases were not approved for service because they did not meet case goal criteria (10 cases) 

or did not have parental rights terminated (2), or for an unknown reason (13). Of the resulting 453

approved FF referrals, a total of 375 were served.11 In addition to the FF services, CHS served 113 

children through CSR services.12 (See Table 1).

Table 1. Number of cases (children) served by program type 

Family Finding CSR

Tier 1 Tier 2 Tier 3

Open 2 9 144 79

Closed 51 33 136 34

Total 53 42 280 113

About half of the counties in North Carolina made referrals. As to be expected, some counties referred 

more cases than others; for instance, Mecklenburg County, which accounts for well over half of all 

children in out-of-home care in the state, accounted for 55 percent of Tier 1 cases, 62 percent of Tier 2 

cases, and 9 percent of Tier 3 cases. Eight counties accounted for over 30 percent of the Tier 3 cases.13

Reasons for referring. Overall, DSS social workers referred children for FF services because the children 

were deemed to have lost connections with their family, and the social workers hoped to find family 

members willing to be part of a support network or become a permanent placement for the children. In 

some cases, children explicitly expressed the desire to reconnect with family. Social workers reported

they were especially interested in establishing support systems for older youth at risk of aging out of 

foster care. As one DSS supervisor reported, they had already “turned over every stone that exists” and 

need FF to “find a new stone.” DSS and CHS staff in counties both new to and with several years’ of 

experience with FF reported feeling pressure from supervisors or administrators to refer cases to FF. 

 

11 Thirty-six Family Finding cases were considered not being served due to their Family Finding case being closed within a 

month of referral. 

12 Due to data limitations, it was not possible to report the total number of cases referred and accepted for CSR services. 

13 Beaufort, Buncombe, Caldwell, Durham, Gaston, Harnett, Henderson, and Onslow counties. 
11

Some social workers reported they would not have made a referral had their supervisor not directed 

them to do so. 

Reasons for not referring. According to CHS leadership, they struggled early on in the project to meet 

the referral goals in each county. DSS staff reported four main reasons for not referring cases to FF: (1) 

social workers did not feel the need for the services; (2) there were few children in the county that met 

the referral criteria; (3) general reluctance on the part of the social worker; and (4) contractual and 

financial parameters. 

The demand for FF services was not present in all counties. With regard to Tier 1 services, in some cases 

the child’s parents had already provided family members’ contact information or the child had ongoing 

contact with family members, thus negating the need for the services. In other cases, DSS social workers 

reported they had already done their due diligence in identifying and engaging family members. In a few 

counties, DSS staff reported they conduct FF in-house, thus limiting the need for the Tier 1 services. A 

DSS supervisor in another county found Tier 1 “useless,” as social workers are already overwhelmed in 

general and receiving a long list of contact information to “weed” through is not helpful. Social workers 

in another county only wanted to refer cases to Tier 1, finding Tiers 2 and 3 unnecessary as they felt it 

best that the social worker engage the family members. 

Another reason reported by DSS for a lack of referrals was insufficient numbers of children who meet 

the referral criteria. Many children were reported to already have an identified permanent placement, 

or an established family support network. According to DSS staff, other children did not meet the 

criteria because FF had already been recently conducted on their case. As mentioned above, a few 

counties perform FF in-house, and many children had already been served in the counties that 

participated in the previous evaluation. Referrals were also affected by judicial practices, with DSS 

workers reporting numerous continuances to allow parents more time to complete case plans. DSS 

workers also reported that judges are hesitant to terminate parents’ rights until there is an identified 

permanent placement for a child. This practice limits referrals for CSR services.

In addition to children not meeting the referral criteria, some social workers reported being reluctant to 

make referrals. Some social workers were admittedly leery of the services (e.g., had not seen success 

with other cases, were protective of the children on their caseload), or found the services time 

consuming (e.g., communicating with the many family members discovered). Others may not have had a 

full understanding of FF and its potential benefits. In a couple of instances the social worker did not 

want to refer again based on negative experiences with the FF specialist (e.g., lack of responsiveness). 

Contractual and financial factors also affected referrals to FF. In a few counties, DSS staff reported they 

would have referred more children but were only allowed a certain number based on their contract with 

CHS. The main reason for not referring cases to Tiers 1 and 2, in fact, was simply that DSS did not have a 

contract with CHS for those services. Some DSS social workers reported the cost of the Tier 1 and 2 

services as a deterrent to referring children. In addition, CHS reported some counties were hesitant to 
12

use Tier 3 because if the child was adopted, the state would not be able to count the adoption for 

federal adoption incentive purposes (since CHS services were paid by the state). 

 

Efforts to increase referrals. In July 2014, CHS intensified its marketing efforts to increase the number of 

referrals. CHS executives met face-to-face with DSS administrators in each county under the PII contract 

with the aim of not only describing the services, but highlighting the benefits to children and DSS. CHS 

staff reported a major part of their work with DSS was building relationships—they wanted to make 

certain the county DSS agencies were satisfied with the FF services, and wanted to identify areas in 

which CHS could improve. 

Another major part of CHS’s “deliberate and aggressive outreach” was training for DSS staff on FF

services, how to make a referral, etc. The CHS trainer worked closely with DSS to identify specific FF

training needs, even creating trainings tailored to each county. For example, for counties that were 

familiar with the model but had yet to buy into or use the services, CHS provided a training called 

“Family Finding: A Closer Look” that helped DSS staff understand the model’s potential and highlighted 

FF success stories. 

More informal outreach occurred at the supervisor level, as FF supervisors would notify DSS supervisors 

via email when CHS had an opening for a new FF case. They would also offer one-on-one consultations 

with DSS staff to identify cases that were appropriate for referral. One FF supervisor talked of doing a 

“road show,” going to each county to talk with DSS directors and administrators about increasing 

referrals. News of successful FF outcomes also spread via word-of-mouth, which encouraged social 

workers to make referrals. 

CHS staff reported the intensive outreach efforts were successful and increases in referrals would occur 

following a presentation or training event. DSS staff regularly mentioned these CHS presentations when 

describing how they learned about the services or their decision to refer cases. The intensive marketing 

efforts were integral to maintaining an adequate number of referrals. 

Statewide variation. In general, DSS staff reported being aware of the services offered by CHS but not all 

staff understood the differences between tiers. More counties referred children for Tier 3 services (53 

counties) compared to Tiers 1 and 2 (8 and 6 counties, respectively). This is not surprising given that the 

Tier 3 services were provided at no cost, while Tiers 1 and 2 required contractual arrangements between 

CHS and the DSS county agency. One county referred more than half of all children referred to Tier 1 

and 2 services. There were no clear referral patterns by county characteristics (e.g., larger vs. smaller, 

new to FF vs. history of FF).

C. Description of children served
13

As expected, there were differences between children served by each program across a variety of 

characteristics (See Table 2, below).14 Children served by Tier 3 and CSR were older, have been in care 

longer, and had more prior placements than children served by Tier 1 and Tier 2. Tier 3 and CSR cases 

also had higher rates of medical, developmental or behavioral impairments than Tier 2 cases. A 

comparison of the Tier 3 and CSR children with children served during the previous evaluation finds 

children from the previous study were, on average, older than children from the current study and in 

care longer than children served by Tiers 1, 2, or 3. This suggests that children served by FF during the 

previous evaluation were at greater risk of not experiencing positive outcomes than children served 

during this study. 

Table 2. Child characteristics, by program/tier 

Family Finding

Tier 1

(N=52)a

Tier 2

(N=39)

Tier 3

(N=274)

CSR

(N=112)

Previous

Study

(N=561)

Genderb

Male (%) 56 59 57 50 58

Race/Ethnicity (%)

Black 543,C 49 391,P 381,P 503,C

White 293,C 263,C 451,2 501,2 40

Hispanic 8 213,C,P 7

2

 8

2

 6

2

Other 10 5 9

P

 4 4

3

Ageb

Average age 10.73,C,P 9.53,C,P 12.41,2,P 12.31,2,P 14.01,2,3,C

% under 9 313,C,P 493,C,P 141,2,P 131,2,P 6

1,2,3,C

% between 9-17 693,C,P 513,C,P 871,2,P 871,2 921,2,3

% over 17 0 0 0

P

 1 2

3

Medical/developmental or behavioral 

impairments (%)c

163 5

3,C 311,2 232 29

Avg. number of removals at referrald 1.13 1.1 1.31 1.2 1.2

Time in foster care at referral (years)e 1.33,C,P 1.03,C,P 2.01,2,C,P 3.91,2,3,P 3.31,2,3,C

Parental rights terminated at referral (%) 0

3,C

 3

3,C 151,2,C 691,2,3

Case goal at referral (%)f

Another Planned Permanent Living 

Arrangement (APPLA)

 2 0 0 0

Adoption 4

3,C

 5

3,C 241,2,C 831,2,3

Guardianship with other court approved 

caretaker

 4 15C 12C 3

2,3

Custody with non-removal parent/relative 2 0 3 0

Custody with other court approved 

caretaker

 0 3 7 3

Emancipation 0 0 1 0

Family reunification 753,C 643,C 451,2,C 111,2,3

Guardianship with relative 10C 10C 8

C

 1

1,2,3

Goal not yet established 0 3 0 0

 

14 As with any administrative and programmatic data, there were some missing data for these variables. The amount of 

missing data was generally low (exceptions are noted in the notes for Table 2) therefore, the issue of missing data was 

not addressed for the purposes of this formative evaluation. 
14

Family Finding

Tier 1

(N=52)a

Tier 2

(N=39)

Tier 3

(N=274)

CSR

(N=112)

Previous

Study

(N=561)

Prevention 43

Placement setting at referral (%)g

Home of parent(s) or trial home visit with 0

parents

Home of relative-regular or specialized 10C,P

Home of non-relative-regular, specialized 52

or therapeutic

Small congregate care setting 8

Large congregate care setting 27

Emergency shelter 2

Number of prior placements at referralh 3.93,C

1.03,C Number of prior therapeutic placements at 

referralh

Reason for latest entry into foster care (%)

Physical or sexual abuse 10

Neglect 832

Parent alcohol use 15P

293,P Parent drug addiction

Child behavioral problem 12

Parent incarceration 10P

153,C Parent unable to cope

Abandonment 6

Inadequate housing 62

i Other 2

Prepared for the adoption & recruitment process

(feelings toward adoption at referral)j

% opposed

% in favor

% feelings unknown

Extended family extensively explored 

before referral/extensive recruitment 

efforts made before referral (%)k 0

 0

 0

 0

72

 8

21

 0

3.83,C 

1.13,C 

23

641,3,C

 5

333,P

15C

 0

21C

15P

311,3,C,P

 8

 6

 01

 0

6C,P 

59

14

20

 2

 6.01,2

2.41,2 

12

812,C,P

 7

151,2,C

17C

 6P

341,P

 7

72,C,P 

 5P

 4

 0

 0

01,3 

68

12

20

 0

7.21,2 

3.11,2 

16

902,3,P

 8

303,P

42,3,P 

 6

431,2,P

 7

152,3

 4P

 3

25

71

 8

 1

21,3 

63

16

16

 1

14

743,C

 61

161,2,C

19C

31,3 

253,C

 72

112,3

93,C 

All tests of statistical significance in this table were chi-square tests, except (1) Fisher’s exact tests were used in cases when 

cell sizes were less than 5, and (2) ANOVA was used for continuous variables: age, number of removals, time in foster care, 

number of prior placements, and number of prior therapeutic placements. Superscripts show which programs were different 

from each other, per row, at the p<0.05 level (1=Tier 1, 2=Tier 2, 3=Tier 3, C=CSR, P=previous study).For example, a 

superscript of “1” in the Tier 3 column denotes that Tier 1 and Tier 3 values for that variable are statistically significantly 

different.

a The sample sizes presented in the header apply to all variables unless otherwise noted.

b The sample size for the previous study is 567. c Definition varied from the previous evaluation so comparisons cannot be made. See Appendix H for detailed information.

The sample sizes were: Tier 1, n=51; Tier 2, n=38; Tier 3, n=275; CSR, n=112; previous study, n=567. d Sample sizes were CSR, n=111; previous study, n=567. e Sample sizes were CSR, n=111; previous study, n=564.

f Information on case goal was obtained from NC administrative data and often varied from case goal data available in CHS 

case management (ECHO) system. The case goal information entered into ECHO is obtained from FF referral forms 

(completed by DSS social workers) and is believed to be more current than NC administrative data. We used the NC 

administrative data in order to compare FF and CSR cases.

g The sample sizes were: Tier 1, n=48; Tier 2, n=39; Tier 3, n=270; CSR, n=106; previous study, n=553.
Family Finding

Tier 1

(N=52)a

Tier 2

(N=39)

Tier 3

(N=274)

CSR

(N=112)

Previous

Study

(N=561) h Due to limitations of the administrative data, we were unable to determine the number of unique placements. This variable 

represents the number of placements a child entered prior to referral, which may include re-entries into the same placement 

following a placement elsewhere. The sample size for CSR was n=111. i ‘Other’ refers to the following entry reasons: child alcohol abuse, child drug addiction, death of parent, relinquishment, or 

child disability.

j

The sample size was 59. k For FF cases, this is defined has having nine or more relatives known at referral. For CSR cases, this is defined as having 

extensive (as opposed to minimal) past recruitment efforts. Therefore, comparisons cannot be made between FF and CSR 

cases. The sample sizes were: Tier 1, n=41; Tier 2, n=36; Tier 3, n=242; CSR, n=60.

To put this sample of children into perspective, we also made comparisons to publicly available data on 

all children in foster care in North Carolina.15 Compared to the statewide data, children served 

experienced more placements, were more likely to experience congregate care, and were less likely to 

experience placement with a relative. In terms of the time in foster care, Tier 1 and Tier 2 children 

reflected the statewide averages, but children receiving Tier 3 and CSR, as well as children served during 

the previous evaluation, had been in care longer than the state average. This comparison suggests that, 

even with a varied sample, children served during this study and during the previous evaluation have at 

least some characteristics that make them less likely to experience positive permanency outcomes than 

the average child in out-of-home care in North Carolina. When compared to the specific referral criteria 

CHS had for each program, we see Tier 3 and CSR services targeted the intended populations to a 

greater degree than Tier 1 or Tier 2 (See Table 3). 

15

16 Table 3. Target criteria

Criteria Criteria 

Met? Details

Tier 1

Receiving in-home services

New to out-of-home care

No

Yes

No Tier 1 children were receiving in-home services

Average of 1.1 prior removals

Tier 2

Receiving in-home services

Youth aging out of the foster care 

system

Children who have been in foster 

care for a long period of time

New to out-of-home care

No

No

No

Yes

No Tier 2 children were receiving in-home services

Average age of children served was 9.5 years

Average length of time in care was one year

Average of 1.1 prior removals

Tier 3

Children age 9-1717

Legally free or have an adoption, 

guardianship or custody case goal 

(those with reunification case goals 

Yes

Yes

87% of children served were 9 or over

Only about one percent had another case goal; more Tier 3 

cases were legally free than Tier 1 and 2

 

15 Duncan et al., 2016 and The Annie E. Casey Foundation, 2016. 

16 An additional criterion for Tier 3 and CSR services is “child is able to reside in a single-family home.” This criterion is not 

presented in this table because it was not possible to measure. 

17 Younger siblings of 9- to 17-year-old children were accepted for Tier 3 services as well. 
Criteria Criteria 

Met? Details

were accepted starting late in 

201418)

Extended family has not been 

extensively explored in the last one Yes Only 4% had their families extensively explored

to two years

CSR

Children age 9-1719

Legally free or have an adoption 

case goal and TPR has been filed 

with the court

Yes

Yes

87% of children served were 9 or over

83% had adoption as a case goal; 69% were legally free

Prepared for the adoption and 

recruitment process

Special needs (meaning being over 

age 13, has medical/developmental 

or behavioral impairments, and/or is 

of minority race/ethnicity)

Have spent two or more years in 

foster care

Yes

Yes

Yes

Only 3% of children opposed to adoption at referral

79% of the children served were over 13, had impairments, or 

were non-white

Average of 4 years in foster care

16

D. Program fidelity 

Given the extent to which CHS modified and developed enhancements to the original model, examining 

program fidelity is a priority. In the section below, we present findings on program fidelity in terms of 

whether or not all model components were delivered, as well as the timing with which the components 

were implemented. 

Throughout the evaluation period CHS managers oversaw data entry into the new fidelity fields within 

the ECHO case management system. Child Trends conducted analyses of the fidelity measures and met 

frequently with CHS managers to discuss preliminary data. This continuous feedback allowed for a 

comprehensive, collaborative examination of model fidelity. The various fidelity items reported by FF 

specialists are categorized into seven components: (1) discovery and engagement activities; (2) 

assessments; (3) child and family preparation; (4) blended perspectives meeting; (5) decision-making 

meeting; (6) post-decision-making activities; and (7) team member relationships. 

Appendix B presents data on the fidelity with which Tier 2 and Tier 3 services were implemented. We 

report on the percentage of closed cases implementing each fidelity item in each of these seven 

components, as well as the average value of several fidelity items when a percentage was not 

appropriate (e.g., average number of people attending meetings).20 It is important to consider that our 

 

18 Cases with reunification case goal could not make up more than 20 percent of the cases served between July 1, 2014 

and June 30, 2015. 

19 Younger siblings of 9- to 17-year-old children were accepted for CSR services as well. 

20 Fidelity items are recorded by FF specialists as being implemented or not. For most fidelity items, supervisors then 

approve the FF specialist’s entry. When supervisor approval was required, we only counted a fidelity item as being 

delivered or not if the supervisor approved the FF specialists’ entry. For some fidelity items, while information was 

captured in the Fidelity Index section of ECHO, we pulled data from other areas of ECHO in order to ensure the data on 
17

operationalization of fidelity only considers whether activities were completed, not the quality with 

which the activities were completed. Fidelity data is reported for Tier 2 and Tier 3 cases overall, and for 

Tier 2 and Tier 3 cases that opened early in the service period and later in the service period to see if 

fidelity improved over time.21 In addition to examining fidelity items individually, we also created a 

component-level measure for each closed case. This measure represents the percentage of items in a 

particular component that the case had completed. An important limitation is missing data. Some 

fidelity items suffered from large amounts of missing data (over 10 percent), limiting our ability to draw 

conclusions. The amount of missing data varied across the fidelity items; as a result, fidelity measures 

and resulting findings should be considered preliminary. See Appendix C for a full description of how 

component-level fidelity was calculated.22

We describe fidelity in more detail below, but in summary, the fidelity with which FF was implemented 

varies. Most of the items in the discovery and engagement activities, assessments, blended perspectives 

meeting activities, decision-making meeting activities, and team member relationships components 

were implemented with fidelity for the majority of cases. Post-decision-making meeting activities and 

child and family preparation activities were implemented with lower fidelity. Given that these two 

components were the newly added Tier 3 components, it is not surprising that they were implemented 

with lower fidelity than the other activities. Further, previous FF evaluations have shown that workers 

struggle to implement activities after the decision-making meeting given that such activities require 

more collaboration and there is a lack of clarity about roles in the latter stages of the model (Vandivere, 

2015). 

Overall, Tier 3 cases tended to experience better fidelity to the model than Tier 2 cases. However, 

among cases that implemented fidelity items, Tier 2 cases implemented fidelity items within the 

prescribed timeframe more often than Tier 3 cases. 

E. Fidelity by model component 

Below, we present fidelity findings for each model component. Appendix B presents component-level 

fidelity by tier. 

Discovery and engagement. Most discovery and engagement activities were completed for the majority 

of Tier 2 and 3 cases. FF specialists completed an average of 73 percent of the required discovery and 

engagement activities for each Tier 3 case, compared to 60 percent for Tier 2 cases. The most commonly 

completed items were the initial meeting with the youth, mobility mapping, and mining the child’s case 

file, which were completed on at least 83 percent of cases. An area of improvement is contacting 

 

the fidelity item was as complete and accurate as possible. For instance, instead of relying on the Fidelity Index field that 

indicates whether at least 40 connections were discovered, we pulled data on the number of connections discovered 

from a separate area in ECHO that lists every discovered connection for each child. 

21 We defined “early in the service period” as January 1, 2014 through July 31, 2014 and “later in the service period” as 

August 1, 2014 through February 19, 2016. 

22 Note that component-level fidelity is calculated based on items that were expected of all Tier 3 cases; activities that 

were not applicable for some cases were excluded from calculations. 
18

(defined as having two-way contact) at least five family members between file mining and the blended 

perspective meeting. This item was not implemented frequently (28 percent of Tier 2 cases and 61

percent of Tier 3 cases), and the percentage of cases completing this activity decreased over time. 

However, since this component was heavily reliant on family members engaging with the FF specialists, 

performance on this item was often out of the FF specialists’ direct control. At least 40 connections were 

identified in the majority of cases (81 percent for Tier 2 and 70 percent for Tier 3), and for Tier 2 cases, 

the achievement of this benchmark improved over time (57 percent for early cases compared to 100

percent for later cases). Tier 3 cases were implemented with significantly better fidelity than Tier 2 cases 

on two of the discovery and engagement activity-related items (initial meeting with youth and 

contacted at least five family members after file mining and prior to holding the blended perspective 

meeting).

Many of the challenges to maintaining fidelity to the model are outside of the FF specialists’ control; for 

example, being able to make contact with family members before the first blended perspectives

meeting. FF specialists cited issues such as not being able to locate family members, (e.g., only having 

old information and therefore “chasing the white rabbit”), family members living far away (e.g., out of 

state or even out of the country), and family members not expressing interest in involvement. Even 

when FF specialists were able to make initial contact with family members, family members often “drop 

off the radar” and stop responding to the FF specialist. Some family members were reported to be 

“scared” of DSS or hesitant to share information about other relatives. Family dynamics can also play a 

part; one DSS social worker recounted an incident when a biological parent threatened other relatives if 

they engaged with the agency. Family members may also not understand the urgency of the child’s 

situation or may be overwhelmed by the child’s needs.

While the aforementioned issues were a struggle for FF specialists, they were equipped with strategies 

to overcome them. Conducting home visits was seen as a good way of building rapport with families, 

which increased their likelihood of engaging with the FF specialist and sharing contact information on 

other relatives. Other effective ways of building rapport included being patient with the family, listening 

to their stories, and in general being understanding of their point of view. According to DSS staff, the FF 

specialist being from CHS and not DSS in and of itself made the family more likely to engage. 

The Stories from the Heart videos—which were completed for almost two thirds (65 percent) of 

children—were reported to be an extremely useful tool for engaging families, by both DSS and CHS staff. 

The videos helped the family reconnect with the child, and were often very moving to the families. FF 

specialists would sometimes mail the videos to family members, which is especially helpful for family 

that lives out of state. However, sometimes the child was placed far away from the FF specialist’s work 

location, thus making it more difficult to complete the Stories from the Heart videos. FF specialists also 

reported not being able to engage all children; some refused to be filmed for Stories from the Heart 

while others were not emotionally or mentally stable enough to engage.

Assessments. As described above, FF specialists are expected to conduct a variety of assessments during 

the FF case. FF specialists completed an average of 73 percent of the required assessment activities for 
19

each Tier 3 case, compared to 53 percent for Tier 2 cases. Some assessments were administered more 

regularly than others. The initial child assessment was conducted for all cases; the well-being baseline 

interview was administered for about half (55 percent) of Tier 2 cases and far more (87 percent) of Tier 

3 cases. The subsequent well-being follow-up interview was rarely administered (no Tier 2 cases and 

only 12 percent of Tier 3 cases) and the percentage of cases having this survey administered decreased 

over time (19 percent of early Tier 3 cases compared to 4 percent of later cases). CANS assessments 

(only completed on Tier 3 cases) were completed for most (95 percent of) children; however, the 

caregiver section23 was only completed for a little more than one third (39 percent) of cases. The 

Permanency Readiness Measure was completed for almost half (46 percent) of Tier 2 cases but almost 

all (95 percent of) Tier 3 cases. Two thirds (66 percent) of Tier 3 cases had a PRM completed on a 

quarterly basis. The percentage increased over time, with 52 percent of early cases having quarterly 

PRMs compared to 83 percent of later cases. Overall, Tier 3 cases were implemented with significantly 

better fidelity than Tier 2 cases on two assessment-related items (completed the well-being baseline 

interview and PRM).

Some FF specialists reported that completing the assessments was time consuming—both administering 

them and also entering the information into the case management system. Other FF specialists reported

not understanding the purpose, while others reported not being confident they have sufficient or the 

correct information to complete the assessments, in particular the CANS.

Child and family preparation. FF specialists are expected to complete regular child and family 

preparation activities. While the first child preparation activity was completed for almost three quarters 

(71 percent) of Tier 3 cases, less than one quarter (23 percent) of cases had preparation activities 

completed on a monthly basis. The low percentage of cases completing monthly preparation activities 

may not reflect a lack of effort on the part of the FF specialist; rather, it could be attributed to other 

factors, such as lack of access to the child. FF specialists conducted preparation activities more regularly 

for cases opened later in the evaluation period than cases opened earlier. This is not surprising, given 

that some FF specialists reported struggling with these activities early on because they had less 

experience with these types of activities and had a difficult time conceptualizing and/or operationalizing 

the activities. 

The newness of these tasks, coupled with youth reported to be difficult to engage, made it more difficult 

to complete preparation activities in accordance with the model timeline. While most children were 

open to participating in such activities, FF specialists came across older youth who wanted to opt out, 

most likely due to fear of rejection from family members. FF specialists tried to “think outside the box” 

when working with the children, using creative strategies that were tailored to the interest of each child 

(e.g., sports, arts/crafts, soundtrack of your life, etc.). Another supervisor mentioned caseload size as 

another challenge in completing this phase of the model with fidelity; the activities are time-intensive, 

making them difficult to complete with a full caseload.

 

23 As mentioned earlier, the caregiver section of the CANS was only completed on cases where an individual had been 

identified as a potential placement resource, which explains the low percentage of cases with this item completed.
20

Similar patterns appear with regard to family or family/child preparation for permanency activities. The 

first activity was completed for 65 percent of cases; however, far fewer completed the activities monthly 

(16 percent). Distance and scheduling posed challenges in completing these activities, with children and 

families living far away from one another. FF specialists often used technology (e.g., Skype or conference 

calls) when distance was a barrier to completing preparation activities. They were also willing to travel 

out of state and meet with family on weekends in order to accommodate families’ work schedules. 

Blended perspectives and decision-making meetings. Blended perspectives meetings were held for 

approximately two thirds (68 percent) of Tier 2 cases and almost three quarters (74 percent) of Tier 3 

cases. Required meeting components—connect-o-gram, greatest unmet need statement, review of 

child’s history, strengths and needs—were delivered in the majority of cases that had a blended 

perspectives meeting. Tier 3 cases were implemented with significantly better fidelity on some items 

(e.g., social worker in attendance at meeting, review of child’s history, strengths and needs) compared 

to Tier 2 cases. Cases opened later experienced significantly weaker fidelity on other items (e.g., 

meeting with and preparing the social worker and actually holding the meeting) compared to earlier 

cases.24

Similarly, most Tier 2 (65 percent) and Tier 3 (72 percent) cases had a decision-making meeting held, and 

among those that had a decision-making meeting, most of the required meeting components (e.g., 

review of mobility mapping and greatest unmet need statement, social worker in attendance) were 

delivered. Tier 3 cases were implemented with significantly better fidelity on some items (e.g., review of 

mobility mapping and greatest unmet need statement) compared to Tier 2 cases. Later cases 

experienced significantly lower fidelity on items related to preparing the family and social worker for the 

decision-making meeting, and even holding the meeting, compared to earlier cases.25 Later Tier 3 cases 

experienced significantly higher fidelity in terms of securing commitments to legal permanency from 

connections than earlier Tier 3 cases. 

FF specialists and supervisors reported having a difficult time reaching the planning and decision-making 

phases of the model. Approximately two thirds of the people invited to the blended perspectives and 

decision-making meetings attended (five participants on average), and FF specialists did not always feel 

the number was sufficient for a productive meeting. Challenges to attendance included logistical issues 

(e.g., distance, work schedules). In addition, FF specialists reported family members simply 

“disappearing” before attending a meeting. FF specialists acknowledged that identifying a family leader 

(which was accomplished for approximately three quarters of cases) was a good way to make 

connections to more extended family members. To overcome logistical challenges, FF specialists 

reported being flexible with meeting times (e.g., meet on weekends or after regular business hours) and 

locations (e.g., use Skype, churches, CHS offices, home of relative). However, meeting during nonregular business hours posed a challenge for county social workers, some of whom were not willing to 

 

24 These findings varied by Tier. See Appendix B for details.

25 These findings varied by Tier. See Appendix B for details.
21

meet during these times or were not allowed to do so. FF specialists reported the absence of the social 

worker at the meetings limits the meeting’s productivity.

Family dynamics can also be a barrier to conducting the blended perspectives and decision-making 

meetings with fidelity. As a FF supervisor reported, sometimes family history (e.g., tension between the 

maternal and paternal sides, conflict between two members of the family) “bubbles up” at meetings, 

thus limiting their productivity. In such cases, FF specialists post rules for the meeting (e.g., only one 

person speaking at a time, be respectful of the opinions of others), invite another FF specialist to attend 

as a mediator, and will hold two separate meetings (e.g., one for maternal and another for paternal 

members), if necessary. 

FF specialists reported they regularly secure commitments from at least five family members to provide 

emotional (65 percent of Tier 3 cases) and legal permanency (53 percent of Tier 3 cases). FF specialists 

shared that meeting activities (e.g., connect-o-gram, Stories from the Heart video, greatest unmet needs 

statement) are effective in helping family members understand the importance of being an emotional 

support to the child, and often influence the family members’ decision to commit to being an emotional 

support. FF specialists reported difficulty getting family members to take the next step towards a legal 

commitment, especially if the DSS social worker is not supportive of such commitments. Even though 

social workers are thought to have a “laser focus” on legal permanency, FF specialists reported the 

workers are sometimes hesitant to disrupt a child’s stable placement with a non-relative foster family to 

move the child to a relative placement. 

Post-decision-making meeting activities. The degree to which post-decision-making meeting activities 

occurred for Tier 3 cases varied, and only half (50 percent) of the required activities were completed. 

Among Tier 3 cases that had a decision-making meeting, most had the family connections report given 

to the family leader (69 percent), were referred to resources before the case closed (71 percent), and 

had plans created during the decision-making incorporated into the DSS case plan (67 percent). While 

not an official fidelity item, FF specialists are also expected to stay involved with Tier 3 cases until the 

child welfare agency divests custody. This occurred in approximately 19 percent of the closed Tier 3 

cases.26

According to FF specialists and their supervisors, the decision-making meeting is the point at which the 

bulk of the work on a case starts, and continues to be a learning process. In addition to continuing with 

child and family preparation activities, the FF specialist is expected to work with the DSS social worker to 

keep the family engaged in the process and ensure that plans made during the planning and decisionmaking phases are incorporated into the child’s DSS case service plan. The decision to do so, however, 

ultimately lies with the DSS social worker. FF supervisors reported that the DSS social workers did not 

always keep the FF specialist updated on the child welfare case, and in a couple of cases decided to 

move forward with a different placement option and not tell the FF specialist. In light of such challenges, 

 

26 We considered a FF specialist to have remained involved in a case if (1) the FF case ended after the child welfare 

agency divested custody, or (2) the FF case closed 14 or fewer days before the child welfare agency divested custody.
22

FF specialists try to be persistent in moving the ball forward towards permanency. While some DSS 

social workers may be viewed as not being “on board” in terms of moving forward with plans, others 

valued the guidance provided by FF specialists as they shepherd the family through the process. 

Team member relationships. In general, the FF specialists were able to maintain good relationships with 

DSS social workers and other professional team members (completing approximately 90 percent of the 

required activities), and rated the social worker’s participation in the process as moderate (3.5 to 4.0 on 

a scale of 1 to 5). FF specialists had an initial meeting with the social worker for every case (both Tier 2 

and 3), and a collaborative team meeting for almost all cases. They were able to maintain weekly 

contact with the child’s social worker for most cases (84 percent for Tier 3 cases), with a few social 

workers reporting infrequent contact from the FF specialist.27 

F. Timing and fidelity 

In addition to examining the completion of fidelity items, we explored the percentage of cases for which 

items were completed within the timeframe prescribed (see Appendix B). This analysis was only possible 

for fidelity items that had both a prescribed timeframe and dates of completion. In general, for each 

item about half of Tier 2 cases with an item completed had it completed within the prescribed 

timeframe. Fewer Tier 3 cases had items completed in the prescribed timeframe. When comparing cases 

opened earlier in the study period with those opened more recently, the cases opened more recently 

tended to see items completed within the prescribed timeframe more regularly. 

FF specialists and supervisors also reported they struggled to meet discovery and engagement timelines, 

for reasons outside of their control. For example, the only item FF specialists were able to routinely 

complete within the specified timeframe was file mining (completed on time for 93 percent of Tier 3 

cases). Meeting with the child is often difficult, and was not regularly completed on time (only 19 

percent of Tier 3 cases). These activities are expected to happen very early in the FF case, so their delay 

usually has a domino effect on subsequent activities. Sometimes the child is placed far away from the FF 

specialist’s work location, thus making it more difficult to visit the child and complete mobility mapping 

or Stories from the Heart videos; only completed on time for 63 and 27 percent of Tier 3 cases, 

respectively. This struggle to meet timelines is exacerbated when FF specialists feel they have too many 

cases in the discovery and engagement phase at one time. 

FF specialists often struggle to complete the assessments within the specified timeframes as some 

depend on the cooperation of the social worker or child. For example, the initial child assessment was 

only conducted within three business days for 35 percent of Tier 3 cases, and the well-being baseline 

interview was only held with the youth within two weeks for 22 percent of Tier 3 cases. 

Another challenge was having the initial meeting with the social worker and the collaborative team 

meetings on time; the initial meeting with the social worker was only held within three business days for 

38 percent of Tier 3 cases, and the collaborative team meeting was only held within two weeks for 39 

 

27 There is a significant amount of missing data for Tier 2 cases on this item, which limits our ability to draw conclusions. 
23

percent of Tier 3 cases. However, the ability to meet these benchmarks on time increased significantly 

as the project went on. As one FF supervisor reported, social workers are busy and overloaded, making 

it difficult to schedule meetings. As mentioned above, FF specialists tried to combine meetings 

whenever possible, particularly in the early stages of the model, thus reducing the burden on social 

workers. 
24

G. Implementation drivers 

The successful implementation of FF services was driven by several factors, including staff competency 

(both CHS and DSS staff), leadership, program outreach, and permanency practices at DSS. These 

implementation drivers are discussed in further detail below. 

CHS staff training and supervision. Overall, FF specialists reported they received adequate training and 

supervision to do their jobs effectively. Training was available formally from CHS and informally from 

supervisors or colleagues; one FF specialist reported that the best way to learn is “on the job.” FF 

specialists and their supervisors reported the need for continued training on child and family 

preparation and post-decision making supports, especially for the veteran workers who had limited 

experience with the new additions to the model. Supervisors met with staff regularly on a monthly basis, 

as well as on an ad-hoc basis as needed. FF specialists in some areas found it helpful to have group 

supervision, saying that it is “reassuring to talk to someone who is going through the same thing.”

DSS training on and understanding of the model. As a part of program outreach (described earlier), CHS 

offered a variety of FF trainings to DSS staff in each county under the PII contract, and was able to 

provide these trainings in most counties. These trainings covered not only a description of the services, 

but walked social workers through the referral process and outlined the roles and responsibilities of CHS 

and DSS staff in the process. Most counties participated in the trainings, although one DSS supervisor 

commented they did not need to be trained on the services, but only needed “the avenues to get the 

services.” Interestingly enough, social workers in that county reported they would have liked more

training on the services. DSS administrators and supervisors were complimentary of the trainings, but 

did not think the amount of training was sufficient, citing high rates of turnover among DSS staff and 

thus the need for multiple trainings. 

DSS staff’s understanding of the different service tiers and the associated eligibility criteria varied widely 

by county. Some staff (usually supervisors) could quickly describe the different tiers and eligibility 

criteria, where others only knew that there were different tiers, but not the differences among them. 

The lack of familiarity with the services was partly attributed to high levels of staff turnover; people 

were new and therefore not familiar with the services. Staff in the counties with longer histories with FF

were more familiar with the FF model in general, but less familiar with the different service tiers. DSS 

supervisors in some counties also lacked a full understanding of what services were available to their 

social workers under the county’s contract with CHS. The trainings mentioned above may not have been 

successful in increasing social worker understanding of the service tiers, but they were successful for the 

most part in increasing awareness that the services were available. 

Leadership. In general, DSS administrators and CHS executives reported the state was supportive of the 

FF initiative, which in turn encouraged the initiative’s acceptance by DSS leadership. CHS leadership 

reported that DSS administrators were supportive of the FF initiative and actively encouraged staff to 

use the services (e.g., by requesting CHS training on FF, emailing reminders of service availability to 

supervisors). However, this encouragement may have been misconstrued by DSS supervisors and social 

workers as a requirement or pressure to refer cases, which may not foster authentic support for FF
25

efforts. Support from leadership alone is not enough; without the buy-in of the front line workers, the 

program cannot be successful. 

Program outreach. CHS has a history of investment in FF, and according to a CHS executive, as an agency 

they “believe in the program and embrace it wholeheartedly.” As described above, CHS made a 

concerted effort to increase the number of FF referrals. CHS recognized that if DSS staff were unaware 

of the program and its potential benefits, no one would make referrals. CHS also understood the 

importance of the relationship between the two agencies, and that DSS would not refer cases (even 

when they are free) if DSS staff did not trust CHS programs. These outreach efforts were mentioned by 

both CHS and DSS staff in every county, which is a testament to their success. 

DSS permanency practices. A system-wide focus on permanency is critical for FF to be successful in 

moving children to permanency, as the final decision about a child’s permanent placement resides with 

DSS. In general, CHS and DSS staff reported DSS is doing a good, but not excellent, job of achieving 

permanency for children. This varies by county; some DSS staff reported they struggle to find permanent 

homes for children, others reported they have a difficult role but are slowly improving. Below we 

describe specific agency practices that affect children’s achieving permanency and therefore affected

the implementation of FF.

 Definition of permanency. According to DSS and court staff, permanency is unique for every 

child. DSS social workers reported mixed opinions on whether legal or relational permanency 

should be the priority. Most social workers agreed that children in care need some sort of 

supportive network of adults (including relatives, mentors, teachers, previous foster parents, 

etc.) to help them feel like they belong, or to call when they are in trouble. For older youth, 

some social workers prioritize relational permanency in an effort to build such a network the 

youth can rely on once they leave care. However, social workers also reported prioritizing legal 

permanency because that is the standard to which they are held accountable. 

 Reunification efforts. In most counties, DSS social workers reported their primary focus for legal 

permanency for children is reunification. Social workers in a few counties reported they have 

intensive family preservation or reunification programs. One judge noted an increased emphasis 

on locating family earlier in the process to help be a support to biological families as they work 

toward reunification. 

 Focus on family. DSS staff at all levels reported that when a child cannot successfully reunify 

with their parents, the child should be placed with family. While DSS social workers in some 

counties reported their county has always prioritized placing children with relatives, workers in 

other counties reported more emphasis on relative placements in recent years. While, in 

general, DSS staff reported being family focused, some judicial staff and CHS staff reported that 

is not always the case. 

 Court practices. DSS social workers reported they do their best to move children into permanent 

placements but frequent court delays and continuances lengthen the process. Workers also 

reported some judges are hesitant to terminate parental rights until there is an identified 

placement for the child, because they do not want to make the child a legal orphan. 
26

III. Program Outcomes 

This section presents a description of the outcomes experienced by children served, and whether there 

are differences across subgroups of children. We explored whether outcomes varied by service and 

whether any factors beyond service receipt, such as degree of fidelity, explain the outcomes 

experienced. Below we categorize the outcomes into 1) placement and permanency related and 2) childrelated. Placement and permanency outcomes include achievement of legal permanency (i.e., adoption, 

guardianship, reunification); step downs in placement; moves to kinship care from a non-kin placement; 

and commitments by adults to legal permanency or relational permanency (ongoing support). Childrelated outcomes include the number of family connections discovered; the child’s knowledge of their 

family history; the child’s overall permanency readiness; and well-being.28 In addition to our exploration 

of outcomes for children in this study, we compared outcomes experienced by children and youth 

served during this study period with the outcomes of children served during the previous evaluation 

period. 

Given CHS’s extensive efforts over the past two years to both revise and enhance the original FF model, 

we explored whether or not model fidelity is associated with child outcomes. Following the discussion of 

outcomes, we present results from this exploration of fidelity. When interpreting findings, it is 

important to keep in mind that more than half of the Tier 2 cases were served in one county. Both CHS 

staff and DSS staff participating in our interviews and focus groups reported this county highly values—

perhaps more than other counties—placing children with relatives. Therefore, differences seen between 

Tier 2 and other services may be due, in part, to this difference. 

A. Permanency and placement outcomes 

Table 4 presents the first three permanency and placement outcomes experienced by children (with 

closed FF/CSR cases) served by the three tiers of FF and the CSR program.29 

Legal permanency. We examined whether children were discharged to permanency—defined as 

discharge to adoption, guardianship, or reunification—at some point following entry into FF or CSR 

services. As shown in Table 4, among closed FF/CSR cases, the percentage of children who achieved 

permanency ranged from 14 percent of Tier 1 cases to 37 percent of Tier 2 cases. Almost 20 percent of 

Tier 3 cases achieved legal permanency during the study period. Comparisons across the three FF tiers 

and CSR cases show Tier 2 cases were more likely than Tier 3 cases to be discharged to permanency. 

 

28 As with any administrative and programmatic data, there was some missing data for the outcome variables. The 

amount of missing data was generally low (child well-being outcomes based on the Permanency Readiness Measure and 

CANS had more missing data, but those outcomes are not central to the analyses), and therefore missing data was not 

addressed in these analyses. 

29 For all tables in this section, tests of statistical significance were chi-square tests except Fisher’s exact tests were used 

in cases when cell sizes were less than 5. Superscripts show which programs were different from each other, per row, at 

the p<0.05 level (1=Tier 1, 2=Tier 2, 3=Tier 3, C=CSR). For example, a superscript of “1” in the Tier 3 column denotes that 

Tier 1 and Tier 3 values for that variable were statistically significantly different.
27

Table 4. Percentage of closed FF/CSR cases that experienced positive placement and permanency 

outcomes

Family Finding

Tier 1 Tier 2 Tier 3 CSR

% % % %

Achieved legal permanency 142 371,3 192 27

(n=50) (n=30) (n=135) (n=34)

Experienced a stepdown 40 60 42 48

(n=45) (n=30) (n=128) (n=31) a 501,3,C Experienced a move from non-relative to relative care 222 162 132

aAmong cases placed in non-relative care at FF/CSR opening. (n=41) (n=30) (n=125) (n=32)

When analyzing specific discharge to permanency outcomes, we found there were differences in the 

outcomes experienced by cases served by each program. For example, Tier 2 cases were more likely to 

experience reunification than Tier 3 cases. This is not surprising given Tier 3 cases were more likely to 

have parental rights terminated than Tier 2 cases (15%, compared with 3%, see Table 2). Also, Tier 3 

cases were less likely than CSR cases, but equally as likely as Tier 2 cases, to be discharged to adoption. 

There were no differences between programs in terms of discharge to guardianship. See Appendix D for 

a table showing the specific discharge outcomes experienced by children, by program. 

Given the descriptive nature of the findings, there are likely a number of factors associated with 

outcomes. Other factors include how time (i.e., length of time since the FF/CSR case opened) and case 

status may affect whether we observed an outcome or not, child characteristics, and the level of 

experience the county in which the child is receiving the services has with FF.

30 The length of time since 

FF/CSR opening is associated with permanency outcomes in a number of ways. First, children who failed 

to attain permanency during the study period may ultimately attain permanency after the study’s 

observation period. In addition, a child who began FF/CSR services shortly before the observation period 

ended may be less likely to experience a permanency outcome than a child who began services during 

the initial months of the study. The status of the case—whether or not the case was closed due to 

reasons other than permanency (e.g., aging out of care)—also affects whether the child achieved 

permanency. Child characteristics are also associated with outcomes. For instance, older children are 

more likely to age out of care than younger children. Lastly, children being served in counties with a 

longer history of using FF services may experience better outcomes than children served in counties new 

to FF.

31 See Appendix E for results of analyses that take into account these factors.32

 

 

30 Another factor is county of origin. Different counties may have different policies and cultures that affect the outcomes 

experienced by the children they serve. We were unable to account for county in our multivariate models due to small 

sample sizes. 

31 The more experienced counties are: Buncombe, Catawba, Cumberland, Durham, Gaston, Guilford, Mecklenburg, New 

Hanover, and Wake counties. 

32 Appendix E does not show the results from comparisons between more and less experienced Family Finding counties 

and from analyses of well-being outcomes because there were no noteworthy differences found. 
28

After accounting for time/case status33 and child characteristics,34 our findings with regard to legal 

permanency remained consistent—Tier 2 cases were more likely than Tier 3 cases to reach legal 

permanency. Among all closed FF cases (Tiers 1-3), there were no differences between more and less 

experienced counties in terms of whether cases ever experienced permanency after FF opening.

We also compared our findings to the findings from the previous evaluation. In the previous evaluation, 

23 percent of children assigned to the treatment group were discharged to permanency.35 This 

compares to 37 percent of Tier 2 cases and 19 percent of Tier 3 cases in this study, suggesting that 

children served by Tier 2 experienced better outcomes than children in the previous study even with a 

shorter observation period within which to experience these outcomes.

36 However, as stated above, a 

comparison of children served during the previous evaluation with children served during the current 

study found that children from the previous study tended to be older than children from the current 

study (14 years old compared with 10 and 12, for Tiers 2 and 3, respectively) and in care longer prior to 

referral to FF (3 years compared with 1 and 2 years for Tier 2 and Tier 3, respectively), suggesting that 

children from the previous study were even more challenging to serve. 

Placement step downs. Another outcome of interest was whether the child experienced a step down in 

placement (defined broadly as the child moving from a more restrictive level of care to a less restrictive 

level of care).37 First, we examined whether a child ever experienced a step down after FF/CSR case 

opening. As shown in Table 4, there were no significant differences between the different FF tiers or CSR 

cases in whether they ever experienced a step down. After controlling for time/case status, our findings 

remained consistent.38 However, after controlling for child and case characteristics, Tier 2 cases were 

shown to be more likely than Tier 3 cases to experience a stepdown.39 This finding suggests that there is 

something inherent to Tier 2 services, or some other factor besides child/case characteristics associated 

with the receipt of Tier 2 services (such as county of origin), explaining why Tier 2 cases were more likely 

 

33 We used competing risks survival analysis to account for time/case status. This method enabled us to account for both 

the influence of time and case status on children’s outcomes. Specifically, competing risks survival analysis allowed us to 

account for the fact that some cases were still open and we did not observe the outcome yet (censored cases), some 

cases experienced the outcome of interest (for example, stepping down), and some cases cannot experience the 

outcome because they experienced a competing event (for example, having their case closed before they experienced a 

stepdown).

34 We added several variables to our competing risk models to account for child age, foster care history, and more. 

35 Malm, Vandivere, Allen, Williams, & McKlindon, 2014.

36 It is important to consider that the previous evaluation measured outcomes for all children assigned to the treatment 

group, regardless of whether or not their FF case was still open. In the current study, we present the percentage of 

children achieving various outcomes only among closed FF/CSR cases. We believe the previous study’s treatment group 

and children with closed FF/CSR cases from the current study are comparable because in the previous study, random 

assignment ended about a year and a half prior to the end of the study period. Therefore the majority of children in the 

treatment group likely had their FF case closed when outcomes were measured. It is also important to consider that the 

observation period for the previous study was just over 4 years (June 2008 - October 2012) and the observation period 

for the current study is just over 2 years (January 2014–February 2016). Therefore, children from the previous study had 

more time to realize an outcome than children in the current study. Given a longer observation period for the current 

study, we could expect more children to experience positive outcomes. 

37 See Appendix H for a more complete definition of stepdown.

38 Competing risks survival analysis was used.

39 We added to our competing risk models several variables to account for child age, foster care history, and more.
29

to experience step downs than Tier 3 cases. We found no differences in the likelihood of step downs 

among closed FF cases (Tiers 1-3) between more and less experienced counties. 

In the previous evaluation, 43 percent of the treatment group experienced a stepdown in placement 

during the study period.40 This is comparable to the 42 percent of Tier 3 cases that experienced a step 

down in the current study. The 60 percent of Tier 2 cases represents a much higher percentage than the 

percent of children that experienced a step down in the previous evaluation. Again, this is likely due in 

part to differences in child characteristics. 

We also examined an alternate method of defining stepdown. Table 5 presents the percentage of cases 

that had a stepdown as a final placement among cases that ever experienced a stepdown.41 There were 

no differences between programs in terms of whether children who experienced a stepdown were in a 

stepdown at child welfare case closure (or the end of the study period for child welfare cases that were 

still open). After controlling for time and child characteristics, findings remained consistent.42 We found 

no differences in the likelihood of experiencing a step down as a final placement among closed FF cases 

(Tiers 1-3) between more and less experienced counties. 

Table 5. Percentage of all closed FF/CSR cases with a stepdown as a final 

ever experienced a stepdown

placement, among cases that 

Family Finding

Tier 1 Tier 2 Tier 3 CSR

% % % %

Stepdown as final placement 88

(n=16)

82

(n=17)

76

(n=50)

79

(n=14)

Moves to relative care. We also examined whether children who started in non-relative placements ever 

moved to a relative placement after FF/CSR case opening.43 As presented in Table 4, Tier 2 cases were 

more likely than Tier 3 cases to have ever experienced a move to relative care. After accounting for 

time/case status44 and child characteristics,45 our findings remained consistent, except Tier 1 cases, in 

addition to Tier 2 cases, were more likely than Tier 3 cases to ever experience a move to relative care. 

Among all closed FF cases (Tiers 1-3), there were no differences between more and less experienced 

counties in terms of whether cases moved from non-relative to relative care. As we mentioned 

 

40 Malm et al., 2014. 

41 We compared first placement at FF/CSR case opening to the final placement at child welfare case closure (or the end 

of the study period) to determine if a stepdown occurred between those two placements. 

42 Logistic regression was used. To account for time, we controlled for the length of time between FF/CSR case opening 

and child welfare case closure or the end of the study period (for open child welfare cases). To account for child 

characteristics, we added several variables to our logistic regression model to account for child age, foster care history, 

and more. Competing risks survival analysis was not used to account for case status because we were interested in final 

placement at the end of the child welfare case or the end of the study period, therefore, there was no need to account 

for censoring. 

43 We conducted this analysis only among cases that were in non-relative care at FF/CSR case opening. See Appendix H 

for a more complete definition of this outcome variable. 

44 Competing risks survival analysis was used. 

45 We added several variables to our competing risk models to account for child age, foster care history, and more. 
30

previously, one possible explanation for the difference between Tier 2 and Tier 3 cases on this outcome 

is the county from which the cases originate. 

We examined an alternate method of defining moves to relative care, specifically, whether children who 

started in non-relative care and moved to relative care, were in relative care as their final placement. 

There were no differences between programs in terms of whether children who started in non-relative 

care and ever moved to relative care were in relative care as their final placement (See Table 6). After 

controlling for time and child characteristics, our findings remained consistent.46 Among closed FF cases 

(Tiers 1-3), we found no differences in the likelihood of being in relative care as a final placement 

between more and less experienced counties. 

Table 6. Percentage of all closed FF/CSR cases with a relative as a final placement, among 

started in non-relative care and ever moved to relative care

cases that 

Family Finding

Tier 1 Tier 2 Tier 3 CSR

% % % %

Relative as final placement 67

(n=9)

87

(n=15)

65

(n=20)

100

(n=4)

Commitment to legal and relational permanency. By the close of the FF case, FF specialists record 

whether commitments to legal and relational permanency were made by adults (kin or non-kin) 

engaged through the FF process. We found no statistically significant differences between Tier 2 and 

Tier 3 cases in the likelihood of these commitments (see Table 7). While we found that Tier 1 cases were 

less likely than Tiers 2 and 3 to secure these commitments, this is not surprising, given that obtaining 

commitments was not an explicit goal of Tier 1 services. After accounting for time and child 

characteristics, we find that Tier 2 cases were more likely than Tier 3 cases to obtain commitments to 

legal permanency (but not relational permanency).47 In addition, among all closed FF cases (Tiers 1-3), 

there were no differences between more and less experienced counties in terms of whether cases 

obtained these commitments. Finally, when compared to children in the previous evaluation, children in 

the current study were more likely to obtain at least one commitment to relational permanency (84 

percent for Tier 2, 74 percent for Tier 3 compared with 63 percent for the earlier evaluation sample).48

Again, this is likely due to differences in child characteristics. 

 

46 Logistic regression was used. To account for time, we controlled for the length of time between FF/CSR case opening 

and child welfare case closure or the end of the study period (for open child welfare cases). To account for child 

characteristics, we added variables to our model to account for child age, foster care history, and more. Competing risks 

survival analysis was not used to account for case status because we were interested in final placement at the end of the

child welfare case or the end of the study period, therefore, there was no need to account for censoring. 

47Logistic regression was used. Only closed FF cases were analyzed because commitments to legal and relational 

permanency were only measured at case closure. To account for time, we controlled for the time the FF case was open. 

To account for child characteristics, we added variables to our model to account for child age, foster care history, and 

more. Competing risks survival analysis was not used because these outcomes were only available for closed cases 

therefore there was no need for censoring. 

48 Malm et al., 2014. 
31

Table 7. Percentage of all closed FF cases that obtained commitments to legal and relational 

a permanency

Family Finding

Tier 1 Tier 2 Tier 3

(n=48) (n=31) (n=132)

% % %

At least one adult made a commitment to legal permanency

At least one relative made a commitment to legal permanency

At least one adult made a commitment to relational permanency

02,3

02,3

22,3

711

521

841

551

461

741

a Commitments made prior to FF case closure.

Challenges to achieving permanency outcomes. One of challenges identified throughout our evaluations 

of FF—as well as other interventions aimed at engaging family members in case planning and 

permanency outcomes for children in foster care—is the extent to which the child and family 

characteristics are barriers to success. For example, during our field work we heard how families are 

disconnected from one another and the FF specialists simply cannot locate them. Others are truly 

dysfunctional, with histories of generational child protective services involvement or other issues that 

would prevent them from being a viable placement or support option for a child (e.g., criminal 

background, substance abuse issues, etc.). Family dynamics also plays a role, where relatives do not 

want to risk disrupting their relationship with the child’s biological parents (e.g., a grandmother not 

wanting to upset her daughter by taking the grandchild). A few social workers recounted incidents 

where the biological parents would threaten family members against engaging with the FF specialist, 

thus creating roadblocks. 

Other CHS staff speculated that some family members do not understand the urgency of the child’s 

situation or how they can be a support to the child, even if they cannot be a permanent placement. 

Family members are also hesitant to become involved because of the certain characteristics, which can 

also be barriers to achieving permanency in general (e.g., problematic behaviors including delinquency, 

promiscuity, aggression; developmental delays; special physical or mental health needs). While most 

children were reported to be open to achieving permanency, some were not interested, particularly in 

adoption. According to CHS staff, the less the youth understand the importance of permanency, the less 

likely they are to achieve it. 

Another challenge to achieving permanency for children is a lack of services available to help children 

and families overcome barriers to reunification or stable placements. Depending on the county, there 

seems to be a dearth of mental health treatment services, particularly in rural counties. If parents are 

unable to access resources such as mental health or substance abuse treatment programs that are a 

part of the case plan, they are not able to reunify with their children. If children are not able to access 

services to meet their therapeutic needs, they may be less likely to get problem behaviors under control, 

thus decreasing their likelihood of finding permanency. 

While, in general, there are relatively few supports offered to relatives who are not licensed foster 

parents, during site visits DSS workers did not raise this as a particular concern. Some counties 
32

encourage relatives to become licensed, and a couple of counties even require them to be licensed. 

However, many relatives are not able to meet standard licensing requirements, such as criminal 

background checks, financial and housing requirements, or previous parenting experience (for 

therapeutic placements). All children placed with relatives are monitored by the court. Most counties 

can use discretionary funds for things like buying clothes, furniture, or car seats. All families are entitled 

to apply for public benefits (e.g., TANF or SNAP) on behalf of children living in their home, or to receive 

any child support payments for the child. A few counties reported providing relatives with a kinship 

placement guide/manual to help them navigate the child welfare system. Other counties invite relatives 

to participate in other periodic events for children in care, such as back to school drives or monthly visits 

to the foster parent food pantry. In general, across the service areas, kinship supports were thought to 

be lacking and in need of improvement but there were no known efforts underway to improve them. In 

some cases, the lack of supports may act as a barrier to achieving permanency with relatives. 

B. Child-related outcomes 

As noted above, child-related outcomes include (1) the number of family connections discovered (an 

expanded family network), (2) the child’s knowledge of their family history and their permanency 

readiness; and (3) child well-being. We present findings from our examination of each outcome below. 

Expanded family network. One way to measure whether knowledge of family increased is to examine 

the average number of connections the FF specialist discovered. Table 8 shows on average 50 

connections were discovered for each child (by closure of the FF case) and there were no significant 

differences between tiers. After accounting for time and child characteristics,49 our findings remained 

consistent. In addition, among all closed FF cases (Tiers 1-3), there were no differences between more 

and less experienced counties in terms of the number of discovered connections. Finally, the 50 

connections discovered was an increase from the number of family members discovered during the 

previous evaluation (average 34 connections).50 

Table 8. Average number of discovered connections, among closed FF cases

Family Finding

Tier 1 Tier 2 Tier 3

(n=48) (n=31) (n=132)

Number of discovered connections 49 48 54

Child knowledge of family and overall permanency readiness. As part of their practice, FF specialists 

administered an instrument (the Permanency Readiness Measure) that assesses the child’s knowledge 

 

49 OLS regression was used. Only closed FF cases were analyzed because counts of discovered connections were only 

measured at case closure. To account for time, we controlled for the time the FF case was open. To account for child 

characteristics, we added variables to our model to account for child age, foster care history, and more. Competing risks 

survival analysis was not used because it was not appropriate (because this data was only available for closed cases 

therefore there was no need for censoring, and because the outcome was not dichotomous). 

50 Malm et al., 2014. 
33

of their family history and their overall permanency readiness.51 On average, child scores on the family 

history questions from the time of the first assessment to the last assessment (average of 8 months) 

improved 17 percentage points, among closed FF cases. On average, child scores on overall permanency 

readiness improved 20 percentage points, among closed FF cases. There were no differences between 

more and less experienced counties in terms of the degree of change in knowledge of family history or 

permanency readiness. 

Well-Being. Finally, we examined child well-being for children receiving Tier 3 and CSR through 

administration of the Child and Adolescent Needs and Strengths (CANS) assessment.52 In general, CANS 

scores improved over time, though by very little.53 There were no statistically significant differences in 

the average change in score between closed Tier 3 and CSR cases. Specific to trauma symptoms, most 

children served by Tier 3 and CSR experienced no change or a slight improvement in adjusting to 

trauma. With regard to children’s medical and physical well-being, most children served by Tier 3 and 

CSR services saw scores improve or hold steady. After controlling for time and child characteristics, 

findings were consistent.54 There were no noteworthy differences between more and less experienced 

counties in terms of child well-being. 

C. Fidelity and outcomes 

We also explored whether model fidelity was associated with outcomes, by conducting bivariate and 

multivariate analyses. 

 

51 See Appendix I for a copy of this instrument. See Appendix H for a complete definition of these outcome variables. 

When measuring change in PRM scores, we only analyzed cases in which the first and latest PRM were administered at 

least 90 days apart. The Permanency Readiness Measure was only administered to children receiving Tier 3 services. We 

did preliminary work to validate the PRM. We found that higher overall PRM scores and family history scores at the 

last/latest PRM are statistically significantly associated with greater odds of getting a commitment to legal permanency 

and getting a commitment to relational permanency. No statistically significant relationships were detected with 

permanency or stepdown, but this may be due to small sample sizes. 

52 See Appendix J for a copy of this instrument. The CANS allows workers to see if a child has needs in areas such as 

health, education, culture, behavior, emotions, and more. There are six CANS domains: Life Domain Functioning, Youth 

Strengths, Acculturation, Caregiver Strengths and Needs, Youth Behavioral/Emotional Needs, Youth Risk Behaviors. For 

each item within each of the six CANS domains, workers are asked to give a child a score from 0 to 3 with 0 indicating the 

item is not problematic and 3 indicating a severe problem. A score of 2 or 3 indicates an “actionable item.” For each 

domain, we summed the number of actionable items and looked at how the child’s number of actionable items changed 

between their first and last/latest CANS. 

53 When measuring change in CANS scores, we only analyzed cases in which the first and latest CANS were administered 

at least 90 days apart. 

54 For models measuring change in trauma symptoms, logistic regression was used. For models measuring change in 

CANS domains, OLS regression was used. Competing risks survival analysis was not appropriate for these outcomes 

because the outcome was not binary, or we were more interested in change between the first and latest surveys rather 

than experiencing any improvement in survey score. To account for time for all CANS outcome measures, we controlled 

for the time between the CANS surveys. To account for child characteristics, we added variables to our model to account 

for child age, foster care history, and more. The physical health score and Caregiver Strengths and Needs score were not 

analyzed due to small sample sizes and limited variation in the outcome variable. 
34

Bivariate analyses. First, for each model component, we analyzed the percentage of Tier 3 cases that 

achieved each outcome by the number of fidelity items the case completed.55 Appendix F presents the 

results. Table 9 summarizes for which components we found statistically significant differences between 

the number of items completed and child outcomes. As shown, for many components and many 

outcomes, children with more fidelity items completed tended to have better outcomes than children 

who had fewer items completed. This was true for commitments to legal and relational permanency, as 

well as for permanency readiness. Looking across outcomes, the components for which there was the 

most consistent positive relationship between number of items completed and outcomes was postdecision-making meeting activities and child and family preparation activities. Although cases with more 

items completed tended to be associated with more positive outcomes than cases with fewer items 

completed, this was not true for achievement of legal permanency. For this outcome, the more blended 

perspectives meeting, decision-making meeting, and team member relationships items completed, the 

poorer the child’s outcomes. 

1 Table 9. Bivariate associations between components and outcomes

Achieved 

Legal 

Permanency

Commitment 

to Legal 

Permanency

Commitment 

to Relational 

Permanency

Ever 

Stepdown

Stepdown 

as Final 

Placement2

Permanency 3 Readiness

Discovery and 

engagement 

activities

Assessments

Child and family 

preparation

Blended 

perspectives -

meeting

Decision-making - meeting

Post-decisionmaking activities

Team member - relationships

+

+

+

+

+

+

+

+

+

+ +

+

+

1 The pluses and minuses represent the direction of statistically significant relationships between outcomes and the 

number of fidelity items completed within each component. 2 Among cases that experienced a stepdown.

3 We examined the association between components and whether the score on the Permanency Readiness Measure 

(PRM) held steady or improved between their first and last PRM (as opposed to whether the score worsened).

In general, the results align with our understanding of how the model works. For example, the fact that 

child outcomes varied most frequently when examining commitments to legal and relational 

permanency, as well as with permanency readiness, is consistent with the focus of FF since those 

outcomes were the ones most targeted by FF and most in the control of the FF specialist. Of particular 

interest is the finding that post-decision-making meeting activities and the child and family preparation 

 

55 This analysis was conducted on Tier 3 cases that had their FF case closed. Their child welfare case may still be open. 
35

activities, which were newly added to the Tier 3 program model, appeared to be positively associated 

with half of the outcomes. As discussed earlier, CHS’s prior implementation of FF did not fully 

implement these activities. If the implementation of these activities plays a role in achieving positive 

outcomes, this may help explain why the previous study did not find positive impacts. Finally, the 

negative relationship between child outcomes and the implementation of the blended perspectives

meeting, decision-making meeting, and team member relationships was unexpected. We have 

anecdotal evidence that, in several instances, commitments to legal permanency were obtained early in 

a case and as a result, the DSS social worker closed the FF case and led the case through to permanency

before all model components were delivered. This means these cases were implemented with less 

fidelity, yet achieved permanency. Given the small sample sizes, these cases could account for why we 

found a negative relationship between some fidelity components and achievement of permanency.

Multivariate analyses. In order to determine how the implementation of each model component is 

independently associated with outcomes, it is important to account for the fidelity with which other 

model components are implemented because some fidelity components are more closely aligned with 

one another in terms of fidelity of implementation. For example, cases in which a blended perspectives

meeting is implemented with high fidelity could be the same cases for which the decision-making 

meeting is implemented with high fidelity. While the decision-making meeting may be the activity 

influencing the outcome, the bivariate analyses alone cannot account for that. 

Accounting for child characteristics is also important because FF specialists may implement the model 

differently depending on child characteristics. For example, it may be that they work particularly 

diligently on behalf of children for whom positive outcomes may be difficult to achieve. If this occurs, a 

cursory look at the relationship between fidelity and outcomes could show that greater fidelity is 

associated with poorer outcomes. 

Thus, to determine how the implementation of each component is associated with the outcomes of 

interest, we regressed each outcome, one at a time, on the all of the component-level fidelity measures, 

while also controlling for child/case characteristics. This approach allowed us to isolate the association 

between individual components and outcomes, while also accounting for child/case characteristics. (This 

analysis was limited to Tier 3 cases with closed FF cases). When possible, we also accounted for 

time/case status.56

 

56 We carried out one regression model for each outcome, in which the outcome was regressed on each measure of 

component-level fidelity. When examining commitment to legal permanency and relational permanency, we used 

logistic regression and controlled for the time the FF case was open (competing risks survival analysis was not 

appropriate because there was no need to account for censoring since these outcomes are measured at FF closure). For 

permanency readiness, we used OLS regression and restricted the sample to cases in which the first and last PRM survey 

were at least 90 days apart and controlled for the length of time between the surveys (competing risks survival analysis 

was not appropriate because there was no need to account for censoring and because the outcome was continuous). We 

used competing risks survival analysis for the “achieved legal permanency” and “ever step down” models. We do not 

present results for the “step down as final placement” outcome since the sample sizes were too small to run the 

multivariate models presented in this section. We treated the component-level fidelity measures as continuous 

independent variables in these models. This assumes that the relationship between component fidelity and outcomes is 
36

See Table 10 for the results (additional detail is provided in Appendix G). The fidelity with which the 

model was implemented does not explain variation in obtaining a commitment to relational 

permanency, nor in experiencing a placement step down during the study period. However, greater 

fidelity in the implementation of discovery and engagement activities, assessments, and the decisionmaking meeting was associated with the achievement of legal permanency. Greater fidelity in the 

implementation of assessments and the decision-making meeting was also associated with a 

commitment to legal permanency, and the fidelity with which discovery and engagement activities and 

the decision-making meetings were implemented was positively associated with permanency readiness. 

Net of the effect of the implementation of the other components, and net of the effect of child 

characteristics, greater fidelity of activities related to team member relationships was related to a lower

likelihood of obtaining a commitment to legal permanency. Also, greater fidelity of blended perspectives 

meeting implementation was associated with a lower likelihood of achieving permanency and a lower 

likelihood of obtaining a commitment to legal permanency. Again, this may be due to some cases being 

closed by the social worker due to achieving permanency early in the case, and the FF specialist unable 

to implement the latter components of the model. 

Table 10. Multivariate associations between components and outcomes1

Achieved Commitment Commitment Ever Stepdown Permanency 

Legal to Legal to Relational Stepdown as Final Readiness

Permanency Permanency Permanency Placement

Discovery and + +

engagement 

activities

Assessments + +

Child and family 

preparation

Blended - -

perspectives

meeting

Decision-making + + +

meeting

Post-decisionmaking activities

Team member - relationships

1 These models controlled for child/case characteristics a

and when possible, time and case status.

nd the fidelity with which other components wer e delivered,

This would suggest that the relationship between the decision-making meeting and outcomes would 

remain negative (as in the bivariate results); however, multivariate models find the greater the fidelity 

with which the decision-making meeting was implemented, the better the child outcomes. This could be 

 

linear (i.e., a change from 1 to 2 fidelity items has the same relationship with an outcome as going from 4 to 5 fidelity 

items). We acknowledge that this relationship may not be linear, but we lacked sufficient sample size to conduct a more 

fine-grained analysis. 
37

a function of the strength of relationships between different model components and outcomes. As 

shown in Appendix G, the magnitude of the relationship between components and outcomes was larger 

for the decision-making meeting component than for the blended perspectives meeting component. As 

a result, a positive association was still found between the decision-making meeting component and 

positive outcomes despite some cases reaching permanency earlier in the case. 

D. Summary of findings 

The results presented in this section are descriptive and should be interpreted with caution. When 

comparing outcomes achieved between children served by different programs, we were able to account 

for some child/case characteristics, and when possible, for the length of time youth have been served, 

the fact that some cases were still open and therefore the child may still experience a positive outcome 

in the future, and that children may have exited care without experiencing the outcome of interest. 

However, many other confounding issues likely exist. For example, we know that over half of Tier 2 

cases were served in one particularly kin-friendly county and we were unable to account for that in our 

analyses due to small sample sizes. 

With that caveat in mind, our findings showed that Tier 2 cases were more likely than Tier 3 cases to 

achieve permanency, experience a stepdown, or experience a move to relative care after FF case 

opening. Tier 2 cases were also more likely than Tier 3 to obtain a commitment to legal permanency. 

Tier 2 and Tier 3 cases fared similarly well in terms of commitments to relational permanency and the 

number of discovered connections found. These findings support the notion that county of origin may 

help explain differences seen between Tier 2 and Tier 3 cases. Tier 2 cases (the majority of which come 

from one county known as being very supportive of kinship care) experienced better outcomes than Tier 

3 cases on outcomes the county caseworker ultimately controls (placements and discharges to 

permanency). On the other hand, the two tiers were comparable on the outcomes over which the FF 

specialist has more control (commitments to relational permanency obtained and connections 

discovered). This suggests that implementing FF in a more kin-friendly jurisdiction yields more positive 

results, although more research would be necessary to state this conclusively. 

Also, while Tier 2 cases achieved better outcomes, on average, than did Tier 3 cases overall, it is 

important to keep in mind that it was children traditionally less likely to experience permanency 

outcomes who were referred for Tier 3 services, and in the absence of the intervention, we would 

expect very few of the Tier 3-eligible children to achieve positive outcomes. Yet, among children who 

received Tier 3 services, 20 percent of the closed cases achieved permanency, 42 percent stepped down, 

and 16 percent moved from non-relative to relative care. Half gained a commitment to legal 

permanency, three-quarters obtained at least one commitment for relational permanency, and they 

averaged a 17 percentage point improvement in permanency readiness. While it is not possible to know 

how these children would have fared in the absence of FF, a sizeable number of the Tier 3 children 

served did experience positive outcomes, despite their elevated risk. 

This study also showed that in general, cases implemented with greater fidelity experienced better 

outcomes. The model components associated with positive outcomes are the discovery and 
38

engagement activities, assessments, and the decision-making meeting. Given the considerable effort 

aimed at ensuring the enhanced FF model was carefully designed and implemented, it is tempting to 

want to infer a causal relationship between fidelity and better outcomes. While we did control for child 

and case characteristics to the extent possible, the findings represent only an association, not a causal 

relationship. 

Children in the current study typically experienced comparable or better outcomes than did children in 

the previous study—despite the shorter observation period. This is not surprising given that, in general, 

children served during the previous evaluation displayed more characteristics associated with a lower 

likelihood of achieving a positive outcome (i.e., older and had spent more time in care prior to referral 

to services) than children served by Tier 2 and Tier 3 services in the present study. The Tier 2 children 

served in the present study presumably needed less intensive services than did children served in the 

prior study. The lower level of needs, together with the greater fidelity of services provided, may explain 

why Tier 2 cases experienced better outcomes than did cases from the previous study. Reserving the 

more intensive services for Tier 3 cases (who displayed more characteristics associated with a lower 

likelihood of positive outcomes than Tier 2 cases), and implementing the original model’s components 

more fully may help explain the somewhat better outcomes experienced by children receiving Tier 3 

services compared to children in the previous evaluation. 

In summary, we observed better outcomes, on average, among Tier 2 than among Tier 3 children, but 

this difference does not address the relative effectiveness of Tier 2 versus Tier 3 services. While we do 

not know how Tier 2 or 3 children would have fared had they not received the services, the fact that a 

substantial portion of Tier 3 children achieved positive outcomes, and Tier 2 and Tier 3 children on 

average tended to achieve better outcomes than did children in the prior study (within a shorter 

observation period), provides preliminary evidence that the enhancements to the model may have 

increased its effectiveness. In fact, the poorer outcomes observed among Tier 3 children, compared with 

Tier 2 children, does not rule out the possibility that the impact of the services for Tier 3 children may 

well be larger than that for Tier 2 children, though the opposite may be true instead. We suggest that 

readers interpret the findings from this study with a sense of very cautious optimism. Additional 

research, specifically, a subsequent rigorous evaluation, would be needed to explore the impact of FF

for children receiving Tier 2 and Tier 3 services, as well as the overall effectiveness of these services 

relative to non-FF services. 
39

IV. Discussion and Recommendations 

Since 2013 the Children’s Home Society of North Carolina (CHS) has been working to modify and 

enhance the FF model, implement the new model with fidelity, and expand their FF program across the 

state. CHS spent considerable time and effort enhancing the model in order to better affect children’s 

permanency outcomes. The purpose of this formative evaluation—initiated after the previous 

evaluation—and the partnership with Child Trends was to provide a comprehensive examination of the 

modifications and enhancements made to the model, to develop measures to assess fidelity to the 

model, and to examine the outcomes experienced by children served during the study period. Below we 

describe implications and recommendations for continued implementation and expansion of CHS’ FF

model. We also suggest additional research that may be warranted prior to a subsequent rigorous 

evaluation of the program. 

A. Referrals 

While children referred for all tiers of FF services and CSR services generally met the criteria established 

by CHS, there were challenges with maintaining a sufficient number of referrals. Intensified outreach 

efforts by CHS increased referrals resulting in a waiting list by the end of the evaluation period; 

however, some counties referred very few children during the study period, reporting they did not have 

children who met the criteria on their caseload.57 In addition, there appear to be duplication of efforts in 

some counties, either informally through routine casework practice (e.g., searching for relatives when a 

child first enters out-of-home care or when a case is first opened per Federal relative notification 

mandates), or formally, through in-house FF services. Some DSS agencies report other intensive 

permanency planning approaches (e.g., Permanency Roundtables and Permanency Planning Action 

Team Meetings) that overlap with some of the FF model components. 

CHS’ outreach generally targeted DSS administrators, but training efforts targeted all staffing levels. CHS 

might want future outreach efforts to target even more levels of staff so front-line DSS social workers 

better understand the unique contributions of the FF model to their casework. While CHS’ efforts to 

engage DSS administrators were impressive, the fact that some social workers reported feeling 

pressured by supervisors or other managers to refer children is of concern as it indicates social workers 

did not value the FF services or felt they can, and do, provide adequate similar services. 

In some states in which private agency workers provide FF services, workers are co-located within DSS 

locations. While this may not be feasible in all parts of the state, CHS might consider this possibility at 

certain locations or at certain times during the expansion process. Private provider organizations such as 

CHS often have to overcome skepticism by public agency workers when offering new services. By 

definition, FF requires collaboration between the FF specialist and DSS social worker, therefore building 

trust is essential. Embedding the FF specialists physically within the DSS agencies, even for a limited 

time, may facilitate the relationship building that is essential to the practice. In addition, CHS should 

continue to focus on the model’s new components during their outreach efforts. Ensuring DSS social 

 

57 According to CHS, counties that made few, if any, referrals during the study period were doing so by the beginning of 

2016. 
40

workers understand the importance of the new components, as well as the expertise and time needed 

to do the work, may encourage referrals as social workers will understand they lack the time and 

expertise to perform the work themselves. 

The changing structure of the contracts between CHS and DSS may also have unintentionally 

complicated the referral process and affected the pace of and/or overall number of referrals. DSS social 

workers may have been told about the CHS contract but misconstrued some of the information or 

formed misperceptions of the program that affected their referrals to the services. 

Finally, while DSS social workers expressed genuine support of family engagement practice and relative 

placements, according to CHS staff, social workers’ actions do not always reflect this support. We found 

similar discrepancies in our field work in other states. By definition, children referred for FF services 

represent a distinct subgroup of children in foster care, one for which initial efforts to engage family 

members or place the child with relatives were either unsuccessful, or not pursued. Social workers’ bias 

toward and distrust of relatives, as viewed by CHS staff, often represents their past experience with 

family members. The complex and delicate relationships DSS social workers must navigate with families 

as part of their casework, and the family information they learn along the way, is likely not entirely 

captured in electronic or written case files reviewed by FF specialists. While philosophically supportive 

of family engagement, DSS social workers may find the practical—the complicated DSS histories of 

families, or the lack of relatives’ responses to earlier requests for engagement—insurmountable, 

regardless of new information discovered by the FF specialist. In addition, lack of relative supports or 

community supports in general may also weigh heavily on DSS workers’ decisions. One recommendation 

for training would be to allow time for facilitated discussion about recent cases during which CHS and 

DSS staff disagreed on the placement decisions and to explore workers’ feelings and actions. Another 

would be to expand training efforts beyond DSS staff to include judges, GALs, and other stakeholders. 

This could help enhance support for placing children with relatives in the counties in which FF is being 

implemented. 

B. Implementation supports 

Throughout the study period, CHS implemented the new tiered FF program in a purposeful and 

supportive manner. CHS leaders supported the program by providing guidance and offering their 

expertise to FF staff. In addition, CHS’s reputation for quality services and the high degree of trust 

between DSS and CHS leadership provided a foundation for strong collaboration. The comprehensive 

internal training and support provided to the FF specialists and supervisors was impressive. In addition, 

DSS agency administrators and staff genuinely appreciated the trainings provided by CHS and requested 

additional trainings due to high turnover. 

Implementation science finds it takes between 2 and 4 years to reach full model fidelity. While CHS has 

been delivering some form of FF for more than 4 years, their experience implementing their enhanced 

model represents only 2 years. Not only did they modify and expand the model activities, they expanded 

the size of their staff and the service area tremendously. Prior to undergoing a subsequent rigorous 
41

evaluation it would benefit CHS (and the field) to be assured their FF practice is being implemented with 

full fidelity, described in more detail below. 

C. Program fidelity 

CHS, in collaboration with Child Trends, assessed model fidelity—within and across each of the 

components—throughout the study period. Extensive modifications to CHS’s case management system 

allowed for the routine entry and systematic collection of practice-specific information for assessment 

purposes. CHS carefully reviewed the data, cleaning and updating data where necessary. Child Trends 

analyzed the data, and an ongoing, collaborative approach to reviewing the findings allowed for 

productive feedback and discussion about how the enhanced model was being implemented. 

These analyses showed the earlier model components (i.e., discovery and engagement, assessments, 

blended perspective and decision-making meetings) were implemented with overall greater fidelity than 

the later components (i.e., post-decision-making activities and child and family preparation). This finding 

is not surprising given that many of the FF specialists and supervisors had several years’ experience 

conducting the former model of FF, which included these earlier components. However, FF specialists 

were able to implement the newer components of the model for some cases, demonstrating it is 

possible to implement all components of the model. Also, similar to findings from the earlier evaluation, 

the ability with which FF specialists were able to schedule and conduct productive blended perspectives 

and decision-making meetings varied. As with other family engagement models, the meeting 

components are dependent, in no small part, on the extent to which family members and other adults 

can come together on behalf of the child. Unfortunately, dysfunctional family dynamics often presented

challenges at this stage of the model and were difficult to overcome, even for the best trained FF 

specialist. One suggestion is for FF specialists and DSS social workers to explicitly discuss their roles and 

the goals of FF early in the case to ensure that the professionals involved are in agreement. 

We recommend continued monitoring of fidelity to track changes and improvements over time. Over 

the past two years, CHS has developed, installed, and scaled-up new and enhanced FF services through a 

purposeful and thoughtful approach. We believe it is important to allow sufficient time to monitor 

fidelity to determine whether, over time, fidelity can be improved and/or sustained. Continued fidelity 

monitoring will allow CHS to target additional outreach to counties, and training and consultation with 

both FF staff and DSS staff. And, while it is commendable that CHS was able to implement each of the 

model components, we recommend strengthening the training on child and family preparation and post 

decision making meeting activities given the lower fidelity with which those activities were 

implemented. 

Also, we heard from FF specialists that receiving many new cases at once can impede their ability to 

implement the model with fidelity. We recommend that CHS continue to attempt to stagger referrals to 

each caseworker, and attempt to assess the difficulty of each case so each FF specialist has a 

manageable caseload. 
42

D. Outcomes 

While the evaluation’s primary purpose was to examine how the enhanced FF model was implemented, 

we also tracked permanency and placement outcomes for the children served. We examined a variety of 

outcomes for the children served by CHS, including: achievement of legal permanency; experiencing a 

placement step down; experiencing a move from a non-relative placement to a relative home; relative 

or other adult making a commitment to legal or relational permanency; the number of family 

connections discovered; a child’s knowledge of their family history; a child’s overall permanency 

readiness; and general child well-being. We then compared outcomes experienced by children served 

during this study period to those experienced by children served during the previous evaluation. 

Compared to findings from the previous evaluation, the new findings suggest that children served by 

CHS during this study period (receiving Tier 2 and Tier 3) experienced better permanency and placement 

outcomes. This is especially noteworthy given the shorter observation period. We found no differences 

in the likelihood of step downs comparing children served by Tier 3 in this study to children in the 

previous evaluation (children served by Tier 2 were more likely to experience a step down). These 

findings should be interpreted with caution since we were not able to make a rigorous comparison 

between previous and current study findings. Further, our examination of the association between 

fidelity and outcomes found that, in general, when FF services were implemented with greater fidelity, 

children experienced better outcomes. This was particularly true for the discovery and engagement 

activities, assessments, and the decision-making meeting. These findings, too, should be interpreted 

with caution since our measures of fidelity (especially for the post decision making meeting activities) 

are limited. In addition, findings represent only an association, not a causal relationship between tier 

and outcomes or fidelity and outcomes. With these cautions in mind, these findings suggest that the 

extensive, ongoing efforts to modify and enhance the original FF model may be paying off in terms of 

better outcomes for children who generally do not experience permanency through traditional services. 

E. Framework for rigorous evaluation 

As noted above, our findings suggest that CHS’ efforts to modify and enhance the original FF model, and

to expand FF services across the state, were worthwhile. CHS and Child Trends worked collaboratively to

measure program fidelity as implementation was occurring. The methods developed to measure fidelity

and the continuous assessment of the resulting findings, provide a strong foundation upon which CHS 

can continue to improve the model and further expand its services. In addition, we collected important 

information about the referral process as part of the study. The detailed information on the types of 

children referred to each tier of FF and to CSR provides CHS with important data for modifying their 

outreach efforts. This information is also important as it tells us that additional, more targeted outreach 

efforts may be necessary in order to obtain an adequate sample for a rigorous impact study. 

Within that framework, we recommend the following smaller-scale analyses prior to conducting a 

subsequent rigorous evaluation: 

 More fully examine the referral process. Additional analyses of statewide DSS administrative 

data could inform our knowledge of the referral process, identifying counties that may not be 

, 
43

referring all eligible children and providing further information on the types of children who are 

and are not being referred to CHS for FF and CSR services. Identifying additional eligible children 

and extending the study period would increase our sample and allow for more nuanced 

analyses, including an exploration of how the relationship between fidelity and outcomes may 

vary based on child/case characteristics. Additional qualitative data collection (i.e., interviews 

and focus groups) may also be necessary to further examine the reasons DSS social workers 

decide not to refer eligible children. 

 Conduct a more comprehensive fidelity assessment. We recommend developing a more nuanced 

tool to assess fidelity that gives more weight to certain activities over others, includes measures 

of quality based on objective observation of implementation, and creates measures of the 

“principles” behind the activities (e.g., belief that children thrive when placed with relatives and 

that every effort should be made to place children with relatives); make enhancements to CHS’s 

case management system to promote better data quality; further analyze the timing of activities 

and association between timing and outcomes; examine whether there is an association 

between caseload size and fidelity and outcomes; and explore how fidelity may vary based on FF 

specialist and county. 

 Allow for more time for children to experience outcomes. This study’s observation period was 

two years shorter than the previous evaluation’s observation period. We recommend re-running 

our outcome analyses, using updated administrative data (requesting an August 2016 extract 

will expand the observation period by six months). Updated analyses will provide more accurate 

information on children’s final permanency outcomes. We know from our evaluation of the 

Wendy’s Wonderful Kids program that less than two thirds of the children adopted, experienced 

the outcome within two years following referral to services. 

 Conduct a quasi-experimental design. We recommend using propensity score matching to 

construct a comparison group of children served across the participating DSS agencies over the 

same time period and compare outcomes for this group of children with children served by Tier 

2 and Tier 3. 

The additional analyses can continue to inform CHS as they strive to improve implementation fidelity 

and expand the practice across the state. Once the practice is fully implemented as intended, more 

rigorous measures of fidelity are constructed and tested, and there is a better understanding of how the 

services translate into outcomes, CHS will be well-poised for a subsequent rigorous evaluation. A second 

randomized controlled trial will more clearly demonstrate whether the enhanced model of FF has a 

positive impact on children’s permanency outcomes. 
44

References

Child Welfare Information Gateway. (n.d.). Lifebooks. U.S. Department of Health & Human Services, 

Administration for Children & Families, Children’s Bureau. Retrieved from 

https://www.childwelfare.gov/topics/adoption/postplacement/lifebooks/

Duncan, D.F., Kum, H.C., Flair, K.A., Stewart, C.J., Vaughn, J.S., Guest, S., Rose, R.A., Gwaltney, A.Y., and 

Gogan, H.C. (2016). Management Assistance for Child Welfare, Work First, and Food & Nutrition 

Services in North Carolina (v3.2). Retrieved from http://ssw.unc.edu/ma/

Henry, D.L. (2007). 3-5-7 Model: A Practice Approach for Permanency Work with Children & Youth 

[PowerPoint slides]. Retrieved from http://74.81.204.52/Files/3-5-7Model.pdf

Malm, K., Vandivere, S., Allen, T., Williams, S.C., & McKlindon, A. (2014). A Rigorous Evaluation of Family

Finding in North Carolina. Bethesda, MD: Child Trends.

National Institute for Permanent Family Connectedness. (n.d.). Six Steps for Family Finding. Retrieved 

from http://www.familyfinding.org/assets/files/FF%20Six%20Steps.doc

The Annie E. Casey Foundation. (2016). KIDS COUNT Data Center. Retrieved from 

http://datacenter.kidscount.org/data/tables/6247-children-in-foster-care-by-placementtype?loc=35&loct=2#detailed/2/35/false/36,868,867,133,38/2623,2620,2622,2625,2624,2626,2

621/12994,12995