Anchor

Scaling the Birth and Beyond (B&B) 

intervention: Insights from the experiences 

of the Child Abuse Prevention Council 

(CAPC) 

June 15, 2020 

Katie Eddins, Mary Anne Anderson, Karen Needels, and Scott Richman

Submitted to:

Corporation for National and Community Service 

Office of Research and Evaluation 

250 E Street SW

Washington, DC 20525

Project Officer: Anthony Nerino 

Contract Number: GS10F0050L/CNSHQ16F0049

Submitted by:

Mathematica 

1100 First Street, NE, 12th Floor 

Washington, DC 20002-4221 

Phone: 202-484-9220 

Project Director: Scott Richman

Reference Number: 50332
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

iii

CONTENTS 

EXECUTIVE SUMMARY .............................................................................................................................vii

Research questions, site selection, and data collection methods................................................vii

Overview of the B&B intervention..................................................................................................ix

Key findings from this case study report ....................................................................................... x

I. INTRODUCTION................................................................................................................................... 1

A. Overview of the SEBM process study ........................................................................................... 2

B. Overview of the Birth and Beyond intervention ............................................................................. 4

II. HOW DID CAPC AND ITS COLLABORATIVE PARTNERS DEFINE AND 

OPERATIONALIZE SCALING? ............................................................................................................ 9

A. Replicating B&B........................................................................................................................... 10

B. Adapting B&B .............................................................................................................................. 11

III. HOW DID CAPC AND ITS COLLABORATIVE PARTNERS SCALE B&B? ....................................... 15

A. Organizational workforce............................................................................................................. 15

1. Approach to structuring and training the workforce............................................................. 15

2. Facilitators and challenges to structuring and training the workforce ................................. 19

B. Systems to monitor implementation and facilitate communication ............................................. 21

1. Monitoring implementation .................................................................................................. 22

2. Communication systems ..................................................................................................... 23

C. Funding and other resources to support scaling ......................................................................... 25

1. Funding for B&B scaling ...................................................................................................... 26

2. Other resources ................................................................................................................... 28

D. Use of data systems and evaluation during scaling .................................................................... 30

1. Data systems ....................................................................................................................... 30

2. Use of evaluation................................................................................................................. 32

IV. SUMMARY OF KEY FINDINGS ......................................................................................................... 36

REFERENCES............................................................................................................................................ 40

APPENDIX: PROCESS STUDY METHODOLOGY ................................................................................... 44
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

v 

EXHIBITS 

1 Family Resource Center personnel structure for implementing Birth and Beyond, as of 

September 2019............................................................................................................................. 17

A.1 Scaling readiness framework developed under the SEBM project................................................ 46

A.2 Characteristics of those interviewed .............................................................................................. 51
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

vii

EXECUTIVE SUMMARY

Scaling interventions that have demonstrated effectiveness can increase the likelihood that an 

organization’s activities will improve participants’ lives (National Implementation Research 

Network 2016). At the same time, scaling can be challenging because it goes beyond 

implementation—the focus is not only on implementing an intervention but also on reproducing 

the same effects for a larger or different population, in a new or different location, or perhaps 

while modifying some of the intervention’s components. 

The Corporation for National and Community Service (CNCS) has invested significant 

resources in supporting implementation of interventions designed to improve lives and 

strengthen communities through its AmeriCorps and Social Innovation Fund (SIF) programs. 

CNCS also invests in evaluating the effectiveness of these interventions and supporting the 

scaling of those that have evidence of being effective to serve new communities or populations. 

Recognizing that an increasing number of CNCS-funded grants were being used to scale 

interventions, CNCS contracted with Mathematica in 2016 to conduct the Scaling EvidenceBased Models (SEBM) project to deepen the agency’s understanding of the interventions and 

its knowledge base on scaling them. The project was also funded to generate systematic 

analysis on how the grantees planned to scale and what their experiences have been when 

doing so.

Using information gathered through the SEBM project’s 

process study, this report presents a case study of the 

Child Abuse Prevention Council (CAPC), a CNCS 

grantee implementing the Birth and Beyond (B&B) 

intervention in Sacramento, California. Through home 

visits or group classes in parenting education, crisis 

intervention services, and enhanced services to support 

families, B&B seeks to educate and support parents in 

order to reduce their risk of child maltreatment. This case 

study provides insights about how CAPC is scaling B&B, 

as well as the factors that appear to facilitate and hinder 

scaling.

The intervention and 

grantee

This report describes the 

scaling of Birth and Beyond, 

a parenting education and 

support intervention 

designed to reduce child 

maltreatment, by the Child 

Abuse Prevention Council of 

Sacramento and its partners. 

Research questions, site selection, and data 

collection methods

More generally, the SEBM project’s process study examined how three organizations that 

received CNCS grant funding and that were selected for in-depth investigation scaled their 

evidence-based interventions. We define evidence-based interventions as those that have been 

demonstrated, through rigorous evaluation studies, to improve participant outcomes. The 

process study focused on how grantees viewed scaling, the actions they took when they scaled, 

and what factors appeared to facilitate or challenge scaling. The process study aimed to 

address two overarching research questions: 
Scaling insights: CAPC’s experience Mathematica

viii

1. How did selected CNCS grantees define and 

operationalize scaling? For this research 

question, we describe the type of scaling that each 

grantee selected for the process study planned to 

undertake. The three types of scaling considered 

under the SEBM project are briefly defined in the 

box to the right. (See the appendix for more 

information about these definitions.)

Types of scaling

Expansion extends the 

intervention to more people in 

the same target population in 

the same location.

Replication extends the 

intervention to the same target 

population in a new location.

Adaptation extends the 

intervention to a different target 

population in either the same or 

different location or modifies the 

intervention for the same 

population in either the same or 

different location.

2. How did selected CNCS grantees scale 

evidence-based interventions? To answer this 

research question, we describe how the grantees 

selected for the process study used organizational 

and implementation supports—including the 

organizational workforce,1 systems to monitor 

implementation and facilitate communication, 

funding and other resources, and use of data 

systems and evaluation—to facilitate scaling. We 

drew from the implementation science literature 

(see box below) to identify supports that are 

typically needed. In documenting the extent to 

which grantees drew upon organizational and implementation supports, the process study 

also identified factors that appeared to facilitate and hinder scaling. 

To answer these research questions, CNCS, with input from Mathematica, selected three 

grantees that were implementing evidence-based interventions. The grantees and the 

interventions they were implementing also demonstrated a 

higher degree of scaling readiness than did other CNCS 

grantees. This meant that the grantees and interventions met 

the conditions expected to lead to successful scaling—that is, 

scaling the intervention while maintaining or exceeding the 

beneficial impacts documented in evidence about its 

effectiveness.

What is 

implementation 

science?

Implementation science is 

the scientific investigation 

of factors associated with 

effective implementation of 

an evidence-based 

intervention or practice 

(Franks and Schroeder 

2013).

Mathematica collected and analyzed data from four sources: 

(1) a review of documents relevant to each intervention and its 

scaling and supplied by the grantees or their partners, (2) a 

two-day visit during October 2018 to each grantee and local 

partners involved in scaling interventions, (3) brief telephone 

calls with grantee personnel shortly before and after the visits, 

and (4) follow-up telephone interviews conducted with grantee 

personnel about 12 months after the visits (that is, in September 2019). Information from these 

sources was compiled to identify insights about scaling that are particular to each grantee. 

1 We use the terms personnel and workforce to refer to organization and partner personnel as well as AmeriCorps 

members who deliver intervention services.
Scaling insights: CAPC’s experience Mathematica

ix

Overview of the B&B intervention

CAPC implements the B&B intervention in Sacramento, California. B&B is an evidence-based 

parenting education and support intervention designed to reduce child maltreatment. A 

countywide collaborative network of partners (called the Collaborative) developed B&B in 1998 

to address child abuse and neglect in Sacramento County. The Collaborative consists of CAPC, 

which supports the administration and operations of B&B, and six partner organizations that 

implement B&B at nine local Family Resource Centers (FRCs) throughout Sacramento County. 

Local funders of B&B also participate in some Collaborative meetings. B&B serves families with 

children ages 0 to 5 living in Sacramento County with one or more risk factors for child 

maltreatment, such as inadequate financial resources, prior history with Child Protective 

Services (CPS), or an active CPS case with a substantiated claim of child abuse or neglect.2

 

The intervention consists of four primary components: (1) one-on-one home visitations, 

conducted by home visitors for up to 55 weeks using the Nurturing Parent Program (NPP) 

curriculum3

; (2) group classes in parenting education, conducted by FRC aides for up to 16 

weeks using NPP or another curriculum; (3) crisis intervention services, delivered by an FRC 

personnel member called a crisis intervention specialist to assist families in crisis; and (4) 

enhanced services, delivered by FRC aides or home visitors to support families’ other needs.4

Home visitors and FRC aides include both FRC personnel and AmeriCorps members —people

participating in local service programs funded by CNCS who commit their time to addressing 

critical community needs through engaging in national service. In addition to home visitors and 

FRC aides, the following personnel support B&B in each FRC: (1) a program manager who

oversees B&B; (2) an FRC coordinator, who supervises the FRC aides and the crisis 

intervention specialist; (3) a team leader, who supervises home visitors; (4) a school readiness 

home visitor who provides additional support to families with children younger than age 5; and 

(5) an engagement specialist who serves in the way that best fits the site, such as supporting 

the FRC coordinator or team leader, conducting community outreach, or coordinating with other 

sites. 

CAPC administers grants on behalf of the Collaborative, which implements and supports B&B, 

including CNCS grants. CAPC and the partner agencies make all decisions about implementing 

and scaling B&B as a Collaborative; therefore, findings in this report reflect how the 

Collaborative was implementing and scaling B&B at the time of the site visit in October 2018

and the follow-up telephone interviews in September 2019. 

2 After scaling in 2015, the intervention was adapted to serve families with the same risk factors with children ages 

6 to 17. Since 2019, the intervention has been adapted to serve pregnant women.

3 Since 2019, pregnant and parenting women with children younger than 2 years old are eligible to receive an 

additional evidence-based service component called Healthy Families America (HFA) for a minimum of three 

years. As of the follow-up telephone interviews in September 2019, HFA families continue to receive the NPP 

curriculum. However, the Collaborative is replacing NPP with the Parents as Teachers (PAT) curriculum over 

time, starting with HFA families and then all families with pregnant women and children ages 0 to 5 years old. 

Families with children ages 6 to 17 years old in home visitation and families in group education classes will 

continue to receive the NPP curriculum. 

4 After scaling in 2015, the role of an existing FRC personnel member called an engagement specialist was 

expanded to provide enhanced services to families.
Scaling insights: CAPC’s experience Mathematica

x 

Key findings from this case study report

The key findings in this report pertain to two areas. First, we discuss the types of scaling that the 

Collaborative has pursued while implementing B&B in Sacramento, California. Second, we 

discuss how the Collaborative draws on organizational and implementation supports—including 

the organizational workforce, systems to monitor implementation and facilitate communication, 

funding and other resources, and use of data systems and evaluation—to scale B&B. We also 

discuss the facilitators and challenges that the Collaborative has experienced while scaling.

Approaches to scaling. In scaling B&B, the Collaborative has replicated, expanded, and 

adapted the intervention. The Collaborative replicated B&B by re-opening a site that had been 

closed during the recession in 2008 due to funding shortages. The Collaborative drew on a 

previous experience with scaling back services and decided to close one site entirely rather 

than cut back some services at all sites. In 2014, an existing funder increased available funding 

in order to re-open the site in what had become a high-need area. Also, beginning in 2013, new 

funding became available; that funding was designated for improving outcomes for African 

American families. The Collaborative used these funds to expand the number of families served 

at one existing FRC, which serves a large population of African American families.5 In addition, 

the Collaborative has adapted B&B in several ways. As of the follow-up interviews in 

September 2019, the Collaborative was adapting B&B in three ways: (1) to evolve from an 

intervention serving parents with children ages 0 to 5 to one serving parents with children ages 

0 to 17; (2) to improve its cultural responsiveness to African American families and other 

demographic groups, such as recent refugees; and (3) to incorporate an additional evidencebased approach, called Healthy Families America (HFA), into its home visitation services for 

pregnant or parenting women with children younger than age 2. Furthermore, the Collaborative 

was in the process of replacing NPP with the Parents as Teachers (PAT) curriculum, starting 

with pregnant or parenting women with children younger than age 2, as required by HFA, and 

extending to families with children ages 0 to 5 in home visitation over time. The Collaborative 

plans to continue using the NPP with families with children ages 6 to 17 in home visitation and 

in group education classes. 

Organizational workforce. Implementation and scaling of B&B is supported at three levels. 

First, at the organizational level, CAPC serves as the backbone organization of the 

Collaborative and provides supports and develops common organizational practices that 

facilitate scaling, such as centralized training for frontline personnel. Second, the executive 

directors of each of the partner organizations implementing B&B, along with key CAPC 

personnel and B&B program managers, form a B&B Management Committee that collectively 

makes decisions about implementing and scaling B&B. Grantee and frontline personnel noted 

that Collaborative leaders were often the impetus for scaling efforts. For example, respondents 

noted that leadership support was a major catalyst for the efforts to scale through the inclusion 

of the HFA approach. Third, at the FRC level, all nine FRCs share a common structure for 

personnel implementing B&B that facilitates implementation fidelity and scaling across diverse 

sites by defining personnel roles and responsibilities, while being flexible to scaling needs. For 

5 Although respondents noted that one FRC expanded B&B, we did not visit this FRC during the site visit in 

October 2018. Therefore, we do not discuss this type of scaling and the facilitators and challenges associated with 

this scaling in this report. 
Scaling insights: CAPC’s experience Mathematica

xi

example, during scaling to serve families with children ages 6 to 17, the Collaborative modified 

the engagement specialist position at all FRCs to provide services focused specifically on 

school readiness and continued success while in school.

Having common organizational practices in place across the Collaborative agencies, such as 

the uniform FRC personnel structure, facilitated scaling. Through the scaling process, the 

Collaborative learned of ways it could further strengthen these practices. For example, through 

the cultural responsiveness initiative, the Collaborative learned that creating common recruiting 

and hiring practices would serve two purposes: facilitate recruiting more diverse and culturally 

responsive personnel and formalize hiring processes that had not previously been formalized in

order to facilitate standard practices across sites. 

Systems to monitor implementation and facilitate communication. The Collaborative sets 

out the standards for implementing B&B with fidelity, which the FRCs then implement and 

monitor. The Collaborative primarily does this through providing a standard program manual to 

all FRCs, which it updates every year. The FRCs each follow a uniform implementation 

monitoring system that includes quarterly observations of home visits and group classes and 

reviews of individual cases or paperwork during supervision of frontline personnel. In addition to 

monitoring that FRC supervisors conduct, external advisors, funders, and CAPC monitor 

implementation. No respondents described formal implementation monitoring systems for two 

intervention components: crisis intervention and enhanced services, although informal 

monitoring through individual supervision does occur. 

The Collaborative has an extensive communication system consisting of committees and 

subcommittees at every personnel level that meet on specific schedules and have defined 

relationships to one another. The Collaborative’s communication system appears to support 

transparency, collaboration, shared decision making, and support for scaling efforts. For 

example, although the cultural responsiveness initiative was initiated by Collaborative leaders, 

the creation of the cultural responsiveness subcommittee appears to have helped frontline 

personnel feel a sense of involvement in and ownership of the initiative, which has contributed 

to its success. 

Funding and other resources to support scaling. Most of the scaling efforts the Collaborative 

pursued required the availability of additional funds. In some cases, funding availability 

dovetailed with a need identified by the Collaborative. For example, the grantee was aware of a 

need to extend services to families with children older than age 5, and new funding offered a 

way to do so. In other cases, funding requirements determined the scaling approach the 

Collaborative would take. For example, to qualify for funding to serve pregnant or parenting 

women with children younger than age 2, the Collaborative is incorporating the evidence-based 

HFA approach into its service delivery options to meet the funder’s standards. Over time, 

changes in funding have also required the Collaborative to scale back B&B services, as in the 

case of the replication site that was previously closed to preserve funding for other B&B sites.

CAPC serves as the central hub for materials for FRCs, including coordinating the creation and 

dissemination of Collaborative-wide materials, such as a policy and procedures program 

manual, and purchasing NPP curriculum materials for home visitation and parent education 

classes and distributing them to the FRCs. Through implementing the cultural responsiveness 
Scaling insights: CAPC’s experience Mathematica

xii

initiative, providing services to families with children ages 6 to 17, and incorporating HFA, the 

Collaborative realized that NPP might not be the best fit for all participants. As of the September 

2019 follow-up interviews, the Collaborative is in the process of replacing NPP with PAT in 

home visitation for families with children ages 0 to 5. 

Use of data systems and evaluation during scaling. At the time of the October 2018 site 

visits, the Collaborative was in the middle of transitioning data systems, using a stopgap system 

to do so. As of the follow-up interviews in September 2019, the Collaborative had decided to 

continue using the stopgap system provided by a funder. During the site visit, respondents 

described the processes, strengths, and challenges of the old system, and grantee personnel 

confirmed in the follow-up interviews that the processes are largely the same with the new 

system. The Collaborative conducts different evaluations and produces different reports for 

different funders, based on what questions are of the most interest to the funder. This presents 

a challenge for gathering a complete picture about the effectiveness of B&B and evaluating the 

effectiveness of scaling efforts. Evaluations conducted for CNCS included families served by 

AmeriCorps members only, not those served by FRC personnel. Evaluations for another funder 

focused on families with a CPS history, which is not the full range of families that B&B serves. 

Collaborative leaders also expressed a desire to conduct more formal continuous quality 

improvement activities but cited limited personnel capacity as a barrier to doing so.

External and internal evaluation influenced the scaling undertaken by the Collaborative.

Findings from an external evaluation by Sacramento County about disproportionately high rates 

of death among children in African American families in Sacramento prompted the Collaborative 

to explore this issue in its own evaluation results and program data. The Collaborative found 

disparities across demographic groups in its evaluation results and found that engagement and 

retention rates for African American families were lower than those of other groups. To better 

serve and retain this population, the Collaborative began adapting B&B to be more culturally

responsive to demographic groups, particularly African American families. External evaluation 

results also influenced the decision to transition from NPP to PAT in home visitation for families 

with children ages 0 to 5, while retaining NPP in home visitation for families with children ages 6 

to 17 and group education classes. An external evidence review found positive outcomes for the 

NPP for families with school-age children. The latest round of review did not rate the NPP for 

families with children ages 0 to 5, while other external evidence sources found positive 

outcomes for PAT for families with young children.

Conclusion. The Collaborative’s scaling of B&B reveals both successes and challenges in 

replicating, expanding, and adapting an evidence-based intervention. This report seeks to help 

stakeholders understand the factors that might facilitate and hinder scaling, based on insights 

from the experiences of one CNCS grantee scaling an evidence-based intervention. Two 

companion reports provide further insights on the scaling experiences of the other two CNCS 

grantees included in the process study. One report presents a case study of the Parent 

Possible’s scaling of the Home Instruction for Parents of Preschool Youngsters intervention, 

which seeks to provide home visitation services to help parents improve their young children’s 

development (Anderson et al. 2020). The other report presents a case study of the United Ways 

of Iowa’s scaling of the Reading Corps intervention, a standardized literacy intervention that 

provides one-on-one tutoring to students enrolled in pre-kindergarten through third grade to help 

them achieve reading proficiency (Jones et al. 2020). Additionally, a fourth report (Needels et al. 
Scaling insights: CAPC’s experience Mathematica

xiii

2020) presents a cross-grantee analysis of information collected from all three grantees; that 

report includes insights on the commonalities and differences in how grantees scaled evidencebased interventions, and the challenges and facilitators they faced while scaling.
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

1 

I. INTRODUCTION 

Scaling interventions that have demonstrated effectiveness can increase the likelihood that an 

organization’s activities will improve participants’ lives (National Implementation Research 

Network 2016). At the same time, scaling can be challenging because it goes beyond 

implementation—the focus is not only on implementing an intervention but also on reproducing 

the same effects for a larger or different population, in a new or different location, or perhaps 

while modifying some of the intervention’s components. 

The Corporation for National and Community Service 

(CNCS) has invested significant resources in supporting 

implementation of interventions designed to improve lives 

and strengthen communities through its AmeriCorps and 

Social Innovation Fund (SIF) programs.6

 CNCS also invests 

in evaluating the effectiveness of these interventions and 

supporting the scaling of those that have evidence of being 

effective to serve new communities or populations. 

Although many of the grantees that CNCS funds are for 

scaling interventions, little systematic analysis has been 

conducted on how the grants have planned to scale and what their experiences have been when 

doing so. Recognizing this, CNCS contracted with Mathematica in 2016 to conduct the Scaling 

Evidence-Based Models (SEBM) project, to deepen the agency’s understanding of interventions 

and its knowledge base on scaling them. 

The project 

The Corporation for 

National and Community 

Service is deepening its 

understanding of how to 

scale interventions deemed 

to be effective through the 

Scaling Evidence-Based 

Models project. 

The intervention and 

grantee

This report describes the 

scaling of Birth and Beyond, a 

parenting education and 

support intervention designed 

to reduce child maltreatment, 

by the Child Abuse Prevention 

Council of Sacramento and its 

partners.

This report presents a case study, using information 

from the SEBM project’s process study, of Child 

Abuse Prevention Council of Sacramento (CAPC), a 

CNCS grantee implementing the Birth and Beyond 

(B&B) intervention in Sacramento, California (see 

box at left). B&B seeks to provide parenting 

education and support to parents of children in order 

to reduce child maltreatment in Sacramento County. 

This case study provides insights about how CAPC 

and its partners are scaling B&B, and the factors that 

appear to facilitate and hinder scaling.

6 AmeriCorps supports a wide range of local service programs through grants to address critical community needs, 

such as those pertaining to education, public safety, health, and the environment. SIF supported programs from 

2010 through 2016. SIF grants were used to fund community-based programs to address challenging social 

problems communities face in the areas of economic opportunity, healthy futures and youth development. CNCS 

(2016, n.d.) provides a detailed description of these programs.
Scaling insights: CAPC’s experience Mathematica

2 

A. Overview of the SEBM process study

As part of the SEBM project, Mathematica conducted a process study examining how 

organizations that received CNCS grant funding scaled evidence-based interventions. We define 

evidence-based interventions as interventions that have been demonstrated, through rigorous 

evaluation studies, to improve participant outcomes. The process study focused on how these 

grantees view scaling, the actions they take when they scale, and what factors appeared to 

facilitate or challenge scaling. Specifically, the 

process study aimed to address two overarching 

research questions: 

1. How did selected CNCS grantees define and 

operationalize scaling? For this research 

question, we describe the type of scaling that 

each grantee planned to undertake. The three 

types of scaling considered under the SEBM 

project are briefly defined in the box to the 

right. (See the appendix for more information 

about these definitions.) 

Types of scaling

Expansion extends the intervention 

to more people in the same target 

population in the same location.

Replication extends the intervention 

to the same target population in a 

new location.

Adaptation extends the intervention 

to a different target population in 

either the same or different location 

or modifies the intervention for the 

same population in either the same 

or different location. 2. How did selected CNCS grantees scale 

evidence-based interventions? For this 

research question, we describe how the 

grantees selected for the process study used 

organizational and implementation supports to facilitate scaling. We drew from the 

implementation science literature (see box below) to identify supports that are typically 

needed. In documenting the extent to which grantees drew upon organizational and 

implementation supports, the process study also identified factors that appeared to facilitate 

and hinder scaling. 

What is implementation 

science?

Implementation science is the 

scientific investigation of factors 

associated with effective 

implementation of an evidencebased intervention or practice 

(Franks and Schroeder 2013).

To answer these research questions, CNCS, with input 

from Mathematica, selected three grantees that were 

implementing interventions with evidence of 

effectiveness, meaning that evaluation studies of those 

interventions used rigorous research designs and had 

consistently favorable findings. The grantees and the 

interventions they were implementing also, when 

compared to other CNCS grantees, demonstrated a higher 

degree of scaling readiness. This meant that the grantees 

and interventions met the conditions expected to lead to 

successful scaling—that is, scaling the intervention while 

maintaining or exceeding the beneficial impacts documented in evidence about its effectiveness. 

(See the appendix for details of the selection process and data collection). 
Scaling insights: CAPC’s experience Mathematica

3 

The grantees selected for the process study —and the interventions they were scaling—differed 

with respect to the size of the grantee implementing an evidence-based intervention, intervention 

focus areas,

7

 planned types of scaling, how long the grantee had been scaling the intervention, 

reported successes and challenges with their scaling experiences, and the extent to which the 

grantees had attempted to apply lessons learned in the past. Because CNCS did not intend for the 

grantees selected for the process study to offer interventions that were typical of all CNCS 

grantees, the insights from their scaling experiences might not apply to a broader set of CNCSfunded grantees or service providers. Still, because of their scaling readiness strengths, the 

findings from the process study can provide insights about scaling practices that can help 

stakeholders understand the conditions that might facilitate or hinder intervention scaling.

Mathematica collected and analyzed data from four sources: (1) a review of documents relevant 

to each intervention and its scaling supplied by the grantees or their partners, (2) two-day visit 

during October 2018 to each grantee and local partners involved in scaling the interventions, (3) 

brief telephone calls with grantee personnel shortly before and after the visits, and (4) 12-month 

follow-up telephone interviews conducted with grantee personnel in September 2019. 

Information from these sources was compiled to identify insights about scaling particular to each 

grantee. (See the appendix for a more detailed discussion of the data sources.) 

This report presents a case study of one of three grantees included in the process study: CAPC,

implementing the B&B intervention in Sacramento, California. We adapted the process study’s 

research questions for CAPC’s scaling of the B&B intervention in Sacramento. As the CNCS 

grantee, CAPC coordinates and supports implementation of B&B by local agencies, who provide 

B&B services at nine Family Resource Centers (FRCs) in Sacramento County.

8

 

In seeking to answer the process study’s research questions with a focus on this single grantee, 

this report describes how CAPC and its partners are scaling an evidence-based intervention, 

providing an in-depth focus on the grantee’s implementation activities. This report aims to 

deepen understanding among funders, policymakers, and service providers on CAPC’s efforts to 

scale an intervention with evidence of effectiveness. Two companion reports discuss findings 

from our case studies of the other two CNCS grantees included in the process study—Parent 

Possible, implementing the Home Instruction for Parents of Preschool Youngsters intervention 

7 Intervention focus area refers to the topics in which CNCS concentrates its funding: disaster services; economic

opportunity; education; environmental stewardship; healthy futures (including physical and mental health, 

substance abuse, and nutrition); and veterans and military families. 

8 In addition to B&B, CAPC and the partner agencies implement other programs that address child welfare and 

other social services needs of families in Sacramento. CAPC also operates an AmeriCorps mentoring program for 

foster youth, called the Youth Investment Center. Partner agencies operate a variety of other programs, from 

mental health treatment for foster youth to employment services for families. 
Scaling insights: CAPC’s experience Mathematica

4 

(Anderson et al. 2020), and the United Ways of Iowa, implementing the Reading Corps 

interventions (Jones et al. 2020).9

 

B. Overview of the Birth and Beyond intervention

B&B is an evidence-based parenting education and support intervention designed to reduce child 

maltreatment. A countywide collaborative network of partners developed B&B in 1998 to

address child abuse and neglect in Sacramento County (called the Collaborative). This 

Collaborative was convened by the Sacramento County Board of Supervisors, a local 

government office that oversees child welfare efforts in the county. The Collaborative consists of 

CAPC, which supports the administration and operations of B&B, and six partner organizations 

that implement B&B. Local funders of B&B also participate in some Collaborative meetings. 

CAPC serves as the “backbone organization” of the Collaborative by administering AmeriCorps 

and other grants to support B&B services; providing training, materials, and support to the 

partner organizations and personnel10 delivering B&B services; and coordinating program 

evaluation and monitoring program performance. The six partner organizations implement B&B 

at nine local FRCs throughout Sacramento County. The Collaborative scaled down to eight FRCs 

during the recession in 2008 and scaled back up to nine FRCs after receiving new funding in 

2014. The partner organizations implementing B&B are local nonprofits, including: 

• Folsom Cordova Community Partnership, a “one-stop shop” and FRC for parent education

and support and employment services for families in northwest Sacramento

• La Familia Counseling Center, a multicultural counseling and support services center and

FRC in central Sacramento

• Mutual Assistance Network, which operates multiservice centers, including two FRCs, with a

focus on parent education and support, youth and community engagement, and employment

in northern Sacramento

• River Oak Center for Children, primarily a behavioral health provider with locations

throughout the county and one FRC in central Sacramento

• Sacramento Children’s Home, which provides prevention and treatment programs for foster

youth and operates three FRCs in northern and central Sacramento

9 A previously published report (Needels et al. 2020) presented a cross-grantee analysis of information collected 

during the process study visits from all three grantees. The insights from this analysis pertain to two broad areas: 

(1) the approaches that grantees and their partners took to scaling—including how grantees viewed scaling and

their actions when the scaling was taking place; and (2) specific commonalities and differences in how they

scaled, and the challenges and facilitators they faced with these aspects of scaling. In contrast, the three case study

reports provide deeper insights into the scaling experiences of each of these grantees.

10 We use the terms personnel and workforce to refer to organization and partner personnel as well as AmeriCorps 

members who deliver intervention services.
Scaling insights: CAPC’s experience Mathematica

5 

• Well Space Health, a countywide health care provider that operates an FRC in a community 

health center in northern Sacramento 

The Collaborative, particularly the leaders of CAPC and the six implementing organizations, 

make collective decisions about implementing and scaling B&B. An extensive system of 

Collaborative committees and subcommittees with personnel from CAPC and the six partner 

organizations supports shared decision making. Ultimate decision-making power about B&B 

rests with Collaborative leaders who make up a B&B Management Committee, including the 

executive directors of CAPC and the six partner agencies, program manager representatives from 

the FRCs, and key CAPC personnel who support the Collaborative. 

B&B serves families with children ages 0 to 5 years old living in Sacramento County with one or 

more risk factors for child maltreatment, such as inadequate financial resources, prior history 

with Child Protective Services (CPS), or an active CPS case with a substantiated claim of child 

abuse or neglect.

11 B&B consists of four primary parent education components: 

• One-on-one home visitation to provide parenting education to participating families. These 

services are provided by FRC personnel or AmeriCorps members—individuals participating 

in local service programs funded by CNCS who commit their time to addressing critical 

community needs through engaging in national service. Members engage in terms of service, 

which specify the number of hours that they are committed to serve. Families are matched 

with a home visitor in an FRC based on their zip code of residence. Home visitation services 

start with an Adult Adolescent Parenting Inventory (AAPI) assessment. Home visitors use

this to assess risk of child maltreatment and create a Family Nurturing Plan that lays out the 

content home visitation will cover, as well as the frequency and number of visits. Parents 

then progress through a series of lessons focused on parenting competencies. Visits typically 

take place weekly over as many as 55 weeks depending on a family’s need.12 Home visitors 

use the Nurturing Parent Program (NPP) curriculum, created by an external developer, to 

deliver parent education services.13,14 The NPP curriculum includes a variety of programs for 

different populations. FRC personnel use the versions for prenatal parents; parents with 

11 After scaling in 2015, the intervention was adapted to serve families with the same risk factors with children ages 

6 to 17 years old. Since 2019, the intervention has been adapted to serve pregnant women. 

12 After scaling in 2019, families with pregnant and parenting women with children younger than 2 years old are 

eligible to receive home visits for a minimum of three years. 

13 For more information about NPP, please see https://www.nurturingparenting.com/. 

14 Since 2019, pregnant and parenting women with children younger than 2 years old are eligible to receive an 

additional evidence-based service component called Healthy Families America (HFA). As of the follow-up 

telephone interviews in September 2019, HFA families receive the NPP curriculum along with HFA. However, 

the Collaborative is replacing the NPP with the Parents as Teachers (PAT) curriculum over time, starting with 

HFA families and then all families with pregnant women and children ages 0 to 5 years old. Families with 

children ages 6 to 17 years old in home visitation and families in group education classes will continue to receive 

the NPP curriculum.
Scaling insights: CAPC’s experience Mathematica

6 

infants, toddlers and preschoolers; parents with school-age children; and a version for fathers 

only.15 

• Group classes in parenting education, conducted by FRC personnel or AmeriCorps 

members, typically take place weekly for up to 16 weeks. Families are able to attend classes 

at any FRC that is convenient for them and are not restricted by zip code. Group class 

educators, also called FRC aides, also use the NPP or another evidence-based curriculum 

(called Making Parenting a Pleasure) to deliver parenting education.16 

• Crisis intervention services, delivered by an FRC personnel member called a crisis 

intervention specialist, assist families experiencing a crisis that shifts the family’s focus away 

from effective parenting. Services consist of short-term intervention, case management, and 

referrals to outside community resources. 

• Enhanced services, typically delivered by FRC aides or home visitors, including both FRC 

personnel and AmeriCorps members, consist of referrals to and the provision of supports that 

help address families’ other needs, such as assistance related to legal, education, food, 

domestic violence, child safety, and other health concerns.17 Some enhanced services vary 

across FRCs according to participant needs.18 

Families participate in either one-one-one home visitation or group classes and receive crisis 

intervention services, enhanced services, or both as needed throughout their participation in

B&B. Families could move from home visitation to a group class, or vice versa, but do not 

participate in both at once. Families are generally able to choose whether they prefer home 

visitation or group classes, with the exception of families with open cases with CPS. Families 

with an open CPS case and a CPS caseworker cannot receive one-on-one home visitation due to 

a county legal measure to avoid the duplication of one-on-one services provided by CPS case 

workers and other county workers. As a result, families with an open CPS case can only 

participate in group parent education classes and other FRC services. Families receiving services 

other than B&B at an FRC can also access crisis intervention services and enhanced services. 

In addition to home visitors and FRC aides, the following personnel support B&B in each FRC: 

(1) a program manager who oversees B&B; (2) an FRC coordinator, who supervises the FRC 

aides and a crisis intervention specialist; (3) a team leader, who supervises home visitors; (4) a 

15 After scaling in 2015 to serve families with children ages 6 to 17 years old, the Collaborative added an NPP 

program for school-age children.

16 In follow-up interviews, grantee personnel noted PAT also has a group class curriculum, which FRCs may offer 

after staff are trained to deliver PAT. 

17 After scaling in 2015 to serve families with children ages 6 to 17 years old, school support services for children 

ages 6 to 17 are provided by an FRC personnel member called an engagement specialist, who splits his or her time 

between providing school support services and conducting administrative or other activities as needed for each 

site. School support services for children ages 0 to 5 are provided by a school readiness home visitor. 

18 The enhanced services have also changed over time according to participant needs. For example, a domestic 

violence specialist position was added at each FRC in 2014.
Scaling insights: CAPC’s experience Mathematica

7 

school readiness home visitor who provides additional support to families with children younger 

than age 5; and (5) an engagement specialist who serves in the way that best fits the site, such as 

supporting the FRC coordinator or team leader, conducting community outreach, or coordinating 

with other sites. 

Evidence of the B&B’s effectiveness in reducing child maltreatment is based on a rigorous 

research study that examined the provision of home visits by AmeriCorps members using the 

NPP curriculum, along with access to crisis intervention and enhanced services, to families with 

children ages 0 to 5 in Sacramento (Williams and de Watteville 2015).

19 The study did not 

address the effectiveness of group classes in parenting education, though in its scaling plan

submitted to CNCS, CAPC points to a large body of evidence about the effectiveness of the NPP 

curriculum, which is also used in group classes. The study found that at least eight hours of home 

visitation is the minimum point at which improvements in outcomes are detectable, which has 

led the Collaborative to strive for families to receive at least eight hours of home visits, or about 

six weekly visits (Williams and de Watteville 2015). 

In the remainder of this report, we identify the types of scaling pursued by CAPC and its partners 

(Chapter II), describe how they scale B&B and discuss the factors that appear to facilitate and 

hinder scaling (Chapter III), and summarize our findings (Chapter IV). In the appendix, we 

describe the process study’s design and the methodologies used to collect and analyze data for 

this process study. 

19 Evidence of effectiveness including the scaling types discussed in this report is discussed in the “Use of data 

systems and evaluation during scaling” section in Chapter III. 
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

9 

II. HOW DID CAPC AND ITS COLLABORATIVE PARTNERS 

DEFINE AND OPERATIONALIZE SCALING? 

CAPC, a countywide organization in Sacramento, California, administers grants on behalf of a 

Collaborative network of partners that implement and support B&B, including CNCS grants. 

CAPC and the partner agencies make all decisions about implementing and scaling B&B as a 

Collaborative; therefore, findings in this report reflect how the Collaborative was implementing 

and scaling B&B at the time of the site visit in October 2018 and the follow-up telephone 

interviews in September 2019. Where differences exist between the roles of CAPC as the grantee 

administrator and the partner agencies as the implementing entities, we discuss those 

distinctions. 

At the time of the site visit in October 2018 and the follow-up interviews in September 2019, the 

Collaborative was implementing and scaling B&B in nine FRCs throughout Sacramento County. 

CAPC personnel support the FRCs by administering the CNCS grants and other funding sources, 

providing standardized training and materials to FRC personnel, and coordinating program 

evaluation and monitoring program performance. All nine FRCs share a common structure for 

personnel implementing B&B, including a program manager who oversees B&B in that FRC, 

supervisors for home visitors, FRC aides, a crisis intervention specialist, and an engagement 

specialist position that each FRC can use as needed for its site. Home visitors, FRC aides, crisis 

intervention specialists, engagement specialists, or a combination provide B&B services to 

families at risk for child maltreatment. 

In scaling B&B, the Collaborative has replicated, expanded, and adapted the intervention. In 

the following sections, we describe specifically how the 

Collaborative carried out these three forms of scaling. In 

Chapter III, we discuss facilitators and challenges 

associated with two types of scaling—replication and 

adaptation.20 

The Collaborative replicated B&B by re-opening a site 

that had been previously closed during the recession in 

2008 due to funding shortages. In 2014, an existing 

funder increased available funding in order to re-open the 

site in what had become a high-need area. Also, 

beginning in 2013, new funding became available; that 

funding was designated for improving outcomes for 

African American families. The Collaborative used these 

Scaling B&B in 

Sacramento

In Sacramento, California, 

B&B has been replicated

(brought to new locations for 

the same target population), 

expanded (delivered to a 

larger population at the 

same site), and adapted

(modified from its original 

design to better fit the needs 

of the target population).

20 Although respondents noted that one FRC expanded B&B, we did not visit this FRC during the site visit in 

October 2018. Therefore, we do not discuss this type of scaling and the facilitators and challenges associated with 

this scaling in this report. 
Scaling insights: CAPC’s experience Mathematica

10

funds to expand the number of families served at one existing FRC, which serves a large 

population of African American residents. 

The Collaborative has adapted B&B in several ways in order to extend services to additional 

families in need and to better serve certain demographic groups. 

• First, CAPC received a CNCS grant in 2015 on behalf of the Collaborative, along with other 

funding sources, to adapt B&B from an intervention serving parents with children ages 0 to 5 

years old to one serving parents with children ages 0 to 17 years old. 

• Second, prompted by external research findings published in 2013 about disproportionately 

negative outcomes for African American families in Sacramento County (Sacramento 

County Blue Ribbon Commission 2013), the Collaborative is adapting B&B to improve its 

cultural responsiveness to African American families and other demographic groups, such as 

recent refugees. 

• Finally, at the time of the follow-up telephone interviews in September 2019, CAPC had 

received additional funding to adapt B&B to serve pregnant or parenting women with 

children younger than age 2. To qualify for this funding, the Collaborative is modifying B&B 

services for this population by incorporating an additional evidence-based approach into its 

home visitation services, called Healthy Families America (HFA). To meet HFA 

requirements, the Collaborative is transitioning from the NPP curriculum to the Parents as 

Teachers (PAT) curriculum for families receiving HFA. Over time, the Collaborative is 

planning to transition from the NPP to the PAT curriculum in home visitation for all families 

with children ages 0 to 5, based on evidence of PAT’s effectiveness for that population. 

A. Replicating B&B

The Collaborative replicated B&B at a site that had previously closed due to funding shortages. 

Collaborative leaders identified wanting to replicate to new sites in communities other than 

Sacramento as a goal for the future, but no other replication efforts have taken place as of the site 

visit in October 2018. 

During the recession of 2008, the Collaborative had to decide how to respond to funding 

shortages. In making this decision, the Collaborative drew on a previous experience with scaling 

back services and decided to close one site entirely rather than cut back some services at all sites. 

The Collaborative had previously tried cutting back the personnel and services at another site and 

found this effort unsuccessful, as overhead costs and the need for supervisory personnel 

remained the same as at a full site, which made providing services less effective and cost 

efficient. The site eventually went back to a full workforce. Therefore, in 2008, Collaborative 

leaders decided to close one site entirely to preserve the integrity of the intervention at other 

sites. A Collaborative leader described the decision this way: “During the recession when there 

had to be a decision of, ‘Okay, are we going to dummy down the model and keep everything in 

place’ or are we going to say, ‘No, the integrity of the model and what we know about it is such 
Scaling insights: CAPC’s experience Mathematica

11

that we will just not do it in as many places’...we know what’s working so why would we not do 

what’s working?” 

The replication site was chosen for closure because at the time it had the least need for B&B 

services in its neighborhood, as suggested by the site having the lowest indicators of child 

maltreatment risk of all the FRCs. After the replication site closed, indicators of child 

maltreatment risk worsened in the neighborhood, which the Collaborative continued to track. In 

2014, the Collaborative received increased funding from an existing funder, which enabled it to 

re-open the site in what was now a high-need area. 

The fact the replication site had previously closed led to distrust within the community and a 

potential challenge in recruiting participants once it was newly re-opened. To deal with this 

challenge, the FRC conducted extensive engagement and outreach efforts in the community

before starting B&B services. The organization running the site already had experience with the 

B&B intervention and its implementation. The organization hired an experienced B&B program 

manager who was familiar with the work and recruiting AmeriCorps members from the 

community with existing knowledge of the community and relationships, which also facilitated 

replication. 

Leaders at CAPC discussed replicating B&B in other communities as a future goal. They noted 

they have shared the B&B intervention with several other counties in California, but so far no 

efforts to replicate B&B in other counties have come to fruition. They identified some potential 

barriers to replication elsewhere, including the complexity of the B&B intervention, lack of 

awareness of the intervention in other counties, lack of available funding, and the lack of a 

champion for replicating B&B in other communities. 

B. Adapting B&B

As of the site visit in October 2018, the Collaborative had made two adaptations to B&B to 

extend services to additional families in need and to better serve certain demographic groups. At 

the time of the follow-up telephone interviews in September 2019, the Collaborative was in the 

process of implementing a third adaptation to better serve an additional demographic group. 

First, in 2015, the Collaborative began extending B&B services to families with children ages 6 

to 17. The Collaborative had found that families with children older than age 5 (in addition to 

children ages 0 to 5) needed support for their children. One of B&B’s major funders received an 

influx of funds to reduce foster care placements and provided B&B with additional funding to 

serve this older population, based on B&B’s history of effectively reducing child maltreatment 

(more information on this funding change is in the “Funding and other resources to support 

scaling” section in Chapter III). 

To serve this new target population of families with children ages 6 to 17, the Collaborative is 

modifying the B&B intervention in two ways. First, the Collaborative added school support 

services, in addition to NPP-based home visitation and group classes, focused specifically on 
Scaling insights: CAPC’s experience Mathematica

12

school readiness and continued success while in school for children ages 6 to 17. The 

Collaborative added these services to the duties of the engagement specialist, who also conducts 

administrative or other activities as needed for each site. Second, home visitors and FRC aides 

are adapting the existing B&B services and materials for this new population, for example, by 

creating supplementary PowerPoint files to the NPP curriculum or using videos with age-specific 

content. At the time of the site visit in October 2018, the Collaborative was considering working 

with the purveyor of the NPP to modify the curriculum and materials to better suit older youth. 

In follow-up telephone interviews in September 2019, grantee personnel noted that the 

Collaborative is no longer pursuing this as it focuses on moving away from the NPP toward the 

PAT curriculum. Collaborative leaders, supervisors, and frontline personnel expressed 

satisfaction with the ability to serve this new target population and saw this change as fulfilling a 

need in a community. However, some noted that the Collaborative was still trying to effectively 

adapt the NPP to meet the needs and situations of families with older children. Although the 

Collaborative added a version of the NPP for school-age children during scaling, frontline 

personnel noted that it does not adequately address the needs of older youth. 

The second adaptation the Collaborative is pursuing is modifying B&B to be more culturally 

responsive to African American families and other demographic groups, such as recent refugees. 

In response to the findings in 2013 from the Sacramento County Board of Supervisors about high 

rates of child abuse and infant mortality among African American families, the Collaborative 

examined internal data on engaging and retaining African American families in B&B

(Sacramento County Blue Ribbon Commission 2013). The Collaborative found the engagement 

and retention rates of African American families to be lower than those of other groups. This 

finding spurred the Collaborative to launch a cultural responsiveness initiative to better serve 

these families and other groups, including refugees and non-native English speakers. To improve 

B&B’s cultural responsiveness, the Collaborative is modifying aspects of B&B’s 

implementation, including changing workforce training, changing hiring and recruitment 

practices for AmeriCorps members and FRC personnel, and altering strategies to recruit and 

retain participants (more information on these changes is in the “Organizational workforce” and 

“Funding and other resources to support scaling” sections in Chapter III). Several frontline 

personnel, supervisors, and Collaborative leaders noted that they are starting to see successes in 

the Collaborative’s ability to engage and retain African American families. 

Finally, at the time of the follow-up telephone interviews, Collaborative leaders were 

implementing a third adaptation. CAPC received approval of funding from CalWORKs (the 

Temporary Assistance for Needy Families program in California) in January 2019 to extend 

services for pregnant or parenting women with children younger than 2 years old—the 

population eligible for CalWORKs. To qualify for this funding, the Collaborative has begun to 

embed HFA, an additional evidence-based approach, into its home visitation. HFA is an 

approach approved by the Maternal, Infant, and Early Childhood Home Visiting program and the 

Title IV-E Prevention Services Clearinghouse. 
Scaling insights: CAPC’s experience Mathematica

13

The inclusion of HFA has led to modifications to the target population and personnel structure of 

B&B. Scaling by incorporating HFA led to both planned and unplanned changes in the target 

population. Extending services to pregnant women was a planned change when the Collaborative 

decided to apply for CalWORKs funding. An unplanned change occurred after HFA 

implementation—about half of the participants referred to B&B from CalWORKs have been 

homeless, which is a substantially higher proportion than in the past. To serve homeless families, 

personnel meet with participants in any location they prefer, including an FRC, to administer

home visitation services and place more emphasis on crisis intervention services, either in 

conjunction with or prior to, home visitation services. The Collaborative created an HFA-specific 

home visitor position and an additional supervisor of home visitors. 

To support scaling with HFA, the Collaborative is modifying the B&B intervention. The 

Collaborative is transitioning from the NPP to the PAT curriculum to fulfill an HFA requirement 

to use an HFA-approved, evidence-based curriculum. The Collaborative is taking an incremental 

approach to implementing HFA and PAT, by phasing in training on these topics over time and 

extending services first to a subset of the eligible population, before extending the new 

intervention components to all eligible participants. After HFA is fully implemented, the 

Collaborative is planning to transition from the NPP to the PAT curriculum in home visitation 

for all families with children ages 0 to 5. This change will accomplish several goals, including 

maintaining compliance with HFA funding requirements, potentially opening new avenues for 

funding that require the use of certain evidence-based curricula, and being responsive to evolving 

external evidence that did not find positive outcomes for the NPP with families with young 

children. Grantee personnel also noted they believe PAT is more culturally responsive and easier 

to use in languages other than English than the NPP and will improve the cultural responsiveness 

of home visitation services. External evidence did show positive outcomes for the NPP with 

families with older children, and the Collaborative will continue to use the NPP with families 

with children ages 6 to 17 and in group education classes (more information on this change is in 

the “Organizational workforce” and “Funding and other resources to support scaling” sections in 

Chapter III). 
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

15

III. HOW DID CAPC AND ITS COLLABORATIVE PARTNERS 

SCALE B&B? 

To understand how the Collaborative supported scaling of the B&B intervention in Sacramento, 

California, we describe aspects of implementation that are identified as having key roles in 

scaling interventions. Each of these components is shown to help organizations scale 

interventions while they seek to generate the same beneficial participant outcomes that occurred 

before scaling (National Implementation Research Network n.d). Namely, we examined the 

following: 

• How the workforce helped to carry out B&B implementation 

• How grantee and partner personnel used monitoring and communication systems to support 

implementation as intended 

• The sufficiency of funding and other resources, such as materials and physical space 

• The use of data systems to monitor ongoing implementation and inform any changes that 

might need to be made, and evaluation to assess whether a scaled intervention is still 

producing the same outcomes observed in prior research

A. Organizational workforce

Engaging supportive leadership and sufficient personnel members, who have been appropriately 

trained in their duties, can support intervention scaling. Strong leaders can provide creative 

solutions to implementation problems as well as other meaningful implementation supports 

during scaling (Bernfeld 2006). Implementation science literature also suggests that specifying 

workforce characteristics, such as requirements around the types of education and experience 

that personnel should have, supports strong implementation (Fixsen et al. 2005, 2013).

Additionally, procedures to train personnel have been shown to facilitate scaling the intervention 

with fidelity, meaning the extent to which implementation of an intervention matches the 

intervention as designed (Breitenstein et al. 2010; National Implementation Research Network 

n.d.). 

1. Approach to structuring and training the workforce

At the organizational level, CAPC serves as the backbone organization of the Collaborative that 

implements and scales B&B. CAPC personnel, including the agency executive and program 

managers, support the implementation and scaling of B&B, along with other interventions that 

CAPC supports. CAPC has additional personnel who focus on B&B training, AmeriCorps grant 

management, and supporting the operations of the Collaborative. Each of the six partner 

organizations in the Collaborative has an executive director who oversees all services, including 

B&B, and participates in Collaborative decision making about implementing and scaling B&B. 

Most partner organizations operate one FRC, though two organizations operate multiple FRCs in 

different locations. 
Scaling insights: CAPC’s experience Mathematica

16

All nine FRCs share a common structure for personnel implementing B&B that facilitates 

implementation fidelity across diverse sites (Exhibit 1). Since before scaling in 2019, each FRC 

has had a program manager who oversees B&B in that site and reports to the organization 

executive. Under the program manager is (1) an FRC coordinator, who supervises the FRC aides

(who conduct group education classes) and a crisis intervention specialist; (2) a team leader, who 

supervises home visitors; (3) a school readiness home visitor who provides additional support to 

families with children younger than age 5; and (4) an engagement specialist. Each FRC can 

choose to use the engagement specialist in the way that best fits its site, such as supporting the 

FRC coordinator or team leader, conducting community outreach, or coordinating with other 

sites, in addition to providing additional school-age support for families with children ages 6 to 

17. Both FRC personnel and AmeriCorps members provide home visitation and group classes in 

parenting education. At a minimum, each FRC must have four FRC aides (one FRC personnel 

and three AmeriCorps members) and nine home visitors (five FRC personnel and four 

AmeriCorps members). After scaling in 2019 to incorporate HFA into home visitation for 

pregnant and parenting women with children younger than age 2, the Collaborative added a team 

leader at each FRC focused on HFA, for a total of two team leaders per FRC. The additional 

team leader oversees the HFA home visitors, as well as some traditional home visitors, in order 

to reduce the ratio of home visitors to supervisors. Each FRC also created three HFA-specific 

home visitor positions (two FRC personnel and one AmeriCorps member). The HFA home 

visitors tend to be experienced personnel who have worked with high-risk families in the past. At 

the time of the site visit, 105 AmeriCorps members were serving as home visitors and FRC aides 

across the nine FRCs. At the time of the follow-up telephone interviews in September 2019, the 

Collaborative was in the midst of recruiting AmeriCorps members for the upcoming program. 

The FRCs recruit AmeriCorps members and frontline personnel from the communities they serve 

in order to reflect the characteristics of families living in those communities. In addition to open 

recruitment of AmeriCorps members and agency personnel, the FRCs and CAPC have a process 

akin to a pipeline for hiring and developing personnel. CAPC also sees this process as an avenue 

to reflecting the community it serves. As one CAPC leader noted, “We like to recruit from our 

AmeriCorps members. Through our hiring process we try to mirror the programs we are 

operating.” In the most comprehensive example, a former participant could become an 

AmeriCorps home visitor, then an agency personnel home visitor, then go on to serve in a 

leadership role at an FRC or CAPC. AmeriCorps members serve for terms of AmeriCorps 

service with a certain number of assigned service hours, typically for one to two years. At the 

FRCs, AmeriCorps members generally all start in the fall and serve as a cohort until the next fall. 

Because AmeriCorps members start and end their terms of service at the same time, there is 

often high turnover each fall. 
Scaling insights: CAPC’s experience Mathematica

17

Exhibit 1. Family Resource Center personnel structure for implementing Birth and 

Beyond, as of September 2019

Notes: B&B = Birth and Beyond; FRC = Family Resource Center; HFA = Healthy Families America. 

 * Role was added or modified during scaling. 

Although FRCs have traditionally recruited frontline personnel from the communities in which 

they serve, recruiting diverse frontline personnel became an increased area of focus with 

adapting the intervention to be more culturally responsive. CAPC and the FRCs have worked to 

change recruitment and hiring practices in order to recruit a more diverse workforce. One 

supervisor noted that at the time of the site visit, the FRC had recruited the most diverse 

candidate group yet, including several African American men and Afghan-, Chinese-, Hmong-, 

and Spanish-speaking candidates. The FRC successfully recruited these personnel by pushing 

their organization’s human resources department to advertise on online job boards in addition to 

the traditional postings on agency websites and at community locations. As of the follow-up 

interviews in September 2019, the Collaborative had developed new, standardized procedures for 

recruitment and hiring to promote a more diverse workforce. These include the following: 

standard position descriptions that are welcoming to applicants from diverse backgrounds, 

including those with little academic experience; policies for the outlets in which the FRCs should 

advertise AmeriCorps and agency positions; and standard interview questions that convey the 
Scaling insights: CAPC’s experience Mathematica

18

importance of cultural responsiveness in B&B. CAPC also made changes to its internal hiring 

procedures to include a panel of personnel from the FRCs in hiring decisions for positions in 

CAPC that support the Collaborative. However, as of these interviews, grantee personnel noted it 

was too early to comment on the effectiveness of these policies because new AmeriCorps 

members and FRC personnel were still being recruited for the upcoming program year. 

Training for personnel is delivered centrally by CAPC, though FRCs can add training 

components specific to their site. Home visitors and FRC aides 

receive about 200 hours of initial training by CAPC personnel 

covering all aspects of B&B, such as the relevant curriculum, home 

visitation skills, class facilitation skills, and AmeriCorps policies.

Ongoing training for first-term personnel occurs throughout the first 

year, though it is heavily concentrated at the beginning of the term 

in the fall. In all subsequent years of service or employment, 

frontline personnel participate in a leadership academy to continue 

training, as well as to learn from one another’s experiences. FRCs 

can add trainings specific to their sites or organizations. For 

example, one FRC that is part of a mental health organization offers 

additional training to frontline personnel on mental health topics. In 

addition to formal training through CAPC and FRC-specific trainings, frontline personnel 

receive supervision from FRC supervisors or team leaders that serves a dual role of training and 

implementation monitoring. 

Promising practice 

in training

Providing training that 

builds over time as 

personnel gain more 

experience can be 

particularly beneficial 

for more experienced 

personnel.

The training system for positions other than frontline personnel is less well developed. CAPC 

provides some specific trainings for certain positions, such as team leaders and FRC 

coordinators, but other positions, such as the crisis intervention specialist, do not have positionspecific training. New supervisors attend the same training as frontline personnel and receive a 

specific training for new AmeriCorps supervisors. There is no formal ongoing training for 

supervisors, though some training occurs through Collaborative meetings that serve a dual role of 

communication and training. Monthly, the Collaborative hosts subcommittee meetings for 

personnel in specialized positions from each FRC to come together. Each meeting lasts two 

hours, with one of those hours typically reserved for a training or outside presenter. 

CAPC regularly updates the training it provides to frontline personnel and trains personnel on 

new topics as needs arise. CAPC personnel consult with the partner organizations and FRC 

personnel about any improvements or changes to the training. CAPC also regularly conducts 

surveys and other more informal data collection with frontline personnel to gather their feedback 

on training. The Collaborative uses this feedback to plan for new trainings or improve existing 

trainings in the subsequent year. For example, as of the follow-up interviews in September 2019, 

CAPC added a mental health first-aid training in response to interest from personnel. Also, as of 

September 2019, CAPC hired a training program manager who conducted an additional survey 

of frontline personnel and supervisors and was in the process of redesigning CAPC’s training 
Scaling insights: CAPC’s experience Mathematica

19

curriculum. According to grantee personnel, the new training manager is focused on redesigning 

training to focus on competency building, in addition to knowledge gain. 

While scaling, CAPC began offering training to frontline personnel on adapting B&B, including 

working with children ages 6 to 17 and integrating cultural responsiveness. As of the follow-up 

interviews in September 2019, CAPC provided training on HFA to all frontline personnel and 

supervisors. Grantee personnel noted that CAPC provided training to all personnel, not just the 

HFA home visitors, so that supervisors can support the HFA home visitors and other frontline 

personnel can learn from the HFA approach. One grantee personnel member noted that, in 

particular, HFA strategies to encourage participant retention are relevant to all frontline 

personnel and dovetail with other scaling efforts to improve the cultural responsiveness of B&B 

and increase retention of disadvantaged groups. As of the follow-up interviews in September 

2019, CAPC was preparing training on the PAT curriculum but had not yet implemented the 

training. The PAT training will first be provided to HFA home visitors and supervisors and then 

extended to other personnel. Grantee personnel noted that the decision to train personnel first on 

HFA and then in PAT was intentional, so as to foster personnel’s openness to change in the 

intervention while avoiding overwhelming them with too many changes at one time. 

2. Facilitators and challenges to structuring and training the workforce

Leadership support for change appears to facilitate scaling. The executive director of CAPC, 

along with the leaders of the other Collaborative organizations, are viewed by grantee and 

frontline personnel as strong and capable leaders for garnering funding and political support 

from other entities for B&B. Respondents noted that leadership support was a major catalyst for 

the planned efforts to scale through including HFA. Respondents described the inclusion of HFA 

as coming from the Collaborative leaders in response to funding changes. This change is viewed 

as a necessity, as the Collaborative will need to close a site if it does not incorporate this new 

intervention component and receive the anticipated new funding. One Collaborative leader noted 

that all the Collaborative leaders are flexible but also conscientious about sticking to the B&B 

intervention and will not pursue funding that does not fit in the intervention. 

Recruiting frontline personnel from the communities in which they serve is an avenue for 

scaling. Respondents noted that FRCs have been more successful in recruiting diverse frontline 

personnel since adapting B&B to be more 

culturally responsive became a Collaborative 

priority. Improving and standardizing 

recruitment and hiring practices is a key 

strategy of the Collaborative in order to 

make B&B services more culturally 

responsive to a range of demographic 

groups, particularly African American 

families. One frontline supervisor noted, “With some of the cultures that we work with, when we 

have that proper cultural match [between families and personnel], we have a better understanding 

“With some of the cultures that we work with, 

when we have that proper cultural match [between 

families and personnel], we have a better 

understanding of what [the families] have gone 

through” – Partner frontline supervisor
Scaling insights: CAPC’s experience Mathematica

20

of what they [the families] have gone through.” Without that match, it is harder for frontline 

personnel to connect with and retain families. During the follow-up interviews in September 

2019, one grantee personnel member noted that the focus on cultural responsiveness may have 

helped improve frontline personnel retention, in addition to recruitment, because personnel value 

the Collaborative’s efforts in this area. The FRCs also continue to be successful in recruiting 

frontline personnel from the pipeline of former B&B participants. As one Collaborative leader 

noted, “In recruiting AmeriCorps members, we find success in individuals who have accessed 

services, stabilized their situation, and want to give back.” 

Centralized, responsive training supports personnel while scaling. The majority of 

respondents we spoke with across all levels thought the Collaborative had a robust training 

system for frontline personnel. Most respondents found this training helpful and comprehensive. 

Particularly more experienced personnel appreciated that training continued beyond the first year 

of service or employment. The Collaborative has added trainings related to adapting B&B to be 

more culturally responsive, such as cultural proficiency, as well as trainings related to adapting 

B&B to extend services to pregnant and parenting women with child younger than 2 years old, 

including HFA and PAT. The Collaborative is incorporating HFA and PAT training gradually in 

order to make the changes more manageable for personnel. The Collaborative has also adjusted 

trainings in response to frontline personnel feedback. For example, they added a self-care portion 

into each training, which gives personnel time to process particularly difficult subjects, such as 

mandated child abuse reporting. 

Challenges with the Collaborative structure could impede scaling. Although the majority of 

respondents believed CAPC’s leadership as the backbone organization of the Collaborative was 

positive and strong, without overly dictating to the FRCs, a few respondents expressed 

perceptions that the Collaborative did not have equal partnership among the agencies. On one 

hand, some respondents felt that CAPC sees itself as able to determine policies and procedures 

on behalf of the FRCs, rather than as one member of the Collaborative that is equal to the other 

members. On the other hand, some grantee personnel thought that the FRCs see CAPC as “just 

the trainer” and as a lesser member of the Collaborative. Collaborative leaders identified 

securing a trusted leader to replace the executive director (who is near retirement age) and 

securing sustainable funding for CAPC as necessary future steps to ensure CAPC can retain its 

status as the backbone organization of the Collaborative.

Respondents reported some gaps and challenges with the centralized training system. Some 

frontline and grantee personnel found CAPC’s extensive training curriculum to be too much 

information at the beginning of the term when visitors and FRC aides do not have the context of 

real-world experience. This training regimen also presents a challenge to FRCs to serve families 

with fewer personnel during this training period. Some frontline personnel noted that although 

continued training beyond the first year is beneficial, separating new and more experienced 

personnel does not allow new personnel to learn from more experienced peers. Many

respondents found the training less robust after their first year for other personnel positions, 

including the crisis intervention specialist and supervisors. 
Scaling insights: CAPC’s experience Mathematica

21

Respondents reported implementation challenges with recruiting AmeriCorps members.

Collaborative leaders and frontline supervisors noted that the quality of applications for 

AmeriCorps positions has declined in recent years as the economy has improved since the 

recession in 2008. In the past, supervisors would see applicants with advanced degrees, but now 

they are primarily seeing applicants with a high school degree or some community college 

experience, which might mean that applicants are less well equipped to work with families in 

crisis and with high needs. As one CAPC personnel noted, “Our programs ask a lot of the 

AmeriCorps members...and so we’re looking for AmeriCorps members that can handle it.” Each 

FRC does its own recruiting, and sometimes the FRCs compete for AmeriCorps members. The 

process can be confusing to applicants and lead to duplicative human resources efforts by the 

FRCs and a missed opportunity to match applicants to the FRC where they might be needed 

most. For example, an applicant with a particular language skill might receive offers of an 

AmeriCorps position from multiple FRCs, and 

may or may not chose the FRC with the 

population that matches his or her skill set. 

Finally, the turnover of AmeriCorps members, 

along with the extensive training system for 

new personnel, can be difficult for the FRCs 

to balance. Frontline personnel noted a 

diminished capacity to deliver services at FRCs when some members end their terms and other 

members are starting training. 

“Our programs ask a lot of the AmeriCorps 

members...and so we’re looking for AmeriCorps 

members that can handle it.” 

– Grantee personnel

Challenges retaining personnel can hinder scaling. Collaborative leaders and frontline 

supervisors noted challenges retaining frontline personnel when the timing of open FRC 

personnel positions does not align with the end of AmeriCorps terms. As FRCs seek to have 

personnel who are reflective of the communities they serve, losing personnel from a specific 

cultural community is particularly detrimental to that goal. For example, when the AmeriCorps 

term of a home visitor recruited to speak a specific language or serve a specific cultural 

community ends and there is no vacant agency personnel position, the FRC loses that home 

visitor and that person’s connection to the community. When the FRC does have an open 

position, it might not be able to find a candidate who speaks the same language or has the same 

skills as the previous AmeriCorps member, potentially hindering the FRC’s ability to serve a 

specific cultural community.

B. Systems to monitor implementation and facilitate communication

In studying the conditions under which evidence-based interventions are implemented, research 

on implementation science has identified specific supports that can help to ensure an 

intervention’s fidelity, which is important to scaling. Lack of fidelity can be a reason why 

interventions might produce good outcomes when initially implemented but then fail to yield the 

same outcomes when scaled (Breitenstein et al. 2010). Robust systems that track measures 

related to fidelity and have processes in place to address challenges that arise can help ensure 

that an intervention maintains the beneficial outcomes that it produced before scaling. 
Scaling insights: CAPC’s experience Mathematica

22

According to implementation science research, a system to foster communication among 

organizational personnel, as well as personnel from partner agencies, can be another critical 

support for fidelity during scaling. Frequent communication should be maintained so that leaders 

can constructively intervene, address challenges, and strengthen implementation supports on an 

ongoing basis (Nord and Tucker 1987). Researchers have found that better adherence to 

intervention components might be related to implementing agencies demonstrating high quality 

communication between stakeholders, including well-specified channels of communication, 

common goals, and clear lines of authority (Mihalic and Irwin 2003, Fagan et al. 2008, Fagan 

and Mihalic 2003). 

1. Monitoring implementation

a. Approach to monitoring implementation 

Respondents described the role of the Collaborative, particularly the Policy and Procedure 

Subcommittee, as setting out the standards for implementing B&B with fidelity, which the FRCs 

then implement and monitor. The Policy and Procedure Subcommittee primarily does this 

through providing a standard program manual to all FRCs, which the subcommittee updates 

every year. According to our document review, the primary purpose of the program manual “is 

to ensure program fidelity through the establishment of consistent and comprehensive policies 

and procedures.” 

CAPC and FRC personnel described a uniform implementation monitoring system that largely 

occurs at the individual FRC level. Quarterly, FRC supervisors conduct joint visits with home 

visitors, observe group classes, and perform quality assurance calls with a sample of families. If 

home visitors or FRC aides need additional support, supervisors can observe more often as 

needed. After each observation, supervisors share an evaluation with frontline personnel. 

Individual FRCs perform additional monitoring during individual supervision of frontline 

personnel, for example, through reviewing individual cases or reviewing paperwork and data 

entry. Frontline and grantee personnel noted that the majority of monitoring occurs through this 

type of supervision, though frontline personnel noted that the frequency and quality of this 

supervision might vary by FRC. The primary purpose of this monitoring appears to be to ensure 

that visits were conducted as scheduled and families receive the appropriate enhanced and crisis 

intervention services. One grantee workforce member noted that team leaders can tell from the 

paperwork quality if home visitors did not attend a visit and they will check in with the family.

Team leaders also rely on families to call the FRC if a visitor does not attend a scheduled 

visitation appointment, rather than having parents sign off at each visit that the visit occurred. 

Finally, frontline personnel each have annual performance reviews with FRC supervisors during 

which they discuss their intervention delivery, among other things. 

In addition to monitoring that FRC supervisors conduct, external advisors, funders, and CAPC 

personnel monitor implementation. To ensure that families receive the necessary enhanced 

supports, FRC personnel present all cases to a multidisciplinary resource team to review the 

wraparound supports that families receive, including from other providers. Funders of B&B
Scaling insights: CAPC’s experience Mathematica

23

provide an additional level of monitoring by conducting site visits every three years to CAPC 

and each FRC to observe workshops and review case files. CAPC personnel conduct formal site 

visits to each FRC every three years to monitor AmeriCorps adherence to policies and fidelity to 

the procedures and intervention. Site visits include interviews with the FRC leadership team, 

interviews with all AmeriCorps members in the FRC, observations of one AmeriCorps member 

in each position (home visitor and FRC aide), and reviews of AmeriCorps member files. CAPC 

modeled the schedule and interview tool for AmeriCorps site visits on the schedule and tools 

funders use in their monitoring visits. Grantee personnel noted they are in constant 

communication with the FRCs, and therefore they do not find it necessary to conduct monitoring 

visits more frequently. 

b. Facilitators and challenges to monitoring implementation

Gaps in implementation monitoring could hinder implementation fidelity while scaling. No 

respondents described formal implementation monitoring systems for crisis intervention and 

enhanced services, such as supervisors’ observations of home visitation and group classes or 

CAPC’s observations of and interviews with AmeriCorps members. Personnel who provide these 

services do receive some informal monitoring during individual supervision and advice on 

services to provide through presentations to the multidisciplinary resources teams. As discussed 

above, implementation monitoring systems are important to ensure fidelity to an intervention as 

it is implemented and scaled. In the absence of such monitoring systems for crisis intervention 

and enhanced services, there could be differences between how those intervention components 

are intended to be implemented and how they are implemented in practice, which could affect 

whether the intervention consistently produces positive outcomes for all families.

2. Communication systems 

a. Approach to using communication systems 

According to Collaborative leaders and CAPC personnel, 

CAPC and its partner organizations make all decisions 

about B&B as a Collaborative, including performance 

goals, decisions on continuous quality improvement, 

reviews of data and evaluation, training improvements, 

and scaling. To support this shared decision making, the 

Collaborative has an extensive communication system 

consisting of committees and subcommittees at every 

personnel level that meet on specific schedules and have 

defined relationships to one another. There is a 

subcommittee for each common position across the FRCs 

that meet monthly, including B&B program managers, 

crisis intervention specialists, and engagement specialists, 

as well as a parent cabinet—a committee of former parent participants who provide feedback to 

the Collaborative. There are also subcommittees focused on operations that meet quarterly or as 

Promising practice 

in communication

Having regular meetings 

of frontline personnel and 

supervisors across the 

agencies implementing 

an intervention can 

facilitate communication 

and shared decision

making. 
Scaling insights: CAPC’s experience Mathematica

24

needed, such as a policies and procedures 

subcommittee and a cultural responsiveness 

subcommittee. The program manager committee 

serves as a central hub for all the subcommittees and 

includes the B&B program managers, CAPC 

personnel, and representatives from the other 

subcommittees who report out about their 

subcommittee’s activities. Finally, a B&B 

management committee provides overall leadership 

for B&B and includes executive directors from 

partner organizations, CAPC personnel, and a 

representative from the program manager committee. 

The management committee has closed and open 

sessions, which include funders. CAPC supports the 

subcommittees, such as providing a note taker for 

meetings. Individual FRCs also have their own communication systems, including team 

meetings and individual supervision between supervisors and frontline personnel. 

Trade-offs in 

communication

A balance between 

communication 

comprehensiveness and 

efficiency is important. A 

comprehensive 

communication system 

involving many perspectives 

can lead to collaboration and 

transparency but can be 

time-consuming for the 

personnel involved. 

b. Facilitators and challenges involving communication systems 

Collaborative communications support transparency, uniform intervention 

implementation, program improvement, and scaling. Collaborative leaders, grantee 

personnel, and supervisors noted collaboration and transparency as major benefits of the 

subcommittee system, including internal 

transparency and transparency with funders. 

As one CAPC workforce member 

described, “In program manager committee 

meetings, we have a closed session where 

the program managers talk about ‘family

business’ [private Collaborative 

proceedings]. Our funders are also in the 

room for the open session, which is really 

interesting to have them there. It fosters transparency on both sides, because the funders can hear 

about challenges that are arising. We can hear about things that are coming down the pipeline 

from our funders.” Internally, issues that surface in each subcommittee are brought up to B&B 

leaders and discussed collaboratively with the program manager committee, which includes 

representatives from each subcommittee. Collaborative leaders noted that close communication 

with funders ensured that funders were aware and supportive of scaling decisions. For example, 

funders were a part of the conversation about how to scale down B&B during the recession in 

2008 but allowed the Collaborative to make its own decision to close one FRC site rather than to

change the B&B intervention in all sites. 

“We’re [FRC personnel] very open with each other 

and so there’s healthy competition. We look at our 

numbers and see how we’re doing, but always 

with the idea of improving and supporting each 

other.” 

– Partner frontline personnel
Scaling insights: CAPC’s experience Mathematica

25

The Collaborative communication system also supports uniformity in program implementation 

across the nine diverse FRCs. CAPC and FRC personnel noted the main purposes of the 

subcommittee system are to foster collaboration, draw out best practices, and troubleshoot across 

partner agencies. In the past, there had been a sense of competition among partner agencies, 

which the subcommittee system has helped transform into a sense of collaboration and healthy 

competition. Frontline personnel also noted that subcommittee meetings support consistent 

implementation across sites, as personnel troubleshoot issues together and learn from what the 

other FRCs are doing. One frontline personnel noted that during subcommittee meetings “we’re 

very open with each other and so there’s healthy competition. We look at our numbers and see 

how we’re doing, but always with the idea of improving and supporting each other and so we can 

be very open if we’re experiencing challenges whether it’s with staff or with an issue.” 

Athough the subcommittee system is well defined, it is also flexible as scaling-related needs and 

other needs arise. When B&B began scaling to serve families with children ages 6 to 17, the 

policies and procedures committee met more frequently than their regularly scheduled quarterly 

meetings to support this change. The Collaborative also created a new subcommittee for school 

readiness home visitors. After the external findings about particularly negative outcomes for 

African American families were published in 2013, the Collaborative created a cultural 

responsiveness subcommittee to steer the cultural responsiveness initiative. Program leaders 

highlighted the cultural responsiveness committee’s practice of creating a yearly action plan as a 

best practice that facilitates continuous quality improvement and something they could consider 

expanding to other committees. 

Personnel time committed to Collaborative communications is a challenge. A major 

challenge with the Collaborative subcommittee system is the time commitment required for 

committee meetings, as well as the time it can take to reach decisions. As one grantee personnel 

member noted, “It’s a lot of cooks in the kitchen...like if we want to put a new policy into place, 

it takes six months.” CAPC personnel noted that although attendance at meetings is generally 

high, attendance has been a challenge at the subcommittee meetings for frontline personnel. At 

the time of the site visit, Collaborative leaders were planning to assess the subcommittee system 

and determine whether it should be restructured to reduce the time commitment of personnel 

involved. Grantee personnel reported that, as of the follow-up interviews in September 2019, no 

changes had been made to the communication system. 

C. Funding and other resources to support scaling

Providing sufficient and sustainable funding as well as other nonfinancial resources can be 

critical to intervention scaling. According to findings from implementation science research, 

providing adequate resources might be one of the most significant factors influencing 

implementation of an intervention (Wenter et al. 2002). Resources can include a range of 

supports such as funding, physical space, and intervention materials (Klingner et al. 2001, 2003; 

Coolbaugh and Hansel 2000). Organizations might want to ensure the availability of such
Scaling insights: CAPC’s experience Mathematica

26

resources well before implementation begins so that they can develop and put into place any 

needed space, equipment, and other supports (Metz and Albers 2014). 

1. Funding for B&B scaling

a. Approach to funding intervention scaling 

As the backbone organization of the Collaborative, CAPC administers funding support for B&B 

on behalf of the partner organizations implementing B&B in the FRCs. According to 

Collaborative leaders and our document review, major funding sources for implementing and 

scaling B&B include First 5 Sacramento;

21 the Sacramento County Department of Children, 

Family, and Adult Services (DCFAS), which oversees the county’s child protective services; and 

CNCS AmeriCorps grants. These three funding sources have supported B&B for many years, but 

their relative share of B&B’s funding has waxed and waned over the intervention’s 20-year 

history. B&B was once primarily funded by DCFAS and served children ages 0 to 17. Then, 

First 5 Sacramento became the largest funding source, and the B&B target population narrowed 

to children ages 0 to 5, according to First 5’s priorities. During the site visit in October 2018, 

Collaborative leaders noted that funding from First 5 Sacramento has been diminishing over 

time. In the funding cycle that was awarded before the site visit in October 2018, B&B funding 

from First 5 was cut by 15 percent, and Collaborative leaders expect future funding cycles will 

be cut as well. Collaborative leaders are looking to funding from CalWORKs, which was

approved in January 2019, to fill this gap. New funding from CalWORKs funds B&B services 

for pregnant or parenting women with children younger than age 2. In addition to looking for 

direct sources of funding to fund B&B service delivery, Collaborative leaders noted that they 

advocate to state and county governments for more funding to prevent child maltreatment in 

general, but there are no specific plans for new funding sources for B&B in addition to those 

discussed above. 

21 Each county in California has a First 5 commission, which uses money from a state tax on tobacco products to 

fund early childhood development programs for children ages 0 to 5. For more information about First 5 

Sacramento, visit https://first5sacramento.saccounty.net/About-Us/Pages/First5-PuttingKidsFirst.aspx. 
Scaling insights: CAPC’s experience Mathematica

27

b. Facilitators and challenges involving funding for intervention scaling

Changes in funding sources facilitated or necessitated 

scaling over time. According to Collaborative leaders and 

our document review, in 2015, Sacramento DCFAS, along 

with other county agencies in California, received a Title 

IV-E waiver that freed up funds traditionally used for 

foster care to be used for prevention and to reduce foster 

care placements. B&B was identified as an effective 

intervention at reducing child maltreatment, and DCFAS 

provided funding to serve families with children ages 6 to 

17 in B&B. First 5 Sacramento continues to support 

services for families with children ages 0 to 5. Support 

from DCFAS was particularly important in adding school 

support services to the engagement specialist position. 

School support services focus on families’ needs to help 

children be successful in school and allows other frontline 

personnel to focus on delivering the NPP curriculum. Funding from DCFAS has also supported 

improvement to enhanced services over time, including creating a domestic violence specialist 

position at each FRC. 

Promising practice 

in funding

Funding sources might 

change over time and 

new funders might 

require adaptations to an 

intervention. Proactively 

considering and 

preparing for such 

adaptations can help 

organizations be ready 

when new requirements 

go into effect. 

According to Collaborative leaders and grantee personnel, the Collaborative had begun planning 

to embed HFA into its home visitation to qualify for funding from CalWORKs as of the site visit 

in October 2019. As of the follow-up interviews in September 2019, the Collaborative had 

implemented additional workforce training on HFA, changed how the home visiting workforce is 

structured, and started HFA services with a subset of eligible families; it is also transitioning 

from the NPP to PAT to fulfill HFA requirements. Some Collaborative leaders noted that adding 

HFA into B&B will help address a newly identified need—to provide more training to home 

visitors serving a higher proportion of high-risk families than in the past. Collaborative leaders 

noted that they have recently seen a large growth in the percentage of families served with 

current or prior CPS involvement, who are typically at higher risk for child maltreatment. One 

collaborative leader specified that in 2008, about one-third of families had previous contact with 

CPS and that number had grown to 60 percent in the 2017–2018 program year. To help better 

support high-risk families and incorporate other HFA strategies, such as a focus on participant 

retention, the Collaborative provided HFA training to all personnel in 2019. 

In addition to transitioning from the NPP to PAT for HFA families, the Collaborative plans to 

transition to PAT for all families with children ages 0 to 5. According to grantee personnel, HFA 

and PAT are approved home visiting interventions for other potential future funding sources and 

are included in the Title IV-E Prevention Services Clearinghouse of effective child welfare 
Scaling insights: CAPC’s experience Mathematica

28

interventions.22 Using these approved evidence-based service components might make B&B 

eligible for other funding sources in the future, such as new federal funding for prevention 

services.23 A potential challenge with applying for additional federal funding, grantee personnel 

noted, is navigating restrictions on using federal funding to match AmeriCorps funding.24 One 

grantee personnel member summed up all the reasons to transition to PAT as being about 

continuous quality improvement, moving to a curriculum that is approved by funding streams 

other than AmeriCorps, implementing HFA requirements, and improving cultural sensitivity in 

B&B. 

2. Other resources

a. Approach to leveraging other resources 

CAPC provides materials for B&B, while FRCs provide physical space for B&B services not 

offered in participants’ homes. CAPC serves as the central hub for materials for Collaborative 

FRCs. CAPC coordinates the creation and dissemination of Collaborative-wide materials, such 

as a policy and procedures program manual. CAPC purchases curriculum materials for home 

visitation and parent education classes and distributes them to the FRCs. As of the follow-up 

interviews in September 2019, the Collaborative is transitioning from the NPP to PAT for home 

visitation families with children ages 0 to 5. CAPC will coordinate the purchase and distribution 

of PAT materials after the transition is complete. The Collaborative will continue to use the NPP 

in home visitation for families with children ages 6 to 17 and in group education classes, and 

CAPC will continue to coordinate those materials. Materials used for implementing or scaling 

the other components of the B&B intervention—crisis intervention services and enhanced 

services—include a resource and referral guide that is tailored to each FRC. Respondents did not 

note any standard materials used for school support services. Unlike home visitation services, 

which require few space needs because services are delivered in families’ homes, FRCs require 

classroom-like space to conduct group education classes as well as space for personnel who 

provide crisis intervention and enhanced services. Each FRC provides and adjusts this space as 

needed. After receiving funding from CalWORKs to scale by incorporating HFA, about half of 

the participants referred to B&B from CalWORKs were homeless, which is a substantially 

higher proportion than in the past. To serve homeless families, personnel meet with participants 

in any location they prefer, including an FRC, for home visitation services. Personnel also place 

22 For more information about the Title IV-E Prevention Services Clearinghouse, see 

https://preventionservices.abtsites.com/. 

23 For more information about the Family First Prevention Services Act, which authorized additional federal funding 

for the prevention of foster care placements, see Title VII of the Bipartisan Budget Act of 2018, 

https://www.congress.gov/bill/115th-congress/house-bill/1892. 

24 Information about AmeriCorps funding requirements is available in the Electronic Code of Federal Regulations:

https://www.ecfr.gov/cgi-bin/textidx?c=ecfr&sid=8dbe66bef1823e0c40bf15a79be6bde3&rgn=div5&view=text&node=45:4.1.9.11.14&idno=45#se

45.5.2521_140. 
Scaling insights: CAPC’s experience Mathematica

29

more emphasis on crisis intervention services, either in conjunction with or prior to home 

visitation services.

b. Facilitators and challenges to leveraging other resources

Adjusting physical space to scaling needs is a challenge, but external partnerships have 

facilitated needed changes. Related to scaling to serve families with children ages 6 to 17, not 

all FRCs have the ideal space to serve families with older children. For example, some FRCs 

would like to have a playground for older children in addition to their current play area for 

younger children. One FRC recently moved to a larger space to accommodate the expanded 

personnel needed to serve families with older children, as well as to be closer to the families it 

serves because its neighborhood demographics changed and the target population moved to a 

different part of the neighborhood. To facilitate this move, the Sacramento County Board of 

Supervisors assisted this FRC with moving to a community 

collaborative space with rent that is lower than market rate. 

Respondents noted that some parent education 

materials were not particularly well suited to scaling 

needs. Collaborative leaders, CAPC personnel, and 

frontline personnel noted that the NPP materials were out 

of date and not well suited to their scaling needs, both in 

terms of cultural responsiveness to diverse populations and 

in serving families with children ages 6 to 17. For example, 

one frontline personnel noted that parents have commented 

on how old the videos are that accompany the NPP 

curriculum. Other personnel noted that materials might use 

complicated terminology that is not easily understandable 

by all the populations B&B serves, including recent 

refugees. One CAPC personnel member noted, “When we 

first started implementing NPP in B&B it was a great 

evidence-based curriculum for families. As our program as evolved, it hasn’t aged with us.” 

Challenges with 

materials

Regularly monitoring 

materials can help ensure 

they are up to date and 

meet target population and 

personnel needs. If 

materials stop meeting 

these needs, personnel 

might make ad hoc 

modifications, which can be 

a challenge for scaling 

interventions, because

those modifications might 

not be evidence-based. 

To address both cultural fit and to relate content to families with older children, frontline 

personnel have been creating supplementary materials on their own, such as their own 

PowerPoint presentations to accompany lessons. Some frontline and grantee personnel noted that 

FRCs do not share these materials with one another, and frontline personnel within one FRC 

might not use the same materials. The Collaborative has also made changes to NPP materials by 

translating materials into different languages spoken by participating families. But CAPC 

personnel noted that materials might not translate well into all other languages or their 

translations have not been entirely successful. In particular, CAPC personnel noted that the 

translation of the AAPI assessment into languages other than in English might have changed the 

meaning of some questions. Ad hoc or site-specific changes to intervention-wide materials can 

present a challenge to implementing with fidelity and scaling an evidence-based intervention. 
Scaling insights: CAPC’s experience Mathematica

30

Collaborative leaders noted they are considering working with the developer of NPP to modify 

the curriculum and materials to improve their fit for diverse populations and ages. This work 

with the NPP developer had not yet taken place as of September 2019 because the Collaborative 

was focused on implementing HFA. 

During the follow-up interviews in September 2019, grantee personnel noted the Collaborative is 

moving away from the NPP to PAT for home visitation for families with children ages 0 to 5. In 

addition to meeting HFA requirements, grantee personnel believe that PAT is more culturally 

responsive and easier to use in languages other than English. The Collaborative will continue to 

use the NPP for home visitation for families with children ages 6 to 17 and in group education 

classes; therefore, challenges with the NPP and scaling needs might continue. 

D. Use of data systems and evaluation during scaling

Data systems can be used to track, measure, and store information about program 

implementation. These systems typically include a financial data collection and reporting system 

as well as a management information system to record the processes and outcomes of the 

intervention’s core components. According to findings from implementation science research, 

using data systems is critical to monitoring an intervention’s implementation because these 

systems can alert personnel as to whether changes are necessary to improve the intervention’s 

effectiveness or efficiency (National Implementation Research Network n.d.). 

In addition to drawing on data and data systems to monitor scaling in an ongoing way, 

intervention developers and organizations scaling those interventions can use evaluation to 

assess whether scaled evidence-based interventions are maintaining their intended effects—that 

is, producing the same beneficial outcomes that were produced before scaling (Bangser 2014). 

Furthermore, multiple studies of the same intervention in different scaling contexts can be used 

to generate evidence about whether the intervention can produce similarly positive results while 

being scaled across new or adapted settings and populations. 

1. Data systems 

a. Approach to using data systems 

Collaborative leaders and FRC personnel noted that at the time of the site visit in October 2018, 

the Collaborative was in the middle of transitioning data systems. The data system for B&B had 

previously been provided by the Collaborative’s evaluation partner, but the Collaborative had

stopped working with that partner recently. Collaborative leaders were in the process of 

identifying a new partner(s) for evaluation services or to provide a data system. As a stopgap 

measure, the Collaborative was using a funder’s data system (First 5 Sacramento). As of the 

follow-up interviews in September 2019, the Collaborative had decided to continue using the 

First 5 Sacramento data system at the request of First 5. This system meets the Collaborative’s 

needs for reporting to other funders in addition to First 5 and is free of charge to the 

Collaborative. CAPC hired a new data manager to provide in-house technical assistance to sites 
Scaling insights: CAPC’s experience Mathematica

31

on the new data system, monitor data to ensure milestone targets are met, work with First 5 and 

external evaluators, and facilitate a data subcommittee for the Collaborative. 

During the site visit in October 2018, personnel spoke about the processes, strengths, and 

challenges of the old data system, but personnel did not yet know what the future system would 

be, including whether they would stay with First 5’s system or find a new one. During the 

follow-up interviews in September 2019, grantee personnel noted data processes are largely the 

same with the First 5 system as with the previous system. According to CAPC and FRC 

personnel, under the old data system, all FRCs collected the same data. Home visitors or FRC 

aides collected data from families face to face, on paper. Frontline personnel collected an AAPI 

assessment pre- and post-test, as well as other enrollment paperwork. Each family has a case file 

that is updated monthly. FRCs conduct parent satisfaction surveys every other year. In some 

FRCs, data personnel take the paperwork collected by home visitors or FRC aides and enter the 

data into the data system. In other FRCs, frontline personnel enter their own data. In the old 

system, FRCs would receive monthly reports from the evaluation partner and they could pull 

their own reports to look for data issues. In sites with data personnel, that personnel member also 

monitors data. Supervisors review the data collected regularly with frontline personnel for 

accuracy and completeness. 

FRCs and Collaborative leaders primarily use data to monitor progress toward performance 

milestones they are responsible for to funders. Milestone targets include the number of families 

served, frequency of services delivered, service completion, and knowledge gain among parents 

about topics covered in home visits or group classes. At the Collaborative level, CAPC sets 

overall milestone targets by using data from evaluations and discussions with the Collaborative. 

Each FRC has the same milestone targets regardless of population size. CAPC personnel noted 

this can be a challenge for FRCs with smaller target populations. FRCs produce quarterly reports 

in which they compare their actual data to milestones and justify any differences. Respondents 

mainly described the milestones as applying to the FRCs, rather than individual personnel, 

though FRC supervisors might review milestones with frontline personnel if progress toward 

achieving a certain milestone has been slow. The Collaborative would also review data for 

continuous quality improvement, for example, to review who they are serving and how and what 

service improvements they could make. 

b. Facilitators and challenges involving use of data systems 

The data system transition presents an opportunity to streamline data collection and entry. 

Collaborative leaders were grateful that First 5 presented a temporary solution to transitioning 

away from B&B’s legacy system. Personnel now have fewer data systems to use. Before the 

transition, B&B personnel had to upload data from their legacy system to First 5’s system, which 

they no longer have to do. Collaborative leaders noted that they are using the data system 

transition, along with the addition of HFA, as opportunities to review and streamline B&B’s data 

collection and reporting. Both Collaborative leaders and frontline personnel noted that as 

paperwork has been updated over time to address different funder requirements, some data 
Scaling insights: CAPC’s experience Mathematica

32

collection is duplicative or collects outdated questions that funders no longer require. 

Collaborative leaders hope to address these issues during the transition. 

The data system transition presented challenges and learning opportunities. Before 2019, 

personnel received training on the new data system from an external trainer who was 

knowledgeable about the data system but not the specifics of B&B. As of the follow-up 

interviews in September 2019, the new data manager had provided B&B-specific data system 

training to the FRCs and provides ongoing, in-house technical assistance. According to grantee 

personnel, B&B-specific training conducted by the new data manager required a large 

investment of time from CAPC and the FRCs. However, these grantee personnel reported that 

having a data system expert in-house who understands the intervention and can “speak the same 

language” as frontline personnel has been helpful as they learn and use the new system. 

Collaborative leaders and frontline supervisors also discussed challenges with data not correctly 

matching data in the transition between the legacy database and the First 5 database. Grantee 

personnel noted that it might have been better to pilot test the system with a few sites first, rather 

than transition all at once. 

2. Use of evaluation 

a. Approach to using evaluation 

CAPC used evaluation, continuous quality improvement, and other types of research when 

scaling B&B. Collaborative leaders described reviewing the following: evaluation and outcome 

study results about B&B as they are released to the Collaborative (typically annually, but not on 

a set schedule), data submitted to funders quarterly and annually, and external data or research 

from other organizations as needed for issues that need to be addressed or changes that could 

improve the intervention. According to Collaborative leaders, the Collaborative has conducted 

evaluations of B&B since 2000 and has increased the rigor of its evaluations in response to 

requests from CNCS, including moving from a pre-post design to a quasi-experimental design 

study in 2015. As one Collaborative leader stated, “We just didn’t get on the bandwagon with 

impact evaluation because our funder said so.” As of the site visit in October 2018, the 

Collaborative had stopped working with its long-time evaluation partner. As of the follow-up 

interviews in September 2019, the Collaborative, with input from major funders, had selected a 

new evaluation partner that had subcontracted with its prior evaluation partner and thus was 

familiar with the intervention. 

The Collaborative conducts different evaluations for different funders, based on what research 

questions are of the most interest to them. For example, for CNCS, the Collaborative conducted 

an evaluation examining the impact of home visitation services provided by AmeriCorps 

members. This evaluation included families with children ages 0 to 5 because it was conducted 

before scaling to serve families with children ages 6 to 17 (Williams and de Watteville 2015). 

For DCFAS, evaluations focus on the families who have been or are currently involved with that 

agency. As of the follow-up interviews in September 2019, the Collaborative had completed an 

evaluation that included results for families with children 0 to 17 years old. The evaluation 
Scaling insights: CAPC’s experience Mathematica

33

compared rates of re-referral to CPS for B&B participants who were originally referred to B&B 

by CPS, received home visiting services between January 2015 and August 2017, and were 

served by AmeriCorps members to re-referral rates of a comparison group of randomly selected 

CPS clients not served by B&B. Overall, the evaluation found higher rates of re-referral for the

B&B treatment group; however, the authors also noted the following findings: differences in the 

background characteristics of the treatment and comparison groups were statistically significant, 

the treatment group had higher levels of risk than the comparison group, the study design cannot 

account for unobserved confounding factors, and the difference in re-referral rates among lowrisk families was not statistically significant. The evaluation also examined the dosage treatment 

participants received and the participant characteristics associated with positive outcomes

(Huang et al. 2019). As of the follow-up interviews, the Collaborative, along with its funding 

partners, is in the process of reviewing those results and using them to establish future milestone 

targets for the FRCs and identify needs for continuous quality improvement. Thus far, the 

findings confirmed for the Collaborative that it should continue to focus on increasing participant 

retention and confirmed the Collaborative’s anecdotal understanding that the participants 

referred from CPS are the most high-risk. The findings also pointed to better outcomes for 

participants with low risk factors, with younger children, and with higher dosage, which the 

Collaborative is still assessing how to incorporate into continuous quality improvement plans. 

In addition to evaluation results, Collaborative leaders and FRC supervisors noted they review 

internal program data and local community data to continuously improve B&B, though they did 

not describe the processes or frequency of this review in detail. In particular, FRC supervisors 

noted they use data to decide whom they should be serving and whether they should be offering 

different services. CAPC personnel noted they would like to focus more on continuous program 

improvement but feel limited by personnel capacity. 

External evaluation results have played a large role over time in scaling B&B. For example, 

about 10 years ago, the Collaborative worked with First 5 Sacramento to move to an evidencebased curriculum from an approach that was based on HFA but had no curriculum. The 

Collaborative identified NPP from a California evidence-based clearinghouse as having positive 

outcomes in child welfare. As of the follow-up interviews in September 2019, the Collaborative 

reincorporated the HFA approach into the intervention to satisfy new funding requirements. In 

addition to funding requirements, new external evidence led the Collaborative to decide to 

transition from the NPP to PAT for home visitation services for families with children ages 0 to 

5, while keeping the NPP for use with families with children ages 6 to 17. The California 

evidence-based clearinghouse did not rate the NPP for families with young children in its latest 

round of reviews, though it did continue to find positive outcomes for the NPP for families with 

school-age children. PAT, on the other hand, is well-rated in the Title IV-E Prevention Services 

Clearinghouse of effective child welfare interventions for families with young children. 

External data also played a large role in the creation of the Collaborative’s cultural 

responsiveness initiative. Research published by the Sacramento County Board of Supervisors in 

2013 showing different child mortality rates for African American families was a major factor in 
Scaling insights: CAPC’s experience Mathematica

34

identifying the need to better serve African American families and other groups. After the 

Collaborative received the results of this external evaluation, it confirmed this same issue was 

playing out in its own evaluation data, including that engagement and retention rates for African 

American families were lower than those of other groups. These findings prompted the creation 

of the cultural responsiveness initiative to better serve and retain African American families in 

B&B services. 

b. Facilitators and challenges involving use of evaluation 

Positive evaluation results supported continued funding 

for sustaining and scaling B&B. Collaborative leaders see 

the value in rigorous evaluations to help sustain current 

funding by continuing to show evidence-based results and

using positive results to enhance the program’s visibility and 

generate new funding. Collaborative leaders identified a 

longstanding commitment to evaluation as a strength that has 

helped B&B remain successful for so long. Being able to 

demonstrate successful evaluation results helped the 

Collaborative secure funding from DCFAS to adapt to a new 

target population. When DCFAS was distributing new funds to 

reduce foster care placements in the county, it identified B&B 

as an effective intervention at reducing child maltreatment. 

These new funds supported the adaptation of B&B to serve 

families with children ages 6 to 17. 

Lesson learned in 

evaluation

Regularly assessing 

evaluation capacity can 

help ensure capacity is in 

line with evaluation needs 

or goals. Particularly when 

working with external 

evaluation partners, it is 

important to ensure they 

have the capacity to 

conduct new types of 

evaluations when desired.

Capacity of the evaluation partner was a challenge. One challenge with the Collaborative’s 

longstanding commitment to evaluation is the Collaborative outgrew the capabilities of its longtime evaluation partner. For example, as the Collaborative raised the level of rigor in its 

evaluations and began to implement a quasi-experimental design study, it realized its evaluation 

partner did not have the capacity to support this study. At the time of the follow-up interviews in 

September 2019, grantee personnel noted the Collaborative has selected a different evaluation 

partner. The new evaluator is well-regarded by major funders and, as a subcontractor to the 

previous evaluation partner, is familiar with the B&B intervention. The new partner has 

completed one evaluation and is working with the Collaborative on future evaluation plans. 

Conducting multiple different evaluations for specific funders was a challenge for 

determining the overall effectiveness of B&B. The Collaborative conducts different 

evaluations and produces different reports for different funders, which presents a challenge for 

gathering a complete picture about the effectiveness of B&B and evaluating the effectiveness of 

scaling efforts. Evaluations conducted for CNCS included families served by AmeriCorps 

members only, not those served by FRC personnel. Evaluations for DCFAS focused on families 

with a CPS history, which is not the full range of families that B&B serves. Grantee personnel

noted that narrowing the scope of the most recent evaluation to only families with a CPS history 
Scaling insights: CAPC’s experience Mathematica

35

led to issues creating a valid comparison group and to less positive results than prior evaluations, 

but the Collaborative was limited by the interest of the funder and the outcome data the funder 

was willing to share for the comparison group. Based on this experience, the Collaborative has 

started planning for an extensive evaluation including detailed findings about other services 

provided at the FRCs, not just home visiting services. The Collaborative is involving major 

funders and other stakeholders in planning conversations to get everyone on the same page going 

forward. 
Scaling insights: CAPC’s experience Mathematica

36

IV. SUMMARY OF KEY FINDINGS 

In Sacramento, California, CAPC and its Collaborative partners are scaling an evidence-based 

parent education intervention—B&B—by replicating it in a previously closed setting, expanding 

it to reach more families, and adapting aspects of the intervention to serve additional types 

families in need and to better serve certain demographic groups. While scaling B&B, the 

Collaborative has drawn upon supports such as each organization’s workforce, implementation 

monitoring and communication systems, funding and other resources, and data systems and 

evaluation to varying degrees. In this chapter, we highlight key findings in light of what we 

learned about the scaling of B&B by the Collaborative. 

• Funding availability facilitated or drove different types of scaling. Most of the scaling 

efforts the Collaborative pursued required the availability of additional funds. In some cases, 

funding availability dovetailed with a need identified by the Collaborative. For example, the 

grantee was aware of a need to extend services to families with children older than 5 years 

old, and funding from DCFAS offered a way to do so. In other cases, funding requirements 

determined the scaling approach the Collaborative would take. For example, to qualify for 

funding from CalWORKs to serve pregnant or parenting women with children younger than 

age 2, the Collaborative incorporated an additional evidence-based component into its service 

delivery options that meets the funder’s standards. Over time, changes in funding have also 

required the Collaborative to scale back B&B services, as in the case of the replication site 

that was closed to preserve funding for other B&B sites. 

• As the developer of the intervention, the Collaborative had considerable flexibility to 

adapt the intervention. Three of the scaling approaches identified in this report involve 

modifications to the B&B intervention—creating school support services and ad hoc 

supplementary materials to the NPP for school-age children; modifying personnel and human 

resources policies to improve cultural responsiveness; and incorporating HFA, along with the 

PAT curriculum, into home visitation for pregnant and parenting women with children 

younger than age 2. The Collaborative relies on external intervention developers for some 

B&B components, including the NPP and PAT curricula and HFA. However, because it

developed B&B overall, the Collaborative has flexibility to modify the components of the 

B&B intervention that do not rely on an external intervention developer and to replace 

components that are created by an external developer. This flexibility allows the 

Collaborative to quickly respond to changing needs within its target population, to extend to 

new populations, to adapt to new evidence about the effectiveness of B&B and its 

components, and to respond to the targeted interests of new funding sources. However, this 

flexibility and the evolving nature of B&B could contribute to Collaborative leaders’ noted 

perception that B&B is a particularly complex intervention and could hinder replication in 

new locations. The evolving components of B&B can also pose a challenge for implementing 

an evidence-based intervention with fidelity, as many components have been adapted since 

they were originally evaluated. 
Scaling insights: CAPC’s experience Mathematica

37

• Structured resources aided scaling, and the scaling experience helped the Collaborative 

strengthen those resources. Common organizational resources, such as the uniform FRC 

personnel structure, centralized training, and use of standard materials including NPP and a 

program manual, facilitated scaling. Through the scaling process, the Collaborative learned 

of ways it could further strengthen these resources. For example, through the cultural 

responsiveness initiative, the Collaborative learned that creating common recruiting and 

hiring practices would serve two purposes: facilitate recruiting more diverse and culturally 

responsive personnel and formalize hiring processes that had not previously been formalized 

in order to facilitate standard practices across sites. Also, through that initiative, providing 

services to families with children ages 6 to 17, and the inclusion of HFA, the Collaborative 

realized that NPP might not be the best fit for all populations and explored options to 

improve it. As of the follow-up interviews, the Collaborative is in the process of replacing 

NPP with PAT in home visitation for families with children ages 0 to 5. 

• External and internal evaluation influenced scaling. Findings from an external evaluation 

by Sacramento County about disproportionately high child death rates among African 

American families in Sacramento prompted the Collaborative to explore this issue in its own 

evaluation results and program data. The Collaborative found disparities across demographic 

groups in its evaluation results and found that engagement and retention rates for African 

American families were lower than those of other groups. To better serve and retain this 

population, the Collaborative began adapting B&B to be more culturally responsive to 

demographic groups, particularly African American families. External evaluation results also 

influenced the decision to transition from NPP to PAT in home visitation for families with 

children ages 0 to 5, while retaining the NPP in home visitation for families with children 

ages 6 to 17 and group education classes. 

• The grantee faced constraints in conducting evaluation and continuous quality 

improvement. The Collaborative conducts different evaluations and produces different 

reports for different funders, which presents a challenge for gathering a complete picture 

about the effectiveness of B&B and evaluating the effectiveness of scaling efforts. 

Evaluations conducted for CNCS included families served by AmeriCorps members only, 

not those served by FRC personnel. Evaluations for DCFAS focused on families with a CPS 

history, which is not the full range of families that B&B serves. Collaborative leaders also 

expressed a desire to conduct more formal continuous quality improvement activities but 

cited limited personnel capacity as a barrier to doing so. 

• Leader-led initiatives that were communicated through a structure that garnered input 

and support from frontline personnel facilitated scaling. Respondents often described the 

impetus for scaling as coming from the leadership of the Collaborative. For example, 

respondents noted that leadership support was a major catalyst for scaling through the 

inclusion of the HFA approach. In addition to this “top-down” approach, the Collaborative’s 

communication system of committees and subcommittees involving all levels of personnel 

appears to support transparency, collaboration, and support for scaling efforts. For example, 

although the cultural responsiveness initiative started with Collaborative leaders reviewing 
Scaling insights: CAPC’s experience Mathematica

38

external evaluation results, the creation of the cultural responsiveness subcommittee appears 

to have helped frontline personnel feel a sense of involvement in and ownership of the 

initiative, which has contributed to its success. 

• Recruiting and retaining frontline personnel from the communities in which they serve 

facilitated scaling. Respondents noted that FRCs have been more successful in recruiting 

diverse frontline personnel since adapting B&B to be more culturally responsive became a 

Collaborative priority. Improving and standardizing recruitment and hiring practices is a key 

strategy of the Collaborative in order to make B&B services more culturally responsive to a 

range of demographic groups, particularly African American families. During the follow-up 

interviews in September 2019, one grantee personnel member noted that the focus on cultural 

responsiveness may have helped improve frontline personnel retention and recruitment, 

because personnel value the Collaborative’s efforts in this area. Providing responsive training 

is another key strategy of the Collaborative to retain frontline personnel. The Collaborative 

has added trainings related to adapting B&B to be more culturally responsive, such as 

cultural proficiency, and trainings related to adapting B&B to extend services to pregnant and 

parenting women with child younger than 2 years old, including HFA and PAT. The 

Collaborative is incorporating HFA and PAT training gradually, in order to make the changes 

more manageable for personnel and be responsive to personnel needs. The Collaborative has 

also adjusted trainings in response to frontline personnel feedback. For example, it added a 

self-care portion into each training, which gives personnel time to process particularly 

difficult subjects, such as mandated child abuse reporting. 

The Collaborative’s scaling of B&B reveals both successes and challenges in replicating, 

expanding, and adapting an evidence-based intervention. This case study report, along with two 

companion case study reports about other CNCS-funded grantees, was intended to illustrate the 

various experiences that organizations attempting scaling might face (Anderson et al. 2020; 

Jones et al. 2020). Using an implementation science lens, this report sought to help stakeholders 

understand the factors that might facilitate and hinder scaling, and the lessons learned by one 

particular grantee scaling an evidence-based intervention. 
This page intentionally blank for double-sided copying.
Scaling insights: CAPC’s experience Mathematica

40

REFERENCES 

Anderson, Mary Anne, Katie Eddins, Karen Needels, and Scott Richman. “Scaling the Home 

Instruction for Parents of Preschool Youngsters (HIPPY) intervention: Insights from the 

experiences of Parent Possible.” Report submitted to the Corporation for National and 

Community Service. Washington, DC: Mathematica, June 2020. 

Bangser, Michael. “A Funder's Guide to Using Evidence of Program Effectiveness in Scale-Up 

Decisions.” New York: MDRC and the Growth Philanthropy Network, 2014. 

Bernfeld, G. A. “The Struggle for Treatment Integrity in a ‘Dis-integrated’ Service Delivery 

System.” The Behavior Analyst Today, vol. 7, no. 2, 2006, pp. 188–205. 

doi:10.1037/h0100086. 

Blase, Karen, and Dean Fixsen. “Core Intervention Components: Identifying and 

Operationalizing What Makes Programs Work.” Washington, DC: Office of the Assistant 

Secretary for Planning and Evaluation, Office of Human Services Policy, U.S. Department 

of Health and Human Services, February 2013. 

Breitenstein, S. M., D. Gross, C. Garvey, C. Hill, L. Fogg, and B. Resnick. “Implementation 

Fidelity in Community-Based Interventions.” Research in Nursing & Health, vol. 33, no. 2, 

2010, pp. 164–173. doi:10.1002/nur.20373. 

Coolbaugh, Kathleen, and Cynthia Hansel. “The Comprehensive Strategy: Lessons Learned from 

the Pilot Sites.” OJJDP Juvenile Justice Bulletin. Washington, DC: U.S. Department of 

Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention, 

March 2000. 

Corporation for National and Community Service. “AmeriCorps State and National Policy 

Frequently Asked Questions (FAQs).” n.d. Available at

https://www.nationalservice.gov/sites/default/files/documents/AmeriCorps_State_National_

Policy_FAQs.pdf. Accessed September 5, 2019. 

Corporation for National and Community Service. “Metadata Glossary.” 2016. Available at

https://www.nationalservice.gov/sites/default/files/

documents/GLOSSARY-Metadata%20Categories%20Definitions%20for% 

20Publication.pdf. Accessed July 31, 2019. 

Fagan, A. A., K. Hanson, J.D. Hawkins, and M. W. Arthur. “Bridging Science to Practice: 

Achieving Prevention Program Implementation Fidelity in the Community Youth 

Development Study.” American Journal of Community Psychology, vol. 41, no. 3-4, June 

2008, pp. 235–249. 

Fagan, A., and S. Mihalic. “Strategies for Enhancing the Adoption of School-Based Prevention 

Programs: Lessons Learned from the Blueprints for Violence Prevention Replications of the 

Life Skills Training Program.” Journal of Community Psychology, vol. 31, no. 3, 2003, pp. 

235–253. 

Fixsen, D. L., K. Blase, K., A. Metz, and M. Van Dyke. “Statewide Implementation of EvidenceBased Programs.” Exceptional Children, vol. 79, no. 2, 2013, pp. 213–230. 
Scaling insights: CAPC’s experience Mathematica

41

Fixsen, Dean, Sandra Naoom, Karen Blase, Robert Friedman, and Frances Wallace. 

“Implementation Research: A Synthesis of the Literature.” Tampa, FL: University of South 

Florida, 2005. 

Franks, Robert, and Jennifer Schroeder. “Implementation Science: What Do We Know and 

Where Do We Go from Here?” In Applying Implementation Science in Early Childhood 

Programs and Systems, edited by Tamara Halle, Allison Metz, and Ivelisse Martinez-Beck. 

Baltimore, MD: Paul H. Brookes Publishing Company, 2013. 

Huang, Penelope, Yoonyoung Kwak, Christina Branom, and Lisa Colvig-Niclai. “Impact 

Evaluation: Birth and Beyond Home Visitation Program 2015-2017.” ASR, January 2019. 

Jones, Christopher, Daniel Friend, Mary Anne Anderson, Karen Needels, and Scott Richman. 

“Scaling the Reading Corps intervention: Insights from the experiences of United Ways of 

Iowa.” Report submitted to the Corporation for National and Community Service. 

Washington, DC: Mathematica, June 2020. 

Klingner, J., S. Ahwee, P. Pilonieta, and R. Menendez. “Barriers and Facilitators in Scaling Up 

Research-Based Practices.” Exceptional Children, vol. 69, no. 4, 2003, pp. 411–429. 

Klingner, J. K., M. E. Arguelles, M. T. Hughes, and S. Vaughn. “Examining the Schoolwide 

‘Spread’ of Research-Based Practices.” Learning Disability Quarterly, vol. 24, no. 4, 2001, 

pp. 2221–2234. 

Maxwell, N., and S.B. Richman. “Scaling Programs with Research Evidence and Effectiveness 

(SPREE).” The Foundation Review, vol. 11, no. 3, 2019, pp. 55-67. 

Metz, A., and B. Albers. “What Does It Take? How Federal Initiatives Can Support the 

Implementation of Evidence-Based Programs to Improve Outcomes for Adolescents.” 

Journal of Adolescent Health, vol. 54, no. 3, supplement, March 2014, pp. S92–S96. 

Mihalic, S., and K. Irwin. “Blueprints for Violence Prevention: From Research to Real-World 

Settings—Factors Influencing the Successful Replication of Model Programs.” Youth 

Violence and Juvenile Justice, vol. 1, no. 1, January 2003, pp. 1–23. 

National Implementation Research Network. “Active Implementation Science and Practice.” 

Chapel Hill, NC: National Implementation Research Network, October 2016. Available at

https://nirn.fpg.unc.edu/resources/brief-1-active-implementation-practice-and-science. 

Accessed November 14, 2019. 

National Implementation Research Network. (n.d.). “Implementation Drivers.” Available at 

https://nirn.fpg.unc.edu/module-2. Accessed September 19, 2019. 

Necoechea, Denise Marie. “Children At-Risk for Poor School Readiness: The Effect of an Early 

Intervention Home Visiting Program on Children and Parents.” Riverside, CA: University of 

California, Riverside, 2007. 

Needels, Karen, Daniel Friend, Scott Richman, Katie Eddins, and Christopher Jones. “Agency 

Guide for: Scaling Evidence-Based Interventions: Insights from the Experiences of Three 

Grantees.” Report submitted to the Corporation for National and Community Service. 

Oakland, CA: Mathematica, June 2020. 
Scaling insights: CAPC’s experience Mathematica

42

Nord, W. R., and S. Tucker. Implementing Routine and Radical Innovations. Lexington, MA: D. 

C. Heath and Company, 1987. 

Sacramento County Blue Ribbon Commission. “Report on Disproportionate African American 

Child Deaths.” 2013. Available at http://www.philserna.net/wpcontent/uploads/2013/05/Blue-Ribbon-Commission-Report-2013.pdf. Accessed September 

27, 2019. 

Wenter, D. L., S. T. Ennett, K. M. Ribisl, A. A. Vincus, L. Rohrbach, C. L. Ringwalt, et al. 

“Comprehensiveness of Substance Use Prevention Programs in U.S. Middle Schools.” 

Journal of Adolescent Health, vol. 30, 2002, pp. 455–446. 

Williams, Shannon, and Charles de Watteville. “AmeriCorps Impact Evaluation Sacramento 

County: Birth and Beyond Home Visitation Program 2013-2015.” Sacramento, CA: LPC 

Consulting Associates, Inc., December 2015. 
This page intentionally blank for double-sided copying. 
Scaling insights: CAPC’s experience Mathematica

44

APPENDIX: PROCESS STUDY METHODOLOGY 

A. Overview of the process study 

The Corporation for National and Community Service (CNCS) contracted with Mathematica in 

2016 to conduct the Scaling Evidence-Based Models (SEBM) project. The project is intended to 

deepen the agency’s understanding of evidence-based interventions and its knowledge base on 

scaling them. Through the SEBM project, Mathematica has (1) reviewed and evaluated research 

evidence on the effectiveness of AmeriCorps interventions that were funded in 2015 and 2016 

and Social Innovation Fund (SIF) interventions that were funded in 2010 and 2011,25 (2) 

assessed grantees’ plans for scaling, and (3) evaluated the readiness for scaling of CNCS-funded 

interventions that showed research evidence of effectiveness and recommended for further study 

the grantees and interventions that showed evidence of readiness for scaling.

As part of the SEBM project, Mathematica conducted a process study examining how three 

CNCS-funded grantees implementing interventions that Mathematica assessed to have evidence 

of effectiveness and to be ready to scale actually scaled their interventions. This process study 

was structured to identify the types of scaling that grantees undertook and describe how grantees 

drew upon organizational and implementation supports to facilitate scaling. This report presents 

a case study of one of three grantees included in the process study: the Child Abuse Prevention 

Council (CAPC) implementing the Birth and Beyond (B&B) intervention in Sacramento, 

California. 

This appendix details the scaling definitions used for the project and the scaling readiness 

framework that informed grantee selection (Section B); the methods used to conduct the process 

study, including the grantee selection process (Section C); and the methodologies used to collect 

and analyze data for the process study (Section D).

B. Defining scaling and the SEBM scaling readiness framework

To better understand how funders like CNCS and other stakeholders can foster the scaling of 

evidence-based interventions, Mathematica first operationalized the concept of scaling by 

identifying three types of scaling that can be pursued:26

25 AmeriCorps supports a wide range of local service programs through grants to address critical community needs, 

such as those pertaining to education, public safety, health, and the environment. SIF supported programs from 

2010 through 2016. SIF grants were used to fund community-based programs to address challenging social 

problems communities face in the areas of economic opportunity, healthy futures, and youth development. CNCS 

(2016, n.d.) provides a detailed description of these CNCS programs.

26 Mathematica adapted these definitions from Fixsen et al. (2005), a synthesis of implementation research 

published by the National Implementation Research Network. For example, Fixsen et al. (2005) defines 

“adaptation of the program” as “modifications that are made in a program to accommodate the context and 

requirements at an implementation site” and defines “replication” as the implementation of an intervention in new 

sites. Given these existing definitions, Mathematica defined “expansion” as the implementation of an intervention 

in the same site, with the same population, but serving more people. 
Scaling insights: CAPC’s experience Mathematica

45

• Expansion extends the intervention to more people in the same target population in the same 

location. It requires that the intervention and the organization serve a larger number of 

participants with the same service quality and in a consistent manner with the intervention’s 

design. An example of expansion would be increasing the number of unemployed adults 

served at a work center by hiring five more job search specialists who will each serve 20 

more adults. 

• Replication extends the intervention for the same target population to a new location. It 

requires the intervention and the organization maintain service quality and fidelity to the 

intervention in the new location. An example of replication would be implementing a reading 

program designed for 5th graders in a new school district, city, and state, but serving the 

same target population of 5th graders.

• Adaptation extends the intervention to a new target population. It requires that the 

organization adapt the intervention in a way that maintains service quality. An example of 

adaptation would be modifying a parent training curriculum designed for mothers to include 

language that is more inclusive of fathers. 

Scaling is considered to be successful when the intervention (1) is replicated, expanded, and/or 

adapted, and (2) maintains or surpasses its beneficial impacts for participants after the scaling 

has occurred. Drawing on these definitions as well as research from implementation science, 

Mathematica then developed for the SEBM project a framework that identifies five conditions 

that indicate whether an intervention and the organization implementing it are ready for scaling 

(Exhibit A.1). For example, the framework specifies that an intervention might be ready for 

scaling if it is well specified. In the implementation science literature, this means that the core 

elements, or set of activities that is critical for achieving beneficial outcomes for the 

intervention’s participants, are made clear and that for each core intervention element, a 

description exists of the dimensions necessary to produce the intended outcomes (Blase and 

Fixsen 2013). (A comprehensive synthesis of the implementation science literature that supports 

the scaling readiness framework is available in Maxwell and Richman 2019). 
Scaling insights: CAPC’s experience Mathematica

46

Exhibit A.1. Scaling readiness framework developed under the SEBM project

The first three conditions indicate whether an intervention might be ready to be scaled: 

• A well-specified intervention, consisting of a description of the content, mode of service 

delivery, intensity, workforce needs,27 and setting for each core element. A well-specified 

intervention also includes a definition of participation in and completion of the intervention. 

• A well-defined target population, consisting of a description of the population for which 

the intervention was found to be effective. 

• Implementation supports, consisting of a description of supports that can help ensure 

fidelity, such as an implementation monitoring team and performance benchmarks. 

Implementation supports also include a description of the procedures for putting the supports 

into action, such as describing the processes the monitoring team follows and a process for 

measuring performance benchmarks. 

The final two conditions indicate that an organization might be ready to scale an intervention: 

• Enabling context, consisting of a description of the presence of organizational and partner 

agency leadership and culture that supports the scaling effort. Enabling context is 

demonstrated with examples of ways that the organization is innovative and has improved 

upon past interventions, particularly in the face of implementation challenges. 

27 We use the terms personnel and workforce to refer to organization and partner personnel as well as AmeriCorps 

members who deliver intervention services.
Scaling insights: CAPC’s experience Mathematica

47

• Implementation infrastructure, consisting of a description of the organizational 

infrastructure, such as the workforce, materials, and physical space that support 

implementation.

The scaling framework was used to inform selection of the three grantees that participated in the 

process study, because CNCS and Mathematica sought to include interventions with 

implementing grantees that appeared to be ready to scale. We also collected data from the 

grantees included the process study using questions that were informed by the framework. This 

helped us understand whether the requirements for readiness for scaling were indeed present and 

sustained during implementation of each intervention. 

C. Grantee selection for the process study

Mathematica and CNCS used a multistage process to select the interventions and the grantees 

scaling them for inclusion in the process study. In the first stage, Mathematica reviewed the 

evaluation studies that grantees submitted to demonstrate evidence of their intervention’s 

effectiveness and grantees’ plans for scaling those interventions. Grantees submitted these 

documents to CNCS in 2015 and 2016 for AmeriCorps grantees and in 2010 and 2011 for SIF 

grantees. Mathematica used those documents to identify 17 interventions that CNCS grantees 

were scaling that demonstrated evidence of effectiveness. Mathematica identified these 

interventions by assessing whether the evaluation studies used rigorous research designs and had 

consistently favorable findings, and whether the intervention upon which the evidence was based

aligned with the proposed plans for the intervention during scaling. 

In the second stage, Mathematica developed and applied a scoring system to rank the 

interventions, and the grantee(s) scaling them, according to their readiness to scale. The scoring 

system used condition-level scores to operationalize each of the five conditions in the scaling 

readiness framework (see Section B). Mathematica identified eight interventions, associated with 

10 grantees28 that had relatively high scores and represented a mix of scaling types and 

intervention focus areas.29 

In the third stage, CNCS staff in the Office of Research and Evaluation (ORE) conducted 

outreach and telephone screening interviews to learn more information about the eight 

interventions and 10 grantees. ORE staff reached out to the CNCS program officers, AmeriCorps 

State Commission administrative staff, and AmeriCorps National Direct staff who oversaw or 

interacted with each of the 10 identified grantees to understand any grant management issues or 

grantee capacity concerns that might preclude them from participating in the process study. 

Then, ORE staff contacted administrative personnel from the grantees via telephone, using a 

28 One recommended intervention was being scaled by three different grantees; other recommended interventions 

were being scaled by one grantee each.

29 Intervention focus areas are the topics in which CNCS concentrates its funding: disaster services; economic 

opportunity; education; environmental stewardship; healthy futures (including physical and mental health, 

substance abuse, and nutrition); and veterans and military families.
Scaling insights: CAPC’s experience Mathematica

48

protocol to collect information on the grantees’ reported progress toward their intervention 

scaling objectives, barriers and successes they had encountered when scaling, and their interest 

and ability to participate in the process study. One of the 10 grantees asked not to be included in 

the process study, indicating that it could not fulfill the necessary data collection activities 

associated with the process study. 

In the final stage, Mathematica used the information CNCS collected to develop criteria to 

identify three grantees as candidates for the process study. The selection criteria included grantee

size and project age, geographic location, intervention focus areas, types of scaling, reported 

successes and challenges with their scaling experiences, and reported efforts to date to codify 

lessons learned. After applying these criteria to the remaining nine grantees, Mathematica 

recommended three grantees, each implementing a different intervention, as candidates for the 

process study. Based on Mathematica’s recommendations and application of the criteria, CNCS 

ultimately selected three grantees to include in the process study, all of which agreed to 

participate in the study.

30 

At the time of selection into the process study, the grantees varied in the extent to which they 

appeared to fully operationalize the conditions of the scaling framework. The selected sample 

included grantees that had reported, during their initial screening interviews with CNCS, both 

successes and challenges in their scaling execution processes, resource planning issues, 

successes and difficulties generating community support, grant management concerns, and 

successful, mixed, and poor results on various aspects of scaling. Because CNCS did not select 

the grantees at random, and they were not representative of all CNCS grantees, the insights from 

the experiences of the three process study grantees and their partners cannot be interpreted as 

applicable to a broader set of CNCS-funded grantees or service providers. However, the 

grantees—and the interventions they were scaling—were considered to be some of the strongest 

in terms of readiness to scale, and ranged in features such as geographic location, intervention 

focus areas, types of scaling, and the length of time they had been scaling their interventions. 

Because of their scaling readiness strength and range of experiences, the findings from the 

process study allow us to draw lessons learned and illustrative practices that can help 

stakeholders understand the conditions that might facilitate intervention scaling. 

D. Methods for collecting and analyzing data for the process study

1. Methods for collecting data for the process study 

Mathematica staff collected data from all three grantees selected for the process study during 

summer and fall 2018 and fall 2019. In summer 2018, Mathematica held pre-visit telephone calls 

with grantee personnel from all three grantees and reviewed grantees’ program documents. 

Mathematica staff then conducted two-day visits during October 2018 to each grantee as well as 

any partner organizations involved in scaling activities, and brief follow-up telephone interviews 

30 One of the grantees ultimately selected for inclusion in the process study differed from Mathematica’s 

recommendations. Based on its internal conversations, CNCS selected this grantee in light of its own research and 

funding priorities.
Scaling insights: CAPC’s experience Mathematica

49

after each visit. In September 2019, Mathematica staff conducted 12-month follow-up telephone 

interviews with grantee personnel. 

During the visits to each grantee, members of the process study team conducted one-on-one or 

small group interviews with the following types of personnel: (1) the program manager who 

oversaw implementation of the intervention being scaled; (2) grantee executives, such as the 

executive director and deputy directors; (3) data managers from grantees; (4) any other personnel 

from grantees or partners directly involved in supporting scaling activities, including supervisors 

and managers of frontline personnel; (5) frontline personnel, including AmeriCorps members, 

who directly delivered the intervention being scaled; and (6) for two out of three of the grantees 

visited, personnel from the developer of the intervention being scaled. (For this grantee, the 

grantee was scaling an intervention it developed itself). 

Both individual and group interviews during the process study visits generally lasted 30 minutes 

to two hours. The study team held the most comprehensive interviews with the program manager 

for each grantee. These interviews covered all topics related to the five conditions in the scaling 

framework (that is, the presence of a well-specified intervention, a well-defined target 

population, implementation supports, an enabling context, and an implementation infrastructure); 

the type of scaling conducted by the grantee and its partners; and the factors that appeared to 

facilitate and challenge implementation and scaling. Interviews with other types of respondents 

were more limited in scope. During interviews with grantee executives, we focused on topics 

related to planning and funding for scaling, and the use of evidence of intervention effectiveness 

in planning scaling efforts. When speaking with other types of grantee personnel, we focused on 

topics related to their specific function—for example, we concentrated on data systems and 

evaluation efforts when interviewing data managers. When interviewing personnel more closely 

aligned to frontline operations, in both individual and group settings, we focused on topics 

related to direct service provision, implementation supports (such as training, communication 

systems, data systems, and implementation and performance monitoring), use of evaluation, and 

other factors that might facilitate or hinder scaling. When interviewing intervention developers, 

we focused on topics related to implementation supports, evaluation efforts, plans to innovate or 

improve the intervention, scaling efforts beyond the specific grantee visited, and other factors 

that might facilitate or hinder scaling.

The study team collected additional data from the grantees during the follow-up telephone 

interviews held in September 2019. The study team held these interviews with the grantee 

program managers and executives. Across the grantees, these interviews focused on changes that 

had occurred in scaling since the visit in October 2018, any successes or challenges associated 

with scaling that the grantees experienced, and clarifications about information collected during 

the October 2018 visits. 

2. Information used for the analysis of CAPC’s and its partners’ scaling of B&B

We based the analysis for this report on information collected from CAPC and its partners, as 

summarized in the previous section. First, study team members reviewed documents that could 
Scaling insights: CAPC’s experience Mathematica

50

shed light on how CAPC executed scaling of the B&B intervention and the supports the grantee 

had in place to scale it. Examples of such documents include personnel manuals; human resource 

and communication protocols, such as personnel performance evaluation forms; performance 

and fidelity monitoring protocols; and summaries of results of participant assessments. The study 

team summarized the contents of these documents in a detailed write-up that also included notes 

from the data collection that took place during the process study visit (discussed below). 

The study team then visited the grantee in October 2018. During the visit, we conducted one-onone or small group interviews with the following types of personnel: (1) the program managers 

who oversaw implementation of B&B; (2) the executives of CAPC and of three partner 

organizations implementing Family Resource Centers (FRCs); (3) four CAPC workforce

members who provide training and support for B&B; (4) seven program managers, team leaders, 

or coordinators who oversaw frontline personnel; (5) 21 frontline personnel, including both FRC 

personnel and AmeriCorps members who served as home visitors, FRC aides, school readiness 

home visitors, data entry personnel, and crisis intervention specialists (across the three different 

local sites visited); and (6) two representatives from funders of B&B. In total, we interviewed 40 

respondents during the visit. Exhibit A.2 details the characteristics of the 40 interview 

respondents. Some respondents did not respond to every question. We interviewed personnel at 

CAPC’s office or at one of three FRCs we visited. CAPC personnel, Collaborative leaders, and 

FRC leaders were asked to participate by the program managers. Frontline personnel were asked 

to participate by leaders at their respective FRCs. 

Finally, the study team conducted follow-up telephone interviews with CAPC personnel in 

September 2019. The study team held a three-hour interview (split into two parts) with the two

grantee program managers, and one 90-minute interview with the grantee executive. 
Scaling insights: CAPC’s experience Mathematica

51

Exhibit A.2. Characteristics of those interviewed

Respondent characteristic Response category Number of respondents

Type of position Grantee program manager 2 

Grantee and partner organization executive 4 

Other grantee personnel 4 

Partner agency frontline supervisor 7 

Partner agency frontline personnel 21

Funder personnel 2 

Type of personnela AmeriCorps member 6 

Paid organizational personnel (from the 

grantee or FRCs)

23

Experience in current position Fewer than 12 months 19

1 to 2 years 5 

3 to 5 years 7 

More than 5 years 9 

Experience with organizationb Fewer than 12 months 9 

1 to 2 years 7 

3 to 5 years 9 

More than 5 years 12

Experience in the same type of workb Fewer than 12 months 6 

1 to 2 years 6 

3 to 5 years 8 

More than 5 years 17

Highest level of education Less than high school degree 0 

High school degree (including equivalency) 3 

Some college, no degree 7c

Associate’s degree 1 

Bachelor’s degree 20c

Master’s degree or above 9 

Gender Female 34

Male 6 

Race/ethnicityd Hispanic 24

Asian/Pacific Islander American 0 

Black/African American 5 

Native American 1 

White 14

Other 0 

a We asked respondents to report their job title. Twenty-nine respondents indicated whether they were AmeriCorps 

members or paid organizational personnel. 
Scaling insights: CAPC’s experience Mathematica

52

b Three respondents did not respond to this question.

c Includes respondents with higher education experience from outside the United States.

d Four respondents identified with two racial/ethnicity categories, so the sum of respondents in this category is greater

than 40. 

3. Analysis of collected data

We ensured accuracy and thoroughness of data collection in the following ways: (1) preparing 

standardized protocols that were tailored to each respondent type and to the intervention and 

grantee; (2) having both a senior and junior researcher conduct the visits and telephone 

interviews so that one team member could take notes while the other conducted interviews; (3) 

audio recording interviews and taking detailed, near-verbatim notes during interviews; and (4) 

having multiple study team members review and provide feedback on the interview notes as well 

as ask for clarifications on content when necessary and appropriate. 

After finalizing the site visit interview notes, study team members synthesized those notes into a 

detailed write-up based on a standardized template. The template grouped information according 

to (1) scaling readiness conditions; (2) the contextual factors that affect implementation and 

scaling; (3) the accomplishments, challenges, successes, and facilitators of scaling; and (4) the 

sustainability of scaling efforts. The write-up also included a checklist that summarized the 

intervention’s readiness for scaling in each of the five conditions of the scaling framework. 

Members of the study team reviewed these write-ups for completeness, thoroughness, and 

accuracy. Before visiting, the research team also developed detailed descriptions of each

intervention and each grantee’s scaling of it, and asked the respective program managers to 

review and correct the description if needed. Study team members synthesized the follow-up 

telephone interview notes by organizing the notes by respondent and by the topics that aligned 

with the chapters and subsections of each of the case study reports. 

To conduct the analysis across all three grantees, the study team reviewed the grantee-specific 

write-ups to synthesize data according to the data collection topics of interest. (These topics 

related to the research questions and the ways that grantees were approaching aspects of the five 

conditions that indicate scaling readiness.) Because the study team conducted this analysis before 

the September 2019 follow-up telephone interviews, it and the resulting report (Needels et al. 

2020) were based on a more limited set of information than the analysis for the case study 

reports. 

For this report, the authors assessed the topic-specific information relevant to only CAPC from 

the analyses conducted with the data from the site visit and the telephone interviews. The authors 

used these data to identify insights and takeaway conclusions that have the potential to be 

broadly applicable as CNCS seeks to support its grantees in their scaling efforts. Quotes from 

interview respondents also provided illustrative insights. A similar approach was used to analyze 

data for the two companion reports (Anderson et al. 2020; Jones et al. 2020), each of which 

provides in-depth insights about scaling using data from the two other grantees in the process 

study. 
This page intentionally blank for double-sided copying.
This page intentionally blank for double-sided copying.
Mathematica

Princeton, NJ • Ann Arbor, MI • Cambridge, MA 

Chicago, IL • Oakland, CA • Seattle, WA

Tucson, AZ • Woodlawn, MD • Washington, DC 

EDI Global, a Mathematica Company

Bukoba, Tanzania • High Wycombe, United Kingdom

mathematica.org