Scaling Checklists: Assessing Your Level 

of Evidence and Readiness (SCALER)

A Guide for Practitioners

September 25, 2020

Scott Richman, Nan Maxwell, Karen Needels, and Mary Anne Anderson 

Submitted to: Submitted by:

AmeriCorps

250 E Street SW

Washington, DC 20525

Project Officer: Lily Zandniapour

Contract Number: 

GS10F0050L/CNSHQ16F0049

Mathematica

111 East Wacker Drive, Suite 3000

Chicago, IL 60601-4303

Phone: (312) 994-1002

Fax: (312) 994-1003

Project Director: Scott Richman

Reference Number: 50332
This page has been left blank for double-sided copying. 
 

Mathematica iii

Contents

Abstract......................................................................................................................................... v

I. Scaling Effective Interventions...............................................................................................1

A. The SCALER framework................................................................................................. 2

B. Purpose of guide.............................................................................................................3

II. Identifying or Building Evidence of Effectiveness ..................................................................5

A. Identifying evidence of effectiveness through use of research clearinghouses ..............5

B. Building evidence............................................................................................................7

C. Checklist for building evidence of effectiveness ...........................................................11

III. Getting Ready to Scale........................................................................................................13

A. Is my effective intervention ready to scale?..................................................................13

B. Is my organization ready to scale? ...............................................................................18

C. Summarizing across checklists.....................................................................................20

D. Completing and using the SCALER ................................................................................21

IV. Ready, Set, Scale? ..............................................................................................................25

A. Establishing evidence of effectiveness may not be possible at the moment ................25

B. The organization might have an opportunity to scale before it is ready........................26

C. The organization is seeking to adapt an intervention....................................................27

References..................................................................................................................................29

Appendix ...............................................................................................................................33

Tables

Table II.1. Standards to identify effective interventions ................................................................6

Table II.2. Types of evaluation failure and potential solutions ....................................................10
 

Mathematica iv

Figures

Figure I.1. The SCALER framework .............................................................................................2

Figure II.1. Building evidence........................................................................................................7

Figure III.1. Establishing readiness to scale ...............................................................................13

Checklists 

Checklist 1. Identifying an effective intervention ...........................................................................7

Checklist 2. Building evidence of effectiveness ..........................................................................11

Checklist 3. Specifying the intervention ......................................................................................15

Checklist 4. Defining the target population .................................................................................16

Checklist 5. Establishing implementation supports.....................................................................17

Checklist 6. Having an enabling context .....................................................................................19

Checklist 7. Establishing an implementation infrastructure.........................................................20

Checklist 8. Summary checklist ..................................................................................................21

Exhibit

Exhibit III.1. Process for completing the SCALER ......................................................................22
Abstract

Mathematica v 

Abstract

Successfully scaling effective interventions can help organizations improve lives for larger numbers of 

individuals and communities than they could have reached before scaling. Organizations can maximize 

this potential by ensuring that the interventions that they scale have shown evidence of improving 

participant outcomes and by putting in place intervention-level and organizational procedures and 

processes to successfully scale. This guide describes a framework developed from implementation 

science research—Scaling Checklists: Assessing your Level of Evidence and Readiness, or SCALER—

that identifies how organizations can improve both their readiness to scale an intervention and the 

intervention’s readiness to be scaled, so that intervention services are best positioned to improve 

outcomes for a larger number of participants. The SCALER includes checklists that organizations can use 

to determine if they have (1) evidence of their intervention’s effectiveness that meets industry standards 

for rigorous evaluation research and (2) the procedures and processes in place that will enable them to 

successfully scale an intervention. 
This page has been left blank for double-sided copying. 
 

Mathematica 1 

I. Scaling Effective Interventions

Successfully scaling effective interventions can 

help organizations improve lives for larger numbers 

of individuals and communities. Successful 

scaling—defined as maintaining or surpassing the 

beneficial impacts of an intervention that has been 

documented to have evidence of effectiveness—is 

contingent on implementing the intervention as it 

was intended (that is, with fidelity). (See the 

sidebar for a definition of scaling.) AmeriCorps has 

been interested in building a process that it can use 

to help its grantees engage in successful scaling.1

 It 

funded Mathematica to conduct the Scaling 

Evidence-Based Models (SEBM) project, which is 

intended to help deepen the agency’s understanding 

of the interventions it funds and its knowledge base 

on scaling them. 

As part of this project, Mathematica designed and 

implemented the Scaling Programs with Research 

Evidence and Effectiveness, or SPREE, which 

provides a comprehensive methodology for funders 

to use when they want to help organizations scale 

effective interventions (Maxwell and Richman 

2019). SPREE was grounded in implementation 

science research that identified how organizations 

might scale interventions so they improve desired 

outcomes for a larger number of participants 

(Miller et al. 2006; National Implementation 

Research Network [NIRN] 2018). SPREE was 

further informed and refined by the lessons learned 

when applying the SPREE process to a group of 

organizations that received AmeriCorps funding 

that was to be used for scaling. 

Although the SPREE process was initially 

developed for funders to help organizations scale 

effective interventions, Mathematica has adapted it so that organizations can use it directly. In this guide, 

we describe the Scaling Checklists: Assessing your Level of Evidence and Readiness, or SCALER, which 

organizations can use to increase their likelihood of successfully scaling an intervention that increases the 

desired outcomes of participants. Funders can also use the SCALER to help communicate expectations to 

their grantees about scaling readiness and inform how to provide technical assistance or training to one or 

more grantees to help them prepare for scaling. This guide discusses the SCALER framework, describes 

each of its components, and provides checklists that organizations can use to assess their level of 

1 Prior to September 29, 2020, AmeriCorps operated under the name Corporation for National and Community 

Service.

What is scaling?

Scaling involves replicating, expanding, 

and/or adapting an intervention to 

improve lives for larger numbers of 

individuals and communities. It is 

considered successful when the 

intervention maintains or surpasses its 

beneficial impacts for participants after 

the scaling has occurred. The types of 

scaling are:

Expansion extends the intervention to 

more people in the same target 

population in the same location. An 

example would be increasing the number 

of unemployed adults served at a work 

center by hiring five more job search 

specialists who will each serve 20 more 

adults.

Replication extends the intervention for 

the same target population to a new 

location. An example would be 

implementing a reading program 

designed for 5th graders in a new school 

district, city, and state, but serving the 

same target population of 5th graders.

Adaptation extends the intervention to a 

different target population in either the 

same or different location or modifies the 

intervention for the same population in 

either the same or different location. An 

example would be modifying a parent 

training curriculum designed for mothers 

to include language that is more inclusive 

of fathers.
Chapter I Scaling Effective Interventions

Mathematica 2 

preparation in each component or that funders can 

use in helping organizations prepare for scaling. It 

also provides guidance for how organizations might 

fill out these checklists, including who should fill 

out the SCALER, when and how often it should be 

completed, and how to use the results to identify 

next steps.

A. The SCALER framework

The SCALER framework can help organizations 

improve the lives of more people by preparing them to successfully scale effective interventions. The 

SCALER’s two-part process (1) ensures that the intervention to be scaled is likely to produce desired 

outcomes and is therefore worthy of being scaled and (2) identifies whether the effective intervention and 

the organization are ready to scale. 

Figure I.1. The SCALER framework

Figure I.1 outlines the SCALER framework. As the figure shows, the organization wanting to scale an 

intervention should first ask, “Is there evidence that this is an effective intervention?” The answer to this 

question is not always straightforward, in part because different entities might hold different opinions on 

what constitutes evidence (as will be discussed in Chapter II). Still, the answer to the question is critical 

because it determines what needs to be done to prepare an organization to scale an intervention. If the 

organization does not have evidence that its intervention is effective, it should build that evidence before 

scaling. If the intervention has evidence that it is effective, the organization needs to ensure there is a 

readiness to scale before scaling. 

• Building evidence. The SCALER framework includes a five-step process to build evidence that the 

intervention is effective. That evidence is built by showing that the intervention alone—and not other 

factors—has an impact that improves desired participant outcomes. The SCALER framework 

The purpose of this guide

This guide was designed to help 

organizations scale their interventions 

successfully. It describes the Scaling 

Checklists: Assessing your Level of Evidence 

and Readiness (SCALER) tool, which can 

help organizations assess how ready they 

and their interventions are for scaling.
Chapter I Scaling Effective Interventions

Mathematica 3 

suggests that building this evidence involves a process to continuously improve the intervention; it 

does not use evidence to make a “thumbs up/thumbs down” decision about whether to continue 

offering an intervention (Austin and Claassen 2008b). This potential need to tweak an intervention 

until it is shown to be effective is represented by the circular flow depicted in Figure I.1 and discussed 

extensively in Chapter II. This process begins with clearly defining the intervention and the path it 

takes to affect outcomes, and then consistently following the articulated path. Once the intervention is 

implemented consistently and has suggestive evidence that participants’ outcomes improve after 

receipt of intervention services, the process moves to designing and implementing an evaluation that 

meets standards for effectiveness and that demonstrates the intervention’s effectiveness. If the 

evaluation shows evidence of effectiveness, the intervention gets ready to scale. If it does not, the 

intervention is tweaked in ways to increase its likelihood of being effective, and the evidence-building 

process begins again.

• Getting ready to scale. The SCALER framework specifies five conditions that should be present for 

both an intervention and an organization for successful scaling, as discussed in Chapter III. Having 

these five conditions in place helps ensure fidelity after scaling, which can be critical to successful 

scaling, as both fidelity and effectiveness often deteriorate as adjustments are made during scaling 

(Larson et al. 2017). Three of the conditions in the framework are meant to ensure the intervention 

has features that allow it to be implemented with fidelity after scaling—namely, the intervention 

needs to have its core activities and target population clearly specified, and needs to have supports for 

implementation in place so they can be implemented in a manner that will achieve the same positive 

outcomes after scaling. The other two conditions are intended to ensure the organization can support 

scaling the intervention: (1) the organization’s leadership and culture supports innovation, learning, 

and improvement; and (2) the organization has an infrastructure that can provide the resources and 

support the scaling.

B. Purpose of guide 

This guide was designed to help organizations scale 

their interventions successfully. The next two 

chapters discuss the conceptual underpinnings of 

the SCALER framework as well as checklists that 

organizations can use to assess how prepared they 

and their interventions are for scaling. Chapter II 

discusses and provides checklists for both 

identifying and building evidence of effectiveness, 

and Chapter III provides the discussion and 

checklists for getting ready to scale the 

intervention. It also provides guidance on how 

organizations might use the SCALER—how to fill 

it out, who should be involved, how often it should 

be completed, and how to use the checklist scoring 

results to identify next steps for getting ready to 

scale. Both chapters present an idealized version of 

what organizations need for successful scaling. 

Chapter IV examines scaling when either of the two 

parts outlined in the SCALER framework is 

Considerations as you read this 

guide

• The SCALER framework is based on 

successful scaling principles that have 

emerged from implementation science 

research.

• The checklists in this guide were 

developed from rubrics used to assess 

AmeriCorps grantees’ evidence of 

effectiveness for funded interventions 

and readiness to scale.

• The discussion and checklists in this 

guide were developed for AmeriCorps 

grantees to use when scaling their 

funded interventions, though they may 

also be useful to a broader audience.
Chapter I Scaling Effective Interventions

Mathematica 4 

difficult to achieve. This chapter discusses constraints that organizations might face when seeking to 

establish evidence of their interventions or encounter opportunities to scale an intervention before the 

organizations are ready to do so. The appendix compiles all the checklists that appear throughout this 

guide. 
 

Mathematica 5 

II. Identifying or Building Evidence of Effectiveness

Before an intervention is scaled to reach a larger 

number of people, it should have demonstrated 

effectiveness for the population it initially targets. 

An organization can draw on rich information from 

external sources, such as research clearinghouses, 

to understand if existing research shows that an 

intervention it wants to scale has already been 

proven effective. But suppose an intervention has 

not been rigorously evaluated, or the existing 

research does not show evidence of effectiveness 

(because the research design lacked rigor or no 

impacts were found). In such cases, organizations 

must build evidence of the intervention’s 

effectiveness and demonstrate that it has the desired 

effects on anticipated outcomes. This chapter 

provides organizations with a framework and 

checklists for identifying evidence of 

effectiveness—or, if no such evidence exists, for 

building that evidence to ensure that the steps an 

organization is taking will likely result in research 

that meets the standards for showing evidence of 

effectiveness. 

A. Identifying evidence of effectiveness 

through use of research 

clearinghouses 

It can be tricky for organizations to understand 

whether an intervention can be considered 

effective. Simply assessing participant outcomes 

does not demonstrate whether an intervention is 

effective because outcomes can change due to a 

variety of factors, including the economic context 

in which the intervention was implemented or 

participant-specific characteristics. Rigorous 

research can be difficult to conduct because it 

requires isolating whether the intervention itself 

contributed to changes in participant outcomes. 

Yet, knowing that an intervention and not some 

other factor changed participant outcomes provides 

the evidence that the intervention is effective and 

warrants scaling. Organizations therefore need to 

ensure that their interventions are effective before 

investing resources in scaling them. 

Research Clearinghouses

• What Works Clearinghouse (WWC, 

https://ies.ed.gov/ncee/wwc/) reviews 

the existing research on educational 

programs, products, practices, and 

policies to help educators make 

evidence-based decisions.

• Clearinghouse for Labor Evaluation 

and Research (CLEAR, 

https://clear.dol.gov/) reviews studies 

of labor programs to inform decisions 

about employment-related policies 

and programs.

• Home Visiting Evidence of 

Effectiveness

(https://homvee.acf.hhs.gov/) reviews 

research on home visiting models that 

serve pregnant women or families with 

children up to kindergarten age.

• Pathways to Work

(https://pathwaystowork.acf.hhs.gov/) 

reviews studies of interventions 

designed to help low-income job 

seekers succeed in the labor market.

• Results First

(https://www.pewtrusts.org/en/researc

h-and-analysis/datavisualizations/2015/results-firstclearinghouse-database) centralizes 

information from nine national 

clearinghouses on the effectiveness of 

social policy programs.

• For information on other research 

clearinghouses, see AmeriCorps’ 

“Clearinghouses and Evidence 

Reviews for Social Benefit Programs” 

(https://www.nationalservice.gov/

sites/default/files/documents/Clearingh

ouses%20and%20Evidence%20Revie

ws.pdf).
Chapter II Identifying or Building Evidence of Effectiveness

Mathematica 6 

The ability to find external information about interventions’ effectiveness has increased dramatically over 

the past decade. Most prominently, research clearinghouses such as those listed in the sidebar provide 

information to help organizations and funders identify research on the effectiveness of a wide variety of 

interventions. When considering scaling an intervention and looking for evidence of its effectiveness, 

organizations can: 

1. Seek to understand what is needed to conduct rigorous research to demonstrate an 

intervention’s effectiveness. Understanding whether a research study can establish a causal 

beneficial impact on participants’ outcomes is critical because the quality of evaluation research 

varies greatly. Although different clearinghouses use slightly different criteria for making this 

determination, they use common principles (Table II.1).2 These standards define the level of rigor that 

needs to be demonstrated in an evaluation study to show with confidence that an intervention caused 

beneficial outcomes for participants. 

Table II.1. Standards to identify effective interventions

Evidence standards Definition

Evaluator independence The evaluator is external to the organization or intervention with no vested interest in 

the evaluation findings. 

Design has a counterfactual The research design has a counterfactual, generally meaning a comparison or control 

group that is ideally developed through random assignment and that is used to 

provide evidence about participant outcomes in the absence of the intervention. 

Design has no confounding 

factors

The research design enables the evaluators to distinguish between outcome changes 

that are due to the intervention and changes that are due to other factors.

The research has: 

• Low attrition

• No reassignment

• Baseline equivalence

• Few people in the treatment or comparison group left the study.

• No people in the comparison group switched to the treatment group or vice versa.

• The final treatment and comparison groups used in the outcomes analysis did not 

differ on key characteristics, other than one group was given access to program 

services. 

2. Determine whether an intervention they want to scale has research showing that it is effective.

For example, an organization helping high schools implement a career and technical education (CTE) 

intervention could use both CLEAR and the WWC to determine which CTE interventions have 

evidence of effectiveness. This information can help an organization decide whether it wants to 

pursue scaling of a particular intervention, though it may still want to weigh additional information or 

seek out other studies on intervention effectiveness. Organizations must decide if the current amount 

and rigor of the research on their intervention is adequate for them to consider the intervention 

effective, or if more evidence of effectiveness is needed. 

3. Learn about the context or environment in which the intervention they want to scale was 

evaluated. This information can help organizations decide if the research is relevant to the context in 

which their intervention will be scaled. Showing that an intervention was effective 25 years ago or 

that it was effective for 1st- and 2nd-graders in inner city schools, for example, does not necessarily 

2 All clearinghouses provide explicit criteria for meeting each of these standards. In addition, two practitioner guides 

developed for the SEBM project (Anderson and Maxwell 2018; Anderson and Maxwell 2019) are resources that 

organizations can use to understand this terminology: “Baseline Equivalence: What It Is and Why It Is Needed,” 

https://nationalservice.gov/impact-our-nation/evidence-exchange/baseline-equivalence-what-it-and-what-it-needed; 

“What Makes for a Well-Designed, Well-Implemented Impact Study,” https://nationalservice.gov/impact-ournation/evidence-exchange/what-makes-well-designed-well-implemented-impact-study. 
Chapter II Identifying or Building Evidence of Effectiveness

Mathematica 7 

mean it will be effective if scaled today for 4th-graders in rural America. Therefore, organizations can 

learn about the locations or populations that were used to establish evidence for their intervention’s 

effectiveness and determine the extent to which these contexts apply to the plans the organizations

have for scaling their interventions.

The sidebar provides Checklist 1, which 

organizations can use to assess whether their 

intervention has evidence of effectiveness. 

The first four categories align with the 

clearinghouse standards that guide evaluation 

research. If a research clearinghouse has 

identified evaluations of the intervention in 

which it has confidence in the results, the 

first four categories would be rated “yes.” An 

organization must then decide if that research 

meets its needs for evidence of the 

intervention’s effectiveness and if the 

research is relevant to the environment in 

which it is planning to scale. Should one or 

more of the answers be a “no,” meaning the 

organization does not have evidence of the 

intervention’s effectiveness, then such 

evidence should be built, as discussed in the 

next section. If all answers are a “yes,” the 

organization should next determine its readiness to scale the intervention (Chapter III). 

B. Building evidence

When rigorous evidence about an intervention’s effectiveness does not exist or does not meet the needs of 

an organization interested in scaling that 

intervention, evidence of effectiveness must be 

built. The SCALER framework (Figure II.1)

includes a five-step process for building evidence 

of an intervention’s effectiveness that is grounded 

in the use of research and evaluation to gain 

knowledge about an intervention’s implementation 

and impact. Such a process can also become a 

powerful tool for improving intervention design,

as organizations use the results from research to 

test and retest the assumptions underlying the 

intervention (the intervention’s core components 

that are deemed necessary to cause beneficial 

outcomes for participants). 

1. Define the intervention

Research and evaluation can help an organization 

continuously improve their intervention and 

Figure II.1. Building evidence

Checklist 1. Identifying an effective intervention

Check one

Components Yes No 

1. Research rigor 

Research conducted by a third party ○ ○

Research includes a counterfactual ○ ○

Research has no confounding factors ○ ○

Research has low attrition, no reassignment, and 

baseline equivalence with final sample 

○ ○

2. Intervention effectiveness 

The research showed evidence of effectiveness ○ ○

3. Intervention relevance 

The research is appropriate to the context or 

environment in which scaling is planned (the 

research is generalizable or the intervention will 

be scaled within the specific environment studied) 

○ ○

If “no” in one or more categories, there is a continued need for 

evidence (see Chapter II, Section B)

If “yes” in all categories, determine readiness to scale (see Chapter III)
Chapter II Identifying or Building Evidence of Effectiveness

Mathematica 8 

provide evidence of its effectiveness. A well-articulated theory of change (TOC) is central to this 

effort. It defines an intervention by demonstrating how the intervention’s resources and activities are 

structured with the intent of changing participant knowledge, attitudes, behaviors, and/or conditions in 

ways that improve outcomes. Developing a TOC with this level of detail generally begins with thinking 

about the desired outcomes that the intervention will affect and then identifying all the assumptions about 

how the intervention might do so (Hernandez and Hodges 2003; Yampolskaya et al. 2004). Articulating 

the assumptions for how the outcomes will be affected is critical for the intervention’s success because it 

helps establish the inputs, activities, outputs, and outcomes needed for the intervention to succeed. 

Such a TOC provides a road map that identifies the human, financial, and physical resources (inputs) 

required to implement the intervention’s core components (activities).3 These activities result in services 

provided (outputs) that ultimately yield the desired outcomes. It is important that the desired outcomes be 

shown as measurable changes that are expected to occur as a result of the intervention—this designates 

them as goals for the organization to achieve. These four components of a TOC (inputs, activities, 

outputs, and outcomes) are often influenced by situations, forces, or circumstances that may exist within 

or outside the organization (contextual factors). 

Identifying these components in a TOC provides a clear sense of what is needed to produce outputs and 

what needs to be measured to determine that the intervention improves participant outcomes (Hodges et 

al. 2002). In addition to helping the organization ensure that the intervention is being implemented 

as planned, this clarity will show what measures, as reflected in the outputs and outcomes, are needed for 

monitoring and evaluating the intervention. 

In addition to providing clarity about the intervention and its implementation, the intervention must be 

“mature and stable” when it is evaluated. This means that the inputs and activities identified in the TOC 

must be implemented and supported as part of a regular routine, that personnel implementing the 

intervention must be proficient and skilled and have managerial and administrative support, and that

output and outcome goals must be met (Bertram et al. 2011). Conducting an evaluation before an 

intervention is mature and stable might not provide an accurate assessment of its potential to improve 

participant outcomes (Macallair and Males 2004). 

An organization may seek to make modifications to an intervention in the hopes of improving upon the 

outcomes for the individuals they serve (based on prior evaluations of those efforts) or to maximize buyin from communities where the intervention might take place. Ideally, these adaptations would be 

informed by other existing research and evidence to illustrate how the adapted intervention can 

produce the desired outcomes for participants. Once the evaluation is underway, the organization can 

assess whether the modifications were implemented as intended and whether they improved the 

intervention’s effectiveness. (For more information about considerations for adaptation, see Chapter IV.)

2. Lay evaluation groundwork

Building research evidence and using it to improve an intervention requires a culture of measurement, 

learning, and evaluation. In such a culture, stakeholders have a common understanding about the 

value of and need for research and evaluation in decision making (Austin and Claassen 2008a). In 

practice, this means that stakeholders understand the value of using a TOC to improve the design and 

3 The W.K. Kellogg Foundation has developed a Logic Model Development guide (W.K. Kellogg Foundation 2004) 

that organizations can use to construct or strengthen the logic model for their interventions. See 

https://www.wkkf.org/resource-directory/resources/2004/01/logic-model-development-guide.
Chapter II Identifying or Building Evidence of Effectiveness

Mathematica 9 

implementation of an intervention and, subsequently, generating estimates of the intervention’s impact. 

This culture might take time to develop, but the value of this process is often highest when developed 

over time rather than imposed in the short term (Walker and Soule 2017). 

Even within such a culture, an organization needs to build a consensus among key stakeholders, such as 

organizational leaders and board members, on the outcomes the intervention is trying to influence 

and the goals for how much outcomes should be improved. For example, a funder might want to see a 

study that compares participant outcomes to those of a similar group of individuals who did not 

participate in the program. The organization’s board of directors might want to see that the benefits of the 

program outweigh its costs. To identify the goals of an evaluation, the organization should first determine 

how its stakeholders define the effectiveness that the intervention must display, which will help determine 

what resources and personnel would be necessary for evaluating the intervention.

All stakeholders, including organizational leaders and personnel involved in implementing the 

intervention, must support research and evaluation efforts before they are designed and begin (Fixsen et 

al. 2005). Agreement must exist on the TOC components, implementation goals for the intervention 

(outputs and outcomes), and what constitutes evidence of effectiveness. Such agreement not only 

determines the design of the research but also its ability to obtain accurate information about the 

intervention, including background and context. Personnel implementing the intervention—or managing 

those who are—are critical for providing this information. If organizational leadership, personnel, or other 

key stakeholders implementing the intervention are not willing and able to participate in an evaluation, 

the evaluation will likely not be feasible—or as useful as possible, even if it is conducted. 

3. Prepare for evaluation

Research and evaluation require financial resources and personnel.

4 AmeriCorps found that evaluations 

that can provide evidence of effectiveness tend to cost 15 to 25 percent of a grant’s budget depending on 

the scope of the evaluation (Zandniapour and Vicinanza 2013). Costs include infrastructure expenses for 

implementing the evaluation (such as preparing systems to record data that will be analyzed for the 

evaluation), fees for experts in research design and implementation, and personnel time in collecting data 

for the evaluation. Not allocating sufficient funding to evaluation activities could lead to less rigorous or 

poorly implemented research that does not meet the standards of rigor for showing an intervention is 

effective (as discussed in the previous chapter). The latter scenario might arise because money is not 

spent to fully engage personnel in the evaluation. For example, personnel are not trained about the 

specifics of their assigned evaluation tasks or do not carry out evaluation tasks because they have 

concerns that have not been addressed about collecting evidence for causal research (Despard 2016; 

Gondolf 2015).

4. Conduct evaluation to assess effectiveness

Once it has prepared for its evaluation, an organization can move on to conducting its study, which would 

ideally be rigorous enough to allow the organization to assess whether its intervention has evidence of 

effectiveness (see Checklist 1). AmeriCorps refers to this type of rigorous evaluation study as an impact 

4 AmeriCorps has developed an Impact Evaluability Assessment Tool (Zandniapour and JBS International 2014) 

that organizations can use to determine their readiness to engage in a rigorous impact evaluation of their 

interventions. See 

https://www.nationalservice.gov/sites/default/files/resource/Laying_the_Groundwork_Before_Your_First_Evaluatio

n_Tool.pdf. 
Chapter II Identifying or Building Evidence of Effectiveness

Mathematica 10

evaluation. Impact evaluations should be conducted by an objective third party; this tends to bring greater 

objectivity because there is less risk of a real or perceived conflict of interest and, as a result, stronger 

evidence of an intervention’s effectiveness. However, an organization’s internal personnel are also critical 

for the measurement, evaluation, and learning undertaken almost every day. For example, during the 

evaluation design phase, organizational personnel must work with evaluators on three key tasks: (1) 

clarify the intervention’s TOC so that evaluators understand the inputs, activities, outputs, and outcomes; 

(2) ensure that the evaluation explores the essential components in the TOC; and (3) create processes to 

translate information from the evaluation into organizational learning and improvement. In addition, when 

the evaluation is being implemented, organizations and evaluators will work together to make sure the 

research tasks are carried out as planned.

5. Continued need for evidence

Upon completing its evaluation, an organization may find that its intervention did not improve participant 

outcomes, or it may be unable to establish evidence of effectiveness due to lack of rigor. In those 

instances, the organization is faced with the continued need for evidence, and it should start over 

with the five-step process for building evidence of an intervention’s effectiveness (see Figure II.1). 

Three reasons might explain why research and evaluation efforts do not show evidence of effectiveness 

(Stame 2010): method failure, implementation failure, and theory failure. How the organization proceeds 

with building evidence of effectiveness depends on the type of failure (also highlighted in Table II.2). 

• Method failure occurs when an evaluation 

does not show evidence of effectiveness 

because its design or implementation was 

flawed. For example, the evaluation might not 

have contained a large enough sample to 

demonstrate statistical significance (Lipsey 

2000), or issues may have arisen when the 

evaluation was carried out—such as random 

assignment was not properly conducted—that 

inhibited the evaluators from adequately 

assessing the intervention’s effectiveness

(Greene et al. 1989). In such cases, another 

evaluation might be in order. 

• Implementation failure occurs when an 

intervention is not implemented as intended. 

For example, personnel might find it 

challenging to implement the intervention as designed (Donkoh et al. 2006), are intentionally 

adapting the intervention to their own context (Patton 2010), or disagree on what the intervention’s 

core components are or define the intervention in a variety of ways (Lipsky 1980). In all cases, the 

intervention would not be implemented with fidelity to the intended model, and the evaluation would 

not be able to capture its potential impact. In such cases, intervention implementation would need to 

be improved to ensure fidelity. 

• Theory failure occurs when the intervention is not actually linked to the desired outcome. In some 

cases, this causal link might not exist, and the intervention or key components need to be revisited. 

Table II.2. Types of evaluation failure and 

potential solutions 

Type What it is Potential solution

Method failure Occurs when an 

evaluation does not 

show evidence of 

effectiveness because 

its design or 

implementation was 

flawed 

Conduct a new 

evaluation 

Implementation 

failure 

Occurs when an 

intervention is not 

implemented as 

intended 

Improve intervention 

implementation to 

ensure fidelity 

Theory failure Occurs when the 

intervention is not 

actually linked to the 

desired outcome 

Revisit the design of 

the intervention or 

its key components 
Chapter II Identifying or Building Evidence of Effectiveness

Mathematica 11

This can occur when organizations 

implement and test adaptations to an existing 

intervention to serve a new target population 

of participants or to deliver services in a new 

type of setting. In other cases, the causal link 

might be dependent on another intervention 

(Rogers 2008). For example, an evaluation of 

a complicated intervention with multiple 

components might assess only some 

components and not others and might show 

no impacts, but the intervention as a whole, if 

evaluated, might generate an impact. The 

intervention might also work in conjunction 

with other interventions and might not show 

an impact when evaluated alone. 

C. Checklist for building evidence of 

effectiveness 

The sidebar provides Checklist 2, which 

organizations can use as they seek to build 

evidence of effectiveness for their 

interventions. An organization can use the 

first group of checklist items to ascertain 

whether it has an appropriately defined 

intervention. If it does, it moves on to laying 

the groundwork for the evaluation (that is, 

gaining stakeholder agreement). Once the 

organization lays the groundwork, it can start 

preparing for the evaluation by ensuring 

adequate resources are in place. After all of 

the resources are in place, the organization 

can conduct the evaluation and assess the 

intervention’s effectiveness. Should the 

intervention be shown to be effective, the 

organization would get ready to scale 

(Chapter III). Should the evidence not 

indicate effectiveness, such evidence must be 

built through a process that identifies and corrects any failures or weaknesses that emerged along the way. 

Checklist 2. Building evidence of effectiveness

Check one

Components Yes No 

1. Define intervention 

Defined theory of change (TOC) for the 

intervention 

○ ○

Intervention mature and stable ○ ○

Intervention implemented as described in TOC 

(with fidelity) 

○ ○

Intervention, including any modifications, informed 

by research 

○ ○

If “yes” in all categories, lay groundwork . .

2. Lay evaluation groundwork

Agreement among organization, partners, and 

funders on need for evidence 

○ ○

Agreement among organization, partners, and 

funders on outcomes and measures 

○ ○

Agreement among organization, partners, and 

funders on evaluation goals 

○ ○

Organization partners, and funders support 

evaluation 

○ ○

If “yes” in all categories, prepare for evaluation . .

3. Prepare for evaluation

Financial resources available ○ ○

Personnel available to work with evaluators ○ ○

If “yes” in all categories, conduct evaluation . .

4. Conduct evaluation and assess effectiveness

Research conducted in a manner that meets 

standards for rigor (see Checklist 1 for an 

effective intervention and standards to identify 

effective interventions) 

○ ○

The research showed beneficial results for 

participants 

○ ○

The context or environment in which the research 

was conducted is generalizable to the 

organization’s plans for scaling its intervention 

○ ○

If “yes” in all categories, get ready to scale . .

5. Continued need for evidence

If “no” in one or more categories, make 

corrections and repeat process, as needed 

. .
This page has been left blank for double-sided copying. 
Mathematica 13

III. Getting Ready to Scale 

The SCALER framework includes five conditions that should be met for both an intervention and the 

organization implementing it to be ready to scale successfully (Figure III.1). Having these five conditions 

in place before scaling begins helps ensure that organizations can implement the intervention with fidelity 

to the TOC after scaling. This advantage 

is critical, as both fidelity and 

effectiveness often flounder when 

adjustments are being made while 

delivering intervention service during 

scaling (Larson et al. 2017). Maintaining 

fidelity to the intervention model after 

scaling helps ensure that the intervention 

will generate the same beneficial 

participant outcomes that occurred before 

scaling.

Each of the five conditions includes 

guidance for components, processes, and 

procedures that need to be in place for an 

organization and its intervention to be 

ready to scale. This chapter provides 

organizations with a framework for why each of these conditions is essential for scaling, as well as a 

checklist to assess whether the components, procedures, and processes within each condition are in place 

to foster scaling readiness. 

Each checklist in the SCALER provides summary scores to reflect how ready an intervention and 

organization might be for scaling. The closer a score is to the maximum total for a particular condition, 

the more ready for scaling the intervention or organization is, with regard to that condition. Lower total 

scores indicate the intervention or organization might not be as ready for scaling. To strengthen their 

scaling readiness, organizations can examine each row in the checklist to identify the components (or 

specific areas) within the conditions that they may need to better define or put into place.

In this chapter, we first present the checklists associated with intervention readiness for scaling (Section 

A) and the checklists associated with organizational scaling readiness components (Section B). Scoring 

instructions and explanations of the components within the conditions accompany the checklists. We then 

include a summary checklist (Section C) that allows organizations to assess scaling readiness across all 

five conditions. We close this chapter by offering guidance on how organizations might approach using 

the SCALER (Section D).

A. Is my effective intervention ready to scale?

The first three conditions for successful scaling indicate whether the intervention has the features that 

could allow it to be implemented with fidelity after scaling. Meeting these first three conditions indicates 

an intervention is ready to be scaled with fidelity.

Figure III.1. Establishing readiness to scale
Chapter III Getting Ready to Scale

Mathematica 14

1. Well-specified intervention

A well-specified intervention clearly identifies the core set of activities that are critical for achieving 

beneficial outcomes (Fixsen et al. 2005). Typically, core activities are identified in an intervention’s 

theory of change, as the mechanisms by which an intervention produces the expected outcomes (Blase 

and Fixsen 2013). For example, for a school-based intervention seeking to increase elementary-age 

students’ reading scores, a core activity might be delivering one-on-one tutoring using a specific reading 

curriculum in the classroom whereas a non-core, or supplemental, activity could be disseminating 

monthly newsletters to parents or caregivers with book recommendations that could be read to children 

during non-school hours. Each core activity must have a description of what it takes to produce the 

intended outcomes, including its content (for example, the service focus or curriculum topics); how it is to 

be delivered; how much of it participants should receive; the requirements for the personnel delivering it; 

and the setting in which it will take place (Blase and Fixsen 2013).5 These details provide structure that 

ensures the intervention is delivered with fidelity both before and during scaling. Without such 

specificity, new organizations, personnel, or sites may not be able to implement core activities 

consistently and with fidelity to the TOC. Such a scenario would decrease the likelihood that the 

intervention will improve participants’ outcomes to the extent expected given the intervention’s prior 

success. Four components are needed to ensure that the organization’s specification of the intervention is 

enough to facilitate successful scaling: 

• Clearly identifies core activities

• Clearly defines personnel roles

• Clearly defines the setting

• Clearly defines completion

Furthermore, the specifications of the intervention’s core activities, personnel roles, setting, and 

completion should be aligned to the research that showed evidence of the intervention’s effectiveness. If 

an organization defines these components in a way that differs from how they were defined when the 

intervention produced beneficial outcomes, then the organization might not be able to produce the same 

beneficial results when scaling the intervention. (In some cases, organizations might choose to adapt their 

interventions, which means the intervention could be different from the one that produced evidence of 

effectiveness. See Chapter IV for more information on considerations for adapting interventions.)

Organizations can use Checklist 3 to assess whether they have a well-specified intervention. For each 

row, organizations should circle the response option that best characterizes their status: a 0 indicates that 

the organization has not yet defined this aspect of its intervention; a 1 indicates that a definition does not 

exist but the organization has plans for developing it; a 2 indicates that the organization is actively 

working on defining this aspect of the intervention; and a 3 indicates that a definition for this aspect of the 

intervention is already in place. Organizations can then sum their scores for each of the four conditions to 

gauge their status with respect to having a well-specified intervention. 

5 A practitioner guide developed for the SEBM project (Maxwell 2019) is a resource organizations can use for more 

information on how to thoroughly describe their intervention: “How to Fully Describe an Intervention,” 

https://nationalservice.gov/impact-our-nation/evidence-exchange/how-fully-describe-intervention. 
Chapter III Getting Ready to Scale

Mathematica 15

Checklist 3. Specifying the intervention

Circle one 

Components

Not yet 

defined 

No, but 

making 

plans to 

develop a 

definition 

No, but 

actively 

developing 

a definition

Yes, this Is 

defined 

Summary 

score 

Identifying core activities 

Core activities (services) of the intervention 0 1 2 3 ___/9 

The mode of delivery (for example, in person or online, 

one on one or small group) for each core activity 

0 1 2 3 ___/9 

The intensity (that is, amount, frequency, and duration) of 

each core activity 

0 1 2 3 ___/9 

Clearly defining personnel roles 

The number of personnel needed to offer core 

components of the intervention as they are intended to be 

offered 

0 1 2 3 ___/9 

Roles for personnel (at the lead organization and its 

partners) implementing the intervention 

0 1 2 3 ___/9 

Qualifications for personnel (at the lead organization and 

its partners) implementing the intervention 

0 1 2 3 ___/9 

Clearly defining the setting

The geographic setting (for example, rural/urban, 

geographic region) where the intervention will be 

implemented during scaling 

0 1 2 3 ___/6 

The venue for service delivery (such as a community 

center, participants’ homes, nonprofit organization, or 

school) where the intervention will be implemented during 

scaling 

0 1 2 3 ___/6 

Clearly defining participation and completion

Who is considered a program participant (for example, is 

everyone who enrolls considered a participant, or is there 

a minimum length of time that a person must be in the 

program to be considered a participant?) 

0 1 2 3 ___/6 

Participant completion of the intervention (for example, 

minimum services completed, certification acquired, 

particular score achieved on an assessment) 

0 1 2 3 ___/6 

2. Well-defined target population

When scaling, an organization might want to offer an intervention to the same population for whom the 

intervention was designed and shown to be effective. Doing so makes it more likely that the organization 

will be successful in scaling, because interventions might not be as effective when used with populations 

or groups that are different from the target population for which the intervention demonstrated evidence. 

(At the same time, organizations might choose to target an intervention to a different population from the 

one that was targeted when the intervention produced evidence of effectiveness. For more information on 

adaptation, see Chapter IV.) To do so, the population must be well-defined. This definition must be 

specific and include the characteristics people must have to participate in the intervention (Garg 2016; 

McElroy and Ladner 2014). 

Organizations can use Checklist 4 to assess whether they have a well-defined target population. The 

checklist should be completed in the same manner as the checklist for having a well-specified 
Chapter III Getting Ready to Scale

Mathematica 16

intervention. Within each row, organizations should circle the response option that best characterizes their 

status of defining various aspects of the target population for the intervention. 

Checklist 4. Defining the target population

Circle one 

Components

Not yet 

defined 

No, but 

making 

plans to 

develop a 

definition 

No, but 

actively 

developing 

a definition

Yes, this Is 

defined 

Summary 

score 

The characteristics of the population to be served by the 

intervention (target population) 

0 1 2 3 ___/9 

The criteria for determining who is eligible to participate 

in the intervention (inclusion criteria) 

0 1 2 3 ___/9 

The criteria for determining who is not eligible to 

participate in the intervention (exclusion criteria) 

0 1 2 3 ___/9 

3. Implementation supports 

An organization must have supports in place to ensure that an intervention can be implemented with 

fidelity during scaling (Breitenstein et al. 2010; NIRN n.d.). An intervention that does not have such 

supports might not be able to produce beneficial outcomes when scaled.6 These six supports will help an 

organization implement the intervention with fidelity: 

• A monitoring system

• Performance goals

• Continuous quality improvement processes (a systematic process for improving implementation 

service delivery, processes, and outcomes)

• Preservice and in-service training

• Communication systems

• Data systems

Organizations can use Checklist 5 to assess the status of these implementation supports in place. Within 

each row, organizations should circle the response option that best characterizes their status of setting up 

these supports: a 0 indicates that the organization has not yet set up that aspect of the implementation 

support; a 1 indicates that the organization is making plans to set up that aspect; a 2 indicates that the 

organization is actively setting up that aspect; and a 3 indicates that the organization already has that 

aspect in place. Organizations can then sum their scores to gauge their status with respect to having the 

implementation supports needed for scaling their intervention.

6 Two practitioner guides developed for the SEBM project (Eddins and Needels 2019; Friend and Needels 2019) are 

resources that discuss considerations organizations should be mindful of when developing implementation supports 

for their intervention: “How to Structure Implementation Supports,” https://nationalservice.gov/impact-ournation/evidence-exchange/how-structure-implementation-supports; “Making the Most of Data,” 

https://nationalservice.gov/impact-our-nation/evidence-exchange/making-most-data. 
Chapter III Getting Ready to Scale

Mathematica 17

Checklist 5. Establishing implementation supports 

Circle one 

Components

No plans 

in place 

No, but 

making 

plans to 

set up 

No, but 

actively 

setting it 

up

Yes, this 

is in place 

Summary 

score 

Monitoring 

Personnel who will monitor the intervention (the monitoring team) 

are identified 

0 1 2 3 ___/9 

Monitoring team will assess whether core intervention 

components are implemented as specified 

0 1 2 3 ___/9 

Monitoring team will report on implementation issues or 

challenges to leadership and personnel delivering the intervention 

0 1 2 3 ___/9 

Establishing performance goals

Benchmarks for service delivery are clearly established 0 1 2 3 ___/6 

Benchmarks are monitored for achievement 0 1 2 3 ___/6 

Engaging in continuous quality improvement (CQI)

Processes for engaging in CQI are clearly articulated 0 1 2 3 ___/9 

The intervention will be part of conducted CQI efforts 0 1 2 3 ___/9 

The intervention procedures will be updated based on results of 

the CQI process 

0 1 2 3 ___/9 

Conducting preservice and in-service training 

Training to be taken before scaling the intervention begins is 

clearly articulated (preservice training) 

0 1 2 3 ___/12 

All personnel needing preservice training will receive it in a timely 

manner 

0 1 2 3 ___/12 

Training needed by individuals working on the intervention is 

clearly articulated (in-service training) 

0 1 2 3 ___/12 

All personnel needing in-service training will receive it in a timely 

manner 

0 1 2 3 ___/12 

Establishing communication systems 

A communication system exists that supports coordination among 

personnel working on the intervention 

0 1 2 3 ___/9 

A process exists to ensure that communications will function 

smoothly and as intended 

0 1 2 3 ___/9 

A process exists to resolve communication problems shortly after 

they occur 

0 1 2 3 ___/9 

Establishing data systems 

A data system supports collection, analysis, and decision making 

about the intervention 

0 1 2 3 ___/12 

A process exists to ensure that data are entered into the system 

in a timely manner and are of high quality 

0 1 2 3 ___/12 

A process exists to ensure that data are analyzed appropriately 0 1 2 3 ___/12 

A process exists to ensure that data are used in decision making 0 1 2 3 ___/12 
Chapter III Getting Ready to Scale

Mathematica 18

B. Is my organization ready to scale?

Even if the intervention is ready to be scaled, the organization might not be able to support the scaling. 

As a result, scaling might not be successful. For an organization to support successful scaling, it must 

have an environment that is conducive to scaling and have sufficient supports in place.7 Meeting these 

final two conditions for scaling readiness indicates that an organization is ready to successfully scale an 

intervention. 

1. Enabling context 

To foster successful scaling of effective interventions, an organization’s leadership and culture must 

support innovation, learning, and improvement (Metz 2016). (External context can also play a role in 

scaling; considerations related to external context are discussed in Chapter IV.) For example, 

organizations with a history of engaging in efforts to improve upon their intervention can apply those 

experiences to adjust how intervention services are being delivered as the intervention is scaled. Although 

an organization’s enabling context can develop in different ways, having successfully tackled challenges 

in the past is one way a supportive context can develop. The organization’s structures, roles, and 

functions should also facilitate (rather than hinder) service delivery as well as the achievement of 

beneficial participant outcomes. Two components will enable an organization to scale an intervention: 

• Support by organization leaders and key stakeholders

• A culture of innovation and learning

Organizations can use Checklist 6 to assess whether their context will likely enable successful scaling. 

Within each row, organizations should circle the response option that best describes the extent to which 

they meet the characteristic: a 0 indicates that the aspect of enabling context is not at all present at the 

organization; a 1 indicates that the aspect somewhat describes the organization; a 2 indicates that the 

aspect mostly describes the organization; and a 3 indicates that the aspect fully describes the organization. 

Organizations can then sum their scores to gauge their organizations’ status with respect to having the 

enabling context needed for scaling their intervention.

7 A practitioner guide developed for the SEBM project (Jones and Needels 2019) is a resource organizations can use 

for more information on the enabling context and implementation infrastructure conditions organizations needs to 

implement their intervention: “Build Organizational Capacity to Implement an Intervention,” 

https://nationalservice.gov/impact-our-nation/evidence-exchange/build-organizational-capacity-implementintervention. 
Chapter III Getting Ready to Scale

Mathematica 19

Checklist 6. Having an enabling context

Circle one 

Components

Not at all 

descriptive 

of the 

organization

Somewhat 

descriptive 

of the 

organization

Mostly 

descriptive 

of the 

organization

Fully 

descriptive 

of the 

organization

Summary 

score 

Supporting the intervention 

Organizational leaders demonstrate support for the 

intervention 

0 1 2 3 ___/6 

Key stakeholders demonstrate support for the 

intervention 

0 1 2 3 ___/6 

Engaging in innovation and learning

The organization has an established history of innovation 0 1 2 3 ___/12 

The organization is innovative in finding ways to improve 

the intervention model 

0 1 2 3 ___/12 

The intervention's successes are discussed among 

personnel involved with the intervention 

0 1 2 3 ___/12 

The intervention's challenges are discussed among 

personnel who can help to overcome them 

0 1 2 3 ___/12 

2. Implementation infrastructure

An organization’s infrastructure must ensure that it has sufficient financial, human, and physical 

resources to support the intervention (Bernfeld 2006; Fixsen et al. 2009; Klingner et al. 2003) and its 

successful implementation (Mihalic and Irwin 2003) during scaling. To effectively support the scaled 

intervention, the organization’s infrastructure must facilitate hiring the personnel necessary for scaling, 

support supervision and personnel development through a human resources management system, and 

provide funding and other needed resources for implementing the intervention (for example, materials and

physical space). Of note, infrastructure could include resources external to the organization. For example, 

if partners play a key role in implementation, their policies, priorities, and systems must also support 

successful scaling of the intervention. Two support components will help an organization implement the 

intervention with fidelity: 

• Financial and human resources 

• Materials and physical space

Organizations can use Checklist 7 to assess whether they have the infrastructure in place to support 

scaling. The checklist should be completed in the same manner as the checklist for having implementation 

supports for the intervention. 
Chapter III Getting Ready to Scale

Mathematica 20

Checklist 7. Establishing an implementation infrastructure 

Circle one 

Components

No plans 

in place 

No, but 

making 

plans to 

set up 

No, but 

actively 

setting it 

up

Yes, this 

is in place 

Summary 

score 

Procuring financial and human resources 

The organization has adequate funding for the intervention 0 1 2 3 ___/9 

The organization provides enough dedicated personnel for the 

intervention 

0 1 2 3 ___/9 

A human resource system is in place to hire, supervise, and 

develop personnel implementing the intervention 

0 1 2 3 ___/9 

Providing materials and physical space 

A thorough description of the materials and physical space 

needed to implement the intervention exists 

0 1 2 3 ___/9 

The materials needed to implement the intervention are available 0 1 2 3 ___/9 

Enough physical space exists for implementing the intervention 0 1 2 3 ___/9 

C. Summarizing across checklists

Checklist 8, a summary checklist for scaling readiness, can help organizations quickly identify areas in 

which such assistance might be needed. This summary checklist uses information from each scaling 

readiness condition checklist (listed in bold in the summary checklist) and the various components within

each condition. As a reminder, Checklist 8 and the other checklists in this chapter are appropriate for 

organizations that are seeking to scale an intervention that already has rigorous evidence of effectiveness. 

Organizations that are seeking to establish evidence of effectiveness for their intervention should 

complete the process and checklist described in Chapter II.

To complete this form, organizations should enter the summary score they received for each of the 

various components within each scaling readiness condition. Organizations should then tally these scores 

to create a total summary score (which can be entered into the box highlighted with a border) for each 

condition. The closer a summary score is to the maximum total for that condition, the more ready for 

scaling the intervention or organization appears to be, in that condition. 

These scores can quickly identify the intervention or organization conditions in which an intervention or 

organization might not yet be ready to scale. Organizations can then develop the needed procedures or 

processes to build their capacity in these areas or look to identify new partners to help gain this capacity. 
Chapter III Getting Ready to Scale

Mathematica 21

Checklist 8. Summary checklist

Evidence of effectiveness Guidance

For identifying an effective intervention, are all items under research rigor, 

intervention effectiveness, and intervention relevance marked yes? 

Yes: Proceed to scaling readiness conditions 

No: Proceed to building evidence 

For building evidence of effectiveness for your intervention, are all items 

under define intervention, lay evaluation groundwork, prepare for evaluation, 

and conduct evaluation and assess effectiveness marked yes? 

Yes: Proceed to scaling readiness conditions 

No: Continue building evidence for your 

intervention 

Scaling readiness conditions Summary scores 

Intervention

Well-specified intervention

Identifying core components ___/9 

Clearly defining personnel roles ___/9 

Clearly defining the setting ___/6 

Clearly defining completion ___/6 

TOTAL SUMMARY SCORE ___/30 

Well-defined target population

TOTAL SUMMARY SCORE ___/9 

Implementation supports

Monitoring implementation ___/9

Establishing performance goals ___/6

Engaging in continuous quality improvement ___/9

Conducting preservice and in-service training ___/12

Establishing communication systems ___/9

Establishing data systems ___/12

TOTAL SUMMARY SCORE ___/57

Organization

Enabling context

Supporting the intervention ___/6

Engaging in innovation and learning ___/12

TOTAL SUMMARY SCORE ___/18

Implementation infrastructure

Procuring financial and human resources ___/9

Providing materials and physical space ___/9 

TOTAL SUMMARY SCORE ___/18 

D. Completing and using the SCALER 

When using the SCALER, organization personnel should seek to capture full and complete information 

on each of the conditions for scaling readiness. Because personnel at all levels can be involved in 

intervention scaling, organizations might want to involve multiple personnel members in the process of 

completing the SCALER and discussing its results, including identifying and resolving, through 

discussion, any differences of opinion. Organization members who might be well-positioned to help 

complete the SCALER include: 

• Executive directors, who might be most knowledgeable about implementation supports and the 

enabling context
Chapter III Getting Ready to Scale

Mathematica 22

• Program managers or frontline supervisors, who could have most familiarity with how the 

intervention and target population are defined, in addition to the other conditions

• Measurement, learning, and evaluation personnel (such as data managers or internal evaluation 

specialists), who might have the most knowledge of existing evidence of an intervention’s 

effectiveness, or building evidence of effectiveness, as well as the implementation supports related to 

monitoring, CQI, performance goals, and data systems

• Administrative personnel, such as human resources, development, or purchasing personnel, who 

might have firsthand knowledge of the organization’s implementation infrastructure 

Although it might be prudent to have multiple personnel involved in completing the SCALER, to ensure 

that someone has ownership over the tool, we recommend designating one personnel member or role to 

be in charge of the SCALER—such as the intervention’s program manager or the measurement, learning, 

and evaluation personnel member. That person would track whether and when the SCALER needs to be 

completed, identify and bring together the appropriate personnel to complete it, have personnel complete 

it, synthesize the scores and produce output to share with personnel on action steps to address the 

SCALER results, and then determine when the tool should be filled out again. Exhibit III.1 offers a 

suggested process that organizations can take to complete the SCALER on an ongoing basis or as needed.

Exhibit III.1. Process for completing the SCALER

The personnel member who has ownership over the SCALER (“the SCALER owner”) should take 

these steps to complete it as needed:

Who at the organization should complete the SCALER?

• The SCALER owner should consult with appropriate organizational personnel—such as leaders and the 

intervention program manager—to decide who should help to fill out the SCALER.

• Once these personnel are identified, the SCALER owner will let them know they are responsible for helping to fill 

out the tool and the deadline for doing so.

• The SCALER owner should set up a timeline for completing the tool.

How should organizational personnel complete the SCALER?

• We recommend that the SCALER owner have each personnel member fill out his or her own version of the tool 

and then set up a meeting for everyone to discuss and come to agreement on scores.

• Organizations might prefer to take different approaches, such as having several personnel fill it out together while 

discussing it during a meeting.

What happens after the SCALER is completed?

• After scoring is completed, the SCALER owner documents the scores and shares them with key stakeholders 

and organizational leaders involved with preparing to scale. 

• Working with key stakeholders and organizational leaders, the SCALER owner can also suggest next steps—for 

example, defining the target population if it is found to lack a definition.

When should the SCALER be filled out again?

• Depending on the priorities of the organization, its interest in or plans for scaling, and any upcoming changes to 

the organization or its external context, the SCALER owner should identify an appropriate time to complete the 

SCALER again.

• The SCALER is intended to be a living document, and reflective of one moment in time. As changes or progress 

is being made to scaling readiness conditions, the document should be updated accordingly.
Chapter III Getting Ready to Scale

Mathematica 23

As noted in Exhibit III.1, personnel can identify next steps to strengthen scaling readiness as a result of 

the scores. Some discussion questions that personnel might want to discuss in interpreting their scores and 

identifying what actions to take as a result of completing the SCALER are:

• In what areas are we the strongest? In which areas do we appear to need work? Are there specific 

items (rows) on which we should concentrate our efforts for improvement? (This may include 

identifying evidence of effectiveness, building evidence of effectiveness, or any of the scaling 

readiness conditions.)

• Do we need any additional information to complete any parts of the SCALER? Are there any items 

that we don’t know the answers to? If so, how can we go about answering those items?

• In what conditions or areas, if any, do we have differences of opinion? What information can we draw 

upon to resolve those differences?

• Who are the stakeholders (organization personnel or external stakeholders) with whom we should 

discuss these results (that is, people who were not involved in completing the SCALER)? 

• What next steps should we take to identify or build evidence of intervention effectiveness, or 

strengthen our scaling readiness (as appropriate)? Which steps are highest and lowest priority? What 

is our timeline for taking action? 

• What partners could we bring in to build capacity in certain components? Should we seek to identify 

new partners to help gain this capacity? 

The frequency with which organizations complete the SCALER depends on their context and priorities. 

Organizations that score low in certain items might want to complete the SCALER again soon after 

making changes to address those low-scoring items. Organizations that have higher scores overall might 

want to complete the SCALER again after implementing changes, such as personnel restructuring, or after 

some external event, such as receiving a large grant that allows it to bolster its implementation supports 

and infrastructure. The frequency with which an organization completes the SCALER depends on its 

intervention’s and organization’s scaling readiness and other factors, but in general, completing the 

SCALER two or three times a year can allow organizations to closely track their scaling readiness and 

needs to strengthen scaling readiness over time.
This page has been left blank for double-sided copying. 
Mathematica 25

IV. Ready, Set, Scale?

Successful scaling requires planning and preparation. This guide—with the SCALER framework and 

checklists—was developed to help organizations do both. In Chapters II and III, we discussed the 

hallmarks of an ideal scenario—what would happen if an organization wanted to scale an intervention 

that has been evaluated in a rigorous, well-designed study that shows evidence of effectiveness, and both 

the intervention and the organization meet all the conditions for scaling readiness. This ideal scenario is 

intended to help organizations understand what considerations they should weigh before scaling and 

determine whether scaling efforts should begin. 

However, we recognize that no organization operates under ideal circumstances. In understanding the 

ideal, organizations will be better equipped to identify the trade-offs and compromises that must be made 

in a world that is far from ideal. This chapter covers practical guidance for scaling by discussing 

constraints that organizations might face when seeking to establish evidence of their interventions. We 

also discuss how the SCALER framework can be applied when circumstances might lead organizations to 

scale before they are ready to do so. 

A. Establishing evidence of effectiveness may not be possible at the moment

All research and evaluation efforts include trade-offs and concessions when they are being designed and 

implemented. Organizations utilize these trade-offs and concessions to structure research in a way that 

best serves their needs within the time and resources that they have. Organizations might face some of the 

following scenarios, each of which inhibits them from conducting an evaluation that meets the evidence

standards of rigor and objectivity required by research clearinghouses. Yet, they can still conduct research 

that can strengthen their interventions and prepare them for a more rigorous evaluation in the future.

1. The intervention is not ready for rigorous evaluation because it is not yet mature. As noted in 

Chapter II, research cannot accurately assess whether an intervention is effective until the 

intervention is mature. Indeed, new and innovative intervention models may not be ready for 

evaluation for years. When an organization develops and offers a new intervention model, it can build 

a research plan and process that includes formative and outcome evaluations to gather the information 

needed to refine the intervention. These types of evaluations can serve as the foundation for a 

subsequent, more rigorous evaluation of the intervention’s impact (Bertram et al. 2011). During a 

formative evaluation, organizations typically seek to determine if an intervention is feasible before 

fully implementing it. To do so, an organization might interview a range of stakeholders about their 

perceptions of the intervention and how to strengthen it, and draw on data about whether participants 

are satisfied with the intervention as well as whether they are engaging with services as intended 

(U.S. Centers for Disease Control and Prevention n.d.). For an outcome evaluation, organizations can 

use the intervention’s TOC to identify measures for each TOC component (inputs, activities, outputs, 

and outcomes), collect data for each measure, and use those data to monitor whether the intervention 

is achieving the desired outcomes (Connell and Kubisch 1998). If it is not achieving the desired 

outcomes, organizations can use the data collected on other components to assess what might need to 

change to improve outcomes. Organizations might decide that additional or different inputs are 

needed, or that the intervention’s activities need to be tweaked, for example. 

2. The organization does not have resources for a rigorous evaluation but wants to monitor 

intervention performance. An organization might have an intervention that is ready or close to 

ready for a rigorous evaluation but lacks the resources to carry out that evaluation. Continuing to 

demonstrate that the intervention is making progress toward its intended outcomes can help an 
Chapter IV Ready, Set, Scale?

Mathematica 26

organization continue to build the intervention’s proof of concept while it seeks funding for a rigorous 

study. For example, funders often require organizations to provide data that track their progress 

toward outcomes; these data can also be used to demonstrate an intervention’s promise. Although 

these data are sometimes funder specific and may or may not be consistent with the data the 

organizations would otherwise choose to collect, they can still be used to generate insights for 

monitoring intervention performance. The trick is to convert the data into usable information (Slotnik 

and Orland 2010). For example, a funder might require organizations to administer periodic 

assessments to participants as a way of monitoring growth. In addition to reporting results of these 

assessments to the funder, organizations might also use them as a case management tool to discuss 

progress with individual participants, or use them in discussions with case managers to explain why 

progress is or is not as expected. 

Although it may seem that directing monetary resources and personnel toward research and 

evaluation efforts can reduce the capacity of an organization to serve participants, using information 

from such efforts within a culture of data-driven decision making increases performance (LaValle et 

al. 2010), output, and productivity (Brynjolfsson et al. 2011). Furthermore, using data to monitor and 

report on performance can lay the groundwork for future scaling efforts by uncovering how the 

intervention might be strengthened and what types of adaptations might be appropriate in the context 

of scaling (Manno and Gaubert 2018). 

3. The intervention is hard to rigorously evaluate because of the circumstances of the target 

population being served or some other ethical challenge. An intervention might be hard to 

evaluate in a rigorous manner because its target population is hard to reach or otherwise hard to 

measure using traditional techniques such as surveys. For example, an intervention that attempts to 

reduce drug use could face the challenge of locating people for whom the intervention did not work; 

as a result, the intervention’s true outcomes cannot be captured. The evaluation would not meet 

standards for rigor because of attrition. In addition, some organizations may not be able to use 

random assignment to create comparison groups, which are needed for establishing causal impact. 

Legislation might mandate that members of a certain class of individuals receive an intervention, for 

instance, which precludes randomly assigning some people in that class to a comparison group. 

Likewise, organizations or their funders may have ethical concerns about withholding services from 

those who need them for the sake of using random assignment to conduct research. Under these 

circumstances, organizations might want to employ quasi-experimental design (QED) studies, which 

allow them to avoid random assignment—while maintaining rigor and objectivity—to measure 

intervention impact. 

B. The organization might have an opportunity to scale before it is ready

Sometimes, organizations have an opportunity to scale an intervention before they are ready to do so. For 

instance, political pressures to scale an intervention into a new community or for a new target population 

might arise before the intervention is completely mature. Or funders might offer resources to scale an 

intervention today, and the funding may not be available next year when the organization feels it might be 

in a better position to scale successfully. 

Under such circumstances, the checklists in this guide can help organizations identify areas in which they 

might need support to successfully scale the intervention. Such support may come from partnering with 

other organizations or from additional funding for scaling efforts. For example, an organization scaling a 

promising after-school intervention for youth might have limited funds and not enough physical space for 

all the new students it expects to enroll in the next year. The organization may want to partner with the 
Chapter IV Ready, Set, Scale?

Mathematica 27

school district, local community-based organizations that serve youth, or other local systems to acquire 

that space for low or no cost. Organizations might also want to draw on outside expertise to help 

strengthen some of their implementation supports—for example, technical assistance providers or 

consultants can help to set up data systems, establish continuous quality improvement processes, and 

provide appropriate personnel training. Using the checklists, an organization might also realize that some 

aspect of its intervention or target population is not clearly defined, and it may choose to engage its 

personnel in an exercise to develop or refine its TOC to clarify those definitions.

Organizations can encounter many reasons for scaling before they have met all of the scaling readiness 

conditions. This guide should help organizations prioritize their next steps to strengthen their scaling 

efforts and improve their chances for success. The SCALER framework can be used to help organizations 

navigate their own circumstances and decide where to place resources and attention to help put them on a 

path to successfully scaling their interventions. After scaling, organizations should continue to evaluate 

their intervention’s effectiveness to ensure it is maintaining benefits as expected for participants. If an 

organization finds that its intervention is no longer producing impacts as expected, it might draw on the 

SCALER to understand if it should seek to improve aspects of scaling readiness.

C. The organization is seeking to adapt an intervention

Adaptation is a type of scaling that occurs when an intervention is extended to a different target 

population in either the same or different location or modified for the same population in either the same 

or different location. An organization may seek to adapt an intervention in the hopes of improving upon 

participant outcomes. It might also make adaptations due to external contextual factors—for instance, 

funding might become available to target a new population, and an organization can seek to adapt its 

intervention to apply for and use that funding.

The field of implementation science has not yet come to a consensus around the limits of adaptation. In 

other words, we cannot readily determine when an intervention can be considered to be adapted versus 

when it is considered to be a new or different intervention. However, in general, when an organization 

changes an intervention’s core activities or how they are implemented—for example, modifies the 

activities’ dosage or intensity—it runs the risk of no longer scaling the intervention that produced 

evidence of effectiveness. While most adaptations should be tested to understand how they might change 

the effectiveness of an intervention, the following types of adaptations can typically be made while 

maintaining the integrity of an intervention: 

• Modifying or adding non-core components to an intervention. An intervention’s core activities 

are those that are deemed critical for achieving beneficial outcomes (see Chapter III, Section A). 

Using the example given in Chapter III—a school-based intervention to increase elementary-age 

students’ reading scores—an organization might choose to supplement a core, in-school reading 

curriculum with sending out monthly newsletters to parents and caregivers that suggest books they 

can read to children at night or on the weekends.

• Making the intervention accessible to the target population. Making the intervention accessible 

can consist of translating materials only offered in English for non-English speakers or making 

modifications related to a target population’s culture. Using the same example as before, the reading 

curriculum could be translated from English into Spanish, while keeping the content the same. Or, 

examples in the curriculum might be changed so they are more relatable to the new target 

population—for instance, changing mentions of types of foods to ones that the children in the target 

population are likely to eat. 
This page has been left blank for double-sided copying. 
Mathematica 29

References

Anderson, Mary Anne, and Nan Maxwell. “Baseline Equivalence: What It Is and Why It Is Needed.” 

Paper submitted to AmeriCorps. Chicago, IL: Mathematica, September 2018. Available at

https://nationalservice.gov/impact-our-nation/evidence-exchange/baseline-equivalence-what-it-andwhat-it-needed. Accessed September 14, 2020.

Anderson, Mary Anne, and Nan Maxwell. “What Makes for a Well-Designed, Well-Implemented Impact 

Study.” Paper submitted to AmeriCorps. Chicago, IL: Mathematica, May 2019. Available at

https://nationalservice.gov/impact-our-nation/evidence-exchange/what-makes-well-designed-wellimplemented-impact-study. Accessed September 14, 2020.

Austin, M. J., and J. Claassen. “Impact of Organizational Change on Organizational Culture.” Journal of 

Evidence-Based Social Work, vol. 5, nos. 1–2, 2008a, pp. 321–359. doi:10.1300/J394v05n01_12. 

Austin, M. J., and J. Claassen. “Implementing Evidence-Based Practice in Human Service 

Organizations.” Journal of Evidence-Based Social Work, vol. 5, nos. 1–2, 2008b, pp. 271–293. 

doi:10.1300/J394v05n01_10. 

Bernfeld, G. A. “The Struggle for Treatment Integrity in a ‘Dis-integrated’ Service Delivery System.” The 

Behavior Analyst Today, vol. 7, no. 2, 2006, pp. 188–205. doi:10.1037/h0100086.

Bertram, Rosalyn, Karen Blase, David Shern, Pat Shea, and Dean Fixsen. “Policy Research Brief: 

Implementation Opportunities and Challenges for Prevention and Promotion Initiatives.” Alexandria, 

VA: National Association of State Mental Health Program Directors (NASMHPD), 2011. Available at 

https://www.researchgate.net/publication/233885310_Policy_Research_Brief_Implementation_Opportun

ities_and_Challenges_for_Prevention_and_Promotion_Initiatives. Accessed November 5, 2019.

Blase, K., and D. Fixsen. “Core Intervention Components: Identifying and Operationalizing What Makes 

Programs Work.” Washington, DC: Office of the Assistant Secretary for Planning and Evaluation, 

Office of Human Services Policy, U.S. Department of Health and Human Services, 2013.

Breitenstein, S. M., D. Gross, C. Garvey, C. Hill, L. Fogg, and B. Resnick. “Implementation Fidelity in 

Community-Based Interventions.” Research in Nursing and Health, vol. 33, no. 2, 2010, pp. 164–

173. doi:10.1002/nur.20373. 

Brynjolfsson, E., L. M. Hitt, and H. H. Kim. “Strength in Numbers: How Does Data-Driven 

Decisionmaking Affect Firm Performance?” Social Science Research Network, April 22, 2011. 

Available at http://dx.doi.org/10.2139/ssrn.1819486. Accessed June 13, 2013. 

Connell, James P., and Anne C. Kubisch. “Applying a Theory of Change Approach to the Evaluation of 

Comprehensive Community Initiatives: Progress, Prospects, and Problems.” Washington, DC: Aspen 

Institute, 1988. Available at 

http://www.dmeforpeace.org/sites/default/files/080713%20Applying+Theory+of+Change+Approach.

pdf. Accessed December 26, 2019.

Despard, M. “Challenges in Implementing Evidence-Based Practices and Programs in Nonprofit Human

Services Organizations.” Journal of Evidence-Informed Social Work ̧ vol. 13, no. 6, 2016, pp. 502–

522. doi:10.1080/23761407.2015.1086719. 

Donkoh, C., K. Undershill, and P. Montgomery. “Independent Living Programs for Improving Outcomes 

for Young People Leaving the Care System.” Campbell Systematic Reviews, Campbell 

Collaboration, 2006. Available at https://www.cochrane.org/CD005558/BEHAV_independent-livingprogrammes-for-improving-outcomes-for-young-people-leaving-the-care-system. 
References

Mathematica 30

Eddins, Katie, and Karen Needels. “How to Structure Implementation Supports.” Paper submitted to 

AmeriCorps. Chicago, IL: Mathematica, September 2019. Available at 

https://nationalservice.gov/impact-our-nation/evidence-exchange/how-structure-implementationsupports. Accessed September 14, 2020.

Fixsen, A. M. “Defining Scaling Up Across Disciplines: An Annotated Bibliography.” Semantic Scholar, 

2009. Available at 

https://pdfs.semanticscholar.org/57fc/57337970444b7bfc536adee7d0791a36471b.pdf. Accessed 

November 23, 2018.

Fixsen, D. L., S. F. Naoom, K. A. Blase, R. M. Friedman, and F. Wallace. “Implementation Research: A 

Synthesis of the Literature.” FMHI Publication #231. Tampa, FL: University of South Florida, Louis 

de la Parte Florida Mental Health Institute, The National Implementation Research Network, 2005. 

Available at https://nirn.fpg.unc.edu/resources/implementation-research-synthesis-literature. 

Accessed November 28, 2018.

Friend, Daniel, and Karen Needels. “Making the Most of Data.” Paper submitted to AmeriCorps. 

Chicago, IL: Mathematica, September 2019. Available at https://nationalservice.gov/impact-ournation/evidence-exchange/making-most-data. Accessed September 14, 2020.

Garg, R. “Methodology for Research I.” Indian Journal of Anaesthesia, vol. 60, no. 9, 2016, pp. 640–645. 

doi:10.4103/0019-5049.190619. 

Greene, J. C., V. Caracelli, and W. F. Graham. “Toward a Conceptual Framework for Mixed-Method 

Evaluation Designs.” Educational Evaluation and Policy Analysis, vol. 11, no. 3, 1989, pp. 255–274.

Gondolf, E. W. “The Evidence-Based Practice Movement: Contributions, Controversies, and 

Recommendations.” In Emerging Trends in the Social and Behavioral Sciences, edited by R. A. 

Scott, S. M. Kosslyn, and N. Pinkerton. Hoboken, NJ: John Wiley & Sons, 2015. 

Hernandez, Mario, and Sharon Hodges. “Building Upon the Theory of Change for Systems of Care.” 

Journal of Emotional and Behavioral Disorders, vol. 11, no. 1, 2003, pp. 19–26.

Hodges, Sharon, Mario Hernandez, Teresa Nessman, and Lodi Lipien. “Creating Change and Keeping It 

Real: How Excellent Child-Serving Organizations Carry Out Their Goals. Cross-site Findings for 

Phase I Community-Based Theories of Change.” Tampa, FL: Louis de la Parte Florida Mental Health 

Institute, Research and Training Center for Children’s Mental Health, University of South Florida, 

December 2002. Available at http://rtckids.fmhi.usf.edu/rtcpubs/creatingchange/CreatingChange.pdf. 

Accessed November 5, 2019. 

Jones, Christopher, and Karen Needels. “Build Organizational Capacity to Implement an Intervention.” 

Paper submitted to AmeriCorps. Chicago, IL: Mathematica, September 2019. Available at 

https://nationalservice.gov/impact-our-nation/evidence-exchange/build-organizational-capacityimplement-intervention. Accessed September 14, 2020.

Klingner, J., S. Ahwee, P. Pilonieta, and R. Menendez. “Barriers and Facilitators in Scaling Up ResearchBased Practices.” Exceptional Children, vol. 69, no. 4, 2003, pp. 411–429.

Larson, S. R., J. W. Dearing, and T. E. Backer. “Strategies to Scale Up Social Programs: Pathways, 

Partnerships, and Fidelity.” New York, NY: The Wallace Foundation, 2017. Available at 

https://www.wallacefoundation.org/knowledge-center/pages/how-to-scale-up-social-programs-thatwork.aspx. Accessed November 28, 2018.

LaValle, S., E. Lesser, R. Shockley, M. S. Hopkins, and N. Kruschwitz. “Big Data, Analytics and the 

Path from Insights to Value.” MIT Sloan Management Review, vol. 52, no. 2, 2010, pp. 21–31.
References

Mathematica 31

Lipsky, M. Street-Level Bureaucracy. New York, NY: Russell Sage Foundation, 1980.

Lipsey, M. “Statistical Conclusion Validity for Intervention Research: A Significant (p <.05) Problem.”

In Validity and Social Experimentation (vol. 1 of Donald Campbell’s Legacy), edited by L. Bickman. 

Thousand Oaks, CA: Sage, 2000.

Macallair, Daniel, and Mike Males. “A Failure of Good Intentions: An Analysis of Juvenile Justice 

Reform in San Francisco during the 1990s.” Review of Policy Research, vol. 21, no. 1, 2004, pp. 63–

78.

Manno, Michelle S., and Jennifer Miller Gaubert. “How Early Implementation Research Can Inform 

Program Scale-Up Efforts.” Implementation Research Incubator Blog, 2018. Available at 

https://www.mdrc.org/publication/how-early-implementation-research-can-inform-program-scaleefforts. Accessed November 5, 2019.

Maxwell, Nan. “How to Fully Describe an Intervention.” Paper submitted to AmeriCorps. Chicago, IL: 

Mathematica, May 2019. Available at https://nationalservice.gov/impact-our-nation/evidenceexchange/how-fully-describe-intervention. Accessed September 14, 2020.

Maxwell, Nan L., and Scott Richman. “Scaling Programs with Research Evidence and Effectiveness 

(SPREE).” Foundation Review, vol. 11, no. 3, November 2019, pp. 55–67. Available at 

https://nationalservice.gov/impact-our-nation/evidence-exchange/scaling-programs-researchevidence-and-effectiveness. Accessed September 14, 2020.

McElroy, L. M., and D. P. Ladner. “Defining the Study Cohort: Inclusion and Exclusion Criteria.” In 

Success in Academic Surgery: Clinical Trials, edited by T. Pawlik and J. Sosa. London: Springer, 

2014. 

Metz, Allison. “Practice Profiles: A Process for Capturing Evidence and Operationalizing Innovations.” 

Chapel Hill, NC: National Implementation Research Network, 2016. Available at 

https://nirn.fpg.unc.edu/sites/nirn.fpg.unc.edu/files/resources/NIRN-Metz-WhitePaperPracticeProfiles.pdf. Accessed September 25, 2020.

Mihalic, S., and K. Irwin. “Blueprints for Violence Prevention: From Research to Real-World Settings—

Factors Influencing the Successful Replication of Model Programs.” Youth Violence and Juvenile 

Justice, vol. 1, no. 1, 2003, pp. 1–23.

Miller, W. R., J. L. Sorensen, J. A. Selzer, and G. S. Brigham. “Disseminating Evidence-Based Practices 

in Substance Abuse Treatment: A Review with Suggestions.” Journal of Substance Abuse Treatment, 

vol. 31, no. 1, 2006, pp. 25–39.

National Implementation Research Network. “The Hexagon: An Exploration Tool.” August 2018. 

Available at https://nirn.fpg.unc.edu/resources/hexagon-exploration-tool. Accessed January 17, 2019.

National Implementation Research Network. “Implementation Drivers.” n.d. Available at 

https://nirn.fpg.unc.edu/module-1/implementation-drivers. Accessed September 25, 2020. 

Patton, M. Q. Developmental Evaluation. New York, NY: Guildford Press, 2010.

Rogers, P. “Using Program Theories to Evaluate Complicated and Complex Aspects of Interventions.” 

Evaluation, vol. 14, no. 1, 2008, pp. 29–48. 

Slotnik, William J., and Martin Orland. “Data Rich but Information Poor.” Education Week, May 6, 2010. 

Available at https://www.edweek.org/technology/opinion-data-rich-but-information-poor/2010/05. 

Accessed December 26, 2019.
References

Mathematica 32

Stame, Nicholetta. “What Doesn’t Work? Three Failures, Many Answers.” Evaluation, vol. 16, no. 4, 

2010, pp. 371–387. https://doi.org/10.1177/1356389010381914. 

U.S. Centers for Disease Control and Prevention. “Building Our Understanding: Key Concepts of 

Evaluation.” Available at 

https://www.cdc.gov/nccdphp/dch/programs/healthycommunitiesprogram/tools/pdf/eval_planning.pdf. 

Accessed August 7, 2020.

W.K. Kellogg Foundation. “Logic Model Development Guide.” Battle Creek, MI: W.K. Kellogg 

Foundation, 2004. Available at https://www.wkkf.org/resource-directory/resources/2004/01/logicmodel-development-guide. Accessed May 23, 2020.

Walker, B., and S. A. Soule. “Changing Company Culture Requires a Movement, Not a Mandate.” 

Harvard Business Review, June 20, 2017, pp. 2–6. Available at https://hbr.org/2017/06/changingcompany-culture-requires-a-movement-not-a-mandate. Accessed November 28, 2018.

Yampolskaya, Svetlana, Teresa M. Nesman, Mario Hernandez, and Diane Koch. “Using Concept 

Mapping to Develop a Logic Model and Articulate a Program Theory: A Case Example.” American 

Journal of Evaluation, vol. 25, no. 2, 2004, pp. 191–207. 

Zandniapour, L., and N. Vicinanza. “Budgeting for Rigorous Evaluation: Insights from the Social 

Innovation Fund.” Washington, DC: AmeriCorps, October 2013. Available at 

https://www.nationalservice.gov/sites/default/files/documents/Budgeting_for_Evaluation.pdf.

Accessed January 17, 2019.

Zandniapour, L., and JBS International. “Impact Evaluability Assessment Tool.” Washington, DC: 

AmeriCorps, 2014. Available at 

https://www.nationalservice.gov/sites/default/files/resource/Laying_the_Groundwork_Before_Your_

First_Evaluation_Tool.pdf. Accessed May 4, 2020.
APPENDIX

The SCALER Assessment Tool
This page has been left blank for double-sided copying. 
Appendix: The SCALER Assessment Tool

Mathematica 35

The SCALER Assessment Tool

This appendix compiles the eight checklists included in the Scaling Checklists: Assessing your Level of 

Evidence and Readiness, or SCALER. Organizations can use these checklists to determine if they have 

(1) evidence of their intervention’s effectiveness that meets industry standards for rigorous evaluation 

research and (2) the procedures and processes in place that will enable them to successfully scale an 

intervention. This appendix also offers a suggested process for implementing the SCALER as well as a 

guide that organizations can use to structure their discussions about the results of the SCALER and 

identify next steps.

SCALER Checklists 

Checklist 1. Identifying an effective intervention (also see p. 9) 

Checklist 2. Building evidence of effectiveness (also see p. 13) 

Checklist 3. Specifying the intervention (also see p. 17) 

Checklist 4. Defining the target population (also see p. 18) 

Checklist 5. Establishing implementation supports (also see p. 19) 

Checklist 6. Having an enabling context (also see p. 21) 

Checklist 7. Establishing an implementation infrastructure (also see p. 22) 

Checklist 8. Summary of scaling readiness (also see p. 23) 

Process for completing the SCALER (also see p. 24) 

Discussion guide for the SCALER (also see p. 25) 
Appendix: The SCALER Assessment Tool

Mathematica 36

Checklist 1. Identifying an effective intervention

Organizations can use this checklist to assess 

whether their intervention has evidence of 

effectiveness. Should one or more of the 

answers in the checklist be a “no,” meaning 

the organization does not have evidence of 

the intervention’s effectiveness, such

evidence must be built (see Checklist 2). 

Should all answers be a “yes,” the 

organization should next determine its 

readiness to scale the intervention (see 

Checklist 3). 

Checklist 1. Identifying an effective intervention

Check one

Components Yes No 

1. Research rigor 

Research conducted by a third party ○ ○

Research includes a counterfactual ○ ○

Research has no confounding factors ○ ○

Research has low attrition, no reassignment, and 

baseline equivalence with final sample 

○ ○

2. Intervention effectiveness 

The research showed evidence of effectiveness ○ ○

3. Intervention relevance 

The research is appropriate to the context or 

environment in which scaling is planned (the 

research is generalizable or the intervention will 

be scaled within the specific environment studied) 

○ ○

If “no” in one or more categories, there is a continued need for 

evidence (see Chapter II, Section B)

If “yes” in all categories, determine readiness to scale (see Chapter III)
Appendix: The SCALER Assessment Tool

Mathematica 37

Checklist 2. Building evidence of effectiveness 

Organizations can use this checklist as they 

seek to build evidence of effectiveness for 

their interventions. The checklist assesses 

four groups of conditions: (1) whether the 

intervention is appropriately defined; (2) 

whether the groundwork has been laid (that 

is, stakeholder agreement has been obtained) 

for an evaluation; (3) whether adequate 

resources are in place for the evaluation; and 

(4) whether the evaluation is conducted in a 

rigorous manner and produces positive 

effects in a relevant context. Should the 

intervention be shown to be effective, the 

organization would get ready to scale (see 

Chapter III). Should the evidence not indicate 

effectiveness, the organization should 

continue to build evidence (no. 5 in the 

checklist). 

Checklist 2. Building evidence of effectiveness

Check one

Components Yes No 

1. Define intervention 

Defined theory of change (TOC) for the 

intervention 

○ ○

Intervention mature and stable ○ ○

Intervention implemented as described in TOC 

(with fidelity) 

○ ○

Intervention, including any modifications, informed 

by research 

○ ○

If “yes” in all categories, lay groundwork . .

2. Lay evaluation groundwork

Agreement among organization, partners, and 

funders on need for evidence 

○ ○

Agreement among organization, partners, and 

funders on outcomes and measures 

○ ○

Agreement among organization, partners, and 

funders on evaluation goals 

○ ○

Organization partners, and funders support 

evaluation 

○ ○

If “yes” in all categories, prepare for evaluation . .

3. Prepare for evaluation

Financial resources available ○ ○

Personnel available to work with evaluators ○ ○

If “yes” in all categories, conduct evaluation . .

4. Conduct evaluation and assess effectiveness

Research conducted in a manner that meets 

standards for rigor (see Checklist 1 for an 

effective intervention and standards to identify 

effective interventions) 

○ ○

The research showed beneficial results for 

participants 

○ ○

The context or environment in which the research 

was conducted is generalizable to the 

organization’s plans for scaling its intervention 

○ ○

If “yes” in all categories, get ready to scale . .

5. Continued need for evidence

If “no” in one or more categories, make 

corrections and repeat process, as needed 

. .
Appendix: The SCALER Assessment Tool

Mathematica 38

Checklist 3. Specifying the intervention

Organizations can use the checklist below to assess whether they have a well-specified intervention. The 

checklist assesses items within four components that are needed to facilitate successful scaling: (1) clearly 

identifies core components; (2) clearly defines personnel roles; (3) clearly defines the setting; and (4) 

clearly defines completion.

Checklist 3. Specifying the intervention

Circle one 

Components

Not yet 

defined 

No, but 

making 

plans to 

develop a 

definition 

No, but 

actively 

developing 

a definition

Yes, this Is 

defined 

Summary 

score 

Identifying core activities 

Core activities (services) of the intervention 0 1 2 3 ___/9 

The mode of delivery (for example, in person or online, 

one on one or small group) for each core activity 

0 1 2 3 ___/9 

The intensity (that is, amount, frequency, and duration) of 

each core activity 

0 1 2 3 ___/9 

Clearly defining personnel roles 

The number of personnel needed to offer core 

components of the intervention as they are intended to be 

offered 

0 1 2 3 ___/9 

Roles for personnel (at the lead organization and its 

partners) implementing the intervention 

0 1 2 3 ___/9 

Qualifications for personnel (at the lead organization and 

its partners) implementing the intervention 

0 1 2 3 ___/9 

Clearly defining the setting

The geographic setting (for example, rural/urban, 

geographic region) where the intervention will be 

implemented during scaling 

0 1 2 3 ___/6 

The venue for service delivery (such as a community 

center, participants’ homes, nonprofit organization, or 

school) where the intervention will be implemented during 

scaling 

0 1 2 3 ___/6 

Clearly defining participation and completion

Who is considered a program participant (for example, is 

everyone who enrolls considered a participant, or is there 

a minimum length of time that a person must be in the 

program to be considered a participant?) 

0 1 2 3 ___/6 

Participant completion of the intervention (for example, 

minimum services completed, certification acquired, 

particular score achieved on an assessment) 

0 1 2 3 ___/6 
Appendix: The SCALER Assessment Tool

Mathematica 39

Checklist 4. Defining the target population

Organizations can use the checklist below to assess whether they have a well-defined target population. 

This checklist assesses whether the definition of the target population includes the characteristics that 

people must have to participate in the intervention, including any inclusion or exclusion criteria, and how 

well the population aligns to the one for whom the intervention was shown to be effective. 

Checklist 4. Defining the target population

Circle one 

Components

Not yet 

defined 

No, but 

making 

plans to 

develop a 

definition 

No, but 

actively 

developing 

a definition

Yes, this Is 

defined 

Summary 

score 

The characteristics of the population to be served by the 

intervention (target population) 

0 1 2 3 ___/9 

The criteria for determining who is eligible to participate 

in the intervention (inclusion criteria) 

0 1 2 3 ___/9 

The criteria for determining who is not eligible to 

participate in the intervention (exclusion criteria) 

0 1 2 3 ___/9 
Appendix: The SCALER Assessment Tool

Mathematica 40

Checklist 5. Establishing implementation supports

Organizations can use the checklist to assess the status of its implementation supports for scaling. The 

checklist assesses six supports: (1) a monitoring system, (2) performance goals, (3) continuous quality 

improvement processes, (4) preservice and in-service training, (5) communications systems, and (6) data 

systems.

Checklist 5. Establishing implementation supports 

Circle one 

Components

No plans 

in place 

No, but 

making 

plans to 

set up 

No, but 

actively 

setting it 

up

Yes, this 

is in place 

Summary 

score 

Monitoring 

Personnel who will monitor the intervention (the monitoring team) 

are identified 

0 1 2 3 ___/9 

Monitoring team will assess whether core intervention 

components are implemented as specified 

0 1 2 3 ___/9 

Monitoring team will report on implementation issues or 

challenges to leadership and personnel delivering the intervention 

0 1 2 3 ___/9 

Establishing performance goals

Benchmarks for service delivery are clearly established 0 1 2 3 ___/6 

Benchmarks are monitored for achievement 0 1 2 3 ___/6 

Engaging in continuous quality improvement (CQI)

Processes for engaging in CQI are clearly articulated 0 1 2 3 ___/9 

The intervention will be part of conducted CQI efforts 0 1 2 3 ___/9 

The intervention procedures will be updated based on results of 

the CQI process 

0 1 2 3 ___/9 

Conducting preservice and in-service training 

Training to be taken before scaling the intervention begins is 

clearly articulated (preservice training) 

0 1 2 3 ___/12 

All personnel needing preservice training will receive it in a timely 

manner 

0 1 2 3 ___/12 

Training needed by individuals working on the intervention is 

clearly articulated (in-service training) 

0 1 2 3 ___/12 

All personnel needing in-service training will receive it in a timely 

manner 

0 1 2 3 ___/12 

Establishing communication systems 

A communication system exists that supports coordination among 

personnel working on the intervention 

0 1 2 3 ___/9 

A process exists to ensure that communications will function 

smoothly and as intended 

0 1 2 3 ___/9 

A process exists to resolve communication problems shortly after 

they occur 

0 1 2 3 ___/9 

Establishing data systems 

A data system supports collection, analysis, and decision making 

about the intervention 

0 1 2 3 ___/12 

A process exists to ensure that data are entered into the system 

in a timely manner and are of high quality 

0 1 2 3 ___/12 

A process exists to ensure that data are analyzed appropriately 0 1 2 3 ___/12 

A process exists to ensure that data are used in decision making 0 1 2 3 ___/12 
Appendix: The SCALER Assessment Tool

Mathematica 41

Checklist 6. Having an enabling context 

Organizations can use the checklist below to assess whether their context will likely enable successful 

scaling. The checklist assesses two components: (1) support by organization leaders and key stakeholders 

and (2) a culture of innovation and learning.

Checklist 6. Having an enabling context

Circle one 

Components

Not at all 

descriptive 

of the 

organization

Somewhat 

descriptive 

of the 

organization

Mostly 

descriptive 

of the 

organization

Fully 

descriptive 

of the 

organization

Summary 

score 

Supporting the intervention 

Organizational leaders demonstrate support for the 

intervention 

0 1 2 3 ___/6 

Key stakeholders demonstrate support for the 

intervention 

0 1 2 3 ___/6 

Engaging in innovation and learning

The organization has an established history of innovation 0 1 2 3 ___/12 

The organization is innovative in finding ways to improve 

the intervention model 

0 1 2 3 ___/12 

The intervention's successes are discussed among 

personnel involved with the intervention 

0 1 2 3 ___/12 

The intervention's challenges are discussed among 

personnel who can help to overcome them 

0 1 2 3 ___/12 
Appendix: The SCALER Assessment Tool

Mathematica 42

Checklist 7. Establishing an implementation infrastructure

Organizations can use Checklist 7 to assess whether they have the infrastructure in place to support 

scaling. The checklist assesses two components: (1) financial and human resources and (2) materials and 

physical space. 

Checklist 7. Establishing an implementation infrastructure

Circle one 

Components

No plans 

in place 

No, but 

making 

plans to 

set up 

No, but 

actively 

setting it 

up

Yes, this 

is in place 

Summary 

score 

Procuring financial and human resources 

The organization has adequate funding for the intervention 0 1 2 3 ___/9 

The organization provides enough dedicated personnel for the 

intervention 

0 1 2 3 ___/9 

A human resource system is in place to hire, supervise, and 

develop personnel implementing the intervention 

0 1 2 3 ___/9 

Providing materials and physical space 

A thorough description of the materials and physical space 

needed to implement the intervention exists 

0 1 2 3 ___/9 

The materials needed to implement the intervention are available 0 1 2 3 ___/9 

Enough physical space exists for implementing the intervention 0 1 2 3 ___/9 
Appendix: The SCALER Assessment Tool

Mathematica 43

Checklist 8. Summarizing across checklists

Organizations can use the summary checklist for scaling readiness to quickly identify areas in which such 

assistance might be needed. The summary checklist compiles information from each scaling readiness 

condition checklist (listed in bold) and the various components within each condition. To complete the 

summary checklist, organizations should (1) enter the summary score they received for each of the 

various scaling readiness components, then (2) tally these scores to create a total summary score (which 

can be entered into the box highlighted with a border) for each condition. These scores can help 

organizations quickly identify the conditions in which an intervention or the organization itself might not 

yet be ready to scale and might seek to build their capacity before scaling. 

Checklist 8. Summary checklist

Evidence of effectiveness Guidance

For identifying an effective intervention, are all items under research rigor, 

intervention effectiveness, and intervention relevance marked yes? 

Yes: Proceed to scaling readiness conditions 

No: Proceed to building evidence 

For building evidence of effectiveness for your intervention, are all items 

under define intervention, lay evaluation groundwork, prepare for evaluation, 

and conduct evaluation and assess effectiveness marked yes? 

Yes: Proceed to scaling readiness conditions 

No: Continue building evidence for your 

intervention 

Scaling readiness conditions Summary scores 

Intervention

Well-specified intervention

Identifying core components ___/9 

Clearly defining personnel roles ___/9 

Clearly defining the setting ___/6 

Clearly defining completion ___/6 

TOTAL SUMMARY SCORE ___/30 

Well-defined target population

TOTAL SUMMARY SCORE ___/9 

Implementation supports

Monitoring implementation ___/9

Establishing performance goals ___/6

Engaging in continuous quality improvement ___/9

Conducting preservice and in-service training ___/12

Establishing communication systems ___/9

Establishing data systems ___/12

TOTAL SUMMARY SCORE ___/57

Organization

Enabling context

Supporting the intervention ___/6

Engaging in innovation and learning ___/12

TOTAL SUMMARY SCORE ___/18

Implementation infrastructure

Procuring financial and human resources ___/9

Providing materials and physical space ___/9 

TOTAL SUMMARY SCORE ___/18 
Appendix: The SCALER Assessment Tool

Mathematica 44

Process for completing the SCALER

The personnel member who has ownership over the SCALER (“the SCALER owner”) should track 

whether and when the SCALER needs to be completed, identify and bring together the appropriate 

personnel to complete it, have the personnel complete it, synthesize the scores and produce output to 

share with personnel on action steps to address the SCALER results, and then determine when the tool 

should be filled out again. See below for a suggested process that organizations can take to complete the 

SCALER on an ongoing basis or as needed.

Exhibit A.1. Process for completing the SCALER

The personnel member who has ownership over the SCALER (“the SCALER owner”) should take 

these steps to complete it as needed:

Who at the organization should complete the SCALER?

• The SCALER owner should consult with appropriate organizational personnel—such as leaders and the 

intervention program manager—to decide who should help to fill out the SCALER.

• Once these personnel are identified, the SCALER owner will let them know they are responsible for helping to fill 

out the tool and the deadline for doing so.

• The SCALER owner should set up a timeline for completing the tool.

How should organizational personnel complete the SCALER?

• We recommend that the SCALER owner have each personnel member fill out his or her own version of the tool 

and then set up a meeting for everyone to discuss and come to agreement on scores.

• Organizations might prefer to take different approaches, such as having several personnel fill it out together while 

discussing it during a meeting.

What happens after the SCALER is completed?

• After scoring is completed, the SCALER owner documents the scores and shares them with key stakeholders 

and organizational leaders involved with preparing to scale. 

• Working with key stakeholders and organizational leaders, the SCALER owner can also suggest next steps—for 

example, defining the target population if it is found to lack a definition.

When should the SCALER be filled out again?

• Depending on the priorities of the organization, its interest in or plans for scaling, and any upcoming changes to 

the organization or its external context, the SCALER owner should identify an appropriate time to complete the 

SCALER again.

• The SCALER is intended to be a living document, and reflective of one moment in time. As changes or progress 

is being made to scaling readiness conditions, the document should be updated accordingly.
Appendix: The SCALER Assessment Tool

Mathematica 45

Discussion guide for the SCALER

After scoring their scaling readiness using the SCALER, organization personnel will likely want to 

identify next steps to strengthen scaling readiness as a result of the scores. See below for suggested 

discussion questions to help organizations assess their scores and identify actions to take after completing 

the SCALER.

• In what areas are we the strongest? In which areas do we appear to need work? Are there specific 

items (rows) on which we should concentrate our efforts for improvement? (This may include 

identifying evidence of effectiveness, building evidence of effectiveness, or any of the scaling 

readiness conditions.)

• Do we need any additional information to complete any parts of the SCALER? Are there any items 

that we don’t know the answers to? If so, how can we go about answering those items?

• In what conditions or areas, if any, do we have differences of opinion? What information can we draw 

upon to resolve those differences?

• Who are the stakeholders (organization personnel or external stakeholders) with whom we should 

discuss these results (that is, people who were not involved in completing the SCALER)? 

• What next steps should we take to identify or build evidence of intervention effectiveness, or 

strengthen our scaling readiness (as appropriate)? Which steps are highest and lowest priority? What 

is our timeline for taking action? 

• What partners could we bring in to build capacity in certain components? Should we seek to identify 

new partners to help gain this capacity? 
This page has been left blank for double-sided copying. 
This page has been left blank for double-sided copying. 
Mathematica

Princeton, NJ • Ann Arbor, MI • Cambridge, MA 

Chicago, IL • Oakland, CA • Seattle, WA

Tucson, AZ • Woodlawn, MD • Washington, DC 

EDI Global, a Mathematica Company

Bukoba, Tanzania • High Wycombe, United Kingdom

mathematica.org