November 3, 2021

Deepening Our Understanding of What it 

Takes to Scale an Effective Program

1

Scaling Evidence-Based Models Initiative

Office of Research and Evaluation (Evaluation@cns.gov)
Please be advised that all lines 

have been muted to avoid 

background noises and to allow for 

greater engagement.

Please use chat feature to submit 

any questions throughout the 

presentations. Select Everyone

when submitting your questions. 

We will read your questions for you.

Spanish captions are available. 

To activate closed caption please 

select click on captions and select 

show subtitles.

Welcome!

2
Mary Hyde, PhD

Director, Office of Research and Evaluation

AmeriCorps

Scaling Evidence-Based Models Initiative

3

Welcome Remarks
Scaling Evidence-Based Models Initiative

4

Introductory Remarks

Diana Epstein, PhD

Evidence Team Lead

Office of Management and Budget
Scaling Evidence-Based Models Initiative

5

Scaling Evidence-Based Models 

(SEBM) Project and SCALER Tool: 

An Introduction

Lily Zandniapour, PhD

Research and Evaluation Manager

AmeriCorps
• The Edward M. Kennedy Serve America Act 

(Public Law 111-13, enacted April 21, 2009)

 Social Innovation Fund 2010-2016 (Tiered-Evidence-Based 

Grantmaking): 100% of grant funds supported evidencebased models and rigorous evaluation requirements were 

implemented

 AmeriCorps State & National implemented evaluations 

requirements based on funding levels in the grant 

application process and rolled out Evidence-Based Planning 

Grants

 AmeriCorps Seniors offered Augmentation Grants to 

grantees that selected evidence-based models for 

implementation

 AmeriCorps provided evaluation training and technical 

assistance to support evidence building

AmeriCorps Evidence Journey

6

Scaling Evidence-Based Models Initiative 
• Scaling Evidence-Based Models Project (SEBM) was implemented in partnership with Mathematica (2016 through 2021) 

• Purpose: To deepen the agency’s understanding of the most 

effective program innovations and the knowledge base on 

scaling them

• Opportunity: Be more intentional/strategic about use of 

evidence to inform our work and how best to leverage national 

service in solving pressing social issues

• Vision/Goal: Scaling of AmeriCorps’ most effective supported 

solutions nationwide & use of national service to facilitate that 

growth & impact

Scaling Evidence-Based Models Initiative 

Project Background
• Which AmeriCorps-supported interventions demonstrate strongest 

evidence of effectiveness? 

• How do AmeriCorps-funded organizations define scaling? 

• How do organizations scale effective interventions?

• How can scaling of effective interventions be best supported?

Scaling Evidence-Based Models Initiative 

Key Questions of Interest
• Document/database review including a study of 

previous systematic reviews of evaluation reports

• Yearly meta-synthesis and meta-analysis of existing and 

emerging evidence-based models

• Yearly review and analysis of scaling plans and grant 

applications

• Three in-depth process studies of selected grantees 

implementing and scaling an Evidence-Based Program 

and a cross site analysis

• Development and engagement of a Field Workgroup 

• Development of policy & practice guides/technical 

assistance materials on scaling

Scaling Evidence-Based Models Initiative 

Components
Scaling Evidence-Based Models Initiative

10

Mathematica Research Team

Scott Richman, PhD

Senior Researcher

Mathematica

Mary Anne Anderson, MPP

Researcher (Qualitative Specialist)

Mathematica
• Overview of the Scaling Checklists: Assessing your Level of Evidence 

and Readiness (SCALER) tool

• Discuss the SCALER framework and components

• Demonstration of the SCALER

Scaling Evidence-Based Models Initiative

Agenda

11
Overview of the SCALER

Scaling Evidence-Based Models Initiative

12
• Successfully scaling effective interventions means:

• Maintaining or surpassing the beneficial impacts of an intervention

• Helping to improve lives for larger numbers of individuals and 

communities

• Organizations can maximize this potential by: 

• Ensuring the interventions have evidence of improving participant 

outcomes 

• Having intervention-level and organization-level procedures and 

processes to successfully scale

Scaling Evidence-Based Models Initiative

What is successful scaling?

13
• The Scaling Checklists: Assessing your Level of Evidence and Readiness 

(SCALER) tool is a framework and set of checklists organizations can use to 

increase their likelihood of successfully scaling an intervention

• The primary goal is to support organizations in getting ready to scale their 

interventions by helping them to: 

• Identify or build the evidence needed for scaling an effective intervention

• Identify the intervention-level and organization-level procedures and processes that will 

enable successful scaling

• Funders can also use the SCALER to communicate expectations to 

grantees about scaling readiness and inform technical assistance or 

training

Scaling Evidence-Based Models Initiative

SCALER overview

14
• The SCALER is a tool that can be completed online: 

https://americorps.gov/grantees-sponsors/evaluationresources/scaler 

• To complete the SCALER, organizations need: 

• Familiarity with any research studies on the intervention

• Knowledge of the organization’s support for building evidence

• Understanding of the intervention’s activities, target population, and 

implementation supports, and of the organization’s culture and infrastructure

• Recommend designating one person or role to be the SCALER 

owner who completes the tool and seeks input from other 

organization personnel 

Scaling Evidence-Based Models Initiative

Who completes the SCALER

15
• The online tool will help identify areas in which organizations 

are more and less ready to scale

• Organizations can take steps to build capacity in areas of 

growth 

• Organizations can identify realistic and short-term goals to 

gauge progress

• Organizations can revisit the SCALER regularly to assess 

scaling readiness

Scaling Evidence-Based Models Initiative

Action planning with the SCALER

16
The SCALER framework and components

Scaling Evidence-Based Models Initiative

17
Scaling Evidence-Based Models Initiative

The SCALER Framework

18
• Understand what rigorous research is

• Determine whether an intervention has 

research showing that it is effective

• Learn about the context or environment 

in which the intervention was evaluated

Scaling Evidence-Based Models Initiative

Step 1: Identifying evidence of effectiveness

19
• A well-articulated theory of change is needed for a well-defined intervention

• Stakeholder consensus should have a 

common understanding and s about 

research and evaluation

• Financial resources and personnel should 

be available

• Organizations conduct rigorous evaluation 

and determine if intervention improved 

participant outcomes 

• Continue process if further research is 

needed

Scaling Evidence-Based Models Initiative

Step 2: Building evidence of effectiveness

20
• Five conditions can facilitate scaling 

readiness

• Maintaining fidelity after scaling helps an 

intervention generate the same 

outcomes that occurred before scaling

Scaling Evidence-Based Models Initiative

Step 3: Getting ready to scale

21
• A well-specified intervention clearly identifies the 

core components critical for achieving beneficial 

outcomes after scaling

• Clearly identified core activities

• Clearly defined personnel roles

• Clearly defined setting

• Clearly defined completion 

Scaling Evidence-Based Models Initiative

Step 3: Having a well-specified intervention

22
• When scaling, offering an intervention to the 

same population for whom the intervention 

was designed and shown to be effective 

leads to the highest likelihood of success 

• The population must be well-defined in terms 

of the characteristics people must have to 

participate in the intervention

Scaling Evidence-Based Models Initiative

Step 3: Having a well-defined target population

23
• Six supports can help an organization implement 

with fidelity: 

• A monitoring system

• Performance goals

• Continuous quality improvement processes 

• Preservice and in-service training

• Communication systems

• Data systems

Scaling Evidence-Based Models Initiative

Step 3: Establishing implementation supports

24
• An organization’s culture should support 

innovation, learning, and improvement during 

scaling

• Two components will enable scaling: 

• Support by organization leaders and key stakeholders

• A culture of innovation and learning

Scaling Evidence-Based Models Initiative

Step 3: Having an enabling context

25
• An organization’s infrastructure should ensure 

sufficient resources to support fidelity during 

scaling 

• Infrastructure should include sufficient: 

• Financial and human resources 

• Materials and physical space

Scaling Evidence-Based Models Initiative

Step 3: Establishing an implementation infrastructure

26
Demonstration of the SCALER

Scaling Evidence-Based Models Initiative

27
• Flag Football for Fitness (F4) seeks to combat childhood obesity, promote healthy 

eating and exercise habits, and foster positive youth development 

• Students improve football skills while learning about healthy dietary and exercise 

habits

• F4 works with students in grades K–8 at schools receiving Title I funds in urban districts 

throughout the country

• Sessions are 90 minutes per day, 3 days a week, for 12 weeks during the school year 

and are conducted in a face-to-face in a group format 

• Site coordinators, who have at least 5 years of experience and at least a bachelor’s 

degree, oversee administrative and programmatic functions at each school 

• A coach, with demonstrated experience working with youth, runs the sessions at 

each school

Scaling Evidence-Based Models Initiative

Hypothetical program for demonstration

28
Scaling Evidence-Based Models Initiative

Questions and Discussion 

Mathematica Research Team and 

AmeriCorps’ Office of Research and Evaluation

29
Lily Zandniapour

lzandniapour@cns.gov

Scott Richman

SRichman@mathematica-mpr.com

Mary Anne Anderson

MAAnderson@mathematica-mpr.com

Scaling Evidence-Based Models Initiative

For more information

30

Thank You!

• Recording will be available at 

AmeriCorps Impact Webinars

• Visit the AmeriCorps Scaler Tool 

on the AmeriCorps Evaluation 

Resources Page

For more information on AmeriCorps Research and Evaluation: evaluation@cns.gov