ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

1 

[Music] 

JENELLE AZORE: Hello, everyone. I am Jenelle Azore and 

the webinar will start soon. Before we begin, we 

would like to cover a few housekeeping items. This 

webinar will be recorded and posted online on 

AmeriCorps Impact webinars page. Please be advised 

that all lines have been muted to avoid background 

noises and to allow for greater engagement. You can 

submit questions at any time during the presentation 

by using the chat feature below. Select everyone when 

submitting your questions. We will read your 

questions for you during the Q and A session 

following the presentation. Spanish captions are 

available. To activate, please select the live 

transcript icon on the Zoom menu and select show 

subtitles. As mentioned earlier, this webinar is 

being recorded and if you have any questions or 

experience technical difficulties, please let us know 

using the chat box feature. That takes care of all 

our housekeeping items. The webinar will start now. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

2 

And here to welcome us is Mary Hyde, the Director of 

AmeriCorps’ Office of Research and Evaluation. 

MARY HYDE: Good afternoon. We are thrilled to be hosting 

this webinar and delighted to welcome Dr. Diana 

Epstein with the Office of Management and Budget, who 

will offer introductory remarks today. Today we are 

featuring the SCALER tool as a resource that will 

have utility both for practioners and funders and 

grant makers. The tool is meant to gauge readiness to 

scale evidence based models and provide guidance on 

ways readiness to replicate can be improved. We 

anticipate that this resource will be beneficial to 

AmeriCorps grantees and sponsors, but we also see a 

much wider application beyond our agency. We are 

hopeful that this tool will prove to be valuable to 

our federal partners, as well as community-based 

organizations, non-profits, and the philanthropic 

community. The tool is grounded in applied research, 

driven by state of the art implementation science and 

was informed by national service grantees who 

graciously agreed to provider user testing. This 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

3 

allowed us to bridge fieldwork and scholarship. This 

tool is part of a larger suite of products developed 

through an investment in research by AmeriCorps. This 

work has been a long time coming, and we are so glad 

to be hosting this webinar today. I would like to 

introduce Diana Epstein, Evidence Team Lead with the 

Office of Management and Budget for some introductory 

remarks. Diana? 

DIANA EPSTEIN: The Biden-Harris administration has made 

evidence-based policymaking a priority, starting with 

the Presidential Memorandum issued in January on 

scientific integrity and evidence-based policymaking. 

This strong support for evidence-based policymaking 

further amplifies the work that agencies across the 

federal government, including AmeriCorps, have been 

engaged in over the past few years, including 

implementation of the Foundations for Evidence-based 

Policymaking Act. We know we have a lot of work left 

to do, but we’re really encouraged by the tremendous 

progress that we’ve seen agencies make, and their 

commitment to taking on what is often really 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

4 

difficult culture change. You’re going to learn all 

about the SCALER tool in today’s session, so I won’t 

go into those details. But I did want to point out a 

couple of connections to the broader focus on 

evidence-based policymaking that I think are 

particularly important. So first, we need to 

understand if the programs we’re running, if the 

interventions we’re implementing are effective. Are 

they producing the intended results for the 

populations and the communities that they’re designed 

to serve? Are they working well in some places, but 

not others and do we know why? We need to build 

evidence, often through rigorous evaluation and other 

methods, to make these determinations. This focus on 

high quality evidence building is something we’ve 

been emphasizing where I work at OMB. If the study 

finds conclusive evidence of effectiveness, that’s 

great. You know, you can use that information along 

with other pieces of information to figure out how 

you might want to scale to other populations. As you 

all know, communities are facing a lot of challenges, 

now more so than ever. We want to be able to share 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

5 

and spread effective practices when we can and this 

is what SCALER is all about. On the other hand, maybe 

you don't currently have evidence that your program 

or your intervention is effective or maybe an 

evaluation led to surprising or unexpected results. 

That’s not necessarily a bad thing. It just means 

it’s time to build evidence. We have to understand 

what our programs are doing, if they’re being 

implemented as designed, and if they’re having the 

intended results. This is the only way we’ll know if 

we’re serving communities well. We need to build 

evidence where it doesn’t exist and there are still 

many programs and many policy areas where we lack the 

evidence we need. So we want you to embrace evidence 

building in thoughtful and rigorous ways. It’s not 

about showing that something doesn’t work; rather 

it’s about understanding what’s happening, learning 

and improving where we need to. So to close, I’m 

really pleased that AmeriCorps invested the time and 

energy in building the SCALER tool. It’s a great 

model of how a focus on evidence can be infused 

throughout the work of a federal agency and then 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

6 

shared with grantees and community partners. With 

that, I will pass it off to Lily Zandniapour to tell 

us more about SCALER and how AmeriCorps got here. 

LILY ZANDNIAPOUR: Thank you so much, Diana. Good 

afternoon, everyone. It is a pleasure to be with you. 

I am Lily Zandniapour, and I oversee the scaling 

initiative at AmeriCorps. And before our main 

speakers begin their presentation and introduce you 

to this tool, which we are really, really excited 

about, I just wanted to give you a little bit of 

context and background about the scaling initiative 

and sort of give you a sense of the journey that 

really got us to this point. I think it would be sort 

of arguably the passage of the Serve America Act in 

2009 was a pivotal moment for AmeriCorps. Following 

the passage of Serve America Act, you know, there was 

a renewed sort of emphasis on evidence, evidence 

building, and use of evidence in our work. And since 

then, there’s been incredible amount of investments 

made, both by the agency and by our grantees, in 

program evaluation and evidence building. I think one 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

7 

of the catalysts for this change and shift in focus 

was that the Serve America Act really called for the 

implementation of the Social Innovation Fund. This 

was a tiered evidence program where 100 percent of 

the grant funds went to programs that had some 

evidence that they worked and there was a very robust 

evaluation program attached to this initiative and a 

high rigorous evaluation bar was placed on it. And 

that was very - it was a catalyst for a lot of change 

in our agency. AmeriCorps State and National was one 

of the first programs and the largest program in the 

agency to really adopt the evidence framework and 

over successive years really started to incorporate 

the role of evidence in grant-making at AmeriCorps. I 

just want to draw your attention to this infographic. 

And if someone can put the - thank you - the link to 

this infographic in the chat box, you’ll see that 

over time the share of investments that - the 

competitive grants have made with AmeriCorps State 

and National, have gone to evidence-based programs. 

And that change is really noteworthy, it’s 

interesting. So I suggest you take a look at this 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

8 

infographic, because that gives you a sense. Other 

programs in the agency too really started after the 

passage of Serve America Act, become more evidencefocused. For example, AmeriCorps Seniors started a 

very large research project to build its evidence 

base, and over time implemented a number of things. 

And I’m going to just stop there, but there’s more 

that I can say about the National and the Community 

Corps program and also Vista and their efforts. But 

this is just to give you a flavor of this pivotal 

moment and this shift in focus in evaluation. Another 

thing that the agency did which I think was very 

influential and sort of a game-changer was the fact 

that the agency focused on evaluation, training, and 

technical assistance during this time, to support 

evidence building. So we not just asked for - put 

that in a requirement but really tried to support it. 

If you could go to the next slide, please. A few 

years after that passage of Serve America Act we 

really came to a point where there was a lot of 

evidence built and so there was an emerging evidence 

around the programs that the agency supported. And 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

9 

what we did then was we started to think about how we 

can be more intentional and leverage all of these 

investments. And think about how with sort of all 

this evidence that had emerged how we could take 

solutions, things that work, interventions that 

really affect lives, and try to bring that to other 

communities, communities that were underserved, that 

didn’t have these effective program models to help 

communities and different groups across the country. 

So, our vision was really to bring that, bring 

solutions to all of America and really use national 

service as a way to grow that impact and facilitate 

that. So we started this scaling evidence-based 

models project in 2016. And we’re sort of about to 

wrap it up in this format, although the journey is 

going to continue. But this project, a lot of people 

really contributed to that. I was just thinking of 

Diana who was with our agency at the time and was 

critical in sort of the conceptual piece of this 

work. So we’re grateful for that. I wanted to give or 

acknowledge three other people that really are team 

members past and present that worked on this 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

10

initiative on our end. Anthony Nurino managed this 

contract and this initiative on our end for several 

years. Roshni Mennan who’s now since has moved to HHS 

also was very supportive of this work. And one of our 

current team members, Katie Hussy-Slonecker, has been 

just instrumental in getting a SCALER tool out and 

really supporting the initiative in more recent 

years. So I really want to thank them for their 

contribution. So, with that, what we’ve ended up 

doing was partnering with a great firm, Mathematica, 

that had deep expertise in this space and we started 

to sort of engage on answering some of the key 

questions that we had at that point. So if you could 

go to the next slide please. What you’ll see here is 

some of the key questions that drove this project. We 

wanted to know where we have the strongest evidence 

of effectiveness, what are the programs that the 

agency had supported over time that had the strongest 

evidence of effectiveness. We also wanted to learn 

how organizations, national service organizations and 

the organizations that the agency had funded over 

time, defined scaling and how they operationalized 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

11

it. And we also definitely wanted to know how this 

work can be supported, how we can support scaling 

effectively so they are successful. In relation to 

these questions, these were the main driving 

questions, but alongside of this, many other things 

came to light, like our interest to know the role, 

for example, of fidelity and adaptation as you 

increase - as you try to take a solution to a new 

community. About factors that facilitate or hinder 

scaling. The use of evaluation to inform the scaling 

process - that was a big thing. And as I said, 

readiness was a big factor there. If you could go to 

the next slide, please. The evidence snapshot you 

see, this - and I hope that there is a link to that 

in the chat box, this document provides you with an 

overview of this project, more detailed, because 

there’s no time to really go into all of it. But this 

effort has been a multi-year, multi-component 

project. It had different phases. There was a very 

targeted attempt, as Mary mentioned in the beginning, 

to make sure that the experiences of the practioners 

and experts on the ground were incorporated in this 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

12

work. As one of the things, for example, that we did 

and you see some of the components listed on this 

slide, is to conduct deep analysis and case studies 

on examples where our grantees had attempted scaling, 

to learn from that. But we also drew insights from a 

broader group of practioners through a field work 

group to inform this work. And again, as Mary 

mentioned, a lot of our grantees helped to assist us 

in the tweaking and finessing of the SCALER tool, 

which we’ll be talking about shortly. So many 

resources, such a generative process and created so 

many tools and resources. The evidence snapshot that 

you see there and you have the link too really lists 

out all of those resources and links to them so you 

will have them and you can sort of map it back to the 

phases of this project. And with that, I am delighted 

to turn it over to our partners, Dr. Scott Richman 

from Mathematica has been managing this project and 

has been a great partner with us. And Mary Anne 

Anderson has been the lead researcher on 

Mathematica’s side who has also worked on the SCALER 

tool, and everything else that you see in this suite 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

13

of products that are offered by this. So with that, 

I’m going to turn it over to Scott and Mary Anne. 

Thank you. 

SCOTT RICHMAN: Thank you so much, Lily - we really 

appreciate it. And good afternoon, everyone, and 

thank you for joining us today for the webinar. Mary 

Anne and I are excited to discuss the new SCALER tool 

that Mathematica has developed in collaboration with 

AmeriCorps’ Office of Research and Evaluation. The 

purpose of this part of the webinar is to provide an 

overview of the SCALER where we’ll discuss the 

purpose of the tool, along with some of the practical 

aspects of how the tool can be completed and used. 

We’ll also discuss the SCALER’s framework and 

underlying components that are the basis of the selfassessment tool. And finally, we’ll conduct a brief 

demonstration of the SCALER tool itself as it’s now 

live on the AmeriCorps website and we’ll highlight 

the tool’s features and run through an example of how 

users can complete the tool. So let’s get started 

with a general overview of the SCALER, which was 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

14

designed to support the successful scaling of an 

intervention. So when we say successfully scaling an 

intervention, that means that you’re taking an 

intervention with documented evidence of 

effectiveness, and then maintaining or surpassing its 

beneficial impacts. And you’re doing so, for example, 

when replicating the intervention in a new location, 

expanding the number of existing individuals being 

served, or adapting the intervention to serve a new 

target population. Broadly speaking, when an 

organization successfully scales an intervention, 

it’s helping to improve the lives for a larger number 

of individuals and communities than organizations 

could have otherwise done before scaling. So the ways 

in which organizations can maximize their potential 

to successfully scale their interventions is to first 

ensure that the interventions have rigorous evidence. 

Once such evidence is established, the intervention 

and the organization need to have certain 

definitions, procedures and processes in place in 

order for that organization to take that intervention 

and scale it successfully. However, sometimes an 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

15

organization may try to scale an intervention that 

doesn’t yet have evidence of beneficial impacts, or 

even if such evidence is in place, the organization 

doesn’t have everything that it needs and that the 

intervention needs to support successful scaling. In 

these circumstances, organizations may find 

difficulties in achieving their goal of successful 

scaling. So with this in mind, Mathematica’s scaling 

evidence based models developed a tool called the 

SCALER which stands Scaling Checklists: Assessing 

your Level of Evidence and Readiness. The tool can 

help organizations identify whether they have the 

evidence and conditions necessary for scaling. It 

also helps identify areas that organizations can 

focus on to better prepare for scaling. Funders can 

also make use of this SCALER to communicate 

expectation to grantees about their scaling 

readiness, or to inform the types of training and 

technical assistance they provide to grantees based 

on the areas of need that the SCALER can help 

identify. So the SCALER tool can be completed online 

and can be found on the AmeriCorps website, along 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

16

with the other evaluation resources that the agency 

provides to its grantees. And the primary audience 

for the SCALER are the organizations who are 

interested in scaling their interventions. So to 

complete the SCALER, users should have some 

familiarity with any research studies that have been 

conducted on the intervention. They should have 

knowledge of their organization’s preparedness and 

support for building evidence for the intervention 

if, in fact, evidence is needed. And of course, 

having an understanding of the intervention’s core 

activities, target population and implementation 

supports, along with their organization’s culture and 

infrastructure that can supporting scaling. We would 

recommend designating one person or role at an 

organization to be the SCALER owner who could help 

ensure that the tool is completed. And this 

individual can be the person who’s most knowledgeable 

about the various components that the SCALER covers, 

or it can be a person who can seek out input from 

other team members who have that necessary knowledge 

and expertise. Personnel who might be well positioned 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

17

to lead or help complete the SCALER can be executive 

directors, program managers or project directors, 

frontline supervisors or measurement learning and 

evaluation team members. So after completing this 

self-assessment tool, the SCALER will help 

organizations by summarizing areas of strength, and 

different scaling components in which an intervention 

or organization might not yet be ready to scale. The 

results can also help organizations to develop 

procedures or processes to build capacity in these 

areas, or to help to identify new partners to help 

gain this capacity. And organizations can also 

identify specific next steps to take to further their 

scaling preparations and set short-term goals to 

gauge their progress. For example, reviewing their 

SCALER results can help organizations to identify and 

discuss their areas of strength and biggest needs, 

and how to prioritize their next steps to advancing 

their scaling readiness. And lastly, organizations 

can visit the SCALER regularly based on their own 

priorities or previous use of the tool. For example, 

some may wish to complete the SCALER on a periodic 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

18

basis, such as two or three times a year to make sure 

that certain scaling components remain in place. 

Others may wish to visit the SCALER again soon after 

making changes to address a scaling condition that 

they scored low on when we previously took the 

SCALER. What the SCALER is there for organizations to 

use as often as makes sense to them, and they can 

access their previous results which we’ll discuss 

later on in the demonstration. With that, let’s dig 

into a little bit more about the SCALER tool in terms 

of the larger framework that informs the tool’s 

design, along with the specific components of the 

tool. So here’s the framework that helps organize and 

sequence the different components of the SCALER. An 

organization wanting to scale an intervention should 

first ask the question in the red box, is there 

evidence that this is an effective intervention? The 

answer to this question, which is what step one of 

the SCALER is all about, is critical because it 

determines what needs to be done to prepare an 

organization to scale their intervention. So the 

SCALER framework begins with guidance and a checklist 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

19

to help organizations identify what rigorous evidence 

for their intervention looks like. And then if that 

evidence doesn’t exist, the organization should then 

seek to build that evidence before scaling. So on the 

right hand side of the graphic, the SCALER framework 

includes step two, which is a process to help 

organizations build evidence that the intervention is 

affected. To this end, the SCALER includes guidance 

and a checklist to help organizations move towards 

conducting research that can help establish evidence 

for their intervention. And finally, once the 

evidence has been identified or built, the SCALER 

step three then provides guidance and a series of 

checklists that are designed to help organizations 

get ready to scale. More specifically, the SCALER 

discusses five conditions that should be present for 

an intervention in an organization to have scaling 

readiness. But let’s unpack these different steps a 

bit more, starting with identifying evidence of 

effectiveness. So rigorous research can be difficult 

to conduct because it requires isolating, whether the 

intervention itself contributed to changes in 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

20

participant outcomes. Yet knowing that intervention 

and not some other factors change participant 

outcomes, this provides the confidence that 

intervention is effective and then it warrants 

scaling. So step one of the scale has a checklist of 

items designed to help organizations identify whether 

an intervention has such evidence. So informed by 

federal evidence review clearing houses, the first 

set of items speaks to the rigor of the research 

which are presented on the table on the right. And 

these items include having an evaluator that is 

external to the organization to conduct the 

evaluation. It includes using a study design that has 

a counter-factual, meaning that there’s a comparison 

group included in the study to show what would have 

happened to participants had they not received the 

intervention. And it also involves making sure there 

are no confounds in the study design, which means 

that evaluators are able to attribute any impacts to 

the intervention and not to other factors that could 

account for these effects. And finally, the research 

needs to be carried out in such a way to maintain its 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

21

strong internal validity. In that regard, the 

conducted research, depending on the study design 

used, would need to show low attrition, no 

reassignment, and it should establish baseline 

equivalence between its study groups. And once an 

organization is able to identify existing studies 

that were carried out with a rigorous design, step 

one of the SCALER then asks whether the studied 

intervention actually produced the beneficial 

participant impacts. If the intervention was not 

shown to be effective, then it’s not yet ready for 

scaling. And the last part of the SCALER captures 

whether the context and the environment used as the 

basis for the research evidence, whether that’s 

generalizable or relevant to the scaling plans that 

an organization has for its intervention. And if 

there is a no on any of these items in the step one 

checklist related to research rigor, effectiveness or 

relevancy, the organization still has a need for 

evidence and should think about building that 

evidence themselves. And if there’s a yes on all 

these items, than that organization can proceed to 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

22

determine their scaling readiness at which Mary Anne 

will speak to shortly. First, when rigorous evidence 

does not yet exist, this evidence must be built 

before an organization can look to scale. The SCALER 

framework highlights a process organizations can use 

to openly gain knowledge about an intervention’s 

impact. The ultimate goal of this process is for an 

organization to come away with an evaluation that 

beats those standards, like just discussed on the 

previous slide. So organizations can use SCALER step 

two checklist as they seek to build evidence for 

their interventions. And this process begins with 

clearly defining the intervention. This involves 

having a well-articulated theory of change, which 

will describe the assumptions for how the 

intervention will successfully affect outcomes for 

participants. And the organization then needs to lay 

the groundwork for evaluation by working to build a 

common understanding among organization stakeholders 

and partners about the value of the evaluation and 

what the evaluation is aiming to achieve. And next, 

organizations need to prepare for the evaluation 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

23

which requires sufficient financial resources, and 

personnel to help evaluators carry out the evaluation 

design. And once an organization has prepared for its 

evaluation, they can then work with a third party 

evaluator to conduct an impact study which would 

ideally be rigorous enough to allow the organization 

to know whether its intervention has evidence of 

effectiveness. And upon completing its evaluations, 

organizations may find that the intervention that did 

not, in fact, improve participant outcomes, or it was 

unable to carry the rigorous study design, as they 

hoped. And that’s okay. In those instances the 

organization is faced with a continued need for 

evidence and should start over with this process for 

building such evidence until they have the impact 

study they’re looking for. And once an organization 

is able to complete each of these steps and the 

evaluation shows evidence of effectiveness, then they 

can think towards getting that intervention and the 

organization itself ready to scale, which Mary Anne 

will speak to next. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

24

MARY ANNE ANDERSON: Thanks, Scott. So once an 

intervention’s effectiveness has been established, 

organizations can seek to get ready to scale it. So 

the SCALER framework has five conditions that guide 

both an organization and an intervention’s readiness 

to scale successfully. The first three conditions, a 

well specified intervention, a well-defined target 

population, and implementation supports indicate 

whether the intervention is ready to scale, which 

means that it has the features that could allow it to 

be implemented with fidelity after scaling. This, as 

Scott mentioned, increases the likelihood of 

achieving the same beneficial outcomes that were seen 

prior to scaling. The last two conditions on the 

slide, having and enabling context that’s conducive 

to scaling and is sufficient implementation 

infrastructure in place speak to whether the 

organization is ready to scale. So I’ll touch a bit 

on each of these five conditions. So first is having 

a well-specified intervention. This includes the core 

components that are thought to be critical for 

achieving beneficial outcomes after scaling. So 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

25

specifying these components helps organizations 

implement interventions with fidelity. The four 

components that we include in the SCALER framework 

are first, clearly identified core activities. So, 

for example, for a school-based reading program, core 

activity might be the curriculum that the program 

uses whereas a non-core or supplement activity might 

be suggestions of books that parents could read to 

their children at home. Second are clearly defined 

personnel roles. So this includes the number of 

personnel needed, what they do and their 

qualifications. Third is a clearly defined setting, 

meaning where the intervention takes place. And then 

fourth is a clear definition of participation, 

participant completion. The next condition is having 

a well-defined target population. So when scaling 

offering an intervention to the same population for 

whom the intervention was designed and shown to be 

effective makes it more likely that the intervention 

will be successful when scaled. To be able to do 

this, the population should be well defined and 

include the characteristics that people must have to 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

26

participate in the intervention. So this, so some 

examples of this might be children within a certain 

range of ages or grades, having a certain health 

condition or being at risk of a certain health 

condition or being a recipient of a specific public 

benefits program. Next, the intervention should have 

implementation supports in place to facilitate 

scaling. So these help to ensure that an intervention 

can be implemented with fidelity as well. These 

include a system to monitor implementation of the 

intervention, performance goals for personnel who are 

implementing the intervention, continuous quality 

improvement or CQI process which are a systematic 

process for improving service delivery and outcomes. 

Pre-service and in-service training for personnel, 

meaning training that occurs before implementation of 

the intervention and while it’s being implemented. 

Systems to ensure communications about the 

intervention are smooth and the appropriate personnel 

are communicating about it, and data systems that 

allow organizations to record, track and analyze data 

and use it for intervention monitoring and 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

27

improvement purposes. So now we move on to the two 

conditions that indicate whether an organization is 

ready to scale an intervention. The first is having 

an enabling context. So that means that an 

organization’s culture supports innovation, learning 

and improvement during scaling. And also that the 

organization’s roles, structures and functions 

facilitate service delivery. So there’s two specific 

components that we look at under having an enabling 

context. First is support by organization leader and 

key stakeholders and second is having a culture of 

innovation and learning within the organization. And 

then the final condition for scaling readiness is 

establishing an implementation infrastructure that 

supports fidelity. So the implementation 

infrastructure includes two components, financial and 

human resources and materials and physical space. So 

these components cover hiring the necessary personnel 

for the intervention, supporting their supervision 

and professional development, and providing funding, 

materials and space for implementing the 

intervention. And now we’re going to jump into a 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

28

demonstration of the SCALER tool so that we can 

highlight the tool’s key features and show how 

organizations can use it. To help with our 

demonstration, we’re going to be referring to a 

fictitious program that we made up named Flag 

Football for Fitness or F-4. So F-4 seeks to combat 

childhood obesity, promote healthy eating and 

exercise habits and foster positive youth 

development. So while in the program, students play 

flag football, while also learning about health 

eating and exercise practices. The F-4 program is an 

after school program. It works with students in 

grades K through 8 who attend schools that receive 

Title I funding in urban school districts throughout 

the country. The F-4 program works with groups of 

students in person, face to face. So 90 minutes per 

day, three days a week for twelve weeks during the 

school year and there are spring and fall sessions. 

And at each F-4 school or site there’s a site 

coordinator who has at least five years of experience 

and at least a bachelor’s. And that person oversees 

administrative and programmatic functions at the 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

29

school. And then there’s also a coach that runs the 

program activities. The coach should have 

demonstrated experience working with youth and they 

are most often college students, parents and 

teachers. So we’ll refer to this fictitious program 

throughout our demonstration. And now we’ll switch 

over to the SCALER tool on the AmeriCorps’ website, 

and Scott is going to walk us through the first few 

screens of that. 

SCOTT RICHMAN: Thanks, Mary Anne. So the SCALER tool can 

be found on the AmeriCorps website on the evaluation 

resources page. So after clicking on the link here, 

you are taken to the SCALER main home page where you 

are immediately greeted with the SCALER framework 

diagram that Mary Anne and I just discussed. And on 

the page users will see some general overview 

information that describes what the SCALER framework 

is designed for. The main home page also provides 

some helpful information to help get users oriented 

to the SCALER tool itself such as providing 

information on who should complete the SCALER, what 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

30

information is needed and how the results can be 

used. And users can expand or collapse on these 

different content panels to access the information 

they need based on the questions they have about the 

SCALER. The SCALER also provides the starting points 

to the SCALER’s three steps. As we mentioned before, 

there’s a sequential nature to the SCALER, with users 

starting at Step 1, then going to Step 2 if needed, 

before ultimately going to Step 3. That said, we 

recognize that identifying or building evidence or 

getting ready to scale, are processes that each take 

time for an organization to do. So we anticipate 

organizations needing to kind of come back to the 

SCALER multiple times. So the SCALER is designed so 

that users can start the SCALER at any of these 

steps, if they want to complete them at those 

different time points. But with this demonstration, 

where we’re thinking about the F-4 program Mary Anne 

discussed, and because we haven’t used the SCALER 

before, I’m going to begin with Step 1. So when I 

click on the step 1 button users will immediately see 

a popup that asks whether they want to start a new 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

31

step 1, or if they want to return back to a prior 

step 1 checklist they may have already started or 

completed. And I’ll come back to that point in a 

little bit, but for now let’s start a new step 1. So 

on the step 1 landing page, we have an overview 

information that tells us what the purpose of this 

step is. And because the step is about identifying 

evidence of effectiveness for my intervention, users 

can click on this middle panel here, definitions 

about the three different components that are 

involved with building - with identifying evidence. 

And on the right hand side you can see we have a 

series of expandable FAQs. For example, if I would 

expand on the what information do I need, the SCALER 

tells us that for this step, users will need to know 

about any research studies that have been conducted 

on their intervention. And if they’re not aware of 

any, it points to the SCALER resource page where they 

can find links to various research clearinghouses to 

see if any studies have been conducted on their 

programs. But for this F-4 program example, I know 

that our big research study about our study that our 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

32

organization did on a program, did it a few years ago 

and I’m not aware of any other studies, so I’m going 

to get started on the step 1 with this one study in 

mind. So here is the step 1 checklist. And I’m going 

to select the items that reflect the nature of the 

research that has been conducted on the F-4 program. 

First, we designed the checklist page so that users 

can have helpful information and guidance if they 

need it and to avoid toggling back and forth between 

screens. So for example, we have these floating 

question marks that users can hover over for 

definitions to key terms. And underneath each of the 

items, we also provide some examples that have some 

user friendly content to further explain what the 

item is asking. So for my F-4 program, I’m thinking 

about that study my organization conducted which was 

a randomized control trial where we compared middle 

school students who received the program to a group 

of middle school students who did not. So the study 

was conducted by my organization, so I’m going to 

leave this first item here unchecked, but I’m going 

to check off the next two items because we did a 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

33

randomized control trial. So it had a comparison 

group as a counter-factual and there were no 

confounding factors in the research design. But the 

study did report having some high attrition because 

we were unable to collect survey data from a good 

number of students. And we didn’t do any baseline 

equivalence analyses afterwards. And because of that, 

I’m going to leave this box unchecked. And then just 

for the purpose of this demonstration, I’m going to 

quickly check off the next two boxes, which are to 

say whether the research study showed positive 

impacts for the F-4 program and whether the study 

context was relevant to the plans that my 

organization has for scaling the F-4 program. And 

with that, I’m going to hit the next button and I am 

taken to the next screen. So this is the results page 

for step 1. And I just want to quickly point out that 

at the top you’ll see a green box which says that 

your results are saved, and I’m going to come back to 

that in a little bit. But organization personnel can 

use this page as a summary of the evidence that 

exists for their intervention. And here we have the 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

34

headline that says, ‘Evidence for Your Intervention’s 

Effectiveness That Aligns To Your Plans For Scaling 

Does not Yet Exist.’ Your organization will need to 

build such evidence. And I can see my results for the 

items I did and did not select. And I’m provided with 

some summary information from my choices, and given 

information on why these intervention evidence 

components are important, and also what my 

organization can do next for the ones that I did not 

select. In this case I can quickly see that I 

received my rating, just because I did not select 

that my intervention evaluation was conducted by a 

third party evaluator and we had some low attrition 

issues with our study design. Scrolling down to the 

bottom, I’m given the instruction that I should 

proceed to Step 2 of the SCALER so that my 

organization can work towards building evidence for 

my F-4 program. I just want to quickly note that had 

I selected all the items in the checklist, I would 

have been given the direction that I can proceed 

directly to Step 3, because I would have identified 

evidence, that was rigor enough, and show positive 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

35

impacts. But in this case before going to Step 2, 

you’ll see that your SCALER has a feature that allows 

users to print off their Step 1 results, which would 

produce a pdf copy of this summary page. And in the 

chat we’ll upload an attachment for you to see what 

an example results page can look like when you print 

it or save it to your own personal computer. But if 

users wanted to stop here or return to their results 

at a later time, they can also obtain a unique URL 

token that will allow them to access their step on 

SCALER later on. So to do this, I can click on the 

copy My Step 1 link button which will give me a 

unique URL token that I can paste into a separate 

document and then save it for accessing the SCALER 

later on. Or I can email that token to myself, which 

for me would open up an Outlook email message that 

would already have the URL pre-populated into the 

message and then I can just email it to myself. But 

I’m going to click on this copy my link button which 

will give me my unique URL token. And if I were to 

then go back quickly to the SCALER homepage, and I’ll 

do that right now. And if you recall, when we go to 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

36

that Step 1 button here, and then enter my token, and 

then access my results from the previous session. So 

this is something we wanted to make sure that it was 

available to users so they can revisit previous 

SCALER results, especially as they’re making progress 

towards SCALER. But I’m going to proceed to the Step 

2. So on this Step 2 landing page, we have an 

overview information that describes the purpose of 

this step, which again is to help organizations work 

towards building evidence for their intervention. And 

as we saw in Step 1 and its landing page, users can 

toggle across the different tabs here, to get more 

information on what these different components are 

for this step, for building evidence. And of course 

we have some useful FAQs on the side for users to 

expand as they need, if they desire more information. 

But for our demonstration let’s get started on Step 

2. So as you can see here, they have all the items 

from the first section, which is defining 

intervention, and all the other components that were 

described on the landing page. And for me I’m going 

to fill out this first section here for the 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

37

demonstration we’re doing right now. And my 

organization does have a theory of change. We have 

been running our program for a while in its current 

form and we haven’t made any tweaks, so I would 

consider it mature and stable. We’ve been able to 

implement it with fidelity according to the model and 

the theory of change. And then previously when we 

developed the intervention, we did so as it was 

grounded in research. We really looked at previous 

studies to really think about the design of our 

intervention. So I’m going to check off that as well. 

And then we also, I’m going to just quickly fill out 

the next section which is saying we’ve been able to 

develop a lot of consensus among our key stakeholders 

about the need for evidence, what we’re going to 

measure, and what outcomes are important to us, what 

are the goals of our evaluation, and just broadly 

speaking our organizational leadership, partners, and 

other key stakeholders - are they in support of our 

evaluation. And then the final two sections of this, 

and again users will likely need to come back at 

later times as they work on these processes which can 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

38

take a little while. Your organizations can 

ultimately check off whether they have the resources 

and personnel available to conduct the evaluation. 

And then once that evaluation is conducted, these are 

the standards that referred back in Step 1 in terms 

of the rigor, the effectiveness, and the relevancy of 

the study. So I’m just, for the purposes of this 

demonstration, I’m just going to check off all of 

these here. And I’m hitting next, you’re going to be 

taken to the Step 2 Results page, which will look 

very similar to what we saw with Step 1. And again, 

users can receive a headline that tells them the 

status of their evidence building efforts in this 

case. And we see here we get check marks across the 

board. Go us. And we can then scroll across different 

tabs here if we wanted to look for more information 

about why these components are important. And had I 

not selected anything, we’d have gotten a red X icon 

here, and it would point me to some things I can 

consider doing next as I continue my efforts towards 

building evidence. And scrolling down to the bottom, 

again users can print off their results as needed. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

39

And here I’m given the direction of continuing to 

Step 3. And I’m going to do that next, and then Mary 

Anne is going to take it from here. 

MARY ANNE ANDERSON: Thanks so much, Scott. So, I’ll 

share an overview of Step 3, which focuses on those 

five conditions of scaling readiness that I talked 

through a few minutes ago. So when you first come to 

the landing page for Step 3, you’ll see an overview 

of what we mean by specifying the intervention, which 

is the first checklist. But if you click through the 

tabs there you can also see an overview of the other 

four conditions and checklists. So if we scroll down, 

and click the start button, we can get going on the 

checklists. So I will just give an overview of each 

one and highlight a few examples of items in each of 

these checklists. So, this first one is for 

specifying the intervention just to orient you all a 

bit. At the top you can see a tracker that shows your 

progress in completing the five checklists on the way 

to receiving your results. So this checklist is split 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

40

up into four different sections. So looking at 

identifying core activities first. Using our F-4 

example, we know that the core activities are playing 

flag football and learning about healthy habits. And 

we also talked through the modes and intensities. So 

we know for our program students and their coaches 

meet face to face in groups. So that’s the node. And 

that they meet for 90 minutes a day, three days a 

week, for 12 weeks. So we also know the intensity. So 

for all three of those we can check yes, this is 

defined. If we scroll down the next set of items is 

on clearly defining personnel roles. And from our F-4 

example we also know that these are well-defined. So, 

we have one site coordinator and one coach, per 

school or per site. We know that the site coordinator 

is - covers administrative and programmatic duties, 

and the coach works directly with the students and we 

also know their qualifications. So, five years of 

experience and a bachelor’s for the site coordinator 

and experience working with youth for the coach. So 

for all three of these we can also check yes, this is 

defined. The next two groups of items, the first is 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

41

on clearly defining the setting, so where the 

intervention takes place. And the last set of items 

is about defining who is a program participant and 

what does it mean to complete the intervention. Just 

in the sake of time we’ll skip talking through those 

and move on to the next checklist. So the next 

checklist is about defining the target population. 

This one actually only has three items. The first two 

are about characteristics of the population, and the 

criteria for who is eligible to participate. So, 

inclusion criteria. And again we had a pretty good 

definition of this for our F-4 example. We said that 

students in grades K through 8 at schools receiving 

Title I funds in urban districts throughout the 

country can participate and are eligible. So we can 

say, yes, this is defined, for both of those items. 

But imagine for the next item is about exclusion 

criteria, so determining who’s not eligible to 

participate. And let’s imagine that F-4 leadership 

wants to set some criterion around this. Right now in 

some schools students can participate as many times 

as they’d like in F-4, whereas in other sites 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

42

students can only participate once, and then they 

can’t participate again. So, they’re going to put 

together some standards around this. So they’re 

making plans to develop a definition here. And that 

completes our target population one, so we can hit 

next, and that moves us on to establishing 

implementation supports. So this is our third 

checklist. This is divided into six sections and 

we’ll start with this first group of items on 

monitoring. So for F-4, we don’t know a lot about 

monitoring the program. Let’s assume here that F-4 

leadership is working on plans for who will monitor, 

how they will monitor, and how they’ll report on 

issues and challenges. So we can say, no, but making 

plans to set up for those items. Scrolling down, the 

next group of items is around establishing 

performance goals. So let’s imagine first that 

benchmarks for service delivery are clearly - are 

being actively established. So, they’re close to, but 

they haven’t quite gotten those in place. And they’re 

also concurrently working on a plan for monitoring 

those benchmarks. So, because these are in the works, 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

43

we can say, actively setting it up for those items. 

And then in the interest of time I won’t - we won’t 

go through and fill in the rest, but I just wanted to 

give a quick overview. The next set is on CQI. So 

this covers having a CQI process, regularly using it, 

and updating the intervention based on CQI results. 

The next one is on engaging in personnel training. So 

this looks at whether training is in place and 

personnel are receiving it in a timely manner, both 

before and during implementation of an intervention. 

Next is on communication systems. So these items look 

at whether a communication system exists that 

supports coordination among personnel, as well as 

whether there’s processes in place to make sure 

communications function smoothly and there’s a 

process to resolve issues if they occur. And then the 

last setoff items is on data systems. So this helps 

organizations assess whether they have a data system 

in place and whether there are processes to ensure 

data are entered in a timely manner, analyzed 

appropriately, and then used in decision making. So 

we can click next, and go to the next checklist, 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

44

which is on enabling context. So here we have two 

sets of items. The first set of items looks at 

support for the intervention. So going back to our F4 example, let’s imagine that organization leaders 

are very supportive of the intervention, so we can 

say fully descriptive. But maybe some community 

groups or some schools and some of our target 

communities need to be more brought on board and 

don't know as much about F-4. So we might say 

somewhat descriptive for that item. The other set of 

items here is on engaging in innovation and learning. 

Just to give you a sense of what they cover, they 

help an organization assess whether there’s a history 

of innovation, whether there’s been ways to or the 

organization has used ways to improve the 

intervention model and whether successes and 

challenges are regularly discussed. But we’ll skip 

actually filling those out. And then if we hit the 

next button, we’ll move to our last checklist, which 

is on establishing an implementation infrastructure. 

So this first set of items is on procuring financial 

and human resources. The first item is on adequate 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

45

funding, so imagine for F-4 that the F-4 leadership 

has some funding in place, but they’re actively 

applying for grants to fill in some funding gaps. So 

we can say actively setting it up for that item. For 

our dedicated personnel, which is the next item here, 

let’s also say that they have most of the personnel 

in place, but they have some hiring gaps that they’re 

looking to fill, so we can say actively setting it up 

for that one as well. And the last item in the group 

here is on having a human resources system, so let’s 

just say that’s in place. We can say yes, there’s 

human resources. And the next group we won’t fill in, 

but this is about providing materials and physical 

space. And the next group we won’t fill in, but this 

is about providing materials and physical space. So 

having a thorough description of those resources and 

then making sure each of those are in place. So that 

brings us to the end of our checklist. You would 

normally hit the next button, but we actually prefilled the separate results page to share with you 

all, so Scott has switched over to that. So this is 

what would populate after you complete Step 3 of the 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

46

SCALER. So this is information that organizations can 

use to understand their scaling readiness for each of 

those five components in the framework. So we have 

circle graphs up here, up at the top here, that 

represent how far along an intervention and an 

organization is towards achieving maximum scaling 

readiness in each of those five conditions. So here 

you can see for F-4, they are relatively stronger on 

having a well-specified intervention and a welldefined target population. But they have some more 

room for growth for their implementation supports, 

the enabling context and the implementation 

infrastructure. And then below that, you can click on 

these five tabs that align with each of those 

conditions to get detailed information about your 

results. So the text on each of these tabs is 

tailored to the answers that were given to the items 

in the checklists. So organizations will see 

different headings and tabs based on their answers. 

So for F-4, for specifying the intervention, if you 

don't mind scrolling up, Scott, the headline says 

you’ve begun to specify your intervention, but 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

47

there’s still work to do. And then you have a 

paragraph with some general guidance on where F-4 

might want to focus their efforts to make sure 

they’re ready to scale. If you scroll down in areas 

where F-4 met the criteria, they get these green 

checkmarks and some guidance that’s saying you’re 

doing good, you’re doing a good job here. The last 

group of items, F-4 received a blue clock that’s 

saying you’re making some progress, but you have some 

progress to go, so they partially met the criteria 

here. So the text next to that symbol gives some 

guidance on next steps that they might want to take 

to make sure that they have definitions of 

participation and completion in this case. And just 

to point out on the left hand side, there’s also bar 

charts that show how many of the items the 

organization answered like yes, this is defined too 

in the actual checklists. So just to give one more 

example. Scott, if you could click on implementation 

infrastructure. Thank you. You’ll see a similar 

structure to the guidance here. The headline says 

keep working to ensure, or you have taken some steps 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

48

to make sure the necessary infrastructure is in 

place, but there’s still some work to do. And here 

you can see that F-4 received a blue clock to say 

they’ve made some progress on financial and human 

resources, but haven't kind of achieved maximum 

progress. And then they did receive a red X here. We 

assume that hardly any progress had been made on 

providing materials and physical space. So they have 

some guidance there to help them - to help them sort 

of get ready for scaling in that area, in both of 

those resources. So when personnel are ready, they 

can scroll down and hit continue to planning your 

next steps. And this takes us - this is a static page 

that just helps organizations digest their SCALER 

results and identify action steps. So we provided 

some guiding questions that personnel might want to 

discuss to build their scaling readiness in the tan 

box. And then if your scroll down, you can also hit 

the discussion guide link to bring up these questions 

in a fillable pdf worksheet, which will also be 

included as a handout in the chat box. And then - 

yeah, you can also navigate to the SCALER resources. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

49

So that brings us to a list of documents, tools and 

guides that organizations can use to address areas of 

growth or challenge after completing the SCALER. So 

we organized these resources to align with the SCALER 

framework, so organizations can find resources on 

identifying evidence, building evidence and getting 

ready to scale. And I believe that concludes our 

demonstration. I just want to pause and say that we 

worked with several AmeriCorps grantees to test this 

site while it was being developed and we really would 

like to appreciate, thank them, and we really 

appreciate their time and effort in participating in 

that user testing and helping us get this fully 

completed and launched. And now we, I believe we can 

open it up for questions. 

LILY ZANDNIAPOUR: Thank you so much, Scott and Mary 

Anne. Really appreciate it. 

Katherine Hussey-Sloniker: So let’s start by asking the 

panel some questions. From Savannah Kid, please 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

50

repeat how you arrived at the study design elements, 

or elements that produced credible evidence. 

SCOTT RICHMAN: And I can take that one. So that’s a 

great question. So really, we looked to the existing 

research clearinghouses that cut across multiple 

content areas, whether it’s clearinghouse, Home-V 

that does child and parenting intervention reviews 

and so forth. And really, these clearinghouses define 

evidence and rigorous research in a variety of ways, 

but they have these common set of principles and 

that’s what we really looked to in terms of creating 

the, what we called the standard for identifying 

evidence in the SCALER. So again, that was, things 

really to internal validity in terms of having a 

comparison group, confounds, how well the internal 

validity was carried out in terms of attrition or 

based on equivalence. And these are common themes and 

principles that cut across the different 

clearinghouses. And of course, they define it a 

little bit different, depending on the context of 

their area, but that’s really where we try to ground 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

51

the Step 1 of the SCALER in terms of really 

leveraging the great work that has been done across 

various federal agencies in this space. 

 KATHERINE HUSSEY-SLONIKER: Thank you, Scott. I’m 

going to read the next question, but I would like to 

remind everyone to please place your questions in the 

chat. The next question comes from Laura Schultz, or 

it’s not been covered yet, is there a checklist that 

supports schools and efforts to place participants in 

an effective intervention that addressed their 

specific needs, and further, how to apply the 

taxonomy of intensity and other variables. 

SCOTT RICHMAN: That’s a really good question as well. 

There isn’t anything in terms of checklist items for 

that. At one level, we wanted to be conscious of 

making the SCALER framework generalizable across a 

lot of different content areas. So there isn’t 

anything, for example, healthy future specific or 

economic opportunity specific, or education specific. 

But what I think about when I hear your question 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

52

though is the importance of as you’re scaling, 

working with those community partners or the context 

in which you’re bringing your intervention to, to 

make sure that not only are you, the organization, 

implementing the intervention as you hoped and 

intended, but that there isn’t then any issues with 

implementation fidelity and with carrying it out in 

the schools. And that might be then working with 

those partners to help ensure that on the ground the 

schools are carrying out the intervention as you had 

designed. Because if that doesn’t happen, then you 

start deviating from the intervention model which 

then would start raising concerns, you know, are you 

going to jeopardize the effectiveness that your 

intervention has had in previous efforts. 

 KATHERINE HUSSEY-SLONIKER: Thank you. Would anyone 

else like to add to the response? All right. The next 

question is, is this tool something funders can use, 

or is this specific to practioners and communitybased organization program leadership only? 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

53

LILY ZANDNIAPOUR: I can take a stab at it, and then 

I’ll let Scott and Mary Anne also weigh in. We 

foresee this being used both by practioners and by 

funders. And you know, first off, I just wanted to 

point something about the tool which, even though 

Scott mentioned it, it wasn’t really overtly 

discussed. One of the things that we did as we put 

this tool together and you know, brought it to you is 

get it through our privacy and security analysis. And 

one of the things that you’ll see is that there’s no 

identifiable information about any organization as 

you fill this out. So the privacy of organizations 

and individuals that fill out the form is fully 

protected and we won’t have any of that information 

at hand. So just one point so folks can be aware of. 

There is a token that is generated when you start 

filling out the form. And that’s the token you use 

to go back to the tool and no one would know it 

unless you share that token number with another one 

of your colleagues and so forth. But one of the ways 

we think this tool can be really helpful is, for 

example, funders can use it to kind of gauge where 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

54

their grantees are in terms of their readiness to 

scale. Findings and results from the SCALER tool can 

be used to perhaps support the grantees, to increase 

their readiness. They can be a foundation for 

technical assistance services, funding supports and 

the like to really make them ready to start scaling 

or replicating an intervention that has evidence 

behind it. You know, it could be that an organization 

needs to go back and do a more robust evaluation, 

that’s another thing, or do it more externally so 

there’s more validity and credibility to the results 

of the evaluation. All of those factors come into 

play. It could also be a facilitative tool that is 

used for just communication engagement, right? And 

then on the practioner side, we think that they can 

use it just to self-assess and see where they are, 

take it back to the organization. You know, take it 

to the leadership, to the program implementers, and 

really figure out how best to get ready to do this. 

So we see it having utility and application, you 

know, both for the funder community and for 

practioners on the ground. And with that, I’m going 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

55

to pass it on to Scott and Mary Anne if they want to 

add something. 

SCOTT RICHMAN: Yeah, that was a wonderful summary. I can 

very much see, for example, if there’s a grant maker 

or someone who’s overseeing a portfolio of grants 

that this could even be a discussion tool. You know, 

that’s something that you and an organization that 

you work with can fill out together. Or, as you think 

about what guidance and supports you’d want to 

provide to the grantees you oversee or organizations 

you have support. If you were to fill this out with 

them in mind, it might identify areas where you might 

want to ask about for further support and to say hey, 

do you have this in place, is there anything we can 

do to help provide training, resources or any tools 

to help kind of build your capacity in this area. So 

it could also be a little bit of a self-reflection 

for those project officers or portfolio managers, for 

example, who really have that intimate knowledge of 

their grantees or organizations they work with to 

kind of guide what level of support they may want to 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

56

provide to those organizations. Mary Anne, I don't 

know if you have anything to add as you reflect on 

that question, too. 

MARY ANNE ANDERSON: Yeah, I will say here at Mathematica 

we do participate in a lot of training and technical 

assistance. And we are adapting parts of the SCALER 

to other projects to kind of do like an assessment of 

grant recipients upfront to determine what kinds of 

evaluation or research support and capacity building 

support we could provide in areas related to scaling. 

So that’s another application of it as well. 

LILY ZANDNIAPOUR: Yeah, and then one final point that 

I wanted to make is you know, within an organization, 

within - whether it be - any organization working in 

different areas, you might also have multiple 

perspectives of people who actually work there. You 

know, depending on who fills out the form and you 

know, the amount of knowledge they have, the 

grounding they have, their perspective on things. So 

it could also be something that, again, as Scott also 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

57

referenced, can be a tool that can be used for 

discussion. If there are differences of opinion about 

where you are, it could be a tool that is used to 

sort of frame that discussion and come to agreement 

or make decisions about what to do next to sort of 

bridge gaps and reach consensus or an understanding 

at least of where you are. 

 KATHERINE HUSSEY-SLONIKER: A follow-up question from 

Savannah Kid. Was anyone in your test group 

representative of the American Indian, Alaska Native 

organizations? 

SCOTT RICHMAN: I believe the answer to that is no. When 

we tested the SCALER during its development, we had a 

small number of users. And the focus there as we were 

thinking about different ways to go about that 

testing was to, one, think about the different focus 

areas that AmeriCorps funds. So making, again, going 

back to what I was saying earlier, like wanting to 

ideally create a tool that could be useful to 

organizations who are working in various focus areas. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

58

And the other part that we wanted to focus on is 

trying to get some diversity with regards to the 

individuals that could potentially use the tool. So 

whether this could be a project director, this could 

be an evaluation lead. So I think given the - what we 

were able to do in terms of usability, that is where 

we focused on. But I do think that’s a great 

suggestion in terms of thinking about the broader 

usability and accessibility and applicability of this 

tool. I think that definitely warrants some future 

testing and feedback. Lily, I don't know if you have 

any other thoughts on that as well. 

LILY ZANDNIAPOUR: Well I think that’s a great question, 

and I like the way you responded to it, Scott. And I 

would just add that we are constantly trying to 

think, you know, how we can support this tool, use 

this tool, maybe do more outreach to our - the 

different organizations that we work with. And the 

grantees - the Native American grantees that you’re 

referencing also are among the groups that we are 

thinking of in terms of how to strategize and 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

59

consider engaging with them with this tool and other 

tools and sort of help in best sort of using it to 

help organizations grow and get better and get more 

ready for scaling. So, it’s in the works. 

 KATHERINE HUSSEY-SLONIKER: Thank you. So, Rani 

Kinashida asks, will AmeriCorps engage in any 

regulatory practices for programs to build their 

evidence of effectiveness? i.e., requiring programs 

to use these tools to gain funding? 

LILY ZANDNIAPOUR: I’ll take that. You know, at this point 

we don’t have any specific plans to use this 

particular tool, the SCALER tool, or tie it to 

funding-related things. You know, our grant 

competitions typically require, have an evidence 

component to it and in a sense that sort of 

marginally - it relates to this tool. But no, we 

think that this tool needs to be a standalone, and 

for folks to use as they see fit. And it’s meant to 

be a resource, and we didn’t want to have it have 

that big baggage attached to it, related to funding. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

60

 KATHERINE HUSSEY-SLONIKER: So a follow from Rani, how 

can we raise awareness of the AmeriCorps evidence 

exchange and the tools available for your program to 

build out programs evidence base? 

LILY ZANDNIAPOUR: That is a great question. Evidence 

exchange, you know, has been a repository that we’ve 

used and all the resources that we’ve put on there. 

There’s just a ton of information there about 

effective programs, a lot about different programs 

that have already begun on the sort of path of 

building their evidence. And there’s many, many 

resources and the SCALER included are available on 

the resources page. We do everything we can to try to 

make sure that the word is out, but this is not 

something that we can do by ourselves. You know, we 

need our agency partners to do that and partners in 

the communities to help spread the word. So I think 

that our team at AmeriCorps and within our unit does, 

we do everything we can to promote evidence exchange 

and make sure that all of this information and these 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

61

tools and resources are in the public domain and are 

magnified. And that is going to be an ongoing effort 

to really bring more attention to it, spotlight it 

and so forth. In fact, one of our hopes is that 

SCALER tool would be a shining and great tool to 

bring more attention to those resources and 

information that is on the website. 

 KATHERINE HUSSEY-SLONIKER: So the next question - 

there are a lot of questions here from the people. So 

be aware, there’s more questions coming. Calvin 

Patterson wants to know, is there more weight, to 

part - more weight to part of the process, e.g., 

financial and personnel is more important than 

materials and setting? 

SCOTT RICHMAN: I can jump in with that one. Yeah, so at 

this stage we think of the SCALER as something where 

everything matters equally. That one thing isn’t more 

important than the other. But I think what you’re 

question suggests also, that could be a great 

empirical question as you think long-term when 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

62

grantees are scaling and whether they have success in 

scaling or not. That might be an interesting thing to 

look at in terms of what factors did they have in 

place or not when they sought to scale. And I think 

that sort of analysis research, can help I think 

illicit ideas of you know, what is more important or 

not. But at this stage, we are thinking of this as 

these are our, you know, the things that need to be 

in place for scaling and not prioritizing one or the 

other. 

 KATHERINE HUSSEY-SLONIKER: All right. And the last 

question that I have in the open chat, please feel 

free, we have a few more minutes before the chat 

closes, is basically will we have access to the 

recording to absorb the information again and use it 

for discussion? The answer is yes. The recording will 

be made available through the impact page, webinar 

page of the AmeriCorps Office of Research and 

Evaluation in about two weeks. It will have support 

materials with it to include the slides as well. All 

right, lots of messages listed in the chat talking 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

63

about how wonderful the SCALER tool is. We have time 

for one more question, if someone would like to place 

it in the chat. As I see none, I guess I’d like to 

ask for closing comments from each one of you here, 

both Mathematica, Scott and Mary Anne, and Lily from 

AmeriCorps. Closing comments? 

SCOTT RICHMAN: Sure. I just want to thank everyone, 

again, for attending. As Lily and Mary Anne said at 

the beginning, our hope is to really just support the 

work that communities, organizations are doing on the 

ground and really our mission at the beginning was to 

create something that was usable to really support 

the, you know, what organizations bring to 

individuals and communities. So we really appreciate 

the questions. And again, as Mary Anne also noted, 

the users who helped to test this along the way and 

kind of where, give us some bearings that we were on 

the right track with creating something useful. And 

we hope that you are able to put this in practice. 

And thank you again for your questions and 

attendance. 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

64

LILY ZANDNIAPOUR: Yeah. On my end, I want to first thank 

Scott and Mary Anne. The partnership with Mathematica 

has been so wonderful. We have learned so much in the 

process and there is just a ton of information. And 

all of you, I thank you for attending today’s webinar 

and really encourage you to engage with the tool and 

check out all these resources that were generated as 

part of this initiative. They are very, very 

informative and useful. And, you know, I think they 

have set the stage for hopefully bigger and better 

things in the future. And again, thanks for the 

questions. As I noted in the beginning, there have 

been many other people that have been involved in 

this effort and I just want to thank all of them too 

in the background and supporting, have been 

supporting this work, both on the agency side and on 

Mathematica’s side to bring this project with all of 

this wonderful products to everyone. And yeah, just 

to thank everyone for their work and contributions. 

And hopefully, you can find it really useful and 

engage with the tool. Mary Anne? 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

65

MARY ANNE ANDERSON: I totally echo everything that’s 

been said. So glad to see how many people were able 

to attend today. And I do, you know, want to 

acknowledge this was not just Scott and me. We had a 

large team on Mathematica’s side as well that helped 

put together the resources that are in report from, 

as well as our kind of more product development side 

that helped to put together the tool for the website. 

So thank you to all those team members. 

 KATHERINE HUSSEY-SLONIKER: Okay. Thank you all very 

much for attending. We’ve provided the speakers’ 

email addresses here on the page. A recording will be 

available in about two weeks. And the AmeriCorps 

SCALER tool is located on the evaluation resources 

page, and was also embedded as part of the chat that 

you see here. All follow-up and support materials 

will be included in the recording as attached - 

attached supplement. There will be the slides, there 

will be all of the official worksheets and documents 

that were placed in that chat as well. So we’d like 
ICF Transcription 

Deepening Our Understanding of What 

It Takes To Scale An Effective Program: 

Scaling Evidence-Based Models Initiative and Tool 

66

to thank you very much for attending the webinar 

today. Thank you. 

[End of File]