Anchor

Scaling Evidence-Based Models: Document 

Review Rubrics

September 24, 2021 

Scott Richman and Rebekah Selekman

Submitted to:

AmeriCorps

Office of Research and Evaluation 

250 E Street, SW

Washington, DC 20525

Project Officer: Lily Zandniapour

Contract Number: GS10F0050L/CNSHQ16F0049

Submitted by:

Mathematica 

111 East Wacker Drive, Suite 3000 

Chicago, IL 60601

Telephone: (312) 994-1002

Facsimile: (312) 994-1003

Project Director: Scott Richman

Reference Number: 50332
This page has been left blank for double-sided copying.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

iii

CONTENTS 

I. INTRODUCTION ................................................................................................................................. VI

II. EVIDENCE REVIEW PROCESS AND RUBRIC .................................................................................. 3

III. SCALING ACTIVITIES REVIEW PROCESS AND RUBRIC ................................................................ 7

III.A. Intervention Review Section.................................................................................................... 10

III.A.1. Context for the intervention ........................................................................................ 10

III.A.2. Well-specified intervention.......................................................................................... 13

III.A.3. Well-defined target population.................................................................................... 18

III.A.4. Implementation supports ............................................................................................ 20

III.A.5. Enabling context ......................................................................................................... 26

III.A.6. Implementation infrastructure..................................................................................... 28

III.B. Scaling Readiness Section...................................................................................................... 30

III.B.1. Scaling approach and its context ............................................................................... 30

III.B.2. Well-specified intervention.......................................................................................... 31

III.B.3. Well-defined target population.................................................................................... 35

III.B.4. Implementation supports ............................................................................................ 36

III.B.5. Enabling context ......................................................................................................... 39

III.B.6. Implementation infrastructure..................................................................................... 41

REFERENCES............................................................................................................................................ 43
This page has been left blank for double-sided copying.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

v 

TABLES

II.1 Standards for categorizing evidence of effectiveness ..................................................................... 3

II.2 Evidence review rubric examples .................................................................................................... 5

III.1 Examples of applying scaling activities rubric questions on specifying the intervention 

proposed for scaling......................................................................................................................... 8

III.A.1 Questions about context for the intervention ................................................................................. 11

III.A.2 Questions about specifying the intervention .................................................................................. 13

III.A.3 Questions about defining the target population ............................................................................. 18

III.A.4 Questions about implementation supports .................................................................................... 20

III.A.5 Questions about the enabling context............................................................................................ 26

III.A.6 Questions about implementation infrastructure ............................................................................. 28

III.B.1 Questions about scaling approach and local context .................................................................... 31

III.B.2 Questions about specifying the intervention proposed for scaling ................................................ 32

III.B.3 Questions about the target population proposed for scaling ......................................................... 36

III.B.4 Questions about implementation supports for the intervention proposed for scaling.................... 37

III.B.5 Questions about the enabling context for the organization proposing the scaling ........................ 40

III.B.6 Questions about implementation infrastructure for the organization proposing the scaling.......... 42

FIGURES

I.1 The Scaling Programs with Research Evidence and Effectiveness (SPREE) process................... 2
This page has been left blank for double-sided copying.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

1 

I. INTRODUCTION 

AmeriCorps was established as a federal agency in 1993 with a mission to improve lives, 

strengthen communities, and foster civic engagement through service and volunteering.1 The 

AmeriCorps State and National program and the three AmeriCorps Seniors programs (RSVP, 

Senior Companions, and Foster Grandparent) are the main avenues through which the agency 

currently achieves this mission; the Social Innovation Fund (SIF) also provided funding to 

innovative community-based programs from 2010 to 2016. 

In 2016, AmeriCorps funded the Scaling Evidence-Based Models project to gain insights about 

which of the AmeriCorps State and National and SIF interventions were effective and might be 

ready to scale. Conducted by Mathematica, this project is designed to generate practical 

knowledge about how AmeriCorps might foster the successful scaling of effective interventions 

and support AmeriCorps’ efforts to identify which interventions work and how they can work for 

more people. This information will also help inform the agency’s interest in identifying the 

intervention components that are critical for an intervention’s effectiveness. 

The project developed and applied the Scaling Programs with Research Evidence and 

Effectiveness (SPREE) process to help AmeriCorps identify which of its funded interventions 

demonstrate evidence of effectiveness and a readiness to scale (Maxwell and Richman 2019). 

The SPREE process has two distinct parts, each of which is represented in Figure I.1. The first 

part is designed to identify AmeriCorps-funded interventions with evidence of effectiveness, 

which we define as those that met the project’s effectiveness standards and produced favorable 

findings on the majority of targeted outcomes among participants receiving the intervention. We 

often refer to interventions that have provided evidence of effectiveness as interventions that 

work. The second part of the process is designed to determine the extent to which the effective 

interventions and the organizations implementing them demonstrate a readiness for scaling. 

1 As of September 29, 2020, the Corporation for National and Community Service is operating under the name 

AmeriCorps.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

2 

Figure I.1. The Scaling Programs with Research Evidence and Effectiveness (SPREE)

process

In this report, we provide an overview of the process, including the rubrics, we applied to the 

interventions funded from 2015 to 2019 by the AmeriCorps State and National program and 

from 2010 to 2015 by the SIF. These programs and grantee cohorts were of interest to 

AmeriCorps and had evidence and scaling documents available for review for this project. 

Chapter II describes the evidence review process that we applied to evidence documents to

identify interventions with evidence of effectiveness. Chapter III describes the scaling review 

process that we used with scaling plans documents to identify which interventions and 

organizations were ready to scale. We discuss this process and how we applied review rubrics to 

the Scaling Evidence-Based Models project so that they can be of use to other funding agencies 

and grantmaking entities that are interested in identifying and supporting the scaling of 

interventions with evidence of effectiveness. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

3 

II. EVIDENCE REVIEW PROCESS AND RUBRIC 

In this chapter, we describe the process that Mathematica used for extracting information on the 

characteristics of the 32 AmeriCorps-funded interventions identified as having evidence that they 

work (Richman and Streke 2020). These interventions were among those proposed by 

AmeriCorps State and National 2015–2019 and SIF 2010–2015 grantees that were judged by an 

independent, third-party evaluator contracted by AmeriCorps as providing strong or moderate 

evidence for their intervention. 

Table II.1 describes the criteria we applied to each evidence document (typically research reports 

or journal articles) AmeriCorps grantees submitted for these interventions to determine eligibility 

for further review. We developed these criteria to align with the overarching goal of the project: 

to identify interventions with evidence of effectiveness that grantees intended to scale to produce 

a wider impact. 

Table II.1. Standards for categorizing evidence of effectiveness 

Criterion Question Yes or No

Project objectives

Study design Did the study use a comparison group in its research design to allow for causal 

inferences to be drawn about the impact of the intervention?

Reporting on 

methods

Did the study sufficiently describe its research design and statistical approach? For 

example, did the study provide adequate information on the formation of its study 

groups and the statistical procedures used to gauge the impacts of the intervention?

Evaluator 

independence

Was the study conducted by an evaluator external to the grantee?

Impact findings Did the study show consistently favorable findings on the intervention’s outcomes of 

interest? Having consistently favorable findings is defined as having intervention 

impacts that are statistically significant across the majority of confirmatory research 

questions examined in the study.

Scaling plan 

alignment

Is the intervention assessed in the study aligned to the intervention the grantee plans 

to scale up? 

Internal validity

Attrition If the study was a randomized controlled trial (RCT), did the study experience high 

attrition?a

Attrition from the study sample (overall and differential attrition between the study 

groups) can compromise the initial equivalence of the groups and lead to biased 

estimates of an intervention’s effects. To determine the potential for attrition bias, we 

used the What Works Clearinghouse (2020) model,b which combines the overall and 

differential attrition rates a study may encounter, and generates acceptable and 

unacceptable levels of bias.

Reassignment If the study was an RCT, did the study use reassignment?

Reassignment, or switching study participants from the comparison group to the 

intervention group (or vice versa) after random assignment, is another major internal 

validity concern for RCTs. Using reassignment would undermine the important 

assumption that the intervention and comparison groups are similar at baseline due 

to random assignment. 

Baseline 

equivalence

For an RCT study that had high attrition or used reassignment or a study that used a 

quasi-experimental design, did the study demonstrate baseline equivalence? That is, 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table II.1 (continued)

4 

Criterion Question Yes or No

did it show that the intervention and comparison groups in the final analytic sample 

are similar on a pre-test outcome measure at baseline?

When intervention and comparison groups are not formed through a randomization 

process (or when RCTs experience high attrition or use reassignment), it is possible 

that groups could differ in unobservable ways at baseline even if they appear similar 

on their measured characteristics. Unmeasured baseline differences can bias 

estimates of the intervention’s impact. As a result, studies must demonstrate baseline 

equivalence, showing that the intervention and comparison groups have similar 

observable characteristics at baseline. For this review, equivalence had to be 

established on at least one primary outcome measure, meaning there were no 

statistically significant differences on primary outcome measures for the analytic 

sample at baseline. If a statistical difference occurred, the study would have to 

statistically control for these baseline differences when analyzing post-test outcomes.

Confounding 

factors

Does the study have the presence of a confounding factor that makes it difficult to 

distinguish between the effect of that factor and the intervention, thus making it 

impossible to attribute any potential impacts solely to the intervention?

One common confounding factor is when some aspect of the research design lines 

up with either the intervention or comparison group, also referred to as an “n = 1 

confound.” For example, if the intervention or comparison group comprises a single 

unit (school, classroom, teacher), the study would not be able to isolate the effect of 

the intervention from other observable or unobservable characteristics of that unit. 

Interventions bundled with other services not being studied can also make it 

impossible to isolate the unique impact of the intervention. 

a Attrition is examined only for RCTs and is not applicable to studies using quasi-experimental designs because for 

this study design type, only the analytic sample with post-test outcomes (not the initial sample present at the onset of 

the study) is considered when determining evidence quality. b We used the What Works Clearinghouse’s “optimistic” assumptions for assessing unacceptable levels of expected 

bias. See https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf. 

When applying these criteria, we did not consult other external sources (for example, we did not 

query authors to collect more information not available in the documents or conduct web 

searches to find additional information on the interventions). Several interventions supported 

their effectiveness by providing more than one evidence document that met the project’s 

effectiveness standards. A separate assessment was created for each evidence document, and the 

information was then combined to provide an intervention-level summary of characteristics that 

informed the evidence supporting each intervention. Table II.2 shows practical examples of how 

the evidence review rubric criteria can be applied to evidence documents submitted for 

(fictitious) interventions/programs and what the final assessments concluded. These examples 

demonstrate the level of detail that reviewers can record to explain the response option they 

selected for a given rubric item. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

5 

Table II.2. Evidence review rubric examples 

Criterion Summer Boost Reading Intervention Youth Fitness Fun Program

Project objectives

Study design Yes Yes 

Reporting on methods Yes Yes 

Evaluator independence Yes Yes 

Impact findings Yes Yes 

Scaling plan alignment Yes Yes 

Internal validity

Low attrition Not applicable

The study used a quasi-experimental design. 

Yes 

No reassignment Not applicable

The study used a quasi-experimental design. 

No

The study discussed how several youth 

who were randomly assigned to the 

comparison group ultimately received the 

program. When examining the impact of 

the fitness program, the study analysis 

switched these youth into the intervention 

group (in other words, the study reassigned 

these youth into a different study group).

Baseline equivalence Yes No

Because the study used a randomized 

controlled trial that experienced participant 

reassignment, it must establish baseline 

equivalence with the final analytic sample 

on a pre-test outcome measure at baseline. 

However, the study did not conduct any 

baseline equivalency analyses. 

No confounding factors Yes Yes 

Final assessment

The study met all the criteria that show 

evidence of the intervention’s effectiveness. 

The study’s randomized controlled trial 

experienced reassignment but did not then 

establish baseline equivalence with the 

final analytic sample on a pre-test outcome 

measure at baseline. As a result, the study 

did not meet all the criteria that show 

evidence of the intervention’s effectiveness. 

Reviewing these documents required systematic procedures to ensure that we reliably extracted 

standardized information from each document. Trained reviewers documented the interventions’ 

characteristics using the rubric; in addition, a third-party consultant, external to Mathematica, 

reviewed all evidence documents authored by Mathematica to reduce any potential conflicts of 

interest. Reviewers classified each intervention and documented the characteristics of each 

intervention being reviewed, describing the intervention as implemented and evaluated. Study 

team leaders also reviewed the evidence documents and completed assessments to ensure the 

information was complete and accurate. The reviewer and study team leader discussed and 

resolved any discrepancies, and an additional study team leader was consulted if a consensus 

could not be reached. As a final quality assurance procedure, a Mathematica researcher external 

to the project reviewed all finalized assessments.
This page has been left blank for double-sided copying.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

7 

III. SCALING ACTIVITIES REVIEW PROCESS AND RUBRIC 

This chapter describes the process that Mathematica used for extracting information on the 

characteristics of the 25 AmeriCorps-funded organizations and their interventions with evidence 

of effectiveness (that were identified using the evidence review rubric described in Chapter II). 

These organizations were among those funded by AmeriCorps State and National 2015–2019 

and SIF 2010–2015 grants. 

AmeriCorps required applicants to submit different types of documents to plan for 

implementation of an intervention based on whether the request for funding was through the 

AmeriCorps State and National program or the SIF program. From 2015 to 2019, those applying 

for AmeriCorps State and National funding submitted applications that explained the rationale 

and approach for their planned interventions, a plan for supervising AmeriCorps members, 

organizational capacity to provide the intervention, and other details of their planned 

implementation approach. We used this application as the scaling plan for these organizations. 

From 2010 to 2015, SIF organizations were required to submit scaling plans that described their 

approach for scaling, their growth goals and indicators, the actions required to achieve their 

growth goals, and a monitoring plan. 

We used a two-part systematic review process to collect and record information about the 

intervention and the extent to which organizations’ scaling plans demonstrated that the 

organizations and their interventions met the conditions necessary for successful scaling. 

Members of the study team developed a two-part rubric for systematically reviewing documents 

submitted by AmeriCorps organizations that (1) provided evidence of an intervention’s 

effectiveness and (2) described plans for scaling the effective intervention.

• Intervention review section. Reviewers used this first part of the rubric to review the 

evaluation reports submitted to AmeriCorps by organizations as support for their 

interventions’ effectiveness. In this section, reviewers described the intervention as 

implemented and evaluated. When completing this section, we did not consult other external 

sources (for example, we did not query authors to collect information not available in the 

documents or conduct web searches to find additional information on the interventions). If 

more than one evidence document that met the project’s effectiveness standards was 

submitted, we created a separate assessment for each evidence document and then combined 

the information to provide an intervention-level summary of characteristics that informed the 

evidence supporting each intervention. We used the completed assessments of the evidence 

documents to provide context for the review of the scaling plan. 

• Scaling readiness section. Reviewers used this second part of the rubric to review the 

scaling plans and application narratives submitted by AmeriCorps organizations. This part of 

the rubric was designed to understand organizations’ planned activities for scaling the 

interventions with evidence of effectiveness by capturing information on the intervention’s 

readiness to be scaled and the organization’s ability to support that scaling. This section 

contains six subsections that align with our framework for successful scaling. The first 

subsection collects information about the approach to scaling and the context in which the 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

8 

intervention will be implemented. Each of the next five subsections collects information on 

one of the five conditions needed for intervention and organizational readiness for scaling: 

(1) a well-specified intervention, (2) a well-defined target population, (3) implementation 

supports, (4) the implementing organization’s enabling context, and (5) the organization’s 

implementation infrastructure.

For each condition of scaling readiness, reviewers determined how well the organization 

described the condition and demonstrated readiness for scaling. Most conditions consisted of 

subcomponents that made up the overall condition (see Selekman et al. 2020 for additional 

information on the scaling readiness conditions). For example, the overall condition of a wellspecified intervention was measured with questions about five dimensions of intervention 

specification: (1) core elements, (2) mode of service delivery, (3) intensity, (4) personnel needs, 

(5) setting, and (6) definition of completion. When a condition consisted of components, we 

measured the specification of each component and used this to describe readiness for the overall 

condition. Reviewers pulled information from the scaling plans to determine whether 

organizations provided a full specification, a limited specification, or no specification for each 

condition. 

Table III.1 shows practical examples of how the scaling readiness rubric section can be applied 

to the scaling plans that organizations submit for (fictitious) their interventions/programs. The 

provided examples are for a subsample of scaling activities rubric questions (those that pertain to 

specifying an intervention). These examples demonstrate the level of detail that reviewers can 

record to explain the response option they selected for a given rubric item by clarifying what 

aspects of scaling readiness the scaling plans have and have not addressed. 

Table III.1. Examples of applying scaling activities rubric questions on specifying the 

intervention proposed for scaling

Question We Read Together Intervention Planning for Success Program

I. Core elements 

1. Are the core elements 

of the intervention well 

specified?

Yes (full), core elements are well 

specified, with details provided.

The intervention will deliver an early 

childhood curriculum taught using roleplay that addresses literacy and 

social/emotional development. The 

curriculum includes weekly activity 

packets and two storybooks for each 

month.

Yes (limited), core elements are 

specified, but with no details provided.

The program participants will receive 

access to financial counselors who will 

conduct financial assessments of each 

participant and offer individualized 

coaching on budgeting, saving, and 

credit building. The scaling plan 

discussed how employment counselors 

will also be available to eligible 

participants but no further information is 

provided on what types of services the 

employment counselors will offer. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.1 (continued)

9 

Question We Read Together Intervention Planning for Success Program

2. Do the proposed core 

elements differ from 

those in the 

evaluation? 

No, proposed core elements for the 

evaluation and intervention are the 

same. 

Yes (limited), proposed core elements 

differ from those evaluated, but with no 

details provided.

The core elements of the intervention 

that were evaluated did not include 

employment counselors. This is a new 

core element that is being added to the 

program.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the core elements? 

Not applicable No, a rationale or support is not 

provided. 

II. Mode of service delivery

4. Is the service delivery 

model well-specified? 

Yes (full), the service delivery model is 

well-specified, with details provided.

The curriculum will be delivered through 

in-person, one-on-one home visits, 

activity packets for parents to complete 

at home with their children, and parent 

group meetings.

Yes (limited), the service delivery model 

is specified, but with no details provided. 

Financial counselors will meet with 

participants in-person, one-on-one. 

Service delivery information on the new 

services provided by employment 

counselors was not described.

5. Does the proposed 

service delivery model 

differ from the service 

delivery model 

implemented during 

the evaluation? 

No, the proposed and evaluation service 

delivery models are the same. 

No, the proposed and evaluation service 

delivery models are the same. 

6. Does a rationale or 

support exist for the 

proposed change(s) to 

the service delivery 

model? 

Not applicable Not applicable

III. Intensity

7. Is the intensity of the 

intervention wellspecified?

Yes (full), the intensity is well-specified, 

with details provided.

Home visits will last 60 minutes and be 

offered weekly over 30 weeks. Parents 

will work with their children on weekly 

packets for 15 minutes per day, five days 

per week. Parent group meetings will be 

held monthly.

No, the intensity is not mentioned.

No information was provided on the 

dosage, duration, and frequency of the 

services provided by the financial 

counselors and employment counselors.

8. Does the proposed 

intensity of the scaled 

intervention differ from 

the intensity of the 

intervention 

evaluated? 

No, the proposed and evaluated intensity 

is the same.

Do not know; the intensity is not defined.

9. Does a rationale or 

support exist for the 

proposed change(s) to 

the intensity of the 

intervention? 

Not applicable Not applicable
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

10

Reviewing the scaling plans required systematic procedures to ensure that the study team reliably 

extracted standardized information from each evaluation report. Study team leaders trained 10 

reviewers, including a third-party consultant, to identify the scaling approach and document 

scaling readiness criteria using the rubric. The consultant was external to Mathematica and 

reviewed all evidence documents in which Mathematica was the evaluator of the intervention to 

avoid any potential conflicts of interest. Reviewers classified the scaling approach of each 

intervention and documented the characteristics related to scaling readiness of each intervention 

being reviewed. Study team leaders reviewed all finalized assessments to ensure the information 

was complete and accurate. As a final quality assurance procedure, a Mathematica researcher

external to the rest of the review process reviewed finalized assessments.

III.A. Intervention Review Section

The intervention review section of the rubric is structured to extract information from documents 

submitted by AmeriCorps grantees that report evaluation findings for AmeriCorps-funded 

interventions. The information extracted through the use of this rubric section describes the 

characteristics of the intervention when it was evaluated. It contains six subsections. The first 

subsection collects information about the context in which an intervention was implemented, and 

each of the next five subsections collects information on one of the five conditions needed for 

scaling. 

III.A.1. Context for the intervention

Table III.A.1 provides the first subsection of the rubric, which includes three types of questions 

about the context in which the intervention unfolds: 

1. Description of the intervention, according to AmeriCorps nomenclature. The nomenclature 

includes the intervention’s (1) focus area, (2) topic area, (3) intended outcome domain(s), and 

(4) intended outcome(s). The appendix provides a list of predetermined response options for 

classifying the intervention per the nomenclature, with the “other” response indicating that 

the predetermined areas do not adequately describe the intervention. 

2. Local area context, including contextual features that could affect implementation of the 

intervention, including any regional, state, and local contextual features; demand for the 

intervention at the local level; and partners.

3. Measures of implementation, including the percentage of participants who complete the 

intervention and cost per participant. The measures require a definition of components 

(participant, completer, and costs). 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

11

Table III.A.1. Questions about context for the intervention

Question Response options Additional guidance

I. Description of the intervention 

1. What is the 

intervention’s focus 

area?

Only one focus area. Focus areas include disaster services, economic 

opportunity, education, environmental 

stewardship, healthy futures, nonprofit 

organizational capacity, and veterans and 

military families.

2. What is (are) the 

intervention’s topic 

area(s)?

Up to two topic areas listed in 

Table 1 in the appendix.

The topic area provides greater specificity about 

the intervention’s broad goals than about the 

focus area. For example, an intervention may fall 

into the “economic opportunity” focus area. If it 

helps low-income people obtain employment, 

however, it would fall into the “employability” 

topic area.

3. What is (are) the 

intervention’s outcome 

domain(s)?

Up to two outcome domains 

listed in Table 1 in appendix that 

fall within the topic area selected.

The outcome domain provides greater specificity 

than the topic area. The intervention aimed at 

helping low-income people obtain employment 

may do so by providing industry skill training, 

which is the intervention’s outcome domain. The 

veterans and military families, disaster services, 

and nonprofit organizational capacity focus 

areas and some of the education topic areas do 

not have outcome domains. 

4. What is (are) the 

intervention’s 

outcome(s)?

Up to four outcomes listed in 

Table 1 in appendix. 

Outcomes are the targets that the intervention 

seeks to affect. In the example of the industry 

skill training intervention, an outcome might be 

certifications/skills obtained by participants. 

Outcomes may span several outcome domains 

within a given topic area. 

II. Local area context

1. Might the regional, 

state, and local context 

facilitate or challenge 

implementation of the 

intervention?

Yes, contextual considerations 

might affect implementation.

No, contextual considerations 

are not mentioned.

Regional, state, and local contextual factors can 

challenge or facilitate implementation of an 

intervention. Such factors may include 

community stakeholders, the political 

environment, or the local economy. For example, 

an affordable housing intervention that was 

implemented during a local economic downturn 

could see increased demand for its services.

Question 2 is skipped with a “no” response.

2. What are the major 

features of the 

regional, state, and 

local context that could 

have affected 

implementation of the 

intervention?

[open-ended] Describe any features of the regional, state, or 

local context that may have been salient in 

implementing the intervention. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.1 (continued)

12

Question Response options Additional guidance

3. Does evidence exist 

for local demand for 

the intervention? 

Yes (full), local demand exists for 

the intervention.

Yes (limited), local demand 

exists, but details are not 

available.

No, local demand for the 

intervention is not mentioned.

Local demand may be identified by conducting a 

needs assessment, landscape analysis, or 

informal data collection through attendance at 

community meetings or conversations with key 

stakeholders. 

4. Do key partners play a 

role in implementing 

the intervention?

Yes, implementation partners are 

identified.

No, partners are not identified.

The organization may use partners to deliver 

services or assist with implementation of the 

intervention. 

Question 5 is skipped with a “no” response. 

5. What role do the key 

partners play in 

implementing the 

intervention?

[open-ended] Describes the partners or contractors that 

assisted with implementation of the intervention, 

including the role played by each.

III. Measures of implementation

1. What is the intended

overall completion rate 

for participants in the 

intervention?

[open-ended] Intended intervention completion rate during a 

specific time period. Interventions often set forth 

completion goals for participants. 

2. What is the actual 

overall completion rate 

for the intervention?

[open-ended] Actual rate of participants’ completion of the 

intervention during a specific time period. 

3. Is cost-per-participant 

information discussed?

Yes, cost per participant is 

discussed.

No, cost per participant is not 

mentioned. 

Cost-per-participant information may be 

presented as part of a cost-benefit analysis.

Question 4 is skipped with a “no” response.

4. What is the cost per 

participant for the 

intervention?

[open-ended] The average cost of delivering the intervention 

per participant, including the year the costs were 

captured and information about the resources 

reflected in the calculation (the cost of personnel

time, materials, space rental, partner services, 

among other items).
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

13

III.A.2. Well-specified intervention

This subsection addresses the first condition of the scaling readiness framework—a wellspecified intervention. It includes questions in six areas (Table III.A.2). The first area describes 

the intervention, and the next four describe a different dimension of each core element. The last 

area describes what it takes a participant to complete the intervention.

1. Core elements, including identification of the intervention by its name. Core elements are 

activities that were part of the intervention and were intended to achieve the intervention’s 

ultimate outcome domain and outcomes. 

2. Mode of service delivery, including how each element was designed to be delivered to 

participants. 

3. Intensity, including how often and for how long each element was offered to participants 

and how much of each element each participant received.

4. Personnel needs, including the intervention’s personnel structure. A personnel structure 

includes the job titles of personnel who delivered the intervention, the number of individuals 

involved in delivery of the intervention, reliance on AmeriCorps national service 

participants, and qualifications of personnel. 

5. Setting, including location (geography) and venue (the place where activities occurred, such 

as a community center, home, nonprofit organization, park, school). 

6. Definition of completion, including criteria for being considered a participant in the 

intervention and what a participant needed to do to be considered a “graduate” or completer 

of the intervention. 

In some cases, the rubric refers to a standardized, name-brand intervention (for example, Teach 

For America). A standardized intervention adheres to a specific model or curriculum in terms of 

the scope and sequence of the intervention’s content. It spells out the same type and amount of 

services that each participant should receive. In a nonstandardized intervention, each 

participant’s need determines the type and amount of services delivered; as a result, each 

participant experiences the intervention differently. 

Table III.A.2. Questions about specifying the intervention

Question Response options Additional guidance

I. Core elements

1. What is the name 

of the 

intervention?

[open-ended] Use of a standardized, name-brand intervention rather than the 

local intervention name. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.2 (continued)

14

Question Response options Additional guidance

2. What are the 

designed core 

elements of the 

intervention?

[open-ended] Core elements are the set of activities or curricula that result in 

participants’ receipt of services, such as a description of services, 

the content area of the intervention, the existence of a 

standardized intervention model, the services or curriculum 

comprising the intervention, and the content area of the 

intervention to be delivered. A standardized intervention is 

considered to have a clearly specified intervention. Examples of 

core elements are tutoring, facilitator-led classes or workshops, 

one-on-one coaching, case management, electronic or telephone 

communication with participants, and building the capacity of an 

organization or partner organization. 

The description must align with the selected categorical responses 

in Table A.1 in the appendix. 

3. Were all of the 

intervention’s core 

elements 

implemented?

Yes, all core elements 

were implemented.

No, some core 

elements were not 

implemented.

Question 4 is skipped with a “yes” response.

4. Which core 

elements of the 

intervention were 

implemented? 

[open-ended] Identify all core elements implemented.

5. Were any 

purposeful

changes made to 

the intervention 

core elements 

implemented as 

compared to an 

earlier version of 

the model 

(including 

adaptations and 

modifications)?

Yes, intervention core 

elements were 

purposefully changed. 

No, intervention core 

elements were not 

purposefully changed. 

Organizations may make changes to the intervention core 

elements prior to implementation, referred to as purposeful 

changes. Purposeful changes could extend to the intervention 

content, service delivery mode, intensity, and personnel needs (for 

example, a model designed for classroom training of service 

providers may have changed to one-on-one training). 

Implementing a “2.0” version is also a purposeful change. 

Differences between intended and actual implementation are not 

considered purposeful changes if, for example, the actual duration 

was shorter than intended, but the intended duration did not differ 

from the intervention model. 

Question 6 is skipped with a “no” response. 

6. What purposeful 

adaptations or 

modifications did 

the organization 

make?

[open-ended] Describes all purposeful changes made to the intervention core 

elements. 

7. Were any 

planned changes 

made to the 

intervention during 

the 

implementation 

period?

Yes, planned changes 

were made to the 

intervention during

implementation. 

No, planned changes 

were not made to the 

intervention during 

implementation. 

During implementation, an organization may decide to make 

changes to an intervention, referred to as planned changes. 

Planned changes may include modifications to activities or 

services, mode of delivery, source of delivery, setting, or intensity. 

For example, an organization may have decided to offer an 

intervention in a new setting, extend the period of service delivery, 

or change the originally planned activities in order to test whether 

such changes would increase participant outcomes. 

Question 8 is skipped with a “no” response.

8. What were the 

planned changes 

to the intervention 

and the reasons 

for them?

[open-ended] Describes all planned changes to intervention core elements.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.2 (continued)

15

Question Response options Additional guidance

II. Mode of service delivery

1. What is the mode 

of delivery for 

each element?

[open-ended] The service delivery model specifies how services are delivered, 

who delivers services, and the mode through which services are 

delivered. Organizations should provide a description of the way in 

which each core element is delivered. For example, peer 

educators or teams of facilitators may deliver services in groups or 

one-on-one meetings and through the use of electronic 

communication or in-person meetings.

III. Intensity

1. What is the 

intended

frequency of 

delivery of each 

element?

[open-ended] For each element implemented, describes how often the element 

is intended to be offered during a specific time period. 

2. What is the 

frequency with 

which each 

element was 

actually 

implemented (on 

average)?

[open-ended] For each element implemented, describes how often the element 

was actually offered during a specific time period. 

3. What is the 

intended duration 

of attendance for 

each element?

[open-ended] For each element implemented, includes the intended length of 

time participants were supposed to receive an element during a 

specific time period.

4. What is the actual

duration of 

attendance for 

each element (on 

average)?

[open-ended] For each element implemented, describes the actual length of time 

participants received an element during a specific time period.

5. What is the 

intended number 

of hours of service 

receipt for each 

session of each 

element?

[open-ended] For each element implemented, describes how many hours 

participants were supposed to receive an element during a specific 

time period.

6. What is the actual

number of hours 

of service receipt 

for each session 

of each element 

(on average)?

[open-ended] For each element implemented, describes the number of hours 

that participants actually received an element during a specific time 

period. 

7. What is the 

intended total 

number of hours 

of programming 

for each element?

[open-ended] For each element implemented, describes the total number of 

hours of services that participants were scheduled to receive. 

8. What is the actual

total number of 

hours of 

programming for 

each element (on 

average)?

[open-ended] For each element implemented, describes the total number of 

hours of services that participants actually received, on average, 

during a specific time period. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.2 (continued)

16

Question Response options Additional guidance

IV. Personnel needs

1. How many 

individuals 

deliver the 

intervention 

(overall and by 

intervention 

element)? 

[open-ended] Describes the number of individuals, by job title, who directly 

delivered the services across all locations, both overall and by 

element. For example, if an after-school reading intervention used 

3 national service participants to lead classroom sessions and 9 

classroom teachers to provide one-on-one tutoring across three 

locations, the response would state, “12 personnel overall: 3

participants (classroom sessions) and 9 teachers (one-on-one 

tutoring).”

2. Who delivers the 

services per 

intervention 

element?

[open-ended] For each core element, lists job titles of individuals who directly 

delivered services, including several job titles as necessary. For 

example, if an after-school reading intervention used national 

service participants to lead classroom sessions and classroom 

teachers to provide one-on-one tutoring, the titles of both the 

participants and teachers are listed, along with the service element 

delivered by each group.

3. Does the 

intervention’s 

personnel

structure include 

AmeriCorps

service 

participants? 

Yes, AmeriCorps

participants are 

included in the 

personnel. 

No, these groups are 

not mentioned.

Question 4 is skipped with a “no” response.

4. What role do 

AmeriCorps

participants play?

[open-ended] Describes the role played by AmeriCorps participants. For 

example, “participants provided tutoring services for an afterschool intervention.”

5. What 

qualifications are 

the required or 

preferred as a 

prerequisite for the 

personnel involved 

in delivering the 

intervention?

[open-ended] Describes qualifications for each job title involved in delivering the 

intervention. Qualifications could include experience or expertise in 

delivering the needed services or education/training. For example, 

a nutrition education intervention requires educators to hold a 

degree in nutrition and demonstrate at least two years of teaching 

experience. 

6. Does the 

organization 

intend or plan for a 

different personnel

structure or 

personnel

qualifications or 

training from what 

was originally 

intended or 

planned?

Yes, personnel plans 

differ from the 

personnel as 

implemented.

No, personnel plans 

do not differ from the 

personnel as 

implemented.

Describes how the intended personnel plan, including the job titles 

of those who delivered the intervention, the number of individuals 

involved in service delivery, whether the personnel included 

AmeriCorps national service participants, personnel qualifications, 

and training plans for personnel, differed from the plan that was 

implemented. 

Question 7 is skipped with a “no” response.

7. How does the 

personnel

structure differ 

from what was 

originally intended 

or planned?

[open-ended] Describes the differences between the planned personnel structure 

and the implemented personnel structure. Includes the job titles of 

those who delivered the intervention, the number of individuals 

involved in service delivery, and whether the personnel included 

AmeriCorps service participants. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.2 (continued)

17

Question Response options Additional guidance

VII. Setting

1. What are the 

geographic 

location(s) of the 

intervention, and 

what sites are 

included in 

implementation?

[open-ended] For all locations, describes the geographic region, location, and 

whether the intervention was implemented in a rural or urban 

setting (or both). For example, an intervention implemented in two 

regions of the United States, three cities or areas, and several 

school districts and schools within each district would state, 

"Northeast and Southern regions, including New York, NY 

(Brooklyn): three school districts, 10 schools; Washington, DC: one 

school district, 8 schools; eight rural counties in southwestern 

Virginia (two school districts, 4 schools).” 

2. In what venue(s) 

are the 

intervention core 

elements 

implemented?

[open-ended] The implementation setting is where an intervention physically 

takes place, such as community center, home, nonprofit 

organization, park, school, and residential facilities. Organizations 

may implement core elements in different settings. For example, 

an intervention may have included job training at an American Job 

Center and job coaching on site with employers; both the American 

Job Center and employer settings, along with the job training and 

job coaching activities, are listed.

3. Did the 

evaluation enroll 

participants from a 

subset of 

intervention 

locations? 

Yes, the locations for 

the evaluation were a 

subset of places 

where the intervention 

was implemented. 

No, the locations for 

the evaluation and 

implementation were 

the same.

The evaluation of the intervention may be conducted with 

participants from a subset of the locations in which the intervention 

was implemented. For example, an intervention may have been 

implemented in two cities, but the evaluation included participants 

in only one city.

Question 4 is skipped with a “no” response.

4. What locations 

were included in 

the evaluation?

[open-ended] Describes geographic locations indicating where the evaluation

occurred and the number of locations. 

VI. Definition of completion

1. Is intervention 

participant 

defined?

Yes, participant is 

defined.

No, participant is not 

defined.

Definition of a participant describes the minimum amount of 

services needed to be received in order to be considered a 

“participant” in the intervention. For example, individuals may enroll 

in an intervention and be considered a participant even if they 

never receive any services. Alternatively, participants may be 

defined as only those individuals who enroll and receive at least 

one service through the intervention.

Question 2 is skipped with a “no” response.

2. What is the 

definition of an 

intervention 

participant?

[open-ended] Describes the amount of services an individual must receive in 

order to be considered a “participant.”

3. Is completion of 

the intervention 

defined?

Yes, completion is 

defined.

No, completion is not 

defined.

Definition of completion indicates what participants must do to be 

considered “completers” or “graduates.” For example, participants 

may have to complete a course and obtain a certificate in order to 

complete an intervention.

Question 4 is skipped with a “no” response. 

4. What is the 

definition of 

intervention 

completion?

[open-ended] Describes what participants must do to be considered “completers” 

or “graduates.”
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

18

III.A.3. Well-defined target population 

This subsection addresses the second condition of the scaling readiness framework—a welldefined target population. It includes questions about the target population along three major 

dimensions (Table III.A.3):

1. Intervention as it was designed, including the characteristics of the target population as the 

intervention was originally designed. 

2. Intervention as it was implemented, including the characteristics of the population that was 

eligible to participate in the intervention as it was implemented.

3. Evaluation sample for the intervention, including the participants in the evaluation sample. 

Table III.A.3. Questions about defining the target population

Question Response options Additional guidance

I. Intervention as it was designed

1. What is the target population 

for the intervention as it was 

designed?

[open-ended] Describes the characteristics of the 

population that the intervention was originally 

designed to serve. Includes risk factors, age 

or grade ranges, participants’ locations, 

gender, and so forth. In the case of several 

target populations, describes each 

population. 

II. Intervention as it was implemented

1. Who is eligible to participate 

in the intervention as it was 

implemented by the 

organization?

[open-ended] Describes who is eligible to participate in the 

intervention as implemented. Includes risk 

factors, age or grade ranges, participants’ 

locations, gender, and so forth, along with 

inclusion and exclusion criteria and 

information for several target populations. 

2. Does the intervention as it is 

implemented enroll 

participants primarily from a 

specific racial or ethnic 

group?

Yes, participants are primarily 

from one of the prespecified 

racial or ethnic categories.

No, participants are not 

primarily from a racial or ethnic 

category.

Racial/ethnic groups include African 

American, Hispanic, and other. An even 

distribution of African American, European 

American, and Hispanic participants is 

considered a “no” response. 

3. Does the intervention as it is 

implemented enroll 

participants primarily from a 

certain age group?

Yes, participants are primarily 

from one of the prespecified 

age group(s). 

No, participants are not 

primarily from a certain age 

group(s).

Age categories include 0-5, 6-9, 10-7, 18-24, 

25 55, and 56+. The intervention may be 

designed to serve participants in a specific 

age category, but, during implementation, 

participants from a different age category 

might enroll. Age group categories include all 

ages in that range. For example, if an 

intervention enrolled 4- to 6-year-old children, 

the response would be "0-5 and 6-9." 

4. Does the intervention as it is 

implemented enroll 

participants primarily from 

any of the key populations?

Yes, participants are primarily 

from one of the prespecified 

key populations.

No, participants are not 

primarily from a key 

population.

Key populations include low-income 

households/individuals, developmentally 

disabled individuals, homeless individuals, 

parents/caregivers of young children, 

parolees, and veterans/military families. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.3 (continued)

19

Question Response options Additional guidance

III. Evaluation sample for the intervention

1. Does the evaluation sample 

differ from the target 

population of the intervention 

as it is implemented?

Yes, the evaluation sample 

and target population differ for 

the intervention as 

implemented. 

No, the evaluation sample and 

target population do not differ 

for the intervention as 

implemented.

Differences may arise if the evaluation 

sample was a subset of the population 

served by the intervention. For example, an 

intervention may have targeted 1,000 

students in grades 9 through 12 in Chicago 

and Detroit, but the evaluation may have 

considered the outcomes of only 250 of those 

students in all grades and in both cities. 

Although the evaluation sample might be 

representative of the population it served, it 

does not include all of those served. 

Questions 2 through 5 are skipped with a 

“no” response.

2. How does the evaluation 

sample differ from the target 

population for the 

intervention as it is 

implemented?

[open-ended] Describes how the evaluation sample differs 

from the population intended to receive the 

intervention. 

3. Does the evaluation 

measure outcomes of 

participants primarily from a 

particular racial or ethnic 

group?

Yes, the evaluation measured 

outcomes of participants 

primarily from a prespecified 

racial or ethnic group(s). 

No, the evaluation did not 

measure outcomes of 

participants primarily from a 

racial or ethnic group(s). 

Racial/ethnic groups include African 

American, Hispanic, and other. An even 

distribution of African American, European 

American, and Hispanic participants is 

considered a “no” response.

4. Does the evaluation 

measure outcomes of 

participants primarily from a 

certain age group?

Yes, the evaluation measured 

outcomes of participants from 

a certain age group(s). 

No, the evaluation did not 

measure outcomes along the 

age dimension.

Age categories include 0-5, 6-9, 10-17, 18-

24, 25-55, and 56+. 

5. Does the evaluation 

measure outcomes of 

participants from any of the 

key populations? 

Yes, the evaluation measured 

outcomes of participants from 

a prespecified key 

populations.

No, the evaluation did not 

measure outcomes of 

participants from the key 

populations.

Key populations include low-income 

households/individuals, developmentally 

disabled individuals, homeless individuals, 

parents/caregivers of young children, 

parolees, and veterans/military families. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

20

III.A.4. Implementation supports

This section of the rubric addresses the third condition of the scaling readiness framework— 

implementation supports. It includes questions in six major areas, each of which describes the 

supports that help ensure quality in implementation (Table III.A.4). 

1. Implementation monitoring team, including how the organization ensures fidelity to the 

intervention model (whether an organization delivered an intervention as intended). It also 

includes whether implementation issues arose, and whether the organization made any 

purposeful changes to the implementation supports. 

2. Performance procedures, including performance goals and benchmarks for how personnel 

deliver the intervention. 

3. CQI, including the ongoing, systematic process of identifying, describing, and analyzing 

strengths and challenges during implementation, along with the collection and use of data to 

improve the intervention’s processes. 

4. Preservice and inservice training for the personnel, including initial and ongoing training 

for those who deliver the intervention as well as the identification of those responsible for the 

delivery of training. 

5. Communication systems, including the elements that facilitate high quality communication 

among intervention leaders, personnel, and partners. 

6. Data systems, including efforts to track, measure, and store information about 

implementation and to use the information to help make decisions for monitoring and CQI. 

Table III.A.4. Questions about implementation supports

Question Response options Additional guidance

I. Implementation monitoring team

1. Is an implementation 

monitoring team described?

Yes (full), an implementation 

monitoring team is described, 

with details provided. 

Yes (limited), an 

implementation monitoring 

team is described, but with no 

details provided.

No, an implementation 

monitoring team is not 

mentioned.

A monitoring team ensures that 

implementation takes place as planned. 

Team members may play other roles in the 

organization, such as supervisors, project 

managers, and project directors, or they 

may be other personnel dedicated 

exclusively to ensuring implementation of 

the intervention takes place with fidelity to 

the model. 

Question 2 is skipped with a “no” response.

2. Which personnel make up 

the implementation 

monitoring team? 

[open-ended] Describes the team members, including 

their titles and responsibilities, who are 

monitoring implementation of the 

intervention with fidelity to the model. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.4 (continued)

21

Question Response options Additional guidance

3. Does a process exist for 

monitoring service delivery 

to assess fidelity to the 

intervention as planned?

Yes, steps exist for monitoring

fidelity.

No, steps to monitor fidelity are 

not mentioned.

A process for monitoring fidelity to the 

intervention model ensures that services are 

being delivered as intended. The process 

for monitoring implementation fidelity could 

include collection of service delivery data 

through observations during site visits or 

regular reviews of service data entered into 

a data system. References to processes for 

assessing whether the “intervention is 

delivered as intended” are considered in 

assessing fidelity. Discussions of monitoring 

quality or other facets of implementation are 

not considered.

Question 4 is skipped with a “no” response.

4. How is implementation 

fidelity monitored?

[open-ended] Describes the steps taken to monitor 

implementation with fidelity, including 

identification of the individual (by job title) 

responsible for assessing fidelity, frequency 

of implementation monitoring, processes for 

managing/supervising personnel involved in 

service delivery, and frequency of personnel

management or supervisory meetings.

5. Are issues or challenges 

noted regarding 

implementation fidelity? 

Yes, issues were noted. 

No, issues regarding fidelity 

were not mentioned.

Fidelity issues include variation among 

instructors in the manner in which they 

deliver the curriculum across classes or 

cohorts and unplanned modifications to the 

intervention during the evaluation, such as 

changing an implementation site or 

adjusting the curriculum to meet the target 

population’s unexpected needs. 

Question 6 is skipped with a “no” response.

6. What issues or challenges 

are noted regarding 

implementation fidelity?

[open-ended] Describes any issues or challenges reported 

with implementation of the intervention with 

fidelity.

7. Who was responsible for 

supervising personnel

involved in service delivery? 

[open-ended] Describes, by job title, who supervised the 

personnel involved in service delivery. If 

more than one personnel member 

responsible for supervising the individuals 

involved in service delivery, identifies each 

title and who was supervised. 

8. How were personnel

involved in service delivery

supervised?

[open-ended] Describes the frequency and format of 

supervision for each job title. For example, 

supervision may take place on a weekly, 

monthly, or as needed basis and may be in 

the form of one-on-one meetings or small 

team meetings. Supervision extends to 

check-in meetings, case note reviews, 

administrative data analysis, 

communication, and supports for 

supervisors. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.4 (continued)

22

Question Response options Additional guidance

9. Did any implementation 

supports purposefully

change from an earlier 

version of the intervention 

model before the start of 

implementation?

Yes, the implementation 

supports changed from an 

earlier version of the model 

before the start of 

implementation.

No, the implementation 

supports did not change from 

an earlier version of the model 

before the start of 

implementation.

Organizations may make changes to 

implementation supports before the start of 

implementation. Purposeful changes to 

implementation supports may include 

changes related to the following: the 

personnel structure, personnel training, 

supervision and performance management, 

implementation monitoring, communication 

processes, and data systems. 

Question 10 is skipped with a “no” 

response.

10.What purposeful changes 

were made to the 

intervention’s 

implementation supports?

[open-ended] Describes implementation supports that 

purposefully differ from an earlier design of 

the intervention. Changes may include any 

support related to the following: steps to 

ensure fidelity, the personnel structure, 

personnel training, supervision and 

performance management, monitoring, 

communication, and data systems. 

II. Performance procedures

1. Are performance 

benchmarks established for 

personnel involved in service 

delivery described?

Yes (full), performance 

benchmarks established for 

personnel involved in service 

delivery are described, with 

details provided.

Yes (limited), performance 

benchmarks established for 

personnel involved in service 

delivery are described, but with 

no details provided.

No, performance benchmarks 

established for personnel

involved in service delivery 

were not mentioned.

Performance benchmarks are specific goals 

that personnel involved in service delivery 

must meet as related to delivery of the 

intervention, such as the number of 

participants contacted per personnel

member or the number of participants 

served per personnel member.

Question 2 is skipped with a “no” response. 

2. What were the performance 

benchmarks established for 

personnel involved in 

service delivery?

[open-ended] Describes the performance benchmarks 

established for personnel involved in service 

delivery, such as the number of participants 

contacted per personnel member or the 

number of participants served per personnel

member. If different performance goals are 

set for specific job titles, goals are listed by 

job title; for example, “case managers are 

required to contact 12 participants per day.” 

3. Are procedures for 

monitoring achievement of 

performance benchmarks 

described?

Yes, procedures for monitoring 

achievement of performance 

benchmarks are described, with 

details provided.

Yes (limited), procedures for 

monitoring achievement of 

performance benchmarks are 

described, but with no details 

provided.

No, procedures for monitoring 

achievement of performance 

benchmarks were not 

mentioned.

Procedures for monitoring achievement of 

performance benchmarks include how 

benchmarks are measured and collected, 

who reviews progress toward benchmarks, 

and the frequency of performance 

monitoring.

Question 4 is skipped with a “no” response.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.4 (continued)

23

Question Response options Additional guidance

4. What were the performance 

benchmarks monitoring 

procedures?

[open-ended] Describes the procedures for monitoring 

achievement of performance benchmarks 

including how benchmarks are measured 

and collected, who reviews progress toward 

goals, and how frequently benchmarks for 

goals are captured and reviewed. If different 

performance monitoring procedures are set 

for specific job titles or roles, monitoring 

procedures are listed by job title.

III. Continuous quality improvement (CQI)

1. Are CQI processes for the 

intervention described?

Yes (full), CQI processes are 

described, with details 

provided.

Yes (limited), CQI processes 

are described, but with no 

details provided.

No, CQI processes are not 

mentioned.

Continuous quality improvement processes 

refer to procedures for continuously 

assessing the quality of the intervention as 

implemented to improve implementation 

practice. CQI includes regularly testing the 

intervention and making adjustments as 

needed, with continual retesting of the 

modifications to ensure that the intervention 

is succeeding as planned. 

Question 2 is skipped with a “no” response.

2. What were the CQI

processes for the 

intervention?

[open-ended] Describes the process for testing the 

intervention, including how frequently CQI 

data is collected. 

3. Were data analyzed to 

support CQI for the 

intervention?

Yes (full), data were analyzed 

and used for CQI, with details 

provided.

Yes (limited), data were 

analyzed and used for CQI, but 

with no details provided.

No, the analysis and use of 

data for CQI was not 

mentioned.

CQI requires the collection of data on 

performance benchmarks and use of the 

data to provide ongoing personnel

development training and the delivery of 

technical assistance to partners. Data 

analysis to support CQI includes examining 

personnel achievement of performance 

benchmarks. 

Question 4 is skipped with a “no” response.

4. How were data analyzed to 

support CQI for the 

intervention?

[open-ended] Describes how performance data were used 

to continuously improve intervention 

implementation.

IV. Preservice and inservice training for personnel

1. Is the initial training to 

deliver the intervention 

received by personnel

described?

Yes (full), initial personnel

training to deliver the 

intervention is described, with 

details provided.

Yes (limited), initial personnel

training to deliver the 

intervention is described, but 

with no details provided.

No, initial personnel training to 

deliver the intervention is not 

mentioned.

Training for service delivery given to 

personnel before implementation begins 

includes training on intervention content 

(such as a training on a curriculum used in 

the intervention), methods for service 

delivery (such as motivational interviewing), 

and client processing procedures (such as 

intake procedures).

Question 2 and 3 are skipped with a “no” 

response.

2. What initial training did

personnel receive to deliver 

the intervention?

[open-ended] Describes, for each job title, initial personnel

training for delivery of the intervention. 

Specifies when training occurred and 

training content and format (such as 

classroom, online). For example, a "summer 

training institute" for teachers occurs in 

person (through classroom sessions and 

practice teaching) in the summer before 

teachers begin teaching. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.4 (continued)

24

Question Response options Additional guidance

3. Who delivered the initial 

training?

[open-ended] Describes, by job title, who delivered the 

initial training (for example, supervisors, 

personnel involved in service delivery, and 

partners). Separately identifies people with 

different job titles who delivered different 

types of training. For example, Teach For 

America personnel might deliver a "summer 

training institute" for teachers. 

4. Is the ongoing training to 

deliver the intervention 

received by personnel

described?

Yes (full), ongoing personnel

training to deliver the 

intervention is described, with 

details provided.

Yes (limited), ongoing 

personnel training to deliver the 

intervention is described, but 

with no details provided.

No, ongoing personnel training 

to deliver the intervention is not 

mentioned.

Training for personnel involved in service 

delivery during implementation and 

throughout the service delivery period 

includes refresher trainings on intervention 

content, methods for service, and client 

processing procedures (such as intake 

procedures).

Question 5 and 6 are skipped with a “no” 

response.

5. What is the ongoing training 

needed to deliver the 

intervention?

[open-ended] Describes, by job title, the ongoing training 

that helps service providers deliver the 

intervention. Details include when and how 

often training occurred, content (for 

example, curriculum content, use of data 

systems), and format (for example, 

classroom, online). 

6. Who delivered the ongoing 

training?

[open-ended] Describes, by job title, who delivered the 

ongoing training (for example, supervisors, 

service providers, and partners). 

V. Communication systems

1. Is a communication system 

to support coordination 

among personnel and 

partners the described?

Yes (full), a communication 

system is described, with 

details provided.

Yes (limited), a communication 

system is described, but with no 

details provided.

No, a communication system is 

not mentioned.

Communication systems support 

coordination among personnel and partners 

and specify the frequency and the expected 

duration of communication, the parties 

responsible for communication, and the 

expected duration of communication. 

Systems may vary in the extent to which 

they specify and standardize 

communication. For example, a detailed 

plan may state that, for the first six months 

of the intervention, service providers must 

meet as a group with their supervisor for 

one hour each week to discuss topics 

related to recruitment and engagement of 

participants; for the next six months, 

meetings must take place monthly in the 

form of hour-long sessions to discuss 

service provision. 

Question 2 is skipped with a “no” response.

2. What was the 

communication system for 

supporting coordination 

personnel and partners? 

[open-ended] Describes the communication system, 

including the frequency and duration of the 

communication between personnel and with 

any partners, and which individuals are 

included in the communication system.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.4 (continued)

25

Question Response options Additional guidance

3. Is a process for ensuring the 

communication system 

functioned as intended 

described?

Yes (full), a process for 

ensuring the communication 

system functioned is described, 

with details provided.

Yes (limited), a process for 

ensuring the communication 

system functioned is described, 

but with no details provided.

No, a process for ensuring the 

communication system 

functioned is not mentioned

Processes to ensure a communication 

system functions as intended include 

obtaining feedback from personnel

regarding the communication system and 

tracking the communication that takes 

place. Details include how feedback was 

obtained regarding the functioning of the 

communication system. For example, a 

communication system may have included 

logs of how often personnel met together\ or 

used regular check-ins with personnel

eliciting feedback on how easily they feel 

they are able to get in contact with each 

other or key intervention partners.

Question 4 is skipped with a “no” response.

4. What was the process for 

ensuring the communication 

system functioned as 

intended?

[open-ended] Describes the process for ensuring the 

communication system functioned as 

intended.

VI. Data systems

1. Is a data system to support 

data collection, analysis, and 

decision making described?

Yes (full), a data system is in 

place, with details provided.

Yes (limited), a data system is 

mentioned, but with no details 

provided.

No, a data system is not 

mentioned.

Data systems (for example, a management 

information system or a standardized Excel 

spreadsheet) capture enrollment and 

participation data and generate reports on 

participation trends. Such systems support 

data collection, analysis, and decision 

making. A detailed description specifies the 

information captured by the system, the 

organization’s capacity to analyze recorded 

data, and how the organization uses the 

data to support its decision-making 

processes.

Question 2 is skipped with a “no” response.

2. What was the data system 

developed to support data 

collection, analysis, and 

decision making?

[open-ended] Describes the data system developed to 

support data collection, analysis, and 

decision making.

3. Is a process for ensuring 

data quality described?

Yes (full), a process for 

ensuring data quality is 

described, with details 

provided.

Yes (limited), a process for 

ensuring data quality is 

described, but with no details 

provided.

No, a process for ensuring data 

quality is not mentioned.

Processes for ensuring that the data 

collected are of high quality, such as 

checking for the completeness, accuracy, 

consistency, and timeliness of the data. For 

example, there is a plan to check 

participation data entered into a data system 

with paper records of attendance or case 

note file reviews.

Question 4 is skipped with a “no” response.

4. What was the process for 

ensuring the quality of the 

data collected through the 

data system?

[open-ended] Describes the process for ensuring that the 

data collected through the data system was 

of high quality.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

26

III.A.5. Enabling context 

This subsection addresses the fourth condition of the scaling readiness framework—the enabling 

context. It includes three aspects of the context in which organizations provide services (Table 

III.A.5): 

1. Support for the intervention, including support from organizational leadership and partner 

agencies for implementing and evaluating the intervention. 

2. Innovation and learning, including earlier activities that the organization might have 

undertaken in an effort to introduce new practices or improve the intervention and the 

organization’s ability to understand the reason for any successes. 

3. Improvement in response to challenges, including the identification of any challenges that 

organizations have encountered during implementation of the intervention and the solutions 

to those challenges. 

Table III.A.5. Questions about the enabling context 

Question Response options Additional guidance

I. Leaders, key stakeholders, and partners support for the intervention

1. Is support from 

organizational leaders 

for the intervention 

described?

Yes (full), support from 

organizational leaders for the 

intervention is described, with 

details provided.

Yes (limited), support from 

organizational leaders for the 

intervention is described, but with no 

details provided.

No, support from organizational 

leaders for the intervention is not 

mentioned. 

Describes organizational leaders and their 

commitment to the intervention. For example, 

organization leaders are described as 

engaging in planning activities or making 

implementation of the intervention a priority 

for the organization. 

Question 2 is skipped with a “no” response.

2. How was support from 

organizational leaders 

for the intervention 

demonstrated?

[open-ended] Describes how support was demonstrated by 

organizational leaders.

3. Is support from 

stakeholders and/or 

partners for the 

intervention 

described?

Yes (full), support from stakeholders 

and/or partners for the intervention 

is described, with details provided.

Yes (limited), support from 

stakeholders and/or partners for the 

intervention is described, but with no 

details provided.

No, support from stakeholders 

and/or partners for the intervention 

was not mentioned.

Describes stakeholders’ and/or partners’ 

commitment to the intervention. Stakeholders 

and partners include service delivery partners 

or leadership at implementation sites. 

Support may be demonstrated by securing 

space for service delivery at local 

implementation sites or removing 

responsibility for non-intervention related 

workload for personnel involved in service 

delivery. 

Question 4 is skipped with a “no” response.

4. How was support from 

organization 

stakeholders and/or 

partners for the 

intervention 

demonstrated?

[open-ended] Describes how support was demonstrated by 

stakeholders and/or partners.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.5 (continued)

27

Question Response options Additional guidance

II. Innovation and learning

1. Is there any description 

of the organization’s 

earlier efforts to be 

innovative?

Yes (full), efforts to be innovative 

are described, with details provided.

Yes (limited), efforts to be innovative 

are described, but with no details 

provided.

No, efforts to be innovative were not 

mentioned.

Describes organizational efforts to identify, 

develop, and implement new ways of 

meeting community needs. For example, 

organizational innovation could include 

offering new products or services or 

identifying novel ways of delivering regularly 

available services. 

Question 2 is skipped with a “no” response.

2. What were the 

innovative efforts 

described?

[open-ended] Describes organization’s efforts to be 

innovative.

3. Is there a description 

of the organization’s 

efforts to improve its 

interventions?

Yes (full), efforts to improve its 

interventions are described, with 

details provided.

Yes (limited), efforts to improve its 

interventions are described, but with 

no details provided.

No, efforts to improve its 

interventions are not mentioned.

Describes the way in which organizations 

have improved interventions. Improvement 

may be demonstrated by a description of how 

the organization advanced its practices and 

enhanced the current service array. For 

instance, the evaluated intervention may 

have improved upon a previous version of 

the intervention by making changes to its 

recruitment strategies. 

Question 4 is skipped with a “no” response. 

4. What were the 

improvement efforts 

described?

[open-ended] Describes organization’s efforts to improve 

interventions.

5. Is there any discussion 

of successes regarding 

implementation of the 

intervention?

Yes (full), implementation 

successes are discussed, with 

details provided.

Yes (limited), implementation 

successes are discussed, but with 

no details provided.

No, implementation successes were 

not mentioned.

Describes successes related to implementing 

intervention core elements, recruiting or 

serving the target population(s), supporting 

implementation, and creating an enabling 

context. 

Question 6 is skipped with a “no” response. 

6. What were the 

implementation 

successes of the 

intervention and the 

reasons for those 

successes?

[open-ended] Describes the implementation successes and 

the reasons for those successes. 

III. Improvement in response to challenges

1. Is there any discussion 

of challenges 

regarding 

implementation of the 

intervention?

Yes, challenges were discussed.

No, challenges were not discussed.

Challenges could be related to intervention 

components, target population(s), 

implementation supports, and the enabling 

context. 

Question 2 is skipped with a “no” response. 

2. What were the 

challenges regarding 

implementation of the 

intervention and 

solutions to those 

challenges?

[open-ended] Includes the implementation challenges and 

the solutions to those challenges, if any. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

28

III.A.6. Implementation infrastructure

This section of the rubric addresses the fifth condition of the scaling readiness framework— 

implementation infrastructure. It includes questions in four major areas, each of which describes 

the organization’s infrastructure in support of implementation of the intervention (Table III.A.6). 

1. Financial resources, including how the organization demonstrates that it or its partners 

successfully supported implementation of the intervention with dedicated financial resources. 

2. Sufficient personnel, including whether the organization dedicated the personnel needed 

either to implement or scale the intervention. 

3. Materials, including the standardization of materials and tailoring of materials to meet 

participants’ needs. 

4. Physical space, including the availability of space necessary for service delivery. 

5. Human resource system, including a human resource system that is critical in typically 

overseeing three highly important, recognized implementation drivers—hiring, training, and 

ongoing supervision. 

Table III.A.6. Questions about implementation infrastructure

Question Response options Additional guidance

I. Financial resources

1. Does the organization provide 

funding for implementation of 

the intervention?

Yes (full), adequate funding is 

in place, with details provided.

Yes (limited), adequate funding 

is in place, but with no details 

provided. 

No, funding is not mentioned.

Describes the amount and source of funds 

and, if applicable, partners’ funding for 

implementation of the intervention.

Question 2 is skipped with a “no” response. 

2. What funding was provided by 

the organization for 

implementation of the 

intervention?

[open-ended] Describes the funding provided by the 

organization for implementation of the 

intervention.

II. Sufficient personnel

1. Does the organization provide 

personnel to implement the 

intervention (including 

dedicated supervisors and 

service providers)?

Yes (full), the organization 

provides dedicated personnel, 

with details provided.

Yes (limited), the organization 

provides dedicated personnel, 

but with no details provided.

No, dedicated personnel are

not mentioned.

Describes the number of personnel needed 

for implementation, whether personnel

worked exclusively on intervention 

implementation or also had nonintervention 

responsibilities, and, if applicable, partners’ 

role in providing personnel for the 

intervention.

Question 2 is skipped with a “no” response.

2. How many personnel were 

provided by the organization to 

implement the intervention?

[open-ended] Describe the number of personnel needed 

for implementation.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.A.6 (continued)

29

Question Response options Additional guidance

III. Materials 

1. Is there a description of 

materials needed for the 

intervention?

Yes (full), intervention materials 

are described, with details 

provided.

Yes (limited), intervention 

materials are mentioned, but 

with no details provided.

No, needed intervention 

materials are not mentioned.

Describes the materials needed to

implement the intervention, such as 

workbooks, culturally sensitive documents, 

or other handouts.

Question 2 is skipped with a “no” response. 

2. What were the materials 

needed for the intervention?

[open-ended] Describes the intervention materials needed 

for implementation.

IV. Physical space

1. Is there a description of the 

physical space needed for the 

intervention?

Yes (full), the physical space is 

described, with details 

provided.

Yes (limited), the physical 

space is described, but with no 

details provided.

No, the physical space is not 

mentioned.

Describes the physical space needed for 

implementation, such as the size or number 

of classrooms needed to fit the target 

number of participants or the availability of 

private meeting spaces for one-on-one

service delivery.

Question 2 is skipped with a “no” response.

2. What physical space was 

needed for implementation of 

the intervention?

[open-ended] Describes the physical space needed for 

implementation.

V. Human resource system

1. Is a human resource system in 

place to hire, supervise, and 

develop the personnel? 

Yes (full), a human resource 

system is in place, with details 

provided.

Yes (limited), a human resource 

system is in place, but with no 

details provided.

No, a human resource system 

is not mentioned.

A human resource system supports 

implementation through processes and 

procedures for hiring personnel who meet 

qualifications for implementation with 

fidelity; for defining a supervisory structure 

(including the identification of who reports to 

whom, the frequency of supervisory 

meetings, mechanisms for reporting on 

personnel progress); and for providing 

personnel development (including planned 

training for personnel and opportunities for 

performance monitoring and improvement). 

A clearly defined human resource system 

specifies how these supports are routinized 

and whether procedural guidelines are 

developed to standardize the supports.

Question 2 is skipped with a “no” response.

2. What was the human resource 

system put in place for the 

intervention?

[open-ended] Describes the human resource system put 

in place for the intervention.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

30

III.B. Scaling Readiness Section

The scaling readiness section of the rubric is used to assess an intervention’s readiness to be 

scaled and the organization’s ability to support that scaling as described in the organization’s 

narrative application for funding or in the organization’s scaling plan. This section is used 

primarily to assess the degree to which the organization specifies details critical for assessing 

that the intervention and the organization will be able to successfully implement the intervention 

and to what degree the intervention proposed for scaling differs from the intervention that was 

evaluated (and described using the previous section of the rubric that was applied to the 

intervention’s evaluation report). The section includes six subsections. One subsection collects 

background information about the approach to scaling and the context in which implementation 

will occur, and the next five subsections collect information on each of the five conditions 

needed for scaling. 

III.B.1. Scaling approach and its context 

Table III.B.1 provides the first subsection of the rubric, which includes two types of questions 

about the organization’s approach to scaling and the context in which it occurs: 

1. Scaling approach planned for the intervention. Research from implementation science

includes three forms of scaling. (1) Expansion extends the intervention to more people in the 

same target population in the same location. Successful expansion requires the intervention 

and the organization to serve a larger number of participants with the same service quality 

and in a manner demonstrating fidelity to the model’s design. (2) Replication extends the 

intervention to the same target population, but in a new location. Successful replication 

requires the intervention and the organization to maintain service quality and fidelity to the 

intervention in the new location. (3) Adaptation extends the intervention to a new target 

population. Successful adaption requires the organization to change the intervention in a way 

that maintains service quality. AmeriCorps also funded some AmeriCorps State and National 

grantees to sustain or deepen the intervention services instead of scaling the intervention. 

Some grantees received funds to sustain services, which means that the intervention will 

continue serving the same target population in the current location without any purposeful 

changes to it. Some grantees received funds to deepen services, which means that the

intervention will serve the same target population in the current location, but with enhanced 

services (for example, more hours of job coaching). 

2. Local context and demand associated with the geographic area where the intervention is 

proposed for scaling. Local contextual factors include any regional, state, and local features 

that may affect implementation of the intervention, and local demand factors include 

evidence of demand for the intervention in the local area.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

31

Table III.B.1. Questions about scaling approach and local context

Question Response options Additional guidance

I. Scaling approach

1. What is the organization’s 

proposed intervention?

[open-ended] Describes the proposed intervention, 

including its core components and 

activities and plans for implementation. 

2. How does the organization 

self-categorize its scaling 

approach? 

Expansion

Replication

Adaption

Sustaining services

Deepening services

Not described

Describes the self-categorization of the 

scaling approach.

3. Based on the description of 

the proposed intervention, 

what is the scaling 

approach? 

Expansion

Replication

Adaption

Sustaining services

Deepening services

Describes the scaling approach as 

defined by implementation science.

4. Will the intervention 

proposed for scaling be 

implemented by the 

organization that developed 

it?

Yes, the intervention proposed for 

scaling will be implemented by the 

organization developing it.

No, the intervention proposed for 

scaling was developed by a 

different organization.

Describes whether the scaled 

intervention was developed by the 

implementing organization or by another 

entity.

II. Local context and demand

1. Does the organization 

provide evidence of demand 

in the local area for the 

intervention that is being 

scaled? 

Yes (full), evidence exists for local 

demand for the intervention being 

scaled, with details provided.

Yes (limited), demand exists for the 

intervention being scaled, but with

no details provided.

No, local demand is not mentioned 

or does not exist.

Means of illustrating local demand 

include the results of a needs 

assessment, a landscape analysis 

specifying the scope of the intervention, 

or informal data collection via attendance 

at community meetings or conversations 

with key stakeholders. The results of 

such efforts must motivate the proposed 

scaling approach. 

2. Does the organization 

provide evidence that the 

local, regional, or state 

context will be conducive to 

scaling the intervention?

Yes (full), the context will be 

conducive to scaling, with details 

provided. 

Yes (limited), the context will be 

conducive to scaling, but with no 

details provided.

No, the context is not mentioned or 

not conducive to scaling.

Evidence exists that key community 

partners will value the intervention and 

that stakeholders at the local, regional, 

or state level will not impede 

implementation. Such evidence might 

include memorandum of understanding, 

grants, contracts or more formal 

arrangements, as well as tacit

relationships, such as board 

memberships.

III.B.2. Well-specified intervention

This subsection addresses the first condition of the scaling readiness framework—the wellspecified intervention—as described in the grantee’s plan for scaling. It includes overall 

questions on the core elements of the intervention proposed for scaling and their dimensions for 

service delivery, as well as the definition of participants completing the intervention (subsection 

III.A.2 describes these dimensions). The subsection asks questions about five dimensions of 

intervention specification (Table III.B.2): (1) core elements (services, the intervention’s content 

area, and whether a standardized intervention model is used), (2) mode of service delivery, (3) 

intensity, (4) personnel needs, (5) setting, and (6) definition of completion. For each, questions 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

32

(1) categorize the specificity of the proposed intervention elements, (2) identify differences 

between the proposed intervention element and the evaluated intervention element (as described 

in the intervention evaluation reports and captured in subsection III.A.2 of the rubric), and, if 

applicable, (3) report the rationale for differences between the proposed intervention element and 

evaluated intervention element. If the organization proposes to expand to a new population or 

replicate the intervention, it may not make any changes to the intervention itself. 

Table III.B.2. Questions about specifying the intervention proposed for scaling

Question Response options Additional guidance

I. Core elements 

1. Are the core elements 

of the intervention well 

specified?

Yes (full), core elements are well 

specified, with details provided.

Yes (limited), core elements are 

specified, but with no details provided.

No, core elements are not mentioned.

Core elements are the set of activities or 

curricula that result in participants’ 

receipt of services, such as a description 

of services, the content area of the 

intervention, the existence of a 

standardized intervention model, the 

services or curriculum comprising the 

intervention, and the content area of the 

intervention to be delivered. A 

standardized intervention is considered 

to have a clearly specified intervention. 

2. Do the proposed core 

elements differ from 

those in the 

evaluation? 

Yes (full), proposed core elements differ 

from those evaluated, with details of the 

differences provided.

Yes (limited), proposed core elements 

differ from those evaluated, but with no 

details provided.

Do not know; proposed core elements 

are not defined.

No, proposed core elements for the 

evaluation and intervention are the 

same. 

The proposed core elements of the 

intervention for scaling may differ from 

the core elements of the intervention that 

were evaluated. Changes to the 

intervention could include the addition of 

services, the modification of existing 

services, the discontinuation of some 

services, the addition of content, or the 

use of a new curriculum, all described in 

sufficient detail to allow an external 

audience to replicate them.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the core elements? 

Yes, a rationale or support for the 

change(s) is provided.

No, a rationale or support is not 

provided. 

Organizations may provide a reason for 

differences between the proposed 

intervention core elements and the 

evaluated core elements. Support might 

refer, for example, to research showing 

that a proposed change has positive 

impacts on outcomes.

II. Mode of service delivery

1. Is the service delivery 

model well-specified? 

Yes (full), the service delivery model is 

well-specified, with details provided.

Yes (limited), the service delivery model 

is specified, but with no details provided. 

No, the service delivery model is not 

mentioned.

The service delivery model specifies how 

services are delivered, who delivers 

services, and the mode through which 

services are delivered. For example, 

services may be delivered in groups or 

one-on-one meetings, by peer educators 

or teams of facilitators, and the through 

the use of electronic communication or 

in-person meetings. A standardized 

intervention clearly specifies the service 

delivery model. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.B.2 (continued)

33

Question Response options Additional guidance

2. Does the proposed 

service delivery model 

differ from the service 

delivery model 

implemented during 

the evaluation? 

Yes (full), the proposed service delivery 

model differs from the intervention 

evaluated, with details provided.

Yes (limited), the proposed service 

delivery model differs from the 

intervention evaluated, but with no 

details provided.

Do not know; the service delivery model 

is not defined.

No, the proposed and evaluation service 

delivery models are the same. 

The proposed service delivery model for 

the intervention for scaling may differ 

from the service delivery model of the 

intervention that was evaluated. 

Changes to the service delivery model 

could include the type of delivery (such 

as face-to-face meetings or electronic 

communications) and the nature of 

delivery (such as in a group or one-onone meetings). 

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the service delivery 

model? 

Yes, a rationale or support for the 

change(s) is provided.

No, a rationale or support for the 

proposed change(s) is not provided.

Organizations may provide a reason for 

differences between the proposed 

interventions’ service delivery mode and 

the evaluated service delivery mode. 

Support could include research showing 

that a proposed change has positive 

impacts on outcomes.

III. Intensity

1. Is the intensity of the 

intervention wellspecified?

Yes (full), the intensity is well-specified, 

with details provided.

Yes (limited), the intensity is specified, 

but with no details provided.

No, the intensity is not mentioned.

Describes the total number of hours of 

intervention programming participants 

will receive. This is based on how long 

an intervention will last (duration) and 

the amount of services participants will 

receive (dosage), including the 

frequency of intervention interaction and 

hours of service. 

2. Does the proposed 

intensity of the scaled 

intervention differ from 

the intensity of the 

intervention 

evaluated? 

Yes (full), the proposed intensity differs 

from the intervention evaluated, with 

details of the differences provided.

Yes (limited), the proposed intensity 

differs from the intervention evaluated, 

but with no details provided.

Do not know; the intensity is not defined.

No, the proposed and evaluated intensity 

is the same.

The proposed intensity of the 

intervention for scaling may differ from 

the intensity of the intervention that were 

evaluated. Differences may include 

changes to how long an intervention

lasts, how many times a participant and 

a service provider meet, and how many 

hours of service content a participant 

receives.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the intensity of the 

intervention? 

Yes, a rationale for the change(s) is 

included, with details provided.

No, a rationale for the change(s) to 

dosage or duration is not mentioned.

Organizations may provide a reason for 

differences between the proposed 

interventions’ intensity and the evaluated 

intensity. Support might include research 

showing that a proposed change has 

positive impacts on outcomes.

IV. Personnel needs

1. Are the qualifications 

clearly specified for the 

personnel involved in 

delivering the 

intervention?

Yes (full), personnel qualifications are 

clearly specified, with details provided.

Yes (limited), personnel qualifications 

are specified, but with no details 

provided.

No, personnel qualifications are not 

mentioned.

Clearly specified qualifications for each 

job title involved in delivering services for 

the scaled intervention. Qualifications 

could include required training or 

education and experience levels of the 

personnel delivering the intervention 

services. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.B.2 (continued)

34

Question Response options Additional guidance

2. Do the proposed 

personnel

qualifications after 

scaling differ from the 

qualifications of 

personnel delivering 

services as part of the 

evaluation? 

Yes (full), the proposed personnel

qualifications differ from the intervention 

evaluated, with details of the differences 

provided.

Yes (limited), the proposed personnel

qualifications differ, but with no details of 

the differences provided.

Do not know; the personnel

qualifications are not defined.

No, the proposed and evaluation 

personnel are the same.

The proposed personnel of the 

intervention for scaling may differ from 

the personnel of the intervention that 

were evaluated. Differences in personnel

qualifications may include education, 

experience, or various levels of training.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the personnel

qualifications? 

Yes, a rationale or support exists for the 

personnel qualifications change(s).

No, a rationale for the proposed 

change(s) is not mentioned.

Organizations may provide a reason for 

differences between the proposed 

interventions’ personnel and the 

personnel of the evaluated intervention. 

A rationale could include research 

showing that a proposed change has 

positive impacts on outcomes. 

V. Setting 

1. Is the implementation 

setting clearly 

specified?

Yes (full), the implementation setting is 

clearly specified, with details provided.

Yes (limited), the implementation setting 

is specified, but with no details provided.

No, the implementation setting is not 

mentioned.

Implementation setting includes the 

geographic location of implementation 

and where an intervention takes place. 

The description of the implementation 

setting may include the city and state; 

whether the intervention is being 

implemented in a school, communitybased organization, or workforce center; 

and the number of implementation 

settings. The number of implementation 

sites is not considered a change unless 

the new sites are located in a new city, 

state, or type of setting (such as a 

different implementing agency).

2. Does the proposed 

implementation setting 

differ from the 

evaluation’s 

implementation 

setting?

Yes (full), the proposed setting differs 

from the evaluation setting, with details 

provided.

Yes (limited), the proposed setting differs 

from the evaluation setting, but with no 

details provided.

Do not know; the setting is not 

described.

No, the proposed and evaluation setting 

are the same.

The proposed implementation setting of 

the intervention for scaling may differ 

from the implementation setting of the 

intervention that was evaluated. 

Differences in the implementation setting 

might include delivering services in a 

new city, state, region, or urban/rural 

area as well as switching from a 

community-based to a school-based 

intervention. 

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the implementation 

setting?

Yes, a rationale exists for the change(s). 

No, a rationale is not provided for the 

proposed change(s).

Organizations may provide a reason for 

differences between the implementation 

setting of the proposed intervention and 

the setting of the evaluated intervention. 

A rationale could include research 

showing that a proposed change has 

positive impacts on outcomes.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.B.2 (continued)

35

Question Response options Additional guidance

VI. Definition of completion

1. Are the criteria for 

completion wellspecified?

Yes (full), the criteria for completion are 

well-specified, with details provided.

Yes (limited), the criteria for completion 

are specified, but with no details 

provided.

No, the criteria for completion are not 

mentioned. 

Criteria for completion indicates what 

participants must do to be considered 

“completers” or “graduates” for the 

intervention for scaling. To sufficiently 

specify the completion criteria, there 

must also be a definition of a 

“participant,” including the minimum 

amount of services needed to be 

received in order to be considered a 

“participant.” For example, to be 

considered a participant, a youth must 

attend at least 1 case manager meeting; 

to be considered a completer or 

graduate, participants must attend 10 

case manager meetings.

2. Do the criteria for 

completion differ from 

the criteria described in 

the evaluated 

intervention?

Yes (full), the proposed completion 

criteria differ from the criteria for the 

intervention evaluated, with details 

provided.

Yes (limited), the proposed completion 

criteria differ from the criteria for the 

intervention evaluated, but no details are 

provided.

Do not know; the completion criteria are 

not described.

No, the proposed and evaluated criteria 

for completion are the same.

The completion criteria of the 

intervention for scaling may differ from 

the completion criteria of the evaluated 

intervention. Changes to the criteria for 

completion might include requiring a new 

minimum amount of services to be 

attended to be considered a participant 

or a change in the number of services 

received to be considered a graduate of 

the intervention.

Question 3 is skipped with a “don’t know” 

or “no” response.

3. Does a rationale or 

support exist for the 

proposed change(s) to 

the participant 

completion definition?

Yes, a rationale or support for the 

change(s) is provided.

No, a rationale or support for the 

proposed change(s) is not provided.

Organizations may provide a reason for 

differences between the completion 

criteria of the proposed intervention and 

the completion criteria of the evaluated 

intervention. A rationale could include 

research showing that a proposed 

change has positive impacts on 

outcomes.

III.B.3. Well-defined target population 

This subsection of the rubric addresses the second condition of the scaling readiness 

framework—well-defined target population. It includes questions discussed in subsection III.A.3 

about the target population proposed for scaling (Table III.B.3) and describes differences 

between the proposed target population after scaling and the population studied in the evaluation. 

If the organization proposes to expand the population size or replicate the intervention, it may 

not make any changes to the target population. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

36

Table III.B.3. Questions about the target population proposed for scaling

Question Response options Additional guidance

1. Is the target population 

well-specified?

Yes (full), the target population is 

clearly specified, with details provided.

Yes (limited), the target population is 

specified, but with no details provided.

No, the target population is not 

mentioned.

Describes who is eligible to participate 

in the intervention, including inclusion 

and exclusion criteria. For example, a 

proposed intervention to serve 

unemployed adults should include 

examples of eligible age ranges (for 

example, 18-64), length of 

unemployment, and any other 

inclusion or exclusion criteria such as 

no felony conviction.

2. Does the proposed target 

population differ from the 

evaluation sample? 

Yes (full), the proposed and evaluation 

populations differ, with details of the 

differences provided.

Yes (limited), the proposed and 

evaluation populations differ, but with 

no details of the differences provided.

Do not know; the populations are not 

defined.

No, the proposed and evaluation 

populations are the same. 

Changes to the target population 

include age requirements (for 

example, expanding the eligible age 

range from 11- to 13-year-old children 

to 10- to 14-year-old children), 

personal characteristics (for example, 

risk factors or education levels), or 

eligibility (for example, veterans or 

youth). 

Question 3 is skipped with a “don’t 

know” or “no” response. 

3. Does a rationale or support 

exist for the proposed 

change(s) to the target 

population?

Yes, a rationale exists for the 

change(s), with details provided.

No, no rationale for the change(s) is 

mentioned.

A rationale could include research 

showing that a proposed change has 

had positive impacts on outcomes in 

other studies or that modifications 

could be based on the 

recommendations. 

III.B.4. Implementation supports

This section of the rubric addresses the third condition of the scaling readiness framework—

implementation supports. It includes questions about the implementation supports discussed in 

subsection III.A.4 that are available for the scaled intervention (Table III.B.4): (1) 

implementation monitoring team, (2) performance procedures, (3) CQI, (4) preservice and 

inservice training for the personnel, (5) communication system, and (6) data system. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

37

Table III.B.4. Questions about implementation supports for the intervention proposed for 

scaling

Question Response options Additional guidance

I. Implementation monitoring team

1. Is a team proposed to 

monitor implementation of 

the scaled intervention? 

Yes (full), a team is proposed, with 

details provided.

Yes (limited), a team is described, 

but with no details provided,

No, a team is not mentioned. 

A monitoring team ensures that 

implementation takes place as planned. 

Team members may play other roles in the 

organization (such as supervisors, project 

managers, and project directors) or they 

may be other personnel members 

dedicated exclusively to ensuring that the 

intervention is implemented with fidelity to 

the model. Details include specific 

information about members of the 

implementation team. 

2. Does a process for 

monitoring fidelity to the 

intervention model exist 

for the scaled 

intervention?

Yes (full), a process for monitoring 

fidelity to the intervention model is 

described, with details provided.

Yes (limited), a process for 

monitoring fidelity to the 

intervention is described, but with 

no details provided.

No, a process for monitoring fidelity 

is not mentioned

A process for monitoring fidelity to the 

intervention model ensures that services 

are being delivered as intended. The 

process for monitoring implementation 

fidelity could include collecting service 

delivery data through observations during 

site visits or regular reviews of service data 

entered into a data system. Details include 

identification of the individual (by job title) 

responsible for assessing fidelity, 

frequency of implementation monitoring, 

processes for managing/supervising 

personnel members, and frequency of 

personnel management or supervisory 

meetings.

II. Performance procedures

1. Do performance 

benchmarks exist for 

personnel involved in 

service delivery of the 

scaled intervention?

Yes (full), performance 

benchmarks are described, with 

details provided.

Yes (limited), performance 

benchmarks are described, but 

with no details provided.

No, performance benchmarks are 

not mentioned.

Performance benchmarks are specific 

goals that personnel must meet as related 

to delivery of the scaled intervention, such 

as such as the number of participants 

contacted per personnel member or the 

number of participants served per 

personnel member.

2. Do procedures for 

monitoring achievement of 

performance benchmarks 

exist for the scaled 

intervention?

Yes (full), procedures for 

monitoring achievement of 

performance benchmarks are 

described, with details provided.

Yes (limited), procedures for 

monitoring achievement of 

performance benchmarks are 

described, but with no details 

provided.

No, procedures for monitoring 

achievement of performance 

benchmarks are not mentioned. 

Procedures for monitoring achievement of 

performance benchmarks include how 

benchmarks are measured and collected, 

who reviews progress toward benchmarks, 

and the frequency of performance 

monitoring.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.B.4 (continued)

38

Question Response options Additional guidance

III. Continuous quality improvement (CQI)

1. Are CQI processes 

proposed for the scaled 

intervention? 

Yes (full), CQI processes are 

proposed, with details provided

Yes (limited), CQI processes are 

proposed, but with no details 

provided.

No, CQI processes are not 

mentioned.

CQI processes refer to procedures for 

continuously assessing the quality of the 

intervention as implemented to improve 

implementation practice. CQI includes 

regularly testing the intervention and 

making adjustments as needed, with 

continual retesting of the modifications to 

ensure that the intervention is succeeding 

as planned. Details include a description of 

the data collected throughout this process. 

2. Does a plan exist to 

analyze data to support 

CQI for the scaled 

intervention?

Yes (full), data analysis to support 

CQI are described, with details 

provided.

Yes (limited), data analysis to 

support CQI are described, but 

with no details provided.

No, data analysis to support CQI is 

not mentioned.

CQI requires the collection of data on 

performance benchmarks and use of the 

data to provide ongoing personnel

development training and the delivery of 

technical assistance to partners. Describes 

plans for using data to support CQI 

includes examining personnel achievement 

of performance benchmarks. 

IV. Preservice and inservice training for personnel

1. Do plans exist to provide 

initial training to personnel 

to deliver the scaled 

intervention? 

Yes (full), initial training plans are 

described, with details provided.

Yes (limited), initial training plans 

are described, but with no details 

provided.

No, initial training plans are not 

mentioned.

Training for service delivery given to 

personnel before implementation begins 

includes training on intervention content 

(such as a training on a curriculum used in 

the intervention), methods for service 

delivery (such as motivational 

interviewing), and client processing 

procedures (such as intake procedures).

2. Do plans exist to provide 

ongoing training to 

personnel to implement 

the scaled intervention?

Yes (full), ongoing training plans 

are described, with details 

provided.

Yes (limited), ongoing training 

plans are described, but with no 

details provided.

No, ongoing training plans are not 

mentioned.

Training for service delivery given to 

personnel during implementation and 

throughout the service delivery period, 

includes refresher trainings on intervention 

content, methods for service, and client 

processing procedures (such as intake 

procedures).

V. Communication system

1. Does a communication 

system exist to support 

coordination among 

personnel and partners for 

the scaled intervention?

Yes (full), a communication system 

exists, with details provided. 

Yes (limited), a communication 

system is mentioned, but with no 

details provided.

No, a communication system is not 

mentioned.

Communication systems support 

coordination among personnel and 

partners and specify the frequency and 

expected duration of communication and 

the parties responsible for communication. 

Systems may vary in the extent to which 

they specify and standardize 

communication. For example, a detailed 

plan may state that, for the first six months 

of the intervention, service providers must 

meet as a group with their supervisor for 

one hour each week to discuss topics 

related to recruitment and engagement of 

participants; for the next six months, 

meetings must take place monthly in the 

form of hour-long sessions to discuss 

service provision. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.B.4 (continued)

39

Question Response options Additional guidance

2. Do processes exist for 

ensuring the 

communication system is 

functioning as intended for 

the scaled intervention?

Yes (full), processes for ensuring 

the communication system is 

functioning as intended are 

described, with details provided.

Yes (limited), processes for 

ensuring the communication 

system is functioning as intended 

are described, but with no details 

provided.

No, processes for ensuring the 

communication system is 

functioning as intended are not 

mentioned

Processes to ensure a communication 

system functions as intended include 

obtaining feedback from personnel

regarding the communication system and 

tracking the communication that takes 

place. Details include how feedback was 

obtained regarding the functioning of the 

communication system. For example, a 

communication system may have included 

logs of how often personnel met together 

or used regular check-ins with personnel

eliciting feedback on how easily they feel 

they are able to get in contact with each 

other or key intervention partners.

VI. Data system

1. Does a data system exist 

to support data collection, 

analysis, and decision 

making for the scaled 

intervention?

Yes (full), a data system is 

described, with details provided.

Yes (limited), a data system is 

described, but with no details 

provided.

No, a data system is not 

mentioned.

Data systems capture enrollment and 

participation data and generate reports on 

participation trends. Such systems support 

data collection, analysis, and decision 

making. Details include the information 

captured by the system, the organization’s 

capacity to analyze recorded data, and 

how the organization uses the data to 

support its decision-making processes.

2. Does a process exist for 

ensuring the data quality 

for the scaled 

intervention?

Yes (full), a process for ensuring 

the data quality is described, with 

details provided.

Yes (limited), a process for 

ensuring the data quality is 

described, but with no details 

provided.

No, a process for ensuring the data 

quality is not mentioned.

Processes for ensuring that the data 

collected are of high quality, such as 

checking for the completeness, accuracy, 

consistency, and timelines of the data. For 

example, there is a plan to check 

participation data entered into a data 

system with paper records of attendance or 

case note file reviews.

III.B.5. Enabling context 

This subsection addresses the fourth condition of the scaling readiness framework—the enabling 

context. It includes questions about the organizational supports discussed in subsection III.A.5 

that are available for the scaled intervention (Table III.B.5). Support for scaling pertains to the 

organization’s leaders and key stakeholders, as well as to the organization’s culture and its focus 

on innovation, learning, and improvement. The enabling context involves questions in three 

major areas about the system in which organizations provide services: (1) organizational 

leadership and partner support, (2) innovation and learning and (3) improvements in response to 

challenges. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

40

Table III.B.5. Questions about the enabling context for the organization proposing the 

scaling

Question Response options Additional guidance

I. Leaders, key stakeholders, and partners support for the intervention

1. Is support from 

organizational leaders for 

scaling the intervention 

described?

Yes (full), support from organizational 

leaders for scaling the intervention is 

described, with details provided.

Yes (limited), support from 

organizational leaders for scaling the 

intervention is described, but with no 

details provided.

No, support from organizational 

leaders for scaling is not mentioned.

Describes organizational leaders and 

their commitment to scaling the 

intervention. For example, 

organization leaders are part of the 

team that is overseeing the 

AmeriCorps State and National grant 

and the scaling plan for the 

intervention.

Question 2 is skipped with a “no” 

response.

2. Is support from 

stakeholders and/or 

partners for scaling the 

intervention described?

Yes (full), support from stakeholders 

and/or partners for scaling the 

intervention is described, with details 

provided.

Yes (limited), support from 

stakeholders and/or partners for 

scaling the intervention is described, 

but with no details provided. 

No, support from stakeholders and/or 

partners for scaling the intervention is 

not mentioned.

Describes stakeholders and/or 

partners’ commitment to scaling the 

intervention. Stakeholders and 

partners include service delivery 

partners or leadership at 

implementation sites. Support may be 

demonstrated by securing space for 

service delivery or removing 

responsibility for non-intervention 

related workload for service providers

so they can focus on scaling the 

intervention.

II. Innovation and learning

1. Has the organization 

previously participated in 

efforts to be innovative? 

Yes (full), efforts to be innovative are 

described, with details provided.

Yes (limited), efforts to be innovative 

are described, but with no details 

provided.

No, efforts to be innovative were not 

mentioned,

Describes organizational efforts to 

identify, develop, and implement new 

ways of meeting community needs. 

For example, organizational innovation 

could include offering new products or 

services or identifying novel ways of 

delivering regularly available services. 

2. Has the organization 

previously participated in 

activities to improve its 

interventions? 

Yes (full), efforts to improve its 

interventions were described, with 

details provided.

Yes (limited), efforts to improve its 

interventions were described, but with 

no details provided.

No, efforts to improve its interventions

were not mentioned.

Describes the way in which 

organizations have improved 

interventions. Improvement may be 

demonstrated by a description of how 

the organization advanced its 

practices and enhanced the current 

service array. For instance, the 

evaluated intervention may have 

improved upon a previous version of 

the intervention by making changes to 

its recruitment strategies. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

Table III.B.5 (continued)

41

Question Response options Additional guidance

II. Improvements in response to challenges

1. Did the organization 

previously face challenges 

in supporting intervention 

implementation?

Yes (full), the organization faced 

challenge(s), with details provided.

Yes (limited), the organization faced 

challenges, but with no details 

provided.

No, no challenges are described.

Describes earlier challenges 

experienced by the organization in 

providing organizational support for 

implementation of the intervention. For 

example, organizational leaders might 

have changed or a previous leader 

might not have endorsed the 

intervention and thus refused to 

dedicate personnel to its 

implementation. 

Question 2 is skipped with a “no” 

response. 

2. Did the organization make 

improvements to address 

earlier challenges in 

providing organizational 

support for the scaled 

intervention?

Yes (full), the organization made 

improvements to address 

challenge(s), with details provided.

Yes (limited), the organization made 

improvements to address challenges, 

but with no details provided.

No, improvements to address 

challenges were not mentioned.

Describes improvements made to 

address challenges that may include a 

change in organizational structure or 

the identification of new funding 

sources.

III.B.6. Implementation infrastructure

This subsection addresses the fifth condition of the scaling readiness framework—

implementation infrastructure. It includes questions about the implementation infrastructure

discussed in subsection III.A.6 that are available for the scaled intervention (Table III.B.6): (1) 

financial resources, (2) sufficient personnel to implement the intervention, (3) materials, (4) 

physical space, and (5) a human resource system.
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

42

Table III.B.6. Questions about implementation infrastructure for the organization 

proposing the scaling

Question Response options Additional guidance

I. Financial resources

1. Will the organization 

provide funding for the 

scaled intervention? 

Yes (full), funding for scaling 

exists, with details provided.

Yes (limited), funding after 

scaling exists, but with no details 

provided.

No, funding for scaling is not 

mentioned.

Describes how the organization and, if 

applicable, its partners will provide funding to 

implement the intervention after scaling. Details 

include the amount and source of funds and the 

plan to continue providing funds in the future.

II. Sufficient personnel

1. Will the organization 

provide dedicated 

personnel to 

implement the scaled 

intervention? 

Yes (full), personnel plans are 

described, with details provided.

Yes (limited), personnel plans 

exist, but with no details 

provided.

No, personnel plans are not 

mentioned.

Describes the number of personnel who will 

work on implementation (as compared to the 

number required by the intervention model), 

whether personnel will work exclusively on 

intervention implementation, and, if applicable, 

partners’ role in providing personnel to 

implement the intervention.

III. Materials 

1. Will the organization 

have the materials 

needed for the scaled 

intervention?

Yes (full), materials are 

described, with details provided.

Yes (limited), materials are 

mentioned, but with no details 

provided.

No, materials are not mentioned.

Describes how the organization will obtain 

materials needed for implementation after 

scaling, such as workbooks, culturally sensitive 

documents, or other handouts. 

IV. Physical space

1. Will the organization 

have the physical 

space needed for the 

scaled intervention?

Yes (full), the physical space is 

described, with details provided.

Yes (limited), the physical space 

is mentioned, but with no details 

provided.

No, the physical space is not 

mentioned.

Describes how the organization has or will obtain 

the physical space needed for implementation 

after scaling, such as classrooms to fit the target 

number of participants or private meeting spaces 

for one-on-one service delivery.

IV. Human resource system

1. Is a human resource 

system in place to hire, 

supervise, and develop 

the personnel for the 

scaled intervention? 

Yes (full), a human resource 

system is in place, with details 

provided.

Yes (limited), a human resource

system is mentioned, but with no 

details provided.

No, a human resource system is 

not mentioned.

Describes a human resource system to support 

implementation after scaling. Details specify how 

the human resource system supports processes 

and procedures for hiring appropriate personnel, 

describe a supervisory structure, and provide for 

personnel development, including a description 

of how supports are routinized and standardized. 
Scaling Evidence-Based Models: Document Review Rubrics Mathematica

43

REFERENCES 

Maxwell, Nan L., and Scott Richman. “Scaling Programs with Research Evidence and 

Effectiveness (SPREE).” Foundation Review, vol. 11, no. 3, November 2019, pp. 55–67. 

Available at https://nationalservice.gov/impact-our-nation/evidence-exchange/scalingprograms-research-evidence-and-effectiveness. Accessed May 18, 2021. 

Richman, Scott, and Andrei Streke. “Evidence of Effectiveness in AmeriCorps-Funded 

Interventions.” Submitted to AmeriCorps by Mathematica. Chicago, IL, September 2020. 

Available at https://americorps.gov/evidence-exchange/evidence-effectiveness-americorpsfunded-interventions. Accessed May 18, 2021. 

Selekman, Rebekah, Katie Eddins, Natalie Reid, and Scott Richman. “Planned Scaling Activities 

of AmeriCorps-Funded Organizations.” Submitted to AmeriCorps by Mathematica. Chicago, 

IL, September 2020. Available at https://americorps.gov/evidence-exchange/planned-scalingactivities-americorps-funded-organizations. Accessed May 18, 2021. 
This page has been left blank for double-sided copying. 
This page has been left blank for double-sided copying. 
Mathematica

Princeton, NJ • Ann Arbor, MI • Cambridge, MA 

Chicago, IL • Oakland, CA • Seattle, WA

Tucson, AZ • Woodlawn, MD • Washington, DC 

EDI Global, a Mathematica Company

Bukoba, Tanzania • High Wycombe, United Kingdom

mathematica.org