U.S. Department of Health & Human Services | National Institutes of Health

Implementation
Science
at a Glance
A Guide for Cancer Control Practitioners

Foreword
Since the National Cancer Institute (NCI) Implementation Science Team was formed within the
Division of Cancer Control and Population Sciences some fifteen years ago, we have seen theimportance of improving the resources needed to support implementation of evidence-based
cancer control interventions. While the team also focuses on efforts to advance the scienceof implementation, and considers the need to integrate implementation science within the
broader field of cancer control and population sciences, we know that unless we support the
adoption, implementation, and sustainment of research-tested interventions in communityand clinical settings, we will not move very far in reducing the burden of cancer. To that end,
we recognize that the advances in our understanding of implementation processes in recent
years will have greater benefit if communicated in a way that supports and informs theimportant work of cancer control practitioners.
This resource, Implementation Science at a Glance, is intended to help practitioners and policy
makers gain familiarity with the building blocks of implementation science. Developed byour team and informed by our ongoing collaborations with practitioners and policy makers,
Implementation Science at a Glance introduces core implementation science concepts, tools,
and resources, packaged in a way that maps to the various stages that practitioners may
find themselves in as they seek to use evidence-based interventions to meet the needs ofpatients, families, and communities. This resource also includes several case examples of
how cancer control organizations have gone through the process of exploring evidencebased interventions, preparing for their integration into varied practice settings, actively
implementing them, and evaluating their impact over time.
While we know that the volume of implementation science topics can fill many books, wehope that this resource provides an initial set of valuable and digestible information, alongwith suggested resources for those interested in learning more. In addition, we hope that
this resource can continue to be refined over time, and that you share your experiences inapplying this to the betterment of your communities and key constituents. Thank you for all
your efforts to address cancer control needs, and thanks in advance for your guidance as we
improve the impact of our research.

David A. Chambers, DPhil

Deputy Director for Implementation Science
Division of Cancer Control and Population Sciences
National Cancer Institutei

Implementation Science at a GlanceAcknowledgments
The National Cancer Institute would like to thank the many cancer control practitioners,
implementation scientists, and program partners whose thoughtful comments and robust
contributions were integral to the creation of this resource.
We particularly thank Amy Allen, Stephenie Kennedy-Rea, and Mary Ellen Conn; Ann-Hilary
Heston; Kathryn Braun; and Suzanne Miller Halegoua and Erin Tagai for their contributions tothe case studies.
We are grateful to the NCI Implementation Science Team, particularly Margaret Farrell, Wynne
Norton, and Prajakta Adsul, who provided hours of review and consultation and guided
this resource to its completion. We note with gratitude the work of Dalena Nguyen, whose
attention to detail and commitment to excellence greatly enhanced the resource’s content andquality, as well.

Implementation Science at a GlanceiiImplementation Science at a GlanceContents
1

Foreword
Acknowledgments
Introduction

1
1
2

Who Is This Guide For?
How Do I Use This Guide?
What Is Implementation Science and Why Is It Important?

4

Assess5Engaging Stakeholders and Partners
Confirming Evidence for an Intervention
Choosing an Intervention

i
ii

6
8

10
11
11

16
17
18
19
21

24

PrepareMaintaining Fidelity
Adapting an Intervention

Implement
Diffusion of Innovations
Consolidated Framework for Implementation ResearchInteractive Systems Framework for Dissemination and Implementation
Implementation Strategies

Evaluate

32

What to Evaluate
How to Evaluate
Sustainability
Scaling Up
De-Implementing

33

Case Studies

33
35
37
39

West Virginia Program to Increase Colorectal Cancer Screening
Kakui Ahi (Light the Way): Patient Navigation
Tailored Communication for Cervical Cancer Risk
LIVESTRONG® at the YMCA

41

Implementation Resources for Practitioners
Glossary of Terms
References

25
27
29
30

43
45

Implementation Science at a GlanceIntroduction
Implementation Science at a Glance details how greater use of implementation science
methods, models, and approaches can improve cancer control practice.
While many effective interventions can reduce cancer risk, incidence, and death, as well as
enhance quality of life, they are of no benefit if they cannot be delivered to those in need.
Implementation strategies are essential to improve public health. In the face of increasingly
dynamic and resource-constrained conditions, implementation science plays a critical role indelivering cancer control practices.
Implementation Science at a Glance provides a single, concise summary of key theories, methods,
and considerations that support the adoption of evidence-based cancer control interventions.

Who Is This Guide For?
We wrote this guide for cancer control practitioners who seek an overview of implementation
science that is neither superficial nor overwhelming.
Implementation science is a rapidly advancing field. Researchers from many disciplines are
studying and evaluating how evidence-based guidelines, interventions, and programs are put
into practice.

How Do I Use This Guide?
Implementation Science at a Glance offers a systematic approach to implement your evidencebased, public health program, regardless of where you are in your implementation process.
We organized this guide into a four-stage framework: assess, prepare, implement, andevaluate. Each stage poses important questions for practical considerations.
ADDITIONAL RESOURCES
Look forend of this workbook.

, which links to a list of additional resources at the

While this guide is organized into four distinct stages, these components blend and overlap
in practice. We have also included four case studies to illustrate how implementation scienceplays out in real-world settings.

1

Implementation Science at a GlanceASSESSPREPARE»» Evidence-Based Interventions»» Adaptations»» Stakeholder Engagement and»» Fidelity

EVALUATE

IMPLEMENT»» Sustainability»» Theories»» Scale-Up»» Models»» De-Implementation»» Frameworks»» Return on Investment»» Implementation Strategies

Partnerships

What Is Implementation Science and
Why Is It Important?
Implementation science is the study of methods to promote the adoption and integrationof evidence-based practices, interventions, and policies into routine health care and publichealth settings to improve the impact on population health.1
Implementation science examines how evidence-based programs work in the real world. By
using implementation science and implementation strategies, you can help bridge the divide
between research and practice—and bring programs that work to communities in need.
Applying implementation science may help you understand how to best use specific strategies
that have been shown to work in your (or similar) settings.
By applying implementation science frameworks and models, you may:»» Reduce program costs»» Improve health outcomes»» Decrease health disparities in your community

Implementation Science at a Glance2

3

Implementation Science at a Glance

Implement

Assess

Implement

Implementation Science at a Glance

4

Engaging Stakeholders and Partners
People and place matter. Therefore, it isimportant to seek out stakeholder input
throughout your process of preparation andimplementation. Consider partnering withresearchers to advance your goals. Creating
meaningful partnerships will help you:»» Enhance the quality and practicality

Stakeholders are people, communities,
and organizations that could be
affected by a situation. While internal
stakeholders participate throughcoordinating, funding, and supporting
implementation efforts, external
stakeholders contribute views andexperiences in addressing the issues
important to them as patients,
participants, and members of thecommunity.2,3»» Disseminate your evaluation findings

Ask these questions to identify key
stakeholders:»» Better understand your community andits strengths and weaknesses, assets,values, culture, traditions, leaders, andfeelings on change»» Increase the likelihood thatyour intervention will be adopted
and sustained»» Ensure that your intervention isrelevant to stakeholdersof your efforts»» Create relationships with academic centersthat can help sustain your program

Creating and leveraging partnerships withresearchers and academic programs canempower communities and create socialchange. Successful research and community
collaboration can be particularly effective in:»» Fostering a willingness to learn fromone another»» Building community members’
involvement in research»» Benefiting all partners withresearch outcomes»» Increasing buy-in for your program

5

What You Can Do:
Engage Stakeholders

Implementation Science at a Glance»» Who will be affected by what we aredoing or proposing?

»» Who are the relevant officials?
»» What are the relevant organizations?
»» Who has been involved in similar
situations in the past?

»» Who or what is frequently

associated with relevant topic areas?

Measuring and assessing outcomes
important to stakeholders can havea significant impact on the adoption,
implementation, and sustainment ofevidence-based practices.
Remember that stakeholder
engagement is necessary throughout
the entire implementation process.
These questions can help guide you as
you move through the assess, prepare,
implement, and evaluate stages.

AssessKey Questions
ASSESS

PREPARE

IMPLEMENTEVALUATE»» Does your selected»» How will stakeholder»» Did you identify and»» How will you adjust yourevidence-based
intervention takestakeholders’ goals andneeds into account?

engagement help
achieve program
objectives?

»» Does your plan
include stakeholder
engagement on key
decisions?

»» Can your team engage
stakeholders effectively,
or will you need to hire
experts?

communicate clear roles
for all stakeholders?

»» Did you include
resources forstakeholder
engagement in yourimplementation plan?

implementation plan in
response to stakeholder
feedback?

»» What outcomes aremost important to eachgroup of stakeholders
and each partner?

»» Are you planning toinclude stakeholder
engagement as you
monitor, review,and evaluate yourimplementation?

Confirming Evidence for an Intervention
Cancer control practitioners make decisions based on various types of evidence: from more
subjective evidence—such as their direct experience with the populations they work with—tomore objective sources of evidence—including the results of well-designed research studies.
In this resource, we use the terms “interventions,” “practices,” and “programs”
interchangeably. We generally consider an intervention to be a combination of program
components, while a program often groups several interventions together.
An evidence-based intervention is a health-focused intervention, practice, policy, or guideline
with evidence demonstrating its ability to change a health-related behavior or outcome. 4
Using evidence-based interventions can not only increase your effectiveness but also help
save time and resources.
The less robust the body of evidence supporting a program’s effectiveness, the more
important it is to evaluate the program and share your results.

Assess

Implementation Science at a Glance

6

OBJECTIVESYSTEMATIC REVIEWS
Summaries of a body of evidence
made up of multiple studies andrecommendations

What You Can Do:
Make Sure the Intervention Is
Evidence-Based
How will you know if an intervention
is evidence-based? A quick internet
search may suggest a wide variety ofinterventions, which may or may not
be evidence-based. It is important toevaluate these potential sources.
Evaluate existing information on theintervention, and consider:4»» Who created the information?
RESEARCH STUDIES»» What types of interventions are

Individual studies that test a
specific intervention»» What methods were used to review

highlighted?

the evidence?

»» What criteria were applied to assess
an intervention?

PRACTITIONER REPORTS
Reports, briefs, or evaluations of astrategy in practice»» How current is the evidence?
»» Are resources available to help you
implement the intervention?

The following resources may provide
evidence to support your intervention:

EXPERT OPINION/PERSONAL EXPERIENCE
Recommendations made by credible
groups or individuals that have notyet been tested»» United States Preventive Services
Task Force»» NCI Research-Tested Intervention
Programs»» Healthy People Tools and Resources(Healthy People 2020)»» Pew-MacArthur Results FirstSUBJECTIVE

Initiative»» The Guide to Community Preventive
Services (The Community Guide)

Figure 1. A continuum of evidence to support interventions5

7

Implementation Science at a Glance

AssessChoosing an Intervention
While it is important for interventions to be grounded in research and evidence, your program
will only be effective if it “fits” your community population and your resources.
When choosing an evidence-based intervention to implement, consider the following:6» Does this intervention fit our community’s demographics, needs, values, and risk factors?
» Different interventions will take different amounts of money, labor, and time.
–– Does our organization have the capacity and resources this intervention requires?
–– Do we have the expertise to implement this intervention?
–– Can we engage partners or leverage other resources?
» Does this intervention target our overall goal?
After selecting your intervention, be wary about recommending adaptations without specificguidance, such as from the original developers. Adapting some aspects of an intervention canlead to a “voltage drop”: a change in expected outcome when an intervention moves from a
research setting into a real-world context.
The following sections can help guide you as you decide to implement an intervention as-is, if
it is a good fit, or to first adapt it to your local community.

Population
characteristics, needs,
values, and preferences

DECISION
MAKING
Resources, including
practitioner expertise

Best available
research evidence

Figure 2. Components to consider when selecting an intervention7
ADDITIONAL RESOURCES

Assess

Implementation Science at a Glance89Implementation Science at a Glance

ImplementPrepare

Implement

Implementation Science at a Glance

10

Maintaining Fidelity
Fidelity refers to the degree to which an evidence-based intervention is implemented
without compromising the core components essential for the program’s effectiveness. 8,9Lack of fidelity to the original design or intent makes it difficult to know which version of theintervention was implemented and, therefore, what exactly caused the outcomes.

Why is fidelity important?
One of the most common reasons that practitioners do not get the results they anticipate isthat they have not properly implemented the practice or program. To avoid this problem, andto get better results, you must understand the importance of implementing the practice orprogram as intended.
When interventions implemented with fidelity are compared to those not implemented withfidelity, the difference in effectiveness can be profound. Those implemented with fidelity will
have a greater impact on outcomes than those implemented without fidelity.10

The core components of an intervention relate to its:11»» Content – the substance, service, information, or other material that the intervention
provides (e.g., screening tests)»» Delivery – how the intervention is implemented (e.g., setting, format, channels, providers)»» Method – how the intervention will affect participants’ behavior or environment

Adapting an Intervention
Evidence-based interventions are not one size fits all. You may have to adapt them to better fit
the population or local conditions.12 Adaptations may involve the addition, deletion, expansion,
reduction, or substitution of various intervention components. 8,12,13 Core components to adapt
to may include the setting, target audience, delivery, or culture.14
Adapting an intervention can help improve health equity. For example, organizations thatserve communities with limited economic resources, such as local health departments
or safety-net health centers, may adapt some parts of the intervention to leverage theirresources while still working toward similar outcomes.
Additionally, sharing information about your adaptations and results can contribute to agreater understanding of the full range of factors that impact implementation in high-needand under-resourced areas.
There are many areas in which changes to the original intervention can take place. See Figure 3.

11

Implementation Science at a Glance

PrepareBefore adapting an intervention, consider
the following:

SERVICE SETTING»» Are adaptations necessary?

TARGET AUDIENCE»» How important is it to your partners toadapt this intervention?

MODE OF DELIVERY»» What adaptation would you make?

CULTURE»» Do you have the resources to implement
the adapted intervention?

CORE COMPONENTS
Figure 3. Sources of intervention adaptation14

What You Can Do: Balance Fidelity and Adaptations
Making too many changes toan intervention can reduce
its original effectiveness, orworse, introduce unintended
and harmful outcomes.
Before making adaptations
to the intervention, you
should think about how
the change to the original
intervention can improve
the fit to your community,
setting, or target population,
and at the same time,
maintain fidelity to the corecomponents of the original
intervention. Think ofpossible adaptations as you
would a green, yellow, or
red traffic light: green lightchanges are usually OK to
make; yellow light changes
should be approachedwith caution; and red lightchanges should be avoided
when possible.12

Prepare»» Usually minor»» Made to increase the reach, receptivity, andparticipation of the community

GREEN LIGHT
CHANGES»» May include:–– Program names–– Updated and relevant statistics or healthinformation–– Tailored language, pictures, cultural

indicators, scenarios, and other content»» Typically add or modify intervention components
and contents, rather than deleting them

YELLOW LIGHT
CHANGES»» May include:–– Substituting activities–– Adding activities–– Changing session sequence–– Shifting or expanding the primary
audience–– Changing the delivery format–– Changing who delivers the program

RED LIGHT
CHANGES»» Changes to core components of the intervention»» May include:–– Changing a health behavior model ortheory–– Changing a health topic or behavior–– Deleting core components–– Cutting the program timeline–– Cutting the program dosage

Implementation Science at a Glance

12What You Can Do: Use a Systematic Approach to Adaptations
Try this five-step process when adapting an intervention.12 The more adaptations you make,the more you will need to re-evaluate the effectiveness of the intervention.

1
ASSESS THE ACCEPTABILITY

and importance of adaptationand consider adaptation

2
3

MAKE ADAPTATIONSASSESS FIT

MAKE FINAL DECISIONS
about what and how to adapt

4
5

PRETEST AND PILOT TEST

Figure 4. A systematic approach to adapt your intervention1213Implementation Science at a Glance

PrepareADDITIONAL RESOURCES

PrepareImplementation Science at a Glance

14

15

Implementation Science at a Glance

ImplementImplement

Implement

Implementation Science at a Glance

16

There are multiple theories, models, and frameworks frequently used in implementation sciencethat can guide you as you plan, implement, and evaluate your intervention. While theories,
models, and frameworks are distinct concepts, this resource uses them interchangeably.
Implementation science models provide guidance for understanding how to address the gap
between identifying an intervention and ensuring its adoption (the research-to-practice gap)
and later sustaining the intervention.
By spending the time to understand these underlying processes, you will be better preparedto more rapidly move effective programs, practices, or policies into communities.
Models can help you understand the logic of how your implementation effort creates animpact and offer clear constructs to measure that impact.
Using models can also help you find problem areas at your setting and help guide theselection of implementation strategies. Some models include:» Diffusion of Innovations» Consolidated Framework for Implementation Research» Interactive Systems Framework for Dissemination and Implementation

Diffusion of Innovations
Diffusion of Innovations theory refers to the process by which an innovation is communicatedover time through members of a social network.15 Diffusion consists of four elements:» The innovation, idea, practice, or object that is intended to be spread» Communication or the exchange of messages» A social system, structure, or group of individuals that interact» A process of dissemination or diffusion that occurs over time
This theory suggests that an innovation, like an evidence-based intervention, will be
successful or adopted by individuals when the innovation is diffused or distributed throughcommunities.16 Tables 1 and 2 describe the theory further.16
Table 1. Concepts of Diffusion of Innovations
Constructs

17

Definition

Innovation

An idea, object, or practice that an individual, organization, or community believes is new

Communication channels

The means of transmitting the new idea from one person to anotherSocial systems

A group of individuals who together adopt the innovationTime

How long it takes to adopt the innovation

Implementation Science at a Glance

ImplementTable 2. Key Attributes Affecting the Speed and Extent of an Innovation’s Diffusion
Constructs

Definition

Relative advantage

Is the innovation better than what it will replace?

Compatibility

Does the innovation fit with the intended audience?

Complexity

Is the innovation easy to use?

Testability

Can you test the innovation before deciding to adopt?

Observability

Are the results of the innovation observable and easily measurable?

Consolidated Framework
for Implementation Research
The Consolidated Framework for
Implementation Research (CFIR) can help
you identify what aspects of your context
you should assess during the planning
process. CFIR has five domains:17

Each CFIR domain provides a menu of
key factors for you to choose from. The
factors have been linked with the effective
implementation of interventions. Examples
include, but are not limited to:» Intervention characteristics» Intervention characteristics: relativeadvantage, complexity, cost

» Characteristics of individuals involved» Characteristics of individuals involved:» Inner setting

self-efficacy, knowledge/beliefs about
intervention» Outer setting» Process» Inner setting: culture, readiness for
implementation» Outer setting: external policy and incentives

INNER SETTING
CHARACTERISTICS OF INDIVIDUALSONTI
E N D)
RV TE
TE AP
E
IN AD
BL Y
(
TA E RA P PHA DE R IPOUTER SETTINGCORE
COMPONENTS

IN
T(U E RNA VE
DA N TADP T IONED
PE A P)R I TAPH B LER E
Y
CORECOMPONENTS» Process: planning, champions

PROCESS
Figure 5. Consolidated Framework for Implementation Research17

Implement

Implementation Science at a Glance18Interactive Systems Framework for
Dissemination and Implementation
The Interactive Systems Framework (ISF) for Dissemination and Implementation wasdeveloped to address the “how-to” gap between scientifically determining what works andmoving that knowledge into the field for the benefit of the public.18
Figure 6 shows the ISF and how it connects three systems to work together for successful
dissemination and implementation. The term “system” is used broadly to describe a set of
activities that accomplish one of the three identified functions that make dissemination andimplementation possible. These systems are:19» SYNTHESIS AND TRANSLATION SYSTEM – Here, scientific knowledge is distilled into

understandable and actionable information. Research institutions, universities, and NCI
are all institutional examples of this system.

» SUPPORT SYSTEM – This system supports the work of the other two systems by building

the capacity to carry out prevention activities. Agencies like state health departments orstate cancer control coalitions are often in the role of prevention support for grantees or
local programs.

» DELIVERY SYSTEM – This is where innovations are implemented or where “the rubber
meets the road.” Community-based organizations often function in the role of theprevention delivery system.

As depicted in Figure 6, these three systems work together and are embedded within anunderlying context that influences decision making and the adoption of interventions. These
underlying conditions include:» Legislation that supports funding for cancer prevention and control» The best available theory and research evidence» The community or organizational context in which interventions are implemented» Macro-level policy factors such as state or federal budget constraints or legislative changes
These underlying considerations are graphically displayed as the climate in which the threesystems exist, and all of these have an impact on successful dissemination and implementation.
Each system within the ISF also builds upon or influences the functions of the other two
systems. These relationships and influences are represented by the arrows that connect thesystems to each other.
The ISF can offer you a non-exhaustive list of practical considerations and strategies toaddress each of the three systems involved. These strategies will make up the implementation
effort, leading to the population health and implementation outcomes you seek to change.19,20

19

Implementation Science at a Glance

Implement

FUNDING
DELIVERY SYSTEM

MOTIVATIONINNOVATIONSPECIFIC CAPACITYSUPPORT SYSTEM
GENERALCAPACITY

MOTIVATIONINNOVATIONSPECIFIC CAPACITY

CLIMATE

MACRO-POLICY

GENERALCAPACITYOUTCOMESSYNTHESIS & TRANSLATION SYSTEMSYNTHESIS

TRANSLATION

Figure 6. The Interactive Systems Framework for Dissemination and Implementation19

Implement

Implementation Science at a Glance20Implementation Strategies
Implementation strategies are the “how-to” components of interventions. 4,21 Think of these as
ways to implement evidence-based practices, programs, and policies.
Implementation strategies are the essential components of implementation science but areoften not adequately described nor labeled properly. A commonly used definition pitches
them as “specific methods or techniques used to enhance the adoption, implementation,and sustainability of a public health program or practice.”21 It is important to note that an
implementation strategy focuses on improving implementation outcomes such as acceptability,
adoption, appropriateness, feasibility, costs, fidelity, penetration, and sustainability.
Recent progress has been made to identify and define strategies relevant to the health care
context, which resulted in a list of seventy-three distinct strategies. 22 These strategies can begrouped into eleven categories as shown in Figure 7. This list provides a good starting point tounderstand the different types of strategies that have been used and tested previously, andalso facilitates the selection of strategies that might be relevant to your practice context.

DEVELOP STAKEHOLDER
INTERRELATIONSHIPSCONVENE TEAMSUTILIZE FINANCIAL STRATEGIESPRACTICE FACILITATIONENGAGE CONSUMERSPROVIDE INTERACTIVE ASSISTANCEUSE EVALUATION PLAN
AND INTERACTIVE STRATEGIESSUPPORT PRACTITIONERSCHANGE INFRASTRUCTURE
ADAPT AND TAILOR TO CONTEXTTRAIN AND EDUCATESTAKEHOLDERS

Examples of “Train and Educate
Stakeholders” Strategies» Conduct educational outreach visits» Use train-the-trainer strategies» Create a learning collaborative» Provide ongoing consultation

Figure 7. Implementation strategy categories and examples23

21

Implementation Science at a Glance

Implement

What You Can Do: Identify Implementation Strategies
To maximize the potential of your implementation efforts, it is important that you select strategies
that fit your local context.
Discuss with stakeholders the factors that may influence how your intervention is implemented.
Their perspectives can provide important insights about the community and other contexts.
Generating a list of these contextual considerations can be an important step to determine which
implementation strategies best fit the local context. Table 3 illustrates how, once you have this list,
you may select strategies to address these determinants.
It is rare to use a single strategy during implementation. Selecting multiple strategies to address
multiple barriers to implementing the intervention may be necessary. You may also need to selectdifferent strategies in different phases of implementation.
Methods such as concept mapping and intervention mapping may also help you select relevant
implementation strategies.24
Table 3. Selecting Strategies Based on Influential Factors
Identified Factor

Implementation Strategy

YOUR DETERMINANT

YOUR STRATEGY

Lack of knowledge

Interactive education sessions

Beliefs or attitudes

Peer influence or opinion leaders

Community-based services

Process redesign

ADDITIONAL RESOURCES

Implement

Implementation Science at a Glance

22

23

Implementation Science at a Glance

Implement

Evaluate

Implement

Implementation Science at a Glance24What to Evaluate
Is what we’re
doing working?

Why or why not?

How do we show the valueof the work we do?

Evaluation is the systematic collection of information about activities, characteristics, andresults of programs to assess the program and implementation outcomes. 25
Depending on what you and your organization prioritize, you can choose specific outcomes
you want to target, define what success would look like, and then evaluate your success.

Key Outcomes
There are many outcomes that you can evaluate to assess or determine whether yourimplementation efforts were successful.
There are distinct categories of outcomes you can evaluate in implementation science:» Implementation outcomes» Community outcomes» Program outcomes» Individual outcomes

The differences between these are important. Implementation outcomes assess the effects
of implementation efforts, while community and individual outcomes assess the effects of
the intervention. If your intervention does not achieve community or individual outcomesas expected, it is important to know whether the failure is due to ineffectiveness of theintervention in your setting or to ineffective implementation of the intervention. To evaluate
implementation, you must assess implementation outcomes.
As depicted in Table 4, implementation outcomes have three important functions:26» Indicators of implementation success» Proximal indicators of implementation process» Key intermediate outcomes that impact community- and individual-level outcomes
Table 4. Key Outcomes in Implementation Science
Implementation
Outcomes» Acceptability» Adaptation» Adoption» Appropriateness» Feasibility» Fidelity» Maintenance» Penetration» Sustainability

25

Program
Outcomes» Cost-effectiveness» Effectiveness» Equity» ReachImplementation Science at a GlanceCommunity
Outcomes

Individual
Outcomes» Access to care
» Access to fresh» Longevity» Physical activity and» Built environment» Disease incidence» Disease prevalence» Health disparities» Immunization and» Social connectedness» Quality of life

producefitness

vaccination» Walkability

Evaluate

Measurement Tools
Measurements of some implementation outcomes can be captured by examining attitudes,
opinions, intentions, and behaviors. 26 Additional measures for assessing implementation
outcomes can be found through the Society for Implementation Research Collaboration.

What You Can Do: Choose Outcomes to Measure
Table 5 defines nine implementation outcomes, their most relevant stage during implementation,and some methods to measure them.26
Table 5. Implementation Outcomes
Implementation
OutcomeAcceptability

Adoption

Definition
Perception among stakeholders
that the program is agreeable tothe intervention
Intention among stakeholders to
employ an intervention

Implementation
Stage
Early for adoption
Ongoing for penetration
Late for sustainability

SurveySurveyEffectivenessImpact of an intervention on
important outcomes

Mid to late

Feasibility

Extent to which the intervention
can be successfully used within agiven setting

Early (during adoption)

Fidelity

Degree to which an intervention
was implemented as intended by
the program developers

Early to mid

Cost impact of an
implementation effort

Administrative data

Observation interviews

Early (prior to adoption)

Implementation cost

Survey interviews

Administrative data
Early to mid

Perceived fit of the innovation orintervention for a given setting/population/problem

AppropriatenessWays to Measure

Interviews
Focus groups
Observation
Interviews
Surveys
Administrative data
Observation checklists
Self-reporting

Early for adoption andfeasibility
Mid for penetration

Administrative data

Late for sustainability
Penetration

Integration of an intervention
within a community,organization, or system

Sustainability

Extent to which the intervention
is maintained over time

Evaluate

Mid to late

Program audits
Checklists
Program audits

LateInterviews
Checklists

Implementation Science at a Glance

26

How to Evaluate
A number of approaches are available to structure your evaluation and provide a better
understanding of how and why your implementation efforts succeed or fail. 27 While thefollowing is not an exhaustive list, it provides a summary of evaluation approaches andframeworks often used by practitioners to address the pragmatic needs of their context.

Logic Models
Logic models are a visual representation of how a program is expected to produce desired
outcomes. 28 A logic model can be used in the development, planning, and evaluation phasesof your program implementation, but for the purposes of this guide, logic models will be used
as an evaluation tool. In using a logic model, you can identify the interrelationships of yourinputs and activities, and how they relate to your desired short-term, mid-term, and long-term
outcomes to be measured.

INPUTSINTERVENTIONACTIVITIESOUTPUTS &OUTCOMES

LONG-TERMOUTCOMESCONTEXTUAL AND EXTERNAL FACTORS
Figure 8. Simplified logic model28

Evaluability Assessments
Evaluability assessments are a useful evaluation approach if your program is new or
premature for evaluation. 29,30 These assessments require developing a logic model with all
the stakeholders involved. Evaluability assessments are highly participatory and result in
stakeholders reporting on five findings:» Plausibility» Areas of program development» Evaluation feasibility» Options for further evaluation» Critique of current data availability

27

Implementation Science at a Glance

Evaluate

Reach, Effectiveness, Adoption, Implementation, Maintenance (RE-AIM) Framework
You can use the RE-AIM framework to inform your planning, evaluation, and reporting, but
it is most often used for implementation evaluation. 31 RE-AIM is an especially useful tool forpractitioners because it was created in response to the need for attention to external validity,or the extent to which the implemented intervention would be generalizable to other realworld settings. 32,33
Table 6. Defining RE-AIM Constructs
Construct

Definition

Reach

The number, proportion, and representativeness of individuals who are willing toparticipate in an intervention

EffectivenessThe impact of an intervention on important outcomes, including potential negative
effects, quality of life, and economic outcomes

Adoption

The absolute number, proportion, and representation of settings and intervention
agents who are willing to initiate an intervention

Implementation

The intervention agents’ fidelity to the various components of an intervention’s
protocol (e.g., delivery as intended)

Maintenance

The extent to which an intervention or policy becomes institutionalized or part ofroutine practice and policy

Economic Evaluation
Economic evaluations will be useful to you if you are interested in the affordability of yourimplementation efforts in achieving individual and community outcomes. 34 A very expensive
intervention that produces small improvement in outcomes is less appealing than anotherintervention that produces the same outcome at a fraction of the cost. Economic evaluations help
you quantify cost-effectiveness and can help justify scaling up the intervention in the future.

Evaluate

Implementation Science at a Glance28

SustainabilityYour intervention can only deliver population benefits if you are able to sustain your activities
over time. Sustainability describes the extent to which an evidence-based intervention cancontinue to be delivered, especially if external support or funding ends. 35
You will only be able to sustain effective implementation efforts if you keep evaluating andadapting it to your setting and population. Therefore, after you evaluate your efforts, you
should reassess and continue sustaining the implementation.

What You Can Do: Sustain Your Intervention Program
Consider the following eight core domains to increase the intervention’s capacity forsustainability. 36,37 These domains were developed by practitioners, scientists, and funders
from several public health areas.
You can use the Program Sustainability Assessment Tool to understand factors that influence
your intervention’s capacity for sustainability and develop an action plan to increase thelikelihood of sustainability. The tool helps identify your organization’s sustainability strengths
and weaknesses and can guide your sustainability planning.
Factors Influencing Sustainability

29

FUNDING STABILITY

PARTNERSHIPS

Establishing a consistent financial basefor your program

Cultivating connections between yourprogram and its stakeholders

PROGRAM ADAPTATIONSTRATEGIC PLANNING

Changing your program to ensure its
ongoing effectiveness

Using processes that guide yourprogram’s direction, methods, and goals

POLITICAL SUPPORTORGANIZATION CAPACITY

Maintaining relationships with internaland external stakeholders who support
your program

Having the internal support and resources
needed to effectively manage your programand its activities

PROGRAM EVALUATIONCOMMUNICATIONSAssessing your program to informplanning and document results

Exchanging information about yourprogram with stakeholders and the public

Implementation Science at a Glance

Evaluate

Scaling Up
If the intervention has been successful in your setting, you or your organization might beconsidering “scale-up.” Scaling up is the deliberate effort to increase the impact of successfulinterventions so that they can benefit more people and foster sustainability. 38 You can scaleup your implementation effort in three ways, as shown in Figure 9.
Scaling up requires a new examination of your partnerships and resources to decide if there isevidence to support the adapted intervention.

VERTICAL
Adoption by different jurisdictions for policy-based,
systematic, and structural change

HORIZONTAL
Expansion across the same system levels, suchas departments, organizations, sectors

DEPTH
Addition of new components to an existing innovation

Figure 9. Potential directions for scaling up in population public health39

Evaluate

Implementation Science at a Glance30

What You Can Do: Proceed in Increments and Closely Monitor Your Progress
Multiple conditions and external institutions affect the process and prospects for scale-up.
The environment presents many opportunities and obstacles that must be identified andaddressed when deciding how you are going to scale up. You can use these steps and thescale-up framework to systematically plan and manage the scale-up process:40
1. Plan actions to increase scalability of your intervention.
2. Increase the user organizations’ capacity to scale up.
3. Assess the conditions of your environment (e.g., policies, bureaucracy, health and othersectors, socioeconomic and cultural context, people’s needs and rights).
4. Increase the capacity of the resource team and implementers.
5. Make strategic choices appropriate for your scale-up.
ENVIRONMENTTHEINNOVATIONSCALE-UP
STRATEGYRESOURCETEAM

TYPES OFSCALING UP

DISSEMINATION& ADVOCACYORGANIZATIONALCHOICESUSER
ORGANIZATIONSCOSTS/RESOURCE
MOBILIZATIONMONITORING& EVALUATION

Figure 10. The ExpandNet/WHO framework for scaling up

Table 7. Considerations for Scale-Up
Strategic Choices
Types of scaling up

Dissemination and advocacy

Organizational process

Cost/resource mobilization

Monitoring and evaluation

31

Issues to Consider When Choosing Strategies» Vertical scaling up – institutionalization through policy, political, legal,budgetary, or other health systems change» Horizontal scaling up – expansion, replication» Personal – training, technical assistance, policy dialogue, cultivating
champions and gatekeepers» Impersonal – websites, publications, policy briefs, toolkits» Scope of scaling up (the extent of geographic expansion and levels within thehealth system)» Pace of scaling up (gradual or rapid)» Number of agencies involved» Assessing costs» Linking scale-up to macro-level funding mechanisms» Ensuring adequate budgetary allocation» Special indicators to assess the process» Outcome and impact of scaling-up» Service statistics» Local assessments

Implementation Science at a Glance

EvaluateDe-Implementing
De-implementation is the process of reducing
or stopping the use of a practice, intervention,or program. There are many reasons why
a public health agency, organization, or
department may purposely choose to reduce (interms of frequency or intensity) the delivery ofa practice to a target population, or choose tostop offering the practice to a target population
entirely.
Practices that may be appropriate for deimplementation include those that are:» Ineffective (e.g., evidence shows thepractice does not work)» Contradicted (e.g., new and stronger ormore robust evidence shows the practicedoesn’t work)» Mixed (e.g., some evidence shows that thepractice works but other evidence showsthat it doesn’t work)» Untested (e.g., programs that have not yet
been evaluated in a research study)

Determining what practice to de-implement
and how quickly is influenced by many factors,
including how widespread the practice is in use,
what resources are allocated for implementing
the program that might otherwise be spent onoffering effective practices, and the needs ofthe target population.
De-implementation also should include multiple
stakeholders, planning, and considerationof multi-level factors that can influence thede-implementation of a practice. In addition,frameworks, models, and theories that can help
inform and guide the use of strategies facilitate
the de-implementation process. Frameworks
specifically focused on de-implementation,and identification of strategies most effective
for facilitating the de-implementation process,are of increasing interest among researchers,
practitioners, and policy makers.

What You Can Do:
Follow These Steps for
De-Implementation
1. Identify and prioritize practices
that may be appropriate forde-implementation.
a. Is your organization offering practices
that are no longer needed by thecommunity?
b. Is there a more pressing or important
health issue that should be addressed
instead?

2. Gather information on potential
barriers to the de-implementation
process.
a. Will personnel or organizationalchanges be needed if the practice is nolonger offered?
b. Will de-implementing the practice
reduce collaborative opportunities with
community partners?

3. Identify strategies that areneeded to overcome the deimplementation barriers.
a. Will an alternative practice beintroduced to replace the one thatis being removed? What training isavailable for the new practice?
b. What communication is needed toeducate the community on why apractice is no longer being offered?

4. Implement and evaluate strategies
to support de-implementation.
a. Can you identify alternative practices
that could be used to meet the needsof the community while maintaining
strong community linkages?
b. Can you allocate resources to anotherimportant issue or public health practice?

ADDITIONAL RESOURCES

Evaluate

Implementation Science at a Glance32

Case Studies
See how cancer control practitioners use implementation science todeliver effective interventions in their communities.

West Virginia Program to
Increase Colorectal Cancer Screening
The West Virginia Program to Increase Colorectal Cancer Screening (WV PICCS) was fundedby the Centers for Disease Control and Prevention and directed through the West Virginia
University Cancer Institute. The program aimed to increase colorectal cancer screening rates
in persons aged 50‒75 by partnering with health care systems across West Virginia. Theintervention sought to change protocols within health care systems, such as primary carepractices, to increase referral and completion of colorectal cancer screenings.
Each year, WV PICCS partnered with a different cohort of primary care clinics to help increase
their colorectal cancer screening rates. The program prioritized partner clinics serving areas
with high rates of colorectal cancer mortality and late-stage diagnosis. To date, WV PICCS has
partnered with forty-four clinics around the state.
WV PICCS further partnered with health systems to implement at least two evidence-based
interventions and supportive activities shown to increase colorectal cancer screening. As
part of the WV PICCS project, every clinic undertook a provider assessment and feedback
intervention. Each clinic chose to deliver an additional intervention method they believed
would be the best “fit” for both their clinic as well as their patient population.

ASSESSTo help clinics identify and select the second intervention, each health system and WV PICCS
collaboratively assessed the clinic’s capacity, interests, and workflow, among other factors. As
a result, clinics could choose to implement enhanced client reminders, provider reminders, or
structural barriers reduction programs to increase the recommendation for and completion of
screening. The most frequently implemented project was the enhanced call reminder program
to encourage patients to complete and return fecal immunochemical tests to the clinic.

PREPAREWV PICCS staff worked with the health care systems to help tailor the enhanced call reminderintervention to best fit the clinics’ workflow and ensured the protocol was adapted to fit thesetting. For example, while several clinics opted to have nurses place the reminder calls, indifferent clinics, lab technicians or care coordinators made the calls.

33

Implementation Science at a Glance

IMPLEMENTWV PICCS worked with the clinics over a two-year, two-phase implementation period. WV
PICCS leveraged multiple implementation strategies, such as patient navigation and media
outreach, to enhance the intervention implementation and uptake.
WV PICCS provided technical assistance by a staff member to clinics extensively during the
project’s first year. Assigning a staff member to provide the assistance for each clinic provided
tailored technical support and monthly facilitation meetings and helped monitor changes toeach clinic’s care delivery system.
In the second year, the technical assistance was reduced to once a month. This tapering
allowed the clinic and WV PICCS to assess the clinic’s capability to sustain the newinterventions over time.

EVALUATEWV PICCS used the “plan, do, study, act” evaluation cycles to support their implementationand evaluation efforts. In keeping with their model, during the second year of the initiative,
data collection began to assess clinical capacity to sustain the improved colorectal cancer
screening rate.

LESSONS LEARNEDThe call reminders were originally proposed as up to threecalls followed by a letter. Knowing it is important thatinterventions fit the clinical culture, WV PICCS adapted theintervention to allow clinics to send reminders via letter first,
if it better suited their workflow. WV PICCS listened to theclinics when they provided feedback and adjusted the protocol.

As partners,
you need to listen.

Implementation Science at a Glance

34

Kukui Ahi (Light the Way): Patient Navigation
Racial and ethnic disparities affect rates of cancer screening. For example, Asian Americans are lesslikely than non-Hispanic Whites to undergo timely cervical and colorectal screening, and Native
Hawaiians are less likely than non-Hispanic Whites to get a mammogram.41 These disparities may
be due to a combination of health system, provider, and patient factors that decrease access to careand lower patients’ capacity to advocate for their needs. Interventions that promote accessible
and coordinated care may have the potential to increase screening and reduce delays in time todiagnosis and treatment after abnormal screenings.
The intervention “Kukui Ahi (Light the Way): Patient Navigation” used lay-patient navigatorsfrom the local community. In coordination with health care providers, the lay-patientnavigators support Medicare recipients through education, coordinating screenings, providing
transportation, assisting with paperwork, and finding ways to pay for care. They aimed to increase
screening rates for colorectal, cervical, breast, and prostate cancers among Asian and PacificIslander Medicare beneficiaries.

ASSESS
The idea of implementing a patient navigation intervention came from community partners of the‘Imi Hale Native Hawaiian Cancer Network (‘Imi Hale). The ‘Imi Hale team consequently examinedcurrent patient navigation programs, such as the patient navigation intervention at Harlem Hospital,and looked for ways in which they could adapt it for the populations of the Hawaiian Islands.

PREPARETo create a comprehensive training curriculum for the lay-patient navigators implementing the
program, ‘Imi Hale conducted a needs assessment that involved interviewing and collaboratingwith public health practitioners, doctors, community health workers, cancer nurses, and otherstakeholders at the hospital. Additionally, they maximized opportunities to build relationships
between community outreach workers and hospital-based providers by inviting both to the training.
Given the diversity of the population that ‘Imi Hale served, there were three different training
modules created for the lay-patient navigators. Although there was variety in intervention
implementation, there were fourteen key navigator competencies that were consistently included
across all trainings.

IMPLEMENTOne of the reasons why ‘Imi Hale believes they were able to recruit and retain such dedicatedlay-patient navigators is because of their investment in building the navigators’ capacity throughongoing training. The team sent their navigators to trainings that occurred in places like New York
City and Michigan so that their navigators could build their network and capacity. The team also
created a dynamic training atmosphere where they emphasized that the lay-patient navigators were
not expected to know everything necessary during the implementation process; rather, their role
was to be an active learner and find the resources needed to assist their patients.

35

Implementation Science at a GlanceEVALUATE
With many programs and interventions, an important factor for sustainability is theavailability of funding. Aside from collaborating with community health centers and their
grant writers, who help in attaining continual funding, the navigation curriculum wasconverted into a community college course where many navigators received training. Creatinga standardized training was useful in the move to certify navigators and establish mechanisms
for reimbursement for navigation services. 42 Additionally, the team used a task list to ensure
fidelity of program delivery. This list included tasks required across the cancer care continuum
related to evaluation and quality assurance of intervention implementation. 42

LESSONS LEARNEDCreating meaningful partnerships for successful program
implementation and thinking ahead about sustainability canseem like a daunting task. However, the one lesson that theteam came to understand is that when you are proactive insharing the vision with everyone on your team, unexpectedresources and people power will subsequently present
themselves. When your team shares that vision of program
implementation, the tasks become less daunting.

Share yourvision with everyone
on your team.

Implementation Science at a Glance

36

Tailored Communication for Cervical Cancer Risk
While overall rates of cervical cancer have decreased in the United States, racial and ethnic
minorities face a greater incidence of and death from the disease. 41 The most widely usedscreening for cervical cancer is the Pap test. Women who receive an abnormal Pap test result
are referred for follow-up testing (colposcopy). 41 Low rates of follow-up after an abnormal
Pap test result may contribute to the higher incidence of cervical cancer among low-income,minority women. Interventions that target barriers faced by low-income minority women are
essential to redress cervical cancer.
Tailored Communication for Cervical Cancer Risk is a telephone counseling intervention
developed by Fox Chase Cancer Center. The intervention targeted at-risk women who received
an abnormal Pap test result and were scheduled for follow-up testing. Two to four weeks
prior to their colposcopy appointment, interviewers called each woman and, using a scripted
questionnaire, identified barriers to follow-up and provided tailored counseling messages.
The messages address these barriers and encouraged women to attend an initial colposcopy
appointment and, six and twelve months later, repeat Pap test and colposcopy appointments.

ASSESS
Delivering this intervention over the telephone leveraged readily accessible resources.

PREPAREIn advance of the project, Fox Chase Cancer Center created a scripted questionnaire to ensurethat the intervention was delivered as planned.
Fox Chase established partnerships with health care providers—particularly those inleadership positions. By working with nurse managers, they learned about the clinic workflow
and how best to manage nurses’ competing priorities during intervention implementation.
Additionally, their community engagement approach allowed them to identify different
populations’ perceived notions of the best times to call, how often to call, and the most
relevant and appropriate counseling messages to deliver.

IMPLEMENT
Many strategies supported the program’s successful implementation. For example, Fox Chase
engaged a new clinical team to conduct recruitment and deliver the intervention at an offsite
facility. This ensured that there was dedicated staff trained to deliver the intervention andthat support was available. Insights from community members and their review of the tailoredmessages were also key in ensuring the intervention fit the target population.

37

Implementation Science at a GlanceEVALUATE
The evaluation found the intervention to be successful in both increasing the number ofwomen who attended an initial colposcopy appointment as well as longer-term medical followup. 41 Integrating the intervention into the standard care practice ensured its uptake andsustainability. Hiring designated staff to deliver the intervention and engaging stakeholders
throughout the process further ensured program fidelity.

LESSONS LEARNEDProgram director Suzanne Miller reflected that involving
research staff, clinic leadership, and community members
from the onset supported their success. “It will be exhausting
and time consuming, but that is what is going to set you up
for success.”

Have all stakeholders
involved on day one.

Implementation Science at a Glance38

LIVESTRONG® at the YMCA
Physical activity plays an essential role in enhancing the length and quality of life for cancer
survivors. Aside from cancer-specific outcomes, the benefits of exercise include increased
flexibility and physical functioning as well as improvements in patient-reported outcomessuch as fatigue.
The LIVESTRONG at the YMCA program was developed to improve the well-being of adultcancer survivors following a cancer diagnosis. 43 The twelve-week physical activity program
included two ninety-minute personalized exercise sessions per week, delivered in a small,supportive environment.

ASSESSLIVESTRONG at the YMCA began as an evidence-informed physical activity programthat drew from studies showing that physical activity is safe for and beneficial to cancer
survivors. 43 Since then, LIVESTRONG at the YMCA has had wide-reaching impact and has
shown to improve the physical activity, fitness, quality of life, and reduce cancer-related
fatigue in its participants. 44

PREPARE
Although LIVESTRONG at the YMCA has spread across 245 different Y associations, the program
has maintained its centralized vision and program goal. Core components of the program—afunctional assessment at the beginning and end of the program and the delivery of the programby a certified YMCA instructor—are consistently implemented across the local Ys.
YMCA-certified instructors are usually existing staff, external individuals, past participants,
or volunteers who meet the certification requirements. The flexibility of who delivered theprogram as well as the required training has allowed for the adoption of this program in 700
communities across the nation.
One of the challenges that LIVESTRONG at the YMCA faced is that its onsite mode of delivery
may make access to the program difficult for some YMCAs. YMCAs that did not have thecapacity to host the program due to a lack of trained staff or available exercise equipment wereable to sign on to an agreement with larger YMCAs to help build their readiness to implement.
Program directors at the Y attributed their continued success to meaningful partnerships they
formed and nurtured with LIVESTRONG. LIVESTRONG provides not only financial support but
also additional services such as patient navigation and resource books. The Y integrated these
services into their usual operations.

39

Implementation Science at a Glance

IMPLEMENT
The national partnership between LIVESTRONG and YMCA is enhanced by strong local
partnerships. One key implementation strategy was a six-month capacity-building training
required prior to program delivery. A key aspect of this training focused on helping local Ys
collaborate with their community’s health system. These local-level partnerships engaged the Y
with cancer survivors, health care providers, and patients, expanding the reach of the program.

EVALUATE
The goal of LIVESTRONG at the Y is to reach 100,000 cancer survivors by the year 2022.
Participating YMCAs regularly collect data to track their progress and program delivery. The
YMCA will soon launch a centralized reporting system to help with consistent evaluations
across the Ys.

LESSONS LEARNED
When asked to reflect on the factors that
contribute to the spread and uptake of
LIVESTRONG at the YMCA, program directors
and practitioners credited the time spent in
preparation and in giving organizations thetime to build the program. Investing in thetime necessary to identify local staff andpartners, develop a partnership pathway, andsustain meaningful relationships was central
to their success.

You cannot undervalue
the laying of groundwork
and giving organizations thetime to build.

Implementation Science at a Glance40Implementation Resources
for Practitioners
Here are some resources to help further your implementation efforts.
CANCER PREVENTION AND CONTROL RESEARCH NETWORK – PUTTING PUBLIC HEALTH
EVIDENCE IN ACTION TRAINING
An interactive training curriculum to teach community program planners and health educators
to use evidence-based approaches, including how to adapt programs.
http://cpcrn.org/pub/evidence-in-action/#
CANCER CONTROL P.L.A.N.E.T. (PLAN, LINK, ACT, NETWORK WITH EVIDENCE-BASED TOOLS)A portal that provides access to data and resources that can help planners, program staff, andresearchers design, implement, and evaluate evidence-based cancer control programs.
https://cancercontrolplanet.cancer.gov/planet
THE COMMUNITY GUIDEA searchable collection of evidence-based findings of the Community Preventive Services Task
Force. It is a resource to help select interventions to improve health and prevent disease in astate, community, community organization, business, health care organization, or school.
www.thecommunityguide.org
CONSOLIDATED FRAMEWORK FOR IMPLEMENTATION RESEARCH –TECHNICAL ASSISTANCE WEBSITE
A site created for individuals considering the use of this framework to evaluate an
implementation or design an implementation study.
http://cfirguide.org
DISSEMINATION AND IMPLEMENTATION MODELS IN HEALTH RESEARCH AND PRACTICE
An interactive database to help researchers and practitioners select, adapt, and integrate thedissemination and implementation model that best fits their research question or practice problem.
http://dissemination-implementation.org
EXPANDNET/WHO SCALING-UP GUIDE
Tools that provide a more comprehensive examination of scaling-up. Includes guides,
worksheets, briefs, and more.
http://expandnet.net/tools.htm
HEALTHY PEOPLE 2020 EVIDENCE-BASED RESOURCES
A searchable database with interventions and resources to improve the health of your community.
www.healthypeople.gov/2020/tools-resources/Evidence-Based-Resources

41

Implementation Science at a GlanceLIVESTRONG AT THE YMCA
A physical activity program developed by the YMCA and the LIVESTRONG Foundation.
The program assists those who are living with, through, or beyond cancer to strengthen
their spirit, mind, and body.
www.ymca.net/livestrong-at-the-ymca
NIH EVIDENCE-BASED PRACTICE AND PROGRAMS
A collection of several databases and other resources with information on evidence-based
disease prevention services, programs, and practices with the potential to impact public health.
https://prevention.nih.gov/resources-for-researchers/dissemination-and-implementationresources/evidence-based-programs-practices#topic-13
PARTNERSHIPS ENGAGING STAKEHOLDERS TOOLKITA set of resources to help program planners who are working in partnerships to improve policy.
www.pmc.gov.au/sites/default/files/files/pmc/implementation-toolkit-3-engaging-stakeholders.pdf
PEW-MACARTHUR RESULTS FIRST INITIATIVE
A resource that brings together information on the effectiveness of social policy programsfrom nine national clearinghouses.
https://www.pewtrusts.org/en/projects/pew-macarthur-results-first-initiative
PROGRAM SUSTAINABILITY ASSESSMENT TOOL (PSAT)
A 40-question self-assessment that program staff and stakeholders can take to evaluate thesustainability capacity of a program. Use the results to help with sustainability planning.
https://sustaintool.org
REACH, EFFECTIVENESS, ADOPTION, IMPLEMENTATION, AND MAINTENANCE (RE-AIM)
FRAMEWORK
Resources and tools for those wanting to apply the RE-AIM framework. Includes planning tools,
calculation tools, measures, checklists, visual displays, figures, an online RE-AIM module, and more.
http://www.re-aim.org
A REFINED COMPILATION OF IMPLEMENTATION STRATEGIES: (ERIC) PROJECT
A list of strategies you can use to implement your program.
https://implementation-science.biomedcentral.com/articles/10.1186/s13012-015-0209-1
RESEARCH-TESTED INTERVENTION PROGRAMS
A searchable database with evidence-based cancer control interventions and programs
specifically for program planners and public health practitioners.
https://rtips.cancer.gov/rtips/index.do
UNITED NATIONS OFFICE ON DRUGS AND CRIME (UNODC) EVALUABILITY ASSESSMENTS
A template to examine whether your program can be evaluated in a reliable and credible way.
www.unodc.org/documents/evaluation/Guidelines/Evaluability_Assessment_Template.pdf

Implementation Science at a Glance42

Glossary of Terms
ADAPTATION – The degree to which an evidence-based intervention is changed to suit theneeds of the setting or the target population. 8,13
ADOPTION – A decision to make full use of an innovation, intervention, or program as thebest course of action available. Also defined as the decision of an organization or community
to commit to and initiate an evidence-based intervention.13
COMMUNITY-BASED PARTICIPATORY RESEARCH (CBPR) – A collaborative approach toresearch that equally involves all partners in the research process and recognizes the uniquestrengths that each brings. CBPR begins with a research topic of importance to the community
and aims to combine knowledge with action to drive social change to improve healthoutcomes and eliminate health disparities. 45
CONSTRUCTS – Concepts developed or adopted for use in a theory. The key concepts of agiven theory are its constructs. 46
DE-IMPLEMENTATION – Reducing or stopping the use of a guideline, practice, intervention, or
policy in health care or public health settings. 47
DISSEMINATION SCIENCE – The study of targeted distribution of information and intervention
materials to a specific public health or clinical practice audience. The intent is to understandhow best to spread and sustain knowledge and the associated evidence-based interventions. 48
EVIDENCE-BASED INTERVENTION – Health-focused intervention, practice, program, or
guideline with evidence demonstrating the ability of the intervention to change a healthrelated behavior or outcome. 4
FIDELITY – Degree to which an intervention or program is implemented as intended by thedevelopers and as prescribed in the original protocol. 8,9IMPLEMENTATION OUTCOMES – The effects of deliberate and purposive actions to implement
new treatments, practices, and services. Implementation outcomes may include acceptability,
feasibility, adoption, penetration, appropriateness, cost, fidelity, and sustainability. 26
IMPLEMENTATION SCIENCE – The study of methods to promote the adoption and integrationof evidence-based practices, interventions, and policies into routine health care and publichealth settings to improve the impact on population health.
IMPLEMENTATION STRATEGIES – Methods or techniques to enhance the adoption,
implementation, and sustainability of a program or practice. 21,49
KNOWLEDGE SYNTHESIS – A process for obtaining and summarizing scientifically derived
information, including evidence of effectiveness (risk and protective factors, core components,
and key features, etc.).18

43

Implementation Science at a Glance

KNOWLEDGE TRANSLATION – The process of converting scientific and technically complex
research into everyday language and applicable actionable concepts in the practice setting.18
REACH – The absolute number, proportion, and representativeness of individuals who
participate in a given initiative or receive a specific intervention. 26
SCALE-UP – Deliberate efforts to increase the spread and use of innovations successfully
tested in pilot or experimental projects to benefit more people and to foster policy andprogram development. 38
SUSTAINABILITY – The continued use of program components and activities to achieve
desirable outcomes. 35
SERVICE EFFECTIVENESS OUTCOMES – Intervention results examined at the system level,
including efficiency, safety, effectiveness, equity, patient-centeredness, and timeliness. 26,50

Implementation Science at a Glance

44References1.

About Implementation Science (IS). National Cancer Institute: Division of Cancer Control &Population Sciences. https://cancercontrol.cancer.gov/IS/about.html. March 8, 2018.

2. Cabinet Implementation Unity Toolkit: Engaging Stakeholders. Australian Government.
https://www.pmc.gov.au/sites/default/files/files/pmc/implementation-toolkit-3-engagingstakeholders.pdf. June 2013.
3. Identifying and managing internal and external stakeholder interests. HealthKnowledge.
https://www.healthknowledge.org.uk/public-health-textbook/organisationmanagement/5b-understanding-ofs/managing-internal-external-stakeholders.
4. Chambers DA, Vinson CA, Norton WE. Advancing the Science of Implementation Across theCancer Continuum. New York, NY: Oxford University Press; 2019.
5. Brownson RC, Fielding JE, Maylahn CM. Evidence-based public health: a fundamental
concept for public health practice. Annu Rev Public Health. 2009;30:175-201. doi:10.1146/
annurev.publhealth.031308.100134.
6. Session 1: Defining Evidence. Putting Public Health Evidence in Action Training Workshop.
Cancer Prevention and Control Research Network (CPCRN). http://cpcrn.org/pub/evidencein-action/. November 2017.
7.

Satterfield JM, Spring B, Brownson RC, et al. Toward a transdisciplinary model of evidencebased practice. Milbank Q. 2009;87(2):368-390. doi:10.1111/j.1468-0009.2009.00561.x.

8. Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary fordissemination and implementation research in health. J Public Health Manag Pract.
2008;14(2):117-123. doi:10.1097/01.PHH.0000311888.06252.bb.
9. Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of
implementation: implications for drug abuse prevention in school settings. Health Educ Res.
2003;18(2):237-256.
10. Durlak, JA, DuPre, EP. Implementation Matters: A Review of Research on the Influenceof Implementation on Program Outcomes and the Factors Affecting Implementation.
American Journal of Community Psychology. 2008; 41(3-4):327-50. doi:10.1007/s10464-0089165-0.
11. Evidence-Based Practices (Part 2): Implementing a Practice or Program with Fidelity. The
IRIS Center. https://iris.peabody.vanderbilt.edu/module/ebp_02/. 2014.
12. Session 5: Adapting an Evidence-based Intervention to Fit Your Community. Putting Public
Health Evidence in Action Training Workshop. Cancer Prevention and Control Research
Network (CPCRN). http://cpcrn.org/pub/evidence-in-action/. November 2017.

45

Implementation Science at a Glance13. Rogers EM. Diffusion of Innovations. New York, NY: Simon & Schuster; 2010.
14. Chambers DA, Norton WE. The adaptome: advancing the science of intervention
adaptation. Am J Prev Med. 2016;51(4)(suppl 2):S124-131. doi:10.1016/j.amepre.2016.05.011.
15. Glanz, K. & Rimer, B.K. (2005). Theory at a glance: A guide for health promotion practice.
National Cancer Institute. National Institutes of Health. US Department of Health andHuman Services. NIH Pub. No. 05-3896. Washington, DC: NIH.
16. Rogers, EM. Diffusion of innovation. New York, NY: Free Press of Glencoe; 1962.
17. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering
implementation of health services research findings into practice: a consolidated
framework for advancing implementation science. Implement Sci. 2009;4:50.
doi:10.1186/1748-5908-4-50.
18. Understanding the Interactive Systems Framework for Dissemination and Implementation.
Applying Science. Advancing Practice. (ASAP). https://www.cdc.gov/violenceprevention/
pdf/ASAP_Issue-a.pdf.
19. Wandersman A, Duffy J, Flaspohler P, et al. Bridging the gap between prevention researchand practice: the interactive systems framework for dissemination and implementation.
Am J Community Psychol. 2008;41(3-4):171-181. doi:10.1007/s10464-008-9174-z.
20. Chambers DA. The interactive systems framework for dissemination and implementation:enhancing the opportunity for implementation science. Am J Community Psychol. 2012;50(34):282-284. doi:10.1007/s10464-012-9528-4.
21. Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual
distinctions, measurement challenges, and research agenda. Adm Policy Ment Health.
2011;38(2):65-76. doi:10.1007/s10488-010-0319-7.
22. Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies:
results from the Expert Recommendations for Implementing Change (ERIC) project.
Implement Sci. 2015;10:21. doi:10.1186/s13012-015-0209-1.
23. Waltz TJ, Powell BJ, Matthieu MM, et al. Use of concept mapping to characterize
relationships among implementation strategies and assess their feasibility andimportance: results from the Expert Recommendations for Implementing Change (ERIC)
study. Implement Sci. 2015;10:109. doi:10.1186/s13012-015-0295-0.
24. Powell BJ, Beidas RS, Lewis CC, et al. Methods to improve the selection and tailoring of
implementation strategies. J Behav Health Serv Res. 2017;44(2):177-194. doi:10.1007/s11414015-9475-6.
25. Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide.
Centers for Disease Control and Prevention (CDC). https://www.cdc.gov/eval/guide/
introduction/index.htm. May 11, 2012.

Implementation Science at a Glance4626. Proctor EK, Powell BJ, McMillen JC. Implementation strategies: recommendations for
specifying and reporting. Implement Sci. 2013;8:139.
27. Tabak RG, Khoong EC, Chambers DA, Brownson RC. Bridging research and practice:models for dissemination and implementation research. Am J Prev Med. 2012;43(3):337-350.
doi:10.1016/j.amepre.2012.05.024.
28. Petersen D, Taylor EF, Peikes D. The Logic Model: The Foundation to Implement, Study, andRefine Patient-Centered Medical Home Models. Rockville, MD: Agency for Healthcare Research
and Quality; 2013.
29. Brownson RC, Colditz GA, Proctor EK, eds. Dissemination and Implementation Research in
Health: Translating Science to Practice. Oxford; New York: Oxford University Press; 2018.
30. Evaluability Assessment Template. United Nations Office on Drugs and Crime. https://www.unodc.org/documents/evaluation/Guidelines/Evaluability_Assessment_Template.pdf.
January 26, 2012.
31. RE-AIM Resources and Tools. Reach Effectiveness Adoption Implementation Maintenance.
http://www.re-aim.org/resources-and-tools/.
32. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impactof health promotion programs: using the RE-AIM framework to form summary measuresfor decision making involving complex issues. Health Educ Res. 2006;21(5):688-694. doi:
10.1093/her/cyl081.
33. Five-Part Webcast on Economic Evaluation. Centers for Disease Control and Prevention(CDC). https://www.cdc.gov/dhdsp/evaluation_resources/economic_evaluation/index.htm.
April 26, 2017.
34. Understand Sustainability. Program Sustainability Assessment Tool.
https://www.sustaintool.org/understand/#fundingstability.
35. Scheirer MA, Dearing JW. An agenda for research on the sustainability of public healthprograms. Am J Public Health. 2011;101(11):2059-2067. doi:10.2105/AJPH.2011.300193.
36. Schell SF, Luke DA, Schooley MW, et al. Public health program capacity for sustainability: a
new framework. Implement Sci. 2013;8:15. doi:10.1186/1748-5908-8-15.
37. Willis, Cameron D., et al. "Scaling up complex interventions: insights from a realist
synthesis." Health Research Policy and Systems, vol. 14, no. 1, 2016. Academic OneFile,
Accessed 19 Feb. 2019.
38. Simmons R, Fajans P, Ghiron L. Scaling up health service delivery: from pilot innovations topolicies and programmes. Geneva, Switzerland: World Health Organization; 2007.
39. Worksheets for developing a scaling-up strategy. ExpandNet/World Health Organization.
http://www.expandnet.net/PDFs/ExpandNet-WHO%20Worksheets%20-%20July%202012.pdf.

47

Implementation Science at a Glance40. McKay VR, Morshed AB, Brownson RC, Proctor EK, Prusaczyk B. Letting go: conceptualizing
intervention de-implementation in public health and social service settings. Am J
Community Psychol. 2018;62(1-2):189-202. doi:10.1002/ajcp.12258.
41. Tailored Communication for Cervical Cancer Risk. National Cancer Institute (NCI). https://
rtips.cancer.gov/rtips/programDetails.do?programId=26767713. May 23, 2018.
42. Miller SM, Hui SK, Wen KY, et al. Tailored telephone counseling to improve adherence tofollow-up regimens after an abnormal pap smear among minority, underserved women.
Patient Educ Couns. 2013;93(3):488-495. doi:10.1016/j.pec.2013.08.008.
43. Heston AH, Schwartz AL, Justice-Gardiner H, Hohman KH. Addressing physical activity
needs of survivors by developing a community-based exercise program: LIVESTRONG® at
the YMCA. Clin J Oncol Nurs. 2015;19(2):213-217. doi:10.1188/15.CJON.213-217.
44. Irwin ML, Cartmel B, Harrigan M, et al. Effect of the LIVESTRONG at the YMCA exercise
program on physical activity, fitness, quality of life, and fatigue in cancer survivors. Cancer.
2017;123(7):1249-1258. doi:10.1002/cncr.30456.
45. Program goals and competencies. Community Health Scholars website.
http://www.kellogghealthscholars.org/about/community.php#competencies.
46. What are constructs? National Cancer Institute: Division of Cancer Control & Population Sciences.
https://cancercontrol.cancer.gov/brp/research/constructs/what-are-constructs.html.
April 28, 2008.
47. Norton WE, Kennedy AE, Chambers DA. Studying de-implementation in health: an analysis
of funded research grants. Implement Sci. 2017;12(1):144. doi:10.1186/s13012-017-0655-z.
48. Dissemination and implementation research in health. National Institutes of Health website
2017; https://grants.nih.gov/grants/guide/pa-files/PAR-18-007.html.
49. Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing
clinical innovations in health and mental health. Med Care Res Rev. 2012;69(2):123-157.
doi:10.1186/1748-5908-8-139.
50. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation
research in mental health services: an emerging science with conceptual, methodological,
and training challenges. Adm Policy Ment Health. 2009;36(1):24-34. doi:10.1007/s10488-0080197-4.

Implementation Science at a Glance48

NIH Publication Number 19-CA-8055
49

Implementation Science at a Glance

Implement