SEPTEMBER 2014

USING EVIDENCE-BASED CONSTRUCTS TO 

ASSESS EXTENT OF IMPLEMENTATION OF 

EVIDENCE-BASED INTERVENTIONS

ABOUT THIS ISSUE 

BRIEF

This issue brief was written by

Gene Hall, Ph.D., of the 

University of Nevada Las Vegas;

Allison Dymnicki, Ph.D., of the 

American Institutes for 

Research; Jennifer Coffey, 

Ph.D., of the Office of Special 

Education Programs, and 

Melissa Brodowski, Ph.D., of the 

Children’s Bureau. 

In 2012, ASPE awarded the 

American Institutes for Research 

to manage the Investing in What 

Works (IWW) project, to 

continue ASPE’s efforts to keep 

building the knowledge and 

supports that evidence-based 

programs and initiatives can use 

to improve the quality and 

outcomes of interventions funded 

through federal investments. 

Office of the Assistant Secretary 

for Planning and Evaluation

Office of Human 

Services Policy

US Department of Health 

and Human Services

Washington, DC 20201 

This issue brief was developed under contract with ASPE (Contract #: HHSP23320095626SWC).

Executive Summary

Implementing interventions in communities and organizations 

and with individuals requires an understanding of the 

complexities involved with making any major change. A good 

idea and evidence that an intervention has been effective 

elsewhere do not automatically lead to the widespread adoption 

and use of an intervention. Studying the implementation of 

interventions has increased our understanding of how to monitor 

more effectively what happens during implementation to 

promote success. 

In this brief, we introduce several constructs that can be used to 

describe, monitor, and facilitate implementation. Then we 

describe how to apply these constructs in several ways, 

including data collection to monitor progress and developing an 

implementation monitoring plan. We also address how much 

time each phase of implementation is expected to take. The brief 

concludes with implications for policy and for supporting 

practice. 

Key Take Away-Messages

 Scholars who study change have identified and categorized

implementation milestones common to most interventions.

These can be used to improve the likelihood of achieving

full implementation of interventions.

 Policymakers could require plans for identifying

implementation milestones, monitoring progress, and using

data to inform continuous quality improvement during

implementation.

 Practitioners should assess implementation progress by

collecting a variety of indicators from different

perspectives (e.g., frontline staff, leadership) to pinpoint

barriers to implementation and ways to overcome them.
 Policymakers evaluating intervention implementation should be realistic about what can be 

done within certain time frames, but they should also encourage grantees to move 

thoughtfully and quickly through the first phase, which is often overlooked. 

What Is the Change Process?

One important aspect of understanding change is that it is a process, not an event. An 

intervention1

1 In other related work, we refer to the terms “innovation” versus interventions. In this brief, we define the term “intervention” as: (a) a 

procedure, or set of procedures (b) designed for use in a specific context (or set of contexts) (c) by a specific set of users (d) to achieve 

defined outcomes (e) for (a) defined population(s) (c.f. Cook, Tankersley, & Landrum, 2009; Dunst, Trivette, & Cutspec, 2002; Flay et 

al., 2005; Horner, Sugai & Anderson, 2010). Interventions are what we do to achieve desired outcomes. 

 involves three major phases of change: 

1. Creating includes the research and development activities that provide the knowledge 

base for developing interventions, programs, and innovations. This phase also includes 

establishing strategies for making interventions available for use, assessing readiness, and

scaling up. 

2. Implementing is when individuals, agencies, or communities apply the new intervention, 

program, technology, or method. 

3. Sustaining is when the entire intervention, or select components of it, becomes the 

established way of doing things; use of the intervention has been “institutionalized.” 

Figure 1 illustrates how these three phases unfold. It also illustrates several ideas important to 

understanding the change process in general and implementation in particular. One especially 

important idea in figure 1 is the Giant Leap. Sometimes, there is an implicit expectation that 

accomplishing a change will be simple and quick, that practitioners who are implementing an 

intervention need only to jump across the gap and start using the new way. This perception does 

not accurately represent the complexity of how change occurs in organizations, in communities, 

or with individuals. Also, most interventions require implementing a set of practices and 

structures simultaneously. A common misunderstanding is the expectation that all practitioners 

in an organization or large system make the Giant Leap at the same time.

Figure 1. Three Phases of the Change Process and the Implementation Bridge 

Source: Hall & Hord (in press). 

ASPE ISSUE BRIEF | 2 

 
Figure 1 represents another idea about change: the size of the gap and the distance that must be 

covered in one leap. Most expectations for implementing new interventions require leaps too 

long for many to achieve. Change initiatives will be more successful if there is an 

Implementation Bridge. As a metaphor, the Implementation Bridge can help us examine the 

activities, challenges, and barriers involved in achieving successful implementation. For 

example, each practitioner who implements an intervention has to make it across the bridge. As 

practitioners move across the bridge, they gain knowledge, skills, and confidence in using the 

new intervention. Individuals and organizations also need ways to measure, monitor, and assess 

how far across the Implementation Bridge an intervention has progressed. These steps and the 

information gathered during these steps are important to inform further progress, identify where 

additional support is needed, gauge the effectiveness of an intervention to produce desired 

outcomes, and identify unintended consequences. The effort and level of support required during 

the implementation phase will vary dramatically based on characteristics of the change and the 

context in which it is introduced. 

How To Measure Implementation Milestones 

Several constructs can be used to monitor progress in implementing interventions: 

Fidelity of Implementation—Today’s interventions are complex and have many parts, 

which can lead to varying aspects of the intervention being implemented at different times.

Competence in Use—The expertise and competence of practitioners carrying out an 

intervention increase during the implementation process. 

Feeling and Perceptions—The personal side to the change process is reflected in 

practitioners’ increasing confidence in or resistance to implementing an intervention. 

Context of the Organization and Community—Implementation takes place within a

broader context that can support or impede implementation efforts. 

Supporting Implementation—Addressing factors that promote the effective 

implementation of interventions can advance its progress. 

In this section, we describe these constructs and demonstrate how each can be used to monitor 

implementation. 

Fidelity of Implementation: To what extent are the core content, pedagogical, 

and implementation components of an intervention being implemented? 

One set of implementation milestones involves Fidelity of Implementation. “Fidelity is the 

faithfulness with which an intervention is implemented; that is, how well the intervention is 

implemented without compromising its core content, pedagogical, and implementation 

components which are essential for the intervention’s effectiveness” (National Resource Center 

for HIV/AIDS Prevention Among Adolescents, n.d.). Implementing an intervention with fidelity 

improves the likelihood of replicating positive intervention effects (see the Administration for 

Children and Families, n.d., for more information). 

ASPE ISSUE BRIEF | 3 
One way to think about and measure fidelity is to identify the major components of a change. In 

a recent research brief developed by the Office of the Assistant Secretary of Planning and 

Evaluation titled Core Intervention Components: Identifying and Operationalizing What Makes 

Programs Work, Blase & Fixsen (2013) described core components as “the essential functions or 

principles, and associated elements and intervention activities (e.g., active ingredients, 

behavioral kernels) that are judged necessary to produce desired outcomes” (Embry, 2004, p. 2). 

Century, Cassata, Rudnick, and Freeman (2012) have proposed a framework for measuring the 

“enactment of innovations,” which includes identifying the characteristics of fidelity. In these 

works there is an expectation that practitioners develop fidelity gradually with practice instead of 

suddenly halting what they are doing and fully implementing a new approach. 

An important first step when implementing an intervention is to establish what achieving fidelity 

will look like for that intervention and then establish what indicators will suggest progress in 

achieving fidelity. This involves identifying the critical components of an intervention, and then 

identifying steps during the implementation process that practitioners are likely to take as they 

move toward implementing intervention practices. 

We recognize that at any point in time there will be variation in how well each component of the 

innovation is being used. For example, in a counseling program the assessment component might 

be implemented very well, but the feedback sessions leave key elements left out, suggesting 

partial fidelity of the whole program. This can be described in terms of different “Innovation 

Configurations” (Hall & Loucks, 1981; Hall & Hord, 2015), where each configuration represents 

a different combination of the components. As fidelity increases, more and more core 

components are included and are used as intended. In the final, "ideal" configuration, all critical 

components are in use and full Fidelity of Implementation has been achieved. 

Important questions about fidelity must be addressed for each change initiative. How will the key 

components of the change be identified? Who will monitor the extent to which core components 

have been established and are being performed consistently? What data collection method will 

be used? How much information is needed? How frequently should fidelity milestones be 

measured? What standards will be used to interpret what is found? These are challenging 

decisions that have to be answered on a case-by-case basis; answers depend on the particular 

change being introduced. For example, Chinman et al. (2005) addressed some of these questions 

in their work about building community capacity for effective preventive interventions, including 

how to identify core components. 

Developing Competence: How do practitioners progress from nonusers to 

expert users?

Another set of implementation milestones describes practitioners’ increasing competence in 

using an intervention. Studies conducted from the practitioners’ perspective suggest that 

developing the knowledge and skill to implement an intervention effectively requires time and 

experience (Bellg et al., 2004; Durlak & DuPre, 2008). As practitioners approach and move 

across the bridge, different levels of expertise can be described:

ASPE ISSUE BRIEF | 4 
Nonuser—Is not engaged in using the change and is using the “old” way. (In other words, is 

not on the bridge.) 

Novice—Is applying the change but struggling to use all pieces in effective ways. (Is on the 

bridge.) 

Expert—Has mastered the complexity and understands everything entailed in full use. (Is 

across the bridge.) 

Change process researchers have identified eight unique behavioral profiles, called “Levels of 

Use” (LoUs), which include three behavioral profiles of nonusers and five profiles of users (Hall 

& Hord, 2015). Each of these profiles describes a different level of understanding and degree of 

skill in using (or not using) an intervention. The LoUs range from the nonuser, who knows 

nothing and is taking no action to learn about an intervention (LoU 0, Nonuse), to the novice, 

whose use is inefficient and disjointed (LoU III, Mechanical Use), to the skilled expert who 

makes deliberate efforts to coordinate with others in using the intervention (LoU V, Integration). 

LoUs can be used to identify several important implementation milestones and to ensure that 

progress continues toward full implementation and sustainability. For example, one very 

important milestone is LoU III Mechanical Use, in which practitioners have begun to implement 

certain practices but their use is disjointed, inefficient, and incomplete. During this phase, 

practitioners might plan lessons 1 day in advance, follow the user’s manual very closely, and 

possibly be uncomfortable with the technology involved in the program. 

These stages have important implications for training and technical assistance (TTA) providers, 

administrators, and evaluators. In the LoU III Mechanical Use stage, the TTA should focus on 

“how to do it,” not on philosophy or theory. Administrators should plan problem-solving 

meetings with staff to understand the challenges being encountered. Evaluators should be 

documenting the extent and duration of this stage and what facilitates progress. 

Developing Confidence: How do feelings, emotion, and motivation develop as 

interventions are being implemented?

Charting feelings and perceptions represents a third way to establish implementation milestones. 

Change science scholars have identified and described the different concerns that people can 

have when engaged with a change process; these can be used to chart movement across the 

Implementation Bridge. Four major categories of concern have been identified (Cheung, Hattie, 

& Ng, 2001; George, Hall, & Stiegelbauer, 2008; Hall & Hord, 2015; Yan & Huang, 2008):

Unconcerned—At the beginning of a change process, practitioners will not be concerned 

about an intervention; they will be more concerned about other things. “I think I heard 

something about it, but I'm too busy right now with other priorities to be concerned about it.” 

Self—As practitioners get close to implementation, concerns about what the change could 

mean for them personally become intense. “I don’t know whether I can do this.” 

Task—After implementation has begun, concerns about time, organizing tasks, and fitting 

everything in become intense. “I have to spend a lot of time preparing to use this tomorrow.” 

ASPE ISSUE BRIEF | 5 
Impact—Ultimately, as practitioners become more familiar with the intervention, they have 

more intense concerns about how well the intervention is working for clients and what can be 

done to produce even better outcomes. “I just heard about this app that might be useful with 

several clients who don’t seem to be making good progress.” 

Different TTA and coaching are needed for each area of concern. For example, consider the 

resistance often experienced during change processes that can occur at any time. The source of 

the resistance will vary by concern. For example, Self-Concerns (e.g., “Well, I don’t see how this 

is better than what I have been doing) can lead to disagreement with engaging the change at all. 

When overwhelmed by complexities (Task Concerns, e.g., “It takes too much time to get ready 

and nothing works out as it is supposed to”), the resistance will have a different theme. Table 1 

presents examples of different types of concern-based resistance that might occur and some 

possible resolutions. Understanding the sources of resistance helps inform the development of 

more targeted solutions. 

Table 1. Examples of Resistance and Solutions for Difference Stages of Concern

Concern Source of Resistance Solution

Unconcerned “I don’t think this intervention will 

last.”

Document leadership’s long-term commitment to 

the intervention. 

Self “I don’t know whether I can do 

this. Will it be worth the effort?”

Describe the supports that will be provided and the 

benefits to be gained.

Task “This is taking too much time 

away from what’s important.”

Provide how-to-do-it TTA; simplify schedules and 

structures; provide joint planning and problemsolving time for staff.

Impact “This new way is terrific, but it’s 

not working for certain 

populations.” 

Target TTA and coaching to discuss how the 

intervention can reach these populations.

Organization and Community Context: What system factors are key to 

implementation success?

Another important consideration is the broader context in which an intervention is being 

implemented. Different aspects of the context can support or inhibit successful implementation. 

One aspect of context is the “unit of change.” The primary implementer can be an individual 

(e.g., a case manager in a foster care agency), an organization (all staff in a community-based 

organization), or a whole community (city or state). It is useful to monitor progress in 

implementing interventions when considering whole-organization and systemic change. The 

same three constructs (fidelity, competence, and concerns) can be applied to the whole system as 

well as the various subgroups. The same considerations regarding TTA, coaching, and assessing 

movement during implementation apply, but now it is to groups and whole communities. 

ASPE ISSUE BRIEF | 6 
Figure 2: Understanding Practitioners in Their Broader Context 

Source: Goodman, 2011.

Implementation will be more successful when there is clear and consistent support for using an 

intervention across the system. The plan for monitoring implementation should include 

measuring how well the system is supporting its use. Figure 2 illustrates an education system in 

one state, but the concept behind the graphic can be applied to any state system because most 

large-scale change efforts should be supported at the organizational, local, and state levels, which 

can be considered units of change. Implementation progress needs to be monitored within each 

unit, and overall. When a change initiative is being led from the “top,” there is a “cascading” 

relationship in which each unit of change supports the next. In addition, needs and barriers are 

expressed to the next unit (e.g., the building team expresses its needs and barriers to the district 

team) so the right supports will be provided. Regular measures of fidelity, use, and concerns 

within each unit are important to maintain an informed, strategic view of progress and needs. 

Supporting Implementation: What supports can help advance progress in 

implementing interventions? 

Another use of implementation milestones is as diagnostics for determining which supports will 

advance the implementation of interventions. The idea of “implementation drivers” proposed by 

Fixsen, Naoom, Blase, Friedman, and Wallace (2005) represents one way to think about actions 

and processes to support implementation at different levels (organization, local, and state). Their 

model of support is based on a set of implementation drivers, which have three main areas of 

focus: (a) drivers that build competence (selection, training, coaching); (b) drivers that support 

ASPE ISSUE BRIEF | 7 
that competence through organizational structures (systems interventions, administrative 

facilitation, decision support data systems); and (c) leadership drivers (technical drivers such as 

time, and adaptive drivers like motivation; see http://nirn.fpg.unc.edu/learnimplementation/implementation-drivers). 

These drivers can be applied to each level of the system. For example, if state mental health 

providers are providing multisystemic therapy, they might rely on regional TTA providers, who 

in turn rely on state agency personnel to offer them the training, guidance, and facilitative 

policies that help them do their work. Staff at the state, regional, and local levels need 

appropriate training and coaching and the use of a data system (that can provide an assessment of 

fidelity, use, and concerns) to help them make decisions about how to change or continue to do 

their work. 

Collecting information about implementation milestones and context supports the diagnostic 

assessments that identify the implementation supports needed for each level, system, subunit, 

and individual. In combination, these constructs and perspectives provide a complex yet coherent 

understanding of what is involved in moving implementation forward. 

Implementation Milestones: Applying These Constructs

In this section, we explore the following questions: 

4. What data should be collected to determine progress across the Implementation Bridge?

5. What would an implementation monitoring plan look like? 

6. How much time does it take?

What data should be collected to determine progress across the 

Implementation Bridge?

Several evidence-based constructs have been introduced for thinking about and developing 

implementation milestones. Applying these milestones to an intervention involves (1) identifying 

core components (i.e., What will the practitioners be doing?); (2) projecting a timeline of the 

likely rate of movement across the LoUs; (3) anticipating the Stages of Concern and what might 

cause resistance; and (4) identifying TTA that will promote the use of critical components, 

address concerns as they arise, and support movement toward optimal implementation. All of 

these actions and steps can be aligned along a timeline. We present several suggestions regarding 

different types of data to collect and ways to use these data during the implementation process, 

including data about: 

 Practitioners’ use of different intervention practices to inform conversations about 

intervention practices that are and are not being implemented consistently, and why 

 Practitioners’ skills and knowledge of intervention practices, to understand their 

competence in different intervention practices and where additional TTA is needed 

 Practitioners’ personal concerns that arise during the implementation process, to pinpoint 

sources of resistance and possible solutions 

ASPE ISSUE BRIEF | 8 
 The organization and community where the intervention is being implemented, to 

understand how aspects of an organization or community support or inhibit the 

implementation of an intervention

What would an implementation monitoring plan look like?

Creating an implementation monitoring plan could involve tracking specific activities anticipated 

during each stage of implementation. To illustrate, table 2 presents sample implementation 

monitoring activities during a trial that implemented multidimensional treatment foster care 

(MTFC) at 3 sites in 51 counties (Chamberlain, Brown, & Saldana, 2011). Researchers involved 

in this work also conceptualized three phases of the change process similar to the three phases 

listed on page 3 of this brief. They collected information about a number of implementation 

activities associated with each phase, the amount of time associated with completing each 

implementation stage, and the percentage of activities completed. Table 2 provides readers with 

activities that could be included in an implementation monitoring plan for an evidence-based 

intervention in a human services setting.2

 

2 We acknowledge that the dynamic nature of change is not reflected in this table (e.g., there is not necessarily a linear relationship 

among phases, fidelity, and levels) and that all people in an organization or community might not be at the same level or stage.

ASPE ISSUE BRIEF | 9 

 
Table 2. Sample Implementation Activities Completed During Different Implementation Stages in a Multidimensional 

Treatment Foster Care Trial

Phase

Fidelity of 

Implementation

Levels of Use (Action 

Taking)

Stages of Concerns 

(Feelings/Perceptions) Sample Implementation Activities Completed

Creating Original/traditional 

practices

0. Nonuse: Doing 

nothing to learn 

about it.

Orientation: 

Exploring what it is.

Preparation: Deciding 

to use it. 

Unconcerned: About 

what might be 

coming. 

Informational: What is 

it?

Personal: What will it 

mean for me?

Date site is informed service/program is available.

Date of interest indicated.

Date agreed to consider implementation.

Date to first contact for preimplementation 

planning. 

Date of first in-person meeting/feasibility call.

Date Feasibility Questionnaire is completed.

Date of cost/funding plan review.

Date of staff sequence, timeline, hiring plan 

review.

Date of foster parent recruitment review.

Date written implementation plan is complete.

Date service provider is selected.

Implementing Partial/limited 

configurations 

moving toward 

configurations that 

include more/all of 

the intervention

Mechanical Use: 

Disjointed and 

inefficient, shortterm focus.

Task/management: It 

takes a lot of time.

Date agency checklist is completed.

Date first staff is hired.

Date Program Supervisor is trained. 

Date clinical training is held. 

Date foster parent training is held. 

Date data tracking system training is held.

Date of first program administrator call.

Date of first placement.

Date of first consult call.

Date first clinical meeting video is reviewed.

ASPE ISSUE BRIEF | 10
Phase

Fidelity of 

Implementation

Levels of Use (Action 

Taking)

Stages of Concerns 

(Feelings/Perceptions) Sample Implementation Activities Completed

Sustaining Full configuration—

full fidelity

Advanced/higher 

fidelity 

configurations 

IVA. Routine use: 

Stabilized and 

ongoing. 

IVB. Refinement: 

Making adaptations 

to increase 

outcomes. 

V. Integration: 

Teaming to use the 

intervention. 

Renewal: Looking 

for the next thing. 

Impact: These practices 

are successful with 

clients.

Impact: By working 

with my colleagues, 

we are having even 

greater success.

Impact: I wonder 

whether another 

way is needed now.

Date of first site visit.

Date of implementation review.

Date of first program assessment.

Date of certification application.

Date certified.

ASPE ISSUE BRIEF | 11
How much time does it take to move across the bridge?

The amount of time it takes to progress across the Implementation Bridge can vary greatly. Data 

from MTFC suggest that the creating and implementing phase could take 2 to 3 years, and other 

work suggests that the creating phase could take between 6 months and a year (Blase & Fixsen, 

2013). Implementing new programs, curricula, and reforms in schools takes 3 to 5 years (Hall & 

Hord, 2015). 

The time it takes to achieve implementation success is based on a number of factors, including 

(a) the initial readiness of an organization or setting (Dymnicki et al., 2014), (b) intervention 

complexity, (c) how new or different the intervention is from existing practices, (d) the resistance 

expressed by different stakeholders and sources of that resistance along the way, and (e) how 

valuable or beneficial people consider the intervention will be once it is being implemented. 

Information collected during the MTFC trial provides an example of the time needed to complete 

each stage (Chamberlain et al., 2011). Progression through the stages differed markedly by site. 

For example, counties in Site 1 spent almost 2 years in the creating stage but then moved quickly 

through the implementing stage in about a year before applying for and achieving certification. 

Counties in Site 2 moved more quickly through the creating stage (in about 8 months) but took 

about 4 years to implement and achieve certification. Counties in Site 3 discontinued 

implementation activities during the creating stage and skipped 7 of 13 recommended activities 

in that stage. Followup analyses found that sites were more likely to be successful in 

implementing MTFC if they progressed quickly and thoroughly completed activities in the 

creating stage (Saldana, Chamberlain, Wang, & Brown, 2012). 

Implications for Policy 

Policy can be related to identifying and monitoring implementation milestones in several ways: 

(a) Include expectations about the likely time and supports necessary to meet implementation 

milestones in funding announcements and interim project deliverables; (b) align performance 

management and continuous quality improvement (CQI) with implementation milestones; (c) 

Encourage funding for proactive and ongoing technical assistance; and (d) invest in support for 

intervention infrastructure.

Include Expectations To Meet Implementation Milestones in Funding 

Announcements and Interim Project Deliverables

Policymakers and funders can play an important role in fostering the successful implementation 

of interventions by supporting the use of implementation milestones as a key monitoring strategy 

across the life cycle of a program. At the front end, funding announcements can outline 

(a) expectations around the exploration work that applicants must complete before submitting 

their applications and (b) specific activities and implementation milestones expected for each 

year of the grant. Some federal agencies have incorporated an initial planning year for grant 

funding, which culminates in the development of a detailed implementation plan to guide 

subsequent years’ activities. In other federal projects, after 2 years of funding, a panel of external 

ASPE ISSUE BRIEF | 12
reviewers hears evidence of a project’s outputs and preliminary outcomes and provides 

recommendations about activities or modifications to activities that will help the project move 

forward with implementation. A recommendation to terminate or continue funding is based on 

the project’s ability to incorporate these recommendations and make additional progress in 

implementation. In this way, continued funding depends on an approved implementation plan 

and the achievement of major milestones such as staff recruitment and training, identifying 

referral sources, establishing a Memorandum of Understanding with partner agencies, and 

identifying TTA resources. 

Align Performance Management and CQI Efforts With 

Implementation Milestones

Many funded interventions have required—and in some cases, legislatively mandated—that 

participant-level outcomes, benchmarks, and indicators be tracked. However, these rarely include 

implementation milestones. Adding implementation milestones to grant or contract requirements 

provides an opportunity to embed these measures into a comprehensive framework for assessing 

program performance. As different sectors develop data systems for performance management, 

policymakers and funders are in a position to require that such indicators be included and tracked 

by programs. Once data are available, it will be important for policymakers to provide incentives 

for programs to use their data and engage key stakeholders in ongoing conversations about the 

data. Information sharing with funders and others who support the program can be facilitated 

through practice-to-policy feedback loops. These data can also advance CQI efforts throughout 

the project period. CQI is the complete process of identifying, describing, and analyzing 

strengths and problems and then testing, implementing, learning from, and revising solutions 

(Children’s Bureau, 2012). Such data can also help distinguish between programs that do not 

work and sites that have implemented a potentially effective program inadequately. 

Encourage Funding for Proactive and Ongoing Technical Assistance

Policymakers need to be prepared to provide funding for TTA proactively and in response to 

identified implementation needs. Developing effective and usable implementation monitoring 

systems takes time. These systems need to carefully consider the specific context and larger 

system in which a program is being implemented. Policymakers understand that programs often 

experience barriers and challenges to achieving implementation milestones at various points 

during a program’s life cycle. This is part of the implementation process and should not be 

considered failure. Technical assistance can help programs reach critical milestones and also 

provide targeted consultation to identify the reasons that certain programs cannot achieve these 

milestones within a specific time frame. Some funders require remedial action or program 

improvement plans for projects that may need more time to reach the next milestone. For 

example, the Maternal, Infant, and Early Childhood Home Visiting Program legislation requires 

TTA for states that cannot measurably improve in required benchmark areas within 3 years. 

Invest in Systems Support for Interventions

Policymakers’ awareness that investments in evidence-based interventions must include support 

for implementing interventions will help ensure success. Differences in context, capacity, and 

readiness should be considered. Identifying and tracking implementation milestones in these 

ASPE ISSUE BRIEF | 13
settings will involve different timelines and resources. Settings that are less ready will need 

different starting points and development structures. Infrastructure activities might include 

(1) creating activities such as strategic planning and relationship building with partners; 

(2) implementation and delivery services and workforce supports; and (3) sustaining activities 

such as financing, communication, evaluation, and monitoring (Hargreaves, Cole, CoffeeBorden, Paulsell, & Boller, 2013). These costs are not insubstantial and should be factored into 

intervention budgets as part of initial startup and ongoing implementation costs. 

Implications for Supporting Practice

Several implications for supporting practice are related to identifying and monitoring 

implementation milestones: (a) the importance of engaging in planning activities early in 

implementation, (b) the importance of providing support for implementing an intervention, and 

(c) the importance of measuring infrastructure for implementation support.

Importance of Engaging in Planning Activities Early in Implementation

Sometimes, activities such as including stakeholders in planning and assessing the fit of an 

intervention, identifying the knowledge and skills needed, anticipating likely concerns, exploring 

the availability of resources to support the intervention, and building the infrastructure to support 

the intervention might seem like a waste of time and causes of delay. However, if such activities 

are absent, it is less likely that key milestones will be met later. Initial implementation is 

typically an awkward learning period for practitioners. Novice implementers need TTA support 

to resolve challenges that arise during implementation; they must also have the necessary time, 

energy, and support to incorporate intervention practices into their daily behaviors. Matching the 

right implementation supports to these stages and milestones can be encouraging and motivating. 

Importance of Providing Support for Implementing an Intervention

Implementation drivers can support implementation and provide information about the training, 

coaching, administrative facilitation, systems intervention, and performance assessment needed 

to remove barriers to progress. These drivers build staff competence through selection, training, 

and coaching; they assess performance; and they provide organizational support through datadriven decision making, facilitative administration, and systems interventions. For example, if 

offsite training is needed, it may be necessary for alternate staff members to substitute for those 

who are attending training. An administrator would need to facilitate this type of systems 

intervention. Leaders can support implementation by demonstrating how an intervention 

connects to the mission and vision of an organization and convincing staff of the organization’s 

long-term support of and commitment to the intervention. 

Importance of Measuring Infrastructure To Support Implementation

Measuring improvements in the infrastructure to support implementation can produce feedback 

that helps decision making and encourages those building the infrastructure to keep moving 

toward the next milestone (Blase & Fixsen, 2013). Each part of the system that supports the 

practitioner implementing an intervention should have the appropriate infrastructure to support 

ASPE ISSUE BRIEF | 14
the larger initiative. Information should be collected at multiple levels (e.g., organization, local, 

and state) of the support system. This enables one level of the system to prevent barriers that are 

impeding successful implementation from occurring at another level. 

Conclusion

Studying the implementation of interventions can help us understand the challenges and 

opportunities that arise during the implementation process. Several research-verified constructs, 

including Fidelity of Implementation, Levels of Use, and Stages of Concern, can be used to 

measure the progress of systems, communities, organizations, and practitioners as they 

implement interventions. Each offers a way to understand an important element of the change 

process. Determining milestones at regular intervals can help to identify appropriate supports and 

necessary resources. Milestones can also help practitioners understand where they are now and 

what is required to take the next steps. Charting milestones for different stages of 

implementation and measuring success in reaching those milestones can provide motivation to 

continue the work and promote more successful implementation of interventions. 

References

Administration for Children & Families (ACF). Fidelity monitoring tip sheet. Retrieved from

http://www.acf.hhs.gov/sites/default/files/fysb/prep-fidelity-monitoring-ts.pdf 

Bellg, A. J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D. S., Ory, M., et al. (2004). 

Enhancing treatment fidelity in health behavior change studies: Best practices and 

recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 

443. 

Blase, K., & Fixsen, D. (2013). Core intervention components: Identifying and operationalizing 

what makes programs work. ASPE Research Brief, Office of the Assistant Secretary for 

Planning and Evaluation, Office of Human Services Policy, United States Department of 

Health and Human Services. Retrieved from

http://aspe.hhs.gov/hsp/13/KeyIssuesforChildrenYouth/CoreIntervention/rb_CoreInterve

ntion.pdf

Century, J., Cassata, A., Rudnick, M., & Freeman, C. (2012). Measuring enactment of 

innovations and the factors that affect implementation and sustainability: Moving toward 

common language and shared conceptual understanding. The Journal of Behavioral 

Health Services and Research, 39(4), 343–361. 

Chamberlain, P., Brown, C. H., & Saldana, L. (2011). Observational measure of implementation 

progress: The Stages of Implementation Completion (SIC). Implementation Science, 6. 

Cheung, D., Hattie, J., & Ng, D. (2001). Reexamining the stages of concern questionnaire: A test 

of alternative models. The Journal of Educational Research, 94(4), 226–236. 

ASPE ISSUE BRIEF | 15
Children’s Bureau. (2012). Establishing and maintaining continuous quality improvement (CQI) 

systems in state child welfare agencies. Information Memorandum, ACYF-CB-IM-12-07.

Retrieved from http://www.acf.hhs.gov/sites/default/files/cb/im1207.pdf

Chinman, M., Hannah, G., Wandersman, A., Ebener, Hunter, S. B., Imm, P., et al. (2005). 

Developing a community science research agenda for building community capacity for 

effective preventive interventions. American Journal of Community Psychology, 35(3/4), 

143–157. 

Cook, B., Tankersley, M., & Landrum, T. (2009). Determining evidence-based practices in 

special education. Exceptional Children, 75(3), 365–383. 

Dunst, C. J., Trivette, C. M., & Cutspec, P. A. (2002). Toward an operational definition of 

evidence-based practices. Centerscope, 1(1), 1–10. Retrieved from

http://www.wbpress.com/index.php?main_page=popup_image&pID=369&zenid=2027ca

87d08ae7a6f6d5729a380e5389 

Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the 

influence of implementation on program outcomes and the factors affecting 

implementation. American Journal of Community Psychology, 41(3-4), 327–350. 

Dymnicki, A., Wandersman, A., Osher, D., Grigorescu, V., Huang, L. & Meyer, A. (2014). 

Willing, able→ ready: Basics and policy implications of readiness as a key component 

for implementation of evidence-based practices. ASPE Issue Brief, Office of the 

Assistant Secretary for Planning and Evaluation, Office of Human Services Policy, 

United States Department of Health and Human Services.

Embry, D. D. (2004). Community‐based prevention using simple, low‐cost, evidence‐based 

kernels and behavior vaccines. Journal of Community Psychology, 32(5), 575–591. 

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). 

Implementation research: A synthesis of the literature. Tampa, FL: University of South 

Florida, Louis de la Parte Florida Mental Health Institute, National Implementation 

Research Network. (FMHI Publication No. 231). 

Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). 

Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention 

Science, 6(3), 151–175.

George, A. A., Hall, G. E., & Stiegelbauer, S. M. (2008). Measuring implementation in schools: 

The stages of concern questionnaire. Austin, TX: Southwest Educational Development 

Laboratory.

Hall, G. E., & Hord, S. M. (in press). Implementing change: Patterns, principles and potholes 

(4th ed.). Upper Saddle River, NJ: Pearson. 

Hall, G. E., & Loucks, S. F. (1981). Program definition and adaptation: Implications for 

inservice. Journal of Research and Development in Education, 14(2), 46–58.

ASPE ISSUE BRIEF | 16
Hargreaves, M., Cole, R., Coffee-Borden, B., Paulsell, D., & Boller, K. (2013). Evaluating 

infrastructure development in complex home visiting systems. American Journal of 

Evaluation, 34, 147–169. 

Horner, R. H., Sugai, G., & Anderson, C. M. (2010). Examining the evidence base for schoolwide positive behavior support. Focus on Exceptional Children, 42(8), 1–14. Retrieved 

from

http://www.dropoutprevention.org/sites/default/files/horner_sugai_anderson_2010_evide

nce.pdf

National Resource Center for HIV/AIDS Prevention Among Adolescents. Promoting evidencebased approaches: Adaptation guidelines. Retrieved from

https://preventyouthhiv.org/content/promoting-evidence-based-approaches-adaptationguidelines

Saldana, L., Chamberlain, P., Wang, W., & Brown, H. (2012). Predicting program start-up using 

the Stages of Implementation Measure. Administration and Policy in Mental Health and 

Mental Health Services Research. 39(6), 419–425. 

Yan, S.C. & Huang, Y. F. (2008). A study of high school English teachers’ behaviors, concerns 

and beliefs in integrating information technology into English instruction. Computers in 

Human Behavior, 24 (3), 1085-1103.

ASPE ISSUE BRIEF | 17
DEPARTMENT OF HEALTH

& HUMAN SERVICES

Office of the Secretary

Washington, DC

OFFICIAL BUSINESS

Penalty for Private Use $300