Model Programs Guide

Implementation Guides:
 

Background and User Perspectives on
 

Implementing Evidence-Based Programs
 

Submitted by 

Submitted to 

Office of Juvenile Justice and Delinquency Prevention 

810 Seventh Street, NW 

Washington, DC 20531 

Cooperative Agreement No. 2013–JF–FX–K002 

December 2014 

Rachel Stephenson, Marcia Cohen, Chase Montagnet, 

Amanda Bobnis, Stephen Gies, and Martha Yeide 

Development Services Group, Inc. 

7315 Wisconsin Avenue, Suite 800E 

Bethesda, MD 20814 

iii

Model Programs Guide

Implementation Guides:

Background and User Perspectives on

Implementing Evidence-Based Programs

2014

Prepared by:

Rachel Stephenson

Marcia Cohen

Chase Montagnet

Amanda Bobnis

Stephen Gies

Martha Yeide

Development Services Group, Inc.

7315 Wisconsin Avenue, Suite 800E

Bethesda, MD 20814

http://www.dsgonline.com/

This project was supported by Cooperative Agreement No. 2013–JF–FX–K002 awarded by the Office of

Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice.

Points of view in this document do not necessarily represent the official position or policies of the U.S.

Department of Justice.
iv

Acknowledgments

Development Services Group, Inc., has many people to thank for helping us complete this

report and begin the development of the MPG Implementation Guides. First, we thank the staff

at the Office of Juvenile Justice and Delinquency Prevention (OJJDP) and the National Institute

of Justice who took time from their busy schedules to participate in a focus group meeting and

share their thoughts at the beginning of the project. We especially thank Jen Tyson, our Program

Manager at OJJDP, who provided important guidance and useful feedback throughout the

entire process.

We thank the Coalition for Juvenile Justice, which helped organize numerous focus groups that

were conducted during their annual conference. We also thank Blueprints for Healthy Youth

Development, the National Council on Crime and Delinquency, and the National Council on

Juvenile and Family Court Judges, each of which put on a great conference this year that

allowed us to connect face to face with important individuals who contributed a great deal of

information to the project.

Finally, we express our deep gratitude to all of the Juvenile Justice Specialists, all the State

Advisory Group members, and all of the other individuals across the country working with at- risk youth who made time to speak with us about their experiences implementing evidence- based programs and practices. The insight that was gained from their experience and

knowledge was very valuable to the development of the MPG Implementation Guides, and we

truly appreciate all that they contributed.
v

Contents

Acknowledgements .................................................................................................................iv

1. Background on Developing MPG Implementation Guides.......................................1

Search of the Literature ................................................................................................2

Specific Models of Implementation............................................................................2

Focus, Scope, and Breadth of Included Implementation Information..................3

Specific Formatting of Implementation Guides .......................................................5

Conclusion .....................................................................................................................5

References ......................................................................................................................6

2. Methodology of Qualitative Interviews and Focus Groups.......................................7

Development .................................................................................................................7

Design .............................................................................................................................7

Instruments ....................................................................................................................9

Sample ............................................................................................................................9

Data Analysis/Procedure ............................................................................................11

3. User Perspectives: Findings from the Qualitative Interviews ...................................12

Frequently Occurring Words ......................................................................................12

Key Themes....................................................................................................................13

Specific Recommendations to Improve MPG and CrimeSolutions.gov ..................22

MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

1

1. Background on Developing MPG Implementation Guides

The Office of Juvenile Justice and Delinquency Prevention’s (OJJDP’s) Model Programs Guide

(MPG) contains information about evidence-based juvenile justice and youth prevention,

intervention, and reentry programs. The MPG is a resource for practitioners and communities

about what works, what is promising, and what does not work in juvenile justice, delinquency

prevention, and child protection and safety.

A major objective of the first year of the new MPG cooperative agreement (no. 2013–JF–FX– K002) tasked Development Services Group, Inc. (DSG), with developing Implementation

Guides to be featured on the Web site. The goal of the Implementation Guides is to enhance the

information and resources available on MPG to better support MPG users (policymakers and

practitioners) in implementing evidence-based programs and practices.

During the past year, DSG developed an Implementation Guide Development Plan in

conjunction with OJJDP’s Program Manager. The document outlined the approach DSG would

take to gather background information on implementation science research, user perspectives,

and other pertinent information, which would be used to outline the MPG Implementation

Guide Protocols and Standards. The Protocols and Standards would include recommendations

about the type of information to be included in the Implementation Guides, the standards of

evidence with regard to research used to develop the Implementation Guides, the general

format and layout of the Implementation Guides, and other procedures and standards that will

guide development of all future Implementation Guides. This document provides a summary of

this background work, including the research, data collection and analysis, and internal

discussions held on the MPG Implementation Guides. When available, additional reports will

be published with the plan for developing the new Implementation Guides.

Part 1 (this section of the document) provides an overview of some of the implementation

guides and research already available to policymakers and practitioners. Part 2 provides a

description of the methodology used to collect qualitative interview and focus group data from

38 practitioners, researchers, and various other professionals in the juvenile justice field (or

related fields) who have experience in implementing a program or practice. The qualitative data

was designed to supplement the information and knowledge gained from the research

described in Part 1. Part 3 is an overview of the findings that were identified, based on the

responses from the 38 interviewees.
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

2

Figure 1.2. The 10 GTO Steps as

a Painter’s Palette

(from Mattox et al. 2013)

Search of the Literature

The process of developing Implementation Guides for the MPG began with a search of relevant

literature. This included searching for resources and information currently available from peer- reviewed journals, government agencies, and other organizations, such as the National

Implementation Research Network (NIRN), to get a general understanding of what was already

known about implementation and implementation science. The search

resulted in the development of an annotated bibliography. A brief

overview of some of the identified research is provided below. The

research is discussed in context of the proposed framework of the MPG

Implementation Guides. The information discussed below (in addition to

many other implementation research articles and projects identified

during the literature search) was considered during internal DSG

meetings and conference calls between DSG and OJJDP in terms of

whether the MPG Implementation Guides would follow a specific

implementation model already developed and, if not, how much influence

the other implementation models would have on the framework of the

MPG Implementation Guides.

Specific Models of Implementation

Many implementation models provide practitioners with an ordered or

sequential Implementation Process to follow. Most models will emphasize

the importance of not viewing the Implementation Process as linear, but

rather a process that often goes back and forth between stages. However,

visual representations (for example, arrows pointing from one step to

another) are often used to illustrate the various steps or components of a

specific model, which seems to contradict the suggestion that the process

should not be viewed as linear. This may suggest to practitioners that they

should follow a prescriptive formula when implementing a program or

practice, even if they are instructed otherwise. For example, Fixsen and

colleagues (2005, 15) conducted a comprehensive review of the literature and developed six

stages of the Implementation Process based on prior research

(see Figure 1.1). The report emphasized: “[I]mplementation is a

process, not an event. Implementation will not happen all at

once or proceed smoothly, at last not at first.” Similarly, RAND’s Getting to Outcomes (GTO) series presents

10 steps to address issues before and after the

implementation of a program has begun. The model (see

Figure 1.2) provides numbered steps across two phases: the

Planning process and the Evaluating and Improving process.

The steps are: 1) identify needs and resources in your

community; 2) identify goals and desires outcomes; 3) find

programs that help achieve your goals; 4) review program

choices for best fit; 5) determine the capacities needed for

implementation; 6) make a plan for implementation; 7)

evaluate the implementation process; 8) evaluate outcomes

of the program; 9) improve the program with continuous quality improvement; and 10) plan for

Figure 1.1. Stages of

Implementation

Process

(from Fixsen et al. 2005)
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

3

Figure 1.3. The Dynamic

Interplay Among Critical Steps of

QIF (from Meyers, Durlak, and

Wandersman 2012)

program sustainability. The guide does explain: “While the steps are numbered and there is an

overall order to them, you may need to

skip around and pick and choose

depending on your particular needs”

(Mattox et al. 2013, 3).

The Quality of Implementation

Framework (QIF) developed from a

literature synthesis concentrated on 25

frameworks of implementation from

myriad fields, including health care;

substance abuse prevention and

treatment; school-based innovations;

technological innovations; community- based preventions services; and mental

health promotion. When discussing the development and overview of the QIF, Meyers, Durlak,

and Wandersman explain in a footnote of the article:

The arrows from one phase to the next are intended to suggest that steps in each of the phases could

continue to be addressed throughout the implementation process. Steps in each of the phases may

need to be strengthened, revisited, or adapted throughout the use of an innovation in an

organization/community. While a logical order in which the critical steps unfold was needed to

develop a coherent framework, we believe the manner in which they are implemented in practice will

depend on many factors.” [2012, 475]

The phases of QIF, presented in Figure 1.3, also suggest a prescribed order to addressing the

various factors of implementing a program or practice, although narratively the authors

emphasize that the advancement of these steps depends on the specific situations and contexts

of the organization or community. The QIF is meant to represent the “dynamic interplay” of the

important phases and steps of implementation within a structured framework.

Moreover, if a practitioner is searching for information on a particular component or step of the

implementation model—such as how to conduct a needs assessment—this may require

familiarity and understanding about the entire implementation model to find information

specific to the practitioner’s needs. For example, a reader may not know the specific

components that fall under each of the six stages of the Implementation Process developed by

NIRN or the 10 steps of RAND’s GTO model.

Finally, although the models described above allow for practitioners in various fields to use the

information about implementation, it may be difficult for those in the field to translate the

general information into something more specific for their particular program or practice. As

one interviewee noted: “One caveat to using a guide like NIRN is they may not be as useful on

small projects. However, the key steps/pieces are the same.”

Focus, Scope, and Breadth of Included Implementation Information

In addition to considering whether to follow a specific implementation framework, another

important issue was the focus of the MPG Implementation Guides and the amount of
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

4

information that should be provided. The focus refers to the phases, stages, or steps of the

Implementation Process that would be discussed in the Implementation Guides. Many guides

focused on a general framework for the implementation of various types of programs. The

stages of the Implementation Process (Figure 1.1) developed by NIRN is an example of a

general model that could be used by practitioners and policymakers in diverse fields (Fixsen et

al. 2005). The NIRN model centers on the entire Implementation Process and not on a particular

stage. Similarly, the GTO model guides readers through a 10-step process across the spectrum

of the implementation, from planning to evaluation. The QIF (Figure 1.3, above) is another

example of a conceptual framework that illustrates all of the phases and steps involved in the

process of quality implementation, though the authors note that “most of these of steps (10 of

the 14) should be addressed before implementation begins” (Fixsen et al. 2005, 468).

The scope refers to the level at which the Implementation Guides would provide specific

information about implementation. For example, the guides could look at implementation

about specific programs, or take a broader scope and look at certain topic areas or practices. The

GTO series includes guides developed for various topic areas including home visitation,

services for homeless veterans, underage drinking prevention, and developmental assets

(RAND Corporation 2014). Conversely, the Blueprints for Healthy Youth Development database (a

registry of violence, delinquency, and drug prevention programs that have been demonstrated

as effective) provides implementation information, such as program costs and funding

strategies, for each program listed (Blueprints for Healthy Youth Development 2014).

With regard to the breadth of information provided to readers, many guides offer an abundance

of information about implementation. For example, the Getting to Outcomes for Home Visiting

guide by RAND provides readers with specific information about planning, implementing, and

evaluating a home visiting program. It takes readers through its 10 steps of the Implementation

Process (see Figure 1.2, above), in addition to providing checklists; examples; tools; and tips,

resources, and other helpful extras during each step. The information is comprehensive, and the

guide is lengthy (Mattox et al. 2013). This may be daunting to readers, who may not take the

time to explore such a guide even if it provides valuable information about the Implementation

Process. Like RAND, NIRN provides a great deal of valuable information on implementation.

One example is a synthesis of the reviewed literature and research on implementation (Fixsen et

al. 2005). The report provides the results from an extensive search of the literature, but is also

lengthy. However, additional research and resources provided by NIRN do take shorter forms

(see Blase, Kiser, and Van Dyke 2013; Blase and Fixsen 2013). Another example is the

Community Tool Box Web site, developed by the Work Group for Community Health and

Development at the University of Kansas. The Community Tool Box is a free online resource for

policymakers and practitioners looking to build healthier communities and bring about social

change (University of Kansas 2014). The Tool Box provides an immense number of resources,

including 46 chapters through which users can obtain step-by-step guidance in community- building skills, including information on implementing programs and practices. Although the

Web site provides very useful and thorough information, navigating through 46 chapters may

be overwhelming to users. Within each chapter, there are as many as 10 separate sections that

users need to further dig through to find the information they may need.
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

5

Specific Formatting of Implementation Guides

Another factor that was explored was the specific format for the MPG Implementation Guides.

Many guides take the form of a booklet or a journal article (see Fixsen et al. 2005; Mattox et al.

2013). However, other Web sites, such as SAMHSA’s National Registry of Evidence-based Programs

and Practices (NREPP), provide an interactive site for users to explore through the various topics

related to implementation (SAMHSA 2014). NREPP’s interactive Web site also offers users the

opportunity to print the entire course as a guidebook, by downloading a printer-friendly PDF

version of the Web pages (see Figure 1.4 for an example of the first Web page in the course

provided by NREPP). The Community Tool Box available from the University of Kansas is

another example of a more interactive approach to providing users with information. Each

section of the 46 chapters includes a Main section, which provides a narrative explanation about

the Implementation Process; a Checklist section (if relevant for the topic under discussion),

which practitioners can use in the field; an Examples section, which includes an interview with

a practitioner who discusses the implementation experience with regard to the topic under

discussion; and a PowerPoint presentation, which summarizes the major points in the section.

The Community Tool Box also provides users with a printer-friendly version of each section

(University of Kansas 2014). However, because each of the 46 chapters can have as many as 10

sections each, printing out each section may be time consuming.

Conclusion

There is a great deal of information available on numerous frameworks and models that can

guide the Implementation Process for practitioners in juvenile justice and other fields. What is

notably absent from much of the currently available literature on the Implementation Process is

empirical research exploring the application and effectiveness of specific models. Some

evaluation work has recently been conducted. One example of this limited research is the

PROmoting School–community–university Partnerships to Enhance Resilience (PROSPER)

model, a program-delivery system in which universities partner with community teams to

implement evidence-based programs aimed at preventing substance abuse and other problem

behaviors among youth. Although randomized controlled trials were conducted to evaluate the

impact of PROSPER (Spoth et al. 2013), the model represents a specific approach to implement a

select number of prevention programs, as opposed to a general framework that can guide

Figure 1.4. A Roadmap to Implementing

Evidence-based Programs (from SAMHSA)

2014
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

6

practitioners looking to implement any type of program. There has been limited research on the

effectiveness of a general framework (such as those developed by NIRN and RAND).

References

Blase, Karen A., and Dean L. Fixsen. 2013. Stages of Implementation Analysis: Where Are We?

Chapel Hill, N.C.: University of North Carolina at Chapel Hill, FPG Child Development

Institute, National Implementation Research Network.

Blase, Karen A., Laurel Kiser, and Melissa K. Van Dyke. 2013. The Hexagon Tool: Exploring

Context. Chapel Hill, N.C.: University of North Carolina at Chapel Hill, FPG Child

Development Institute, National Implementation Research Network.

Blueprints for Healthy Youth Development. 2014. Blueprints for Healthy Youth Development.

Boulder, Colo.: University of Colorado, Boulder, Institute for Behavioral Science, Center for

the Study and Prevention of Violence. http://www.blueprintsprograms.com/

Fixsen, Dean L., Sandra F. Naoom, Karen A. Blase, Robert M. Friedman, and Frances Wallace.

2005. Implementation Research: A Synthesis of the Literature. Tampa, Fla.: University of South

Florida, Louis de la Parte Florida Mental Health Institute, National Implementation

Research Network.

Mattox, Teryn, Sarah B. Hunter, M. Rebecca Kilburn, and Shelley H. Wiseman. 2013. Getting to

Outcomes for Home Visiting: How to Plan, Implement, and Evaluate a Program in Your

Community to Support Parents and Their Young Children. Santa Monica, Calif.: RAND

Corporation.

Meyers, Duncan C., Joseph A. Durlak, and Abraham Wandersman. 2012. “The Quality

Implementation Framework: A Synthesis of Critical Steps in the Implementation Process.”

American Journal of Community Psychology 50:462–80.

RAND Corporation. 2014. “Getting to Outcomes: Improving Community-Based Prevention.”

Santa Monica, Calif.: RAND Corporation. http://www.rand.org/health/projects/gettingto-outcomes.html

SAMHSA (Substance Abuse and Mental Health Services Administration). 2014. “A Roadmap to

Implementing Evidence-Based Programs.” Washington, D.C.: U.S. Department of Health

and Human Services, SAMHSA.

http://www.nrepp.samhsa.gov/Courses/Implementations/NREPP_0101_0010.html

Spoth, Richard L., Cleve Redmond, Chungyeol Shin, Mark T. Greenberg, Mark E. Feinberg, and

Lisa M. Schainker. 2013. “PROSER Community-university Partnership Delivery System

Substance Misuse Outcomes Through 61⁄2 Years Past Baseline.” Prevention Science 56:190–6.

University of Kansas. 2014. “Community Tool Box.” Lawrence, Kan.: University of Kansas,

Workgroup for Community Health and Development. http://ctb.ku.edu/en
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

7

2. Methodology of Qualitative Interviews and Focus Groups

Development

To determine the common elements in program implementation and the needs of those

involved in the process, Model Programs Guide (MPG) staff conducted interviews and focus

group discussions with practitioners, researchers, policymakers, and program developers who

have successfully implemented programs or who have otherwise been involved in the

implementation process.

An initial focus group discussion was held on Feb. 18, 2014, which included only federal staff

from the Office of Juvenile Justice and Delinquency Prevention (OJJDP) and the National

Institute of Justice (NIJ). The MPG Program Manager sent out an invitation to staff members

from both agencies, and 12 staff members from OJJDP and four from NIJ participated. The

primary goals of the focus group were to a) discuss the possible scope, focus, and format of the

MPG Implementation Guides; b) get feedback from participants about other implementation

models and frameworks identified during the literature search (NIRN, RAND, etc.); and c) learn

about what other agencies and projects within the Office of Justice Programs were doing with

regard to implementation.

Although the ideas from the initial focus group discussion were considered when developing

the recommendations for the Implementation Guides, the meeting notes were not included in

the coding with the rest of the interviews and focus groups, as the focus group with federal staff

had a different focus. However, this initial focus group informed the MPG Implementation

Guide Development Plan, which laid out the approach to conduct informal, qualitative

interviews with a small number of practitioners at four conferences.

Design

Conferences that occurred between April and August 2014 and would draw an interested

audience were chosen as the interview/focus group sites, as this would allow for the maximum

number of in-person interviews. In addition, numerous phone interviews were conducted to

gather input from various OJJDP grantees who did not attend the conferences. The following

conferences were chosen:  Blueprints Conference in Denver, Colo. The Blueprints Conference took place April 14–

16, 2014. The target audience included state child welfare, juvenile justice, education,

mental and behavioral health, and public directors interested in learning about

evidence-based programs, as well as policymakers, researchers, and program developers

of some of the name-brand programs included on the Blueprints for Health Youth

Development Web site, such as Functional Family Therapy. The conference provided

professionals with support, guidance, and tools to help implement Blueprints programs

successfully in their own communities.  National Council on Crime and Delinquency (NCCD) Conference on Children, Youth,

and Families in San Diego, Calif. The NCCD conference took place during May 14–16,

2014. The conference concentrated primarily on child welfare and sought to engage all
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

8

systems working with children and families, including child protection, foster care,

juvenile justice, and education. The overall goal of the conference was to bring together

those involved in these systems to create stronger connections and thereby better serve

youths and families.  OJJDP’s Core Requirements Training and the Coalition for Juvenile Justice (CJJ) Annual

Conference in Washington, D.C. The OJJDP Core Requirements Training and the 2014

CJJ Annual Conference took place June 17–21, 2014. The CJJ conference brought together

Juvenile Justice Specialists (JJSes), State Advisory Group (SAG) Members, and other

pertinent state juvenile justice personnel. The conference included workshops,

informative sessions and plenaries, and provided an opportunity for conversations

among those involved in juvenile justice in their state, as well as those who provide

grants to local governments and nonprofit organizations in an effort to prevent crime

and delinquency. At the CJJ conference, Development Services Group, Inc. (DSG), held a

focus group with SAG members, and, at the Core Requirements Training, DSG held

another focus group with JJSes.  National Council of Juvenile and Family Court Judges (NCJFCJ) Annual Conference in

Chicago, Ill. The NCJFCJ conference took place during July 13–16, 2014, and featured a

wide range of juvenile- and family-law topics including child abuse and neglect, trauma,

custody and visitation, judicial leadership, juvenile justice, sex trafficking of minors,

family violence, drug courts, psychotropic medications, children testifying in court,

detention alternatives, substance abuse, and the adolescent brain. This conference

provided the opportunity to speak with judges: a group that had not been present at

previous conferences.  As a result of discussions with OJJDP, DSG staff also interviewed members of the

Federal Advisory Committee on Juvenile Justice (FACJJ), which has an evidence-based

subcommittee. The staff also conducted interviews with tribal grantees, so that

experiences of program implementation in tribal communities were captured. Moreover,

DSG staff was able to leverage contacts through the OJJDP National Forum on Youth

Violence Prevention to obtain other potential interviewees.

Once the conferences were selected, DSG staff contacted conference organizers to obtain the

participant lists so that interviews with conference attendees could be scheduled. DSG was able

to obtain the lists for the OJJDP Core Requirements Training and the CJJ Annual Conference

and invite specific JJSes and SAG members. Invitations to the JJSes were approved and sent out

by the OJJDP MPG program manager and invitations to the SAG members were sent out by the

DSG MPG project director. Unfortunately, for the other three conferences, participant lists were

not available beforehand. Instead, DSG staff contacted presenters at each of the conferences, as

this information was available on the conference Web sites. The lists of presenters and their

available biographies from each of the conferences’ Web sites were reviewed, and potential

interviewees were selected if they seemed to have substantial experience with program

implementation or were currently in the process of implementing a program. DSG contacted

these potential interviewees by email and provided them with information about the MPG and

the Implementation Guides project. Potential interviewees were asked if they had time for a 30- minute, one-on-one interview during the conference. Interviews were set up around presenters’
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

9

schedules when possible; follow-up phone interviews occurred for some presenters who were

unable to meet during the conferences.

Instruments

The interviews were loosely structured around an interview Discussion Guide and the overall

goals of developing the MPG Implementation Guides. The interview guides were developed to

capture specific ideas and concepts related to implementation and the interviewees’ experiences

implementing a program or practice. The questions were semistructured and open ended,

allowing for interviewers to guide the discussion based on the respondents’ answers. Each

interview included two interviewers: one functioning as the lead interviewer, the other

functioning as the secondary interviewer. The lead interviewer asked most questions, while the

secondary interviewer took notes and asked additional questions as needed. For each interview,

the Discussion Guide was modified based on the background and expertise of the interviewees.

For example, state- or county-level employees of the juvenile and child welfare systems were

asked different questions from those asked of direct-service providers. The interviews lasted

between 30 and 40 minutes.

Similarly, there was a loosely structured Focus Group Discussion Guide, which had fewer

questions and captured broader ideas related to implementation. This guide was used to

facilitate the discussion during the focus groups at the NCCD conference, and during the State

Advisory Group and Juvenile Justice Specialist focus groups at the Coalition for Juvenile Justice

Conference. Overall, the questions varied among the different focus groups; for example, the

conversation with the JJSes and SAG members included questions related to how the focus

group members used the MPG and encouraged communities in their states to use the site. The

focus groups lasted approximately 1 hour.

All one-on-one interviews and both focus groups were audio-recorded (with respondents’

permission) and were covered by note takers. To ensure confidentiality, interviewees were

informed that any identifying information would not be included in any documents made

available to the public.

Sample

Thirty-eight individuals were interviewed through one-on-one interviews or in focus groups.

Table 2.1 shows the number of respondents interviewed across conferences and phone

interviews.
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

10

Table 2.1. Number of Respondents by Conference

Conference/Target Population One-on-One

Interviews

Focus Groups Total

Interviewed

Blueprints Conference 3 0 3

NCCD Conference 4 1

(including 5 respondents)

9

OJJDP Core Requirements Training:

Juvenile Justice Specialists

0 1

(including 9 respondents)

9

CJJ Conference: SAG Members 0 1

(including 9 respondents)

9

NCJFCJ Conference 3 0 3

FACJJ Phone Interview 1 0 1

Tribal Grantees Phone Interviews 2 0 2

National Forum Grantees Phone

Interviews

2 0 2

38

Of the 38 respondents, there were almost even numbers of women (20) and men (18). The

interviewees had expertise primarily in the following three fields: juvenile justice (76 percent);

child and family services (19 percent); and education (5 percent); as shown in Figure 2.1. With regard to participants’ specific jobs, the bulk of

the interviewees were SAG members and Juvenile

Justice Specialists (about 24 percent for both SAG

members and JJSes). The rest of the sample (see Figure

2.2) comprised direct service providers (16 percent);

child and welfare services personnel (16 percent);

courts/probation (13 percent); training and technical

assistance (5 percent); and education (3 percent).

Figure 2.2

Figure 2.1
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

11

The interviewees represented the District of Columbia and 21 U.S. states (see Figure 2.3):

California, Colorado, Delaware, Georgia, Hawaii, Idaho, Kentucky, Maine, Maryland,

Minnesota, Montana, Nebraska, New Jersey, New York, North Carolina, Ohio, Tennessee,

Texas, Utah, Vermont, and Virginia. The regional diversity among interviewees ensured that

the data reflected program implementation experience from across the United States.

Data Analysis/Procedures

All notes from the interviews and focus groups were analyzed by two DSG research assistants

(RAs) using QSR NVivo 10 for Windows, a well-known and state-of-the-art qualitative data– processing software package. The RAs imported the detailed written notes from the interviews

and focus groups into the program. Individually, each RA read through the text and coded the

responses from the interviewees. The responses were coded and categorized based on similar

topics that had been discussed during the interviews and focus groups. This made it easier to

analyze the text for specific themes.

Once the text was coded and organized, the responses were analyzed for common themes and

key ideas or components of the implementation process. Queries were run through NVivo to

search the text for specific words or phrases, identify the most frequently coded words or

phrases, create charts and graphs corresponding to the coded text, and run tests to ensure

interrater reliability.

With regard to interrater reliability, because all of the text was double coded, a query was run to

compare the coding of the two RAs. Some differences in coding or categorizing some responses

were discussed, and a consensus was usually reached. The results of the interrater analysis

showed Kappa=0.67. Using the rule of thumb when interpreting values of Kappa, this means

that there was substantial agreement between the RAs.

Figure 2.3
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

12

Part 3. User Perspectives: Findings From the Qualitative Interviews

The interviewees and focus group members provided important and helpful information about

implementation. Below is an overview of the responses given to the questions asked during the

interviews and focus groups. The responses are grouped by the major themes that were found

through analyzing the responses using the NVivo program. There were a total of 38

respondents across three focus groups and 15 one-on-one interviews.1

Frequently Occurring Words

Using the query option in the NVivo program, the 10 most frequently occurring words in the

coded responses of the interviewees were identified. The query was run to exclude certain

common words (such as “the”) and words often spoken during all interviews (such as

“implementation”). The query was run to include stem words (for example, if the primary word

is “work” the stem words would be “working,” “worked,” and “works”). The 10 most common

words were 1) needs, 2) using, 3) provide, 4) community, 5) trainings, 6) project, 7) staff, 8)

works, 9) model, and 10) grants.

A word cloud is shown below in Figure 3.1 to represent the 10 most common words that were

spoken by interviewees (with the most frequently occurring words in larger fonts). The results

from the query helped guide the identification and categorization of key themes (discussed

below) by providing a list of important words that should be focused on when analyzing the

response from interviewees. For example, the word “needs” guided DSG to concentrate on

responses about using particular needs assessments as well as discussions with regards to the

specific needs of interviewees’ communities or jurisdictions.

1The responses from the focus group conducted with staff members from the Office of Juvenile Justice and

Delinquency Prevention (OJJDP) and the National Institute of Justice (NIJ) were not coded with the responses from

the 38 interviewees. However, several responses from OJJDP/NIJ focus group members are noted in this section.

Figure 3.1
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

13

Key Themes

The results below are based on coding specific words or phrases that were said by interviewees.

For example, some interviewees specifically talked about the issue of adaptability, while others

talked about changing a program to fit the needs and resources of their community. Both

responses were coded as discussing the theme of adaptation.

Many of the major themes discussed below overlap in certain ways. For instance, when

discussing the importance of getting staff buy-in, many interviewees noted that providing

training was a useful method to keep staff members involved in and informed of the

implementation process. When considering the number of respondents who discussed each of

the themes, respondents were counted if they mentioned a specific theme. In the previous

example, an interviewee would be counted as having discussed two major themes: staff buy-in

and training. If a respondent mentioned staff buy-in and did not mention training as an option,

then the respondent was counted as having discussed staff buy-in but not training.

1. Funding. Almost half of the interviewees (17 respondents) discussed the importance of

securing funding. Overall, interviewees received funding from a variety of sources, including

grants from the federal government, state governments, private foundations, nonprofit

organizations, and businesses.

Numerous interviewees had implemented a particular program or practice because of specific

funding they received. For example, a few interviewees’ jurisdictions were presented with the

opportunity to receive funding from research agencies or nonprofit organizations (such as the

Annie E. Casey or Robert Wood Johnson Foundation) to address certain target populations or

youth-related problems in the community. In these instances, it appears funding drove the

decision to implement a program and also affected which programs were implemented. For

instance, one interviewee explained that in her jurisdiction an initiative targeting crossover

youth (i.e., youth involved in the juvenile justice and child welfare systems) was implemented

because a grant opportunity to fund a pilot year of the program became available. However,

had the funding not been available, her jurisdiction might not have implemented a program

targeting crossover youth.

Others described identifying a particular problem in their jurisdiction and finding appropriate

funding to address the problem. For example, one interviewee was interested in delivering

gender-specific programming to juvenile girls in her jurisdiction. After conducting a needs

assessment to determine the services that were currently provided to girls involved in the

system, she searched for funding to support the pilot program. In the end, the county received

funding from a federal grant.

Some interviewees discussed the idea of planning from the beginning of the implementation

process for ways to secure funding (past the initial funding amount) for the long term, instead

of waiting for funding to run out before looking for other sources. Various respondents said

that grants or other types of funding from the state or federal government will typically last 2 to

3 years, a brief period for any program to be fully implemented. As a result, jurisdictions face

the challenge of finding ways to continue the program or locating new sources of funding.
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

14

A few interviewees mentioned the importance of understanding the financial costs of

implementing an evidence-based program or practice, and noted that this information is not

often available. For example, a few Juvenile Justice Specialists stated that it would be beneficial

if the MPG included additional information about the costs of implementing a program, such as

the costs of buying materials, training staff, and evaluating the fidelity and effectiveness of the

program. Moreover, a tribal grantee emphasized that not only does it cost a great amount to

initially train staff members to implement a specific therapeutic program aimed at youths, it

costs even more to maintain staff certification every year following the initial training. Such

information might not be readily available in an MPG program profile.

2. Adaptation. Almost half of the interviewees (16 respondents) discussed the importance of

considering the adaptability of programs and practices during the implementation process.

Some interviewees discussed how they adapted a particular program or model to fit the needs

and resources of their community. For example, one interviewee discussed a structured

decision-making tool that was implemented in her state’s child welfare system. However, the

tool needed to be “tweaked” to account for the state’s child welfare statutes and definitions that

differed from other states. Another interviewee learned, in the course of implementing a

prevention program that targeted a specific problem behavior, of other related problems faced

by program recipients that needed to be addressed. This problem was discussed with the

federal agency funding the pilot program, and the agency later changed future RFPs to require

grantees to account for unanticipated issues that may arise in the course of implementing a

prevention program.

One interviewee (a researcher/TA provider) explained that his organization supports the idea

of “flexible fidelity.” He acknowledged that certain components of most programs must be

included in the implementation process, but he also believed that each community and

population has different needs that must be considered. He noted the importance of

understanding those specific needs and allowing for flexible fidelity.

While acknowledging that adaptation was important, some interviewees questioned how much

of a program model could be changed. A few interviewees specifically noted that they did not

necessarily support the idea of adapting a program model or curriculum. Another interviewee

noted that she was currently responding to a proposal that required potential grantees to select

an evidence-based program; yet the RFP specifically stated that adaptations were not allowed.

Some interviewees mentioned that sites they managed were implementing components of

evidence-based programs rather than implementing the entire program. For example, one

interviewee explained that her jurisdiction was beginning to implement programs that included

aspects of positive youth development. However, the jurisdiction was not implementing

specific PYD programs; rather the emphasis was on implementing important aspects of PYD,

such as focusing on youth’s assets rather than their deficits.

During the Juvenile Justice Specialists focus group, a participant suggested that “it would be

useful for the MPG to include information on using particular program components rather than

the whole program, as sometimes it’s not practical to implement an entire program.” Most focus

group members agreed that it was important to consider issues such as location, cultures, and

subpopulations when thinking about adaptation.
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

15

Finally, during the initial focus group with staff members of OJJDP and NIJ, participants

frequently mentioned adaptation as an important topic. One staffer from OJJDP stated that

adaptation was important to consider, but wondered if the evidence was really available to tell

communities what active ingredients or components of programs or practices could be

implemented effectively. Although focus group members shared different opinions about

adaptation, overall the group seemed to agree that it was an important topic that should be

addressed in the MPG Implementation Guides.

3. Working With Developers, Using Program Manuals, and Receiving Technical Assistance.

About half of the interviewees (16 respondents) discussed working with program developers,

using specific program manuals, or receiving technical assistance in some form to assist in the

program implementation process. Some interviewees were implementing specific program

models (including some programs featured on MPG), such as Functional Family Therapy, Teen

Outreach Program, Structured Decision Making, and Raising Healthy Children. Many

interviewees mentioned contacting the program developers directly to ask questions about

implementation (such as costs and necessary resources) and adaptability of the program model.

For example, some interviewees were working directly with the program developers to

implement the program, while others were working without technical assistance or program

manuals from the developers. One interviewee noted that because of the success of the program

being implemented in her jurisdiction, a manual was being developed for other jurisdictions

interested in implementing a similar program.

Numerous the interviewees were receiving technical assistance from outside organizations (for

example, the Annie E. Casey Foundation, the National Council on Crime and Delinquency, the

Robert Wood Johnson Foundation, and Georgetown University). The technical assistance

received from these organizations was usually tied to specific initiatives the organizations were

supporting in the grantees’ jurisdictions. For example, Georgetown University was working

with many the interviewees to implement a crossover youth initiative in jurisdictions across the

country.

Two respondents (who were working at the state-level) explained that they provided technical

assistance to local jurisdictions (rather than provide them with outside assistance). One

interviewee stated that she had traveled around her state to provide technical assistance to

various communities on concepts such as how to select an evidence-based program, and the

types of data that needs to be collected to evaluate the program.

4. Speaking With Others Who Have Experienced Implementation. Just over one third of the

interviewees (14 respondents) specifically mentioned meeting with other professionals (in their

jurisdictions or in other states) who have implemented programs similar to the ones being

considered for implementation in their jurisdictions. Through talking with others, interviewees

were able to learn about the challenges, mistakes, and successes that other jurisdictions

experienced during their implementation process. The extent of talking to others varied from

basic networking, to site visits, to receiving coaching or training from others who had

previously implemented the program. One interviewee explained: “By working with others,

you can learn from their mistakes and their successes; you don’t have to learn the lessons on

your own.”
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

16

A few interviewees said they wished they had talked to other jurisdictions or communities

before beginning the implementation process, because it would have been beneficial to learn

from the experiences of others. For example, one interviewee was in the process of

implementing a crossover youth initiative in four counties in her jurisdiction’s court system. She

had not reached out to other jurisdictions in her state that had implemented a similar initiative

to get their feedback before taking on the major initiative, and as a result she encountered some

issues when the program was implemented in the first of the four counties. The first county in

her jurisdiction to start the program experienced communication and bonding issues among

staff members from various agencies involved in the initiative (including juvenile justice and

child welfare) that affected program implementation. However, the interviewee noted that staff

in the three counties that were about to begin implementation were already talking and

coordinating among themselves, learning from the experience of the first county.

A few interviewees also mentioned the idea of “coaching.” One interviewee was implementing

a program funded by the Robert Wood Johnson Foundation. In addition to providing financial

support, the Foundation also offered coaching to new sites beginning the implementation

process. The new sites were given the opportunity to meet with a professional from another site

that had already implemented the program to discuss any concerns or questions about the

implementation process.

5. Getting Staff Buy-In. About one third of the interviewees (12 respondents) said that staff

buy-in was critical to the implementation process. Some interviewees explained that it was

important to keep staff informed of the changes that would be happening and why they were

happening, while still being responsive to their concerns. As one interviewee stated, “Staff need

to have an understanding of why.”

A few interviewees noted that it was particularly important to get buy-in from frontline staff

because implementing a new program or practice could impact their workload. One

interviewee stated that a key step to the implementation process was having “good marketing

strategies.” The interviewee further explained that good marketing strategies are a way to

explain why a particular program will work or why the state is mandating a particular program

or change.

There were several suggested methods to gain buy-in from staff. One common suggestion was

to provide training to staff. This can include training on the new program or practice model,

and/or training on why an evidence-based approach is being taken in the first place. By

participating in trainings, staff members are more likely to understand the entire process.

A few interviewees specifically noted the challenge of dealing with high staff turnover, which

often resulted from staff not accepting the new program. One interviewee suggested combating

employee turnover by explaining to the workers and staff members the benefits of the proposed

changes. However, she also noted that workers who want to leave are probably not dedicated to

the organizational change. Additionally, the interviewee said it was important to give everyone

space to learn in a reactive system. Coaching and support are important factors in overcoming

these challenges.
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

17

A few interviewees explained that buy-in may be internal (i.e., the staff directly implements the

program) or external (i.e., the treatment providers may have to implement a new model or

process). When discussing the issue of staff buy-in, the interviewees also noted the importance

of building trust and relationships and of maintaining good internal and external

communication.

6. Creating and Garnering Support From Managers/Executives/Leadership. Similar to the idea of

getting stakeholder buy-in, several interviewees (including those in managerial positions)

emphasized the importance of getting support from executive level staff before implementing a

new program or practice. One interviewed explained: “Ensure that you have upper

management buy-in. Without upper management buy-in, the implementation process will be

an uphill battle.”

Two interviewees specifically mentioned receiving support to implement new programs and

practices from their states’ governors. One interviewee described how her state governor

provided a substantial amount of funding in the beginning, and has recently increased the

funding to continue the work past the first year. Another interviewee explained that in her

state, the governor worked alongside child welfare staff members because he wanted to

understand how the changes to the system actually worked.

7. Finding an Advocate/Champion. About one fourth of interviewees (9 respondents) mentioned

the importance of finding an advocate or champion of the program. This was important to

garner support to implement the program, but it also establishes a go-to person who can field

questions and provide a clear explanation of the program or practice. One interviewee, who

worked in the child welfare system, described her agency’s director as the champion for change

in her state. Once the director was appointed to the position, she pushed the agency to become

more research oriented. When the state began to implement new evidence-based programs and

practices, the director worked to get buy-in at the executive-level (i.e., support from the state’s

governor), explained the new vision of the agency to all stakeholders, spearheaded the funding

efforts, and pushed for fidelity reviews and evaluations to show that the new programs were

having a positive effect. Similarly, some interviewees suggested that having a project coordinator, or someone in a

similar position, was important to ensure at least one consistent person who is knowledgeable

about the program model and its components. One interviewee who was implementing a

multiagency initiative thought that it was necessary to have one lead person for the entire

project or at least one lead person from each of the various agencies involved. She described the

coordinator as someone “who would be the backbone of the program and who would worry

about the details. You need someone who is passionate about the position.”

Some interviewees were advocates and champions of the programs and practices they had

implemented (although they did not initially identify themselves as such). For example, one

interviewee described her efforts to provide support for an initiative in her state that focused on

reducing the number of youth who were incarcerated. The interviewee stated that she provided

training and technical assistance to stakeholders involved in the initiative through many site

visits and phone calls, to answer questions from various individuals. She also handled the
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

18

initial pushback from some stakeholders, such as juvenile court judges, when the initiative was

first proposed.

8. Specific Jurisdictional Issues. One third of the interviewees (13 respondents) discussed

specific issues in their jurisdictions that affected the implementation process. Examples of

specific jurisdictional issues provided by interviewees included challenges associated with

implementing programs in different settings (i.e., rural, urban, and suburban), implementing

programs for specific targeted populations (i.e., tribal youth, females, and crossover youth), and

handling pushback from certain individuals or organizations involved in the implementation

process. As one interviewee noted, “There are variations in implementation across the regions

because of differences in many factors, such as urban and rural variances, transportation

available, services provided, etc.” Another interviewee explained that the issue of

transportation was an obstacle to implementation in her state. “A lot of communities in Georgia

have no public transportation options. The state wants to widen the network of providers using

the trainings to close the gap between access for community members and service providers.”

In many instances, these jurisdictional problems were not identified beforehand and affected

the implementation process. For example, one interviewee was involved in implementing a

multiagency initiative that targeted crossover youth. The initiative included collaboration

among court services personnel and judges in both the juvenile justice and child welfare

systems. She described how defense bar leadership and defense attorneys were resistant to

change because they did not want judges in different agencies (i.e., juvenile justice judges and

child welfare judges) to obtain too much information about their clients, which could affect a

client’s plea deal or chances of getting released.

Another interviewee described an initiative in her state to decrease the number of youth sent to

detention and increase the number of evidence-based programs used by the juvenile court

system, especially the juvenile judges. She noted that there was initial pushback from juvenile

court judges because the judges were not familiar with the programs that they were allowed to

implement, so they felt they were being told what to do. One way she addressed this problem

was to provide training and technical assistance to the judges about the initiative. This also gave

the judges an opportunity to express their reservations about the move to evidence-based

programming.

In a few cases, jurisdictional issues can affect whether a program or practice is implemented at

all. For example, many Juvenile Justice Specialists noted the lack of evidence-based programs

specifically targeting tribal youth. This has hindered attempts by some jurisdictions to

implement programs that fit the needs of the youth population, as there are few evidence-based

programs available for them to choose from. In other instances, the jurisdictional setting has

been an issue. For example, one focus group member mentioned contacting a name-brand

therapy provider to begin services in her community; however, the provider was not willing to

travel to such a remote location to train only a few staff members who would work with a small

number of youths.

9. Sustainability. Almost 30 percent of the interviewees (11 respondents) specifically discussed

the importance of sustainability in the implementation process. Most of these interviewees
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

19

further emphasized the importance of considering sustainability from the start of the process,

versus waiting until after a program has been fully implemented.

Many ideas about sustainability were discussed, such as capacity-building (i.e., training staff),

building strong partnerships, and securing funding. As one interviewee explained: “Strong

partners in sustainability is key. It’s key that funding be the number one concern, with

sustainability.” Another interviewee, who provided training/technical assistance, explained

that his organization provides grantees with information up front on barriers related to

sustainability, such as the amount of time, money, and staff turnover that may occur when

implementing a new program.

Numerous interviewees mentioned high staff turnover as one obstacle to sustainability of a

program or practice, as it creates the added complication of training new staff. One interviewee,

when discussing staff turnover, said “it can slow down the project, momentum, and the

passion, especially when leadership leaves.”

Funding for the program or practice was another issue mentioned by respondents as an obstacle

to sustainability. A few interviewees talked about the difficulty of securing funding, especially

once initial grants or other funding sources are done. Interviewees noted the importance of

ensuring continued funding of a program well beyond the grant period during the initial

implementation stages.

10. Doing Research/Homework. One third of the interviewees (13 respondents) mentioned the

importance of doing research and “homework” before launching the implementation process.

Interviewees suggested that practitioners and policymakers interested in implementing a

program or practice should research all the facts before getting too far into the implementation

process. The specific information that needed to be researched seemed to vary by interviewee.

For example, a few interviewees discussed researching specific programs or practices that were

being considered for implementation. One interviewee explained that his jurisdiction created a

partnership between court staff, treatment providers, and representatives from the mental

health services board to identify treatment programs that could be implemented. The

collaboration team analyzed the research and identified Functional Family Therapy as a

program that best fit the community needs, in terms of how much funding was required to

launch and sustain the project.

Several interviewees also discussed the importance of researching the costs associated with

implementing a program. During the Juvenile Justice Specialists focus group, several

participants discussed the problem of gauging implementation costs for the program or

practice, specific to their community, as often that information is not readily available (even on

Web sites run by the program developers). Numerous interviewees talked about contacting the

program developers directly to obtain information (such as specific program costs or

adaptability of a program) to determine whether implementing the program in their jurisdiction

was possible. Others discussed contacting other jurisdictions that had already implemented the

same program to learn more about the process and associated costs.

Some interviewees mentioned doing research on more general information, such as Dr. Mark

Lipsey’s Standardized Program Evaluation Protocol (SPEP) or research done by Dr. Edward
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

20

Latessa. The importance of Dr. Lipsey’s work was also discussed during the focus group with

staff from OJJDP and NIJ. One focus group member mentioned that OJJDP was evaluating

SPEP, but that the research would likely not be done for a few years.

11. Needs Assessments. About one fourth of the interviewees (10 respondents) discussed

conducting needs assessments. The needs assessments were sometimes conducted internally,

looking at data already available, while other assessments were conducted by outside

organizations.

Interviewees described conducting a needs assessment to determine what services were needed

that were not already available. Other interviewees examined the services currently being

provided to determine if the services were adequate. If it was found that the services were not

adequate, the needs assessment helped interviewees make adjustments to current services

and/or add services.

For example, one interviewee worked in a jurisdiction in which a nonprofit agency was

interested in assisting with the implementation of gender-specific programming. The

interviewee mentioned that a needs assessment was conducted with the probation department

to examine the services currently being delivered to girls. She mentioned the use of a gender- specific assessment tool to determine whether services offered at the county level were

sufficient to handle the issues of girls in the justice system. The needs assessment revealed that

the services were not sufficient, and gender-specific programming was implemented. Another

interviewee discussed a yearly needs assessment that is conducted across the state to help

regions determine the services they need to provide. Another interviewee explained that every

year her jurisdiction administers a tool to youths on probation called the Risk Resiliency

Assessment, to determine a youth’s need for treatment and services.

Whether a needs assessment had been conducted was a specific question that was asked during

the interviews. While several interviewees discussed conducting needs assessments, several

also stated that no formal assessments were completed.

The importance of needs assessments was also discussed during the focus group with staff

members from OJJDP and NIJ. For example, one focus group member talked about providing a

needs assessment tool for MPG users to help them determine their problems and the resources

available. This information could then be used to help users determine which MPG programs

they should focus on.

12. Establish Clear Goals. More than 20 percent of interviewees (8 respondents) mentioned the

importance of establishing and communicating clear goals at the beginning of the

implementation process. Respondents mentioned developing “plans of action” or creating

“frameworks” to help guide the implementation process from the start.

Establishing clear goals when implementing a new program or practice was considered

important to the interviewees for a few reasons. First, when establishing the goals, a clear plan

of action can also be developed. This plan can include important information such as

identifying the key stakeholders involved, determining each stakeholder’s responsibility and its

role in implementing the program, and developing a potential timeline of when certain action
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

21

items will occur. Many interviewees specifically mentioned the importance of implementation

planning.

Second, establishing clear goals also provides a way to gain buy-in from all stakeholders with

the implementation process, especially in situations that involve cross-agency collaboration. By

creating well-defined goals that are known by all those involved, everyone can be on the same

page about the course of action.

Third, establishing clear goals at the beginning of the implementation process allowed for some

interviewees to develop outcome or performance measures they used to evaluate the impact of

the program or practice. As one interviewee explained: “Having a well-defined plan is very

important. Without a well-defined plan, it would almost be impossible to apply for funding.

Your plan also shows how you will evaluate your program.”

13. Implementation Teams. Almost 30 percent of the interviewees (11 respondents) mentioned

the use of an implementation team. Interviewees expressed the importance of implementation

teams for programs or initiatives implemented across multiple agencies (e.g., juvenile justice,

child welfare), as the teams provided a platform for collaboration and communication. For

example, one interviewee from the juvenile court system explained that an implementation

team created for an initiative implemented in his community included representatives from the

alcohol, drug addiction, and mental health services board; the court system; the probation

department; and the treatment provider.

Another interviewee also described an implementation team that was established in her

jurisdiction to help with implementing a program targeting crossover youth. The initiative

included the juvenile court system, child welfare services, and the probation department. She

explained that as many as 40 people would attend the implementation planning meetings, but

within the team there was a smaller core group of individuals who “did all the heavy lifting.”

She also expressed the importance of having the right leadership and people on the

implementation team; interviewees reiterated this idea as well.

14. Handling Unanticipated Setbacks. About 18 percent of interviewees (7 respondents)

discussed unanticipated setbacks or obstacles they encountered during the implementation

process, such as unexpected delays in implementing the program or receiving funding. Some

interviewees believed the setbacks may have been preventable, while others did not think it was

possible to know ahead of time what could happen.

Echoing the importance of having a clear and established implementation plan from the start of

the process, other interviewees noted the importance of creating a plan that is flexible enough to

address unanticipated setbacks and allows for potential changes. One interviewee

recommended that “there should always be a Plan B ready to go if Plan A doesn’t work.”

15. Program Evaluation/Performance Measures. More than one third of the interviewees (14

respondents) discussed evaluating the impact of the implemented program or practice and/or

fidelity to the program design. However, not all interviewees were at the stage to collect

outcome data. Outcome data may have been collected internally, or it may have been collected

by an outside organization that was assisting in the implementation process. For example,
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

22

interviewees from the NCCD focus group discussed working with a private research

organization that was providing training and technical assistance to help implement the

program, as well as collecting performance measures to examine the impact of the program.

Several interviewees discussed collecting performance measures as a requirement of the

funding they were receiving. Members from the Juvenile Justice Specialists and SAG focus

groups discussed the performance measures that were required of grantees, and mentioned

specific databases used to report this information [including OJJDP’s Data Reporting Tool

(DCTAT)]. One interviewee explained that jurisdictions in her state were required to select

evidence-based programs from CrimeSolutions.gov to receive funding. The interviewee stated

that one of the main reasons for this requirement was the description of the outcome measures

in the program profiles listed on CrimeSolutions.gov, which could be used to evaluate the success

of implementing those programs in her state.

As discussed before, a few interviewees mentioned the importance of defining outcomes at the

beginning of the implementation process. While many interviewees discussed measuring the

impact of the program, none of the interviewees talked about conducting an evaluation that

included a comparison group. Most of the evaluations mentioned by interviewees were

evaluations that collected predata and postdata from program participants.

16. Program Training. About 15 percent of interviewees (5 respondents) discussed the

importance of program training during the implementation process. For example, one

interviewee explained that her jurisdiction was implementing a gender-specific program. In

addition to the training on the specific program model, staff also received general training

about gender-specific programming so they could understand why the approach was

necessary. Most staff did not understand the research behind gender-specific programming, or

why it was important to consider implementing these types of programs. As a result, staff

received training on the importance of research and need for gender-specific programs in their

jurisdiction.

“Some people did not accept the gender responsive programming so we turned to Girls Matter,

which was training provided by NGI [the National Girls Institute], to show why gender-specific

programming was needed.” This helped with getting staff to buy in, as discussed previously.

Another interviewee described her experiencing working with the developers of the Raising a

Healthy Child program and the training provided to teachers to ensure the program was

delivered as designed in the classrooms.

Numerous interviewees discussed “training the trainers” as an approach to build internal

capacity and to ensure sustainability of a program. This approach helped ensure that staff could

continue to run a program, even after the program developers or technical assistance providers

left.

Specific Recommendation to Improve MPG and CrimeSolutions.gov

A few interviewees familiar with the MPG and CrimeSolutions.gov gave suggestions to improve

the Web sites. For the MPG, participants in the Juvenile Justice Specialists focus group

suggested 1) adding a search option to filter programs by community size, 2) providing

information about the effectiveness of implementing program components (rather than just
MPG Implementation Guides: Background and User Perspectives on Implementing Evidence-Based Programs

23

information on implementing the whole program), 3) providing information about racial and

ethnic disparities and tools to help communities evaluate disproportionate minority contact, and 4) adding more information about the cultural competencies of programs (i.e., explaining

whether a program worked for a certain subpopulation, such as tribal youth).

One interviewee was involved in a statewide initiative to implement evidence-based programs

to reduce the number of youths who were detained or incarcerated. The initiative provided

funding to communities with the highest incarceration rates of juveniles in the state; however,

to receive the funding, the communities had to select an Effective-rated program from

CrimeSolutions.gov. The interviewee had spent considerable time on the Web site, and had a few

suggestions for improvement: 1) streamlining the Web site so that it is easier to navigate (for

example, the interviewee thought there were many tabs to choose from, which made finding

specific information a daunting process), 2) rewriting some of the program profiles to be more

succinct and less “wordy,” 3) creating individual Web pages to show all of the programs

available for individual treatment and group treatment, 4) providing information about costs

and implementation (such as certification requirements) earlier in the program profiles (i.e., not

at the bottom of the Web page), 5) adding more information about costs and implementation

(such as the initial start-up costs, the annual costs, and costs to sustain a program—a concern

echoed by some in the focus group of Juvenile Justice Specialists), and 6) providing direct links

or email addresses for the program developers, so users can contact them directly with

questions.