1

NJJEC Bulletin January 2012

Welcome to the NJJEC Bulletin!

Welcome to the inaugural edition of the NJJEC Bulletin! NJJEC is a project of the Justice Research and 

Statistics Association funded by the Office of Juvenile Justice and Delinquency Prevention (OJJDP).

NJJEC’s purpose is to improve the evaluation capacity of states, localities, and tribes and facilitate the 

use of evidence-based programs and practices in juvenile justice. The NJJEC Bulletin, which will be 

disseminated directly to subscribers and posted on the NJJEC website, will provide information related 

to juvenile justice evaluation that is of interest to state, local, and tribal juvenile justice professionals. 

Specifically, future issues of the Bulletin will include:

• Updates on NJJEC project activities

• Upcoming conferences, trainings, or other events relevant to those working in the juvenile 

justice field

• Information about practitioner-friendly resources of interest

• A State, Local, or Tribal spotlight sharing information about performance measurement and 

evaluation activities occurring across the country

• Articles discussing evaluation-related concepts in recent juvenile justice evaluation research or 

recent presentations and projects that are of interest to the field.

We welcome your participation! If you would like to submit an article, suggest a news item for the 

Bulletin, or ask a question about evaluation, e-mail njjec@jrsa.org . For more information about the 

resources available through the National Juvenile Justice Evaluation Center, visit

http://www.jrsa.org/njjec/. 

What’s Going On at NJJEC

Online tutorial: Improving Juvenile Justice Programming: An Introduction to Evaluation and 

Evidence-Based Practices

NJJEC is currently revising an online tutorial for state, local and tribal OJJDP grantees interested in 

learning about evaluation, performance measurement, and evidence-based practices. This tutorial will 

be an updated version of the Juvenile Justice Evaluation Center (JJEC) tutorial entitled, “Evaluation: A 

Tool for Program Improvement,” which is currently available on the NJJEC site: 

http://www.jrsa.org/njjec/trainings-presentations.htm. 

NJJEC Webinars Begin in January 2012

In collaboration with OJJDP, NJJEC will be hosting a series of webinars called Assessing Project 

Performance: Building Blocks of Evaluation and Performance Measurement on evaluation, performance 

measurement, logic models, and data collection. Webinar One: Basics of Performance Measurement and 

Evaluation will be held on January 19, 2012 from 2:00-3:30 EST. Check the NJJEC website soon for 

additional information and to register!
2

NJJEC Needs Assessment Update

We have completed data collection for the 2011 NJJEC Needs Assessment and presented preliminary 

findings at the OJJDP 2011 National Conference poster session in National Harbor, MD. View the poster 

here: http://www.jrsa.org/njjec/trainings-presentations.htm. A report summarizing the findings of the 

survey will be posted to the website when it is completed. Thank you to all OJJDP grantees who 

responded to the NJJEC Needs Assessment! 

Upcoming Events of Interest

Upcoming Conference on Mentoring- January 2012

The 2012 National Mentoring Summit will be held January 24-25, 2012 in Washington, DC. The 

conference is co-sponsored by OJJDP and includes sessions on mentoring research and evaluation as 

well as best practices in mentoring programs. For more information, see 

http://www.mentoring.org/program_resources/training_and_events/national_mentoring_summit. 

Featured Resources

New Juvenile Justice Journal Launched

OJJDP has published the first issue of the

Journal of Juvenile Justice. The peer-reviewed 

journal’s Fall 2011 issue contains articles and 

commentary across a wide range of juvenile 

justice topics, including the results from a study 

of Utah youth participating in juvenile drug 

courts as well as an examination of the 

reliability of a juvenile justice risk assessment 

tool used in New Mexico. The journal can be 

accessed at http://www.journalofjuvjustice.org/. 

New Disproportionate Minority Contact 

Resource Center

OJJDP has launched a new DMC Virtual 

Resource Center, which offers DMC-related 

materials and information and spotlights state 

and local efforts to reduce Disproportionate 

Minority Contact. Check out the Tennessee 

spotlight here: 

https://www.nttac.org/index.cfm?event=dmc.eff

orts. 

EBP Resources, Systematic Reviews and Meta-Analysis

In the last several years there has been 

increasing interest in the identification and use 

of evidence-based practices (EBPs, although the 

“P” can also stand for “programs” or “policies”) 

in juvenile justice. EBPs can be defined as those 

policies, practices and programs for which there 

is scientific evidence of their effectiveness.

There are a couple of different approaches that 

have been used to compile research and 

evaluation and draw conclusions about 

effectiveness. Examples include

CrimeSolutions.gov, a resource recently 

developed by the Office of Justice Programs to 

inform practitioners and policymakers about 

what works in criminal justice, juvenile justice, 

and crime victim services, and OJJDP’s Model 

Programs Guide, which focuses on evidencebased programs in juvenile justice.

Another approach to examining research 

evidence is the use of systematic reviews of 

published (and sometimes unpublished) 

evaluation studies regarding particular areas of 

the justice system. Systematic reviews use 

objective, published criteria to find, assess, and 

synthesize the results of research and evaluation 

studies relevant to a particular area. A reviewer 

or group of reviewers reads each study and code 

it according to various components, such as the 

type and strength of the research design used, 

the number and nature of the individuals 
3

included in the study, and the nature of the 

specific findings of the effort. These findings are 

then synthesized by the reviewer(s), who 

attempt to draw conclusions regarding the 

effectiveness of the program or practice. The

Campbell Collaboration is an international 

research network that provides systematic 

reviews of the effects of social networks.

A third approach to synthesizing the results of 

research and evaluation studies is known as 

meta-analysis. Meta-analysis employs a 

statistical method for combining the findings of 

similar studies to produce an overall assessment 

of the direction and magnitude of the findings 

when taken as a whole. This assessment of 

direction and magnitude, known as the effect 

size, indicates whether the program or initiative 

under review has had a positive or negative 

impact on the observed outcomes, and how 

large that impact has been, across all of the 

studies being reviewed. Proponents of metaanalysis argue that they are more objective than 

systematic reviews and can represent findings in 

a more sophisticated way. Reviews by the 

Campbell Collaboration often include metaanalyses, and the Journal of Experimental 

Criminology often publishes meta-analyses of 

specific justice issues. 

There are a number of resources available to 

juvenile justice practitioners and policymakers 

regarding EBPs. Look for discussions of these 

resources over the next several issues of the 

newsletter. 

State Spotlight

Florida

The Florida Department of Juvenile Justice produces a Comprehensive Accountability Report (CAR) 

annually containing performance measurement data for nearly 1,000 programs and case management 

units in the state. Detailed descriptions of data sources and methods used during collection are 

included with each report. To access these reports, check the FDJJ website at 

http://www.djj.state.fl.us/Research/PMIndex.html.

Featured Evaluation

Implementation Outcomes of Multidimensional Family Therapy-Detention to Community: A 

Reintegration Program for Drug-Using Juvenile Detainees

Howard A. Liddle, Gayle A. Dakof, Craig Henderson and Cindy Rowe, 2011

International Journal of Offender Therapy and Comparative Criminology 55(4); pp. 587-604

Implementation analysis is a valuable tool for 

assessing program performance and making 

improvements to program function. Rather 

than focusing on program outcomes, 

implementation analysis examines the way 

program activities have been executed—for 

example, whether the desired amount of the 

intervention was administered in the correct 

way to the appropriate target population. The 

data used in an implementation analysis are

called process measures (also called output 

measures). These measures focus on the day-today activities of a program that are critical to 

achieving desired outcomes. 

Implementation analysis is an important but 

frequently neglected component in program 

evaluation. If an evaluator examines outcomes

without implementation analysis, he or she 

makes assumptions that have not yet been 

supported by data—specifically, that the 

program’s activities adhered to the plan. 

Implementation analysis may show ways the 

execution of a program can be improved, or 

lend support that program activities were 

responsible for the outcomes. 
4

Liddle et al. conducted an implementation 

analysis of a Multidimensional Family TherapyDetention to Community (MDFT-DC) program, 

a modified version of the traditional MDFT 

program, which has consistently been 

demonstrated to be effective. MDFT addresses 

juvenile delinquency through a comprehensive 

approach that includes active participation of 

the youth’s family. Since substance abuse, 

delinquency, and HIV/STD infection are 

frequently co-occurring, MDFT-DC therapy 

adds an HIV/STD prevention component and 

seeks to address these elements simultaneously 

and through a more cohesive, continuous form 

of treatment both during detention and the 

post-release period.

Liddle et al.’s implementation analysis of 

MDFT-DC for drug-using juvenile detainees 

focuses on how well the important elements of 

the program were executed compared to 

elements of the enhanced services as usual 

(ESAU), the reentry program provided for 

juveniles not receiving MDFT-DC. Below are 

brief descriptions of the process measures

collected on the components of the MDFT-DC 

program that are considered to be essential to 

its success:

• enrollment—the number and percent 

of youth assigned to each program who 

actually enrolled,

• retention—number and percent of 

youth who remained in their treatment 

program (MDFT-DC or ESAU) for 3 

months or more, using treatment 

enrollment and discharge dates,

• satisfaction with services—total 

satisfaction scores from the Services 

Satisfaction Scale (SSS-16)

• substance abuse treatment provider 

and juvenile probation 

collaboration—a version of the Index of 

Interdisciplinary Collaboration (ICC), 

adapted for treatment therapists and 

juvenile justice professionals, and

• time in treatment—percent of youth 

receiving at least the minimum number 

of hours of treatment required for the 

recommended treatment type

The MDFT-DC program was implemented more 

effectively than ESAU in a variety of ways. More 

youth recommended for treatment enrolled and 

remained in MDFT-DC compared to ESAU, and 

MDFT-DC youth and parents were more 

satisfied with the services received compared to 

those in ESAU. The authors also found a high 

level of collaboration among substance abuse 

treatment providers and juvenile justice 

professionals involved in administering the 

MDFT-DC program. 

While Liddle et al.’s study does not examine the 

outcomes of youth involved in these programs, 

the authors provide valuable information about 

program performance for each type of 

treatment provided. By demonstrating high 

enrollment, retention, adequate dosage of 

treatment, and satisfaction with treatment 

services, the study shows that the MDFT-DC 

program has a solid foundation for producing 

its intended outcomes.

NJJEC Bulletin is the newsletter of the National Juvenile Justice Evaluation Center. It provides information on NJJEC 

project activities, evaluation-related activities of OJJDP grantees, and upcoming trainings and conferences of interest.

To access NJJEC project information and resources: http://www.jrsa.org/njjec/

To subscribe: http://www.jrsa.org/njjec/newsletter.htm

To unsubscribe: e-mail njjec@jrsa.org

NJJEC is a Project of the Justice Research and Statistics Association. 

Notice of Federal Funding and Federal Disclaimer: This document was prepared under Grant Number 2010-JF-FX-0063 from 

the Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. Points of view or 

opinions in this document are those of the authors and do not necessarily represent the official position or policies of the U.S. 

Department of Justice.