NJJEC Bulletin January 2013

The NJJEC Bulletin

NJJEC is a project of the 

Justice Research and 

Statistics Association 

funded by the Office of 

Juvenile Justice and 

Delinquency Prevention

(OJJDP). NJJEC’s purpose is 

to improve the evaluation 

capacity of states, localities, 

and tribes and facilitate the 

use of evidence-based 

programs and practices in 

juvenile justice. The NJJEC 

Bulletin is disseminated 

directly to subscribers and 

posted on the NJJEC 

website, and provides 

information related to 

juvenile justice evaluation 

that is of interest to state, 

local, and tribal juvenile 

justice professionals. 

We welcome your 

participation! If you would 

like to submit an article, 

suggest a news item for the 

Bulletin, or ask a question 

about evaluation, e-mail 

njjec@jrsa.org. For more 

information about the 

resources available through 

the National Juvenile Justice 

Evaluation Center, visit the 

NJJEC website. 

What’s Going On at 

NJJEC?

On December 6, 2012, NJJEC 

staff delivered a webinar 

entitled Sustainability: A 

Collaborative Approach. 

Panelists discussed the 

importance of performance 

measurement in 

sustainability, and ways grantmaking agencies and serviceproviders can work together 

to support program success. 

Thank you to all who 

attended! Webinar slides and 

narration will be available on 

the OJJDP NTTAC website

shortly. To view archived 

NJJEC webinars, check the 

NJJEC Trainings and 

Presentations page.

We are developing our spring 

webinars, and creating a series 

of short, narrated resources 

on program evaluation and 

performance measurement. 

The first in the series is 

entitled Hiring and Working 

With An Evaluator. This 

resource discusses the 

advantages of hiring an 

evaluator, how to select an 

evaluator, and how to develop 

and evaluation plan. Check 

the March edition of the 

Bulletin for more information 

and a link!

Upcoming Events of 

Interest

The 38th Annual Juvenile 

Justice Symposium and 

National DMC Conference

February 13-15

In conjunction with the 

Mississippi State Advisory 

Group, The Coalition for 

Juvenile Justice will hold its 

Annual Symposium and 

National DMC Conference 

February 13-15 in Biloxi, 

Mississippi. Sessions cover a 

wide variety of topics related 

to disproportionate minority 

contact and other issues of 

interest to the juvenile justice 

field, including evidencebased practice. Check the 

conference website for more 

information and to register!

2013 National Conference on 

Bullying

February 27-March 1

The School Safety Advocacy 

Council will host the 2013 

National Conference on 

Bullying February 27-March 1 

in Orlando, Florida. Session 

topics include methods for 

reporting and tracking 

bullying as well as evidencebased interventions. A full 

schedule and registration 

information are available 

online. 
NJJEC Bulletin Page 2

National Youth-At-Risk Conference

March 3-6

The College of Education and The Division of 

Continuing Education at Georgia Southern 

University will host the 24th Annual National 

Youth-At-Risk Conference March 3 through 6 in 

Savannah, Georgia. Conference session topics 

will include academic achievement, violence 

prevention, and youth mental and physical 

health. Check the conference website for 

registration information and to view the full 

program.

Featured Resources:

The MacArthur Foundation’s

Models For Change 

Knowledge Brief Series

The MacArthur Foundation’s Knowledge Brief 

Series provides information on effective 

practices in juvenile justice. The Briefs span a 

wide range of topics relevant to the field, 

including mental health issues, 

Disproportionate Minority Contact (DMC), costbenefit of juvenile interventions, and the work 

of the child welfare system. 

The briefings provide plain-language summaries 

of studies and evaluations conducted by the 

MacArthur Foundation, and include background 

information about the issue being examined as 

well as easy-to-interpret graphical displays of 

evaluation results. 

For example, the briefing Can Risk Assessment 

Improve Juvenile Justice Practice? provides 

results from a Foundation study of the use of 

risk assessment tools by juvenile probation 

officers. Risk assessment is generally viewed as 

an effective way to deliver targeted services to 

youth, and to avoid unnecessary interaction 

with the juvenile justice system for youth at low 

risk of reoffending. The study results include

the responsiveness of probation officers to the 

assessment recommendations, the effect of 

the use of these tools on the intensity of 

system response for high-risk and low-risk 

youth, and the effect of the tools on 

reoffending. The study found that probation 

officers were in fact making use of the risk 

assessment recommendations, and allocating 

more resources and services towards high-risk 

youth. However, at completion of the study 

there was not yet a significant reduction in 

reoffending. 

Check the MacArthur Foundation’s website to 

access the Knowledge Brief Series. 

Evaluation Feature:

Understanding Control Variables

One of the key challenges for practitioners 

who wish to participate in a program 

evaluation is having adequate data to provide 

an evaluator. We frequently discuss how to 

collect program data, but it is also very 

important to understand what to collect. 

Preparing for a true evaluation requires the 

collection of more than basic performance 

measures. In order to know what to collect, 

practitioners need to understand some of the 

components of an evaluation.

In an evaluation, a control variable is a 

characteristic that is known to be related to 

the outcome of interest. Because it is known 

to affect the outcome, a statistical analysis 

must be used to “control” that variable. 

Controlling on a variable means it is held 

constant so it is not a contributing factor on 

the effect of an intervention. Evaluators can 

use program participant data to make sure 

individual characteristics (control variables) 

are not affecting observed outcomes, and this 
NJJEC Bulletin Page 3

helps to establish that program activities

(independent variables) caused the results. It is 

important for program managers and staff to 

know variables that should be controlled for in 

an evaluation so they can collect the relevant 

data on program participants whenever possible. 

Collecting these data will help an evaluator work 

more efficiently and effectively to perform an 

outcome evaluation.

A quick look at publicly available resources can 

assist practitioners in identifying characteristics 

of youth that might be controlled for in an 

evaluation. For example, Can Risk Assessment 

Improve Juvenile Justice Practice?, the Knowledge 

Brief by the MacArthur Foundation which we 

referenced in the Featured Resources section, 

identifies dynamic risk factors for delinquency 

like poor school performance and a lack of social 

ties. The Center for Juvenile Justice Reform at 

Georgetown University has a wealth of resources 

on this topic. A 2012 report entitled Addressing 

the Unmet Educational Needs of Children and 

Youth in the Juvenile Justice and Child Welfare 

Systems identifies risk factors for system 

involvement such as poverty, learning 

disabilities, and adverse childhood experiences 

such as abuse. 

Understanding risk factors that should be used 

as control variables in an evaluation can help 

practitioners prepare to work with an evaluator. 

Service providers and program managers who 

are able to collect this information on program 

participants will be better prepared to equip an 

evaluator with the necessary tools to conduct an 

outcome evaluation.

Local Spotlight: 

Chittenden County, Vermont

Substance abuse is a complex problem and a 

pervasive issue in the justice system. Frequently, 

treatment is not readily available or easily 

accessed by justice-involved youth and adults. 

Chittenden County, Vermont has found a 

unique way to respond to this issue, and to 

reduce the length of time to referral for 

treatment. 

In partnership with Spectrum Youth and 

Family Services in Burlington, Vermont, the 

Chittenden County Criminal Court created the 

Rapid Referral program. The purpose of the 

program is to support pretrial substance abuse 

assessment. Spectrum provides substance 

abuse and mental health outpatient treatment 

for individuals up to age 23 using techniques 

such as motivational enhancement and 

cognitive behavioral therapies. Using pre-trial 

conditions of release, Rapid Referral assesses 

the dependency levels and treatment needs of 

offenders, and requires that those exhibiting a 

need for treatment participate in that 

treatment during the pretrial period. 

Defendants are assessed at arraignment, and 

identified candidates are provided with 

therapy immediately rather than delaying 

services until their adjudication dates. Within 

72 hours of arraignment, individuals with an 

alcohol or drug-related charge are ordered to 

an appointment for an evidence-based 

substance use/co-occurring assessment and 

treatment if indicated using American Society 

of Addiction Medicine (ASAM) placement 

criteria. The program is a new approach to the 

longstanding problem of repeat offenders with 

substance dependency. 

An outcome assessment of program 

participants showed a low recidivism rate of 

18.7 percent, but because there was no control 

group for comparison, that assessment could 

not rule out other factors beyond participation 

in Rapid Referral that might have affected 

recidivism for program participants. While 

the Court and Spectrum saw positive results in 
their participants, additional evaluation was 

needed to establish the link between program 

activities—in this case, pretrial treatment and

reductions in reoffending.

This is a problem faced frequently by youth 

service providers: providers see positive results, 

but are not able to establish through evaluation 

that program activities actually caused those 

results. Juvenile justice practitioners often find 

program evaluation to be infeasible, and one of 

the most common challenges is that they are 

unable to meet the “gold standard” of research 

methodology and randomly assign youth to one 

condition or another. For example, it might be

considered unethical or even illegal to 

randomly assign some youth to mandatory 

treatment and not others, other characteristics 

being equal.

To establish a link between program 

participation and lower recidivism, researchers 

at the Vermont Center for Justice Research

created a control group after the program had 

been implemented using criminal history and 

demographic characteristics available in the 

Vermont Criminal Information Center. The 

control group consisted of individuals 

processed through Chittenden County Court 

during the same period that the Rapid Referral 

program was evaluated who were similar to 

program participants with regard to fourteen

criminal history and demographic factors.

Statistical tests were conducted to confirm 

that the groups were equal on all of these 

important dimensions, so that an additional 

evaluation could compare Rapid Referral 

participants to offenders who were similar but 

had not received pretrial treatment. While 

this design is not a “true” experiment (also 

known as a Randomized Controlled Trial) 

because it did not involve random assignment, 

a matched-group comparison is still a strong 

way to evaluate program effectiveness. 

The evaluation results showed that the control 

group had a recidivism rate of 84.3 percent, 

substantially higher than the 18.7 percent of 

the treatment group who reoffended. Analysis

showed that the differences were statistically 

significant, meaning they were large enough 

that they were unlikely to be due to chance.

Because the treatment and comparison groups 

were similar on characteristics related to 

recidivism and the results were statistically 

significant, the study established that Rapid 

Referral does indeed lower the likelihood that 

program participants will reoffend.

The final report by the Vermont Center for 

Justice Research is available online.

NJJEC Bulletin Page 4

NJJEC Bulletin is the newsletter of the National Juvenile Justice Evaluation Center. It provides information on NJJEC 

project activities, evaluation-related activities of OJJDP grantees, and upcoming trainings and conferences of interest.

To access NJJEC project information and resources: http://www.jrsa.org/njjec/

To subscribe: http://www.jrsa.org/njjec/newsletter.htm

To unsubscribe: e-mail njjec@jrsa.org

NJJEC is a Project of the Justice Research and Statistics Association. 

Notice of Federal Funding and Federal Disclaimer: This document was prepared under Grant Number 2010-JF-FX-0063 from the 

Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. Points of view or 

opinions in this document are those of the authors and do not necessarily represent the official position or policies of the 

U.S. Department of Justice.