NJJEC Bulletin September 2012

The NJJEC Bulletin

NJJEC is a project of the 

Justice Research and 

Statistics Association 

funded by the Office of 

Juvenile Justice and 

Delinquency Prevention

(OJJDP). NJJEC’s purpose is 

to improve the evaluation 

capacity of states, localities, 

and tribes and facilitate the 

use of evidence-based 

programs and practices in 

juvenile justice. The NJJEC 

Bulletin is disseminated 

directly to subscribers and 

posted on the NJJEC 

website, and provides 

information related to 

juvenile justice evaluation 

that is of interest to state, 

local, and tribal juvenile 

justice professionals. 

We welcome your 

participation! If you would 

like to submit an article, 

suggest a news item for the 

Bulletin, or ask a question 

about evaluation, e-mail 

njjec@jrsa.org. For more 

information about the 

resources available through 

the National Juvenile Justice 

Evaluation Center, visit the 

NJJEC website. 

What’s Going On at 

NJJEC?

NJJEC has released an online 

tutorial about the basics of 

performance measurement,

evaluation, and program design. 

The tutorial covers such topics as

selecting an evidence-based 

program, developing program 

logic and constructing a logic 

model, and establishing 

performance measures. 

The material is presented 

through the story of a fictional 

program manager trying to 

secure funding for her 

community center by using an 

evidence-based program. The 

tutorial is divided into modules 

so users can easily navigate 

through different topics to 

review the material. Users can 

assess their own knowledge by 

completing two quizzes, and 

have their results e-mailed to 

them upon completion. 

NJJEC staff members continue to

develop two Sustainability 

Toolkits to assist direct service 

providers and grant-making 

agencies. The toolkits will be 

available on the NJJEC website

this fall. 

Check out other trainings and 

presentations on the NJJEC 

website! 

Upcoming Events of 

Interest

American Society of 

Criminology Annual Meeting

November 2012

The American Society of 

Criminology will hold its annual 

meeting November 14-17 in 

Chicago, IL. Topics of interest 

include policing juveniles, 

juvenile corrections and 

detention alternatives, and 

youth-focused risk assessment. 

Be sure to check out the session

Want Data? We Got It! 

Information on and Access to 

OJJDP’s Juvenile Justice Data 

Collections on November 14.

Featured Resources:

Screening and 

Assessment Tools

We have received many 

questions about risk assessment 

and screening tools—where to

find them, how to know if they 

are appropriate for the 

population and problem at hand, 

and how to use them correctly. 

The Technical Assistance 

Partnership for Child and Family 

Mental Health has released a 

resource series dedicated to 

youth mental health issues in 

juvenile justice. One of these 
NJJEC Bulletin Page 2

resources focuses on screening and assessment 

tools to determine both the mental health 

needs of youth and the risk that youth will 

participate in delinquent activities in the 

future. 

Screening and Assessment in Juvenile Justice 

Systems: Identifying Mental Health Needs and 

Risk of Reoffending describes the uses of 

screening and assessment tools and provides 

specific examples of tools used to address 

juvenile mental health issues. The author notes 

that “screening” tools can be used widely to 

determine the general needs of all youth 

involved in the justice system, while 

“assessment” tools provide more in-depth 

information about a youth’s problems and the

appropriate course of treatment when such 

additional information is needed. 

This resource explains that it is important for 

juvenile justice professionals to consider the 

reason for using the tool, when and by whom it 

will be administered, whether it examines a 

question of interest to the administering 

practitioner or agency, and whether it has been 

supported by research evidence. Consider, for 

example, whether the tool will be administered 

by a clinical psychologist or a detention officer. 

The tool must be reliable (consistently 

measuring the same phenomena in the same 

way) and valid (measuring what it is intended 

to measure), but it is also important to consider 

the usefulness of this information once it has 

been collected and whether the accuracy of the 

information has been verified by research.

Let’s take a look the 10-Question Tool that was 

featured in the most recent edition of OJJDP’s 

Journal of Juvenile Justice article entitled, The 

10-Question Tool: A Novel Screening Instrument 

for Runaway Youth. The 10-Question Tool is a 

screening instrument: it is intended to be 

administered to all runaway youth encountered 

by police officers to provide some basic 

information about safety concerns related to 

runaway youth so that the officers can provide 

the most effective service referrals possible. 

Runaway youth have high rates of physical and 

sexual abuse, mental illness, and substance 

abuse, so the 10-Question Tool is designed 

specifically to examine these issues. The 

questions posed to youth by law enforcement 

officers include, “why did you leave home?’ and 

“has anyone hurt you or tried to hurt you while 

you were away from home?” The structure of 

the tool is simple and it can be completely 

quickly, so is appropriate for administration to 

a large number of youth—all runaways 

encountered by law enforcement—and can be 

administered by police officers without 

extensive background in counseling or 

psychology. 

The authors examined the prevalence of 

problems faced by runaway youth by gender 

and race, and attempted to verify much of the 

information provided by the youth and the 

likelihood that youth were referred to 

appropriate social services to assess the 

usefulness of the tool. The study found that 

many teens did in fact disclose drug and 

alcohol use, and the majority of teens who 

reported sexual assault victimization had not 

sought prior help for this abuse. 

The findings indicate that the screening tool 

might assist law enforcement in identifying 

assault victims who do not independently 

report their abuse. More complex, in-depth 

assessments can be used to determine the 

extent of a youth’s problems and the 

appropriate treatments, but the 10 Question 

Tool provides an adequate initial determination 

of how to deal with runaway youth.

Source: Vincent, G.M. (2011). Screening and Assessment in 

Juvenile Justice Systems: Identifying Mental Health Needs 

and Risk of Reoffending. Washington, DC: Technical 

Assistance Partnership for Child and Family Mental 

Health. 
NJJEC Bulletin Page 3

Evaluation Feature:

Using Anecdotal Evidence

Typically, when grant administering agencies 

ask for information about a program, they 

request data that has been systematically 

collected—that is, the same information has 

been collected for each participant involved in 

a program. These data are analyzed

collectively to make determinations about 

how well a program is functioning and its 

achievement of expected outcomes. 

Practitioners sometimes feel that this 

information does not completely illustrate 

some of the issues faced during 

implementation, or ways the program has 

affected participants that are not captured in 

data collection. Often, they want to use 

anecdotal evidence to tell the full story.

Evidence is “anecdotal” when a small, nonrepresentative sample of participating youth is 

used to illustrate program performance. For 

example, a program manager with 100 youth 

in a mentoring program might want to 

highlight the experiences of 5 students whose 

grades and attendance improved dramatically 

over the course of the program, rather than 

presenting grade and attendance changes for 

the entire group, or 50 randomly selected 

students. Because non-representative 

samples can tell a biased story about the 

successes and failures of a program, the use of 

anecdotal evidence is often discouraged.

However, there are some ways anecdotal 

evidence can be used to supplement 

systematically collected program data in order 

to illustrate how a program works to outside 

stakeholders. Anecdotal evidence can add 

richness and clarity to process and outcome 

quantitative data, particularly when 

illustrating the challenges involved in 

implementation. 

The Center for Court Innovation recently 

published an evaluation of New York’s QUEST 

Futures program that uses anecdotal evidence 

in conjunction with systematically collected 

data to illustrate program successes and 

challenges. The report includes quantitative 

case-level data (for example, youth 

demographic information, sources of referral 

into the program, youth’s history of mental 

illness) as well as quantitative measures 

aggregated by month (the number of referrals, 

screenings, active cases, etc. in the program). 

Qualitative data are also provided as a 

component of the process evaluation, including 

stakeholder interviews, meeting notes, court 

proceeding observations, and a review of 

relevant documents. 

The QUEST Futures evaluation includes 

anecdotal evidence in the form of six case 

studies of program youth (see page 47). These 

brief stories illustrate some of the complex 

issues involved in implementing the various 

components of the QUEST program plan, 

particularly the intersection of youth, school, 

and family and the interactions between mental 

health and substance abuse issues. 

The case studies include both successful and 

unsuccessful examples, lending credibility to 

the story that is being told throughout the 

narratives. The information might be 

particularly useful for an outsider with little 

knowledge of QUEST Futures or some of the 

compound issues involved in juvenile justice 

and mental health services for youth, and adds 

clarity to the data presented throughout the 

report. The reader is able to see the need for 

changes in treatment plan and approach, and 

the sometimes extensive limitations on the 

ability to complete a course of treatment as 

intended.

Some things to consider when you might use 
anecdotal evidence: 

1. Choose a realistic balance of success 

and failure. Both success and failure can 

be used to make program improvements. 

2. Use anecdotes to supplement 

systematically collected data, and 

clarify aspects of program operation 

that might otherwise be unclear. 

Anecdotal information should highlight or 

further clarify quantitative and qualitative 

data included in your report.

Source: Kenry, K. (2012). Mental Health Services for Youth: 

A Process and Outcome Evaluation of QUEST Futures. New 

York, New York: Center for Court Innovation.

State Spotlight: Arkansas

In recent years, the Arkansas juvenile justice 

system has undergone extensive reform. The 

state’s efforts are documented in Arkansas Youth 

Justice: The Architecture of Reform. Arkansas’ 

juvenile justice reform strategic plan sought to 

address overuse of confinement, the lack of 

information sharing and coordination among 

public agencies interacting with justice-involved 

youth, and more effectively address the variety of 

needs presented by youth in the juvenile justice 

system. The report details the importance of 

data and the use of evidence-based practices in 

improving the effectiveness of the system and 

reducing youth confinement. 

These efforts included the improvement of data 

collection across agencies at multiple points of 

contact through the creation of the Rite Track 

data management system, which is used by both 

the Division of Youth Services and child welfare 

agencies to improve coordination of efforts. 

Arkansas has also increased its use of validated 

risk assessment tools to better determine the 

needs of and most appropriate responses to 

youth in the system. 

DYS has also increased the requirements related 

to data collection for the youth service agencies 

and organizations it funds. Most agencies are 

required to work towards reduction in state 

commitments, and agencies achieving a 

substantially greater reduction (20%) are eligible 

for grant funds otherwise unavailable to them 

had they not met this benchmark. Further, 

service providers are contractually obligated to 

input into the System of Care data system used 

by the Department of Human Services.

The Arkansas reform process illustrates the 

importance of establishing baseline data and 

documenting change. 

Source: Arthur, P. & Hartney, C. (2012). Arkansas Youth 

Justice: The Architecture of Reform. Oakland, CA: National 

Center for Youth Law and National Council on Crime and 

Delinquency.

NJJEC Bulletin Page 4

NJJEC Bulletin is the newsletter of the National Juvenile Justice Evaluation Center. It provides information on NJJEC 

project activities, evaluation-related activities of OJJDP grantees, and upcoming trainings and conferences of interest.

To access NJJEC project information and resources: http://www.jrsa.org/njjec/

To subscribe: http://www.jrsa.org/njjec/newsletter.htm

To unsubscribe: e-mail njjec@jrsa.org

NJJEC is a Project of the Justice Research and Statistics Association. 

Notice of Federal Funding and Federal Disclaimer: This document was prepared under Grant Number 2010-JF-FX-0063 from the 

Office of Juvenile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of Justice. Points of view or 

opinions in this document are those of the authors and do not necessarily represent the official position or policies of the 

U.S. Department of Justice.