Juvenile Justice: DOJ Is Enhancing Information on Effective Programs, but Could Better Assess the Utility of This Information
Published: 2009-12-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: State juvenile justice systems face critical problems when it comes to juvenile delinquency issues such as reentry--when offenders return home from incarceration--and substance abuse. GAO was asked to review juvenile reentry and substance abuse program research and efforts by the Department of Justice's (DOJ) Office of Juvenile Justice and Delinquency Prevention (OJJDP) to provide information on effective programs (i.e., whether a program achieves its intended goal) and cost-beneficial programs (i.e., whether the benefits of programs exceeded their costs). This report addresses (1) expert opinion and available research on these types of reentry and substance abuse programs, (2) the extent to which OJJDP assesses its efforts to disseminate information on effective programs, and (3) OJJDP's plans to accomplish its research and evaluation goals. GAO, among other things, reviewed academic literature, and OJJDP's dissemination efforts and research goals. GAO also interviewed OJJDP officials and a nonprobability sample of 26 juvenile justice experts selected based on their experience with juvenile reentry and substance abuse issues.
section_title: What GAO Found
paragraphs: The majority of the juvenile justice reentry and substance abuse experts GAO interviewed cited evidence that shows cognitive behavioral therapy--programs that help individuals change their beliefs in order to change their behavior--and family therapy--programs that treat juveniles by focusing on improving communication with family members--are effective and cost beneficial when addressing reentry and substance abuse issues. For example, two juvenile reentry experts cited studies showing that 1 year after participating in a cognitive behavioral therapy program, participants were less likely to commit another offense than nonparticipants. Additionally, experts cited a study that reported that a family therapy program provides about $80,000 in savings per participant when accounting for savings from a decline in crime, such as the cost the police would have incurred. Most experts indicated that there was limited evidence on the effectiveness and cost benefits of reentry programs, such as aftercare--programs that assist juvenile offenders in returning to their communities during the reentry process--and substance abuse programs, such as drug courts--specialized courts that provide programs for substance-abusing juveniles and their families. GAO reviewed two OJJDP efforts that provide information on effective programs across the range of juvenile justice issues, the National Training and Technical Assistance Center (NTTAC) and the Model Programs Guide. OJJDP has mechanisms in place to regularly assess the utility of the information provided by NTTAC, but does not have such a mechanism for the guide. OJJDP ensures the utility of NTTAC's information through evaluations in accordance with federal guidelines that highlight the importance of regularly soliciting feedback from users. However, OJJDP could better ensure the utility of the information disseminated by the Model Programs Guide by having a mechanism in place to solicit regular feedback from members of the juvenile justice field--for example, program practitioners--that is specifically related to the guide. OJJDP has articulated research and evaluation goals to support its mission of improving the juvenile justice system and is developing plans to assist in meeting these goals. OJJDP is required under the Juvenile Justice and Delinquency Prevention Act, as amended, to publish an annual program plan that describes planned activities under accounts authorized for research and evaluation activities, among other things. Additionally, the Office of Management and Budget (OMB) recommended that OJJDP develop a comprehensive evaluation plan for juvenile justice programs. While OJJDP has not published an annual program plan since 2002, in December of 2009, it issued a proposed plan for public comment and aims to publish the final program plan once public comments are incorporated. Additionally, although the office has considered developing a comprehensive evaluation plan to address OMB recommendations, it had not previously done so because of a lack of resources. However, OJJDP is committed to developing a comprehensive evaluation plan once the program plan is finalized.

section_title: Letter
paragraphs: Although the juvenile arrest rate is near its lowest point in two decades,  critical problems affecting states’ juvenile justice systems remain, such as  recidivism—the act of committing new offenses after having been arrested  or convicted of a crime—and substance abuse—a pattern of use of illegal,  prescription, or nonprescription drugs leading to significant impairment in  functioning. Reentry, the return of juvenile offenders from residential  facilities back into their communities, aims to reduce recidivism by using  programs that promote the effective reintegration of juvenile offenders  and assist them in acquiring the life skills needed to succeed and become  law-abiding citizens. According to the Department of Justice (DOJ),  approximately 100,000 juvenile offenders are held in residential facilities  in the United States on a given day, and most of these juveniles will likely  leave these facilities and return home to their communities each year.  Further, DOJ has estimated that 62 percent of juveniles in residential  facilities self-reported having had at least one prior commitment in the  juvenile justice system. Additionally, substance abuse plays a significant  role in juvenile crime.
In addition, in a 2008 annual survey of members conducted by the Federal  Advisory Council on Juvenile Justice, reentry of offenders into  communities and schools (28 states) and substance abuse (21 states) were  two of the top three programmatic issues most frequently reported as  affecting these states’ juvenile justice systems. Specifically, concerns  were noted about the minimal number of programs available to help  juveniles reenter communities, the large number of inexperienced  practitioners operating programs, and the limited substance abuse  treatment options. Related to research and policy, the most common issue  state officials cited was the need to identify effective practices for juvenile  programs (23 states) followed by the need to enhance the states’  capacities to conduct juvenile research and collect data about juvenile  programs (20 states). Respondents also cited the need for federal agencies  and state governments to collaborate on conducting research and  evaluations to provide a better explanation of (1) practices that have been  evaluated and have been shown to be effective, (2) the need for more  training from federal agencies on how to conduct evaluations, and   (3) increased funding to establish evaluation procedures (23 states).
The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is  DOJ’s office charged with providing national leadership, coordination, and  resources to help prevent and respond to juvenile delinquency and  victimization. OJJDP supports states and communities in their efforts to  develop and implement effective programs to prevent delinquency and  intervene after a juvenile has offended, among other things. For example,  from fiscal years 2007 through 2009, Congress provided OJJDP almost   $1.1 billion to use for grants to states, localities, and organizations for a  variety of juvenile justice programs. In support of its mission, the office  also disseminates information to states and communities related to a  number of juvenile justice issues, including effective reentry and  substance abuse programs, and funds research and evaluations of juvenile  justice programs.
States and localities develop and implement programs that specifically  target facilitating reentry and reducing substance abuse; therefore, it is  important that federal, state, and local agencies that provide grants for  programs, as well as practitioners operating programs, have information  about which ones are effective and cost beneficial. Having such  information will better position these agencies to help ensure that federal,  state, and local funds are well spent. In general, effectiveness is  determined through program evaluations, which are systematic studies  conducted to assess how well a program is working—that is, whether a  program produced its intended results or effects. Additionally, cost-benefit  analyses can help determine if the dollar value of a program’s success— such as a reduction in recidivism—exceeds the cost of the program. To  help ensure the effective use of grant funds for juvenile reentry and  juvenile substance abuse programs, you asked us to review the available  research as well as OJJDP’s efforts to provide information about effective  programs to the juvenile justice field, that is, program practitioners and  communities. In addition, you asked us to provide information on OJJDP’s  research and evaluation planning. Specifically, this report addresses the  following questions:    What do expert opinion and available research indicate about the types of  juvenile reentry programs and juvenile substance abuse programs that are  effective or cost beneficial?
To what extent does OJJDP have efforts under way to disseminate  information about effective juvenile justice programs and assess the utility  of the information it is providing through these efforts?
To what extent does OJJDP have plans in place to accomplish its juvenile  justice research and evaluation goals?
subsections: section_title: Scope and Methodology
paragraphs: To determine what experts and the available research indicate about the  types of reentry programs and substance abuse programs that are effective  or cost beneficial for juvenile offenders, we reviewed relevant literature,  studies, and federal resources for juvenile justice programs, and  interviewed federal officials and 26 juvenile justice experts. Specifically,  to identify the types of programs to review, we conducted a literature  search for studies and articles, including evaluations of juvenile reentry  and juvenile substance abuse programs in the United States that were  published from May 30, 1999, through May 30, 2009. We chose this time  frame, the past 10 years, because it provided us with an overview of the  available research, including unpublished and ongoing studies, which  assesses the effectiveness of reentry and substance abuse programs. We  also consulted with OJJDP officials who coordinate research on juvenile  justice programs and Department of Health and Human Services officials  who oversee substance abuse and adolescent programs to obtain their  recommendations for repositories—online databases that contain  information on effective programs—and research studies and relevant  Web sites for identifying types of reentry and substance abuse programs.  Using these recommendations, information from relevant literature, and  categories of program types used by OJJDP’s Model Programs Guide, we  identified five types of juvenile justice programs that are used to address  reentry issues and five types of programs that are used to address  substance abuse issues for juvenile offenders.
Specifically related to substance abuse, we focused on substance abuse  programs that involved relapse prevention treatment for juvenile offenders  with substance abuse histories. After consulting with experts and  reviewing the literature, we excluded juvenile alcohol abuse programs and  substance abuse programs for the general juvenile population as well as  at-risk juveniles who are prone to, but have not yet developed, substance  abuse problems. For instance, we excluded after school or recreation  programs, conflict resolution programs, and school or classroom  programs. While all of these programs may have a substance abuse  component, this component is not designed to address juvenile offenders’  actual substance abuse problems.
After identifying the types of programs to be reviewed, we looked at online  databases, academic research, and professional organizations to select  subject matter experts—researchers and practitioners—to obtain their  views on the types of programs that have been shown to be effective or  cost beneficial and the basis they used for making such determinations.  We specifically identified researchers who focus on juvenile reentry issues  or substance abuse issues and practitioners who operate programs that  address these issues. We chose 26 experts to interview as a result of this  process. Specifically, we selected 13 individuals with expertise related to  juvenile reentry programs, 7 individuals with expertise related to juvenile  substance abuse programs, and 6 individuals with both juvenile reentry  and substance abuse program expertise. We selected these experts based  on several criteria, including their employment histories related to juvenile  reentry and substance abuse programs and the number of years they spent  studying, evaluating, or managing programs addressing juvenile reentry or  substance abuse issues. We evaluated their experience by reviewing the  studies the researchers had completed and determining the experience the  practitioners had managing the types of juvenile reentry and substance  abuse programs selected for our review. See appendix I for the list of  experts we interviewed.
We asked these experts to provide their views about the effectiveness of  program types (e.g., drug courts), rather than about the effectiveness of  individual intervention programs (e.g., a specific drug court program that  was implemented in one county). Because the Model Programs Guide,  like other online repositories, contains information about the effectiveness  of individual intervention programs, it does not provide information about  the effectiveness of program types. As a result, we were interested in  obtaining the experts’ consolidated views of the effectiveness of program  types. We also asked the experts to identify other program types—in  addition to those that we explicitly asked about—that they considered to  be effective or cost beneficial, but no additional program types were  mentioned. In addition, we asked the experts to identify factors that in  their view could help programs to achieve intended outcomes, such as  reducing participants’ recidivism, which are summarized in appendix II.  While the results of these interviews cannot be generalized to reflect the  views of all experts knowledgeable about juvenile reentry or substance  abuse programs, we believe the interviews provided us with a good  overview of the available research and valuable information about what  program types are considered to be effective by subject matter experts. In  addition, while we did not assess the methodological rigor of studies and  evaluations in our review, we corroborated expert testimony by reviewing  and summarizing the studies or evaluations that experts cited as the basis  for their opinions. We also provided the experts with a summary of their  opinions to review in order to ensure that we correctly captured their  views.
To identify the extent to which OJJDP has efforts under way to  disseminate information about effective juvenile justice programs and  assess the extent to which OJJDP ensures the utility of the information  provided, we reviewed documentation, such as OJJDP’s annual reports  outlining information dissemination efforts, OJJDP publications, and a  contract related to disseminating training information on effective  programs. We interviewed knowledgeable OJJDP officials, such as the  Training Coordinator and communications policy personnel, about  OJJDP’s efforts to disseminate information about effective programs. We  selected two of OJJDP’s efforts through which it disseminates information  about effective programs—the Model Programs Guide and the National  Training and Technical Assistance Center (NTTAC), which provides  training and support to the juvenile justice field in identifying and  implementing effective programs—because they provide information  about effective programs across the range of issue areas in which OJJDP is  involved, including reentry and substance abuse programs. We then  compared these efforts to guidance articulated by the Office of Justice  Programs (OJP), which oversees OJJDP, and in prior GAO reports that  stresses the importance of assessing whether the information  disseminated is meeting the needs of its users. We also interviewed  representatives from the two organizations that manage these two  information dissemination efforts. Additionally, we asked the 26 juvenile  reentry and substance abuse experts we interviewed about their views  regarding OJJDP’s information dissemination efforts and their opinions  about the effectiveness of these efforts. Although their views cannot be  generalized to the entire juvenile justice field, we believe that the experts  provided us with a good overview of the utility of the information  disseminated by OJJDP. We did not contact recipients of the information  OJJDP disseminates for their views on the usefulness of the information  provided because of the large volume of recipients and the resulting cost  that would be incurred to obtain this input.
To assess the extent to which OJJDP has plans in place for its research  and evaluation efforts, we reviewed relevant laws related to the office’s  role in supporting research and evaluations of juvenile justice programs.  We also reviewed relevant DOJ and OJJDP documentation, such as annual  reports and strategic plans that contain information on OJJDP’s research  and evaluation goals and plans. We interviewed cognizant OJJDP officials  about the office’s planning efforts related to research and evaluation. We  also reviewed criteria found in standard practices for program  management and our prior products that highlight the importance of  developing plans to meet goals and help ensure that resources are used  effectively, and then compared these criteria to OJJDP’s stated plans.  Additionally, we analyzed OJJDP funding and staff data for fiscal years  2005 through 2009 to better understand the resources the office has had  available to support its evaluation activities. We chose these years because  they provide the most recent overview of OJJDP’s research and evaluation  funding.
We conducted this performance audit from April 2009 through December  2009 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Background
paragraphs: Within states’ juvenile justice systems, reentry aims to promote the  effective reintegration of juvenile offenders back into communities upon  release from residential facilities. Reentry is a process that incorporates a  variety of programs to assist juvenile offenders in the transition from  residential facilities to communities. In addition, reentry is intended to  assist juvenile offenders in acquiring the life skills needed to succeed in  the community and become law-abiding citizens and can incorporate the  use of education, mental health, drug rehabilitation, or vocational training  programs. While reentry begins after a juvenile is released back into the  community, to help ensure a seamless transition, a reentry process begins  after sentencing, then continues through incarceration, and into the period  of release back into the community. According to OJJDP, juvenile justice  practitioners and researchers believe that providing supervision and  services to juvenile offenders returning to the community will reduce the  high rate of recidivism among these juveniles. Several types of programs  address juvenile reentry issues, as described in table 1.
Substance abuse includes, but is not limited to, the use or abuse of illegal  drugs (e.g., heroin), prescription drugs, and nonprescription drugs (e.g.,  over-the-counter medications available without a prescription, such as  cough suppressant). Treatment of substance abuse may occur in a variety  of different settings, such as in clinics on an outpatient basis or at a  hospital. Treatment can also occur in short- and long-term residential  facilities that range from secure environments where juveniles’ activities  are physically restricted, to group homes, which are nonsecure settings  where juveniles live and receive services in a homelike environment.  According to the Center for Substance Abuse Treatment, juveniles with  addictions to substances can be helped through programs that specifically  target the factors associated with substance abuse—such as a family  history of such abuse. For example, substance abuse intervention  programs, such as cognitive behavioral therapy and family therapy, aim to  change a juvenile’s behavior by focusing on improving a juvenile’s  response to situations that contributed to prior substance abuse.  Substance abuse intervention programs can be provided to juvenile  offenders throughout the juvenile justice system: after sentencing, during  incarceration, and after release back into the community. Whether  treatment occurs while a juvenile is incarcerated or after the juvenile is  released into the community, according to OJJDP, effective intervention  programs can help addicted juveniles to overcome their substance abuse,  lead crime-free lives, and become productive citizens. Table 2 describes  types of programs—in addition to cognitive behavioral therapy and  wraparound/case management, which are discussed in table 1—that  address juvenile substance abuse issues.
The Juvenile Justice and Delinquency Prevention Act (JJDPA) established  OJJDP in 1974. As the only federal office charged exclusively with  preventing and responding to juvenile delinquency and with helping states  improve their juvenile justice systems, OJJDP supports its mission through  a variety of activities. For example, OJJDP administers a wide variety of  grants to states, territories, localities, and public and private organizations  through formula, block, and discretionary grant programs; provides  training and technical assistance; produces and distributes publications  and other products containing information about juvenile justice topics;  and funds research and evaluation efforts. In fiscal year 2009, the total  appropriation for juvenile justice programs was about $374 million. See  appendix III for more detailed information on OJJDP’s enacted  appropriations for fiscal years 2007 through 2009.
OJJDP, through its various grant programs, has provided funding to states  and organizations to support juvenile reentry and substance abuse  programs, although the JJDPA does not specifically require OJJDP to fund  them. States generally have the authority to determine how formula and  block grants are allocated and may use these funds to support a range of  program areas, including programs specifically for reentry or substance  abuse. For example, from fiscal years 2007 through 2008, OJJDP reported  that states used approximately $7.1 million in applicable formula and  block grant funds for programs that target reentry and $19 million in  formula and block grant funds for programs that target substance abuse,  representing approximately 1.8 percent and 4.5 percent, respectively, of  such funding for those years. Additionally, from fiscal years 2007 through  2009, OJJDP awarded a total of approximately $33 million in discretionary  grants through four juvenile reentry grant programs and three substance  abuse grant programs. Specifically, in the area of reentry, OJJDP awarded  a total of $25.4 million to 38 grantees under 4 programs, and in the area of  substance abuse, OJJDP awarded a total of $7.6 million to 15 grantees  under 3 programs. See appendix IV for more detail on funding for these  reentry and substance abuse programs.
section_title: Experts Cite Evidence from Available Research Indicating That Cognitive Behavioral Therapy and Family Therapy Programs Are Effective and Cost Beneficial When Addressing Reentry and Substance Abuse Issues
paragraphs: Of the five reentry program types we reviewed, reentry experts reported  that there is evidence from available research that cognitive behavioral  therapy reduces recidivism. While experts cited a lack of evidence  demonstrating that wraparound/case management, aftercare, and  vocational/job training were effective in achieving results, such as a  reduction in recidivism, they generally provided positive views on the  potential results of these three types of programs, based on their own  experience or knowledge of them. Similarly, of the five substance abuse  program types we reviewed, juvenile substance abuse experts reported  that there is evidence from available research that cognitive behavioral  therapy along with family therapy are effective at reducing recidivism and  show successful results at reducing substance abuse. However, expert  opinions regarding other substance abuse program types, such as drug  courts, mentoring, and wraparound/case management, were mixed, with  experts stating that these program types could be effective, they were  ineffective, or there was not enough evidence to determine effectiveness.  Furthermore, both reentry and substance abuse experts cited studies  indicating that cognitive behavioral therapy and family therapy programs  are cost beneficial; however, the experts cited limited evidence for  determining the costs and benefits of the other programs we reviewed.
subsections: section_title: Reentry Experts Cited Evidence That Cognitive Behavioral Therapy Is Effective at Reducing Recidivism but Concluded That Other Programs Lack the Evidence Necessary to Determine Their Effectiveness
subsections: section_title: Experts Cited Evidence from Available Research Demonstrating That Cognitive Behavioral Therapy Is Effective at Reducing Recidivism
paragraphs: Eleven of the 12 experts we interviewed who provided comments based  on their knowledge and experience with cognitive behavioral therapy  stated that evidence from available research shows that these programs  can be effective at reducing recidivism. Cognitive behavioral therapy  intervention programs are designed to identify and provide juveniles with  the skills to change thoughts and behaviors that contribute to their  problems. The underlying principle of these programs is that thoughts  affect emotions, which then influence behaviors. These intervention  programs combine two kinds of psychotherapy—cognitive therapy and  behavioral therapy. The strategies of cognitive behavioral therapy have  been used to, among other things, prevent the start of a problem  behavior—such as violence and criminal activity—or stop the problem  behavior from continuing. A juvenile offender can receive this type of  intervention program after sentencing, throughout incarceration, or after  returning to the community. For example, a cognitive behavioral therapy  intervention program may provide individual and family services to treat a  juvenile offender who has mental health and substance abuse issues. The  treatment can occur during the juvenile’s transition from incarceration  back into the community and help the juvenile lower the risk of  recidivism, connect the family with appropriate community support, assist  the juvenile in abstaining from drugs, and improve the mental health of the  juvenile.
Based on their assessment of the available research, these 11 experts  stated that cognitive behavioral therapy programs have been shown to be  effective. Experts identified two meta-analyses of cognitive behavioral  therapy programs that demonstrated effectiveness. One such study  concluded that effective cognitive behavioral therapy programs are  characterized by the low proportion of juveniles who dropped out of the  program, as well as the close monitoring of the quality of the treatment  and adequate training for the providers. In addition, this same study also  found that 12 months after treatment, the likelihood of a juvenile who  received cognitive behavioral therapy not recidivating was about one and  a half times greater than for a juvenile who did not receive the therapy.  This study also reported that the effects of cognitive behavioral therapy  were greater for offenders who had a higher risk of recidivism than those  with a lower risk. Specifically, the best results, in terms of recidivism  reductions, occurred when high-risk offenders received more intensive  treatment that targeted criminal thinking patterns. A second study also  reported that among therapeutic interventions, such as skill building,  cognitive behavioral therapy was most effective at reducing recidivism.  The 12th expert stated that the particular cognitive behavioral therapy  intervention program he was using—aggression replacement training®— had not been evaluated at his particular program site, so he could not  draw conclusions as to its effectiveness.
section_title: Reentry Experts Indicated Positive Views of Wraparound/Case Management, Aftercare, and Vocational/Job Training Programs, but Stated That These Programs Lacked Evidence to Demonstrate Effectiveness
paragraphs: Despite having generally positive views on the results of wraparound/case  management, aftercare, and vocational/job training programs based on  their experience or knowledge of these programs, reentry experts  reported a lack of evaluations that show conclusive evidence about the  effectiveness of these programs. Specifically, of the nine experts who  provided comments on wraparound/case management programs, eight  offered positive opinions about these programs. For example, two of these  experts commented that wraparound/case management can be successful  at reducing recidivism, depending on the quality and availability of  services provided to juveniles. However, two of these eight experts also  stated that there was a lack of evaluations demonstrating the effectiveness  of wraparound/case management programs. One of these experts pointed  us to a study on a specific wraparound/case management intervention  program, Wraparound Milwaukee, that showed potentially promising  results related to a reduction in recidivism rates for juvenile offenders.  However, another expert cautioned that initial evaluations of  wraparound/case management programs did not conclusively demonstrate  the effectiveness of wraparound/case management programs. Finally, the  ninth expert stated that in her experience, wraparound/case management  interventions are not effective because, for example, juveniles are placed  into these interventions based on the availability of program staff and  resources rather than program services being tailored to the individual  needs of each juvenile.
In addition, 7 of the 15 experts who commented about aftercare programs  opined that aftercare interventions are important reentry programs, in  part, because they link a juvenile with the community and provide regular  contact with a caseworker. However, 6 other experts said there was  inconclusive evidence to determine whether these programs can be  effective in achieving results. Three of these experts based their opinions  on an evaluation of the Intensive Aftercare Program that showed  inconclusive results about program effectiveness. Specifically, the study  found no evidence that the program had its intended impact of reducing  recidivism among juveniles who were released back into the community  under supervision in the three states that piloted the program. However,  the evaluation did find that the three states that implemented the Intensive  Aftercare Program model successfully incorporated most of its core  features, which prepared juveniles to transition back into the community.  For instance, these states created new Intensive Aftercare Programs— specific treatment programs that among other things, prepared juveniles  for increased responsibility in the community, facilitated interaction with  the community, and worked with the juveniles’ schools and families. The  state programs had a large percentage of juveniles involved in various  treatment services. Despite the inconclusive results of the study, 1 expert  credited the aftercare program model with addressing the issue of  juveniles interacting with multiple probation officers throughout the entire  reentry process because aftercare programs, in general, assign one  probation officer to a juvenile as a consistent point of contact. The  remaining 2 of 15 experts opined that aftercare intervention programs had  not been shown to be effective at achieving desired results because, for  example, the treatment a juvenile receives depends on the services  available in the community.
With respect to vocational/job training programs, 10 of the 11 reentry  experts who commented on these programs expressed positive opinions  about the programs’ potential outcomes but noted that there had been  limited research conducted to demonstrate their effectiveness.  Specifically, experts noted that vocational/job training programs could be  beneficial if they were applied to older juveniles and if they led to those  juveniles getting jobs. The remaining expert said there is little evidence to  demonstrate the effectiveness of these intervention programs. For a more  detailed description of reentry experts’ opinions about these program  types, see appendix V.
section_title: Substance Abuse Experts Cited Evidence That Cognitive Behavioral Therapy and Family Therapy Are Effective at Reducing Recidivism and Can Help to Reduce Substance Abuse, but Said That Evidence of Effectiveness Was Lacking for the Other Programs
subsections: section_title: Experts Cited Evidence from Available Research Showing Cognitive Behavioral and Family Therapies Effectively Reduce Recidivism and Demonstrate Success at Reducing Substance Abuse
paragraphs: All of the 13 substance abuse experts we interviewed stated that based on  available research, cognitive behavioral therapy effectively reduces  recidivism and has demonstrated success at reducing substance abuse.  Experts cited six studies to support their opinions, two of which were the  same studies cited by reentry experts that demonstrate that cognitive  behavioral therapy is effective at reducing recidivism. Two of the  substance abuse experts noted that few studies have been conducted to  determine whether an intervention program is effective at specifically  reducing substance abuse. However, 3 experts also noted that within the  last decade, newly emerging research has shown promising results with  respect to cognitive behavioral therapy program types reducing substance  abuse. For example, these experts pointed us to three studies that report  that juveniles who participated in these programs showed reductions in  marijuana use.
Twelve of the 13 substance abuse experts we interviewed who provided  comments based on their knowledge and experience stated that family  therapy programs are effective at reducing recidivism or decreasing  substance use. Family therapy uses trained therapists to treat juvenile  offenders with substance abuse problems by including families of  juveniles in the treatment and focusing on improving communication and  interactions among family members and improving overall relationships  between juveniles and their families. This type of therapy focuses on the  family as it is the primary and sometimes only source for emotional  support, moral guidance, and self-esteem for juveniles. Family habits, such  as failing to set clear expectations for children’s behavior, poor monitoring  and supervision, and severe and inconsistent discipline can often lead to  juveniles engaging in delinquency and substance abuse, according to  OJJDP. For example, family drug use often results in adolescent drug use.
Based on their assessment of the available research, these 12 experts  provided positive opinions about the effectiveness of family therapy, and 7  of these experts cited 9 studies that support their opinions. These studies  demonstrated, for example, that multisystemic therapy—a family therapy  intervention program that helps parents identify strengths and develop  natural support systems (e.g., extended family, neighbors, friends, and  church members)—is an effective intervention program for reducing  recidivism and substance use because, for example, juveniles who  participated in multisystemic therapy programs engaged in significantly  less criminal activity than did nonparticipants. Specifically, multisystemic  therapy participants had fewer average convictions per year for violent  crimes than those juveniles who did not participate in the program.  Additionally, analyses of drug tests demonstrated significantly higher rates  of drug abstinence for program participants than for nonparticipants. One  study also showed that participants in functional family therapy, another  family therapy intervention program, had 50 percent reductions in  substance use as compared to juveniles who did not participate in the  program.
section_title: Experts Had Mixed Views on Effectiveness of Drug Courts, Mentoring, and Wraparound/Case Management Types of Programs
paragraphs: According to the 10 experts who commented on drug courts, 5 stated that  there is a lack of evidence to determine program effectiveness, while  another expert stated that drug courts are ineffective types of programs  because they expose first-time offenders to more serious drug users. The  remaining 4 experts stated that drug courts can be effective if, for  example, they are combined with other effective intervention programs,  such as multisystemic therapy. Similarly, according to the 8 experts who  commented on mentoring, 1 stated that there are too few evaluations to  determine effectiveness, while 4 stated that mentoring programs alone are  ineffective or unsuccessful at achieving desired results and that mentoring  intervention programs are more effective at preventing at-risk juveniles  from engaging in delinquent behavior. However, 3 experts thought  mentoring intervention programs could be effective if the programs adhere  to certain factors that have been evaluated and shown to be effective, such  as the mentor being properly trained. Finally, experts also had mixed  views on the effect of wraparound/case management types of programs.  Of the 11 experts who commented on these programs, 7 experts stated  that wraparound/case management is effective or can be effective if, for  example, wraparound/case management is combined with another  intervention program that has been evaluated and has shown to be  effective, such as cognitive behavior therapy. Conversely, 4 experts either  stated that these programs are ineffective because, for example, the  intervention programs lack follow-through as there are no consequences if  a juvenile does not show up for treatment, or there is not sufficient  evidence to determine effectiveness. For a more detailed description of  substance abuse experts’ opinions about these program types, see  appendix VI.
section_title: Reentry and Substance Abuse Experts Cited Available Research That Indicates Cognitive Behavioral and Family Therapies Are Cost Beneficial, but Provided Limited Evidence of the Costs and Benefits of the Other Program Types
paragraphs: While program evaluations establish if a program is effective in producing  its intended results or effects—such as a reduction in recidivism—cost- benefit analyses use program evaluation to determine if the dollar value of  a program’s benefits exceeds the costs to deliver the program. For  example, if a program evaluation shows that an intervention program  reduces the number of offenses committed by juveniles from three to one,  a cost-benefit analysis would first determine a dollar value for each of the  offenses. Then, the cost-benefit analysis would estimate whether the  savings of going from three offenses to one offense is more or less costly  than the amount of money required to deliver the intervention program, as  compared to an alternative program the juvenile would have received. The  intervention may not always be more expensive than the alternative. For  example, if the alternative is incarceration, the intervention program may  be less expensive—meaning that the intervention program can be cost  beneficial even if it does not result in a reduction of offenses. By applying  the same cost-benefit analysis techniques to evaluations of different  program types, decision makers can make comparisons among  alternatives and determine which program types offer the greatest benefits  for the least cost. The results of a cost-benefit analysis are often  represented as a net benefit, meaning total benefits minus total cost.
Of the 26 reentry and substance abuse experts we interviewed, 19  provided information related to the cost benefits of the reentry and  substance abuse program types in our review. These 19 experts identified  five cost-benefit analyses of juvenile justice programs consisting of four  meta-analyses and one systematic review. The studies demonstrate that  various cognitive behavioral therapy and family therapy intervention  programs are cost beneficial because they are effective at reducing crime  and are expected to produce more benefits than costs compared to the  alternative. For example, in one study, the authors reviewed several  program interventions that fall into the family therapy program type, such  as multisystemic therapy and multidimensional treatment foster care.  The authors analyzed three program evaluations of multidimensional  treatment foster care and found that this intervention can be expected to  reduce crime outcomes by 22 percent. Based on this reduction in crime,  the authors of the study predict that the intervention provides about  $80,000 worth of benefits per participant. This dollar value reflects the  savings per participant that result from a decrease in criminal activity,  including savings to crime victims, police and sheriff’s office costs, and  juvenile detention costs, among others.
The four studies cited by the experts show mixed or inconclusive results  for drug courts, vocational/job training, and mentoring program types. For  example, one study found that juvenile drug courts are cost beneficial  because they are expected to have a net benefit of $4,622 per program  participant. The other studies could not determine drug courts’ cost- effectiveness because they either did not include program evaluations of  drug court programs or they found mixed results in the program  evaluations analyzed and therefore could not determine the net benefits.  In addition, two studies found that there are too few evaluations of  vocational/job training or mentoring in juvenile justice programs to  calculate if the benefits of these program types outweigh the costs. The  remaining program types in our review—wraparound/case management,  aftercare, and reentry courts—were not analyzed in these studies. Table 3  presents a summary of these studies.
In addition, seven experts also commented on reentry and substance  abuse programs that were not included in the cited studies. For example,  three experts opined that wraparound/case management programs may  eventually be proven to be cost beneficial, based on preliminary research  and evaluations. For example, one expert cited an unpublished study of a  wraparound program pilot project that showed that recidivism of program  participants was low, and that program costs were approximately 60  percent of the costs of incarcerating juveniles. Additionally, although  experts did not cite cost-benefit analyses of aftercare program types, four  reentry experts stated that such programs could be cost beneficial if the  intervention program being delivered is effective because the cost of  incarceration is so high. Three experts we interviewed stressed that even  though some intervention programs that have been shown to be effective  are expensive, if they reduce recidivism, they might be cost beneficial  because of the high cost of incarcerating juveniles.
section_title: OJJDP Has Mechanisms in Place to Ensure Training and Technical Assistance Meet Users’ Needs, but Regular Feedback on the Model Programs Guide Would Help OJJDP Better Assess Information Utility
paragraphs: Consistent with the JJDPA, OJJDP has several efforts under way to  disseminate information about effective juvenile justice programs. Two of  these efforts—NTTAC and the Model Programs Guide—provide  information about effective programs for a range of juvenile justice issues,  including reentry and substance abuse issues. Consistent with federal  guidelines for ensuring the utility of information, OJJDP has established  mechanisms to ensure that the information provided through its training  and technical assistance efforts meets the needs of the juvenile justice  field. However, OJJDP could better ensure the usefulness of the  information it disseminates through the Model Programs Guide by having  a mechanism in place to solicit regular feedback specifically related to the  guide from the juvenile justice field.
subsections: section_title: OJJDP Disseminates Information about Effective Juvenile Justice Programs through Several Efforts
paragraphs: According to the JJDPA, OJJDP is authorized, but is not required, to  provide information about juvenile justice issues and programs and to  provide training and technical assistance to help the juvenile justice field  implement and replicate such programs. In accordance with this  authority and its mission to support states and communities in their efforts  to develop and implement effective juvenile justice programs, OJJDP  disseminates information related to these programs through a range of  efforts, from those designed to meet the needs of the juvenile justice field  as a whole to those that focus on effective programs in a specific issue  area, such as gang prevention or girls’ delinquency. OJJDP distributes the  broadest range of information on juvenile justice topics through the  Juvenile Justice Clearinghouse (Clearinghouse). Through its services, the  Clearinghouse offers, among other things, the latest research findings and  statistics, publications on juvenile justice issues and programs,  announcements of funding opportunities, and other resources prepared by  a variety of researchers in juvenile justice. As part of its efforts, the  Clearinghouse responds to requests for information about effective  programs by directing users to OJJDP efforts that develop and disseminate  information about effective programs, such as NTTAC and the Model  Programs Guide. Thus, we focused on NTTAC and the Model Programs  Guide because they provide information about effective programs across  the range of issue areas in which OJJDP is involved, including reentry and  substance abuse programs. OJJDP also disseminates information about  effective programs in specific issue areas through various centers, such as  the National Youth Gang Center and the Underage Drinking Enforcement  Center. For a more detailed discussion of these centers and other  information dissemination efforts that focus on specific issues, see  appendix VII.
NTTAC was established in 1995, in part to provide information about  effective juvenile justice programs—such as programs that address issues  related to reentry and substance abuse—through its training and technical  assistance efforts. According to OJJDP, NTTAC works to promote the use  of effective programs in the field through training and technical assistance  programs. Additionally, NTTAC develops training materials and resources,  and customizes the information included in its curricula in an effort to  best meet the needs of its training and technical assistance recipients. In  terms of its efforts specifically related to program effectiveness, NTTAC  provides training and technical assistance for members of the juvenile  justice field on how to develop and sustain effective programs, and to help  the field understand programs that are effective for various juvenile  populations, such as juveniles with mental health issues or female  offenders.
The Model Programs Guide is an online database that contains summary  information about approximately 200 juvenile justice programs, from  prevention programs to reentry programs. It is designed to help  practitioners and communities identify and implement prevention and  intervention programs that have been evaluated and have been shown to  be effective. Programs in the Model Programs Guide may focus on a  range of issues, including delinquency, violence, youth gang involvement,  substance abuse, or academic issues, and can include, but are not limited  to, delinquency prevention, community service, drug courts, or family  therapy. To be included in the Model Programs Guide, programs are  reviewed and rated along several dimensions, including such factors as  whether an evaluation of the program established a causal association  between the treatment and the outcome. Users can search the Model  Programs Guide to find programs that meet their specific needs. For  example, users can look for a program that has been shown to be effective  for juveniles with substance abuse problems who are first-time offenders,  or they can search for a program that has been shown to be effective for  juveniles involved in gang activities who are reentering the community.
section_title: Evaluations and Needs Assessment Help OJJDP Ensure Usefulness of Information Provided by Training and Technical Assistance Efforts, but OJJDP Could Better Ensure the Utility of the Model Programs Guide’s Information through Regular Feedback
paragraphs: In accordance with federal guidelines from OJP and prior GAO work,  OJJDP has mechanisms in place to regularly conduct evaluations and is  currently conducting a needs assessment to ensure the usefulness of the  information provided by its training and technical assistance efforts.  However, OJJDP could better ensure the utility of the information  provided by the Model Programs Guide by establishing a mechanism to  solicit regular feedback from the juvenile justice field. We have previously  reported on the importance of regularly soliciting feedback to assess user  needs and satisfaction. Specifically, we have reported that without  feedback, an agency lacks valuable information from its users and is  hindered in its ability to make improvements to information products that  are relevant to users. Additionally, OJP has published Information Quality  Guidelines for its bureaus, including OJJDP, that highlight the importance  of ensuring the utility of information to be disseminated to the public by  continuously monitoring information needs, among other things.
OJJDP has mechanisms in place to regularly assess the usefulness of the  information disseminated by NTTAC to ensure that it meets the needs of  the juvenile justice field. Specifically, OJJDP has established an evaluation  process for NTTAC that is designed to collect the data necessary to  regularly assess the outcome and impact of the training and technical  assistance NTTAC provides to improve the quality of the information it  disseminates. Officials at NTTAC explained that after every training or  technical assistance event, all participants are given an evaluation form to  complete. This form is intended to capture feedback from participants  about the quality of the event, as well as feedback about the referrals and  resources NTTAC provides. Other evaluation forms are also available on  NTTAC’s Web site so that users can provide feedback about NTTAC’s  services, as well as feedback about the utility of the Web site. NTTAC then  follows up with a sample of these respondents for more in-depth feedback.  According to NTTAC officials, NTTAC analyzes the data collected from  these forms and then provides them to OJJDP. These officials stated that  OJJDP receives this information on at least a quarterly basis, and uses the  information to make changes to existing curricula and guide future  curriculum development, among other things. In accordance with OJP  guidelines and prior GAO work that highlights the importance of assessing  user needs, these evaluation efforts allow OJJDP to regularly monitor the  usefulness of the information it disseminates in order to develop or modify  its information products.
In addition, OJJDP is conducting a needs assessment to solicit additional  information about the utility of the information it disseminates through  NTTAC’s training and technical assistance efforts. NTTAC is administering  the needs assessment and, according to NTTAC officials, it is designed to  determine the training and technical assistance that would be most helpful  to the field. Specifically, the needs assessment is soliciting feedback from  members of the juvenile justice field about OJJDP’s existing efforts. It is  also requesting information regarding issues of interest to the field, any  current training or technical assistance needs, and the specific challenges  that the juvenile justice field is facing in its work. OJJDP officials stated  that they intend to use the results of the needs assessment to influence the  development of training and technical assistance activities and curricula  and the content of national conferences and workshops.
OJJDP’s efforts to conduct evaluations and a needs assessment are  consistent with comments we received from our expert interviews. We  asked all 26 of the juvenile reentry and substance abuse experts we  interviewed to comment on OJJDP’s overall efforts to disseminate  information about effective programs to the juvenile justice field. Thirteen  experts provided responses, and while they did not comment specifically  on NTTAC or the Model Programs Guide, they commented on the utility of  the information OJJDP provides in general about effective programs. Ten  of 13 experts had negative opinions of how useful the information OJJDP  disseminates is to members of the juvenile justice field. For example, 1  expert stated that practitioners often do not have the time to read research  data disseminated by OJJDP, which prevents them from being able to  effectively use it in their work. The expert added that it would be more  useful if OJJDP disseminated information that was practical and could be  applied in the field. In addition, 2 of these 10 experts suggested that it  would be helpful for OJJDP to obtain feedback from members of the  juvenile justice field about what types of information they would find  useful. Thus, OJJDP’s needs assessment should help to address this  concern. The remaining 3 experts who commented on OJJDP’s  information dissemination efforts had generally positive opinions, stating  that the information is useful to researchers and practitioners.
With respect to the Model Programs Guide, although OJJDP has ad hoc  mechanisms in place to solicit feedback about the information it provides,  it does not solicit this feedback on a regular basis or use feedback to help  ensure that the information disseminated by the Model Programs Guide is  useful to the field, in accordance with federal guidelines. For example, the  Model Programs Guide’s Program Director gives several presentations  about the guide each year at juvenile justice conferences. Officials who  operate the Model Programs Guide stated that following these  presentations, they request verbal feedback from participants. Officials  also stated that they regularly receive unsolicited feedback through the e- mail address that is listed on the Model Programs Guide’s Web site, which  they respond to on a case-by-case basis. Additionally, officials said that  they collect feedback about the Model Programs Guide through an annual  e-mail survey that is sent to the program points of contact listed on the  guide to obtain updated program information. Although these efforts to  solicit feedback about the Model Programs Guide provide OJJDP with  some information from its users, according to OJJDP officials, because the  guide does not have a systematic feedback mechanism, information  received cannot be analyzed on an aggregate level in order to regularly  assess how the juvenile justice field views the utility of the information  provided by the Model Programs Guide. Further, while the annual e-mail  survey can help OJJDP confirm that the program information featured in  the Model Programs Guide is accurate, it does not provide information  about whether the guide is useful to the field as a whole since OJJDP  sends the survey’s request for comments about the Model Programs Guide  to a portion of the juvenile justice field whose programs are already  published in the guide, which means that the comments it receives about  the Model Programs Guide do not necessarily reflect the opinions of the  juvenile justice field as a whole.
OJJDP officials agreed that they had not established a systematic  mechanism to obtain feedback from the field regarding the usefulness of  the Model Programs Guide and recognized that such a mechanism would  be useful to have in place. Officials also stated that NTTAC’s needs  assessment might be used as a model to build in more consistent  mechanisms for feedback for the office’s broader efforts. Because NTTAC  uses evaluations and is taking steps to conduct a needs assessment to  monitor the information needs of the juvenile justice field, OJJDP is in a  better position to help ensure that the information provided by NTTAC is  useful to the juvenile justice field. Recognizing that, although there is a  cost associated with gathering and analyzing feedback data, establishing a  cost-effective mechanism to regularly solicit feedback about the Model  Programs Guide should provide OJJDP with the information necessary to  assess whether the information provided by this tool is useful to the  juvenile justice field.
section_title: Finalizing a Program Plan and Developing a Comprehensive Evaluation Plan Would Help OJJDP Achieve Its Research and Evaluation Goals and Use Its Limited Resources Effectively
paragraphs: OJJDP has articulated research and evaluation goals to support its mission  of promoting effective programs and improving the juvenile justice system.  According to OJJDP, one of its three main goals is to promote  improvements in juvenile justice and facilitate the most effective  allocation of resources by conducting research to understand how the  juvenile justice system works in serving children and families. Under the  JJDPA, OJJDP is required to publish an annual program plan that  describes planned activities that are under accounts authorized for  research and evaluation activities and that demonstrate promising  initiatives, among other things. This plan is required to be published  annually in the Federal Register for public comment, and is to describe the  activities the Administrator intends to carry out under parts D and E, the  appropriations accounts that in general are available for research and the  development of new programs and initiatives, respectively. Specifically,  according to the JJDPA, the Administrator must take into account the  public comments received during the 45-day period and develop and  publish a final plan before December 31 of each fiscal year, describing the  particular activities that the Administrator intends to carry out under parts  D and E. While OJJDP has not published an annual program plan since  2002, it issued a proposed plan in the Federal Register to solicit public  comment in December 2009. OJJDP aims to publish the final version once  public comments are incorporated, in accordance with the JJDPA’s  requirements. Although the annual program plan is required to describe  the particular activities the Administrator intends to carry out under parts  D and E of the JJDPA, the proposed program plan includes the office’s  priorities with respect to all discretionary funding, including its research  and evaluation efforts. According to the Acting Administrator, this will, in  part, provide complete transparency for all such funding. According to  OJJDP, the development and publication of the annual program plan is a  first step that will lead to a comprehensive evaluation plan as the annual  program plan outlines the agency’s overall research and evaluation goals.
Additionally, the Office of Management and Budget’s (OMB) fiscal year  2006 Program Assessment Rating Tool found that juvenile justice  programs would benefit from evaluations of their effectiveness but noted  that such evaluations are difficult and expensive to do. As a result, OMB  recommended that OJP develop a comprehensive evaluation plan for  juvenile justice programs to obtain better information about the programs’  impacts. Although OMB’s recommendation was directed at OJP, OJP and  OJJDP officials stated that because OJJDP is the office within OJP  required to conduct juvenile justice evaluations, it is that office’s  responsibility to develop this evaluation plan.
In addition to the above requirement and recommendation, federal  guidelines stipulate the importance of developing a plan to achieve agency  goals. As established in the standard practices for program management,  specific goals of an agency must be conceptualized and defined in a plan.  Specifically, this plan is to contain a description or road map of how the  goals and objectives are to be achieved, including identifying the needed  resources and target milestones or time frames for achieving desired  results. We have also reported on the importance of planning research and  evaluation efforts, in part to ensure that goals are met and resources are  used effectively.
OJJDP’s Research Coordinator stated that such a road map or plan for  conducting research and evaluation would help better target the agency’s  research and evaluation efforts toward achieving their goals. However,  from 2006 to 2009, OJJDP had not developed such a plan, primarily  because of resource constraints. According to this official, in lieu of having  a comprehensive evaluation plan in place to guide its research and  evaluation efforts, the office’s efforts are influenced by a number of  factors, including whether Congress directs the agency to conduct  research in a particular area or whether ideas are generated internally by  staff or externally by members of the juvenile justice field. For example,  OJJDP staff responsible for the mentoring area may generate ideas about  how available research funds could be used, for example, by evaluating a  particular type of mentoring program. In addition, the office may receive  recommendations from the Federal Advisory Council on Juvenile Justice  or feedback from others in the juvenile justice field. While these factors  have influenced OJJDP’s research and evaluation efforts, they have not  provided a framework for helping the office meet its research and  evaluation goals. Therefore, once the program plan is finalized, OJJDP  intends to develop a comprehensive evaluation plan in accordance with  OMB recommendations to provide direction and priorities for its research  and evaluation efforts. According to the Acting Administrator, OJJDP  intends to use this comprehensive evaluation plan to better align and  target available discretionary funds toward achieving its research and  evaluation goals.
In addition to having a road map to help ensure it meets its goals, it is  important for OJJDP to have a comprehensive plan that lays out how the  office will evaluate its juvenile justice programs. Such a plan would help to  ensure that its limited resources are being used effectively. This is  important because OJJDP does not currently receive dedicated funding for  research and instead must make trade-off decisions to balance funding to  implement programs with funding to evaluate which programs are  effective. The office has not received dedicated research funding since  fiscal year 2005 when it received $10 million for its part D appropriations  account—the appropriations account specifically available for research  and evaluation efforts. Without part D funding, OJJDP has relied on funds  it has set aside from its other appropriation accounts to fund its research  and evaluation activities. Specifically, as shown in table 4, OJJDP is  authorized by the appropriations act to set aside up to 10 percent of  certain appropriations accounts for its research and evaluation efforts. In  fiscal year 2008, the last year for which set-aside funding data are  available, the appropriations act authorized OJJDP to set aside over   $23 million for research and evaluation. However, according to OJJDP, the  office set aside approximately $11 million. OJJDP officials stated that this  was, in part, because the JJDPA requires and the agency wants to ensure  that sufficient funds are available to the states for grant programs. In  addition, officials explained that some of OJJDP’s accounts are transferred  to other program offices, such as the Office of Community Oriented  Policing Services, so research funds are not deducted from those  accounts. Of the over $11 million that OJJDP did set aside, officials  reported that the office used nearly $8 million (or 70 percent) for research  and evaluation. Table 4 shows the amounts authorized to be set aside by  the annual appropriations act, as well as the amounts actually set aside  and used by OJJDP.
Additionally, all of the set-asides from these four accounts must be used  for research, evaluation, and statistics activities designed to benefit the  juvenile justice issues that the accounts specify. For example, set-aside  funds from the youth mentoring grant appropriation account must be used  to research or evaluate mentoring programs. For other accounts, OJJDP  can elect to fund research and evaluation efforts in a number of different  areas. For example, under Juvenile Accountability Block Grants, OJJDP  provides funds to states and units of local government to strengthen the  juvenile justice system. The states can use these funds for 17 different  purpose areas, including establishing programs to help the successful  reentry of juvenile offenders from state and local custody in the  community or for hiring staff or developing training programs for  detention and corrections. Consequently, there are limits on the amount of  funds OJJDP can divert to research and evaluation and on its discretion  over how to use of some of these funds. In fiscal year 2008, the  appropriation act allowed OJJDP to set aside more than $23 million that  could be dedicated to research and evaluation efforts on numerous eligible  programs. Because OJJDP has to decide how to split set aside funds  between supporting state and local program implementation and program  evaluation, in accordance with federal guidelines, a comprehensive  evaluation plan that in part identifies its funding resources could help  OJJDP make this determination.
According to OJJDP, the office has spent several years considering  developing a plan to provide a road map for how it would meet its  research and evaluation goals. However, officials stated that it has been  difficult to complete a comprehensive evaluation plan to fulfill OMB’s  Program Assessment Rating Tool recommendation because they have not  had the resources available—that is, funding and staffing—to develop the  plan. Specifically, because funds have not been appropriated for part D  since fiscal year 2005, OJJDP has not had a dedicated source of funding  that could be used to develop a comprehensive evaluation plan or to fund  the research identified by such a plan. Additionally, in 2003, OJJDP  reorganized its divisions and, as part of this, dissolved its research  division, as well as the training and information dissemination units.  According to OJJDP, the intention of the former Administrator who  implemented this reorganization was to better integrate these functions  throughout the agency. OJJDP officials stated that those staff who were  dedicated to research and evaluation work were reassigned to other  divisions. Although some of these staff retained the research projects they  had at the time, they also assumed new grant management duties. Also,  over the past 8 fiscal years, OJJDP’s overall authorized staffing level has  decreased from 95 to 76. Specifically, those staff dedicated to research and  evaluation decreased from 10 in fiscal year 2002 to 3.5 in fiscal year 2009.  According to OJJDP officials, the reduction in staff who were dedicated to  research and evaluation has strained the staffing resources that could be  used for developing a comprehensive evaluation plan.
Although OJJDP cited funding and staffing constraints, the Acting  Administrator has made developing a comprehensive evaluation plan a  priority and the office is committed to moving forward with developing  this plan. Following through with its planning efforts will help OJJDP to  meet its research and evaluation goals and better ensure that its resources  are being used effectively as stipulated by federal guidelines.
section_title: Conclusions
paragraphs: As the juvenile justice field—including states and local communities— works to implement programs to lower juvenile recidivism rates and  address juvenile substance abuse, it is important that the field has  information about which programs have been shown to be effective  through program evaluations. The importance of OJJDP’s goal to research  and evaluate programs to improve juvenile delinquency underscores the  need for a comprehensive plan to evaluate juvenile justice programs, one  that identifies resources to be committed to its research and evaluation  efforts and outlines the details of how OJJDP will accomplish its research  and evaluation goals. OJJDP efforts to publish a fiscal year 2010 program  plan in December are positive steps in developing the comprehensive  evaluation plan that officials have said they are committed to developing.  Having such a plan will provide OJJDP with a road map to help ensure that  it meets its research and evaluation goals, uses its limited resources  effectively, and contributes to identifying effective programs to help  support states and localities. With respect to OJJDP’s efforts to  disseminate information about effective programs, NTTAC’s efforts to  regularly assess the needs for the information it is disseminating through  training and technical assistance are important to helping OJJDP assess  the utility of its efforts and make appropriate improvements. We also  recognize that OJJDP’s efforts to conduct a needs assessment could help  provide important information to NTTAC that can be used in conjunction  with its evaluation efforts. Consistent with federal guidelines from OJP  and prior GAO reports, assessing the utility of the information  disseminated through OJJDP’s Model Programs Guide is also critical to  ensuring that such information meets the needs of the juvenile justice field  so the field can better implement effective programs. Having a mechanism  in place to regularly solicit feedback from the field about the usefulness of  the Model Programs Guide would better position OJJDP to assess whether  the information it is disseminating through the guide on effective programs  regularly meets the needs of its users.
section_title: Recommendation for Executive Action
paragraphs: To help ensure that OJJDP’s Model Programs Guide is regularly meeting  user needs and providing the most helpful information on effective  programs, consistent with federal guidelines, we recommend the  Administrator of OJJDP develop a cost-effective mechanism for regularly  soliciting and incorporating feedback from the juvenile justice field on the  usefulness of the information provided in its Model Programs Guide.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a copy of this report to the Attorney General for review and  comment. On December 3, 2009, OJP provided written comments, which  are reprinted in appendix VIII. OJP stated that it agreed with our  recommendation and intends to develop a mechanism for regularly  soliciting and incorporating feedback from the juvenile justice field on the  usefulness of the information provided in its Model Programs Guide by  March 31, 2010. OJP also provided technical comments, which we  incorporated as appropriate.
We are sending copies of this report to the Attorney General, selected  congressional committees, and other interested parties. The report also is  available at no charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact Eileen Larence at (202) 512-8777 or larencee@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may  be found on the last page of this report. Key contributors to this report are  listed in appendix IX.
section_title: Appendix I: Juvenile Justice Reentry and Substance Abuse Experts We Interviewed
paragraphs: For the purposes of our review, we selected a total of 26 experts to  interview—13 of whom had expertise related to juvenile reentry programs,  7 of whom had expertise related to juvenile substance abuse programs,  and 6 of whom had both juvenile reentry and substance abuse program  expertise. See table 5 for a list of these experts.
section_title: Appendix II: Juvenile Justice Experts’ Views on Factors That Can Help Programs Achieve Intended Outcomes
paragraphs: Of the 26 reentry and substance abuse experts we interviewed, 22  experts—as well as research we reviewed—identified several factors that  can help programs achieve intended outcomes, that is, be effective. The  following factors, while not an exhaustive list of items for programs to  consider when implementing juvenile justice intervention programs, were  the most frequently cited by the experts we interviewed:    maintaining fidelity to the program;    selecting, training, and retaining qualified providers;    conducting needs-based assessments to provide individualized treatment;  and  improving juvenile program participation by engaging and motivating  juvenile and family involvement.
While incorporating these factors into reentry or substance abuse  programs does not guarantee that any particular intervention program will  be successful, existing programs that have been evaluated and found to be  effective have generally included these factors in their designs or  implementation.
subsections: section_title: Maintaining Fidelity to the Program
paragraphs: According to 17 of the 22 experts, maintaining fidelity to the program as it  was intended to be implemented can help programs achieve their intended  objectives. This factor focuses on ensuring that core program services or  intervention components are delivered as they were designed, that is, with  fidelity. For example, for cognitive behavioral therapy, this would entail  that core intervention components, such as cognitive and social skills  training, were provided exactly as they were designed to each participant.  According to one expert’s research, the degree to which an intervention  program is delivered with fidelity is closely related to its effects on  recidivism. Another expert concurred, stating that the more closely core  program services or intervention components are implemented as they  were designed, the more the intervention program will reduce recidivism  rates. For example, one expert emphasized the importance of maintaining  fidelity to the program when replicating the model within a specific  community. In particular, another expert explained that some therapists  tend to substitute their own preferred treatment techniques instead of  using those prescribed by the intervention program, which can affect how  effective a program is at reducing recidivism. This is particularly true if the  intervention program being delivered is a program that has been evaluated  and found to be effective. Furthermore, one of these experts stated that  the specific model chosen has less of an effect on intended outcomes than  the manner in which it is delivered. As another expert explained it, a  weaker intervention—one that has not been evaluated and proven to be  effective—may result in decreased recidivism rates, for example, if it is  implemented as designed, while an effective intervention program that is  implemented poorly may have little or no effect on intended outcomes.
section_title: Selecting, Training, and Retaining Qualified Providers
paragraphs: According to 19 of the 22 experts, selecting, training, and retaining  qualified providers can help intervention programs achieve intended  outcomes. For example, the quality of the services that cognitive  behavioral therapy delivers depends, in part, on the provider’s ability and  whether the provider has been trained on the specific therapies and  components of the intervention program. Three of these experts noted  that if providers are not appropriately trained in the therapy or  intervention being implemented, they may not provide the program as it  was intended, or as one of them noted, may substitute their own preferred  treatment techniques for those prescribed by the intervention program. As  a result, the providers’ failure to deliver the intervention program as it was  designed reduces the ability of the program to achieve intended outcomes.  Furthermore, many intervention programs utilize providers who have  certain educational or clinical experience, such as having a background in  mental health or being a licensed practitioner for the specific therapy  being implemented. One of the 19 experts we interviewed also mentioned  the importance of gaining the support of the juvenile justice community,  as well as agencies’ program management, in the selection and training of  providers.
section_title: Conducting Needs-Based Assessments to Provide Individualized Treatment
paragraphs: According to 18 of the 22 experts, by assessing a juvenile’s specific  treatment needs, program providers can better design intervention  programs that will be targeted to a juvenile’s individual situation. For  example, 4 experts noted that this can help intervention programs achieve  intended outcomes because individualized treatment is more likely to  affect participants’ individual outcomes since it takes into account  differences such as age, gender, culture, environment, and problem  severity. One expert noted that individualized treatment ensures that  juveniles do not receive unnecessary treatment, which in some instances  may produce harmful results. According to this expert, providing juveniles  who do not have substance abuse problems some programs, such as  cognitive behavioral therapy, may lead to harmful results because these  juveniles are exposed to others who have more serious addictions. Four  experts also noted that using needs-based assessments to develop  individualized treatment plans can be more cost beneficial than using  standard treatment plans. Specifically, as one of these experts noted, this  is because individualized treatment plans can help ensure that costly  interventions are not provided to juveniles who do not need extensive  services. In addition, 5 experts stated that conducting a risk-based  assessment is important to determining which juveniles are at higher risk  of reoffending in order to focus programming efforts on them. One of  these experts cited a study that shows that targeting specific treatment  needs of offenders is correlated with recidivism outcomes, that is,  providing targeted treatment needs is generally related to lower  recidivism.
section_title: Improving Juvenile Participation by Engaging and Motivating Juvenile and Family Involvement
paragraphs: According to 16 of the 22 experts, engaging and motivating juvenile and  family involvement can help to improve a juvenile’s program participation,  thereby helping intervention programs to achieve intended outcomes. For  example, 1 expert noted that successful programs rely on staff members to  gain the trust of juvenile offenders. These programs also recognize that  juveniles may experience program fatigue because they are participating  in numerous programs and that motivation may become an issue. In  addition, this expert noted that after being released into the community,  juveniles and their families may not be motivated to participate in  intervention programs. Additionally, research has shown that encouraging  families to participate in the juvenile’s treatment program can reduce  family risk factors for delinquency. Eleven experts also mentioned that  motivating juvenile offenders and their families to participate can assist  juveniles in successfully completing an intervention program. Two of  these experts noted that by involving family members in treatment, some  issues that may contribute to juvenile dropout rates, such as a history of  traumatic stress and family members who also abuse substances, can be  addressed within an intervention program.
section_title: Appendix III: OJJDP’s Enacted Appropriations for Fiscal Years 2007 through 2009
paragraphs: To prevent and respond to juvenile delinquency and help states improve  their juvenile justice systems, the Office of Juvenile Justice and  Delinquency Prevention (OJJDP) administers a wide variety of grants to  states, territories, localities, and public and private organizations through  formula, block, and discretionary grant programs. The office also provides  training and technical assistance, produces and distributes publications  and other products containing information about juvenile justice topics,  and funds research and evaluation efforts. Table 6 shows funding by fiscal  year from 2007 through 2009 for the appropriation accounts for juvenile  justice programs.
section_title: Appendix IV: Juvenile Reentry and Substance Abuse Programs OJJDP Funded through Discretionary Grants
paragraphs: From fiscal years 2007 through 2009, OJJDP allocated approximately   $33 million through discretionary grants to four juvenile reentry grant  programs and three juvenile substance abuse programs. See table 7 for a  description of these reentry and substance abuse programs and the  amounts OJJDP awarded to grantees.
section_title: Appendix V: Experts’ Opinions of Reentry Programs That Lack Conclusive Evidence of Effectiveness
subsections: section_title: Experts Provided Positive Views about Wraparound/Case Management Programs, but Could Not Cite Evidence from Available Research Indicating That They Are Effective at Reducing Recidivism
paragraphs: Of the 19 reentry experts we interviewed, 9 had specific experience or  knowledge related to wraparound/case management and 8 experts had  positive comments about the effectiveness of this program. In general,  wraparound/case management interventions involve making an array of  individualized services and support networks available to juveniles, rather  than requiring them to enroll in treatment programs that may not address  individual needs. According to OJJDP, the goal of wraparound/case  management programs is to keep delinquent juveniles at home and out of  institutions whenever possible. The basic elements that constitute a  wraparound program include, among other things, (1) a collaborative,  community-based interagency team responsible for designing,  implementing, and overseeing the intervention program in a given  jurisdiction; (2) care coordinators who are responsible for helping  juveniles create customized treatment programs, among other things;   (3) juvenile and family teams consisting of family members and  community members who work together to ensure the juvenile’s needs are  met at home, at school, and in the community; and (4) a plan of care  developed and updated by all members of the juvenile and family teams  that identifies the juvenile’s strengths and weaknesses, targets specific  goals such as improved performance in school, and outlines how to  achieve them.
Of these nine experts, eight provided positive opinions of the results of  wraparound/case management intervention programs. For example, an  expert commented about how in one specific wraparound intervention  program, a single case manager is assigned to a juvenile and is responsible  for determining the services the juvenile is to receive based on his or her  specific needs, instead of enrolling the juvenile into a treatment program  that may not be as beneficial for the juvenile. Two of these eight experts  noted that there was a lack of evaluations demonstrating effectiveness of  these intervention programs but pointed us to a study on a specific  wraparound/case management intervention program, Wraparound  Milwaukee, that showed potentially promising results related to a  reduction in recidivism rates for juvenile offenders. However, these  experts stressed that this study alone did not conclusively demonstrate the  effectiveness of wraparound/case management programs. The ninth expert  stated that in her experience, wraparound/case management interventions  are not effective because, for example, a juvenile is placed into this  intervention program based on the availability of program staff and  resources rather than program services being tailored to the individual  needs of the juvenile. Additionally, three of the nine experts cautioned that  these intervention programs are difficult to implement because of such  issues as a lack of quality services or low retention of juveniles and their  families in the intervention being provided. Specifically, one of these  experts noted that the quality of wraparound services can vary depending  on a community’s resources. In addition, another expert emphasized the  importance of obtaining buy-in from diverse service providers who may be  used to working on their own, such as within the welfare, foster care, and  public school systems.
section_title: Reentry Experts Cited Lack of Conclusive Evidence Based on Available Research and Mixed Views of the Effectiveness of Aftercare Programs at Reducing Recidivism
paragraphs: Of 19 reentry experts we interviewed, 15 had specific experience or  knowledge related to aftercare programs and 6 cited a lack of conclusive  evidence of effectiveness of the program type. Aftercare intervention  programs are intended to prepare juvenile offenders to return to the  community during the reentry process by focusing on the delivery of  services and supervision that start while juveniles are incarcerated and  continue after they return to their communities. Specifically, aftercare  programs collaborate with the community and marshal its resources to  help ensure that juvenile offenders receive services that address their  individual needs, such as treatment for a substance abuse problem. These  intervention programs focus on changing individual behavior thereby  preventing further delinquency. For example, an aftercare program might  incorporate the use of techniques from an intervention therapy, such as  motivational enhancement therapy, to engage juvenile offenders in  treatment and increase their commitment to change.
Of these 15 experts, 7 offered positive opinions regarding aftercare  intervention programs, based on their own experience or knowledge of the  intervention programs. For example, 1 expert noted that if aftercare  included intervention programs that were proven to be effective, used  assessment tools that identified the individual needs of the juvenile, and  implemented the therapies as they were designed, then the aftercare  intervention program should be effective at reducing recidivism rates.  Although these experts could not provide examples of studies that had  been conducted to show evidence of the effectiveness of the intervention  programs, all 7 of them agreed based on their own experience or  knowledge that aftercare interventions are important reentry programs, in  part, because they link the juvenile with his or her community and provide  regular contact with a caseworker. Additionally, 3 of these 7 experts stated  that aftercare could be effective depending on the intervention programs  used and if they were delivered as intended. For example, they said that if  aftercare includes intervention programs that have proven to be effective,  such as cognitive behavioral therapy, and identifies the individual needs of  the juvenile, the programs can reduce recidivism. However, 6 of the 15  reentry experts said there was inconclusive evidence to determine  whether these programs can be effective in achieving results. Three of  these experts based their opinions on an evaluation of the Intensive  Aftercare Program that showed inconclusive results about program  effectiveness. Specifically, the study found no evidence that the program  had its intended impact of reducing recidivism among juveniles who were  released back into the community under supervision in the three states  that piloted the program. However, the evaluation did find that the three  states that implemented the Intensive Aftercare Program model did  successfully incorporate most of its core features, which prepared  juveniles to transition back into the community. For instance, these states  created new Intensive Aftercare Program–specific treatment programs  that among other things, prepared juveniles for increased responsibility in  the community, facilitated interaction with the community, and worked  with the juveniles’ schools and families. The state programs had a large  percentage of juveniles involved in various treatment services. Despite the  inconclusive results of the study, one expert credited aftercare programs  with addressing the issue of juveniles having to deal with different  probation officers throughout the reentry process because, in general,  aftercare programs assign one probation officer to a juvenile as a  consistent point of contact. The evaluation also stated that in order for the  general aftercare model to be effective, it must not only provide  supervision and services after a juvenile’s release into the community, but  also focus on preparing a juvenile for release. The remaining 2 experts  opined that aftercare intervention programs had not been shown to be  effective at achieving desired results because, for example, the treatment a  juvenile receives depends on what services are actually available in the  community.
section_title: Experts Indicated Potential Positive Outcomes for Vocational or Job Training Programs
paragraphs: Of the 19 reentry experts we interviewed, 11 had specific experience or  knowledge related to vocational or job training programs and indicated  potential positive outcomes for these programs. According to OJJDP,  providing juveniles with employment opportunities during reentry is a  common strategy used to try to reduce future criminal behavior.  Vocational or job training intervention programs are intended to improve  juveniles’ social and educational functioning by, for example, increasing  earnings, raising self-esteem, and instilling a positive work ethic. Juveniles  can participate in vocational/job training intervention programs while they  are incarcerated and after they return to the community.
Of the 11 reentry experts, 10 of them had positive comments based on  their experience or knowledge of the program type. Specifically, they said  that vocational/job training programs were potentially beneficial, in part, if  they were applied to older juveniles and if they led to those juveniles  getting jobs. The remaining expert said that there is little evidence to  demonstrate the effectiveness of these intervention programs.
section_title: Experts Cited No Evidence for Effectiveness of Reentry Courts
paragraphs: Reentry courts are specialized courts that manage the return of juvenile  offenders to the community after they are released from residential  facilities. The court manages reentry by using its authority to direct  resources to support the offender’s return to the community and promote  positive behavior, among other things. For example, a reentry court would  oversee a juvenile’s release into the community by assigning a judge to  meet with the juvenile once a month. The judge would actively engage the  supervising authority, such as a parole officer, in assessing the juvenile’s  progress. The judge would also oversee sanctions for violations as well as  rewards, like early release from parole, for successful achievement of  goals, such as successfully completing a cognitive behavioral therapy  intervention program.
Of the 19 reentry experts we interviewed, 2 provided comments related to  reentry courts and had differing opinions on their effectiveness. One had a  negative impression of the courts, stating that the reentry courts do not  provide more to a juvenile than a probation officer would. The other  commented that he considers concepts encompassed in reentry courts,  such as intensity of supervision, to be a best practice when it comes to  reentry programs. However, neither was aware of any evaluations of these  types of courts.
section_title: Appendix VI: Experts’ Opinions of Substance Abuse Programs That Lack Conclusive Evidence of Effectiveness
subsections: section_title: Experts Views Were Divided on the Effectiveness of Drug Courts, with Half Indicating That Additional Evidence Is Needed to Determine Effectiveness
paragraphs: Of the 13 substance abuse experts we interviewed, 10 had specific  experience or knowledge related to drug courts that resulted in mixed  views of the effectiveness of this program type. Juvenile drug courts are  specialized courts established within and supervised by juvenile courts to  provide intervention programs, such as cognitive behavioral therapy or  family therapy, for substance-abusing juveniles and their families. Juvenile  offenders assigned to drug courts are identified by a juvenile court as  having problems with alcohol or drugs. The drug court judge maintains  close oversight of each case through frequent—often weekly—status  hearings with the individuals involved. The judge both leads and works as  a member of a team that can comprise representatives from juvenile  justice, social services, school and vocational training programs, law  enforcement, probation, the prosecution, and the defense. Together, the  team determines how best to address the substance abuse and related  problems of the juvenile and his or her family.
Specifically, of these 10 experts, 5 experts described drug courts as having  insufficient evidence to determine program effectiveness. For example, 2  experts mentioned that while some studies show drug courts reducing  substance abuse while juveniles were under court supervision, the results  did not last after juveniles were no longer being supervised by the courts.  Another expert stated that since drug courts tend to be used for juveniles  who have their first or second contact with the juvenile justice system,  they are ineffective at achieving desired results because they expose these  first-time offenders to peers who have more serious substance abuse  addictions and therefore might influence them to continue to abuse  substances. By contrast, the remaining 4 experts stated that drug courts  can be effective at achieving desired results such as reducing substance  abuse if, for example, the juvenile is sent to a community where there are  intervention programs offered that have been evaluated and have been  shown to be effective, such as cognitive behavioral therapy or family  therapy intervention programs. One expert cited a study to support the  opinion that drug courts supplemented with multisystemic therapy  resulted in a decrease in substance abuse by juvenile offenders.
section_title: Experts Report Mixed Views on the Effectiveness of Mentoring Programs
paragraphs: Of the 13 substance abuse experts we interviewed, 8 had specific  experience or knowledge related to mentoring intervention programs that  resulted in mixed views of their effectiveness. Mentoring programs consist  of a relationship between two or more people over a prolonged period of  time, where an older, more experienced individual provides support and  guidance to a juvenile. The goal of mentoring is for the juvenile to develop  positive adult contact, thereby reducing risk factors, such as exposure to  juveniles who use substances, while increasing positive factors, such as  encouragement for abstaining from substance use. In the substance abuse  field, juveniles in need of sobriety are teamed with older sponsors to serve  as positive role models in helping them become sober.
Of these eight experts, four stated that mentoring programs are ineffective  or unsuccessful at achieving desired results, such as reducing substance  abuse, and that these intervention programs are more effective at  preventing at-risk juveniles from engaging in delinquent behavior. Also,  one expert stated that there have been too few evaluations conducted on  mentoring programs to make a general statement about the relative  benefits of mentoring. Conversely, three experts stated that mentoring  programs are effective or can be effective if, for example, mentors are  trained or if mentoring is combined with another intervention program  that has been evaluated and has been shown to be effective, such as  multisystemic therapy.
section_title: Experts Provided Positive Views about the Potential Results of Wraparound/Case Management, but Cited Limited Evidence of Its Effectiveness
paragraphs: Of the 13 substance abuse experts we interviewed, 11 had specific  experience or knowledge related to wraparound/case management  intervention programs that resulted in mixed views of the program type.  Of these 11 experts, 7 stated that wraparound/case management is  effective or can be effective if, for example, it is combined with another  intervention program that has been evaluated and has shown to be  effective, such as multisystemic therapy. Although these experts had  limited evidence demonstrating the effectiveness of wraparound/case  management, 2 experts cited two studies that show potentially promising  results related to a reduction in recidivism. For example, one study  showed that juveniles in wraparound/case management receive a number  of individualized services, such as mental health treatment for those  juveniles who struggle with emotional issues. However, this study stressed  that it is difficult to evaluate wraparound/case management in a controlled  way since treatment plans are individualized for each juvenile. The other 4  experts stated that wraparound/case management intervention programs  are ineffective because, for example, the intervention programs lack  follow-through as there are no consequences if a juvenile does not show  up for treatment, or there is not yet sufficient evidence to determine their  effectiveness.
section_title: Appendix VII: Additional OJJDP Efforts to Disseminate Information about Effective Juvenile Justice Programs
paragraphs: In addition to the National Training and Technical Assistance Center and  the Model Programs Guide, OJJDP disseminates information about  effective programs through a variety of other efforts. Specifically, the  office has developed mechanisms to disseminate information related to  effective programs in specific issue areas, such as youth gang activity,  disproportionate minority contact, and girls’ delinquency, as described in  table 8.
section_title: Appendix VIII: Comments from the Department of Justice
section_title: Appendix IX: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Mary Catherine Hult, Assistant  Director; David Alexander; Elizabeth Blair; Ben Bolitzer; Carissa Bryant;  Katherine Davis; Sean DeBlieck; Allyson Goldstein; Rebecca Guerrero;  Jared Hermalin; Dawn Locke; Lisa Shibata; Janet Temko; and Delia Zee  made key contributions to this report.