Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs
Published: 2009-10-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: This testimony discusses issues related to girls' delinquency--a topic that has attracted the attention of federal, state, and local policymakers for more than a decade as girls have increasingly become involved in the juvenile justice system. For example, from 1995 through 2005, delinquency caseloads for girls in juvenile justice courts nationwide increased 15 percent while boys' caseloads decreased by 12 percent. More recently, in 2007, 29 percent of juvenile arrests--about 641,000 arrests--involved girls, who accounted for 17 percent of juvenile violent crime arrests and 35 percent of juvenile property crime arrests. Further, research on girls has highlighted that delinquent girls have higher rates of mental health problems than delinquent boys, receive fewer special services, and are more likely to abandon treatment programs. The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is the Department of Justice (DOJ) office charged with providing national leadership, coordination, and resources to prevent and respond to juvenile delinquency and victimization. OJJDP supports states and communities in their efforts to develop and implement effective programs to, among other things, prevent delinquency and intervene after a juvenile has offended. For example, from fiscal years 2007 through 2009, Congress provided OJJDP almost $1.1 billion to use for grants to states, localities, and organizations for a variety of juvenile justice programs, including programs for girls. Also, in support of this mission, the office funds research and program evaluations related to a variety of juvenile justice issues. As programs have been developed at the state and local levels in recent years that specifically target preventing girls' delinquency or intervening after girls have become involved in the juvenile justice system, it is important that agencies providing grants and practitioners operating the programs have information about which of these programs are effective. In this way, agencies can help to ensure that limited federal, state, and local funds are well spent. In general, effectiveness is determined through program evaluations, which are systematic studies conducted to assess how well a program is working--that is, whether a program produced its intended effects. To help ensure that grant funds are being used effectively, you asked us to review OJJDP's efforts related to studying and promoting effective girls' delinquency programs. We issued a report on the results of that review on July 24, 2009. This testimony highlights findings from that report and addresses (1) efforts OJJDP has made to assess the effectiveness of girls' delinquency programs, (2) the extent to which these efforts are consistent with generally accepted social science standards and federal standards to communicate with stakeholders, and (3) the findings from OJJDP's efforts and how the office plans to address the findings. This statement is based on our July report and selected updates made in October 2009.
section_title: What GAO Found
paragraphs: With an overall goal of developing research that communities need to make sound decisions about how best to prevent and reduce girls' delinquency, OJJDP established the Girls Study Group (Study Group) in 2004 under a $2.6 million multiyear cooperative agreement with a research institute. OJJDP's objectives for the group, among others, included identifying effective or promising programs, program elements, and implementation principles (i.e., guidelines for developing programs). Objectives also included developing program models to help inform communities of what works in preventing or reducing girls' delinquency, identifying gaps in girls' delinquency research and developing recommendations for future research, and disseminating findings to the girls' delinquency field about effective or promising programs. OJJDP's effort to assess girls' delinquency programs through the use of a study group and the group's methods for assessing studies were consistent with generally accepted social science research practices and standards. In addition, OJJDP's efforts to involve practitioners in Study Group activities and disseminate findings were also consistent with the internal control standard to communicate with external stakeholders, such as practitioners operating programs. The Study Group found that few girls' delinquency programs had been studied and that the available studies lacked conclusive evidence of effective programs; as a result, OJJDP plans to provide technical assistance to help programs be better prepared for evaluations of their effectiveness. However, OJJDP could better address its girls' delinquency goals by more fully developing plans for supporting such evaluations.

section_title: Letter
paragraphs: I am pleased to be here today as you examine issues related to girls’  delinquency—a topic that has attracted the attention of federal, state, and  local policymakers for more than a decade as girls have increasingly  become involved in the juvenile justice system. For example, from 1995  through 2005, delinquency caseloads for girls in juvenile justice courts  nationwide increased 15 percent while boys’ caseloads decreased by 12  percent. More recently, in 2007, 29 percent of juvenile arrests—about  641,000 arrests—involved girls, who accounted for 17 percent of juvenile  violent crime arrests and 35 percent of juvenile property crime arrests.  Further, research on girls has highlighted that delinquent girls have higher  rates of mental health problems than delinquent boys, receive fewer  special services, and are more likely to abandon treatment programs.
The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is the  Department of Justice (DOJ) office charged with providing national  leadership, coordination, and resources to prevent and respond to juvenile  delinquency and victimization. OJJDP supports states and communities in  their efforts to develop and implement effective programs to, among other  things, prevent delinquency and intervene after a juvenile has offended.  For example, from fiscal years 2007 through 2009, Congress provided  OJJDP almost $1.1 billion to use for grants to states, localities, and  organizations for a variety of juvenile justice programs, including  programs for girls. Also, in support of this mission, the office funds  research and program evaluations related to a variety of juvenile justice  issues.
As programs have been developed at the state and local levels in recent  years that specifically target preventing girls’ delinquency or intervening  after girls have become involved in the juvenile justice system, it is  important that agencies providing grants and practitioners operating the  programs have information about which of these programs are effective.  In this way, agencies can help to ensure that limited federal, state, and  local funds are well spent. In general, effectiveness is determined through  program evaluations, which are systematic studies conducted to assess  how well a program is working—that is, whether a program produced its  intended effects. To help ensure that grant funds are being used  effectively, you asked us to review OJJDP’s efforts related to studying and  promoting effective girls’ delinquency programs. We issued a report on the  results of that review on July 24, 2009. My statement today, as requested,  highlights findings from that report and addresses (1) efforts OJJDP has  made to assess the effectiveness of girls’ delinquency programs, (2) the  extent to which these efforts are consistent with generally accepted social  science standards and federal standards to communicate with  stakeholders, and (3) the findings from OJJDP’s efforts and how the office  plans to address the findings.
My statement is based on our July report and selected updates made in  October 2009. For our report, we reviewed documentation about OJJDP’s  establishment of a study group to assess the effectiveness of girl’s  delinquency programs, analyzed the groups’ activities and findings, and  interviewed OJJDP research and program officials and the current and  former principal investigators of the study group. Specifically, we  reviewed the criteria the study group used to assess studies of girls’  delinquency programs and whether the group’s application of those  criteria was consistent with generally accepted social science standards  for evaluation research. We also compared OJJDP’s efforts with criteria in  Standards for Internal Control in the Federal Government, specifically  that agency management should ensure that there are adequate means of  obtaining information from and communicating with external stakeholders  who may have a significant impact on the agency achieving its goals, such  as practitioners operating programs or researchers assessing programs. In  addition, we conducted interviews with 18 girls’ delinquency subject  matter experts, that is, researchers and practitioners, whom we selected  on the basis of their knowledge and experience with girls’ delinquency  issues. While their comments cannot be generalized to all girls’  delinquency experts, we nonetheless believe that their views gave us  useful insights on issues related to girls’ delinquency and OJJDP’s efforts  to assess girls’ programs. Our work was performed in accordance with  generally accepted government auditing standards. More detail about our  scope and methodology is included in our July report.
subsections: section_title: OJJDP Established the Girls Study Group to Assess the Effectiveness of Girls’ Delinquency Programs
paragraphs: With an overall goal of developing research that communities need to  make sound decisions about how best to prevent and reduce girls’  delinquency, OJJDP established the Girls Study Group (Study Group) in  2004 under a $2.6 million multiyear cooperative agreement with a research  institute. OJJDP’s objectives for the group, among others, included  identifying effective or promising programs, program elements, and  implementation principles (i.e., guidelines for developing programs).  Objectives also included developing program models to help inform  communities of what works in preventing or reducing girls’ delinquency,  identifying gaps in girls’ delinquency research and developing  recommendations for future research, and disseminating findings to the  girls’ delinquency field about effective or promising programs. To meet  OJJDP’s objectives, among other activities, the Study Group identified  studies of delinquency programs that specifically targeted girls by  reviewing over 1,000 documents in relevant research areas. These included  criminological and feminist explanations for girls’ delinquency, patterns of  delinquency, and the justice system’s response to girls’ delinquency. As a  result, the group identified 61 programs that specifically targeted  preventing or responding to girls’ delinquency. Then, the group assessed  the methodological quality of the studies of the programs that had been  evaluated using a set of criteria developed by DOJ’s Office of Justice  Programs (OJP) called What Works to determine whether the studies  provided credible evidence that the programs were effective at preventing  or responding to girls’ delinquency. The results of the group’s assessment  are discussed in the following sections.
section_title: OJJDP Efforts to Assess Program Effectiveness Were Consistent with Social Science Practices and Standards, and OJJDP Has Taken Action to Enhance Communication about the Study Group with External Stakeholders
paragraphs: OJJDP’s effort to assess girls’ delinquency programs through the use of a  study group and the group’s methods for assessing studies were consistent  with generally accepted social science research practices and standards.  In addition, OJJDP’s efforts to involve practitioners in Study Group  activities and disseminate findings were also consistent with the internal  control standard to communicate with external stakeholders, such as  practitioners operating programs.
According to OJJDP research and program officials, they formed the Study  Group rather than funding individual studies of programs because study  groups provide a cost-effective method of gaining an overview of the  available research in an issue area. As part of its work, the group  collected, reviewed, and analyzed the methodological quality of research  on girls’ delinquency programs. The use of such a group, including its  review, is an acceptable approach for systematically identifying and  reviewing research conducted in a field of study. This review helped  consolidate the research and provide information to OJJDP for  determining evaluation priorities. Further, we reviewed the criteria the  group used to assess the studies and found that they adhere to generally  accepted social science standards for evaluation research. We also  generally concurred with the group’s assessments of the programs based  on these criteria. According to the group’s former principal investigator,  the Study Group decided to use OJP’s What Works criteria to ensure that  its assessment of program effectiveness would be based on highly rigorous  evaluation standards, thus eliminating the potential that a program that  may do harm would be endorsed by the group. However, 8 of the 18  experts we interviewed said that the criteria created an unrealistically high  standard, which caused the group to overlook potentially promising  programs. OJJDP officials stated that despite such concerns, they  approved the group’s use of the criteria because of the methodological  rigor of the framework and their goal for the group to identify effective  programs.
In accordance with the internal control standard to communicate with  external stakeholders, OJJDP sought to ensure a range of stakeholder  perspectives related to girls’ delinquency by requiring that Study Group  members possess knowledge and experience with girls’ delinquency and  demonstrate expertise in relevant social science disciplines. The initial  Study Group, which was convened by the research institute and approved  by OJJDP, included 12 academic researchers and 1 practitioner; someone  with experience implementing girls’ delinquency programs. However, 11 of  the 18 experts we interviewed stated that this composition was  imbalanced in favor of academic researchers. In addition, 6 of the 11 said  that the composition led the group to focus its efforts on researching  theories of girls’ delinquency rather than gathering and disseminating  actionable information for practitioners. According to OJJDP research  and program officials, they acted to address this issue by adding a second  practitioner as a member and involving two other practitioners in study  group activities. OJJDP officials stated that they plan to more fully involve  practitioners from the beginning when they organize study groups in the  future and to include practitioners in the remaining activities of the Study  Group, such as presenting successful girls’ delinquency program practices  at a national conference. Also, in accordance with the internal control  standard, OJJDP and the Study Group have disseminated findings to the  research community, practitioners in the girls’ delinquency field, and the  public through conference presentations, Web site postings, and published  bulletins. The group plans to issue a final report on all of its activities by  spring 2010.
section_title: The Study Group Found No Evidence of Effective Girls’ Delinquency Programs; in Response OJJDP Plans to Assist Programs in Preparing for Evaluations but Could Strengthen Its Plans for Supporting Such Evaluations
paragraphs: The Study Group found that few girls’ delinquency programs had been  studied and that the available studies lacked conclusive evidence of  effective programs; as a result, OJJDP plans to provide technical  assistance to help programs be better prepared for evaluations of their  effectiveness. However, OJJDP could better address its girls’ delinquency  goals by more fully developing plans for supporting such evaluations.
In its review, the Study Group found that the majority of the girls’  delinquency programs it identified—44 of the 61—had not been studied by  researchers. For the 17 programs that had been studied, the Study Group  reported that none of the studies provided conclusive evidence with which  to determine whether the programs were effective at preventing or  reducing girls’ delinquency. For example, according to the Study Group,  the studies provided insufficient evidence of the effectiveness of 11 of the  17 programs because, for instance, the studies involved research designs  that could not demonstrate whether any positive outcomes, such as  reduced delinquency, were due to program participation rather than other  factors. Based on the results of this review, the Study Group reported that  among other things, there is a need for additional, methodologically  rigorous evaluations of girls’ delinquency programs; training and technical  assistance to help programs prepare for evaluations; and funding to  support girls’ delinquency programs found to be promising.
According to OJJDP officials, in response to the Study Group’s finding  about the need to better prepare programs for evaluation, the office plans  to work with the group and use the remaining funding from the effort— approximately $300,000—to provide a technical assistance workshop by  the end of October 2009. The workshop is intended to help approximately  10 girls’ delinquency programs prepare for evaluation by providing  information about how evaluations are designed and conducted and how  to collect data that will be useful for program evaluators in assessing  outcomes, among other things. In addition, OJJDP officials stated that as a  result of the Study Group’s findings, along with feedback they received  from members of the girls’ delinquency field, OJJDP plans to issue a  solicitation in fiscal year 2010 for funding to support evaluations of girls’  delinquency programs.
OJJDP has also reported that the Study Group’s findings are to provide a  foundation for moving ahead on a comprehensive program related to girls’  delinquency. However, OJJDP has not developed a plan that is  documented, is shared with key stakeholders, and includes specific  funding requirements and commitments and time frames for meeting its  girls’ delinquency goals. Standard practices for program and project  management state that specific desired outcomes or results should be  conceptualized, defined, and documented in the planning process as part  of a road map, along with the appropriate projects needed to achieve those  results, supporting resources, and milestones. In addition, government  internal control standards call for policies and procedures that establish  adequate communication with stakeholders as essential for achieving  desired program goals. According to OJJDP officials, they have not  developed a plan for meeting their girls’ delinquency goals because the  office is in transition and is in the process of developing a plan for its  juvenile justice programs, but the office is taking steps to address its  girls’ delinquency goals, for example, through the technical assistance  workshop. Developing a plan for girls’ delinquency would help OJJDP to  demonstrate leadership to the girls’ delinquency field by clearly  articulating the actions it intends to take to meet its goals and would also  help the office to ensure that the goals are met.
In our July report, we recommended that to help ensure that OJJDP meets  its goals to identify effective or promising girls’ delinquency programs and  supports the development of program models, the Administrator of OJJDP  develop and document a plan that (1) articulates how the office intends to  respond to the findings of the Study Group, (2) includes time frames and  specific funding requirements and commitments, and (3) is shared with  key stakeholders. OJP agreed with our recommendation and outlined  efforts that OJJDP plans to undertake in response to these findings. For  example, OJJDP stated that it anticipates publishing its proposed juvenile  justice program plan, which is to include how it plans to address girls’  delinquency issues, in the Federal Register to solicit public feedback and  comments, which will enable the office to publish a final plan in the  Federal Register by the end of the year (December 31, 2009).
Mr. Chairman, this concludes my statement. I would be pleased to respond  to any questions that you or other Members of the Subcommittee may  have.
section_title: Contacts and Acknowledgements
paragraphs: For questions about this statement, please contact Eileen R. Larence at  (202) 512-8777 or larencee@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this statement. Individuals making key contributions to this statement  include Mary Catherine Hult, Assistant Director; Kevin Copping; and  Katherine Davis. Additionally, key contributors to our July 2009 report  include David Alexander, Elizabeth Blair, and Janet Temko.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.