Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans Will Help Girls' Delinquency Programs
Released: 2009-10-20T00:00:00

Letter

OJJDP Established the Girls Study Group to Assess the Effectiveness of Girls’ Delinquency Programs

With an overall goal of developing research that communities need to make sound decisions about how best to prevent and reduce girls’ delinquency, OJJDP established the Girls Study Group (Study Group) in 2004 under a $2.6 million multiyear cooperative agreement with a research institute. OJJDP’s objectives for the group, among others, included identifying effective or promising programs, program elements, and implementation principles (i.e., guidelines for developing programs). Objectives also included developing program models to help inform communities of what works in preventing or reducing girls’ delinquency, identifying gaps in girls’ delinquency research and developing recommendations for future research, and disseminating findings to the girls’ delinquency field about effective or promising programs. To meet OJJDP’s objectives, among other activities, the Study Group identified studies of delinquency programs that specifically targeted girls by reviewing over 1,000 documents in relevant research areas. These included criminological and feminist explanations for girls’ delinquency, patterns of delinquency, and the justice system’s response to girls’ delinquency. As a result, the group identified 61 programs that specifically targeted preventing or responding to girls’ delinquency. Then, the group assessed the methodological quality of the studies of the programs that had been evaluated using a set of criteria developed by DOJ’s Office of Justice Programs (OJP) called What Works to determine whether the studies provided credible evidence that the programs were effective at preventing or responding to girls’ delinquency. The results of the group’s assessment are discussed in the following sections.

OJJDP Efforts to Assess Program Effectiveness Were Consistent with Social Science Practices and Standards, and OJJDP Has Taken Action to Enhance Communication about the Study Group with External Stakeholders

OJJDP’s effort to assess girls’ delinquency programs through the use of a study group and the group’s methods for assessing studies were consistent with generally accepted social science research practices and standards. In addition, OJJDP’s efforts to involve practitioners in Study Group activities and disseminate findings were also consistent with the internal control standard to communicate with external stakeholders, such as practitioners operating programs.

According to OJJDP research and program officials, they formed the Study Group rather than funding individual studies of programs because study groups provide a cost-effective method of gaining an overview of the available research in an issue area. As part of its work, the group collected, reviewed, and analyzed the methodological quality of research on girls’ delinquency programs. The use of such a group, including its review, is an acceptable approach for systematically identifying and reviewing research conducted in a field of study. This review helped consolidate the research and provide information to OJJDP for determining evaluation priorities. Further, we reviewed the criteria the group used to assess the studies and found that they adhere to generally accepted social science standards for evaluation research. We also generally concurred with the group’s assessments of the programs based on these criteria. According to the group’s former principal investigator, the Study Group decided to use OJP’s What Works criteria to ensure that its assessment of program effectiveness would be based on highly rigorous evaluation standards, thus eliminating the potential that a program that may do harm would be endorsed by the group. However, 8 of the 18 experts we interviewed said that the criteria created an unrealistically high standard, which caused the group to overlook potentially promising programs. OJJDP officials stated that despite such concerns, they approved the group’s use of the criteria because of the methodological rigor of the framework and their goal for the group to identify effective programs.

In accordance with the internal control standard to communicate with external stakeholders, OJJDP sought to ensure a range of stakeholder perspectives related to girls’ delinquency by requiring that Study Group members possess knowledge and experience with girls’ delinquency and demonstrate expertise in relevant social science disciplines. The initial Study Group, which was convened by the research institute and approved by OJJDP, included 12 academic researchers and 1 practitioner; someone with experience implementing girls’ delinquency programs. However, 11 of the 18 experts we interviewed stated that this composition was imbalanced in favor of academic researchers. In addition, 6 of the 11 said that the composition led the group to focus its efforts on researching theories of girls’ delinquency rather than gathering and disseminating actionable information for practitioners. According to OJJDP research and program officials, they acted to address this issue by adding a second practitioner as a member and involving two other practitioners in study group activities. OJJDP officials stated that they plan to more fully involve practitioners from the beginning when they organize study groups in the future and to include practitioners in the remaining activities of the Study Group, such as presenting successful girls’ delinquency program practices at a national conference. Also, in accordance with the internal control standard, OJJDP and the Study Group have disseminated findings to the research community, practitioners in the girls’ delinquency field, and the public through conference presentations, Web site postings, and published bulletins. The group plans to issue a final report on all of its activities by spring 2010.

The Study Group Found No Evidence of Effective Girls’ Delinquency Programs; in Response OJJDP Plans to Assist Programs in Preparing for Evaluations but Could Strengthen Its Plans for Supporting Such Evaluations

The Study Group found that few girls’ delinquency programs had been studied and that the available studies lacked conclusive evidence of effective programs; as a result, OJJDP plans to provide technical assistance to help programs be better prepared for evaluations of their effectiveness. However, OJJDP could better address its girls’ delinquency goals by more fully developing plans for supporting such evaluations.

In its review, the Study Group found that the majority of the girls’ delinquency programs it identified—44 of the 61—had not been studied by researchers. For the 17 programs that had been studied, the Study Group reported that none of the studies provided conclusive evidence with which to determine whether the programs were effective at preventing or reducing girls’ delinquency. For example, according to the Study Group, the studies provided insufficient evidence of the effectiveness of 11 of the 17 programs because, for instance, the studies involved research designs that could not demonstrate whether any positive outcomes, such as reduced delinquency, were due to program participation rather than other factors. Based on the results of this review, the Study Group reported that among other things, there is a need for additional, methodologically rigorous evaluations of girls’ delinquency programs; training and technical assistance to help programs prepare for evaluations; and funding to support girls’ delinquency programs found to be promising.

According to OJJDP officials, in response to the Study Group’s finding about the need to better prepare programs for evaluation, the office plans to work with the group and use the remaining funding from the effort— approximately $300,000—to provide a technical assistance workshop by the end of October 2009. The workshop is intended to help approximately 10 girls’ delinquency programs prepare for evaluation by providing information about how evaluations are designed and conducted and how to collect data that will be useful for program evaluators in assessing outcomes, among other things. In addition, OJJDP officials stated that as a result of the Study Group’s findings, along with feedback they received from members of the girls’ delinquency field, OJJDP plans to issue a solicitation in fiscal year 2010 for funding to support evaluations of girls’ delinquency programs.

OJJDP has also reported that the Study Group’s findings are to provide a foundation for moving ahead on a comprehensive program related to girls’ delinquency. However, OJJDP has not developed a plan that is documented, is shared with key stakeholders, and includes specific funding requirements and commitments and time frames for meeting its girls’ delinquency goals. Standard practices for program and project management state that specific desired outcomes or results should be conceptualized, defined, and documented in the planning process as part of a road map, along with the appropriate projects needed to achieve those results, supporting resources, and milestones. In addition, government internal control standards call for policies and procedures that establish adequate communication with stakeholders as essential for achieving desired program goals. According to OJJDP officials, they have not developed a plan for meeting their girls’ delinquency goals because the office is in transition and is in the process of developing a plan for its juvenile justice programs, but the office is taking steps to address its girls’ delinquency goals, for example, through the technical assistance workshop. Developing a plan for girls’ delinquency would help OJJDP to demonstrate leadership to the girls’ delinquency field by clearly articulating the actions it intends to take to meet its goals and would also help the office to ensure that the goals are met.

In our July report, we recommended that to help ensure that OJJDP meets its goals to identify effective or promising girls’ delinquency programs and supports the development of program models, the Administrator of OJJDP develop and document a plan that (1) articulates how the office intends to respond to the findings of the Study Group, (2) includes time frames and specific funding requirements and commitments, and (3) is shared with key stakeholders. OJP agreed with our recommendation and outlined efforts that OJJDP plans to undertake in response to these findings. For example, OJJDP stated that it anticipates publishing its proposed juvenile justice program plan, which is to include how it plans to address girls’ delinquency issues, in the Federal Register to solicit public feedback and comments, which will enable the office to publish a final plan in the Federal Register by the end of the year (December 31, 2009).

Mr. Chairman, this concludes my statement. I would be pleased to respond to any questions that you or other Members of the Subcommittee may have.

Contacts and Acknowledgements

For questions about this statement, please contact Eileen R. Larence at (202) 512-8777 or larencee@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Individuals making key contributions to this statement include Mary Catherine Hult, Assistant Director; Kevin Copping; and Katherine Davis. Additionally, key contributors to our July 2009 report include David Alexander, Elizabeth Blair, and Janet Temko.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.