Juvenile Justice: OJJDP Reporting Requirements for Discretionary and Formula Grantees and Concerns About Evaluation Studies
Released: 2001-11-29T00:00:00

Letter

Background

OJJDP, one of the components of the U.S. Department of Justice, Office of Justice Programs (OJP), was established by the Juvenile Justice and Delinquency Prevention Act of 1974 (Juvenile Justice Act). Its mission is to provide national leadership, coordination, and resources to prevent and respond appropriately to juvenile delinquency and juvenile victimization. OJJDP accomplishes its mission through developing and implementing prevention programs and a juvenile justice system that protects the public safety, holds juvenile offenders accountable, and provides treatment and rehabilitative services based on the needs of juveniles and their families. OJJDP funds research and evaluation efforts, statistical studies, and demonstration programs; provides technical assistance and training; produces and distributes publications and other products containing information about juvenile justice topics; oversees activities dealing with missing and exploited children; and administers a wide variety of grant programs. OJJDP funds programs that serve juveniles directly as well as those that benefit juveniles more indirectly by focusing on system-wide changes or by increasing the capacity of governmental units or organizations.

OJJDP awards grants to states, territories, localities, and private organizations through five formula and block grant (formula/block grant) programs and numerous discretionary grant programs. OJJDP administers four formula grant programs that provide funds directly to states and territories on the basis of states’ juvenile populations, and one block grant program that awards a fixed level of funds to all states and territories. Under these formula/block grant programs, states may, in turn, make subawards to other organizations such as units of local government. OJJDP awards discretionary grants through a competitive process to state governments, local governments, or individual agencies and organizations. OJP is responsible for the financial monitoring of grantees (i.e., it provides policy guidance, control, and support services in the financial management of grants), whereas OJJDP is responsible for program monitoring. For program monitoring purposes, OJJDP assigns each of its grantees a program manager who is responsible for ensuring administrative and programmatic compliance with relevant statutes, regulations, policies and guidelines of awarded grants. The program manager is also responsible for monitoring grantees’ performance and progress as related to grantees’ stated goals and objectives.

OJJDP’s budget has increased significantly over the last 5 years—from about $188 million in fiscal year 1997 to about $596 million in fiscal year 2001. During this time, the Congress has created new programs and increased appropriations for some existing ones. The Congress has also provided direction each year regarding certain program areas OJJDP should fund. Overall, in fiscal year 2001, 31 percent of OJJDP’s available funds were congressionally earmarked—that is, set aside for an identifiable grantee, specified amount, and/or specific authorized purpose. For fiscal year 2001, $180 million of OJJDP’s funds were available for discretionary grant awards, of which 77 percent was earmarked.

OJJDP’s Fiscal Year 2000 Awards

OJJDP awards the majority of its funds to grantees in its five formula/block grant programs. In fiscal year 2000, the latest year for which awards data were available, OJJDP awarded (1) about $354 million, or 64 percent of the total funds awarded, to these formula/block grant programs and (2) just over $200 million, or 36 percent of the total funds awarded, to a wide range of discretionary grant programs. The programs awarded the most funds in fiscal year 2000 were the Juvenile Accountability Incentive Block Grants Program ($221 million), the Formula Grants Program ($70 million), the Community Prevention Grants Program ($36 million), the Child Abuse and Neglect Program ($32 million), the Missing and Exploited Children Program ($32 million), and the Drug-Free Communities Support Program ($30 million).

OJJDP awards funds to a wide range of recipients, with the majority of awarded funds going to state governments. As shown in figure 1, 67 percent of the funds OJJDP awarded in fiscal year 2000 went to states, 20 percent to nonprofit organizations, 6 percent to school districts or educational institutions, and 5 percent to local governments. However, because many grantees make subawards to other entities, the awards to these grantees do not reflect the ultimate recipients of the funds OJJDP awards. For example, under the Formula Grants Program, states pass through a minimum of two-thirds of their awarded funds to public and private nonprofit organizations. (See app. I for data from fiscal years 1996 through 2000 on (1) OJJDP funds awarded to formula/block grant versus discretionary grant programs, (2) OJJDP awards by program area, (3) OJJDP award recipients, and (4) OJJDP formula/block grant awards by state.)

Scope and Methodology

To identify programmatic reporting requirements for OJJDP grantees, the reasons for these requirements, and examples of information grantees have reported, we reviewed all 5 of OJJDP’s formula/block grant programs and we selected 11 of its major discretionary grant programs to review based on OJJDP officials’ input regarding which programs were “major” (e.g., number of grantees, program funding, and/or importance of the program). To identify grantee reporting requirements, and the purpose of these requirements, we met with the OJJDP program managers who monitor each of these 16 programs or with other key officials. For those programs in which OJJDP funded an outside evaluation, we also met with program managers who oversee the evaluations. We reviewed OJP’s Grant Management Policies and Procedures Manual (January 19, 2001) and Categorical Assistance Progress Report (progress report) form along with the instructions for completing the form. (See app. III for a copy of the progress report form.) We also reviewed OJJDP program documents for each of the 16 programs, including any special reporting conditions. To supplement program documents, we reviewed relevant documents from outside evaluators. We did not assess the adequacy of reporting requirements established by OJJDP or the outside evaluators.

To identify specific examples of performance data that grantees reported, we asked OJJDP officials to provide progress reports for each program demonstrating a range of detail and, in some cases, we asked for reports from specific grantees. For each program, we then reviewed 3 to 15 progress reports (or individual performance reports) submitted between 1998 and 2001. We did not review progress reports from all grantees in every program, nor did we review grantees’ compliance with reporting requirements. For programs being evaluated by an outside evaluator, we reviewed performance data that program grantees reported to those evaluators, when available.

To determine whether OJJDP requires grantees to report the number of juveniles they serve directly and to identify the number of juveniles OJJDP grantees reported serving in fiscal year 2000, we interviewed OJJDP program managers for each of the 16 programs we reviewed and examined relevant program documents, including selected semiannual progress reports. To determine whether other programs directly serve juveniles, obtain available data on the numbers of juveniles served, and learn why OJJDP does or does not require grantees to report such data, we reviewed OJJDP program literature and met with OJJDP division directors. Nevertheless, the list of OJJDP programs we identified as directly serving juveniles may not be comprehensive. We focused our data collection effort on only those programs we identified in which all grantees reported juveniles-served data.

To assess the methodological rigor of the impact evaluations OJJDP has funded since 1995 of its own programs, and to provide information on the other types of evaluations OJJDP has funded, we asked OJJDP to identify all program evaluations it had funded since 1995. For each evaluation, we asked OJJDP to indicate whether it was an impact evaluation, whether the program being evaluated was funded by OJJDP, and whether the evaluation had been completed or was ongoing. Overall, OJJDP identified 35 evaluations funded since 1995.

Eleven of the 35 evaluations were impact evaluations of OJJDP programs, and all were ongoing. For each of the 10 impact evaluations we assessed, we asked OJJDP to provide any documentation relevant to the design and implementation of the evaluation methodologies, such as the initial and supplemental proposals, peer review documents, progress reports, reports of interim results, and correspondence between OJJDP and the evaluators. In addition, we contacted OJJDP officials to resolve any questions that we had regarding the documentation and to request any missing documents. We did not contact the program manager responsible for each evaluation.

To assess the methodological rigor of the 10 impact evaluations, we used a data collection instrument to collect information systematically on each program being evaluated and the features of the evaluation methodology. We based our data collection and assessments on generally accepted social science standards. We examined such factors as whether evaluation data were collected before and after program implementation, how program effects were isolated (i.e., the use of nonprogram participant comparison groups or statistical controls), and the appropriateness of sampling, outcome measures, statistical analyses, and any reported results. A social scientist with training and experience in evaluation research and methodology read and coded the documentation for each evaluation. A second social scientist reviewed each completed data collection instrument and the relevant documentation for the impact evaluation to verify the accuracy of every coded item. We relied on documents OJJDP provided to us in April 2001 in assessing the evaluation methodologies and reporting on each evaluation’s status.

For each of the remaining 24 evaluations, which included nonimpact evaluations of OJJDP-funded programs, as well as evaluations of juvenile justice programs that OJJDP did not fund, we asked OJJDP to provide general descriptive information, such as the type and purpose of the evaluation, the number of sites involved, and whether the evaluation included data on all participants. We did not assess the methodological rigor of these evaluations.

We conducted our work at OJJDP Headquarters in Washington, D.C., from September 2000 to August 2001 in accordance with generally accepted government auditing standards.

All Grantees Receive Standard Reporting Guidance and Some Are Required to Report Specific Data

OJP requires virtually all OJJDP grantees to submit semiannual progress reports, which OJJDP uses to help monitor grantees’ project implementation and achievement of the goals they identified in their grant applications. To this end, OJP provides grantees standard, general guidance on the types of program information they are to report, such as narrative information on the status of each of their project goals and the quantitative results of their projects. In addition to this standard requirement, grantees for some of OJJDP’s programs are subject to additional reporting requirements that apply only to their respective programs. Our review of 16 major programs showed that grantees in 8 of the programs were required to comply only with the standard requirement for information, and grantees in the other 8 programs were required to report additional specified data. The specific reporting requirements were established primarily to help evaluate the results of these programs. Table 1 identifies the 16 programs we reviewed and the reason for the standard or specific reporting requirements for each program.

Our review of selected progress reports from the 16 programs showed that, in all but the Formula Grants Program, grantees reported information on the status of their activities and accomplishments in response to the standard requirements, although the details they reported varied as did the projects themselves. Grantees in the eight programs with specific reporting requirements reported a variety of descriptive information and performance data to OJJDP and/or outside evaluators. (See app. IV for a program description, summary of reporting requirements, and examples of what grantees reported in each of the 16 programs we reviewed.)

Grantees Receiving Standard Guidance Reported a Variety of Information Used for OJJDP Monitoring

All OJJDP grantees are required to report on their project activities and accomplishments to OJP twice a year using the Office of Justice Programs’ Categorical Assistance Progress Report (progress report) form. The form is unstructured and is to be completed in narrative and/or chart form. The standard instructions to grantees for completing the form state that grantees should report information on the status of each of their projects’ goals scheduled to be achieved during the reporting period and set forth in their grant application, including quantitative project results based on performance measures. Grantees are also instructed to report on actions planned to resolve any implementation problems and request any technical assistance they might need. OJJDP program managers are to use reported information to help monitor grantees’ project implementation. OJJDP officials explained that because the progress report is intended as just one of their monitoring tools, this standard, general guidance meets their basic oversight needs. They further explained that guidance needs to be somewhat general given the variation that can occur among projects as grantees tailor them to meet local needs and circumstances. OJJDP encourages grantees to design projects that meet the unique needs of their own communities, and therefore grantees do not always report on the same performance measures.

Although OJJDP program managers have additional ways of keeping abreast of grantees’ projects, such as phone calls and on-site visits, OJJDP officials indicated they would prefer to require and obtain more specific, and even more frequent, information through the progress reports or other reporting mechanisms. However, according to these officials, they are reluctant to impose additional reporting requirements on grantees because of the Paperwork Reduction Act of 1995, which seeks to ensure that federal agencies balance their need to collect information with the reporting and paperwork burden they impose. Under the Act, federal agencies have an obligation to keep the paperwork burden they impose as low as possible, and agencies must receive prior approval from the Office of Management and Budget for information collection requests.

We reviewed selected progress reports that grantees from each of the 16 programs submitted to OJJDP and found that, in all but the Formula Grants Program, grantees reported on the status of their projects. Grantees reported input, output, or outcome data related to the process, implementation, and/or accomplishments of their projects. They included information such as subgrant awards, specific meetings held, staff hired, implementation difficulties, number of project participants, and behavioral change in youths. However, the particular information grantees reported varied, as did their projects. This variation, coupled with the unstructured format of the progress report, makes it difficult to aggregate reported data.

Fourteen of the programs we reviewed had multiple grantees and the information these grantees reported in response to the standard guidance varied, even within each program. For example, we found the following: Under the Tribal Youth Program—a program that recognizes differences among tribes and encourages diversity in their projects—grantees must implement projects in keeping with at least one of four broad purpose areas. One tribe reported that it had completed the renovation of a youth center; another reported that it had collected examples of other tribes’ juvenile law enforcement codes and started drafting model codes adapted for each of its villages. A third tribe reported that the resignation of its community truancy officer had impacted its ability to reduce instances of misbehavior in school.

Under another program—the Juvenile Accountability Incentive Block Grants Program—grantees (states) and their subgrantees (communities) have undertaken a variety of projects and, thus, report different information. In this program, states and their communities can choose from 12 different purpose areas under the broad objective of promoting greater accountability in the juvenile justice system. Thus, one state reported that one community hired a juvenile court intake officer and included that officer’s caseload; the same state reported that another community was unable to start a project because a local agency declined to participate in its project. Another state reported on the number of youths enrolled in one community’s drug testing project and reported the number of drug screening tests performed.

In the Drug-Free Communities Support Program, grantees design projects to meet the needs of their local communities; thus, the projects and the information grantees reported varied. For example, one grantee reported that it helped local students produce a 30-second anti-smoking commercial in collaboration with the local health department and further reported that only 9 of 50 invited members attended a strategic planning meeting it had held. Another grantee reported making presentations on drug abuse to 146 young men at the local juvenile detention facility, and that its pre- and post-assessments continually showed that the young men gained knowledge in the harmful effects of alcohol and drugs.

In the Formula Grants Program, not all grantees reported on the objectives and accomplishments of their subgrantees’ projects, as required. OJJDP requires grantees in this program to complete an Individual Project Report (IPR) for each subgrantee. The instructions for completing IPRs are similar to the instructions grantees in other programs receive for completing semiannual progress reports. Our review of IPRs from selected states showed that, for one state, none of the IPRs contained any information on subgrantees’ accomplishments, and some did not include information on subgrantees’ program objectives. For another state, neither OJJDP nor the state was able to provide us with copies of completed IPRs because OJJDP’s automated reporting system for states was inoperable.

Two of the 16 programs we reviewed had only one grantee each. Although both received only standard reporting guidance, they reported more detailed, quantitative output and outcome data than grantees in the other programs that received only standard guidance. In the first program—the National Clearinghouse and Resource Center for Missing and Exploited Children—the grantee has voluntarily reported detailed information in a structured format. In the second program—the Model Courts Program— OJJDP has emphasized that the grantee should include quantitative performance data in its progress reports, but did not prescribe the specific indicators on which the grantee must report.

OJJDJP has designated the National Center for Missing and Exploited Children (NCMEC) as the grantee for National Clearinghouse and Resource Center for Missing and Exploited Children, and NCMEC has developed its own standardized reporting format that covers 10 categories. This format collects numbers and other information on each category, such as missing children cases, exploited children cases, public affairs, and hotline calls. NCMEC reports to OJJDP quarterly, rather than semiannually, because this timeframe matches the reporting structure of its data management system. For the first quarter of 2001, NCMEC reported various output and outcome data that included receiving 24,983 calls through its hotline; assisting in the recovery of 1,610 missing children; receiving 5,291 tips on its online child pornography tip line; and displaying pictures of 1,399 missing children, which resulted in locating 257 children.

The sole grantee of the Model Courts Program—the National Council of Juvenile and Family Court Judges—-also reports quantitative information in its semiannual progress reports. Although OJJDP has not specified the performance indicators on which the Council must report, it has emphasized to the Council the need for quantitative performance data in the semiannual progress reports. As a result, the Council includes specific quantitative output data in its progress reports. For example, it reported that during the last half of 2000 it distributed 17,818 technical assistance bulletins, conducted 96 training presentations, and made 31 site visits to model courts. In addition, the Council voluntarily publishes an annual report that provides more detailed information on the accomplishments of the individual model courts, such as a reduction in the number of children in court custody. OJJDP officials told us that they do not require the Council to provide this report, but they have instructed it to report detailed performance data on the activities of the model courts, when such data exist. They further explained that if the Council were to stop publishing an annual report, OJJDP would require it to include model court performance data in its progress reports.

Grantees in Some Programs Are Required to Report Specific Performance Data, Often for Evaluative Purposes

In eight of the major programs we reviewed, grantees were given additional, more explicit reporting instructions requiring them to report on the same specific performance measures as other grantees in the same program. In these programs, additional requirements were established to meet the evaluative needs of OJJDP or an outside evaluator. In one of the eight programs, requirements were also established to ensure grantee compliance with certain requirements of the Juvenile Justice Act as well as for program assessment. The specific requirements of each of these eight programs varied, as they were tailored to each program. However, grantees in all these programs were still required to routinely report narrative information on the status of their activities and accomplishments through semi-annual progress reports.

In five of these programs, OJJDP and/or an evaluator have established specific reporting requirements primarily to support an outside program evaluation. For example, as a condition of receiving a Juvenile Mentoring Program grant, OJJDP requires all grantees to participate fully in the evaluation by providing data to the evaluator. This evaluator requires grantees to report their data through quarterly progress reports that are similar to semiannual progress reports. The required data include information on youths participating in each project, participating mentors, and youth-mentor matches. For instance, grantees are required to report family structure information for participating youths. The evaluator aggregates such data from all grantees and has reported, for example, that 56 percent of participating youths lived with their mother only, 20 percent lived with both parents, 4 percent lived with their father only, and 21 percent were in other living arrangements.

In two of the eight programs—the Internet Crimes Against Children Task Force Program and Children’s Advocacy Centers—specific reporting requirements were established so that OJJDP could assess program accomplishments. The governing board of the Internet Crimes Task Force, in agreement with OJJDP, identified monthly performance measures on which grantees must report, such as the number of arrests made, search warrants issued, subpoenas served, and cases opened by the task forces. Under the Children’s Advocacy Centers program, OJJDP prescribed specific performance measures on which grantees must report, such as the number of practitioners trained, training conferences held, and publications distributed. In this program, specific reporting requirements were established not only for OJJDP to assess the program’s overall accomplishments, but also to help grantees assess their own projects.

The eighth program—the Formula Grants Program—has requirements that are statutorily based and further spelled out by OJJDP in program regulations. Program reporting requirements were established to ensure grantees comply with the four core requirements of the Juvenile Justice Act and as a basis for assessing the effects of the program. These core requirements are (1) deinstitutionalization of status offenders, (2) separation of juveniles from adult offenders, (3) removal of juveniles from adult jail and lockup, and (4) addressing efforts to reduce disproportionate minority confinement. OJJDP regulations list in detail the information on which states must report. For instance, regarding the deinstitutionalization of status offenders, states must report the total number of accused and adjudicated status offenders and nonoffenders placed in facilities that are, for example, not near their home community. (See app. VI for a summary of states’ compliance with the core requirements of the Juvenile Justice Act.) According to the compliance monitoring coordinator for the Formula Grants Program, grantees’ reports on compliance with the core requirements also provide the basis for OJJDP to assess the effects of the program.

Some Data Exist on the Number of Juveniles Served Directly, Although Most Grantees Are Not Required to Report This Information

We identified eight programs that serve juveniles directly and whose grantees reported such data for fiscal year 2000. However, OJJDP often does not require grantees to provide this information, in large part because not all of its programs are intended to provide direct services to juveniles.

In Some Programs, All Grantees Report the Number of Juveniles They Directly Serve

We identified eight programs in which grantees directly serve juveniles and in which all grantees report the number of juveniles served to either OJJDP or an outside evaluator. About 400 grantees in these eight programs directly served close to 142,000 juveniles in one year. For example, in fiscal year 2000, the Juvenile Mentoring Program reported serving about 8,500 juveniles, and in calendar year 2000, the Court Appointed Special Advocate Program reported serving 70,348 youths. Table 2 shows the programs we identified as directly serving juveniles and reporting such data for fiscal year 2000.

We also identified a program in which all subgrantees directly serve juveniles, but not all subgrantees report such data. The national granteefor the Children’s Advocacy Centers program reported that its subgrantees served over 100,000 juveniles in calendar year 2000, but this number represents only those juveniles served by subgrantees accredited through a national membership council.

Reasons Exist for Not Requiring Grantees to Report Number of Juveniles Served

For several reasons, OJJDP does not typically require grantees to report the number of juveniles their projects directly serve. First and foremost, many of OJJDP’s programs are not intended to serve juveniles directly. The Juvenile Justice Act established OJJDP for a variety of purposes, many of which involve indirect benefits to juveniles, rather than direct services. Statutorily-established purposes for OJJDP include the following: To provide technical assistance to and training programs for professionals who work with delinquents.

To provide for the evaluations of federally-assisted juvenile justice and delinquency prevention programs.

To establish a centralized research effort on problems of delinquency.

To assist state and local governments in improving the administration of justice and services for juveniles who enter the system.

Some of OJJDP’s programs, in their entirety, provide indirect benefits, rather than direct services, to juveniles. OJJDP’s Model Courts Program, for example, benefits juveniles indirectly by providing training and technical assistance to court personnel to improve their handling of child abuse and neglect cases. The Internet Crimes Against Children Task Force Program also benefits juveniles indirectly by helping to identify and arrest pedophiles and child pornographers who use the Internet to prey on children. Furthermore, in commenting on a draft of this report, the Assistant Attorney General pointed out that although OJJDP’s research projects do not typically provide services directly, their results can potentially help thousands of juveniles.

OJJDP officials provided the following additional reasons for not requiring all grantees to report the number of juveniles their projects directly serve: A common interpretation of “juveniles served” does not exist across, or even within, programs. For example, grantees in one program might consider the number of juveniles served as those assessed for services but referred elsewhere, while grantees in a different program might consider only juveniles who received at least 10 sessions of therapy. Even within the same program grantees may not share a common definition of “juveniles served.” One program grantee might report on the number of juveniles who received intensive one-on-one drug prevention services over an extended period of time, while another grantee in that same program might report on the number who attended a one-time presentation on drug prevention. Without a common interpretation of “juveniles served,” the data grantees report would be inconsistent and would have little value.

For some programs, directly serving juveniles may be only one of a number of intended program purposes and thus, OJJDP does not typically require all grantees within these programs to report such data. For example, in the Formula Grants Program, states and their subgrantees can choose from among 14 different program areas related to preventing and controlling delinquency and improving juvenile justice systems. Under the program area of “planning and administration,” for instance, states can fund planning projects that benefit juveniles indirectly, such as developing a comprehensive state plan to identify juvenile service needs and programs that address those needs over the long term. However, under the area of “illegal drugs and alcohol,” a local subgrantee can serve juveniles directly by establishing a drug and alcohol abuse prevention project.

Juveniles-served data could be used inappropriately to measure the effectiveness of a program whose primary purpose may not be to provide direct services to juveniles. For example, the primary purpose of State Challenge Activities is to stimulate system-wide change, although many of its 10 activity areas also promote projects intended to directly serve juveniles. However, grantees in this program are expected to implement direct service projects within the broader context of promoting system- wide change. For instance, one State Challenge Activities grantee used funds it received under the “deinstitutionalization of status offenders” activity area to establish two community projects that provide housing for runaway juveniles, many of whom are girls. The grantee intends to use its experiences with these two new projects to initiate system-wide change by developing a comprehensive model program expressly geared to serving runaway girls. By focusing on the number of girls served by this program, one might fail to see that its primary purpose was to develop a comprehensive model program for serving runaway girls.

Methodological Concerns Could Adversely Affect Evaluation Studies

OJJDP has funded 35 evaluations since 1995, including 11 evaluations intended to measure the impact of OJJDP-funded programs. We reviewed the methodological rigor of 10 of the 11 evaluations. Half of these 10 evaluations are in formative stages, while the other five are well into implementation. None had been completed at the time of our review. Our in-depth review of these 10 evaluations shows that although several are well-designed and use, or plan to use, sophisticated data analysis methods, others raise concerns as to whether the evaluations will produce definitive results. We recognize that impact evaluations, such as the types that OJJDP are funding, can encounter difficult design and implementation issues. For some of the evaluations we reviewed, program flexibility has added to the complexity of designing evaluations. A lack of comparison groups to aid in isolating the impacts of some programs, and data collection problems could compromise some evaluation results.

According to OJJDP officials, the OJJDP weighs a number of factors when deciding which programs to evaluate and what kind of evaluations to fund.

Given its budget, it considers how much of its discretionary funds to spend in support of evaluation activities. In deciding which of its programs to evaluate, OJJDP gives priority consideration to programs that have been mandated by the Congress. Other criteria OJJDP uses to determine whether a program should be evaluated include the program’s level of funding and its uniqueness, as well as the feasibility and cost of an evaluation and its potential benefits to the field. Similar criteria are also involved with decisions to evaluate non-OJJDP funded programs, as well as congressional interest and other federal agencies’ willingness to co-fund an evaluation.

The 10 impact evaluations of OJJDP-funded programs that we assessed vary in size and scope. The cost to conduct these evaluations ranges from $300,000 to well over $5 million; however, some of these grants involve both impact and process evaluations and the cost of the impact portion alone cannot be separated from the total. All 10 evaluations are multi-year, multi-site projects. The number of evaluation sites ranges from 2 in the Rural Gang Initiative to 175 in the evaluation of the Juvenile Mentoring Program. As of April 2001, three evaluations had produced interim findings of some program impacts. (See app. II for information on OJJDP’s process for disseminating products with interim findings as well as other products.)

Evaluations of OJJDP Programs Are Difficult to Successfully Design and Implement

Program evaluation is an inherently difficult task because the objective is to isolate the impact of a particular program from all other factors that could have caused a change consistent with the intent of the program, or mitigated against that change. Given that programs, such as those funded by OJJDP, operate in an ever-changing environment and involve juveniles and adults who themselves constantly change, producing definitive evaluation results can be arduous. For example, the impact of a hypothetical program intended to improve students’ grades could be confounded by the effects of an outside-of-school mentoring program, the transfer of high-performing students to a magnet program, changes in school faculty, a new scholarship program, a severe flu season that results in widespread student absences from school, and a myriad of other factors.

Project Variation Within a Program Can Complicate Evaluation Design

Our in-depth review of the 10 impact evaluations of OJJDP programs showed that a number of these evaluations are particularly complex because local grantees design their own projects to fit their communities’ needs. (See app. VII for a description of the impact evaluations OJJDP has funded of its own programs since 1995). Although this customization may make sense from a program perspective, it makes it more difficult to evaluate the program. Instead of assessing a single, homogeneous program with multiple grantees, the evaluation must assess the effects of multiple configurations of a program. Although all of the grantees’ projects under each program being evaluated are intended to achieve the same or similar goals, an aggregate analysis could mask differences in individual projects’ effectiveness and, thus, not result in information about which configurations of projects work and which do not.

OJJDP’s evaluation of the Enforcing the Underage Drinking Laws Program (discretionary grant component) exemplifies this situation. In implementing their projects, states and local communities have substantial latitude to employ media campaigns, merchant education, compliance checks, youth leadership training, or a variety of other activities to deter underage drinking. Similarly, under the Positive Action Through Holistic Education program, local educators develop their own ways to prevent student violence and behavior problems based on their assessments of the causes of these problems in their schools. Because of the limited number of sites (two school districts) being evaluated and the likely differences in how each school has developed its own project, the resulting evaluation may not provide information that could be generalized to a broader implementation of the program.

Lack of Appropriate Comparison Groups Could Compromise Some Results

A standard way for evaluators to isolate the impacts of a program from other potential factors that could have influenced change is to use a comparison group as a benchmark. In the hypothetical example cited above concerning a program to improve students’ grades, a second set of students who are not in the program but are matched in academic performance and exposed to all of the same factors (except the program) could provide a baseline from which to assess the impact of the program. The grades of students in the two groups before and after the program would provide the data from which to measure program impacts. Without the benefit of the comparison group as a baseline, it is difficult or impossible to isolate changes resulting from the program from changes due to other factors.

The designs of two of the five evaluations that are well into implementation lack an appropriate comparison group. The evaluation of the Juvenile Mentoring Program—a one-on-one mentoring program for youths—compares youths entering the program to those completing it. However, a variety of other factors, including the fact that youths in the program are likely to mature and, thus, improve somewhat spontaneously, cannot be ruled out as a rival cause of change from the beginning to the end of the program. Although the evaluators are employing multiple and innovative strategies to determine the effectiveness of the program in achieving its objective, the lack of a comparison group of nonparticipant youths is an obstacle to identifying definitive outcomes.

An evaluation of the Partnerships to Reduce Juvenile Gun Violence Program includes a comparison of before and after crime statistics in project communities with crime statistics for the same time frames for the cities in which the projects operate. However, citywide crime statistics would no doubt include data from communities that are similar to the project community as well as from those that are not. Thus, the differences between citywide and project community baselines make it difficult to attribute potential findings to the program.

Of the five programs for which evaluations are still being developed, two (the Safe Start Initiative and the Rural Gang Initiative) did not seem to have plans for comparison groups at the time of our review. Another (Parents Anonymous) anticipates using a comparison group, but as yet had not developed specific plans for one.

Data Collection Problems Could Compromise Some Results

Regardless of the quality of a program evaluation design, data collection problems can compromise the validity of findings. Data collection problems may affect the validity of the findings for three of the five evaluations that are currently completing or have completed data collection. The Juvenile Mentoring Program evaluation has experienced problems obtaining behavioral measures and school performance data with which to gauge program-driven change. The Comprehensive Gang Initiative evaluation has also experienced data collection problems such as the lack of fully adequate comparison youth data at all or most sites, missing police histories, and missing self-reported data.

The Intensive Aftercare evaluation has experienced survey response rate shortfalls, in some cases obtaining response rates of less than 30 percent, which may affect the validity of the findings. In commenting on a draft of this report, the Assistant Attorney General said that the poor response rates for some elements at different sites were particularly disappointing because this evaluation had a strong random assignment design; however, the strategies for obtaining adequate data turned out to be insufficient. She added that the program staff who were required to collect data did not give data collection adequate priority in comparison to their other duties. This was particularly true of data regarding the comparison groups.

OJJDP Has Funded Other Types of Evaluations

In addition to funding impact evaluations of OJJDP programs, OJJDP has funded 24 other evaluations since 1995—11 nonimpact evaluations of OJJDP programs, 9 impact evaluations of programs that were not supported by OJJDP funds, and 4 nonimpact evaluations of programs that were not funded by OJJDP. The nonimpact evaluations are not intended to determine the outcomes of the various programs, but rather how well the programs have been implemented. For example, OJJDP has funded a process evaluation of its SafeFutures program to learn more about the process of community mobilization and collaboration in building a comprehensive program of prevention and intervention for at-risk youths and juvenile offenders.

OJJDP has also funded evaluations of programs that are funded by entities other than OJJDP. For example, although OJJDP does not fund the Act Now Truancy Program, it has funded a nonimpact evaluation of this program. The Act Now Truancy Program grew out of a unique Arizona law that allowed prosecutors to issue citations to parents whose children were chronically truant. Because there was a great deal of interest in this approach and OJJDP believed it provided a unique opportunity to learn about the impact of an unusual approach, it funded an evaluation of the program. Appendix VIII contains brief descriptions of these 24 other evaluations.

Conclusions

Although there is great interest in assessing results of programs, it is extremely difficult to design and execute evaluations that will provide definitive information. Our in-depth review of 10 OJJDP-funded evaluations of OJJDP's own programs undertaken since 1995 has shown that, in some cases, the flexibility that can be beneficial to grantees in tailoring programs to meet their communities' needs has added to the complexities of designing impact evaluations that will result in valid findings. Furthermore, the lack of an appropriate comparison group or sites and/or problems in data collection may compromise the reliability and validity of some of these evaluations. Because half of these 10 evaluations are in relatively early stages, any potential problems with comparison group issues or data collection shortfalls could still be resolved over the course of the evaluation. We recognize that not all evaluation issues that can compromise results are resolvable, including many involving comparison groups and data collection. However, to the extent that appropriate comparison groups can be established and tracked and data collection issues can be overcome, the validity of the evaluation results can be enhanced.

Recommendation for Executive Action

Our review of the recent OJJDP program evaluations has shown that, of the five that are in or near their final stages, some problems with valid comparison groups and/or data collection could compromise the usefulness of some of their results. Five other program evaluations are in a formative stage where comparison group issues and data collection strategies are not yet finalized. Accordingly, we recommend that the Attorney General direct the Administrator of OJJDP to assess the five impact evaluations in the formative stages to address potential comparison group and data collection problems and, on the basis of that assessment, initiate any needed interventions to help ensure that the evaluations produce definitive results.

Agency Comments and Our Evaluation

We provided a copy of this report to the Attorney General for review and comment. In an October 15, 2001 letter, the Assistant Attorney General commented on a draft of this report. Her comments are summarized below and are presented in their entirety in appendix IX. Her detailed comments have been addressed in the report as appropriate.

The Assistant Attorney General said that the draft report provides useful information that highlights areas warranting attention. She added that the draft report would be an important tool that OJP will use to improve the quality of its evaluations and to design programs that will achieve greater impact. Furthermore, OJP will assess the five impact evaluations that are currently in their formative stages to address potential comparison group and data collection problems. On the basis of that assessment, OJJDP will initiate any needed interventions to help ensure that evaluations produce definitive results.

The Assistant Attorney General said that OJP agrees that it should always strive for more rigorous and scientifically sound evaluation designs and that the inclusion of comparison groups would certainly strengthen the interpretation of evaluation results. However, she disagreed with our reliance on the use of comparison groups as the only valid evaluation design for two primary reasons. First, OJJDP seeks to conduct juvenile justice evaluations in a real-world setting, where laboratory-like comparison groups may not be possible. Second, sufficient funding is not available for including comparison groups in every evaluation. The Assistant Attorney General also said that given the choice between conducting far fewer evaluations, all with comparison groups, and conducting a greater number of evaluations under less-than-ideal conditions, OJJDP’s Research and Program Development Division works hard to tread a middle ground that satisfies needs for both quality and quantity. She further pointed out that a growing number of policy makers and evaluators firmly believe that community-based initiatives do not lend themselves to the kind of traditional evaluations that this draft report proposes. Accordingly, some researchers have strongly urged that new approaches to evaluation be developed.

In addition, the Assistant Attorney General said that our report suggests that more evaluations using experimental or quasi-experimental evaluation designs should be funded. She added that many communities reject participation in programs that are evaluated in this way (i.e., with control or comparison groups) because they feel that it requires them to purposely exclude youths from receiving services.

In her comments, the Assistant Attorney General seemed to be using the terms “comparison group” and “control group” interchangeably. However, control groups are commonly associated with experiments involving random assignment. We do not intend our statements regarding the need for comparison groups in impact evaluations to imply that random assignment is necessary for studies to be valid. Furthermore, we recognize that groups can be compared after controlling for differences by methods other than random assignment, including statistical methods and various methods of matching. For impact evaluations, comparisons should be made, and should involve individuals who were not subject to the program or treatment being evaluated. However, not all the evaluations we assessed made such comparisons.

We also recognize that not all evaluation issues that can compromise results are resolvable, even with the use of comparison groups. We also recognize that designing evaluations with comparison groups can be expensive and funding limitations could preclude their use in all evaluations. In addition, obtaining participants can be troublesome, as the Assistant Attorney General pointed out. However, the validity of evaluation results can be enhanced through establishing and tracking comparison groups. If other ways exist to effectively isolate the impacts of a program, comparison groups may not be needed. However, we saw no evidence of other methods being used in the 10 impact evaluations we assessed. While studies that do not have appropriate comparison groups can provide useful information, they should not be considered impact evaluations. Furthermore, we recognize the fact that communities may not favor withholding treatments or programs from individuals in control or comparison groups, however, this problem is commonly handled by phasing in the treatment or program and offering it to comparison group members following the evaluation period.

As we agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. At that time, we will send copies to the Senate Judiciary Committee, the Senate Subcommittee on Youth Violence, the House Committee on Education and the Workforce, the House Subcommittee on Early Childhood, Youth and Families, the Attorney General, and the Director of the Office of Management and Budget.

If you or your staff have any questions about this report, please contact James M. Blume or me at (202) 512-8777. Key contributors to this report are acknowledged in appendix X.

Appendix I: OJJDP Awards Data, Fiscal Years 1996 Through 2000

This appendix provides information on the awards the Office of Juvenile Justice and Delinquency Prevention (OJJDP) made each year from fiscal years 1996 through 2000. It contains data on OJJDP funds awarded to formula/block grant programs versus discretionary grant programs (see fig. 2), OJJDP funds awarded by more specific program areas (see table 3), types of OJJDP award recipients (see table 4), and OJJDP formula/block grant awards by state (see table 5).

We relied on the Office of Justice Program’s (OJP) awards database to analyze data on all OJJDP-administered awards made during this 5-year period. We analyzed awards by the year the award was made—not the year in which the funds were appropriated. We worked with OJJDP officials to identify awards by major program or program area, as the database did not provide sufficiently detailed information. OJP officials advised us that they perform daily quality control checks on all data entered into their database, however, we did not verify the accuracy of the database.

Appendix II: OJJDP’s Process for Disseminating Published Products, Including Interim Results of Impact Evaluations

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) has a process for disseminating published interim results of impact evaluations as well as other publications produced by OJJDP and its grantees. OJJDP publications are available through the Juvenile Justice Clearinghouse.According to an OJJDP official, OJJDP develops a specific strategy for each publication that includes the number of copies to be printed, the methods for announcing availability, and the target audience that will automatically receive copies. OJJDP promotes products through the National Criminal Justice Reference Service (NCJRS) Catalog, OJJDP’s Juvenile Justice journal, the NCJRS and OJJDP Web sites, e-mail lists, the Office of Justice Programs press announcements, conference displays, criminal/juvenile justice newsletters and journals, and flier mailings. Almost all of OJJDP’s publications are made available to the public through OJJDP’s Web sites, which is administered by the Juvenile Justice Clearinghouse. Many publications, depending on their length, are also available through the Clearinghouse’s fax-on-demand service. Individuals can also order copies of publications online or by calling the Clearinghouse’s toll-free number. In addition, the Clearinghouse automatically sends publications to targeted constituents (e.g., juvenile justice policymakers, practitioners, researchers, and community-based organizations) and to individuals who have registered to receive publications based on their specific areas of interest.

As of May 2001, OJJDP had used this dissemination process to share interim results from 5 of the 10 ongoing impact evaluations of OJJDP programs that we assessed. In total, OJJDP had distributed over 400,000 copies of 9 products that contained interim results from the 5 evaluations. Table 6 provides additional information on the distribution of these publications.

Appendix III: Categorical Assistance Progress Report Form

Appendix IV: Descriptions of OJJDP Programs, Reporting Requirements, and Examples of Reported Information

Twice a year, the Office of Juvenile Justice and Delinquency Prevention (OJJDP) grantees are required to complete a Categorical Assistance Progress Report—a narrative report that is to include a summary of the status of their particular projects’ goals, quantitative project results based on performance measures set forth in their grant applications, actions planned to resolve any implementation problems, and any technical assistance they might need. In 8 of the 16 major programs we reviewed, grantees received only this general guidance, and were not subject to any additional reporting requirements. In the other eight programs we reviewed, grantees were required to follow this standard guidance and, in addition, report more specific information. Grantees in all 16 programs reported input, output, and/or outcome data related to the process, implementation, and/or accomplishments of their projects, such as acquisition of additional funding for a project evaluation, the number of project participants, or the number of missing children recovered.

Table 7 provides summary information on the eight programs in which grantees are not subject to additional reporting requirements and examples from grantees’ progress reports. Table 8 provides similar information regarding the other eight programs in which grantees are additionally required to report specified data to OJJDP or outside evaluators, as well as the specific performance measures on which grantees are required to report. Unless otherwise noted, the examples of reported information represent individual grantee or subgrantee data for a 6-month period. Information provided regarding the specific data on which grantees are required to report do not necessarily include all performance data required.

Appendix V: Training and Technical Assistance and Research Performance Data Reported by OJJDP Grantees

The Office of Juvenile Justice and Delinquency Prevention’s (OJJDP) training and technical assistance programs and research programs are unique in that they cut across many of OJJDP’s other programs. Also, grantees in each of these two areas typically report the same types of quantitative performance data as other grantees in their area, even though OJJDP does not usually prescribe the specific performance measures on which the grantees should report. Training and technical assistance grantees maintain the same types of data due to the common support services they provide, and research grantees do the same because they share a common goal of producing research products.

Performance Data for Training and Technical Assistance Grantees

OJJDP awards grants to training and technical assistance providers to support grantees in many of OJJDP’s grant programs. OJJDP administers the vast majority of its training and technical assistance grants through three of the Office’s divisions: (1) the Training and Technical Assistance Division (TTAD), (2) the State and Tribal Assistance Division (STAD), and (3) the Child Protection Division (CPD). Many of the training and technical assistance providers are required to report information on their projects’ activities and accomplishments semiannually using OJP’s Categorical Assistance Progress Report form, as do all OJJDP grantees.OJP provides standard guidance on information to be reported, such as information on the status of each of the grantees’ project goals and quantitative results of their projects. STAD has not imposed additional or more specific reporting requirements on its training and technical assistance providers and, for the most part, neither have TTAD nor CPD.

Officials explained that OJJDP does not require all grantees to routinely report prescribed data because it is reluctant to place additional reporting requirements on grantees due to the Paperwork Reduction Act of 1995,which set goals to reduce the federal government’s reporting and paperwork burden. Although most of these providers are not subject to additional reporting requirements for prescribed data, it is not unusual for them to report on the same or similar quantitative performance measures. Because of the nature of the services they provide, training and technical assistance providers tend to maintain like data that can readily be counted, such as numbers of training events held, practitioners who attended those events (“practitioners trained”), and technical assistance requests filled. Some of these providers also produce publications or materials, such as bulletins, surveys, curricula, brochures, and other support materials, and report such information to OJJDP. Table 9 summarizes performance data we obtained regarding training and technical assistance grants.

OJJDP officials cautioned that not all providers share common definitions of “training” and “technical assistance.” For one thing, the difference between the two is not always clear and, therefore, it is sometimes difficult to definitively categorize a provided service as training versus technical assistance. Furthermore, not all training events are equal. For example, some providers might characterize both a 1-day training conference and a 10-day training workshop as a training event; others might differentiate between the two. Furthermore, one provider might consider a telephone request from a grantee as merely a query, while another might consider it a request for technical assistance.

Performance Data for Research Grantees

OJJDP administers its research grants out of its Research and Program Development Division (RPDD). RPDD sponsors empirical studies on an array of topics related to juveniles and delinquency, from the causes of violence to the impact of victimization. The overall goal of these research grants is to generate credible and useful information to help prevent and reduce juvenile delinquency and victimization. Research grantees are not only expected to collect data but to analyze and disseminate their analyses to the public. RPDD requires all research grantees to produce publishable products and, in some instances, RPDD specifies the type of products to be published depending on the results of the research. Thus, according to OJJDP officials, one measure of a research grantee’s performance is the number of products the grantee has published.

Like most OJJDP grantees, research grantees must report information on their projects’ activities and accomplishments semiannually through progress reports. RPDD does not impose additional, specific reporting requirements on grantees, but it does encourage them to report on products produced through private publishers (as opposed to those published through OJJDP). The division director told us that it is not necessary to impose specific requirements on grantees in addition to the semiannual progress report requirements because officials work closely with grantees throughout the life of the grants.

OJJDP research grantees produce products based on their OJJDP-funded research. Some of these products are approved and published by OJJDP, who in turn disseminates the products through its own distribution process (see app. II for a description of OJJDP’s product dissemination process). Grantees also publish many products that are based on their OJJDP-funded research through private publishers. OJJDP officials told us they give their research grantees latitude to privately publish products because the majority of their research grantees are academics whose funding depends on the number of products they publish, and because grantees often have funding sources in addition to OJJDP.

Tables 10 and 11 summarize the products that active OJJDP research grantees published through OJJDP and private publishers as a direct result of OJJDP-funded research. Table 10 describes the number of research products published by OJJDP from 1993 through September 2000, by topic. Table 11 shows the number of products, by topic, that grantees with active research grants privately published between 1986 and June 2001, or were in the process of publishing in June 2001.

Appendix VI: States’ Compliance With Core Requirements of the Juvenile Justice and Delinquency Prevention Act of 1974

Appendix VII: Impact Evaluations OJJDP Has Funded of Its Own Programs

This appendix contains information on the 10 impact evaluations that the Office of Juvenile Justice and Delinquency Prevention (OJJDP) has funded of its own programs since 1995 and for which we have assessed the methodological rigor, as well as information on one impact evaluation— Teen Courts—that we did not assess. Five of the 10 evaluations are in their formative stages, and five are well into their implementation. For each of the 10, we have included a description of the program being assessed, the evaluating organization, a description of the evaluation and its findings, and our assessment of the evaluation. As discussed in the Scope and Methodology section of this report, we did not assess the methodological rigor of the Teen Courts evaluation. However, we have included a summary of this evaluation at the end of this appendix.

Impact Evaluations of OJJDP Programs in Their Formative Stages

Parents Anonymous

Program Description: Parents Anonymous is a national child abuse prevention program that began in 1970. It consists of 32 state and local organizations and over 1,000 weekly mutual support groups. The principal participants are at-risk parents, though complementary projects exist for children. The cornerstones of the program are mutual support and shared leadership.

Evaluator: National Council on Crime and Delinquency.

Evaluation description: This evaluation, which is in the beginning stages, is based on a proposal to conduct a process evaluation in year 1, and an outcome evaluation in years 2 and 3. The researchers will determine how Parents Anonymous is staffed and operated in different settings, how it attempts to change the behavior and attitudes of parents, and what factors are related to its effectiveness. While the specifics of an outcome evaluation design are yet to be determined, the researchers indicate that they will most likely compare the Parents Anonymous participants with a control group and with Parents Anonymous dropouts. The process evaluation received $300,000 for a 3-year period.

Evaluation findings: It is too early in this evaluation to have reported results.

GAO assessment: No assessment of the impact evaluation is possible because it has not yet been planned. More fully developed proposals will need to be made to OJJDP to obtain funding for the impact evaluation portion of the study.

Positive Action Through Holistic Education (Project PATHE)

Program Description: This program replicates and evaluates Project PATHE, which was first implemented in the Charleston County School District in South Carolina between 1980 and 1983. Project PATHE is a comprehensive school-based program that combines services to students who are at elevated risk for developing problem behaviors with school- wide organizational changes intended to improve both school climate and students’ behavior. Local educators are encouraged to develop their own (1) explanations for the causes of their schools’ violence and behavior problems and (2) specific local objectives and ways to prevent these problems by empowering teachers’ decision-making and fostering collaborative and nonhierarchical efforts. For various reasons, the grantee has had difficulty in selecting a school district for the replication. Funding for this effort began in October 1999 with funds provided by the Centers for Disease Control and Prevention.

In commenting on a draft of this report, the Assistant Attorney General pointed out that all funds for this effort come from the Centers for Disease Control and Prevention through an interagency agreement. OJJDP awarded this grant, which the two agencies jointly manage. OJJDP originally identified Project PATHE as an OJJDP-funded program. However, on the basis of the Assistant Attorney General’s comments, this program appears to be a non-OJJDP-funded impact evaluation. Since we assessed its methodological rigor, we have included it with the other OJJDP-funded evaluations.

Evaluator: Institute of Behavioral Science, University of Colorado.

Evaluation description: Although the impact evaluation was expected to be completed in July 2001, it has yet to begin. Original plans called for one school district to be selected for the replication and evaluation. In a school district, one high school and one middle school would be selected to receive the program, once school principals had been informed and staff surveys had been conducted to determine interest in participating. Comparison schools (one high school and one middle school) in the same district would be selected with similar demographic characteristics of students, levels of problem behaviors, and other unspecified organizational characteristics, as well as a low probability of mounting other school-wide efforts to reduce problem behavior during the study period. Plans are to collect data before and after project implementation. All students and staff in all schools would be surveyed in September and May for 3 consecutive school years. In addition, 200 students from each high school—100 seniors and 100 sophomores—and 200 students from each middle school—100 eighth graders and 100 sixth graders—would be sampled in the first year of the study for followup for 3 years. Each sample is to include 25 students identified as “high risk.” It is not clear, however, how these samples will be selected. Schools are to be visited three times yearly during the study, and school records and teacher ratings of student behavior in each school will be used in establishing differences between the program and comparison schools. Program outcomes are to be selected after determining goals and objectives collaboratively with local school officials. Multivariate statistical analyses, such as logistic regression, are planned. Because of the difficulty in selecting a school district the latest progress report indicates some changes in this design. Agreements have been signed with two school districts (Charleston, SC and Baltimore, MD), instead of one, and plans are to conduct the replication and evaluation in four middle schools (two program and two comparison schools) in each district—high schools have been excluded. The amount of the grant is $875,000. However, additional funds have been requested.

Evaluation findings: It is too early in this evaluation to have reported results.

GAO assessment: The evaluation, as designed, is basically sound. The variation in program structure and implementation between schools may limit generalizability. While the researchers do suggest awareness of potential problems due to students switching schools, they do not clearly indicate, at this early design stage, how possible contamination will be handled.

Rural Gang Initiative

Program Description: The Rural Gang Initiative is a comprehensive strategy to ameliorate gang problems in rural areas. The program was adapted from the comprehensive gang model, developed at the University of Chicago, and implementation began in two rural areas in fall 2000. The program consists of five elements: community mobilization, opportunities provision, social intervention, suppression, and organizational change and development.

Evaluator: National Council on Crime and Delinquency.

Evaluation description: The impact evaluation of this program began in January 2001 and is expected to be completed in December 2003. It focuses on two of the four sites, Mount Vernon (IL) and Glenn County (CA), that were part of a year-long feasibility study that began in April 1999. The other two sites are not part of the evaluation because they did not fully implement the model. Information will be collected from these two sites on gang-involved youths and youths at-risk of gang involvement, all of whom have participated in the program. However, it is unclear how the youths will be sampled or whether all participants will be included. Data will be obtained from a variety of sources, including interviews, organizational surveys, and administrative data from schools and the justice system. No comparison groups are planned, though the researchers indicate that attempts will be made to collect data that will permit an assessment of alternative explanations for program effects. The researchers plan to collect data before and after program implementation; then they will measure any changes and follow-up with program participants for at least 12 months after their participation in the program. Individual sites are to identify specific program outcomes. Although the researchers have described prospective outcome measures, such as the reduction of gang-related crime and the prevention or reduction of gang involvement, they (1) have not chosen the outcome measures that they will use and (2) have not provided information on the types of statistical analyses planned. However, the evaluation is still in the formative stages. At the time of our review, this evaluation had received $525,000 in funding.

Evaluation findings: It is too early in this evaluation to have reported results.

GAO assessment: It is too early to tell from this evaluation how effective it will be. However, the absence of plans for comparison groups and the lack of specificity regarding any other control mechanisms, make it unclear how program effects will be distinguished from alternative explanations at this stage of the evaluation.

Safe Schools/Healthy Students

Program Description: The Safe Schools/Healthy Students program has funded 77 school districts nationwide, with grants ranging up to $3 million, to develop services and activities to promote healthy childhood development and prevent violence and drug abuse. The program also aims to develop greater collaboration and cooperation between communities and schools to enhance their effectiveness in responding to and reducing violence. Each project model is intended to evolve over time.

Evaluator: Research Triangle Institute.

Evaluation description: This impact evaluation began in October 1999 and data will be collected through the Spring of 2005. School and community-based archival records, surveys of key coalition personnel, teachers, superintendents, principals and other school staff, and teacher behavioral checklists for students in selected grades will be gathered in all 77 school districts. The evaluation will compare data from participating sites with national norms and with similar information from matched nonparticipating (comparison) sites that the researchers surveyed in each of two large, nationally representative studies of school districts. The matching will be based on unspecified socio-demographic characteristics and responses to policy-related questions in the baseline survey. Archival data will be collected yearly over a 5-year period. Survey data will generally be collected at three points in time, about 2 years apart. The survey items will be drawn from established instruments and will provide, in conjunction with the archival or administrative data, information on behavioral outcomes, risk factors and inhibiting factors, and indicators of positive development and mental health. At the time of our review, this evaluation was funded at approximately $5.6 million.

Evaluation findings: It is too early in this evaluation to have reported results.

GAO assessment: This evaluation, as designed, is basically sound.

Safe Start Initiative

Program Description: This demonstration program seeks to prevent and reduce the impact of family and community violence on young children, primarily aged 0 to 6, in up to 12 communities. The program plans to create a comprehensive service delivery system that integrates service providers (in the fields of early childhood education/development, health, mental health, and all manner of prevention, intervention, and treatment programs), law enforcement, legal services, and the courts. It also seeks to improve the access, delivery, and quality of services to children exposed to, and at high risk of, violence. Project sites are to be selected through a competitive grant process. Funding for the program began in October 1999.

Evaluator: Caliber Associates.

Evaluation description: This evaluation began in May 2000, and is expected to end in September of 2005. The effect of the Safe Start Initiative will be measured within and across all participating communities at both the community and individual levels. Multiple data collection methods, including focus groups, service agency usage logs and documents, and random-digit-dialing telephone surveys, will be used. Plans are to collect data before program implementation and for 4 years after the program begins in each site. Although specific outcome measures have not yet been identified, they are expected to address such areas as agency referral levels and quality of service, increased interagency collaboration, knowledge and perceptions of police and child protective services, rates of child maltreatment, physical injuries, and mental health problems. No comparison communities are to be studied. Analyses are to include regression and time series models. At the time of our review, this initiative had received $1 million.

Evaluation findings: It is too early in this evaluation to have reported results.

GAO assessment: The absence of any appropriate comparison communities and the variability in program implementation and components across the 12 study sites will make it difficult to find compelling evidence of program effects.

Impact Evaluations of OJJDP Programs Well Into Their Implementation

Comprehensive Communitywide Approach to Gang Prevention, Intervention, and Suppression Program (Comprehensive Gang Initiative)

Program Description: This program aims, in the five sites in which it is being implemented and evaluated (Mesa and Tucson, AZ; Bloomington, IL; San Antonio, TX; and Riverside, CA) to reduce gang-related crime through five interrelated strategies: community mobilization; provision of social, educational, and economic opportunities; suppression of gang violence; social intervention; and organizational innovation. It involves the collaborative efforts of the police, probation officers, prosecutors, judges, schools, youth agencies, churches, housing authorities, and governmental agencies. It targets youths at strong risk of gang membership and crime, and youths already involved in serious gang crime. The evaluation began in 1995.

Evaluator: University of Chicago.

Evaluation description: The impact evaluation of the program began in May 1995, and is expected to be completed in April 2002. Each project site is to be matched with a comparison site. In four of the five sites, the program participants and comparison groups were selected from similar gang problem areas within the same city; in the fifth site, a separate comparison community was selected. Between 100 and 115 youths, ages 12 to 21, who were involved in gangs or at risk of involvement, were selected to participate in the program in each site, and between 77 and 134 similar youths in each site were selected for comparison purposes. Neither program nor comparison group youths were selected randomly. A large, complex, communitywide data collection effort is being employed in each site, through a variety of methods and sources, including organizational surveys, youth surveys, reports from service workers, police and school records, local newspaper reports, and census data. Data were to be collected at baseline and after the first and third years of the program. The principal outcomes to be measured are gang crime patterns at the individual, gang, and community levels. The evaluation will also consider changes in opportunities, as well as integration in and alienation from conventional individuals and institutions. A variety of analyses of the data are planned, including time-series analyses and hierarchical linear models. At the time of our review, the evaluation had received approximately $3 million.

Evaluation findings: Preliminary results have been reported for program participants, but not comparison groups. Because of missing data and other problems, reporting deadlines may not be met, and two of the five project sites and associated comparison sites have been deferred from the current analyses.

GAO assessment: The evaluation, as designed, is basically sound. However, numerous difficulties in obtaining data threaten parts of the evaluation. Also, the way in which subjects were selected for the study may be problematic, and it is unclear how program and comparison youths were matched within sites.

Enforcing the Underage Drinking Laws Program

Program Description: Since 1998, 85 communities and 4 colleges in at least 10 states have been awarded subgrants under the discretionary grant component of this program to enforce underage drinking laws. In most states, a diverse group of stakeholders are involved in planning a variety of projects under this program that can include media campaigns, merchant education, compliance checks and other enforcement, youth leadership training, school-based education, and the development of local coalitions and interventions aimed at reducing underage drinking. States and communities are given substantial latitude in planning their projects; interventions are not standardized across communities.

Evaluator: Wake Forest University School of Medicine.

Evaluation description: The effort to evaluate the discretionary grant component of this program began October 1, 1998, and is expected to be completed December 31, 2001. Data are to be collected—from telephone surveys of police chiefs, sheriffs, and youths in participating communities and matched comparison communities in at least 9 states—before or early on in project implementation and at least 1 year after project initiation. Project sites to be evaluated were initially chosen from all states participating in the program. It is unclear whether these sites are representative of all participating project sites. In the first year, surveys were conducted in 52 participating communities and a similar number of comparison communities. In the second and third years, surveys will be conducted in those same communities and 34 others—17 in each group. The participant and comparison communities were matched on median income, liquor law violations, percentage attending college, and population size. The surveys of the top one or two law enforcement officials in each community will provide information on local law enforcement efforts, including the number of compliance checks conducted in each year. A small number of youths from each site are to be selected at random for the surveys each year. The youth surveys will obtain data on perceptions of alcohol availability, peer and personal alcohol use, and alcohol-related problem behaviors including binge drinking and drunk driving. At the time of our review, this evaluation had received approximately $945,000.

Evaluation findings: While some demographic data have been reported from the baseline survey, no results have been reported involving program effects.

GAO assessment: The researchers suggest aggregating all program communities together and all comparison communities together to diminish community sample size problems, which may mask program effects. In addition, the wide variation allowed in program implementation may compromise the interpretation and generalizability of any findings.

Intensive Aftercare

Program Description: The Intensive Aftercare program provides intensive supervision and services to serious juvenile offenders for 6 months following their release from secure confinement. The goal is to facilitate reintegration and reduce recidivism. The program was implemented, beginning in June of 1993, by various youth service offices and departments of corrections in four states: Colorado, Nevada, New Jersey, and Virginia. New Jersey was eventually dropped because of implementation problems, so the evaluation of the program is being completed in the other three states.

Evaluator: National Council on Crime and Delinquency.

Evaluation description: Beginning in 1995, youths entering correctional facilities in the three states (four counties in Colorado including Metropolitan Denver; Clark County, NV; and Norfolk County, VA) were screened for eligibility and randomly assigned, within each site, to the treatment group (whose members participated in the Intensive Aftercare program upon release) or control group. Between 1995 and 1999, 82 youths were assigned to the program and 68 to the control group in Colorado, 120 youths were assigned to the program and 127 to the control group in Nevada, and 75 youths were assigned to the program and 45 to the control group in Virginia. Information was collected for study participants at baseline (that is, upon entry into the institution), before release from the institution (9 to 12 months after baseline, or entry), immediately after completing the program (6 months after release), and 6 months after completing the program (12 months after release). The data collected, using survey instruments, standardized tests, monthly case management forms, and administrative (police and court) databases, included social and criminal history and demographic data, information on the extent of supervision and services, and the extent of criminal activity following institutional release. Many of the measures being employed in the study, according to the researchers, are standard and have been validated. At the time of our review, the evaluation had received approximately $932,000 in funding.

Evaluation findings: The preliminary findings offered from this evaluation suggest that the Intensive Aftercare participants did receive greater supervision and more services after release than the control group, which suggests some success in implementing the program. Outcome results related to reintegration and recidivism are not complete, and the interim results are mixed as to whether the program is associated with positive outcomes.

GAO assessment: This is a well-designed study, though serious missing data problems, if not corrected, may make it difficult to determine the outcome of this program.

Juvenile Mentoring Program (JUMP)

Program Description: JUMP was established by Part G of the Juvenile Justice and Delinquency Prevention Act of 1974, as amended in 1992. Through that legislation, the Congress authorized OJJDP to award 3-year grants to community-based not-for-profit organizations or to local educational agencies. The grantees are to support one-on-one mentoring projects that match volunteer adult mentors with youths at risk of delinquency, gang involvement, educational failure, and dropping out of school. The legislation also provided funding for a national, cross-site evaluation of JUMP. OJJDP guidelines emphasize the need for projects to recruit, train, supervise, and do thorough background checks for all volunteer mentors; develop procedures for appropriately matching youths and mentors; define the population of at-risk youths to be served; develop guidelines for the type, frequency, and duration of youth and mentor project activities; and establish procedures for gathering and reporting data to support the evaluation process. As of November 2000, 175 JUMP projects had been funded, in amounts ranging from $180,000 to $210,000 over a 3-year period.

Evaluators: Information Technology International and Pacific Institute for Research and Evaluation.

Evaluation description: This evaluation began in May 1997 and is expected to conclude on September 30, 2002. Three approaches are being taken to determine how well JUMP is accomplishing its objectives. The first is a modified pre-post design that involves a within-subject comparison of the characteristics of youths at the time they enter and exit the program and between-subject comparisons of youths entering and exiting the program at the same time. The second approach is a best practices approach that will use structural equation models to estimate what program features or activities, including success in matching mentors and youths, are most likely to contribute to program success in reducing the risk of school and family problems, delinquency, and drug use among youths. The third approach relies on combined youth outcome data and community data to determine community cost offsets. The evaluation was funded at $3.3 million.

Evaluation findings: In their November 2000 JUMP annual report, the evaluators provided considerable descriptive information about the various JUMP projects, the characteristics of the youths and mentors, and information on youth-mentor matching. The only “outcome” information thus far provided, however, is information on how satisfied youths and mentors were with the mentoring experience and how much benefit each perceived was derived from the experience. None of the three analytic approaches described above has been successfully applied to study outcomes because of a variety of pitfalls experienced by the national evaluation team, most notably insufficient data on school performance and behavioral measures (e.g., delinquent behavior and arrests).

GAO assessment: The researchers are employing multiple and innovative strategies to determine the effectiveness of JUMP in achieving its objectives. It is not clear, however, whether definitive evaluation results can be reached in the absence of outcome data on youths who, in the same project areas at the same points in time, do not receive the program. In addition, data limitations, if not corrected, may be serious enough to compromise findings.

Partnerships to Reduce Juvenile Gun Violence Program

Program Description: The Partnerships to Reduce Juvenile Gun Violence Program is a multi-year demonstration program planned for four sites (Baton Rouge and Shreveport, LA; Syracuse, NY; and Oakland, CA). It began in 1997 and is expected to conclude in 2001. However, one site, Shreveport, was dropped from the program early. The program aims to reduce youth gun violence by enhancing, in specific target areas of these cities, prevention and intervention strategies and strengthening partnerships among community residents, law enforcement agencies, and the juvenile justice system. The program involves mobilizing the community, establishing agency linkages, and planning case management for juveniles with gun charges in year 1, linking at-risk youths to services in year 2, and expanding opportunities for youths in year 3.

Evaluator: COSMOS Corporation.

Evaluation description: The strategy for evaluating the impact of this program has evolved as the program has unfolded. An impact evaluation was planned for three sites and was to include (1) a comparison of changes in crime rates in target areas of these cities before and after the implementation of the program, (2) a comparison of responses from high- risk youths in targeted areas surveyed before and after the program was implemented and services were provided, and (3) information on changes in policies and caseloads revealed through focus group meetings and interviews with agency officials. Crime rate information has thus far been reported only for Oakland and Baton Rouge, and surveys have been conducted only in Baton Rouge. In Baton Rouge, surveys were given to 92 high-risk youths in the criminal justice system identified through a variety of processes. The sampling strategies for surveying these high-risk youths were unlikely to yield generalizable results. In addition, fifth-, seventh-, and ninth-grade students in six schools in the target area were surveyed in March of 1999. It is unclear why these students and schools were sampled and what response rates were. In 2000, a small sample of 50 youths in Baton Rouge was identified as a possible matched comparison group for arrest rate comparisons. At the time of our review, this evaluation had received $1.2 million in funding, although a process evaluation is also being conducted with these funds.

Evaluation findings: The researchers report decreases in gun-related homicides and arrests in Oakland that were larger in the target area than for the city as a whole. They also report decreases in gun-related homicides in Baton Rouge. No analyses of results from the survey data have been reported to date.

GAO assessment: Comparisons between crime rates in the target community and the city as a whole may not be appropriate. Student and school selection criteria are unclear, making it difficult to assess their appropriateness for obtaining definitive results. In addition, if supporting survey and administrative data are only gathered in one site, it will be very difficult to generalize findings whether they appear positive or not.

Impact Evaluation of OJJDP Program— Summary of Evaluation We Did Not Assess

Teen Courts

The purpose of the Teen Courts evaluation is to measure the effect of handling young, relatively nonserious violators of the law in teen courts, rather than in traditional juvenile family courts. Although teen courts often include many of the same steps used by formal juvenile courts (for example, intake, preliminary review of charges, court hearing, and sentencing), they differ from formal courts in that young people are able to assist in the community decision-making process for dealing with juvenile offenders. Youths may act as prosecutors, defense counsel, jurors, court clerks, bailiffs, and judge (or as a panel of judges). To evaluate teen courts, both a process and impact evaluation are used, with case studies and comparison groups as part of the research design. In each of the four case study sites (Anchorage, AK; Independence, MO; Maricopa County, AZ; and Rockville, MD), data are collected on about 100 youths handled in teen courts (experimental group) and 100 youths handled in the traditional juvenile justice system (comparison group). Data are also collected on several dimensions of program outcomes, including post-program changes in teens’ perceptions of justice and their ability to make more mature judgements as a result of the program. A process evaluation of the projects—exploring legal, administrative, and case-processing factors that hinder the achieving of project goals—is also being conducted.

Appendix VIII: Twenty-Four Other Evaluations Funded by OJJDP

This appendix contains summaries of the 24 evaluations the Office of Juvenile Justice and Delinquency Prevention (OJJDP) has funded since 1995 (excluding the 11 impact evaluations it has funded of its own programs discussed in app. VII). For 22 summaries, we used descriptions of evaluations that were provided to us by OJJDP; for 2 summaries, we wrote the descriptions based on OJJDP documents. OJJDP categorized the 24 evaluations into the following three groups: OJJDP-funded programs: nonimpact evaluations (11). Non-OJJDP-funded programs: impact evaluations (9). Non-OJJDP-funded programs: nonimpact evaluations (4).

OJJDP-Funded Programs: 11 Nonimpact Evaluations

Community Assessment Centers

The purpose of this evaluation is to test the feasibility and effectiveness of the OJJDP community assessment center concept in different environments. Community assessment centers seek to facilitate earlier and more efficient delivery of prevention and intervention services at the front end of the juvenile justice system. The evaluation uses a two-phase process to (1) measure some outcomes at the two enhancement sites, with quasi-experimental design, and (2) achieve more and better outcome measures. But, according to OJJDP, implementation and data problems will limit the effectiveness of the quantitative methods employed. OJJDP also believes that the attempt to implement a random assignment study at one project site will probably need to be abandoned. The first phase covers the four project sites that comprised the Community Assessment Centers program—two of these sites funded enhancements to existing programs and the other two funded the planning and implementation of new programs. The second phase covers the two project sites—one enhancement and one planning—in which the program is being continued after the end of the first funding cycle. Many of the evaluation measures are at the project or community level rather than at the participant level.

Community Prevention Grants Program (Title V)

The purpose of this evaluation is to examine the viability and effectiveness of the community-based delinquency prevention model used by grantees in the Community Prevention Grants Program. The Community Prevention Grants Program encourages communities to develop comprehensive, collaborative plans to prevent delinquency. The evaluation focuses on two main questions: (1) What is the impact of the program on community planning, service delivery, risk factors, protective factors, and juvenile problem behaviors? (2) What factors and activities lead to the effective implementation of the Community Prevention Grants Program model and to positive program outcomes? This evaluation employs a case study approach supplemented by a basic profile of communities that are participating in the program. Case studies are to be implemented in 11 communities in 6 states. Evaluation measures are to be applied at the project, community, and program levels.

Comprehensive Strategy for Serious, Violent, and Chronic Juvenile Offenders (Comprehensive Strategy)

The purpose of this process evaluation is to address the following questions about the Comprehensive Strategy program: (1) What are the factors associated with successful Comprehensive Strategy planning and implementation? (2) To what extent do project sites adhere to the prescribed Comprehensive Strategy framework? (3) What are the major implementation challenges program grantees face in implementing the Comprehensive Strategy? (4) To what extent does the training and technical assistance provided to project sites help them acquire the knowledge, skill, and tools necessary to develop the Comprehensive Strategy? (5) What role should OJJDP play in the future implementation of the Comprehensive Strategy? The Comprehensive Strategy is OJJDP’s approach for addressing juvenile violence and delinquency at the community, state, and national levels through a systematic plan. It advocates the use of local planning teams to assess the factors and influences that put youths at risk of delinquency, determine available resources, and establish prevention programs to either reduce risk factors or provide protective factors that buffer juveniles from the impact of risk factors. This evaluation uses a multilevel design to assess how project sites implement the Comprehensive Strategy. The evaluation began with telephone interviews with site coordinators from all 48 project sites; 25 of the 48 project sites were randomly selected for a stakeholder survey. One year later, 10 of the 25 project sites are being given a second stakeholder survey. Subsequently, five sites are to be selected for visits, and intensive case studies are being done in three cities.

Drug-Free Communities Support Program

The purpose of this evaluation is to examine (1) community coalitions’ developmental processes from the early planning and adoption stages through implementation and later stages and (2) the impact of coalitions’ prevention efforts concerning risk and resiliency factors and, to the extent feasible, alcohol, tobacco, and other drug use. The Drug-Free Communities Support Program provides grants to community coalitions to strengthen their efforts to prevent and reduce young people’s illegal use of drugs, alcohol, and tobacco. The evaluation is studying two cohorts of program grantees—those that received grants in 1998 or 1999 (cohort 1) and those that received grants in 2000 (cohort 2). The national evaluation sample is comprised of a total of 213 grantees. The sample is divided by years of operation: 1-5 years, 6-9 years, and more than 9 years. Semiannually, cohort 1 grantees are required to submit progress reports to OJJDP and the evaluator that include a special section (Part II), which provides information about the compositions of the coalitions and outcome data collection. Cohort 2 grantees do not have a Part II reporting requirement and submit progress reports semiannually only to OJJDP. In addition, 21 grantees (15 from cohort 1 and 6 from cohort 2) serve as intensive study sites, where interviews with staff and stakeholders provide greater detail about coalition development and local program evaluation.

Juvenile Accountability Incentive Block Grants Program

The purpose of this process evaluation is to provide feedback to OJJDP on the implementation of the Juvenile Accountability Incentive Block Grants Program. The program encourages states and local jurisdictions to implement accountability-based programs and services in 54 states and U.S. territories. The evaluator is surveying state and local practitioners, policy makers, and grant program administrators about their perceptions and attitudes about the program and its administration. Specifically, the evaluation focuses on (1) understanding how states and local units of government plan for and administer program funds and (2) examining the perceptions of states and local units of government about how well the program is achieving congressional intent. In addition, in-depth case studies are conducted at a limited number of sites.

Performance-based Standards Project

The purpose of this evaluation is to (1) provide feedback to the Performance-based Standards Project team on improving design and implementation support to the sites, (2) assist the project team in refining the Performance-based Standards Project model and in maximizing responsiveness to the needs of the participants, that is, those who are implementing the project model, and (3) chronicle the development of the project and summarize lessons learned. OJJDP established the Performance-based Standards Project to improve the quality and conditions of juvenile corrections facilities. Specifically, the project develops and implements outcome standards and an assessment tool. Corrections facilities can use both to monitor progress towards meetings goals in areas of operations, such as health and safety. The evaluation uses a case study approach. This approach consists of the collection of both quantitative and qualitative data describing the processes used to implement the project model in 80 juvenile detention and correctional facilities across the country. Site visits are made and in-depth case studies are planned. An all-site survey is distributed to key participants to determine satisfaction with the supports provided to them in the implementation of the project model. In addition, the survey seeks the participants’ assessment of (1) the impact the project has made on conditions of confinement and management of the facilities and (2) the overall utility of the project model.

Prenatal and Early Childhood Nurse Home Visitation Program

The purpose of this evaluation is to (1) determine the extent to which replication project sites have been able to conform to the original program model, and (2) assess the “prosocial” (that is, positive, socially-oriented behavior) outcomes for mothers and their babies. The Prenatal and Early Childhood Nurse Home Visitation Program consists of intensive and comprehensive home visitation by nurses during a woman’s pregnancy and the first 2 years following the birth of her first child. The evaluation involves six project sites and employs a quasi-experimental design with matched comparison groups.

Partnerships to Reduce Youth Violence and Delinquency Program (SafeFutures)

This purpose of this process evaluation is to document and understand the process of community mobilization and collaboration. SafeFutures is designed to build a comprehensive program of prevention and intervention strategies for at-risk youths and juvenile offenders. The program comprises six project sites that represent urban, rural, and Native American communities. The evaluation is examining all six project sites. Project sites collect and record performance data on program operations and client outcomes using the Client Indicator Data Base. The project sites are required to collect extensive information from selected SafeFutures program components on individual participants’ risk and protective factor profiles, youths’ service utilization, and agencies’ coordination of services for youth during the course of their involvement in the SafeFutures program. In addition, they collect information on outcome measures regarding youths’ educational commitment (that is, school attendance, achievement, and behavior), youths’ involvement in delinquency and crime, and any changes in youths’ risk profiles. Analysis of these data will provide a picture on program performance in three key areas: reaching the intended high-risk youth clientele, coordinating services for youths with multiple problems, and monitoring subsequent school performance problems and involvement in the juvenile justice system.

Safe Kids/Safe Streets

The purpose of this evaluation is to document the lessons learned and factors associated with the successful development and implementation of the Safe Kids/Safe Streets program. The Safe Kids/Safe Streets program is designed to (1) help communities break the cycle of early childhood victimization and later criminality and (2) reduce child abuse and neglect, as well as the child fatalities that often result. The evaluation is surveying five project sites through five data collection strategies: agency administrative data, case tracking, key informant interviews, surveys of agency professionals, and surveys of stakeholders.

Tribal Youth Program

The purpose of this evaluation is to (1) support culturally appropriate process and outcome evaluations of activities funded under Tribal Youth Program grants and (2) build the capacity of tribes to better evaluate their own juvenile justice programs and activities. The Tribal Youth Program assists grantees in developing projects, within tribal communities, for the prevention and control of youth violence and substance abuse. The evaluation is participatory in nature, that is, project personnel and stakeholders will be involved in developing the evaluation designs, with the assistance and guidance of an evaluation facilitator. The five project sites are implementing different projects and have not yet completed their evaluation designs. According to OJJDP, it is too early in the evaluation to tell exactly what designs are to be used. OJJDP has required that all evaluations be designed to examine both program implementation and program outcomes.

Truancy Reduction Demonstration Program

The purpose of this process evaluation is to (1) determine how community collaboration can affect truancy reduction and lead to systemic reform and (2) assist OJJDP in the development of a model for a truancy reduction program, including identifying the essential elements of that model. The Truancy Reduction Demonstration Program encourages communities to develop comprehensive approaches—involving schools, parents, the justice system, law enforcement, and social service agencies—in identifying and tracking truant youths. The evaluation is employing site visits, interviews with key personnel, and case studies of individual sites. Process data are gathered from all seven project sites participating in the evaluation and, from some sites, limited outcome data are gathered.

Non-OJJDP-Funded Programs: Nine Impact Evaluations

Adolescent Female Offenders (Three Programs)

The purpose of this evaluation is to evaluate the efficacy of three Adolescent Female Offenders programs in Wayne County, Michigan. The three programs are (1) a program incorporating gender-specific programming, home-based intervention, and community involvement, including pregnant and parenting adolescents; (2) an intensive probation program with limited gender-specific programming; and (3) a traditional, female-only residential program that provides limited gender-specific training. The evaluation is using a quasi-experimental design. Using random assignment, the home-based intervention model is to be compared with the established intensive probation model; the outcomes of these models are then to be compared with outcomes of the traditional, female-only residential program. The comparison analysis involves at least 50 young women in each of the 3 programs. A wide range of outcomes— including recidivism, substance use, depression, community integration, academic performance and career aspirations, parenting readiness, and responsible sexual behavior—is to be examined. The evaluator is also exploring the relationship of specific program components to these outcomes.

Coping with Life Course

The purpose of this impact evaluation is to evaluate the effectiveness of a cognitive-behavioral group intervention. The Coping With Life Course is aimed at enhancing prosocial coping and problem solving for adolescents incarcerated in youth correctional facilities. To evaluate the program, a minimum of 120 adolescents in one youth correctional facility are randomly assigned to either the Coping with Life Course intervention group or a standard-care control group. Six Coping with Life Course cohort groups of 10 each are followed. The evaluation is allowing for attrition (from the initial 60 participants down to 48) in each of the intervention and control groups. Participant functioning is assessed before and after intervention through a battery of questionnaires. Recidivism, return to close custody, and service utilization are tracked through databases and statewide records.

Creation and Implementation of a Family Index in Riverside County, California, Courts

The purpose of the evaluation is to (1) document the implementation of a new “family index” case management system (through a process evaluation) and (2) examine the impact of the family index on juvenile court case processing (through an impact evaluation). The family index system allows cross-referencing to identify all family members involved in family law; juvenile dependency; juvenile delinquency; and criminal, civil, and probate matters. For the process evaluation, a case study approach is used to describe the implementation of the family index at one project site, the Riverside, California, Court. For the impact evaluation, a pre-post design is used to examine how the family index has affected juvenile court matters (for example, court processing time, coordination between courts, and content of hearings).

Flashpoint

The purpose of this evaluation is to evaluate the effects of Flashpoint on the antisocial patterns of juvenile offenders’ thoughts and actions and high school students’ thoughts and actions. Specifically, changes are assessed for (1) media use and literacy, (2) violence-supporting beliefs and behavior, and (3) substance use and abuse. The Flashpoint program is designed to build critical thinking skills young people need to (1) see through false media portrayals that glorify violence and drug use and (2) apply decision-making in their own lives. The evaluation is using comparison groups, pre-post interventions, and case studies. Participants include 264 juveniles, ages 14 to 17. Treatment groups are compared with no-treatment control groups for baseline-to-posttest changes. Three groups and project sites are involved: (1) repeat and serious offenders in a correctional program, (2) first-time offenders in a diversion program, and (3) students in a public high school.

Free to Grow: Head Start Partnerships to Promote Substance-Free Communities

The purpose of this evaluation is to determine if Free to Grow can reduce substance abuse (alcohol use, smoking, and illegal drug use). The Free to Grow program builds on existing Head Start programs, adding community- strengthening and family-intervention components to address the problem of substance abuse. The evaluation is attempting to determine the independent effects of these two components on substance abuse prevention. It involves 16 project sites and 16 comparison sites and employs a multistage experimental research design.

Gaining Insight Into Relationships for Lifelong Success Project

The purpose of this evaluation is to provide a process and outcome evaluation of the Gaining Insight into Relationships for Lifelong Success Project. The project involves two primary levels of intervention: (1) a psycho-educational counseling group, dealing with relationships and involving girls in four relational domains (relation to self, family, peers, and teachers), and (2) a focus on individual consultations, educational workshops and the policies and procedures of the local juvenile justice system, and the involvement of court service workers from the system. Specifically, the evaluation will (1) investigate the applicability of a relational approach to the treatment of female juvenile offenders; (2) examine the components of the relational approach that deal with relationships to self, family, peers and teachers; (3) evaluate the impact of increasing the knowledgebase of professionals involved in the local juvenile justice system; and (4) provide an empirically based, alternative treatment model that can be replicated in other settings. The evaluation of the first level of intervention—the counseling group—focuses on each of the four relational domains through the use of multimethod data collection; this collection includes self-reports and other reports, school records, and recidivism data. The evaluation of the second level of intervention focuses on the court services that workers use, specifically gender-sensitive treatment recommendations and referrals; qualitative observational data, gathered from monthly meetings, will be used. There is random assignment between girls referred to either the project intervention or to the standard intervention currently being used by the Clark County Court in Athens, Georgia. Approximately 180 girls—90 referred to the project intervention and 90 referred to the standard court intervention—are to be evaluated.

Quantum Opportunities

The purpose of this evaluation is to assess the implementation and the impact of Quantum Opportunities. The Quantum Opportunities program is designed to reduce the incidence of delinquency, criminal behavior, and subsequent involvement in the criminal and juvenile justice systems amongst educationally at-risk inner city youths. The evaluation is using an experimental design with random assignment. Ninth-grade students at six sites are randomly assigned to treatment and control groups, with the treatment group enrolled in the Quantum Opportunities program. The students are followed through their high school careers and 2 years beyond. Information is collected from academic achievement tests, administered each year, and from two questionnaires.

Restorative Justice Conferences

The purpose of this evaluation is to test the impact—on recidivism, program completion, and victim satisfaction—of the Restorative Justice Conferences for a population of youthful offenders (aged 14 and under) in an urban setting (Indianapolis, IN). Restorative Justice Conferences bring together the offender, victim, and supporters of each so as to provide an opportunity for fuller discussion of the offense; the effect of the offense on the victim, the offender’s family, and greater community; and steps the offender can take to make amends. The evaluation is using a single-site evaluation with an experimental design. As part of the design, youths are randomly assigned to a treatment group (Restorative Justice Conferences) or a matched control group.

Risk-Focused Community Policing

The purpose of this evaluation is to determine if the program reduces the amount of delinquency in a city. The Risk-Focused Community Policing program increases protection by the community police, potentially reducing delinquency. The evaluation is using an experimental research design. The project site (city) is divided into approximately 40 census blocks, with 20 blocks randomly selected as program blocks and the other 20 designated as control blocks.

Non-OJJDP-Funded Programs: Four Nonimpact Evaluations

Act Now Truancy Program

The purpose of this evaluation is to study the Act Now Truancy Program. The program is a prosecutor-led truancy reduction program. The evaluation is using a pre-post intervention design involving one project site. Information is collected and aggregated (for example, truancy rates rather than individual truancy behavior) for all participants.

Childhood Violence Prevention Program (Let’s Talk About...)

The purpose of this evaluation, conducted in two schools, is to assess the impact of the Childhood Violence Prevention Program. The program is designed to prevent the legitimization of aggression among pre-adolescent, elementary, and middle school children, with special focus on victims of child maltreatment. The evaluation is using a pre-post intervention design, with comparison groups. The study involves having elementary school students participate in a class activity using a workbook designed to encourage problem solving action rather than aggressive behavior in interactions with peers.

Community-Level Programs for Youth

This project is not an evaluation, per se, but rather a synthesis of existing evidence on community-level interventions and service programs. Its purpose is to identify the strengths and weaknesses of community-level evaluations and to provide recommendations to the field about how to structure and carry out such evaluations. Community-level programs for youths are designed to promote positive youth development. To evaluate the programs, a committee—experts from several disciplines (child and adolescent development, child health, sociology, psychology, evaluation research, youth services, and community development)—is assessing the strengths and limitations of measurements and methodologies that have been used to evaluate these interventions.

Department of Labor’s Education and Training for Youthful Offenders Initiative

The purpose of this evaluation is to assess the activities undertaken by project sites, determine whether they can be evaluated, and ultimately assess the impact of these activities on the youthful offenders participating in the program. The program is intended to (1) enhance school-to-work education and training in juvenile correctional facilities and (2) improve youthful offenders’ transition into the community. The evaluation design has not been completed, but random assignment study is strongly preferred, if feasible. At the time of our review, only one of the three potential sites could be evaluated. One more project site is to be awarded and, if it can be evaluated, it will be added as a second evaluation site.

Appendix IX: Comments from the Department of Justice

The following are GAO’s comments on the Department of Justice’s October 15, 2001, letter.

GAO Comments

1. As we indicated in our report, impact evaluations, such as the types that OJJDP is funding, can encounter difficult design and implementation challenges. (See section titled, Evaluations of OJJDP Programs are Difficult to Successfully Design and Implement.) Also, we are aware that virtually all impact evaluations have limitations. However, where possible, impact evaluations should be designed to mitigate as many rival explanations of program effects as feasible, and potential limitations of the chosen research design should be acknowledged. 2. Our statement that the Juvenile Mentoring Program evaluation “has experienced problems obtaining behavioral measures and school performance data” was not intended to criticize the evaluators’ level of effort, but rather to indicate that their inability to obtain data from school and law enforcement officials in many of the study sites makes it more difficult to evaluate how well the program is achieving its objectives of diminishing delinquency, gang involvement, and school failure. While enhanced analysis of sites with the best data may be warranted, it does not overcome the problem of having a large number of sites with little or no reliable data from school and law enforcement officials. This problem was explicitly recognized by OJJDP in its November 2000 report. 3. During the course of our review, OJJDP officials told us that one measure of a research grantee’s performance is the number of products the grantee has published. These officials provided us a listing of all products published by active research grantees through OJJDP and private publishers as a direct result of OJJDP-funded research. We summarized these voluminous data by topic to facilitate the presentation. 4. Our report points out that the Enforcing the Underage Drinking Laws Program evaluation documents OJJDP provided to us were not clear on whether the sites chosen were representative. Our report does not suggest “there is no way of achieving a legitimate representative sample.” However, we agree with OJJDP that the evaluation may not be evaluating the Enforcing the Underage Drinking Laws Program because there may be no program components common to all project cities. The Assistant Attorney General states that the evaluation will be able to measure impacts on several program areas across each site. However, our point is that the evaluator’s plan to aggregate data across sites may be inappropriate because wide variation allowed by the program means that program activities are not common across all sites. Therefore, interpreting and generalizing results may be problematic.

Appendix X: GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to the above, Lori A. Weiss, Barbara A. Guffy, Michele J. Tong, Leslie C. Bharadwaja, David P. Alexander, Douglas M. Sloane, Shana B. Wallace, Michele C. Fejfar, Charity J. Goodman, and Jerome T. Sandau made key contributions to this report.

Bibliography

Howell, J.C. Youth Gang Programs and Strategies. U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. Washington, D.C.: Aug. 2000.

Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. “Juvenile Mentoring Program: A Progress Review.” Juvenile Justice Bulletin, Sept. 2000.

Sheppard, D., H. Grant, W. Rowe, and N. Jacobs. “Fighting Juvenile Gun Violence.” Juvenile Justice Bulletin, Sept. 2000.

U.S. Department of Justice, Office of Justice Programs, National Institute of Justice. “Reintegrating Juvenile Offenders Into the Community: OJJDP’s Intensive Community-Based Aftercare Demonstration Program.” Research Preview, Dec. 1998.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. FY 2000 OJJDP Discretionary Program Announcement: Juvenile Mentoring Program. Mar. 2000.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. Gang-Free Schools and Communities Initiative: FY 2000 OJJDP Discretionary Program Announcement. July 2000.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. Juvenile Mentoring Program: 1998 Report to the Congress. Washington, D.C.: Dec. 1998.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile Justice and Delinquency Prevention. OJJDP Research 2000. Washington, D.C.: May 2001.

Wiebush, R. G., B. McNulty, and T. Le. “Implementation of the Intensive Community-Based Aftercare Program.” Juvenile Justice Bulletin, July 2000.