Report to the Honorable Bob Schaffer,

House of Representatives

United States General Accounting Office

GAO

October 2001 JUVENILE JUSTICE

OJJDP Reporting

Requirements for

Discretionary and

Formula Grantees and

Concerns About

Evaluation Studies

GAO-02-23
Page i GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Letter 1

Results in Brief 2

Background 4

Scope and Methodology 7

All Grantees Receive Standard Reporting Guidance and Some Are

Required to Report Specific Data 10

Some Data Exist on the Number of Juveniles Served Directly,

Although Most Grantees Are Not Required to Report This

Information 18

Methodological Concerns Could Adversely Affect Evaluation

Studies 22

OJJDP Has Funded Other Types of Evaluations 26

Conclusions 26

Recommendation for Executive Action 27

Agency Comments and Our Evaluation 27

Appendix I OJJDP Awards Data, Fiscal Years 1996 Through 2000 30

Appendix II OJJDP’s Process for Disseminating Published Products,

Including Interim Results of Impact Evaluations 35

Appendix III Categorical Assistance Progress Report Form 37

Appendix IV Descriptions of OJJDP Programs, Reporting

Requirements, and Examples of Reported Information 39

Appendix V Training and Technical Assistance and Research

Performance Data Reported by OJJDP Grantees 48

Performance Data for Training and Technical Assistance Grantees 48

Performance Data for Research Grantees 51

Contents
Page ii GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Appendix VI States’ Compliance With Core Requirements of the

Juvenile Justice and Delinquency Prevention Act

of 1974 55

Appendix VII Impact Evaluations OJJDP Has Funded of Its Own

Programs 56

Impact Evaluations of OJJDP Programs in Their Formative Stages 56

Impact Evaluations of OJJDP Programs Well Into Their

Implementation 62

Impact Evaluation of OJJDP Program—Summary of Evaluation We

Did Not Assess 68

Appendix VIII Twenty-Four Other Evaluations Funded by OJJDP 69

OJJDP-Funded Programs: 11 Nonimpact Evaluations 69

Non-OJJDP-Funded Programs: Nine Impact Evaluations 74

Non-OJJDP-Funded Programs: Four Nonimpact Evaluations 78

Appendix IX Comments from the Department of Justice 80

GAO Comments 93

Appendix X GAO Contacts and Staff Acknowledgments 95

GAO Contacts 95

Acknowledgments 95

Bibliography 96

Tables

Table 1: Reasons for Reporting Requirements for 16 Selected Major

OJJDP Programs 11

Table 2: OJJDP Programs GAO Identified in Which All Grantees

Reported Data on the Number of Juveniles Directly

Served, Fiscal Year 2000 19

Table 3: OJJDP Funds Awarded by Program Area, Fiscal Years

1996 Through 2000 31

Table 4: OJJDP Award Recipients, Fiscal Years 1996 Through 2000 32
Page iii GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 5: OJJDP Formula/Block Grant Awards, by State, Fiscal

Years 1996 Through 2000 33

Table 6: Dissemination of Published Products With Interim Results

of Impact Evaluations, as of May 2001 36

Table 7: Eight Programs GAO Reviewed With General Reporting

Requirements Only 40

Table 8: Eight Programs GAO Reviewed With Specific Reporting

Requirements and General Reporting Requirements 44

Table 9: Training and Technical Assistance Performance Data

Reported by OJJDP Grantees 50

Table 10: Active OJJDP Grantees’ Research Products, Published by

OJJDP 53

Table 11: Active OJJDP Grantees’ Research Products, Published

Externally 53

Table 12: State-Reported Compliance with Core Requirements,

Calendar Year 1998 55

Figures

Figure 1: OJJDP Award Recipients, Fiscal Year 2000 7

Figure 2: OJJDP Formula/Block Grants Versus Discretionary Grant

Awards, Fiscal Years 1996 Through 2000 30

Abbreviations

CPD Child Protection Division

IPR Individual Project Report

JUMP Juvenile Mentoring Program

NCJRS National Criminal Justice Reference Service

NCMEC National Center for Missing and Exploited Children

OJJDP Office of Juvenile Justice and Delinquency Prevention

OJP Office of Justice Programs

PATHE Positive Action Through Holistic Education

RPDD Research and Program Development Division

STAD State and Tribal Assistance Division

TTAD Training and Technical Assistance Division
Page 1 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

October 30, 2001

The Honorable Bob Schaffer

House of Representatives

Dear Mr. Schaffer:

Although the nation’s rates of violent juvenile crime and youth

victimization have declined dramatically over the past 5 years, critical

problems affecting juveniles and the juvenile justice system still remain,

such as drug dependency, the spread of gangs, and child abuse and

neglect. To address these and other issues, the Office of Juvenile Justice

and Delinquency Prevention (OJJDP) has funded a variety of

demonstration, replication, research and evaluation, and training and

technical assistance programs aimed at preventing and responding to

juvenile delinquency and juvenile victimization. Questions have been

raised, however, regarding what these numerous and diverse programs

have accomplished.

You asked us to review OJJDP’s major programs, as well as the

evaluations it has funded. Specifically, we agreed to provide (1)

information on programmatic reporting requirements for OJJDP grantees,

the reasons for these requirements, and examples of the information

grantees have reported; (2) information on how many juveniles OJJDP

grantees reported directly serving in fiscal year 2000, and whether OJJDP

requires grantees to report the number of juveniles they directly serve, and

if not, why; and (3) analysis of the methodological rigor of the impact

evaluations OJJDP has funded of its own programs since 1995, and

information on the other types of evaluations OJJDP has funded. In

addition, you asked us to provide information on how much OJJDP

awarded to various grant programs from fiscal years 1996 through 2000

and the types of organizations that received these awards, and we have

provided this information in appendix I. You asked us how OJJDP

disseminates published interim results of impact evaluations as well as

other published publications OJJDP and its grantees produce, and we have

provided this information in appendix II. You also asked us to review how

OJJDP monitors its grantees, and we have reported to you on this issue.1

 1 Juvenile Justice: Better Documentation of Discretionary Grant Monitoring Is Needed

(GAO-02-65, Oct. 10, 2001).

United States General Accounting Office

Washington, DC 20548
Page 2 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Our review of 16 of OJJDP’s major programs showed that although

virtually all grantees are required to report on their progress twice a year,

the information they reported varied. Grantees receive standard, general

guidance for reporting on their projects and providing OJJDP information

with which to monitor grantees’ projects and accomplishments. According

to OJJDP officials, such guidance needs to be somewhat general, given the

variation that can occur among individual projects as grantees try to meet

local needs and circumstances. For example, one Drug-Free Communities

Support Program grantee reported that it had helped students produce an

anti-smoking commercial, while another grantee in the same program

reported making presentations on drug abuse to young men at the local

juvenile detention facility. Grantees in 8 of these 16 programs are subject

to additional reporting requirements tailored to each of their programs,

and these specific requirements were established primarily to support

outside evaluations or for other program or project assessment purposes.

For example, OJJDP requires all Juvenile Mentoring Program grantees to

cooperate fully with an outside evaluator. Grantees in this program are

required to report prescribed demographic information for both juveniles

and mentors who participate in their programs, as well as other specified

data.

We identified eight programs in which all grantees reported the number of

juveniles they directly served. Grantees in these eight programs reported

serving about 142,000 juveniles in fiscal year 2000.2

 For example, Court

Appointed Special Advocate Program grantees reported serving over

70,000 juveniles through volunteers who advocate for the best interests of

abused and neglected children who come before the court. Juvenile

Mentoring Program grantees reported serving about 8,500 juveniles by

providing one-on-one mentoring for at-risk youths. OJJDP does not require

grantees in all its programs to report on the number of juveniles they serve

directly for several reasons, including that many of its programs are not

intended to serve juveniles directly. Instead, OJJDP programs have a

variety of other purposes that can indirectly benefit juveniles, such as

helping state and local governments improve their juvenile justice systems,

providing training and technical assistance to juvenile justice

professionals, and establishing a centralized research effort on problems

of juvenile delinquency. In commenting on a draft of this report, the

Assistant Attorney General stated that research projects do not typically

 2

 Grantees in three of the eight programs reported data for time periods that did not

correspond precisely with the fiscal year.

Results in Brief
Page 3 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

provide services directly, yet their results can potentially help thousands

of juveniles.

Our in-depth review of 10 of OJJDP’s impact evaluations3

 undertaken since

1995 of its own programs raises some concerns about whether many of the

evaluations will produce definitive results. All of these evaluations are still

ongoing, with half in their formative stages and half well into

implementation. While program evaluation is an inherently difficult task,

in some of these evaluations, the effort is particularly arduous because of

variations across sites in how the programs are implemented. These

variations will make it difficult to interpret evaluation results. Two of the

evaluations that are in their later stages and three of those that are in their

formative stages lack specific plans for comparison groups, which would

aid in isolating the impacts of the program from the effects of other factors

that may have influenced change. Furthermore, three of the five

evaluations that are well into implementation have developed data

collection problems. We are making a recommendation in relation to the

five evaluations that are in their formative stages, intending to ensure that

potential problems related to comparison groups and data collection are

mitigated.

The Assistant Attorney General provided us with written comments on a

draft of this report in a letter dated October 15, 2001. In general, she

agreed with our findings and recommendation. However, she disagreed

with our focus on the use of comparison groups as the only valid

evaluation design. Although she agreed that the inclusion of comparison

groups would strengthen the interpretation of evaluation results, she said

that sufficient funds are not available to include comparison groups in

every evaluation and that laboratory-like comparison groups may not be

possible when evaluating real life community-based programs. We

recognize that not all evaluation issues that can compromise results are

resolvable, including the use of comparison groups. However, the validity

of the evaluation results can be enhanced through establishing and

tracking comparison groups. If other ways exist to effectively isolate the

impacts of a program, comparison groups may not be needed. However, in

the 10 evaluations for which we assessed methodological rigor, we saw no

evidence of other methods being used. While studies that do not have

appropriate comparison groups can provide useful information, they

 3

 Impact evaluations assess the extent to which a program causes changes in the desired

direction in the target population.
Page 4 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

should not be considered impact evaluations. The full text of the Assistant

Attorney General’s comments and our evaluation of them are presented in

appendix IX and elsewhere in this report, as appropriate.

OJJDP, one of the components of the U.S. Department of Justice, Office of

Justice Programs (OJP), was established by the Juvenile Justice and

Delinquency Prevention Act of 1974 (Juvenile Justice Act).4 Its mission is

to provide national leadership, coordination, and resources to prevent and

respond appropriately to juvenile delinquency and juvenile victimization.

OJJDP accomplishes its mission through developing and implementing

prevention programs and a juvenile justice system that protects the public

safety, holds juvenile offenders accountable, and provides treatment and

rehabilitative services based on the needs of juveniles and their families.

OJJDP funds research and evaluation efforts, statistical studies, and

demonstration programs; provides technical assistance and training;

produces and distributes publications and other products containing

information about juvenile justice topics; oversees activities dealing with

missing and exploited children; and administers a wide variety of grant

programs. OJJDP funds programs that serve juveniles directly as well as

those that benefit juveniles more indirectly by focusing on system-wide

changes or by increasing the capacity of governmental units or

organizations.

OJJDP awards grants to states, territories, localities, and private

organizations through five formula and block grant (formula/block grant)

programs and numerous discretionary grant programs. OJJDP administers

four formula grant programs5

 that provide funds directly to states and

territories on the basis of states’ juvenile populations, and one block grant

program6

 that awards a fixed level of funds to all states and territories.

Under these formula/block grant programs, states may, in turn, make

subawards to other organizations such as units of local government.

OJJDP awards discretionary grants through a competitive process to state

 4

 42 U.S.C. 5601, et seq.

5

 OJJDP’s four formula grant programs are the Formula Grants Program, the Community

Prevention Grants Program, State Challenge Activities, and the Juvenile Accountability

Incentive Block Grants Program.

6

 OJJDP’s one block grant program is the Enforcing the Underage Drinking Laws Program.

The Enforcing the Underage Drinking Laws Program has both a block grant component

and a discretionary grant component.

Background
Page 5 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

governments, local governments, or individual agencies and organizations.

OJP is responsible for the financial monitoring7

 of grantees (i.e., it

provides policy guidance, control, and support services in the financial

management of grants), whereas OJJDP is responsible for program

monitoring. For program monitoring purposes, OJJDP assigns each of its

grantees a program manager who is responsible for ensuring

administrative and programmatic compliance with relevant statutes,

regulations, policies and guidelines of awarded grants. The program

manager is also responsible for monitoring grantees’ performance and

progress as related to grantees’ stated goals and objectives.

OJJDP’s budget has increased significantly over the last 5 years—from

about $188 million in fiscal year 1997 to about $596 million in fiscal year

2001. During this time, the Congress has created new programs and

increased appropriations for some existing ones. The Congress has also

provided direction each year regarding certain program areas OJJDP

should fund. Overall, in fiscal year 2001, 31 percent of OJJDP’s available

funds8

 were congressionally earmarked—that is, set aside for an

identifiable grantee, specified amount, and/or specific authorized purpose.

For fiscal year 2001, $180 million of OJJDP’s funds were available for

discretionary grant awards, of which 77 percent was earmarked.

OJJDP awards the majority of its funds to grantees in its five

formula/block grant programs. In fiscal year 2000, the latest year for which

awards data were available, OJJDP awarded (1) about $354 million, or 64

percent of the total funds awarded, to these formula/block grant programs

and (2) just over $200 million, or 36 percent of the total funds awarded, to

a wide range of discretionary grant programs. The programs awarded the

most funds in fiscal year 2000 were the Juvenile Accountability Incentive

Block Grants Program ($221 million), the Formula Grants Program ($70

million), the Community Prevention Grants Program ($36 million), the

Child Abuse and Neglect Program9

 ($32 million), the Missing and

 7

 OJP conducts financial monitoring of grantees through the Monitoring Division of its

Office of the Comptroller.

8

 Of OJJDP’s $597 million dollar appropriation, only $583 million were “available;” about

$12.7 million were designated to support OJJDP administrative costs and $1.3 million

(.0022) was rescinded.

9

 This program encompasses several other programs within OJJDP’s Child Protection

Division including the Court Appointed Special Advocate Program, Children’s Advocacy

Centers, the Model Courts Program, and the Safe Start Initiative.

OJJDP’s Fiscal Year 2000

Awards
Page 6 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Exploited Children Program10 ($32 million), and the Drug-Free

Communities Support Program ($30 million).

OJJDP awards funds to a wide range of recipients, with the majority of

awarded funds going to state governments. As shown in figure 1, 67

percent of the funds OJJDP awarded in fiscal year 2000 went to states, 20

percent to nonprofit organizations, 6 percent to school districts or

educational institutions, and 5 percent to local governments. However,

because many grantees make subawards to other entities, the awards to

these grantees do not reflect the ultimate recipients of the funds OJJDP

awards. For example, under the Formula Grants Program, states pass

through a minimum of two-thirds of their awarded funds to public and

private nonprofit organizations. (See app. I for data from fiscal years 1996

through 2000 on (1) OJJDP funds awarded to formula/block grant versus

discretionary grant programs, (2) OJJDP awards by program area, (3)

OJJDP award recipients, and (4) OJJDP formula/block grant awards by

state.)

 10 This program encompasses several other programs within OJJDP’s Child Protection

Division including the National Clearinghouse and Resource Center for Missing and

Exploited Children, the Missing and Exploited Children Training and Technical Assistance

program, and the Internet Crimes Against Children Task Force Program.
Page 7 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Figure 1: OJJDP Award Recipients, Fiscal Year 2000

Note: Percentages do not total 100 due to rounding.

a

Local governments include county governments, cities, townships, and special district governments.

b

Schools include independent school districts and state and private institutions of higher education.

c

Other includes for-profit organizations, individuals, Indian Tribes, and other miscellaneous recipients.

Source: GAO analysis of OJP database.

To identify programmatic reporting requirements11 for OJJDP grantees, the

reasons for these requirements, and examples of information grantees

have reported, we reviewed all 5 of OJJDP’s formula/block grant programs

and we selected 11 of its major discretionary grant programs to review

based on OJJDP officials’ input regarding which programs were “major”

(e.g., number of grantees, program funding, and/or importance of the

program). To identify grantee reporting requirements, and the purpose of

these requirements, we met with the OJJDP program managers who

monitor each of these 16 programs or with other key officials. For those

 11 We did not review requirements associated with OJP’s financial monitoring of OJJDP

grantees.

Scope and

Methodology

State

Governments

67%

Nonprofit

Organizations

20%

Other c

Schools

6%

b

Local

Governmentsa

5%

3%
Page 8 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

programs in which OJJDP funded an outside evaluation, we also met with

program managers who oversee the evaluations. We reviewed OJP’s Grant

Management Policies and Procedures Manual (January 19, 2001) and

Categorical Assistance Progress Report (progress report) form12 along with

the instructions for completing the form. (See app. III for a copy of the

progress report form.) We also reviewed OJJDP program documents for

each of the 16 programs, including any special reporting conditions. To

supplement program documents, we reviewed relevant documents from

outside evaluators. We did not assess the adequacy of reporting

requirements established by OJJDP or the outside evaluators.

To identify specific examples of performance data that grantees reported,

we asked OJJDP officials to provide progress reports for each program

demonstrating a range of detail and, in some cases, we asked for reports

from specific grantees. For each program, we then reviewed 3 to 15

progress reports (or individual performance reports) submitted between

1998 and 2001. We did not review progress reports from all grantees in

every program, nor did we review grantees’ compliance with reporting

requirements. For programs being evaluated by an outside evaluator, we

reviewed performance data that program grantees reported to those

evaluators, when available.

To determine whether OJJDP requires grantees to report the number of

juveniles they serve directly and to identify the number of juveniles OJJDP

grantees reported serving in fiscal year 2000, we interviewed OJJDP

program managers for each of the 16 programs we reviewed and examined

relevant program documents, including selected semiannual progress

reports. To determine whether other programs directly serve juveniles,

obtain available data on the numbers of juveniles served, and learn why

OJJDP does or does not require grantees to report such data, we reviewed

OJJDP program literature and met with OJJDP division directors.

Nevertheless, the list of OJJDP programs we identified as directly serving

juveniles may not be comprehensive. We focused our data collection effort

on only those programs we identified in which all grantees reported

juveniles-served data.

 12 For the Formula Grants Program, we reviewed OJJDP’s instructions for completing the

annual Performance Report, including the Individual Project Report forms, because

Formula Grants grantees are required to complete these reports in lieu of progress reports.
Page 9 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

To assess the methodological rigor of the impact evaluations OJJDP has

funded since 1995 of its own programs, and to provide information on the

other types of evaluations OJJDP has funded, we asked OJJDP to identify

all program evaluations it had funded since 1995. For each evaluation, we

asked OJJDP to indicate whether it was an impact evaluation, whether the

program being evaluated was funded by OJJDP, and whether the

evaluation had been completed or was ongoing. Overall, OJJDP identified

35 evaluations funded since 1995.

Eleven of the 35 evaluations were impact evaluations of OJJDP programs,

and all were ongoing.13 For each of the 10 impact evaluations we assessed,

we asked OJJDP to provide any documentation relevant to the design and

implementation of the evaluation methodologies, such as the initial and

supplemental proposals, peer review documents, progress reports, reports

of interim results, and correspondence between OJJDP and the evaluators.

In addition, we contacted OJJDP officials to resolve any questions that we

had regarding the documentation and to request any missing documents.

We did not contact the program manager responsible for each evaluation.

To assess the methodological rigor of the 10 impact evaluations, we used a

data collection instrument to collect information systematically on each

program being evaluated and the features of the evaluation methodology.

We based our data collection and assessments on generally accepted

social science standards. We examined such factors as whether evaluation

data were collected before and after program implementation, how

program effects were isolated (i.e., the use of nonprogram participant

comparison groups or statistical controls), and the appropriateness of

sampling, outcome measures, statistical analyses, and any reported

results. A social scientist with training and experience in evaluation

research and methodology read and coded the documentation for each

evaluation. A second social scientist reviewed each completed data

collection instrument and the relevant documentation for the impact

 13 During the course of our review, OJJDP officials identified 10 of their 35 ongoing

evaluations as impact evaluations of OJJDP-funded programs. In commenting on a draft of

this report, the Assistance Attorney General said that the Teen Courts evaluation has a

comparison group and should have also been classified as an impact evaluation of an

OJJDP-funded program, along with the other 10 evaluations. Because this had not been

included as an evaluation of an OJJDP-funded program when we initially requested the

information, we did not assess the methodological rigor of the Teen Courts evaluation as

we did with the other 10 impact evaluations. On the basis of comments provided by the

Assistance Attorney General, we have reclassified the Teen Courts evaluation as an impact

evaluation of an OJJDP-funded evaluation.
Page 10 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

evaluation to verify the accuracy of every coded item. We relied on

documents OJJDP provided to us in April 2001 in assessing the evaluation

methodologies and reporting on each evaluation’s status.

For each of the remaining 24 evaluations, which included nonimpact

evaluations of OJJDP-funded programs, as well as evaluations of juvenile

justice programs that OJJDP did not fund, we asked OJJDP to provide

general descriptive information, such as the type and purpose of the

evaluation, the number of sites involved, and whether the evaluation

included data on all participants. We did not assess the methodological

rigor of these evaluations.

We conducted our work at OJJDP Headquarters in Washington, D.C., from

September 2000 to August 2001 in accordance with generally accepted

government auditing standards.

OJP requires virtually all OJJDP grantees14 to submit semiannual progress

reports, which OJJDP uses to help monitor grantees’ project

implementation and achievement of the goals they identified in their grant

applications. To this end, OJP provides grantees standard, general

guidance on the types of program information they are to report, such as

narrative information on the status of each of their project goals and the

quantitative results of their projects. In addition to this standard

requirement, grantees for some of OJJDP’s programs are subject to

additional reporting requirements that apply only to their respective

programs. Our review of 16 major programs15 showed that grantees in 8 of

the programs were required to comply only with the standard requirement

for information, and grantees in the other 8 programs were required to

report additional specified data. The specific reporting requirements were

established primarily to help evaluate the results of these programs. Table

1 identifies the 16 programs we reviewed and the reason for the standard

or specific reporting requirements for each program.

 14All discretionary and all formula/block grant program grantees, with the exception of

Formula Grants Program grantees, must submit these reports.

15 We reviewed all 5 of OJJDP’s formula/block grant programs and 11 of its discretionary

grant programs.

All Grantees Receive

Standard Reporting

Guidance and Some

Are Required to

Report Specific Data
Page 11 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 1: Reasons for Reporting Requirements for 16 Selected Major OJJDP Programs

Reason for reporting requirement

Program

Standard

requirement Specific requirement

State Challenge Activities Monitoring None

Community Prevention Grants Program (Title V) Monitoring None

Juvenile Accountability Incentive Block Grants Program Monitoring None

Enforcing the Underage Drinking Laws Program (block grant component)a Monitoring None

Enforcing the Underage Drinking Laws Program (discretionary grant

component)a

Monitoring None

Tribal Youth Program Monitoring None

Model Courts Program Monitoring None

National Clearinghouse and Resource Center for Missing and Exploited

Children

Monitoring None

Formula Grants ProgrambMonitoring Statutory compliance,

OJJDP assessment

Comprehensive Gang Initiative Monitoring Outside evaluation,

grantee self-assessment

Rural Gang Initiative Monitoring Outside evaluation,

grantee self-assessment

Juvenile Mentoring Program (JUMP) Monitoring Outside evaluation

SafeFutures Monitoring Outside evaluation

Drug-Free Communities Support Program Monitoring Outside evaluation

Internet Crimes Against Children Task Force Program Monitoring OJJDP assessment

Children’s Advocacy Centers Monitoring OJJDP assessment,

grantee self-assessment

Note: Five programs in this table are formula or block grant programs: State Challenge Activities, the

Community Prevention Grants Program, the Juvenile Accountability Incentive Block Grants Program,

Enforcing the Underage Drinking Laws Program (block grant component), and the Formula Grants

Program. The remaining 11 programs are discretionary grant programs.

a

The Enforcing the Underage Drinking Laws Program has both a block grant program component and

a discretionary grant program component, and OJJDP administers each from separate organizational

units.

b

Rather than reporting semiannually through progress reports, grantees under this program report on

their subgrantees’ projects annually through Individual Project Reports. The standard requirements

for reporting are similar for both types of reports.

Source: GAO analysis.

Our review of selected progress reports from the 16 programs showed

that, in all but the Formula Grants Program, grantees reported information

on the status of their activities and accomplishments in response to the

standard requirements, although the details they reported varied as did the

projects themselves. Grantees in the eight programs with specific

reporting requirements reported a variety of descriptive information and

performance data to OJJDP and/or outside evaluators. (See app. IV for a
Page 12 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

program description, summary of reporting requirements, and examples of

what grantees reported in each of the 16 programs we reviewed.)16

All OJJDP grantees are required to report on their project activities and

accomplishments to OJP twice a year using the Office of Justice Programs’

Categorical Assistance Progress Report (progress report) form.17 The form

is unstructured and is to be completed in narrative and/or chart form.18 The

standard instructions to grantees for completing the form state that

grantees should report information on the status of each of their projects’

goals scheduled to be achieved during the reporting period and set forth in

their grant application, including quantitative project results based on

performance measures. Grantees are also instructed to report on actions

planned to resolve any implementation problems and request any

technical assistance they might need. OJJDP program managers are to use

reported information to help monitor grantees’ project implementation.

OJJDP officials explained that because the progress report is intended as

just one of their monitoring tools, this standard, general guidance meets

their basic oversight needs. They further explained that guidance needs to

be somewhat general given the variation that can occur among projects as

grantees tailor them to meet local needs and circumstances. OJJDP

encourages grantees to design projects that meet the unique needs of their

own communities, and therefore grantees do not always report on the

same performance measures.

Although OJJDP program managers have additional ways of keeping

abreast of grantees’ projects, such as phone calls and on-site visits, OJJDP

officials indicated they would prefer to require and obtain more specific,

 16 In addition, we aggregated performance data from grantees in OJJDP’s training and

technical assistance programs and research programs because such data are quite

significant in terms of OJJDP’s outputs, and would not be fully captured if we reported only

on the 16 programs we selected for review. Although training and technical assistance

grantees and research grantees were typically subject only to standard reporting

requirements, they reported similar types of quantitative performance data. Thus, we were

able to report comprehensive output data, rather than examples of output data, as we did

for selected grantees in the 16 programs. See appendix V for a summary of performance

data from training and technical assistance grantees and research grantees.

17 As previously noted, the Formula Grants Program is the only exception. In lieu of the

semiannual Categorical Assistance Progress Report, Formula Grants Program grantees are

required to report annually on their subgrantees’ activities and accomplishments using

Individual Project Reports.

18 A copy of the progress report form can be found in appendix III.

Grantees Receiving

Standard Guidance

Reported a Variety of

Information Used for

OJJDP Monitoring
Page 13 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

and even more frequent, information through the progress reports or other

reporting mechanisms. However, according to these officials, they are

reluctant to impose additional reporting requirements on grantees because

of the Paperwork Reduction Act of 1995, which seeks to ensure that

federal agencies balance their need to collect information with the

reporting and paperwork burden they impose. Under the Act, federal

agencies have an obligation to keep the paperwork burden they impose as

low as possible, and agencies must receive prior approval from the Office

of Management and Budget for information collection requests.

We reviewed selected progress reports that grantees from each of the 16

programs submitted to OJJDP and found that, in all but the Formula

Grants Program, grantees reported on the status of their projects.

Grantees reported input, output, or outcome data related to the process,

implementation, and/or accomplishments of their projects. They included

information such as subgrant awards, specific meetings held, staff hired,

implementation difficulties, number of project participants, and behavioral

change in youths. However, the particular information grantees reported

varied, as did their projects. This variation, coupled with the unstructured

format of the progress report, makes it difficult to aggregate reported data.

Fourteen of the programs we reviewed had multiple grantees and the

information these grantees reported in response to the standard guidance

varied, even within each program. For example, we found the following:

• Under the Tribal Youth Program—a program that recognizes differences

among tribes and encourages diversity in their projects—grantees must

implement projects in keeping with at least one of four broad purpose

areas.19 One tribe reported that it had completed the renovation of a youth

center; another reported that it had collected examples of other tribes’

juvenile law enforcement codes20 and started drafting model codes

 19 Funds are available for projects that support the following four broad purposes: (1) to

reduce, control, and prevent crime by and against tribal youth; (2) to provide interventions

for court-involved tribal youth; (3) to improve tribal juvenile justice systems; and (4) to

provide prevention programs focusing on alcohol and drugs.

20 To address the severe juvenile crime problem in Indian communities, many tribal

governments have developed juvenile codes that typically include provisions relating to the

disposition of cases involving Indian youth arrested or detained by the Bureau of Indian

Affairs or tribal law enforcement personnel. Tribal juvenile codes are generally tailored to

Indian youth by incorporating traditionally accepted cultural methods of dealing with

juvenile delinquency issues.
Page 14 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

adapted for each of its villages. A third tribe reported that the resignation

of its community truancy officer had impacted its ability to reduce

instances of misbehavior in school.

• Under another program—the Juvenile Accountability Incentive Block

Grants Program—grantees (states) and their subgrantees (communities)

have undertaken a variety of projects and, thus, report different

information. In this program, states and their communities can choose

from 12 different purpose areas21 under the broad objective of promoting

greater accountability in the juvenile justice system. Thus, one state

reported that one community hired a juvenile court intake officer and

included that officer’s caseload; the same state reported that another

community was unable to start a project because a local agency declined

to participate in its project. Another state reported on the number of

youths enrolled in one community’s drug testing project and reported the

number of drug screening tests performed.

• In the Drug-Free Communities Support Program, grantees design projects

to meet the needs of their local communities; thus, the projects and the

information grantees reported varied. For example, one grantee reported

that it helped local students produce a 30-second anti-smoking commercial

in collaboration with the local health department and further reported that

only 9 of 50 invited members attended a strategic planning meeting it had

held. Another grantee reported making presentations on drug abuse to 146

young men at the local juvenile detention facility, and that its pre- and

post-assessments continually showed that the young men gained

knowledge in the harmful effects of alcohol and drugs.

In the Formula Grants Program, not all grantees reported on the objectives

and accomplishments of their subgrantees’ projects, as required. OJJDP

requires grantees in this program to complete an Individual Project Report

 21 Funding is available to support the following 12 purpose areas: (1) construction of

juvenile detention or correctional facilities, including training of personnel; (2)

accountability-based sanctions programs; (3) hiring of judges, probation officers, and

defenders, and funding pretrial services; (4) hiring of prosecutors; (5) funding of

prosecutor-led drug, gang, and violence programs; (6) providing technology, equipment,

and training programs for prosecutors; (7) probation programs; (8) gun courts; (9) drug

courts; (10) interagency information-sharing systems; (11) accountability-based programs

for law enforcement referrals or those designed to protect students and school personnel

from youth violence; and (12) controlled substance testing (including interventions) for

juveniles in the juvenile justice system.
Page 15 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

(IPR)22 for each subgrantee. The instructions for completing IPRs are

similar to the instructions grantees in other programs receive for

completing semiannual progress reports. Our review of IPRs from selected

states showed that, for one state, none of the IPRs contained any

information on subgrantees’ accomplishments, and some did not include

information on subgrantees’ program objectives.23 For another state,

neither OJJDP nor the state was able to provide us with copies of

completed IPRs because OJJDP’s automated reporting system for states

was inoperable.

Two of the 16 programs we reviewed had only one grantee each. Although

both received only standard reporting guidance, they reported more

detailed, quantitative output and outcome data than grantees in the other

programs that received only standard guidance. In the first program—the

National Clearinghouse and Resource Center for Missing and Exploited

Children—the grantee has voluntarily reported detailed information in a

structured format. In the second program—the Model Courts Program—

OJJDP has emphasized that the grantee should include quantitative

performance data in its progress reports, but did not prescribe the specific

indicators on which the grantee must report.

OJJDJP has designated the National Center for Missing and Exploited

Children (NCMEC) as the grantee for National Clearinghouse and

Resource Center for Missing and Exploited Children, and NCMEC has

developed its own standardized reporting format that covers 10 categories.

This format collects numbers and other information on each category,

such as missing children cases, exploited children cases, public affairs,

and hotline calls. NCMEC reports to OJJDP quarterly, rather than

semiannually, because this timeframe matches the reporting structure of

its data management system. For the first quarter of 2001, NCMEC

reported various output and outcome data that included receiving 24,983

calls through its hotline; assisting in the recovery of 1,610 missing

 22 IPRs are part of annual Performance Reports, which Formula Grants Program grantees

are required to complete.

23 Although assessing the quality of OJJDP’s monitoring efforts was outside the scope of

our review, we inquired about the lack of information in this particular state’s IPRs.

According to the responsible OJJDP official, OJJDP staff addressed this issue during a site

visit made subsequent to the state’s annual submission, and also in a follow-up letter to the

state.
Page 16 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

children;24 receiving 5,291 tips on its online child pornography tip line; and

displaying pictures of 1,399 missing children, which resulted in locating

257 children.

The sole grantee of the Model Courts Program—the National Council of

Juvenile and Family Court Judges25—-also reports quantitative information

in its semiannual progress reports. Although OJJDP has not specified the

performance indicators on which the Council must report, it has

emphasized to the Council the need for quantitative performance data in

the semiannual progress reports. As a result, the Council includes specific

quantitative output data in its progress reports. For example, it reported

that during the last half of 2000 it distributed 17,818 technical assistance

bulletins, conducted 96 training presentations, and made 31 site visits to

model courts. In addition, the Council voluntarily publishes an annual

report that provides more detailed information on the accomplishments of

the individual model courts, such as a reduction in the number of children

in court custody. OJJDP officials told us that they do not require the

Council to provide this report, but they have instructed it to report

detailed performance data on the activities of the model courts, when such

data exist. They further explained that if the Council were to stop

publishing an annual report, OJJDP would require it to include model

court performance data in its progress reports.

In eight of the major programs we reviewed, grantees were given

additional, more explicit reporting instructions requiring them to report on

the same specific performance measures as other grantees in the same

program. In these programs, additional requirements were established to

meet the evaluative needs of OJJDP or an outside evaluator. In one of the

eight programs, requirements were also established to ensure grantee

compliance with certain requirements of the Juvenile Justice Act as well

as for program assessment. The specific requirements of each of these

 24 “Missing children” includes nonfamily and family abductions; runaways and

“thrownaways”; and lost, injured, or otherwise missing children. The number of recovered

children includes deceased children.

25 The Council provides training and technical assistance to 23 “model courts” nationwide.

These designated model courts seek to improve how child abuse and neglect cases are

processed through the court and child protection systems. Although their initiatives vary,

all the model courts address certain core issues, such as how well court processes work in

everyday practice and how well the child protection system meets the needs of children

and their families.

Grantees in Some

Programs Are Required to

Report Specific

Performance Data, Often

for Evaluative Purposes
Page 17 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

eight programs varied, as they were tailored to each program. However,

grantees in all these programs were still required to routinely report

narrative information on the status of their activities and accomplishments

through semi-annual progress reports.

In five of these programs, OJJDP and/or an evaluator have established

specific reporting requirements primarily to support an outside program

evaluation.26 For example, as a condition of receiving a Juvenile Mentoring

Program grant, OJJDP requires all grantees to participate fully in the

evaluation by providing data to the evaluator. This evaluator requires

grantees to report their data through quarterly progress reports that are

similar to semiannual progress reports. The required data include

information on youths participating in each project, participating mentors,

and youth-mentor matches. For instance, grantees are required to report

family structure information for participating youths. The evaluator

aggregates such data from all grantees and has reported, for example, that

56 percent of participating youths lived with their mother only, 20 percent

lived with both parents, 4 percent lived with their father only, and 21

percent were in other living arrangements. 27

In two of the eight programs—the Internet Crimes Against Children Task

Force Program and Children’s Advocacy Centers—specific reporting

requirements were established so that OJJDP could assess program

accomplishments. The governing board of the Internet Crimes Task Force,

in agreement with OJJDP, identified monthly performance measures on

which grantees must report, such as the number of arrests made, search

warrants issued, subpoenas served, and cases opened by the task forces.

Under the Children’s Advocacy Centers program,28 OJJDP prescribed

specific performance measures on which grantees must report, such as the

number of practitioners trained, training conferences held, and

publications distributed. In this program, specific reporting requirements

were established not only for OJJDP to assess the program’s overall

accomplishments, but also to help grantees assess their own projects.

 26 In two of these five programs, a secondary purpose for imposing these requirements was

to allow grantees to assess their own programs.

27 Data were reported in a September 2000 OJJDP Juvenile Justice Bulletin entitled

Juvenile Mentoring Program: A Progress Review.

28 This program is, in part, a training and technical assistance program.
Page 18 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

The eighth program—the Formula Grants Program—has requirements that

are statutorily based and further spelled out by OJJDP in program

regulations. Program reporting requirements were established to ensure

grantees comply with the four core requirements of the Juvenile Justice

Act and as a basis for assessing the effects of the program. These core

requirements are (1) deinstitutionalization of status offenders,29 (2)

separation of juveniles from adult offenders, (3) removal of juveniles from

adult jail and lockup, and (4) addressing efforts to reduce disproportionate

minority confinement. OJJDP regulations list in detail the information on

which states must report.30 For instance, regarding the

deinstitutionalization of status offenders, states must report the total

number of accused and adjudicated status offenders and nonoffenders

placed in facilities that are, for example, not near their home community.

(See app. VI for a summary of states’ compliance with the core

requirements of the Juvenile Justice Act.) According to the compliance

monitoring coordinator for the Formula Grants Program, grantees’ reports

on compliance with the core requirements also provide the basis for

OJJDP to assess the effects of the program.

We identified eight programs that serve juveniles directly and whose

grantees reported such data for fiscal year 2000. However, OJJDP often

does not require grantees to provide this information, in large part because

not all of its programs are intended to provide direct services to juveniles.

 29 A status offender is a juvenile who has been charged with or adjudicated for conduct that

would not be a crime if committed by an adult.

30 28 C.F.R. 31.303(f)(5).

Some Data Exist on

the Number of

Juveniles Served

Directly, Although

Most Grantees Are

Not Required to

Report This

Information
Page 19 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

We identified eight programs in which grantees directly serve juveniles

and in which all grantees report the number of juveniles served to either

OJJDP or an outside evaluator. About 400 grantees in these eight programs

directly served close to 142,000 juveniles in one year. For example, in

fiscal year 2000, the Juvenile Mentoring Program reported serving about

8,500 juveniles, and in calendar year 2000, the Court Appointed Special

Advocate Program reported serving 70,348 youths. Table 2 shows the

programs we identified as directly serving juveniles and reporting such

data for fiscal year 2000.

Table 2: OJJDP Programs GAO Identified in Which All Grantees Reported Data on the Number of Juveniles Directly Served,

Fiscal Year 2000

Program Description

Juveniles

directly served Grantees

Arts and At-Risk Youth A program that combines the arts with job training and

conflict resolution skills and provides summer jobs or paid

internships for youths in the program.

381 3

The Arts Program for Juvenile

Offenders in Detention and Corrections

A program that works with arts professionals to help youths

develop art skills to use after release from detention.

1,026 6

Comprehensive Gang Initiative A demonstration program that utilizes a variety of strategies,

including intensive youth and family intervention, community

mobilization, and organizational change, to reduce youth

gang crime and violence.

78a 2

Court Appointed Special Advocate

Program

A program to help ensure that abused and neglected children

receive timely and effective representation in dependency

hearings. Program volunteers are appointed by judges to

advocate in court for the best interests of abused or

neglected children.

70,348b

207

Juvenile Mentoring Program (JUMP) A one-on-one mentoring program for at-risk youths. 8,509 175

Life Skills Training A training and technical assistance program that instructs

teachers on how to replicate the Life Skills Training program

in a large number of diverse jurisdictions. Life Skills Training

is a tested whole-school-immersion model for reducing drug

use by middle-school students in selected communities.

53,685c 1

SafeFutures A program that works to reduce delinquency and youth

violence through the development of a continuum of services.

Grantees in this program use a single grant to draw on a

variety of OJJDP categorical funds.

5,789d 6

Truancy Reduction Demonstration

Program

A program to reduce the number of truant children by

providing services, such as truancy workshops and truancy

case workers.

2,085 7

Total 141,901 407

Note: The programs listed are those we identified in which all grantees reported juveniles-served

data. We identified these programs by reviewing OJJDP literature and interviewing OJJDP division

directors. Nevertheless, the list may not be comprehensive.

In Some Programs, All

Grantees Report the

Number of Juveniles They

Directly Serve
Page 20 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

a

The level of services provided under this program varied widely by juvenile—i.e., some juveniles

received intensive intervention counseling, while others were only monitored by a case worker.

b

This number represents juveniles served in calendar year 2000.

c

This number is associated with the academic year 1999-2000, which roughly corresponds with fiscal

year 2000. Because this is a program to teach drug prevention skills to all middle-school students in

selected communities, the grantee reports the number of “students reached” rather than the number

of “juveniles served.”

d

This number represents a combination of data provided by SafeFutures grantees and by the

program’s national evaluator. In addition, the level of services provided to these juveniles varied

widely by individual, with some grantees including data on juveniles who received a full array of

project services, and others reporting data on juveniles who were assessed but subsequently denied

services or referred elsewhere. Furthermore, one grantee reported data for a slightly different

period—September 1999 through August 2000.

Source: GAO analysis of OJJDP, evaluator, and grantee data.

We also identified a program in which all subgrantees directly serve

juveniles, but not all subgrantees report such data. The national grantee31

for the Children’s Advocacy Centers program reported that its subgrantees

served over 100,000 juveniles in calendar year 2000, but this number

represents only those juveniles served by subgrantees accredited through

a national membership council.32

For several reasons, OJJDP does not typically require grantees to report

the number of juveniles their projects directly serve. First and foremost,

many of OJJDP’s programs are not intended to serve juveniles directly.

The Juvenile Justice Act established OJJDP for a variety of purposes,

many of which involve indirect benefits to juveniles, rather than direct

services. Statutorily-established purposes for OJJDP include the following:

• To provide technical assistance to and training programs for professionals

who work with delinquents.

• To provide for the evaluations of federally-assisted juvenile justice and

delinquency prevention programs.

• To establish a centralized research effort on problems of delinquency.

 31 The national grantee (one of five program grantees) awards funds to subgrantees, who in

turn provide direct services to juveniles. The other four grantees, which are regional,

provide training and technical assistance to local Children’s Advocacy Centers, as does the

national grantee.

32 Accredited subgrantees received 75 percent of funds available to subgrantees.

Reasons Exist for Not

Requiring Grantees to

Report Number of

Juveniles Served
Page 21 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

• To assist state and local governments in improving the administration of

justice and services for juveniles who enter the system.

Some of OJJDP’s programs, in their entirety, provide indirect benefits,

rather than direct services, to juveniles. OJJDP’s Model Courts Program,

for example, benefits juveniles indirectly by providing training and

technical assistance to court personnel to improve their handling of child

abuse and neglect cases. The Internet Crimes Against Children Task Force

Program also benefits juveniles indirectly by helping to identify and arrest

pedophiles and child pornographers who use the Internet to prey on

children. Furthermore, in commenting on a draft of this report, the

Assistant Attorney General pointed out that although OJJDP’s research

projects do not typically provide services directly, their results can

potentially help thousands of juveniles.

OJJDP officials provided the following additional reasons for not requiring

all grantees to report the number of juveniles their projects directly serve:

• A common interpretation of “juveniles served” does not exist across, or

even within, programs. For example, grantees in one program might

consider the number of juveniles served as those assessed for services but

referred elsewhere, while grantees in a different program might consider

only juveniles who received at least 10 sessions of therapy. Even within

the same program grantees may not share a common definition of

“juveniles served.” One program grantee might report on the number of

juveniles who received intensive one-on-one drug prevention services over

an extended period of time, while another grantee in that same program

might report on the number who attended a one-time presentation on drug

prevention. Without a common interpretation of “juveniles served,” the

data grantees report would be inconsistent and would have little value.

• For some programs, directly serving juveniles may be only one of a

number of intended program purposes and thus, OJJDP does not typically

require all grantees within these programs to report such data. For

example, in the Formula Grants Program, states and their subgrantees can

choose from among 14 different program areas related to preventing and

controlling delinquency and improving juvenile justice systems. Under the

program area of “planning and administration,” for instance, states can

fund planning projects that benefit juveniles indirectly, such as developing

a comprehensive state plan to identify juvenile service needs and

programs that address those needs over the long term. However, under the

area of “illegal drugs and alcohol,” a local subgrantee can serve juveniles

directly by establishing a drug and alcohol abuse prevention project.
Page 22 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

• Juveniles-served data could be used inappropriately to measure the

effectiveness of a program whose primary purpose may not be to provide

direct services to juveniles. For example, the primary purpose of State

Challenge Activities is to stimulate system-wide change, although many of

its 10 activity areas also promote projects intended to directly serve

juveniles. However, grantees in this program are expected to implement

direct service projects within the broader context of promoting systemwide change. For instance, one State Challenge Activities grantee used

funds it received under the “deinstitutionalization of status offenders”

activity area to establish two community projects that provide housing for

runaway juveniles, many of whom are girls. The grantee intends to use its

experiences with these two new projects to initiate system-wide change by

developing a comprehensive model program expressly geared to serving

runaway girls. By focusing on the number of girls served by this program,

one might fail to see that its primary purpose was to develop a

comprehensive model program for serving runaway girls.

OJJDP has funded 35 evaluations since 1995, including 11 evaluations

intended to measure the impact of OJJDP-funded programs. We reviewed

the methodological rigor of 10 of the 11 evaluations.33 Half of these 10

evaluations are in formative stages, while the other five are well into

implementation.34 None had been completed at the time of our review. Our

in-depth review of these 10 evaluations shows that although several are

well-designed and use, or plan to use, sophisticated data analysis methods,

others raise concerns as to whether the evaluations will produce definitive

results. We recognize that impact evaluations, such as the types that

OJJDP are funding, can encounter difficult design and implementation

issues. For some of the evaluations we reviewed, program flexibility has

added to the complexity of designing evaluations. A lack of comparison

groups to aid in isolating the impacts of some programs, and data

collection problems could compromise some evaluation results.

According to OJJDP officials, the OJJDP weighs a number of factors when

deciding which programs to evaluate and what kind of evaluations to fund.

 33 As discussed in the Scope and Methodology section of this report, in commenting on a

draft of this report, the Assistant Attorney General pointed out that the evaluation of the

Teen Courts program is also an impact evaluation of an OJJDP-funded program. This

section addresses only the 10 impact evaluations OJJDP had initially identified.

34 We based our analysis of these 10 ongoing evaluations on documentation OJJDP

provided as of April 2001.

Methodological

Concerns Could

Adversely Affect

Evaluation Studies
Page 23 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Given its budget, it considers how much of its discretionary funds to spend

in support of evaluation activities. In deciding which of its programs to

evaluate, OJJDP gives priority consideration to programs that have been

mandated by the Congress. Other criteria OJJDP uses to determine

whether a program should be evaluated include the program’s level of

funding and its uniqueness, as well as the feasibility and cost of an

evaluation and its potential benefits to the field. Similar criteria are also

involved with decisions to evaluate non-OJJDP funded programs, as well

as congressional interest and other federal agencies’ willingness to co-fund

an evaluation.

The 10 impact evaluations of OJJDP-funded programs that we assessed

vary in size and scope. The cost to conduct these evaluations ranges from

$300,000 to well over $5 million; however, some of these grants involve

both impact and process evaluations and the cost of the impact portion

alone cannot be separated from the total. All 10 evaluations are multi-year,

multi-site projects. The number of evaluation sites ranges from 2 in the

Rural Gang Initiative to 175 in the evaluation of the Juvenile Mentoring

Program. As of April 2001, three evaluations had produced interim findings

of some program impacts.35 (See app. II for information on OJJDP’s

process for disseminating products with interim findings as well as other

products.)

Program evaluation is an inherently difficult task because the objective is

to isolate the impact of a particular program from all other factors that

could have caused a change consistent with the intent of the program, or

mitigated against that change. Given that programs, such as those funded

by OJJDP, operate in an ever-changing environment and involve juveniles

and adults who themselves constantly change, producing definitive

evaluation results can be arduous. For example, the impact of a

hypothetical program intended to improve students’ grades could be

confounded by the effects of an outside-of-school mentoring program, the

transfer of high-performing students to a magnet program, changes in

school faculty, a new scholarship program, a severe flu season that results

in widespread student absences from school, and a myriad of other

factors.

 35 Two other evaluations reported preliminary results of some of the data collected as part

of the impact evaluations; however, these did not comprise impact results.

Evaluations of OJJDP

Programs Are Difficult to

Successfully Design and

Implement
Page 24 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Our in-depth review of the 10 impact evaluations of OJJDP programs

showed that a number of these evaluations are particularly complex

because local grantees design their own projects to fit their communities’

needs. (See app. VII for a description of the impact evaluations OJJDP has

funded of its own programs since 1995). Although this customization may

make sense from a program perspective, it makes it more difficult to

evaluate the program. Instead of assessing a single, homogeneous program

with multiple grantees, the evaluation must assess the effects of multiple

configurations of a program. Although all of the grantees’ projects under

each program being evaluated are intended to achieve the same or similar

goals, an aggregate analysis could mask differences in individual projects’

effectiveness and, thus, not result in information about which

configurations of projects work and which do not.

OJJDP’s evaluation of the Enforcing the Underage Drinking Laws Program

(discretionary grant component) exemplifies this situation. In

implementing their projects, states and local communities have substantial

latitude to employ media campaigns, merchant education, compliance

checks, youth leadership training, or a variety of other activities to deter

underage drinking. Similarly, under the Positive Action Through Holistic

Education program, local educators develop their own ways to prevent

student violence and behavior problems based on their assessments of the

causes of these problems in their schools. Because of the limited number

of sites (two school districts) being evaluated and the likely differences in

how each school has developed its own project, the resulting evaluation

may not provide information that could be generalized to a broader

implementation of the program.

A standard way for evaluators to isolate the impacts of a program from

other potential factors that could have influenced change is to use a

comparison group as a benchmark. In the hypothetical example cited

above concerning a program to improve students’ grades, a second set of

students who are not in the program but are matched in academic

performance and exposed to all of the same factors (except the program)

could provide a baseline from which to assess the impact of the program.

The grades of students in the two groups before and after the program

would provide the data from which to measure program impacts. Without

the benefit of the comparison group as a baseline, it is difficult or

impossible to isolate changes resulting from the program from changes

due to other factors.

The designs of two of the five evaluations that are well into

implementation lack an appropriate comparison group. The evaluation of

Project Variation Within a

Program Can Complicate

Evaluation Design

Lack of Appropriate

Comparison Groups Could

Compromise Some Results
Page 25 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

the Juvenile Mentoring Program—a one-on-one mentoring program for

youths—compares youths entering the program to those completing it.

However, a variety of other factors, including the fact that youths in the

program are likely to mature and, thus, improve somewhat spontaneously,

cannot be ruled out as a rival cause of change from the beginning to the

end of the program. Although the evaluators are employing multiple and

innovative strategies to determine the effectiveness of the program in

achieving its objective, the lack of a comparison group of nonparticipant

youths is an obstacle to identifying definitive outcomes.

An evaluation of the Partnerships to Reduce Juvenile Gun Violence

Program includes a comparison of before and after crime statistics in

project communities with crime statistics for the same time frames for the

cities in which the projects operate. However, citywide crime statistics

would no doubt include data from communities that are similar to the

project community as well as from those that are not. Thus, the

differences between citywide and project community baselines make it

difficult to attribute potential findings to the program.

Of the five programs for which evaluations are still being developed, two

(the Safe Start Initiative and the Rural Gang Initiative) did not seem to

have plans for comparison groups at the time of our review. Another

(Parents Anonymous) anticipates using a comparison group, but as yet

had not developed specific plans for one.

Regardless of the quality of a program evaluation design, data collection

problems can compromise the validity of findings. Data collection

problems may affect the validity of the findings for three of the five

evaluations that are currently completing or have completed data

collection. The Juvenile Mentoring Program evaluation has experienced

problems obtaining behavioral measures and school performance data

with which to gauge program-driven change. The Comprehensive Gang

Initiative evaluation has also experienced data collection problems such as

the lack of fully adequate comparison youth data at all or most sites,

missing police histories, and missing self-reported data.

The Intensive Aftercare evaluation has experienced survey response rate

shortfalls, in some cases obtaining response rates of less than 30 percent,

which may affect the validity of the findings. In commenting on a draft of

this report, the Assistant Attorney General said that the poor response

rates for some elements at different sites were particularly disappointing

because this evaluation had a strong random assignment design; however,

the strategies for obtaining adequate data turned out to be insufficient. She

Data Collection Problems

Could Compromise Some

Results
Page 26 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

added that the program staff who were required to collect data did not

give data collection adequate priority in comparison to their other duties.

This was particularly true of data regarding the comparison groups.

In addition to funding impact evaluations of OJJDP programs, OJJDP has

funded 24 other evaluations since 1995—11 nonimpact evaluations of

OJJDP programs, 9 impact evaluations of programs that were not

supported by OJJDP funds, and 4 nonimpact evaluations of programs that

were not funded by OJJDP. The nonimpact evaluations are not intended to

determine the outcomes of the various programs, but rather how well the

programs have been implemented. For example, OJJDP has funded a

process evaluation of its SafeFutures program to learn more about the

process of community mobilization and collaboration in building a

comprehensive program of prevention and intervention for at-risk youths

and juvenile offenders.

OJJDP has also funded evaluations of programs that are funded by entities

other than OJJDP. For example, although OJJDP does not fund the Act

Now Truancy Program, it has funded a nonimpact evaluation of this

program. The Act Now Truancy Program grew out of a unique Arizona law

that allowed prosecutors to issue citations to parents whose children were

chronically truant. Because there was a great deal of interest in this

approach and OJJDP believed it provided a unique opportunity to learn

about the impact of an unusual approach, it funded an evaluation of the

program. Appendix VIII contains brief descriptions of these 24 other

evaluations.

Although there is great interest in assessing results of programs, it is

extremely difficult to design and execute evaluations that will provide

definitive information. Our in-depth review of 10 OJJDP-funded

evaluations of OJJDP's own programs undertaken since 1995 has shown

that, in some cases, the flexibility that can be beneficial to grantees in

tailoring programs to meet their communities' needs has added to the

complexities of designing impact evaluations that will result in valid

findings. Furthermore, the lack of an appropriate comparison group or

sites and/or problems in data collection may compromise the reliability

and validity of some of these evaluations. Because half of these 10

evaluations are in relatively early stages, any potential problems with

comparison group issues or data collection shortfalls could still be

resolved over the course of the evaluation. We recognize that not all

evaluation issues that can compromise results are resolvable, including

OJJDP Has Funded

Other Types of

Evaluations

Conclusions
Page 27 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

many involving comparison groups and data collection. However, to the

extent that appropriate comparison groups can be established and tracked

and data collection issues can be overcome, the validity of the evaluation

results can be enhanced.

Our review of the recent OJJDP program evaluations has shown that, of

the five that are in or near their final stages, some problems with valid

comparison groups and/or data collection could compromise the

usefulness of some of their results. Five other program evaluations are in a

formative stage where comparison group issues and data collection

strategies are not yet finalized. Accordingly, we recommend that the

Attorney General direct the Administrator of OJJDP to assess the five

impact evaluations in the formative stages to address potential

comparison group and data collection problems and, on the basis of that

assessment, initiate any needed interventions to help ensure that the

evaluations produce definitive results.

We provided a copy of this report to the Attorney General for review and

comment. In an October 15, 2001 letter, the Assistant Attorney General

commented on a draft of this report. Her comments are summarized below

and are presented in their entirety in appendix IX. Her detailed comments

have been addressed in the report as appropriate.

The Assistant Attorney General said that the draft report provides useful

information that highlights areas warranting attention. She added that the

draft report would be an important tool that OJP will use to improve the

quality of its evaluations and to design programs that will achieve greater

impact. Furthermore, OJP will assess the five impact evaluations that are

currently in their formative stages to address potential comparison group

and data collection problems. On the basis of that assessment, OJJDP will

initiate any needed interventions to help ensure that evaluations produce

definitive results.

The Assistant Attorney General said that OJP agrees that it should always

strive for more rigorous and scientifically sound evaluation designs and

that the inclusion of comparison groups would certainly strengthen the

interpretation of evaluation results. However, she disagreed with our

reliance on the use of comparison groups as the only valid evaluation

design for two primary reasons. First, OJJDP seeks to conduct juvenile

justice evaluations in a real-world setting, where laboratory-like

comparison groups may not be possible. Second, sufficient funding is not

Recommendation for

Executive Action

Agency Comments

and Our Evaluation
Page 28 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

available for including comparison groups in every evaluation. The

Assistant Attorney General also said that given the choice between

conducting far fewer evaluations, all with comparison groups, and

conducting a greater number of evaluations under less-than-ideal

conditions, OJJDP’s Research and Program Development Division works

hard to tread a middle ground that satisfies needs for both quality and

quantity. She further pointed out that a growing number of policy makers

and evaluators firmly believe that community-based initiatives do not lend

themselves to the kind of traditional evaluations that this draft report

proposes. Accordingly, some researchers have strongly urged that new

approaches to evaluation be developed.

In addition, the Assistant Attorney General said that our report suggests

that more evaluations using experimental or quasi-experimental evaluation

designs should be funded. She added that many communities reject

participation in programs that are evaluated in this way (i.e., with control

or comparison groups) because they feel that it requires them to purposely

exclude youths from receiving services.

In her comments, the Assistant Attorney General seemed to be using the

terms “comparison group” and “control group” interchangeably. However,

control groups are commonly associated with experiments involving

random assignment. We do not intend our statements regarding the need

for comparison groups in impact evaluations to imply that random

assignment is necessary for studies to be valid. Furthermore, we recognize

that groups can be compared after controlling for differences by methods

other than random assignment, including statistical methods and various

methods of matching. For impact evaluations, comparisons should be

made, and should involve individuals who were not subject to the program

or treatment being evaluated. However, not all the evaluations we

assessed made such comparisons.

We also recognize that not all evaluation issues that can compromise

results are resolvable, even with the use of comparison groups. We also

recognize that designing evaluations with comparison groups can be

expensive and funding limitations could preclude their use in all

evaluations. In addition, obtaining participants can be troublesome, as the

Assistant Attorney General pointed out. However, the validity of

evaluation results can be enhanced through establishing and tracking

comparison groups. If other ways exist to effectively isolate the impacts of

a program, comparison groups may not be needed. However, we saw no

evidence of other methods being used in the 10 impact evaluations we

assessed. While studies that do not have appropriate comparison groups
Page 29 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

can provide useful information, they should not be considered impact

evaluations. Furthermore, we recognize the fact that communities may not

favor withholding treatments or programs from individuals in control or

comparison groups, however, this problem is commonly handled by

phasing in the treatment or program and offering it to comparison group

members following the evaluation period.

As we agreed with your office, unless you publicly announce the contents

of this report earlier, we plan no further distribution of it until 30 days

from the date of this letter. At that time, we will send copies to the Senate

Judiciary Committee, the Senate Subcommittee on Youth Violence, the

House Committee on Education and the Workforce, the House

Subcommittee on Early Childhood, Youth and Families, the Attorney

General, and the Director of the Office of Management and Budget.

If you or your staff have any questions about this report, please contact

James M. Blume or me at (202) 512-8777. Key contributors to this report

are acknowledged in appendix X.

Sincerely yours,

Laurie E. Ekstrand

Director, Justice Issues
Appendix I: OJJDP Awards Data, Fiscal Years

1996 Through 2000

Page 30 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

This appendix provides information on the awards the Office of Juvenile

Justice and Delinquency Prevention (OJJDP) made each year from fiscal

years 1996 through 2000. It contains data on OJJDP funds awarded to

formula/block grant programs versus discretionary grant programs (see

fig. 2), OJJDP funds awarded by more specific program areas (see table 3),

types of OJJDP award recipients (see table 4), and OJJDP formula/block

grant awards by state (see table 5).

We relied on the Office of Justice Program’s (OJP) awards database to

analyze data on all OJJDP-administered awards made during this 5-year

period. We analyzed awards by the year the award was made—not the

year in which the funds were appropriated. We worked with OJJDP

officials to identify awards by major program or program area, as the

database did not provide sufficiently detailed information. OJP officials

advised us that they perform daily quality control checks on all data

entered into their database, however, we did not verify the accuracy of the

database.

Figure 2: OJJDP Formula/Block Grants Versus Discretionary Grant Awards, Fiscal

Years 1996 Through 2000

Note: Dollar amounts are rounded to the nearest million.

Source: GAO analysis of OJP database.

Appendix I: OJJDP Awards Data, Fiscal Years

1996 Through 2000
Appendix I: OJJDP Awards Data, Fiscal Years

1996 Through 2000

Page 31 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 3: OJJDP Funds Awarded by Program Area, Fiscal Years 1996 Through 2000

Program area FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

Formula/block grant programs

Formula Grants Program $68,795,296 $77,469,250 $92,069,600 $74,468,599 $69,699,750

State Challenge Activities 9,352,900 10,363,264 9,726,800 9,011,429 8,227,500

Community Prevention Grants Program

(Title V)

19,762,000 18,368,000 18,707,000 39,921,000 36,158,000

Enforcing the Underage Drinking Laws

Program (block grant component)

a a 18,360,000 17,640,000 19,080,000

Juvenile Accountability Incentive Block

Grants Program

a a

232,250,000 232,250,000 221,094,775

Discretionary grant programs

National Institute for Juvenile Justice and Delinquency Preventionb

Comprehensive Strategy for Serious,

Violent, and Chronic Juvenile Offenders 900,730 900,055 1,835,597 1,842,240 2,640,000

Causes and Correlates of Delinquency

Research

299,999 900,000 599,999 600,000 750,000

Statistics and Systems Development

Program 349,978 249,996 449,998 549,923 649,996

Other research, statistics, evaluation,

information dissemination, and training and

technical assistance grants 12,382,088 15,689,329 21,566,738 40,816,294 42,783,923

Demonstration and replication programs

Comprehensive Gang Initiative 2,349,597 2,848,881 3,224,480 2,249,742 1,149,498

Rural Gang Initiative a a 215,000 1,112,754 1,474,473

SafeFutures 2,159,950 5,540,000 4,715,979 9,462,844 10,829,102

Community Assessment Centers a1,074,552 1,699,992 125,000 875,000

Juvenile Mentoring Program (JUMP) a9,688,488 3,363,281 21,059,220 5,760,000

Partnerships to Reduce Juvenile Gun

Violence Program

a

551,060 1,050,230 1,299,975 799,996

Enforcing the Underage Drinking Laws

Program (discretionary grant component)

a a

6,608,019 2,344,986 6,272,003

Drug-Free Communities Support Program a a 8,903,433 19,039,326 29,677,581

Truancy Reduction Demonstration Program aaa 693,849 711,328

Tribal Youth Program aaaa 8,252,019

Other demonstration and replication grants 7,070,330 8,846,421 17,178,085 26,309,408 24,058,149

Missing and Exploited Children Program

National Clearinghouse and Resource

Center for Missing and Exploited Children 3,294,648 4,576,640 7,585,000 8,925,000 19,490,263

Missing and Exploited Children Training &

Technical Assistance program

a

3,448,575 1,749,803 1,848,947 1,949,741

Internet Crimes Against Children Task Force

Program

a a

2,351,829 199,997 8,486,056

Other missing and exploited children grants 1,189,639 1,488,935 1,759,887 978,356 2,074,318

Child Abuse and Neglect Program

Court Appointed Special Advocate Program 6,040,000 5,904,400 7,062,100 9,034,200 9,796,000

Children’s Advocacy Centers 3,199,856 3,099,914 5,691,000 4,733,000 5,824,302
Appendix I: OJJDP Awards Data, Fiscal Years

1996 Through 2000

Page 32 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program area FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

Model Courts Program 742,500 984,100 1,964,900 2,108,097 2,109,200

Safe Start Initiative aaaa 8,038,573

Other child abuse and neglect grants 1,725,000 3,199,878 4,838,152 6,508,938 6,499,976

Total $139,614,511 $175,191,738 $475,526,902 $535,133,124 $555,211,522

Note: Award amounts are not adjusted for de-obligated funds, except in cases in which an entire

award was de-obligated and later re-awarded, as occurred in rare instances of formula/bock grant

awards.

a

Either the program did not receive an appropriation or OJJDP did not award funds.

b

Awards listed under this subheading comprise a variety of research, statistics, evaluation,

information dissemination, and training and technical grants. However, the funds awarded under this

subheading do not represent a comprehensive accounting of all such activities funded by OJJDP. In

some cases these activities are accounted for under other subheadings. For example, the Missing

and Exploited Children Training and Technical Assistance award, which is primarily a training and

technical assistance grant, is listed under the Missing and Exploited Children’s Program. Also,

evaluations for many of the demonstration and replication programs are accounted for under their

respective programs, rather than under the National Institute for Juvenile Justice and Delinquency

Prevention. In commenting on a draft of this report, the Assistant Attorney General pointed out that

combining training and technical assistance funding under the National Institute of Justice and

Delinquency Prevention does not highlight the importance of training and technical assistance.

Although we recognize the importance of training and technical assistance, OJP’s awards database

did not readily lend itself to identifying all training and technical assistance—either awards or portions

of awards—because they are part of many programs across OJJDP divisions.

Source: GAO analysis of OJP database.

Table 4: OJJDP Award Recipients, Fiscal Years 1996 Through 2000

FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

Recipients

Amount

awarded %a

Amount

awarded %a

Amount

awarded %

Amount

awarded %a

Amount

Awarded %a

State governments $98,854,634 71 $109,350,562 62 $377,103,675 79 $377,819,401 71 $369,196,062 67

Local governmentsb2,584,003 2 7,230,243 4 13,534,664 3 18,766,350 4 27,078,615 5

Schoolsc4,936,373 4 14,425,127 8 18,292,717 4 20,926,891 4 34,975,373 6

Nonprofit organizations 32,289,708 23 41,026,686 23 60,476,935 13 106,044,793 20 109,431,488 20

Otherd949,793 1 3,159,120 2 6,118,911 1 11,575,689 2 14,529,984 3

Total $139,614,511 100 $175,191,738 100 $475,526,902 100 $535,133,124 100 $555,211,522 100

a

Percentages may not total 100 due to rounding.

b

Local governments include county governments, cities, townships, and special district governments.

c

Schools include independent school districts and state and private institutions of higher education.

d

Other includes for-profit organizations, individuals, Indian tribes, and other miscellaneous categories.

Source: GAO analysis of OJP database.
Appendix I: OJJDP Awards Data, Fiscal Years

1996 Through 2000

Page 33 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 5: OJJDP Formula/Block Grant Awards, by State, Fiscal Years 1996 Through 2000

Funds awarded

State FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

Alabama $1,424,000 $1,648,900 $5,896,400 $5,855,300 $5,638,100

Alaska 787,500 832,500 2,822,300 2,839,800 2,594,200

Arizona 1,502,000 1,821,000 6,197,500 6,714,900 6,327,800

Arkansas 869,000 1,014,500 4,222,700 4,432,700 4,222,100

California 11,440,000 13,417,100 37,575,000 37,547,300 36,119,800

Colorado 1,279,000 1,497,000 5,579,400 5,649,200 5,474,219

Connecticut 1,017,000 1,197,000 4,744,200 4,824,300 4,612,400

Delaware 787,500 829,500 2,133,100 3,469,600 2,726,200

District of Columbia 788,546 187,500 1,785,400 1,778,000 1,699,100

Florida 4,302,000 5,144,000 15,440,600 15,631,200 15,089,600

Georgia 2,467,000 2,907,000 9,432,800 9,534,600 9,188,600

Hawaii 787,500 860,500 2,984,800 3,402,400 3,096,800

Idaho 787,500 869,500 3,280,000 3,388,200 3,264,700

Illinois 3,605,000 1,626,064 17,824,400 14,320,800 12,886,850

Indiana 1,942,000 2,269,000 7,615,300 7,608,129 7,304,900

Iowa 961,000 1,106,000 4,446,700 4,641,700 4,438,300

Kansas 909,000 1,057,000 4,316,400 4,517,200 4,360,300

Kentucky 278,000 0 4,117,800 5,117,100 6,659,600

Louisiana 1,553,900 1,816,100 6,518,200 6,317,100 6,010,400

Maine 787,500 859,500 2,965,650 3,027,850 3,076,500

Maryland 1,665,000 1,941,000 6,751,400 6,678,100 6,418,700

Massachusetts 1,537,500 893,000 7,789,450 7,375,900 7,035,600

Michigan 5,572,000 3,808,000 11,800,200 11,613,500 11,207,319

Minnesota 1,636,000 1,899,000 6,592,150 6,604,300 6,312,800

Mississippi 997,000 857,000 4,902,400 4,736,500 4,327,600

Missouri 1,797,000 2,088,000 7,169,800 7,188,300 6,835,700

Montana 787,500 843,500 2,957,900 2,619,400 3,223,100

Nebraska 664,500 910,500 3,558,900 3,700,700 3,563,000

Nevada 795,500 888,500 3,481,600 3,695,300 3,630,700

New Hampshire 637,500 1,006,500 3,133,550 3,192,800 3,089,400

New Jersey 2,546,000 2,996,000 7,096,000 7,676,900 9,528,200

New Mexico 830,500 939,500 3,739,300 3,884,500 3,748,900

New York 5,815,000 6,798,000 19,858,900 19,779,400 18,544,600

North Carolina 2,260,000 2,693,000 8,924,300 9,016,600 8,746,900

North Dakota 787,500 827,500 2,779,400 2,766,400 2,682,900

Ohio 3,763,000 4,363,000 13,101,700 13,060,100 9,405,700

Oklahoma 1,160,000 1,340,000 5,102,900 5,170,600 4,946,500

Oregon 1,027,000 1,217,000 4,809,400 4,934,700 3,769,500

Pennsylvania 3,821,000 4,439,000 13,292,600 13,163,000 12,471,600

Rhode Island 787,500 843,500 2,965,000 2,992,300 2,905,700
Appendix I: OJJDP Awards Data, Fiscal Years

1996 Through 2000

Page 34 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Funds awarded

State FY 1996 FY 1997 FY 1998 FY 1999 FY 2000

South Carolina 1,240,000 1,427,000 5,319,300 5,166,300 4,938,550

South Dakota 787,500 100,000 2,013,500 1,993,800 1,921,300

Tennessee 1,710,000 1,999,000 6,897,100 6,905,800 5,332,000

Texas 6,910,000 8,165,000 23,614,200 24,006,200 22,961,500

Utah 885,000 1,030,000 4,281,400 4,518,500 4,372,000

Vermont 787,500 822,500 2,717,300 2,711,200 2,626,619

Virginia 2,113,000 2,460,000 8,155,800 8,205,799 7,794,800

Washington 1,856,000 2,164,000 7,362,500 7,433,800 6,784,350

West Virginia 810,500 899,500 3,497,100 3,590,800 3,432,100

Wisconsin 1,755,000 2,045,000 6,538,150 7,368,750 6,621,600

Wyoming 0 0 1,933,950 1,836,800 1,759,100

American Samoa 148,000 133,000 617,391 601,962 472,248

Guam 115,000 192,000 842,350 841,700 807,800

Northern Mariana Islands 148,000 151,000 381,959 381,738 370,552

Puerto Rico 1,367,750 1,917,000 5,856,900 5,854,100 5,548,800

Republic of Pulau 15,000 21,350 4,300 0 0

Virgin Islands 100,000 123,000 1,374,700 1,407,100 1,331,818

Total $97,910,196 $106,200,514 $371,113,400 $373,291,028 $354,260,025

Source: GAO analysis of OJP database.
Appendix II: OJJDP’s Process for

Disseminating Published Products, Including

Interim Results of Impact Evaluations

Page 35 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

The Office of Juvenile Justice and Delinquency Prevention (OJJDP) has a

process for disseminating published interim results of impact evaluations

as well as other publications produced by OJJDP and its grantees. OJJDP

publications are available through the Juvenile Justice Clearinghouse.1

According to an OJJDP official, OJJDP develops a specific strategy for

each publication that includes the number of copies to be printed, the

methods for announcing availability, and the target audience that will

automatically receive copies. OJJDP promotes products through the

National Criminal Justice Reference Service (NCJRS) Catalog, OJJDP’s

Juvenile Justice journal, the NCJRS and OJJDP Web sites, e-mail lists, the

Office of Justice Programs press announcements, conference displays,

criminal/juvenile justice newsletters and journals, and flier mailings.

Almost all of OJJDP’s publications are made available to the public

through OJJDP’s Web sites, which is administered by the Juvenile Justice

Clearinghouse. Many publications, depending on their length, are also

available through the Clearinghouse’s fax-on-demand service. Individuals

can also order copies of publications online or by calling the

Clearinghouse’s toll-free number. In addition, the Clearinghouse

automatically sends publications to targeted constituents (e.g., juvenile

justice policymakers, practitioners, researchers, and community-based

organizations) and to individuals who have registered to receive

publications based on their specific areas of interest.

As of May 2001, OJJDP had used this dissemination process to share

interim results from 5 of the 10 ongoing impact evaluations2

 of OJJDP

programs that we assessed.3

 In total, OJJDP had distributed over 400,000

copies of 9 products that contained interim results from the 5 evaluations.

Table 6 provides additional information on the distribution of these

publications.

 1

 OJJDP established the Juvenile Justice Clearinghouse to provide individuals and

organizations easy access to information and resources on juvenile justice topics. The

Clearinghouse is a component of NCJRS, a federally-sponsored information clearinghouse

that offers a range of services including online newsletters, access to publications by

partner agencies, and a calendar of upcoming conferences. Its federal sponsors include the

Office of Justice Programs (including OJJDP) and the Office of National Drug Control

Policy. NCJRS maintains a Web site where publications by OJJDP and other federal

sponsors can be accessed.

2

 Although these interim results were based on data collected from impact evaluations, they

did not necessarily comprise impact results.

3

 These 10 impact evaluations were ongoing as of April 2001.

Appendix II: OJJDP’s Process for

Disseminating Published Products, Including

Interim Results of Impact Evaluations
Appendix II: OJJDP’s Process for

Disseminating Published Products, Including

Interim Results of Impact Evaluations

Page 36 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 6: Dissemination of Published Products With Interim Results of Impact Evaluations, as of May 2001

Program Producta

Automatic

mailingb

Other

distributionc

Other

availability

Juvenile Mentoring Program

(JUMP)

Juvenile Mentoring Program: 1998 Report to the

Congress (Report, December 1998).

41,000 7,100 Online

Juvenile Mentoring Program: A Progress Review

(Bulletin, September 2000).

44,000 7,000 Online

Fax-on-demand

FY 2000 OJJDP Discretionary Program

Announcement: Juvenile Mentoring Program

(Solicitation, March 2000).

63,700 9,000 Online

Fax-on-demand

Comprehensive Gang

Initiative

Gang-Free Schools and Communities Initiative: FY

2000 OJJDP Discretionary Program Announcement

(Solicitation, July 2000).

48,700 6,700 Online

Fax-on-demand

Youth Gang Program and Strategies (Summary,

August 2000).

45,000 2,500 Online

Intensive Aftercare Implementation of the Intensive Community-Based

Aftercare Program (Bulletin, July 2000).

32,100 2,500 Online

Fax-on-demand

Reintegrating Juvenile Offenders Into the

Community: OJJDP’s Intensive Community-Based

Aftercare Demonstration Program (National Institute

of Justice Research Preview, December 1998).

Not

available

664 Online

Fax-on-demand

Partnerships to Reduce

Juvenile Gun Violence

Program

Fighting Juvenile Gun Violence (Bulletin, September

2000).

44,000 2,300 Online

Fax-on-demand

Enforcing the Underage

Drinking Laws Program

OJJDP Research 2000 (Report, May 2001). 46,000 1,366 Online

Total 364,500 39,130

a

Full citation information for these products is contained in the bibliography.

b

“Automatic mailing” refers to publications disseminated to targeted and registered individuals and

groups. The Juvenile Justice Clearinghouse maintains 290 targeted mailing lists of key constituents

that include criminal/juvenile justice policymakers, practitioners, researchers, and community-based

organizations. In addition, individuals can register with the Clearinghouse to automatically receive

information based on their specific areas of interest.

c

“Other distribution” refers to orders for specific publications. Anyone can order a publication by calling

the toll-free line or using the NCJRS or OJJDP Web site.

Source: OJJDP.
Appendix III: Categorical Assistance Progress Report Form

Page 37 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Appendix III: Categorical Assistance Progress

Report Form
Appendix III: Categorical Assistance Progress Report Form

Page 38 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 39 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Twice a year, the Office of Juvenile Justice and Delinquency Prevention

(OJJDP) grantees are required to complete a Categorical Assistance

Progress Report—a narrative report that is to include a summary of the

status of their particular projects’ goals, quantitative project results based

on performance measures set forth in their grant applications, actions

planned to resolve any implementation problems, and any technical

assistance they might need. In 8 of the 16 major programs we reviewed,

grantees received only this general guidance, and were not subject to any

additional reporting requirements. In the other eight programs we

reviewed, grantees were required to follow this standard guidance and, in

addition, report more specific information. Grantees in all 16 programs

reported input, output, and/or outcome data related to the process,

implementation, and/or accomplishments of their projects, such as

acquisition of additional funding for a project evaluation, the number of

project participants, or the number of missing children recovered.

Table 7 provides summary information on the eight programs in which

grantees are not subject to additional reporting requirements and

examples from grantees’ progress reports. Table 8 provides similar

information regarding the other eight programs in which grantees are

additionally required to report specified data to OJJDP or outside

evaluators, as well as the specific performance measures on which

grantees are required to report. Unless otherwise noted, the examples of

reported information represent individual grantee or subgrantee data1

 for a

6-month period. Information provided regarding the specific data on which

grantees are required to report do not necessarily include all performance

data required.

 1

 Examples from OJJDP’s five formula/block grant programs were typically reported at the

subgrantee or community level.

Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 40 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 7: Eight Programs GAO Reviewed With General Reporting Requirements Only

Program

name

Active

grantees

(FY2000) Program description

Examples of information reported by grantee(s)

per general requirements

State Challenge

Activities

52 This program, established in 1992 under

Title II, Part E of the Juvenile Justice and

Delinquency Prevention Act of 1974, as

amended, provides incentive grants to

states to improve their juvenile justice

system. States must use funds to develop,

adopt, or improve policies in 10 specified

challenge activity areas, including (1) basic

system services, (2) access to counsel, (3)

community-based alternatives, (4) violent

juvenile offender facilities, (5) genderspecific policies and programs, (7) state

ombudsman, (8) deinstitutionalization of

status offenders, (9) alternatives to school

suspension and expulsion, and (10)

aftercare services/state agency

coordination/case review system. To receive

these grants, states must also participate in

the Title II, Part B Formula Grants Program.

States may subaward State Challenge

Activities funds to communities.

• Implemented an aftercare program for state wards

returning to their community that provided

community reintegration services including

counseling, employment readiness services, family

mediation, and weekly check-in and monitoring

programs.

• Hired a disproportionate minority confinement

coordinator.

• Home visitation research contract delayed due to

paperwork complications.

• Held weekly meetings with female juveniles

identified by the court to receive counseling and

support through a gender-specific services

program.

• Selected and made subaward to private provider

with experience in developing intake services for

youths cited by law enforcement with a focus on

preventing unnecessary placement in attendant

care or detention.

• Provided quarterly training to all case management

staff on more rapidly and successfully returning

youths in state custody from out-of-home

placement into community and in-home placement.

Community

Prevention Grants

Program

(Title V)

54 This program funds collaborative,

community-based delinquency prevention

efforts. The program integrates six

fundamental principles—comprehensive

and multidisciplinary approaches, research

foundation for planning, community control

and decision making, leveraging of

resources and systems, evaluation to

monitor program effectiveness, and a longterm perspective—that combine to form a

strategic approach to reducing juvenile

delinquency.

• Exceeded goal of providing career exploration and

training on work awareness skills to 150 high

school students.

• Trained 11 students to become mentors.

• Provided substance abuse education to six middle

school girls.

• Experienced difficulties starting a delinquency

prevention program to develop self-esteem and

reduce inappropriate school behavior in at-risk

seventh-grade students.

• Hired two outreach workers to work with elementary

and junior high schools.

• Created an instructional video on quality daycare

issues and distributed it to parents and the

community.

• Enrolled and “graduated” 13 children in a 6-week

kindergarten preparation class.

Juvenile

Accountability

Incentive Block

Grants Program

54 The Congress established this formula grant

program in 1997 to promote greater

accountability among juveniles who are

involved in the juveniles justice system. To

qualify for funds, states must demonstrate a

commitment to increasing such

accountability. Funds are allocated to states

based on their relative population of youths

under age 18, and states must pass on at

• Hired a case manager to monitor attendance and

grades of juveniles in a youthful offender program.

• A local office declined to participate in the

mentoring program and the subgrantee is

considering alternative participants.

• Completed a study of achieving a system of

graduated sanctions within the state.

• Enrolled 30 youths in a juvenile drug testing

program and performed 52 drug screens.
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 41 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program

name

Active

grantees

(FY2000) Program description

Examples of information reported by grantee(s)

per general requirements

least 75 percent of their funds to local

governments, absent a waiver. Funds can

be used for 12 program purpose areas.

• Expanded a drug abuse program for juveniles

referred to the court to include a second group

involving both parents and juveniles. To date, the

project has held 3 groups of 6-week classes.

• No juveniles completed a 6-week intensive

educational program for substance abuse during

the reporting period because of problems starting

the program.

• The placement of a full-time police officer at a high

school was delayed due to problems involving the

officer’s training.

• Contracted with a part-time prosecutor to assist fulltime prosecutor with juvenile pre-trial and trial

services, including compiling case information,

preparing witnesses, and tracking cases. The parttime prosecutor works an average of 25 cases per

week. In total, about 234 cases have been handled.

• Experienced difficulties in recruiting a data analyst

due to problems with applicants meeting the

minimum requirements.

Enforcing the

Underage

Drinking Laws

Program (block

grant component)

51 The block grant component of the Enforcing

the Underage Drinking Laws Program helps

states and the District of Columbia develop

comprehensive and coordinated efforts to

enforce state laws that prohibit the sale of

alcoholic beverages to minors and to

prevent the purchase or consumption of

alcoholic beverages to those under age 21.

Grantees may use funds to support

activities in one or more of three priority

areas, including (1) law enforcement and

prosecution efforts that target

establishments suspected of selling alcohol

to minors; (2) public advertising campaigns

to educate businesses and youths about the

prohibitions against illegal alcohol sales and

purchases; and (3) other innovative

programs to prevent and combat underage

drinking.

• Formed 15 youth task groups in order to examine

the community norms and messages that young

people get regarding alcohol.

• Placed billboards to educate the public about

alcohol use in 20 communities for 4 weeks.

• Produced a 60-second radio spot aimed at

preventing underage alcohol use.

• Trained 20 community teams about prevention

approaches to reduce underage drinking.

• Established a support position to improve

coordination and communication between

enforcement, prevention, and treatment systems.

• Held a full-day conference for established task

force groups.

• Experienced implementation problems due to

problems with students’ schedules and limitations

of the school year schedule.

• The local public television station worked with a

youth task force group to design and film a 1-hour

documentary about underage drinking.

Enforcing the

Underage

Drinking Laws

Program

(discretionary

grant component)a

23 The discretionary grant component of the

Enforcing the Underage Drinking Laws

Program provides grants to states and

territories to support the block grant

component of the same program. Its

purpose is to expand the number of

communities taking a comprehensive

approach to the problem of underage

drinking, with an emphasis on increasing

law enforcement activity regarding the sale

of alcohol to minors. States that receive

• Three communities developed a compliance check

database form to be used by law enforcement

agencies.

• Requested technical assistance for a 1-day training

session for local coalitions on environmentallybased activities.

• Published two reports on status of underage

drinking in targeted community.

• Held two advisory committee meetings.

• Implemented new diversion program for underage

alcohol violators and achieved a 32-percent

success rate for youths completing the program.
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 42 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program

name

Active

grantees

(FY2000) Program description

Examples of information reported by grantee(s)

per general requirements

these competitive awards select between

two and eight diverse jurisdictions as

subgrantees.

• Experienced difficulty in bringing one subgrantee on

board; the subgrantee has not yet attended any

planning sessions.

• Four communities hired part-time program

coordinators.

• Conducted six monthly conference calls with four

communities.

• Youth coalition placed 4,000 warning stickers on

alcoholic beverages in 13 stores.

• Youth coalition designed placemats containing

underage drinking facts and figures for restaurant

and grocery store distribution.

• Published two newsletters.

• Was not able to hold banquet recognizing top 100

businesses that had discouraged underage alcohol

sales due to preparation demands of an antialcohol youth conference.

• Arranged for more than 700 youths to sign a pledge

not to drink.

Tribal Youth

Program

81 Under this discretionary award program,

federally recognized Tribes can apply for

funds to (1) reduce, control, and prevent

crime both by and against tribal youths; (2)

provide interventions for court-involved tribal

youths; (3) improve tribal juvenile justice

systems; and (4) provide prevention

programs focusing on alcohol and drugs.

The program is intended to provide a

flexible funding stream for tribes.

• Began keeping juvenile crime data for future

analysis of whether targeted reductions have been

achieved.

• Drafted court diversion protocol.

• Identified tribal judges and court clerks.

• Experienced set-backs in establishing four

functional tribal courts to deal with juvenile crime for

reasons including the resignation of the general

counsel and competing demands of the commercial

and subsistence fishing season.

• Reported conferences and training events

attended.

• Met with state representatives about possible

transfers of cases from state to tribal court.

• Began identifying courtroom equipment needs and

researching purchases.

• Obtained examples of tribal juvenile codes.

• Taught conflict resolution skills to families upon

request.

• Suffered a set-back in reducing cases of

misbehavior and rules violations in the schools

because the community truancy officer resigned

and had not yet been replaced.

• Referred five youths for counseling as a result of

conflict resolution curriculum presented in schools.

Model Courts

Programb

1 This program awards one earmarkedc grant

annually to the National Council of Juvenile

and Family Court Judges to provide training

and technical assistance to 23 “model

courts” around the nation that process child

abuse and neglect cases. The program’s

goal is to design and deliver technical

Information reported by grantee through progress

reports:

• Made 31 on-site staff visits to model courts.

• Held 96 national, regional, state, and local training

events for nonmodel court jurisdictions.

• Supported the establishment of two model courts.

• Trained 9,729 model court and nonmodel court
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 43 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program

name

Active

grantees

(FY2000) Program description

Examples of information reported by grantee(s)

per general requirements

assistance and training programs that

improve dependency courts’ handling of

such cases. Model courts serve as national

“laboratories” for implementing systemswide change in the processing of abuse and

neglect cases. Reforms and initiatives vary

across courts. The Council also supports a

mentoring program in which experienced

model court personnel “mentor” and train

personnel from new model courts.

personnel.

• Distributed 17,818 technical assistance bulletins.

• Conducted 134 telephone consultations.

• Distributed 2,341 resource packets.

Information reported by grantee through annual status

report:d

• Reported one court reduced its caseload of 58,000

abused and neglected children in 1995 (prior to

program implementation) to 27,001 children in

1999.

• Reported one court reduced the average length of

abuse and neglect complaint cases from 3.5 years

in court year 1998 to 2.4 years in court year 1999.

• Reported one court established a child protection

division with two judges who hear all child

protection cases.

• Following this change, the average case length

(from petition to adjudication) for one section of the

division dropped from 140 days to between 29 and

41 days.

National

Clearinghouse

and Resource

Center for Missing

and Exploited

Childrene

1 The National Center for Missing and

Exploited Children (NCMEC) is the national

resource center and clearinghouse

dedicated to missing and exploited

children’s issues. By law, NCMEC is to (1)

establish a toll-free hotline, (2) provide

information on free or low-cost

transportation for missing children and their

families, (3) coordinate programs that

reunite missing children with their families,

(4) disseminate information that benefits

missing and exploited children, and (5)

provide training and technical assistance to

law enforcement and other agencies.

• Upgraded toll-free hotline.

• Received 24,983 hotline calls.

• Assisted in recovering 1,610 missing children.

• Described missing children cases in which NCMEC

assisted.

• Distributed 5,306 pictures of missing children.

• Described public awareness campaigns.

• Trained 1,708 law enforcement and health care

personnel.

• Described joint projects with other agencies.

• Completed 48 age progressions of photographs.

• Received 5,291 child pornography tips online, and

492 tips through the hotline.

• Described technical assistance provided to other

agencies.

a

This program was established in fiscal year 1998 under the name Combating Underage Drinking.

b

OJJDP has emphasized to the single grantee for this program that it should include quantitative

performance data in its progress reports, but did not prescribe the specific indicators on which the

grantee must report.

c Earmarked grants are those for which the Congress sets aside a specified amount for an identifiable

grantee and/or a specific authorized purpose.

d

The grantee chooses to provide performance data regarding model court activities and

accomplishments in an annual status report that is available to nonmodel courts and to the general

public.

e

Although the grantee in this program is only required to submit progress reports semiannually, it

chooses to submit progress reports on a quarterly basis. Examples presented in this table were

excerpted from a recent quarterly progress report.

Source: GAO analysis of OJJDP and grantee data.
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 44 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 8: Eight Programs GAO Reviewed With Specific Reporting Requirements and General Reporting Requirements

Program

name

Active

grantees

(FY2000) Program description

Specific data required from

and reported by program

grantee(s)

Examples of information reported

by grantee(s) per general

requirements

Formula Grants

Program

54 OJJDP has administered this

program since 1975. The

program provides funds to

help states, U.S. territories,

and the District of Columbia

implement a comprehensive

state juvenile justice plan

based on a detailed study of

needs to support state and

local delinquency prevention

and intervention efforts and

juvenile justice system

improvements. To be eligible

for a formula grant, a state

must designate an agency to

prepare a 3-year plan,

establish a State Advisory

Group, and commit to

complying with the four core

requirements of the Juvenile

Justice and Delinquency

Prevention Act of 1974, as

amended.

Grantees (states) must report

data including but not limited

to the following:

• Number of accused status

offenders and nonoffenders

held in any secure

detention or correctional

facility for more than 24

hours.

• Number of accused status

offenders and nonoffenders

securely detained in any

adult jail, lockup, or

nonapproved collocated

facility for any period of

time.

• Number of accused status

offenders and nonoffenders

placed in facilities not near

their home, not the least

restrictive appropriate

alternative, and not

community-based.

• Number of juvenile

offenders and nonoffenders

not separated from adult

criminal offenders in

facilities used for secure

detention and confinement

of both juveniles and adults.

• Number of juveniles

detained in collocated

facilities not approved by

the state that were not sight

and sound separated from

adult criminal offenders.

• For 21 youths involved in a

truancy abatement program,

school attendance increased by an

average of 89 percent.

• A runaway youth program

provided counseling to 129

runaway youths and their families;

of 57 cases closed, 86 percent of

youths returned to home, a friend,

or a relative.

• Discontinued a violence

prevention curriculum for middle

and high school students and a

conflict resolution training program

for eighth-grade students due to

the programs’ lack of

effectiveness.

• Twenty-six status offenders and

their families participated in family

counseling activities that focused

on parenting skills,

communication, and prosocial

behaviors.

• A mentoring program has

continued to have difficulty in

recruiting appropriate mentors.

• Enrolled 40 youths pending

adjudication and/or disposition in

home detention programs in lieu of

incarceration, shelter, or detention

placement.

• Despite efforts, was unable to

gather and analyze data on youths

participating in a tutoring program

due to a lack of school system

cooperation.

Comprehensive

Gang Initiative

2 This program assists

community coalitions in

implementing and testing the

comprehensive gang model—

a collection of strategies

aimed at the prevention,

intervention, and suppression

of youth gang crime and

violence. The model requires

communities to assess local

gang problems, identify key

risk factors among youths,

develop program plans, and

• Number of juveniles served.

• Age, sex, and income level

of participating youths.

• School participation and

academic performance for

participating youths.

• Criminal activity, gang

activity, drug use, and

alcohol use by participating

youths.

• Project staff’s perceptions

of the seriousness of gang

and nongang problems in

• Received university grant to fund

local project evaluation.

• Developed action team to address

problem of female gang members.

• Project youths were not

participating in tattoo removal

program; project staff need to

better promote benefits of tattoo

removal.

• Recruited five new community

leaders to join project.

• Held biweekly staff meetings to

discuss progress of participating
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 45 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program

name

Active

grantees

(FY2000) Program description

Specific data required from

and reported by program

grantee(s)

Examples of information reported

by grantee(s) per general

requirements

evaluate program outcomes.

Coalitions focus on five core

strategies: community

mobilization, academic and

economic opportunities, social

intervention, gang

suppression, and

organizational change and

development.

the community.

• Nature and frequency of

services provided to youths

and families.

• Youth workers’ (probation

officers, case workers)

perceptions of the

effectiveness of services

provided to youths and

families.

youths.

• Inadequate staffing levels caused

delays in processing higher than

expected numbers of youth

referrals and prevented project

expansion to additional

neighborhoods.

• Revised staff procedures for

delivering and coordinating

warrants.

• Recruited 25 girls for selected

projects.

Rural Gang

Initiative

4 A spin-off of the

Comprehensive Gang Initiative

that adapts the comprehensive

gang model to rural

communities. Two sites have

incorporated the full version of

the model and two sites follow

a modified version of the

Comprehensive Gang

Initiative.

• Aggregated demographic

data for community

(unemployment rates,

education attainment, race).

• Community attitude data

(community’s perception of

gang problem).

• Youth attitude data

(perceptions about ganginvolved youths from at-risk

youths).

• Law enforcement crime

data (data on nature of

criminal gang activity in

community—number of

assaults, property crimes,

etc.)

• Community resource data

(inventory of services

available to gang and

community members—e.g.,

conflict resolution, individual

counseling, etc.)

• Experienced delay in hiring a

project coordinator and outreach

staff due to a lack of qualified

applicants; project is preparing for

second recruitment effort.

• Developed steering committee

composed of representatives from

community organizations involved

with youths.

• Attended training on

implementation of the

comprehensive gang model.

• Located and rented a building to

house project staff.

• Insufficient staff to follow up on

court orders against targeted

youths who were at-risk for joining

a gang.

• In the process of forming an

intervention team composed of law

enforcement, school, and social

services personnel.

• Hired a gang crime specialist.

• Computer failure prevented police

department from providing crime

data.

Juvenile

Mentoring

Program

(JUMP)

175 JUMP provides one-to-one

mentoring for youths at risk of

delinquency, gang

involvement, educational

failure, or dropping out of

school. Awards are made to

local educational agencies or

private or nonprofit agencies

that have clearly defined

relationships with local

schools. Grantees must

conduct thorough background

checks on volunteer mentors

to ensure a safe environment

Youth information:

• Demographic information,

such as age, gender, and

ethnicity.

• Living situation.

• Medical or mental health

problems.

• Total number of school

absences.

Mentor information:

• Demographic information

such as age, gender, and

ethnicity.

• Reason for becoming

• Exceeded goal of matching 50 atrisk girls with mentors.

• Mentors and mentees participated

in classes highlighting other

cultures.

• Retained 50 percent of mentors

from the previous year.

• Experienced problems with

parents who were unwilling to

complete project requirements,

such as interviews and home

visits.

• Continued to recruit mentors from

local businesses.
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 46 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program

name

Active

grantees

(FY2000) Program description

Specific data required from

and reported by program

grantee(s)

Examples of information reported

by grantee(s) per general

requirements

for each child. JUMP also

emphasizes projects that are

established in communities

where 60 percent or more of

the youths qualify to receive a

free or reduced-price lunch.

involved in the mentoring

program.

• Highest education

completed.

• Previous mentoring

experience.

• Twenty out of 26 mentors who

responded to a survey said the

program met their expectations

and 23 said they would participate

again.

• Twenty-five youths dropped out of

a project.

SafeFutures 6 This program, based on

OJJDP’s comprehensive

strategy model, targets

communities that have already

begun to reform their juvenile

care systems. Program goals

include developing a full range

of graduated sanctions,

ensuring community safety,

and providing treatment

services for juveniles.

Communities use one grant to

tap into nine OJJDP program

components (e.g., juvenile

mentoring, aftercare, mental

health, delinquency

prevention, gang programs,

etc.)

• Demographic data (age,

race, sex) for youth

participants.

• Family/household

characteristics

(employment, number of

family members, number of

juveniles/adults living at

home, school history, family

rules) for youth participants.

• Self-reported data by youth

participants (neighborhood

characteristics, gang

affiliation, number of

delinquent acts,

characterization of family

relationships, drug and

alcohol abuse, emotional

problems, participation in

project activities, drug and

alcohol abuse by family

members).

• Developed and submitted

applications for additional funding.

• Experienced problems providing

substance abuse treatment

services due to the lack of a fulltime, trained substance abuse

counselor.

• Hired a new local evaluator.

• Referred 249 juveniles for direct

services and care coordination.

• Completed 13 community service

projects involving 394 students

and families.

• Experienced problems

implementing a mentoring

program—i.e., recruiting mentors,

retaining matches, and motivating

mentors to attend sessions.

• Continued coordination between

law enforcement agencies on

gang intelligence resulted in the

arrest of two arson suspects.

• Installed a computer system to

track gang members and gangrelated activities in the community.

Drug-Free

Communities

Support

Program

307 This program is intended to

increase citizen participation

and strengthen community

anti-drug coalition efforts to

reduce substance abuse

among youths in communities

throughout the United States

and, over time, to reduce

substance abuse among

adults.

• Number of organizations

represented on the

grantee’s coalition.

• Types of organizations

represented on the

coalition.

• Number of individuals who

are coalition members.

• Number of youths, parents,

and community leaders who

are coalition members.

• Number of recruitment

activities held.

• Steps taken to engage

members in drug prevention

planning.

• Activities sponsored by the

coalition.

• Number of participants in

activities.

• Exceeded goal of 100 community

organizations participating in

workshops.

• Produced anti-smoking

commercial.

• Invited health professionals to a

strategic planning meeting.

• Staff worked with drug-free clubs.

• Hired first full-time staff.

• Printed quarterly newspaper.

• Distributed 20,000 copies of a

family resource directory.

• Experienced low attendance at a

strategic planning meeting.

• Provided drug prevention

education to 78 after-school

detention students.
Appendix IV: Descriptions of OJJDP

Programs, Reporting Requirements, and

Examples of Reported Information

Page 47 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program

name

Active

grantees

(FY2000) Program description

Specific data required from

and reported by program

grantee(s)

Examples of information reported

by grantee(s) per general

requirements

• Target population for

activities.

Internet Crimes

Against

Children Task

Force Program

50 This program encourages

communities to develop

regional, multi-jurisdictional,

and multi-agency responses to

online enticement and child

pornography cases. These

task forces provide forensic,

prevention and investigative

assistance to parents, law

enforcement, prosecutors,

educators, and other

professionals working on child

victimization issues.

Grantees in this program must

complete a monthly

performance measures report

that collects information

including, but not limited to:

• Number of arrests.

• Number of child

pornography producer

cases.

• Number of child

pornography seller cases.

• Number of search warrants.

• Number of subpoenas.

• Number of referrals from

other agencies.

• Number of law enforcement

personnel trained.

• Number of presentations

given to students, parents,

teachers, citizens or the

media.

• Provided summaries of 24 cases

during the reporting period.

• Hired a child victim service

specialist.

• Achieved goal of implementing

training seminars for local, states

and federal law enforcement

officers.

• Acquired necessary equipment for

investigating online offenders such

as desktop and laptop computers,

cameras, and investigative

software.

• Made presentations on Internet

safety to teachers, parents,

community groups, and schoolage children.

Children’s

Advocacy

Centers

5 This program provides training

and technical assistance to

local children’s advocacy

centers and promotes their

expansion nationwide.

Children’s advocacy centers

coordinate law enforcement

and protective services

responses to victims of child

abuse through multidisciplinary teams. Under this

program four regional centers

and one national association

provide training and technical

assistance to local centers; the

national association also

awards subgrants to local

centers for training, program

development, and other

purposes.

Training and technical

assistance grantees are

required to report on:

• Number of training sessions

conducted.

• Number of practitioners

trained.

• Number of technical

assistance requests filled.

• Number of consultations

(on-site, electronic,

telephone) provided.

• Number of publications

developed and distributed.

National grantee is required to

report subgrantee data on:

• Number of new and

developing local centers.

• Number of on-site

monitoring visits.

• Number of grant

applications approved.

• Type of grants approved

(program development,

program support, training,

and state chapter).

Training and technical assistance

grantees:

• Finalized a diversity plan to ensure

that children receive services in a

culturally appropriate setting.

• Experienced difficulties agreeing

on research methods to evaluate

the children’s advocacy centers’

impact on case outcomes.

• Developed a “multidisciplinary

team development” curriculum.

• Identified and provided assistance

to underserved rural communities.

National grantee:

• Reviewed subgrantees’ biannual

reports on activities held during

the reporting period.

• Unexpected resignation of grants

coordinator hampered data

collection efforts and the awarding

and processing of grants.

• Approved 409 applications for

subgrants and notified recipients.

• Surveyed local accredited centers

and found that 49,350 juveniles

had been served during the

reporting period.

Source: GAO analysis of OJJDP, grantee, and evaluator data.
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 48 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

The Office of Juvenile Justice and Delinquency Prevention’s (OJJDP)

training and technical assistance programs and research programs are

unique in that they cut across many of OJJDP’s other programs. Also,

grantees1

 in each of these two areas typically report the same types of

quantitative performance data as other grantees in their area, even though

OJJDP does not usually prescribe the specific performance measures on

which the grantees should report. Training and technical assistance

grantees maintain the same types of data due to the common support

services they provide, and research grantees do the same because they

share a common goal of producing research products.

OJJDP awards grants to training and technical assistance providers to

support grantees in many of OJJDP’s grant programs. OJJDP administers

the vast majority of its training and technical assistance grants through

three of the Office’s divisions: (1) the Training and Technical Assistance

Division (TTAD), (2) the State and Tribal Assistance Division (STAD), and

(3) the Child Protection Division (CPD).2 Many of the training and

technical assistance providers are required to report information on their

projects’ activities and accomplishments semiannually using OJP’s

Categorical Assistance Progress Report form, as do all OJJDP grantees.3

OJP provides standard guidance on information to be reported, such as

information on the status of each of the grantees’ project goals and

quantitative results of their projects. STAD has not imposed additional or

more specific reporting requirements on its training and technical

assistance providers and, for the most part, neither have TTAD4

 nor CPD.5

 1

 OJJDP awards grants, cooperative agreements, and contracts to training and technical

assistance providers, and award grants and cooperative agreements to research providers.

Officials commonly refer to all of these as “grants” and to all award recipients as

“grantees.”

2

 In commenting on a draft of this report, the Assistant Attorney General noted that these

divisions do not always provide the same types of services.

3

 All training and technical assistance providers that receive grants are required to

complete progress reports. Providers that are under contract or have cooperative

agreements with OJJDP must also report on their accomplishments, but not necessarily

through a progress report. Often, they are required to report their activities on a monthly

basis.

4

 In commenting on a draft of this report, the Assistant Attorney General noted that TTAD

requires Juvenile Accountability Incentive Block Grants Program grantees to submit

monthly reports. These grantees comprise approximately one-fifth of all TTAD grantees

and account for approximately $5 million annually.

Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Performance Data for

Training and

Technical Assistance

Grantees
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 49 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Officials explained that OJJDP does not require all grantees to routinely

report prescribed data because it is reluctant to place additional reporting

requirements on grantees due to the Paperwork Reduction Act of 1995,6

which set goals to reduce the federal government’s reporting and

paperwork burden. Although most of these providers are not subject to

additional reporting requirements for prescribed data, it is not unusual for

them to report on the same or similar quantitative performance measures.

Because of the nature of the services they provide, training and technical

assistance providers tend to maintain like data that can readily be

counted, such as numbers of training events held, practitioners who

attended those events (“practitioners trained”), and technical assistance

requests filled. Some of these providers also produce publications or

materials, such as bulletins, surveys, curricula, brochures, and other

support materials, and report such information to OJJDP. Table 9

summarizes performance data we obtained regarding training and

technical assistance grants.7

 5

 CPD has prescribed additional, specific reporting requirements for providers in two of its

programs—the Children’s Advocacy Centers and the Missing and Exploited Children

Training and Technical Assistance program—such as reporting the number of training

events held.

6

 The Paperwork Reduction Act applies to identical reporting requirements imposed on 10

or more parties. CPD’s training and technical assistance providers that have specific

reporting requirements are not affected by the Act since they involve less than 10

providers.

7

 We attempted to obtain fiscal year 2000 data from all grantees; however, in some cases

OJJDP provided calendar year 2000 or fiscal year 2001 data due to a lack of available or

representative data. Because we obtained publications data that pertained only to some of

the grantees, we did not include such information in the table. For example, State

Challenge Activities training and technical assistance providers reported producing 10

bulletins in fiscal year 2001and Juvenile Accountability Incentive Block Grant providers

reported that in fiscal year 2000 they developed or were developing 628 supportive

materials, such as reports, curricula, and newsletters.
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 50 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 9: Training and Technical Assistance Performance Data Reported by OJJDP

Grantees

Division/program supported Training events

Practitioners

trained

Technical

assistance

requests

filleda

Training and Technical Assistance Division (FY2000)

Juvenile Accountability Incentive

Block Grants Program 227 13,816 1,374

Life Skills Trainingb64 858 16

National Training and Technical

Assistance Centerc d d 329

Other programse f36,000 1,633g

State and Tribal Assistance Division (FY2001)h

Community Prevention Grants

Program (Title V) 55 2,171 4

Formula Grants Program 10 1,148 265

State Challenge Activities 2 69 71

Comprehensive Strategy for

Serious, Violent, and Chronic

Juvenile Offenders

76 2,252 232

Child Protection Division (Calendar Year 2000)

Children’s Advocacy Centers 183 10,939 6,501i

Court Appointed Special Advocate

Program 194 5,312 21,323

Model Courts Program 161 42,537 401

Permanency Planning 124 38,888 4,264

Prosecutor Training f9,300 1,800

Parents Anonymous 3 149 f

Missing and Exploited Children

Training and Technical Assistance

programj

68 2,950 f

a

Data can include the number of in-depth, on-site visits and/or the number of responses to telephone

and e-mail requests.

b

Data for this program were from the academic year 2000-2001, which roughly corresponds with fiscal

year 2001.

c

The Center receives requests from the field for technical assistance, which it then fills in-house or

refers to other technical assistance providers. In fiscal year 2000, it either filled or referred 329

requests for technical assistance. The Center does not provide direct training to the field.

d

Not applicable.

e

Data reported from 19 other training and technical assistance grantees.

f

Data not available.

g

Reflects the number of communities that received technical assistance.
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 51 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

h

At the suggestion of OJJDP officials responsible for the following programs, data for the Community

Prevention Grants Program are prorated from the first 9 months of the fiscal year, training and

technical assistance data for State Challenge Activities are prorated from the first 8 months of the

fiscal year, and data for the Comprehensive Strategy for Serious, Violent, and chronic Juvenile

Offenders are prorated from the first 6 months of the fiscal year.

i

Includes data from the national grantee and three of the four regional grantees.

j

Data are for fiscal year 2000.

Source: GAO analysis of OJJDP and grantee data.

OJJDP officials cautioned that not all providers share common definitions

of “training” and “technical assistance.”

8

 For one thing, the difference

between the two is not always clear and, therefore, it is sometimes

difficult to definitively categorize a provided service as training versus

technical assistance. Furthermore, not all training events are equal. For

example, some providers might characterize both a 1-day training

conference and a 10-day training workshop as a training event; others

might differentiate between the two. Furthermore, one provider might

consider a telephone request from a grantee as merely a query, while

another might consider it a request for technical assistance.

OJJDP administers its research grants out of its Research and Program

Development Division (RPDD).9

 RPDD sponsors empirical studies on an

array of topics related to juveniles and delinquency, from the causes of

violence to the impact of victimization. The overall goal of these research

grants is to generate credible and useful information to help prevent and

reduce juvenile delinquency and victimization. Research grantees are not

only expected to collect data but to analyze and disseminate their analyses

to the public. RPDD requires all research grantees to produce publishable

products and, in some instances, RPDD specifies the type of products to

be published depending on the results of the research. Thus, according to

OJJDP officials, one measure of a research grantee’s performance is the

number of products the grantee has published.

Like most OJJDP grantees, research grantees must report information on

their projects’ activities and accomplishments semiannually through

 8

 In commenting on a draft of this report, the Assistant Attorney General pointed out that

TTAD published the Training, Technical Assistance, and Evaluation Protocols booklet 2

years ago, which defined both training and technical assistance.

9

 Some grants administered through other OJJDP divisions have a research component, but

they are not considered “research grants.”

Performance Data for

Research Grantees
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 52 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

progress reports. RPDD does not impose additional, specific reporting

requirements on grantees, but it does encourage them to report on

products produced through private publishers (as opposed to those

published through OJJDP).10 The division director told us that it is not

necessary to impose specific requirements on grantees in addition to the

semiannual progress report requirements because officials work closely

with grantees throughout the life of the grants.

OJJDP research grantees produce products based on their OJJDP-funded

research. Some of these products are approved and published by OJJDP,

who in turn disseminates the products through its own distribution

process (see app. II for a description of OJJDP’s product dissemination

process). Grantees also publish many products that are based on their

OJJDP-funded research through private publishers. OJJDP officials told us

they give their research grantees latitude to privately publish products

because the majority of their research grantees are academics whose

funding depends on the number of products they publish, and because

grantees often have funding sources in addition to OJJDP.

Tables 10 and 11 summarize the products that active OJJDP research

grantees published through OJJDP and private publishers as a direct result

of OJJDP-funded research. Table 10 describes the number of research

products published by OJJDP from 1993 through September 2000, by

topic. Table 11 shows the number of products, by topic, that grantees with

active research grants privately published between 1986 and June 2001, or

were in the process of publishing in June 2001.

 10 Officials told us that they ask grantees to submit products for review prior to external

publication, but often grantees do not do this. In the past, grant applicants have turned

down funding when it was contingent on OJJDP’s review of privately published products,

citing the importance of academic freedom. Also, grantees oppose submitting external

products to OJJDP prior to publication because many academic journals will not publish

research findings that have been reviewed or disseminated prior to submission.
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 53 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 10: Active OJJDP Grantees’ Research Products, Published by OJJDP

Topic Summaries Reports Fact sheets Bulletins

Aftercare (post-detention) 2 1 1 3

Causes and Correlates of

Delinquencya

12 4 6

Crimes against juveniles 0 0 0 7

Delinquency prevention 1 5 1 6

Detention/corrections 3 2 7 2

Gangs 5 0 4 8

Juvenile courts 1 3 9 4

Juvenile health 0 1 1 2

Juvenile justice system reform 1 2 0 2

Juvenile offenders 3 3 5 10

School safety 0 2 0 0

Other 0 1 1 0

Total 17 22 33 50

Note: Table includes some products published as a result of OJJDP evaluation and statistics grants.

a

This topic is comprised of products from Causes and Correlates of Delinquency grants. In

commenting on a draft of this report, the Assistant Attorney General noted that these products can be

categorized under a number of the topics listed.

Source: GAO analysis of OJJDP data.

Table 11: Active OJJDP Grantees’ Research Products, Published Externally

Topic

Journal

articles Book chapters

Other

publicationsa

Causes and Correlates of

Delinquencyb 57 43 17

Crimes against juveniles 12c

6 2

Delinquency prevention 00 2

Detention/corrections 30 0

Gangs 13d

9 2

Juvenile courts 51 1

Juvenile health 51e

5 7

Juvenile offenders 30 2

School safety 16 0 4

Other 00 5

Total 160 64 42

Note: Table includes some products published as a result of OJJDP evaluation and statistics grants.

a

Includes books, reports, and other publications.

b

This topic is comprised of products from Causes and Correlates of Delinquency grants. In

commenting on a draft of this report, the Assistant Attorney General noted that these products can be

categorized under a number of the topics listed.
Appendix V: Training and Technical

Assistance and Research Performance Data

Reported by OJJDP Grantees

Page 54 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

c

Includes one paper submitted but not yet accepted for publication.

d

Includes one paper presented at an academic conference.

e

Includes three papers submitted but not yet accepted for publication.

Source: GAO analysis of OJJDP and grantee data.
Appendix VI: States’ Compliance With Core

Requirements of the Juvenile Justice and

Delinquency Prevention Act of 1974

Page 55 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Table 12: State-Reported Compliance with Core Requirements, Calendar Year 1998

Core requirements

Number of

jurisdictions Jurisdictionsa

Deinstitutionalization of status offenders

Full compliance – zero violations 9 ME, MT, NY, VT, AS, GU, MP, PR, VI

Full compliance – de minimis exceptionsb39 AL, AK, AR, CA, CO, CT, DE, DC, FL, GA, HI, ID, IL, IN,

IA, KS, KY, LA, MD, MA, MN, MO, NE, NV, NH, NM, NC,

ND, OK, OR, PA, RI, TX, UT, VA, WV, WI

Not in compliance 4 MS, SC, TN, WA

Funds withheld pending additional compliance data 2 NJ, OH

Not participating 2 SD, WY

Separation of juvenile and adult offenders

Full compliance – zero violations 41 AL, AZ, AR, CA, CO, CT, DE, DC, FL, IL, IN, KS, KY, ME,

MD, MA, MI, MN, MO, MT, NE, NV, NH, NM, NC, ND, OK,

OR, PN, RI, SC, UT, VT, VA, WA, WI, AS, GU, MP, PR, VI

Full compliance – exception provision 11 AL, GA, HI, ID, IA, LA, MS, NY, TN, TX, WV

Funds withheld pending additional compliance data 2 NJ, OH

Not participating 2 SD, WY

Jail and lockup removal

Full compliance – zero violations 11 AL, DC, NC, OR, SC, VT, AS, GU, MP, PR, VI

Full compliance – de minimis exceptions 38 AZ, AR, CA, CO, CT, DE, FL, GA, HI, ID, IN, IA, KS, KY,

LA, MD, MA, MI, MN, MS, MO, MT, NE, NV, NH, NM, NY,

ND, OK, PA, RI, TN, TX, UT, VA, WA, WV, WI

Not in compliance 3 AK, IL, ME

Funds withheld pending additional compliance data 2 NJ, OH

Not participating 2 SD, WY

Disproportionate minority confinementc

Completed identification and assessment,

implementing intervention and monitoring

21 AR, CO, CT, FL, GA, ID, IN, IA, LA, MO, NV, NC, ND, OK,

PA, SC, TN, UT, VA, WA, WV

Completed identification and assessment,

implementing intervention

15 AK, AZ, HI, IL, KS, MD, MA, MI, MN, NE, NM, NY, RI, TX,

WI

Completed identification and assessment/updating

data and revising intervention plan

2 CA, DE

Completed identification/implementing

intervention/conducting formal assessment

3 AL, MS, MT

Conducting identification phase 1 NH

Exempt from requirement 7 ME, VT, AS, GU, MP, PR, VI

Status under review 5 DC, KY, NJ, OH, OR

Not participating 2 SD, WY

Note: OJJDP based its initial determination of states’ eligibility to receive fiscal year 2000 Formula

Grants on 1998 monitoring reports—the most recent data available as of March 2001.

a

AS = American Samoa; GU = Guam; MP = Northern Mariana Islands; VI = Virgin Islands.

b

Fewer than 29.4 violations per 10,000 persons under age 18 in the state.

c

States reported data on disproportionate minority confinement in their fiscal year 2000

comprehensive plans for compliance.

Appendix VI: States’ Compliance With Core

Requirements of the Juvenile Justice and

Delinquency Prevention Act of 1974
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 56 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

This appendix contains information on the 10 impact evaluations that the

Office of Juvenile Justice and Delinquency Prevention (OJJDP) has funded

of its own programs since 1995 and for which we have assessed the

methodological rigor, as well as information on one impact evaluation—

Teen Courts—that we did not assess. Five of the 10 evaluations are in their

formative stages, and five are well into their implementation. For each of

the 10, we have included a description of the program being assessed, the

evaluating organization, a description of the evaluation and its findings,

and our assessment of the evaluation.1

 As discussed in the Scope and

Methodology section of this report, we did not assess the methodological

rigor of the Teen Courts evaluation. However, we have included a

summary of this evaluation at the end of this appendix. 2

Program Description: Parents Anonymous is a national child abuse

prevention program that began in 1970. It consists of 32 state and local

organizations and over 1,000 weekly mutual support groups. The principal

participants are at-risk parents, though complementary projects exist for

children. The cornerstones of the program are mutual support and shared

leadership.

Evaluator: National Council on Crime and Delinquency.

Evaluation description: This evaluation, which is in the beginning

stages, is based on a proposal to conduct a process evaluation in year 1,

and an outcome evaluation in years 2 and 3. The researchers will

determine how Parents Anonymous is staffed and operated in different

settings, how it attempts to change the behavior and attitudes of parents,

and what factors are related to its effectiveness. While the specifics of an

outcome evaluation design are yet to be determined, the researchers

 1

 We relied on documents OJJDP provided to us in April 2001 in assessing the evaluation

methodologies and reporting on each evaluation’s status.

2

 OJJDP provided us with this evaluation description.

Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Impact Evaluations of

OJJDP Programs in

Their Formative

Stages

Parents Anonymous
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 57 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

indicate that they will most likely compare the Parents Anonymous

participants with a control group and with Parents Anonymous dropouts.

The process evaluation received $300,000 for a 3-year period.

Evaluation findings: It is too early in this evaluation to have reported

results.

GAO assessment: No assessment of the impact evaluation is possible

because it has not yet been planned. More fully developed proposals will

need to be made to OJJDP to obtain funding for the impact evaluation

portion of the study.

Program Description: This program replicates and evaluates Project

PATHE, which was first implemented in the Charleston County School

District in South Carolina between 1980 and 1983. Project PATHE is a

comprehensive school-based program that combines services to students

who are at elevated risk for developing problem behaviors with schoolwide organizational changes intended to improve both school climate and

students’ behavior. Local educators are encouraged to develop their own

(1) explanations for the causes of their schools’ violence and behavior

problems and (2) specific local objectives and ways to prevent these

problems by empowering teachers’ decision-making and fostering

collaborative and nonhierarchical efforts. For various reasons, the grantee

has had difficulty in selecting a school district for the replication. Funding

for this effort began in October 1999 with funds provided by the Centers

for Disease Control and Prevention.

In commenting on a draft of this report, the Assistant Attorney General

pointed out that all funds for this effort come from the Centers for Disease

Control and Prevention through an interagency agreement. OJJDP

awarded this grant, which the two agencies jointly manage. OJJDP

originally identified Project PATHE as an OJJDP-funded program.

However, on the basis of the Assistant Attorney General’s comments, this

program appears to be a non-OJJDP-funded impact evaluation. Since we

assessed its methodological rigor, we have included it with the other

OJJDP-funded evaluations.

Evaluator: Institute of Behavioral Science, University of Colorado.

Evaluation description: Although the impact evaluation was expected

to be completed in July 2001, it has yet to begin. Original plans called for

one school district to be selected for the replication and evaluation. In a

Positive Action Through

Holistic Education

(Project PATHE)
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 58 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

school district, one high school and one middle school would be selected

to receive the program, once school principals had been informed and

staff surveys had been conducted to determine interest in participating.

Comparison schools (one high school and one middle school) in the same

district would be selected with similar demographic characteristics of

students, levels of problem behaviors, and other unspecified

organizational characteristics, as well as a low probability of mounting

other school-wide efforts to reduce problem behavior during the study

period. Plans are to collect data before and after project implementation.

All students and staff in all schools would be surveyed in September and

May for 3 consecutive school years. In addition, 200 students from each

high school—100 seniors and 100 sophomores—and 200 students from

each middle school—100 eighth graders and 100 sixth graders—would be

sampled in the first year of the study for followup for 3 years. Each sample

is to include 25 students identified as “high risk.” It is not clear, however,

how these samples will be selected. Schools are to be visited three times

yearly during the study, and school records and teacher ratings of student

behavior in each school will be used in establishing differences between

the program and comparison schools. Program outcomes are to be

selected after determining goals and objectives collaboratively with local

school officials. Multivariate statistical analyses, such as logistic

regression, are planned. Because of the difficulty in selecting a school

district the latest progress report indicates some changes in this design.

Agreements have been signed with two school districts (Charleston, SC

and Baltimore, MD), instead of one, and plans are to conduct the

replication and evaluation in four middle schools (two program and two

comparison schools) in each district—high schools have been excluded.

The amount of the grant is $875,000. However, additional funds have been

requested.

Evaluation findings: It is too early in this evaluation to have reported

results.

GAO assessment: The evaluation, as designed, is basically sound. The

variation in program structure and implementation between schools may

limit generalizability. While the researchers do suggest awareness of

potential problems due to students switching schools, they do not clearly

indicate, at this early design stage, how possible contamination will be

handled.

Program Description: The Rural Gang Initiative is a comprehensive

strategy to ameliorate gang problems in rural areas. The program was

Rural Gang Initiative
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 59 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

adapted from the comprehensive gang model, developed at the University

of Chicago, and implementation began in two rural areas in fall 2000. The

program consists of five elements: community mobilization, opportunities

provision, social intervention, suppression, and organizational change and

development.

Evaluator: National Council on Crime and Delinquency.

Evaluation description: The impact evaluation of this program began in

January 2001 and is expected to be completed in December 2003. It

focuses on two of the four sites, Mount Vernon (IL) and Glenn County

(CA), that were part of a year-long feasibility study that began in April

1999. The other two sites are not part of the evaluation because they did

not fully implement the model.3 Information will be collected from these

two sites on gang-involved youths and youths at-risk of gang involvement,

all of whom have participated in the program. However, it is unclear how

the youths will be sampled or whether all participants will be included.

Data will be obtained from a variety of sources, including interviews,

organizational surveys, and administrative data from schools and the

justice system. No comparison groups are planned, though the researchers

indicate that attempts will be made to collect data that will permit an

assessment of alternative explanations for program effects. The

researchers plan to collect data before and after program implementation;

then they will measure any changes and follow-up with program

participants for at least 12 months after their participation in the program.

Individual sites are to identify specific program outcomes. Although the

researchers have described prospective outcome measures, such as the

reduction of gang-related crime and the prevention or reduction of gang

involvement, they (1) have not chosen the outcome measures that they

will use and (2) have not provided information on the types of statistical

analyses planned. However, the evaluation is still in the formative stages.

At the time of our review, this evaluation had received $525,000 in funding.

Evaluation findings: It is too early in this evaluation to have reported

results.

 3

 In commenting on a draft of this report, the Assistant Attorney General pointed out that

these two sites did not implement the comprehensive gang model because their gang crime

problem did not warrant the model’s prescribed intervention approach.
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 60 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

GAO assessment: It is too early to tell from this evaluation how effective

it will be. However, the absence of plans for comparison groups and the

lack of specificity regarding any other control mechanisms, make it

unclear how program effects will be distinguished from alternative

explanations at this stage of the evaluation.

Program Description: The Safe Schools/Healthy Students program has

funded 77 school districts nationwide, with grants ranging up to $3 million,

to develop services and activities to promote healthy childhood

development and prevent violence and drug abuse. The program also aims

to develop greater collaboration and cooperation between communities

and schools to enhance their effectiveness in responding to and reducing

violence. Each project model is intended to evolve over time.

Evaluator: Research Triangle Institute.

Evaluation description: This impact evaluation began in October 1999

and data will be collected through the Spring of 2005. School and

community-based archival records, surveys of key coalition personnel,

teachers, superintendents, principals and other school staff, and teacher

behavioral checklists for students in selected grades will be gathered in all

77 school districts. The evaluation will compare data from participating

sites with national norms and with similar information from matched

nonparticipating (comparison) sites that the researchers surveyed in each

of two large, nationally representative studies of school districts. The

matching will be based on unspecified socio-demographic characteristics

and responses to policy-related questions in the baseline survey. Archival

data will be collected yearly over a 5-year period. Survey data will

generally be collected at three points in time, about 2 years apart. The

survey items will be drawn from established instruments and will provide,

in conjunction with the archival or administrative data, information on

behavioral outcomes, risk factors and inhibiting factors, and indicators of

positive development and mental health. At the time of our review, this

evaluation was funded at approximately $5.6 million.

Evaluation findings: It is too early in this evaluation to have reported

results.

GAO assessment: This evaluation, as designed, is basically sound.

Safe Schools/Healthy

Students
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 61 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program Description: This demonstration program seeks to prevent and

reduce the impact of family and community violence on young children,

primarily aged 0 to 6, in up to 12 communities. The program plans to

create a comprehensive service delivery system that integrates service

providers (in the fields of early childhood education/development, health,

mental health, and all manner of prevention, intervention, and treatment

programs), law enforcement, legal services, and the courts. It also seeks to

improve the access, delivery, and quality of services to children exposed

to, and at high risk of, violence. Project sites are to be selected through a

competitive grant process. Funding for the program began in October

1999.

Evaluator: Caliber Associates.

Evaluation description: This evaluation began in May 2000, and is

expected to end in September of 2005. The effect of the Safe Start

Initiative will be measured within and across all participating communities

at both the community and individual levels. Multiple data collection

methods, including focus groups, service agency usage logs and

documents, and random-digit-dialing telephone surveys, will be used.

Plans are to collect data before program implementation and for 4 years

after the program begins in each site. Although specific outcome measures

have not yet been identified, they are expected to address such areas as

agency referral levels and quality of service, increased interagency

collaboration, knowledge and perceptions of police and child protective

services, rates of child maltreatment, physical injuries, and mental health

problems. No comparison communities are to be studied. Analyses are to

include regression and time series models. At the time of our review, this

initiative had received $1 million.

Evaluation findings: It is too early in this evaluation to have reported

results.

GAO assessment: The absence of any appropriate comparison

communities and the variability in program implementation and

components across the 12 study sites will make it difficult to find

compelling evidence of program effects.

Safe Start Initiative
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 62 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program Description: This program aims, in the five sites in which it is

being implemented and evaluated (Mesa and Tucson, AZ; Bloomington, IL;

San Antonio, TX; and Riverside, CA) to reduce gang-related crime through

five interrelated strategies: community mobilization; provision of social,

educational, and economic opportunities; suppression of gang violence;

social intervention; and organizational innovation. It involves the

collaborative efforts of the police, probation officers, prosecutors, judges,

schools, youth agencies, churches, housing authorities, and governmental

agencies. It targets youths at strong risk of gang membership and crime,

and youths already involved in serious gang crime. The evaluation began

in 1995.

Evaluator: University of Chicago.

Evaluation description: The impact evaluation of the program began in

May 1995, and is expected to be completed in April 2002. Each project site

is to be matched with a comparison site. In four of the five sites, the

program participants and comparison groups were selected from similar

gang problem areas within the same city; in the fifth site, a separate

comparison community was selected. Between 100 and 115 youths, ages

12 to 21, who were involved in gangs or at risk of involvement, were

selected to participate in the program in each site, and between 77 and 134

similar youths in each site were selected for comparison purposes. Neither

program nor comparison group youths were selected randomly. A large,

complex, communitywide data collection effort is being employed in each

site, through a variety of methods and sources, including organizational

surveys, youth surveys, reports from service workers, police and school

records, local newspaper reports, and census data. Data were to be

collected at baseline and after the first and third years of the program. The

principal outcomes to be measured are gang crime patterns at the

individual, gang, and community levels. The evaluation will also consider

changes in opportunities, as well as integration in and alienation from

conventional individuals and institutions. A variety of analyses of the data

are planned, including time-series analyses and hierarchical linear models.

At the time of our review, the evaluation had received approximately $3

million.

Impact Evaluations of

OJJDP Programs Well

Into Their

Implementation

Comprehensive

Communitywide Approach

to Gang Prevention,

Intervention, and

Suppression Program

(Comprehensive Gang

Initiative)
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 63 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Evaluation findings: Preliminary results have been reported for

program participants, but not comparison groups. Because of missing data

and other problems, reporting deadlines may not be met, and two of the

five project sites and associated comparison sites have been deferred from

the current analyses.

GAO assessment: The evaluation, as designed, is basically sound.

However, numerous difficulties in obtaining data threaten parts of the

evaluation. Also, the way in which subjects were selected for the study

may be problematic, and it is unclear how program and comparison

youths were matched within sites.

Program Description: Since 1998, 85 communities and 4 colleges in at

least 10 states have been awarded subgrants under the discretionary grant

component of this program to enforce underage drinking laws. In most

states, a diverse group of stakeholders are involved in planning a variety of

projects under this program that can include media campaigns, merchant

education, compliance checks and other enforcement, youth leadership

training, school-based education, and the development of local coalitions

and interventions aimed at reducing underage drinking. States and

communities are given substantial latitude in planning their projects;

interventions are not standardized across communities.

Evaluator: Wake Forest University School of Medicine.

Evaluation description: The effort to evaluate the discretionary grant

component of this program began October 1, 1998, and is expected to be

completed December 31, 2001. Data are to be collected—from telephone

surveys of police chiefs, sheriffs, and youths in participating communities

and matched comparison communities in at least 9 states—before or early

on in project implementation and at least 1 year after project initiation.

Project sites to be evaluated were initially chosen from all states

participating in the program. It is unclear whether these sites are

representative of all participating project sites. In the first year, surveys

were conducted in 52 participating communities and a similar number of

comparison communities. In the second and third years, surveys will be

conducted in those same communities and 34 others—17 in each group.

The participant and comparison communities were matched on median

income, liquor law violations, percentage attending college, and

population size. The surveys of the top one or two law enforcement

officials in each community will provide information on local law

enforcement efforts, including the number of compliance checks

Enforcing the Underage

Drinking Laws Program
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 64 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

conducted in each year. A small number of youths from each site are to be

selected at random for the surveys each year. The youth surveys will

obtain data on perceptions of alcohol availability, peer and personal

alcohol use, and alcohol-related problem behaviors including binge

drinking and drunk driving. At the time of our review, this evaluation had

received approximately $945,000.

Evaluation findings: While some demographic data have been reported

from the baseline survey, no results have been reported involving program

effects.

GAO assessment: The researchers suggest aggregating all program

communities together and all comparison communities together to

diminish community sample size problems, which may mask program

effects. In addition, the wide variation allowed in program implementation

may compromise the interpretation and generalizability of any findings.

Program Description: The Intensive Aftercare program provides

intensive supervision and services to serious juvenile offenders for 6

months following their release from secure confinement. The goal is to

facilitate reintegration and reduce recidivism. The program was

implemented, beginning in June of 1993, by various youth service offices

and departments of corrections in four states: Colorado, Nevada, New

Jersey, and Virginia. New Jersey was eventually dropped because of

implementation problems, so the evaluation of the program is being

completed in the other three states.

Evaluator: National Council on Crime and Delinquency.

Evaluation description: Beginning in 1995, youths entering correctional

facilities in the three states (four counties in Colorado including

Metropolitan Denver; Clark County, NV; and Norfolk County, VA) were

screened for eligibility and randomly assigned, within each site, to the

treatment group (whose members participated in the Intensive Aftercare

program upon release) or control group. Between 1995 and 1999, 82

youths were assigned to the program and 68 to the control group in

Colorado, 120 youths were assigned to the program and 127 to the control

group in Nevada, and 75 youths were assigned to the program and 45 to

the control group in Virginia. Information was collected for study

participants at baseline (that is, upon entry into the institution), before

release from the institution (9 to 12 months after baseline, or entry),

immediately after completing the program (6 months after release), and 6

Intensive Aftercare
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 65 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

months after completing the program (12 months after release). The data

collected, using survey instruments, standardized tests, monthly case

management forms, and administrative (police and court) databases,

included social and criminal history and demographic data, information on

the extent of supervision and services, and the extent of criminal activity

following institutional release. Many of the measures being employed in

the study, according to the researchers, are standard and have been

validated. At the time of our review, the evaluation had received

approximately $932,000 in funding.

Evaluation findings: The preliminary findings offered from this

evaluation suggest that the Intensive Aftercare participants did receive

greater supervision and more services after release than the control group,

which suggests some success in implementing the program. Outcome

results related to reintegration and recidivism are not complete, and the

interim results are mixed as to whether the program is associated with

positive outcomes.

GAO assessment: This is a well-designed study, though serious missing

data problems, if not corrected, may make it difficult to determine the

outcome of this program.

Program Description: JUMP was established by Part G of the Juvenile

Justice and Delinquency Prevention Act of 1974, as amended in 1992.

Through that legislation, the Congress authorized OJJDP to award 3-year

grants to community-based not-for-profit organizations or to local

educational agencies. The grantees are to support one-on-one mentoring

projects that match volunteer adult mentors with youths at risk of

delinquency, gang involvement, educational failure, and dropping out of

school. The legislation also provided funding for a national, cross-site

evaluation of JUMP. OJJDP guidelines emphasize the need for projects to

recruit, train, supervise, and do thorough background checks for all

volunteer mentors; develop procedures for appropriately matching youths

and mentors; define the population of at-risk youths to be served; develop

guidelines for the type, frequency, and duration of youth and mentor

project activities; and establish procedures for gathering and reporting

data to support the evaluation process. As of November 2000, 175 JUMP

projects had been funded, in amounts ranging from $180,000 to $210,000

over a 3-year period.

Evaluators: Information Technology International and Pacific Institute

for Research and Evaluation.

Juvenile Mentoring

Program (JUMP)
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 66 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Evaluation description: This evaluation began in May 1997 and is

expected to conclude on September 30, 2002.4

 Three approaches are being

taken to determine how well JUMP is accomplishing its objectives. The

first is a modified pre-post design that involves a within-subject

comparison of the characteristics of youths at the time they enter and exit

the program and between-subject comparisons of youths entering and

exiting the program at the same time. The second approach is a best

practices approach that will use structural equation models to estimate

what program features or activities, including success in matching

mentors and youths, are most likely to contribute to program success in

reducing the risk of school and family problems, delinquency, and drug

use among youths. The third approach relies on combined youth outcome

data and community data to determine community cost offsets. The

evaluation was funded at $3.3 million.

Evaluation findings: In their November 2000 JUMP annual report, the

evaluators provided considerable descriptive information about the

various JUMP projects, the characteristics of the youths and mentors, and

information on youth-mentor matching. The only “outcome” information

thus far provided, however, is information on how satisfied youths and

mentors were with the mentoring experience and how much benefit each

perceived was derived from the experience. None of the three analytic

approaches described above has been successfully applied to study

outcomes because of a variety of pitfalls experienced by the national

evaluation team, most notably insufficient data on school performance

and behavioral measures (e.g., delinquent behavior and arrests).

GAO assessment: The researchers are employing multiple and

innovative strategies to determine the effectiveness of JUMP in achieving

its objectives. It is not clear, however, whether definitive evaluation

results can be reached in the absence of outcome data on youths who, in

the same project areas at the same points in time, do not receive the

program. In addition, data limitations, if not corrected, may be serious

enough to compromise findings.

 4

 In commenting on a draft of this report, the Assistant Attorney General pointed out that

the latest award sets project and budget end-dates of September 30, 2002. However, she

expects the evaluation will not be completed before September 30, 2004. OJP does not

move the project period forward beyond the budget period as this may imply an assurance

of future funding.
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 67 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Program Description: The Partnerships to Reduce Juvenile Gun

Violence Program is a multi-year demonstration program planned for four

sites (Baton Rouge and Shreveport, LA; Syracuse, NY; and Oakland, CA). It

began in 1997 and is expected to conclude in 2001. However, one site,

Shreveport, was dropped from the program early. The program aims to

reduce youth gun violence by enhancing, in specific target areas of these

cities, prevention and intervention strategies and strengthening

partnerships among community residents, law enforcement agencies, and

the juvenile justice system. The program involves mobilizing the

community, establishing agency linkages, and planning case management

for juveniles with gun charges in year 1, linking at-risk youths to services

in year 2, and expanding opportunities for youths in year 3.

Evaluator: COSMOS Corporation.

Evaluation description: The strategy for evaluating the impact of this

program has evolved as the program has unfolded. An impact evaluation

was planned for three sites and was to include (1) a comparison of

changes in crime rates in target areas of these cities before and after the

implementation of the program, (2) a comparison of responses from highrisk youths in targeted areas surveyed before and after the program was

implemented and services were provided, and (3) information on changes

in policies and caseloads revealed through focus group meetings and

interviews with agency officials. Crime rate information has thus far been

reported only for Oakland and Baton Rouge, and surveys have been

conducted only in Baton Rouge. In Baton Rouge, surveys were given to 92

high-risk youths in the criminal justice system identified through a variety

of processes. The sampling strategies for surveying these high-risk youths

were unlikely to yield generalizable results. In addition, fifth-, seventh-,

and ninth-grade students in six schools in the target area were surveyed in

March of 1999. It is unclear why these students and schools were sampled

and what response rates were. In 2000, a small sample of 50 youths in

Baton Rouge was identified as a possible matched comparison group for

arrest rate comparisons. At the time of our review, this evaluation had

received $1.2 million in funding, although a process evaluation is also

being conducted with these funds.

Evaluation findings: The researchers report decreases in gun-related

homicides and arrests in Oakland that were larger in the target area than

for the city as a whole. They also report decreases in gun-related

homicides in Baton Rouge. No analyses of results from the survey data

have been reported to date.

Partnerships to Reduce

Juvenile Gun Violence

Program
Appendix VII: Impact Evaluations OJJDP Has

Funded of Its Own Programs

Page 68 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

GAO assessment: Comparisons between crime rates in the target

community and the city as a whole may not be appropriate. Student and

school selection criteria are unclear, making it difficult to assess their

appropriateness for obtaining definitive results. In addition, if supporting

survey and administrative data are only gathered in one site, it will be very

difficult to generalize findings whether they appear positive or not.

The purpose of the Teen Courts evaluation is to measure the effect of

handling young, relatively nonserious violators of the law in teen courts,

rather than in traditional juvenile family courts. Although teen courts often

include many of the same steps used by formal juvenile courts (for

example, intake, preliminary review of charges, court hearing, and

sentencing), they differ from formal courts in that young people are able to

assist in the community decision-making process for dealing with juvenile

offenders. Youths may act as prosecutors, defense counsel, jurors, court

clerks, bailiffs, and judge (or as a panel of judges). To evaluate teen courts,

both a process and impact evaluation are used, with case studies and

comparison groups as part of the research design. In each of the four case

study sites (Anchorage, AK; Independence, MO; Maricopa County, AZ; and

Rockville, MD), data are collected on about 100 youths handled in teen

courts (experimental group) and 100 youths handled in the traditional

juvenile justice system (comparison group). Data are also collected on

several dimensions of program outcomes, including post-program changes

in teens’ perceptions of justice and their ability to make more mature

judgements as a result of the program. A process evaluation of the

projects—exploring legal, administrative, and case-processing factors that

hinder the achieving of project goals—is also being conducted.

Impact Evaluation of

OJJDP Program—

Summary of

Evaluation We Did

Not Assess

Teen Courts
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 69 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

This appendix contains summaries of the 24 evaluations the Office of

Juvenile Justice and Delinquency Prevention (OJJDP) has funded since

1995 (excluding the 11 impact evaluations it has funded of its own

programs discussed in app. VII). For 22 summaries, we used descriptions

of evaluations that were provided to us by OJJDP; for 2 summaries,1

 we

wrote the descriptions based on OJJDP documents. OJJDP categorized the

24 evaluations into the following three groups:

• OJJDP-funded programs: nonimpact evaluations (11).

• Non-OJJDP-funded programs: impact evaluations2 (9).

• Non-OJJDP-funded programs: nonimpact evaluations (4).

The purpose of this evaluation is to test the feasibility and effectiveness of

the OJJDP community assessment center concept in different

environments. Community assessment centers seek to facilitate earlier

and more efficient delivery of prevention and intervention services at the

front end of the juvenile justice system. The evaluation uses a two-phase

process to (1) measure some outcomes at the two enhancement sites, with

quasi-experimental design,3

 and (2) achieve more and better outcome

measures. But, according to OJJDP, implementation and data problems

will limit the effectiveness of the quantitative methods employed. OJJDP

also believes that the attempt to implement a random assignment study at

one project site will probably need to be abandoned. The first phase

covers the four project sites that comprised the Community Assessment

Centers program—two of these sites funded enhancements to existing

 1

 We used OJJDP documents to develop summaries for SafeFutures and Coping With Life

Course.

2

 Impact evaluations assess the extent to which a program causes changes in the desired

direction in the target population.

3

 In a quasi-experimental design, methods other than random assignment are used to create

a comparison group. When randomly assigning individuals to a control group is not

feasible, quasi-experimental impact evaluations can be used to compare the performance

of program participants on various measures to individuals not in the program.

Appendix VIII: Twenty-Four Other

Evaluations Funded by OJJDP

OJJDP-Funded

Programs: 11

Nonimpact

Evaluations

Community Assessment

Centers
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 70 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

programs and the other two funded the planning and implementation of

new programs. The second phase covers the two project sites—one

enhancement and one planning—in which the program is being continued

after the end of the first funding cycle.4

 Many of the evaluation measures

are at the project or community level rather than at the participant level.

The purpose of this evaluation is to examine the viability and effectiveness

of the community-based delinquency prevention model used by grantees in

the Community Prevention Grants Program. The Community Prevention

Grants Program encourages communities to develop comprehensive,

collaborative plans to prevent delinquency. The evaluation focuses on two

main questions: (1) What is the impact of the program on community

planning, service delivery, risk factors, protective factors, and juvenile

problem behaviors? (2) What factors and activities lead to the effective

implementation of the Community Prevention Grants Program model and

to positive program outcomes? This evaluation employs a case study

approach supplemented by a basic profile of communities that are

participating in the program. Case studies are to be implemented in 11

communities in 6 states. Evaluation measures are to be applied at the

project, community, and program levels.

The purpose of this process evaluation5

 is to address the following

questions about the Comprehensive Strategy program: (1) What are the

factors associated with successful Comprehensive Strategy planning and

implementation? (2) To what extent do project sites adhere to the

prescribed Comprehensive Strategy framework? (3) What are the major

implementation challenges program grantees face in implementing the

Comprehensive Strategy? (4) To what extent does the training and

technical assistance provided to project sites help them acquire the

knowledge, skill, and tools necessary to develop the Comprehensive

Strategy? (5) What role should OJJDP play in the future implementation of

the Comprehensive Strategy? The Comprehensive Strategy is OJJDP’s

 4

 In commenting on a draft of this report, the Assistant Attorney General explained that the

other two project sites were not included in the second phase of the evaluation because

they did not receive additional funding. One site is no longer a separate project and the

other project—a planning site—continues to move toward operational status.

5

 A process evaluation uses empirical data to assess delivery of a program and verifies

whether it was delivered as intended to the targeted recipients.

Community Prevention

Grants Program (Title V)

Comprehensive Strategy

for Serious, Violent, and

Chronic Juvenile

Offenders (Comprehensive

Strategy)
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 71 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

approach for addressing juvenile violence and delinquency at the

community, state, and national levels through a systematic plan. It

advocates the use of local planning teams to assess the factors and

influences that put youths at risk of delinquency, determine available

resources, and establish prevention programs to either reduce risk factors

or provide protective factors that buffer juveniles from the impact of risk

factors. This evaluation uses a multilevel design to assess how project

sites implement the Comprehensive Strategy. The evaluation began with

telephone interviews with site coordinators from all 48 project sites; 25 of

the 48 project sites were randomly selected for a stakeholder survey. One

year later, 10 of the 25 project sites are being given a second stakeholder

survey. Subsequently, five sites are to be selected for visits, and intensive

case studies are being done in three cities.

The purpose of this evaluation is to examine (1) community coalitions’

developmental processes from the early planning and adoption stages

through implementation and later stages and (2) the impact of coalitions’

prevention efforts concerning risk and resiliency factors and, to the extent

feasible, alcohol, tobacco, and other drug use. The Drug-Free

Communities Support Program provides grants to community coalitions to

strengthen their efforts to prevent and reduce young people’s illegal use of

drugs, alcohol, and tobacco. The evaluation is studying two cohorts of

program grantees—those that received grants in 1998 or 1999 (cohort 1)

and those that received grants in 2000 (cohort 2). The national evaluation

sample is comprised of a total of 213 grantees. The sample is divided by

years of operation: 1-5 years, 6-9 years, and more than 9 years.

Semiannually, cohort 1 grantees are required to submit progress reports to

OJJDP and the evaluator that include a special section (Part II), which

provides information about the compositions of the coalitions and

outcome data collection. Cohort 2 grantees do not have a Part II reporting

requirement and submit progress reports semiannually only to OJJDP. In

addition, 21 grantees (15 from cohort 1 and 6 from cohort 2) serve as

intensive study sites, where interviews with staff and stakeholders provide

greater detail about coalition development and local program evaluation.

The purpose of this process evaluation is to provide feedback to OJJDP on

the implementation of the Juvenile Accountability Incentive Block Grants

Program. The program encourages states and local jurisdictions to

implement accountability-based programs and services in 54 states and

U.S. territories. The evaluator is surveying state and local practitioners,

policy makers, and grant program administrators about their perceptions

Drug-Free Communities

Support Program

Juvenile Accountability

Incentive Block Grants

Program
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 72 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

and attitudes about the program and its administration. Specifically, the

evaluation focuses on (1) understanding how states and local units of

government plan for and administer program funds and (2) examining the

perceptions of states and local units of government about how well the

program is achieving congressional intent. In addition, in-depth case

studies are conducted at a limited number of sites.

The purpose of this evaluation is to (1) provide feedback to the

Performance-based Standards Project team on improving design and

implementation support to the sites, (2) assist the project team in refining

the Performance-based Standards Project model and in maximizing

responsiveness to the needs of the participants, that is, those who are

implementing the project model, and (3) chronicle the development of the

project and summarize lessons learned. OJJDP established the

Performance-based Standards Project to improve the quality and

conditions of juvenile corrections facilities. Specifically, the project

develops and implements outcome standards and an assessment tool.

Corrections facilities can use both to monitor progress towards meetings

goals in areas of operations, such as health and safety. The evaluation uses

a case study approach. This approach consists of the collection of both

quantitative and qualitative data describing the processes used to

implement the project model in 80 juvenile detention and correctional

facilities across the country. Site visits are made and in-depth case studies

are planned. An all-site survey is distributed to key participants to

determine satisfaction with the supports provided to them in the

implementation of the project model. In addition, the survey seeks the

participants’ assessment of (1) the impact the project has made on

conditions of confinement and management of the facilities and (2) the

overall utility of the project model.

The purpose of this evaluation is to (1) determine the extent to which

replication project sites have been able to conform to the original program

model, and (2) assess the “prosocial” (that is, positive, socially-oriented

behavior) outcomes for mothers and their babies. The Prenatal and Early

Childhood Nurse Home Visitation Program consists of intensive and

comprehensive home visitation by nurses during a woman’s pregnancy

and the first 2 years following the birth of her first child. The evaluation

involves six project sites and employs a quasi-experimental design with

matched comparison groups.

Performance-based

Standards Project

Prenatal and Early

Childhood Nurse Home

Visitation Program
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 73 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

This purpose of this process evaluation is to document and understand the

process of community mobilization and collaboration. SafeFutures is

designed to build a comprehensive program of prevention and intervention

strategies for at-risk youths and juvenile offenders. The program

comprises six project sites that represent urban, rural, and Native

American communities. The evaluation is examining all six project sites.

Project sites collect and record performance data on program operations

and client outcomes using the Client Indicator Data Base. The project sites

are required to collect extensive information from selected SafeFutures

program components on individual participants’ risk and protective factor

profiles, youths’ service utilization, and agencies’ coordination of services

for youth during the course of their involvement in the SafeFutures

program. In addition, they collect information on outcome measures

regarding youths’ educational commitment (that is, school attendance,

achievement, and behavior), youths’ involvement in delinquency and

crime, and any changes in youths’ risk profiles. Analysis of these data will

provide a picture on program performance in three key areas: reaching the

intended high-risk youth clientele, coordinating services for youths with

multiple problems, and monitoring subsequent school performance

problems and involvement in the juvenile justice system.

The purpose of this evaluation is to document the lessons learned and

factors associated with the successful development and implementation of

the Safe Kids/Safe Streets program. The Safe Kids/Safe Streets program is

designed to (1) help communities break the cycle of early childhood

victimization and later criminality and (2) reduce child abuse and neglect,

as well as the child fatalities that often result. The evaluation is surveying

five project sites through five data collection strategies: agency

administrative data, case tracking, key informant interviews, surveys of

agency professionals, and surveys of stakeholders.

The purpose of this evaluation is to (1) support culturally appropriate

process and outcome evaluations of activities funded under Tribal Youth

Program grants and (2) build the capacity of tribes to better evaluate their

own juvenile justice programs and activities. The Tribal Youth Program

assists grantees in developing projects, within tribal communities, for the

prevention and control of youth violence and substance abuse. The

evaluation is participatory in nature, that is, project personnel and

stakeholders will be involved in developing the evaluation designs, with

the assistance and guidance of an evaluation facilitator. The five project

sites are implementing different projects and have not yet completed their

Partnerships to Reduce

Youth Violence and

Delinquency Program

(SafeFutures)

Safe Kids/Safe Streets

Tribal Youth Program
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 74 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

evaluation designs. According to OJJDP, it is too early in the evaluation to

tell exactly what designs are to be used. OJJDP has required that all

evaluations be designed to examine both program implementation and

program outcomes.

The purpose of this process evaluation6

 is to (1) determine how

community collaboration can affect truancy reduction and lead to

systemic reform and (2) assist OJJDP in the development of a model for a

truancy reduction program, including identifying the essential elements of

that model. The Truancy Reduction Demonstration Program encourages

communities to develop comprehensive approaches—involving schools,

parents, the justice system, law enforcement, and social service

agencies—in identifying and tracking truant youths. The evaluation is

employing site visits, interviews with key personnel, and case studies of

individual sites. Process data are gathered from all seven project sites

participating in the evaluation and, from some sites, limited outcome data

are gathered.

The purpose of this evaluation is to evaluate the efficacy of three

Adolescent Female Offenders programs in Wayne County, Michigan. The

three programs are (1) a program incorporating gender-specific

programming, home-based intervention, and community involvement,

including pregnant and parenting adolescents; (2) an intensive probation

program with limited gender-specific programming; and (3) a traditional,

female-only residential program that provides limited gender-specific

training. The evaluation is using a quasi-experimental design. Using

random assignment, the home-based intervention model is to be compared

with the established intensive probation model; the outcomes of these

models are then to be compared with outcomes of the traditional,

 6

 The Truancy Reduction Demonstration Program and its evaluation are jointly funded by

OJJDP, the Office of Justice Program’s Executive Office of Weed and Seed, and the U.S.

Department of Education.

Truancy Reduction

Demonstration Program

Non-OJJDP-Funded

Programs: Nine

Impact Evaluations

Adolescent Female

Offenders (Three

Programs)
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 75 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

female-only residential program. The comparison analysis involves at least

50 young women in each of the 3 programs. A wide range of outcomes—

including recidivism, substance use, depression, community integration,

academic performance and career aspirations, parenting readiness, and

responsible sexual behavior—is to be examined. The evaluator is also

exploring the relationship of specific program components to these

outcomes.

The purpose of this impact evaluation is to evaluate the effectiveness of a

cognitive-behavioral group intervention. The Coping With Life Course is

aimed at enhancing prosocial coping and problem solving for adolescents

incarcerated in youth correctional facilities. To evaluate the program, a

minimum of 120 adolescents in one youth correctional facility are

randomly assigned to either the Coping with Life Course intervention

group or a standard-care control group. Six Coping with Life Course

cohort groups of 10 each are followed. The evaluation is allowing for

attrition (from the initial 60 participants down to 48) in each of the

intervention and control groups. Participant functioning is assessed before

and after intervention through a battery of questionnaires. Recidivism,

return to close custody, and service utilization are tracked through

databases and statewide records.

The purpose of the evaluation is to (1) document the implementation of a

new “family index” case management system (through a process

evaluation) and (2) examine the impact of the family index on juvenile

court case processing (through an impact evaluation). The family index

system allows cross-referencing to identify all family members involved in

family law; juvenile dependency; juvenile delinquency; and criminal, civil,

and probate matters. For the process evaluation, a case study approach is

used to describe the implementation of the family index at one project

site, the Riverside, California, Court. For the impact evaluation, a pre-post

design is used to examine how the family index has affected juvenile court

matters (for example, court processing time, coordination between courts,

and content of hearings).

The purpose of this evaluation is to evaluate the effects of Flashpoint on

the antisocial patterns of juvenile offenders’ thoughts and actions and high

school students’ thoughts and actions. Specifically, changes are assessed

for (1) media use and literacy, (2) violence-supporting beliefs and

behavior, and (3) substance use and abuse. The Flashpoint program is

Coping with Life Course

Creation and

Implementation of a

Family Index in Riverside

County, California, Courts

Flashpoint
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 76 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

designed to build critical thinking skills young people need to (1) see

through false media portrayals that glorify violence and drug use and (2)

apply decision-making in their own lives. The evaluation is using

comparison groups, pre-post interventions, and case studies. Participants

include 264 juveniles, ages 14 to 17. Treatment groups are compared with

no-treatment control groups for baseline-to-posttest changes. Three

groups and project sites are involved: (1) repeat and serious offenders in a

correctional program, (2) first-time offenders in a diversion program, and

(3) students in a public high school.

The purpose of this evaluation is to determine if Free to Grow can reduce

substance abuse (alcohol use, smoking, and illegal drug use). The Free to

Grow program builds on existing Head Start programs, adding communitystrengthening and family-intervention components to address the problem

of substance abuse. The evaluation is attempting to determine the

independent effects of these two components on substance abuse

prevention. It involves 16 project sites and 16 comparison sites and

employs a multistage experimental research design.

The purpose of this evaluation is to provide a process and outcome

evaluation of the Gaining Insight into Relationships for Lifelong Success

Project. The project involves two primary levels of intervention: (1) a

psycho-educational counseling group, dealing with relationships and

involving girls in four relational domains (relation to self, family, peers,

and teachers), and (2) a focus on individual consultations, educational

workshops and the policies and procedures of the local juvenile justice

system, and the involvement of court service workers from the system.

Specifically, the evaluation will (1) investigate the applicability of a

relational approach to the treatment of female juvenile offenders; (2)

examine the components of the relational approach that deal with

relationships to self, family, peers and teachers; (3) evaluate the impact of

increasing the knowledgebase of professionals involved in the local

juvenile justice system; and (4) provide an empirically based, alternative

treatment model that can be replicated in other settings. The evaluation of

the first level of intervention—the counseling group—focuses on each of

the four relational domains through the use of multimethod data

collection; this collection includes self-reports and other reports, school

records, and recidivism data. The evaluation of the second level of

intervention focuses on the court services that workers use, specifically

gender-sensitive treatment recommendations and referrals; qualitative

observational data, gathered from monthly meetings, will be used. There is

Free to Grow: Head Start

Partnerships to Promote

Substance-Free

Communities

Gaining Insight Into

Relationships for Lifelong

Success Project
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 77 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

random assignment between girls referred to either the project

intervention or to the standard intervention currently being used by the

Clark County Court in Athens, Georgia. Approximately 180 girls—90

referred to the project intervention and 90 referred to the standard court

intervention—are to be evaluated.

The purpose of this evaluation is to assess the implementation and the

impact of Quantum Opportunities. The Quantum Opportunities program is

designed to reduce the incidence of delinquency, criminal behavior, and

subsequent involvement in the criminal and juvenile justice systems

amongst educationally at-risk inner city youths. The evaluation is using an

experimental design with random assignment. Ninth-grade students at six

sites are randomly assigned to treatment and control groups, with the

treatment group enrolled in the Quantum Opportunities program. The

students are followed through their high school careers and 2 years

beyond. Information is collected from academic achievement tests,

administered each year, and from two questionnaires.

The purpose of this evaluation is to test the impact—on recidivism,

program completion, and victim satisfaction—of the Restorative Justice

Conferences for a population of youthful offenders (aged 14 and under) in

an urban setting (Indianapolis, IN). Restorative Justice Conferences bring

together the offender, victim, and supporters of each so as to provide an

opportunity for fuller discussion of the offense; the effect of the offense on

the victim, the offender’s family, and greater community; and steps the

offender can take to make amends. The evaluation is using a single-site

evaluation with an experimental design. As part of the design, youths are

randomly assigned to a treatment group (Restorative Justice Conferences)

or a matched control group.

The purpose of this evaluation is to determine if the program reduces the

amount of delinquency in a city. The Risk-Focused Community Policing

program increases protection by the community police, potentially

reducing delinquency. The evaluation is using an experimental research

design. The project site (city) is divided into approximately 40 census

blocks, with 20 blocks randomly selected as program blocks and the other

20 designated as control blocks.

Quantum Opportunities

Restorative Justice

Conferences

Risk-Focused Community

Policing
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 78 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

The purpose of this evaluation is to study the Act Now Truancy Program.

The program is a prosecutor-led truancy reduction program. The

evaluation is using a pre-post intervention design involving one project

site. Information is collected and aggregated (for example, truancy rates

rather than individual truancy behavior) for all participants.

The purpose of this evaluation, conducted in two schools, is to assess the

impact of the Childhood Violence Prevention Program. The program is

designed to prevent the legitimization of aggression among pre-adolescent,

elementary, and middle school children, with special focus on victims of

child maltreatment. The evaluation is using a pre-post intervention design,

with comparison groups. The study involves having elementary school

students participate in a class activity using a workbook designed to

encourage problem solving action rather than aggressive behavior in

interactions with peers.

This project is not an evaluation, per se, but rather a synthesis of existing

evidence on community-level interventions and service programs. Its

purpose is to identify the strengths and weaknesses of community-level

evaluations and to provide recommendations to the field about how to

structure and carry out such evaluations. Community-level programs for

youths are designed to promote positive youth development. To evaluate

the programs, a committee—experts from several disciplines (child and

adolescent development, child health, sociology, psychology, evaluation

research, youth services, and community development)—is assessing the

strengths and limitations of measurements and methodologies that have

been used to evaluate these interventions.

Non-OJJDP-Funded

Programs: Four

Nonimpact

Evaluations

Act Now Truancy Program

Childhood Violence

Prevention Program (Let’s

Talk About...)

Community-Level

Programs for Youth
Appendix VIII: Twenty-Four

Other Evaluations Funded by

OJJDP

Page 79 GAO-02-23 OJJDP Grantee Reporting Requirements and Conce

The purpose of this evaluation is to assess the activities undertaken by

project sites, determine whether they can be evaluated, and ultimately

assess the impact of these activities on the youthful offenders participating

in the program. The program is intended to (1) enhance school-to-work

education and training in juvenile correctional facilities and (2) improve

youthful offenders’ transition into the community. The evaluation design

has not been completed, but random assignment study is strongly

preferred, if feasible. At the time of our review, only one of the three

potential sites could be evaluated. One more project site is to be awarded

and, if it can be evaluated, it will be added as a second evaluation site.

Department of Labor’s

Education and Training for

Youthful Offenders

Initiative
Appendix IX: Comments from the Department

of Justice

Page 80 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Appendix IX: Comments from the

Department of Justice

Note: GAO comments

supplementing those in

the report text appear at

the end of this appendix.
Appendix IX: Comments from the Department

of Justice

Page 81 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on pp. 23-24.

Now on pp. 3, 8-10, and

22-25.
Appendix IX: Comments from the Department

of Justice

Page 82 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department

of Justice

Page 83 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on p. 10.

Now on p. 9.

Now on p. 6.

Now on p. 4.

Now on p. 21.
Appendix IX: Comments from the Department

of Justice

Page 84 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

See comment 1.

Now on p. 22.

Now on pp. 10-18.
Appendix IX: Comments from the Department

of Justice

Page 85 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department

of Justice

Page 86 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on p. 25.
Appendix IX: Comments from the Department

of Justice

Page 87 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on p. 31.

Now on p. 26.

Now on p. 26.

Now on pp. 25-26.

See comment 2.

Now on p. 25.
Appendix IX: Comments from the Department

of Justice

Page 88 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns
Appendix IX: Comments from the Department

of Justice

Page 89 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on pp. 48-51

Now on p. 48.

Now on p. 36, table 6.

Now on p. 35.

Now on p. 35.
Appendix IX: Comments from the Department

of Justice

Page 90 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on p. 66.

Now on p. 57-58.

Now on p. 59.

Now on p. 53, tables 10

and 11.

See comment 3.

Now on p. 51.

Now on p. 51.
Appendix IX: Comments from the Department

of Justice

Page 91 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now on p. 73.

Now on p. 71.

Now on p. 70.

Now on p. 64.

See comment 4.
Appendix IX: Comments from the Department

of Justice

Page 92 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Now in app. VII, p. 68.
Appendix IX: Comments from the Department

of Justice

Page 93 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

The following are GAO’s comments on the Department of Justice’s

October 15, 2001, letter.

1. As we indicated in our report, impact evaluations, such as the types

that OJJDP is funding, can encounter difficult design and

implementation challenges. (See section titled, Evaluations of OJJDP

Programs are Difficult to Successfully Design and Implement.) Also,

we are aware that virtually all impact evaluations have limitations.

However, where possible, impact evaluations should be designed to

mitigate as many rival explanations of program effects as feasible, and

potential limitations of the chosen research design should be

acknowledged.

2. Our statement that the Juvenile Mentoring Program evaluation “has

experienced problems obtaining behavioral measures and school

performance data” was not intended to criticize the evaluators’ level of

effort, but rather to indicate that their inability to obtain data from

school and law enforcement officials in many of the study sites makes

it more difficult to evaluate how well the program is achieving its

objectives of diminishing delinquency, gang involvement, and school

failure. While enhanced analysis of sites with the best data may be

warranted, it does not overcome the problem of having a large number

of sites with little or no reliable data from school and law enforcement

officials. This problem was explicitly recognized by OJJDP in its

November 2000 report.

3. During the course of our review, OJJDP officials told us that one

measure of a research grantee’s performance is the number of

products the grantee has published. These officials provided us a

listing of all products published by active research grantees through

OJJDP and private publishers as a direct result of OJJDP-funded

research. We summarized these voluminous data by topic to facilitate

the presentation.

4. Our report points out that the Enforcing the Underage Drinking Laws

Program evaluation documents OJJDP provided to us were not clear

on whether the sites chosen were representative. Our report does not

suggest “there is no way of achieving a legitimate representative

sample.” However, we agree with OJJDP that the evaluation may not

be evaluating the Enforcing the Underage Drinking Laws Program

because there may be no program components common to all project

cities. The Assistant Attorney General states that the evaluation will be

GAO Comments
Appendix IX: Comments from the Department

of Justice

Page 94 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

able to measure impacts on several program areas across each site.

However, our point is that the evaluator’s plan to aggregate data across

sites may be inappropriate because wide variation allowed by the

program means that program activities are not common across all

sites. Therefore, interpreting and generalizing results may be

problematic.
Appendix X: GAO Contacts and Staff

Acknowledgments

Page 95 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Laurie E. Ekstrand, (202) 512-8777

James M. Blume, (202) 512-8777

In addition to the above, Lori A. Weiss, Barbara A. Guffy, Michele J. Tong,

Leslie C. Bharadwaja, David P. Alexander, Douglas M. Sloane, Shana B.

Wallace, Michele C. Fejfar, Charity J. Goodman, and Jerome T. Sandau

made key contributions to this report.

Appendix X: GAO Contacts and Staff

Acknowledgments

GAO Contacts

Acknowledgments
Bibliography

Page 96 GAO-02-23 OJJDP Grantee Reporting Requirements and Concerns

Howell, J.C. Youth Gang Programs and Strategies. U.S. Department of

Justice, Office of Justice Programs, Office of Juvenile Justice and

Delinquency Prevention. Washington, D.C.: Aug. 2000.

Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. “Juvenile

Mentoring Program: A Progress Review.” Juvenile Justice Bulletin, Sept.

2000.

Sheppard, D., H. Grant, W. Rowe, and N. Jacobs. “Fighting Juvenile Gun

Violence.” Juvenile Justice Bulletin, Sept. 2000.

U.S. Department of Justice, Office of Justice Programs, National Institute

of Justice. “Reintegrating Juvenile Offenders Into the Community: OJJDP’s

Intensive Community-Based Aftercare Demonstration Program.” Research

Preview, Dec. 1998.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile

Justice and Delinquency Prevention. FY 2000 OJJDP Discretionary

Program Announcement: Juvenile Mentoring Program. Mar. 2000.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile

Justice and Delinquency Prevention. Gang-Free Schools and

Communities Initiative: FY 2000 OJJDP Discretionary Program

Announcement. July 2000.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile

Justice and Delinquency Prevention. Juvenile Mentoring Program: 1998

Report to the Congress. Washington, D.C.: Dec. 1998.

U.S. Department of Justice, Office of Justice Programs, Office of Juvenile

Justice and Delinquency Prevention. OJJDP Research 2000. Washington,

D.C.: May 2001.

Wiebush, R. G., B. McNulty, and T. Le. “Implementation of the Intensive

Community-Based Aftercare Program.” Juvenile Justice Bulletin, July

2000.

Bibliography

(185012)
The first copy of each GAO report is free. Additional copies of reports are

$2 each. A check or money order should be made out to the

Superintendent of Documents. VISA and MasterCard credit cards are also

accepted.

Orders for 100 or more copies to be mailed to a single address are

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

Orders by visiting:

Room 1100

700 4th St., NW (corner of 4th and G Sts. NW)

Washington, DC 20013

Orders by phone:

(202) 512-6000

fax: (202) 512-6061

TDD (202) 512-2537

Each day, GAO issues a list of newly available reports and testimony. To

receive facsimile copies of the daily list or any list from the past 30 days,

please call (202) 512-6000 using a touchtone phone. A recorded menu will

provide information on how to obtain these lists.

Orders by Internet

For information on how to access GAO reports on the Internet, send an email message with “info” in the body to:

Info@www.gao.gov

or visit GAO’s World Wide Web home page at:

http://www.gao.gov

Contact one:

• Web site: http://www.gao.gov/fraudnet/fraudnet.htm

• E-mail: fraudnet@gao.gov

• 1-800-424-5454 (automated answering system)

Ordering Information

To Report Fraud,

Waste, and Abuse in

Federal Programs