Drug Courts: Better DOJ Data Collection and Evaluation Efforts Needed to Measure Impact of Drug Court Programs
Released: 2002-04-22T00:00:00

Letter

Background

Since 1989, when the first drug court program was established, the number of drug court programs has increased substantially. In addition, DCPO’s oversight responsibilities and funding to support the planning, implementation, and enhancement of these programs have increased.

As shown in figure 1, the number of operating drug court programs has more than tripled since our prior report from about 250 in 1997 to almost 800 in 2001 based on information available as of December 31, 2001.

The number of operating programs that received DCPO funding, and thus were subject to its oversight, has also grown—from over 150 in fiscal year 1997 to over 560 through fiscal year 2001.

As shown in figure 2, the number of drug court programs started by calendar year since our prior report has also increased. Although the number of drug court programs started in 2001 dropped, over 450 additional programs have been identified as being planned based on information available as of December 31, 2001.

Based on information available as of December 31, 2001, drug court programs were operating in 48 states, the District of Columbia, and Puerto Rico. Only New Hampshire and Vermont had no operating drug court programs. Six states (California, Florida, Louisiana, Missouri, New York, and Ohio) accounted for over 40 percent of the programs. Appendix II provides information on the number of operating drug court programs in each state.

Although there are basic elements common to many drug court programs, the programs vary in terms of approaches used, participant eligibility and program requirements, type of treatment provided, sanctions and rewards, and other practices. Drug court programs also target various populations (adults, juveniles, families, and Native American tribes). Appendix III provides details on the number of drug court programs by targeted population, and appendix IV provides details on the drug court programs by jurisdiction and the types of funding, if any, the programs have received from DCPO.

Federal funding for drug court programs has also continued to increase. As shown in table 1, congressional appropriations for the implementation of DOJ’s drug court program has increased from about $12 million in fiscal year 1995 to $50 million in fiscal years 2001 and 2002. Since fiscal year 1995, Congress has appropriated about $267 million in Violent Crime Act related funding to DOJ for the federal drug court program. DCPO funding in direct support of drug court programs has increased from an average of about $9 million in fiscal years 1995 and 1996 to an average of about $31 million for fiscal years 1997 through 2001. Between fiscal years 1995 and 2001, DCPO has awarded about $174.5 million in grants to fund the planning, implementation, and enhancement of drug court programs. About $21.5 million in technical assistance, training, and evaluations grants were awarded. About $19.6 million were obligated for management and administration purposes and to fund nongrant technical assistance, training, and evaluation efforts. Since the inception of the DCPO drug court program, a total of $3 million in prior year recoveries have been realized. About $4.5 million through fiscal year 2001 had not been obligated. Congress appropriated an additional $50 million for fiscal year 2002. At the time of our review, DCPO was in the process of administering the fiscal year 2002 grant award program.

Appendix V provides details on the number, amount, and types of grants DCPO awarded since the implementation of the federal drug court program.

DOJ Has Not Sufficiently Managed the Collection and Utilization of Performance and Outcome Data Collected from Federally Funded Drug Court Programs

Since 1998, DCPO implementation and enhancement grantees have been required to collect, and starting in 1999, to submit to DCPO, among other things, performance and outcome data on program participants. DCPO collects these data semiannually using a Drug Court Grantee Data Collection Survey. This survey was designed by DCPO to ensure that grantees were collecting critical information about their drug court programs and to assist in the national evaluation of drug court programs. In addition, DOJ intended to use the information to respond to inquiries regarding the effectiveness of drug court programs. However, due to various factors, DCPO has not sufficiently managed the collection and utilization of these data. As a result, DOJ cannot provide Congress, drug court program stakeholders, and others with reliable information on the performance and impact of federally funded drug court programs.

Factors Contributing to Insufficiencies in DOJ’s Management

Various factors contributed to insufficiencies in DOJ’s drug court program data collection effort. These factors included (1) inability of DOJ to readily identify the universe of DCPO-funded drug court programs, including those subject to DCPO’s data collection reporting requirements; (2) inability of DOJ to accurately determine the number of drug court programs that responded to DCPO’s semiannual data collection survey; (3) inefficiencies in the administration of DCPO’s semiannual data collection effort; (4) the elimination of post-program impact questions from the scope of DCPO’s data collection survey effort; and (5) the insufficient use of the Drug Court Clearinghouse.

DOJ Has Been Unable to Readily Identify the Universe of Drug Court Programs It Has Funded

DOJ’s grant management information system, among other things, tracks the number and dollar amount of grants the agency has awarded to state and local jurisdictions and Native American tribes to plan, implement, and enhance drug court programs. This system, however, is unable to readily identify the actual number of drug court programs DCPO has funded. Specifically, the system does not contain a unique drug court program identifier, does not track grants awarded to a single grantee but used for more than one drug court program, and contains data entry errors that impact the reliability of data on the type of grants awarded. For example, at the time of our review, the system contained some incorrectly assigned grant numbers, did not always identify the type of grant awarded, and incorrectly identified several grantees as receiving a planning, implementation, and enhancement grant in fiscal year 2000. These factors made it difficult for DCPO to readily produce an accurate universe of the drug court programs that had received DCPO funding and were subject to DCPO’s data collection reporting requirement.

Although DOJ has been able to provide information to enable an estimate of the universe of DCPO-funded drug court programs to be derived, the accuracy of this information is questionable because DCPO has relied on the Drug Court Clearinghouse to determine the number of DCPO-funded drug court programs and their program implementation dates. One of the Drug Court Clearinghouse’s functions has been to identify DCPO-funded drug court programs. However, the Drug Court Clearinghouse has only been tasked since 1998 with following up with a segment of DCPO grantees to determine their implementation date. Thus, the information provided to DCPO on the universe of DCPO-funded drug court programs is at best an estimate and not a precise count of DCPO drug court program grantees. Noting that its current grant information system was not intended to readily identify and track the number of DCPO-funded drug court programs, DCPO officials said that they plan to develop a new management information system that will enable DOJ to do so. Without an accurate universe of DCPO-funded drug court programs, DCPO is unable to readily determine the actual number of programs or participants it has funded or, as discussed below, the drug court programs that should have responded to its semiannual data collection survey.

DOJ Has Been Unable to Accurately Determine Response Rates for Its Data Collection Survey

According to DCPO officials, grantee response rates to DCPO’s semiannual survey have declined since DCPO began administering the survey in 1998. As shown in figure 3, the information in DCPO’s database indicated that grantee response rates declined from about 78 percent for the first survey reporting period (July to Dec. 1998) to about 32 percent for the July to December 2000 reporting period. However, results from our follow-up structured interviews with a representative sample of the identifiable universe of drug court programs that were DCPO grantees during the 2000 reporting periods revealed that DCPO did not have an accurate account of grantees’ compliance with its semiannual data collection survey.

Based on our structured interviews, we estimate that the response rate to the DCPO data collection survey for the January to June 2000 reporting period was about 60 percent in contrast to the 39 percent response rate DCPO reported. Similarly, the response rate to the DCPO survey for the July to December 2000 reporting period was about 61 percent in contrast to the 32 percent response rate DCPO reported. The remaining programs did not respond or were uncertain as to whether they responded to DCPO’s data collection survey for each of the reporting periods in 2000. DOJ officials said that some of the surveys they did not receive may have been mailed to an incorrect office within DOJ. DCPO officials acknowledged that this type of error could be mitigated if DCPO routinely followed up with the drug court programs from which they did not receive responses.

Furthermore, based on our follow-up structured interviews with a representative sample of DCPO-funded drug court programs that were listed as nonrespondents in DCPO’s database, we estimate that about 61 percent had actually responded to DCPO’s survey for the January to June 2000 reporting period. About two-thirds of these programs could produce evidence that they responded. For the July to December 2000 reporting period, we estimate that about 51 percent of the DCPO-funded drug court programs that were listed as nonrespondents in DCPO’s database had actually responded to the survey. About two-thirds of these programs could produce evidence that they responded.

Inefficiencies Existed in the Administration of DCPO’s Semiannual Data Collection Effort

The requirement for grantees to submit DCPO’s semiannual survey is outlined in DOJ’s grant award notification letter that drug court program grantees receive at the beginning of their grant period. In addition, the survey is made available in the grantee application kit as well as on DCPO’s website. However, other than these steps, DCPO has not consistently notified its drug court program grantees of the semiannual reporting requirements nor has it routinely forwarded the survey to grantees. At the time of our review, DCPO had taken limited action to improve grantees’ compliance with the data collection survey requirements. DCPO officials said that they generally had not followed up with drug court program grantees that did not respond to the survey and had not taken action towards the grantees that did not respond to the semiannual data collection reporting requirement. Results from our follow-up structured interviews showed that DCPO had not followed up to request completed surveys from about 70 percent of the drug court program grantees that were nonrespondents during the January to June 2000 reporting period and from about 76 percent of the nonrespondents for the July to December 2000 reporting period.

DCPO has had other difficulties managing its data collection effort. Specifically, (1) DCPO inadvertently instructed drug court program grantees not to respond to questions about program participants’ criminal recidivism while in the program; (2) confusion existed between DCPO and its contractor, assigned responsibility for the semiannual data collection effort, over who would administer DCPO’s data collection survey during various reporting periods; and (3) some grantees were using different versions of DOJ’s survey instruments to respond to the semiannual data collection reporting requirement.

DCPO Eliminated Post-Program Data from Its Data Collection Effort

The overall success of a drug court programs is dependent on whether defendants in the program stay off drugs and do not commit more crimes when they complete the program. In our 1997 report we recommended that drug court programs funded by discretionary grants administered by DOJ collect and maintain follow-up data on program participants’ criminal recidivism and, to the extent feasible, follow-up data on drug use relapse. In 1998, DCPO required its implementation and enhancement grantees to collect and provide performance and outcome data on program participants, including data on participants’ criminal recidivism and substance abuse relapse after they have left the program. However, in 2000, DCPO revised its survey and eliminated the questions that were intended to collect post-program outcome data.

The DCPO Director said that DCPO’s decision was based on, among other things, drug court program grantees indicating that they were not able to provide post-program outcome data and that they lacked sufficient resources to collect such data. DCPO, however, was unable to produce specific evidence from grantees (i.e., written correspondence) that cited difficulties with providing post-program outcome data. The Director said that difficulties have generally been conveyed by grantees, in person, through telephone conversations, or are evidenced by the lack of responses to the post-program questions on the survey.

Contrary to DCPO’s position, evidence exists that supports the feasibility of collecting post-program performance and outcome data. During our 1997 survey of the drug court programs, 53 percent of the respondents said that they maintained follow-up data on participants’ rearrest or conviction for a nondrug crime. Thirty-three percent said that they maintained follow-up data on participants’ substance abuse relapse.

Recent information collected from DCPO grantees continues to support the feasibility of collecting post-program performance and outcome data. The results of structured interviews we conducted in the year 2001 with a representative sample of DCPO-funded drug court programs showed that an estimated two-thirds of the DCPO-funded drug court programs maintained criminal recidivism data on participants after they left the program. About 84 percent of these programs maintained such data for 6 months or more. Of the remaining one-third that did not maintain post- program recidivism data, it would be feasible for about 63 percent to provide such data. These estimates suggest that about 86 percent of DCPO- funded drug court programs would be able to provide post-program recidivism data if requested.

The results of structured interviews we conducted in the year 2001 with a representative sample of DCPO-funded drug court programs also showed that about one-third of the DCPO-funded drug court programs maintained substance abuse relapse data on participants after they have left the program. About 84 percent of these programs maintained such data for 6 months or more. Of the estimated two-thirds that did not maintain post- program substance abuse relapse data, it would be feasible for about 30 percent to provide such data. These estimates suggest that about 50 percent of DCPO-funded drug court programs would be able to provide post-program substance abuse data if requested.

According to survey results collected by the Drug Court Clearinghouse in 2000 and 2001, a significant number of the drug court programs were able to provide post-program outcome data. For example, about 47 percent of the DCPO-funded adult drug court programs that responded to the Drug Court Clearinghouse’s 2000 operational survey reported that they maintained some type of follow-up data on program participants after they have left the program. Of these drug court programs, about 92 percent said that they maintained follow-up data on recidivism and about 45 percent said that they maintained follow-up data on drug usage.

Of the DCPO-funded adult and juvenile drug court programs operating for at least a year that responded to the Drug Court Clearinghouse’s annual survey that was published in 2001, about 56 percent were able to provide follow-up data on program graduates’ recidivism and about 55 percent were able to provide follow-up data on program graduates’ drug use relapse.

DCPO Has Not Sufficiently Utilized the Drug Court Clearinghouse’s Data Collection Efforts

Operating under a cooperative agreement with DCPO, the Drug Court Clearinghouse has successfully collected performance and outcome data through an annual survey of all operating adult, juvenile, family, and tribal drug court programs, including those funded by DCPO. In addition, as previously noted, the Drug Court Clearinghouse has generally administered an operational survey to adult drug court programs every 3 years, including those funded by DCPO. The Drug Court Clearinghouse annually disseminates the results from its annual survey and has periodically published comprehensive drug court survey reports that provide detailed operational, demographic, and outcome data on the adult drug court programs identified through its data collection efforts. Although funded by DOJ, the Drug Court Clearinghouse has not been required to primarily collect and report separately on the universe of DCPO-funded programs. In addition, no comprehensive or representative report has been produced by DCPO or the Drug Court Clearinghouse that focuses primarily on the performance and outcome of DCPO-funded drug court programs. Instead, DCPO instructed the Drug Court Clearinghouse, in July 2001, to eliminate recidivism data from its survey publications. Although the Drug Court Clearinghouse has developed and implemented survey instruments to periodically collect and disseminate recidivism and relapse data, the DCPO Director had concerns with the quality of the self-reported data collected and the inconsistent time frames for which post-program data were being collected by drug court programs.

DOJ’s Effort to Complete a National Impact Evaluation of DCPO-Funded Drug Court Programs Has Fallen Short of Its Objective

In response to recommendations in our 1997 report, DOJ undertook, through NIJ, an effort to conduct a two-phase national impact evaluation focusing on 14 selected DCPO-funded drug court programs. This effort was intended to include post-program data within its scope and to involve the use of nonparticipant comparison groups. However, various administrative and research factors hampered DOJ’s ability to complete the NIJ-sponsored national impact evaluation, which was originally to be completed by June 30, 2001. As a result, DOJ fell short of its objective, discontinued this effort, and is considering an alternative study that, if implemented, is not expected to provide information on the impact of federally funded drug court programs until year 2007. Unless DOJ takes interim steps to evaluate the impact of drug court programs, the Congress, the public, and other drug court stakeholders will not have sufficient information in the near term to assess the overall impact of federally funded drug court programs.

The Objectives of DOJ’s National Evaluation Effort

The overall objective of the NIJ-sponsored national evaluation was to study the impact of DCPO-funded drug court programs using comparison groups and studying, among other things, criminal recidivism and drug use relapse. This effort was to be undertaken in two phases and to include the collection of post-program outcome data. The objectives for phase I, for which NIJ awarded a grant to RAND in August 1998, were to (1) develop a conceptual framework for evaluating the 14 DCPO-funded drug court programs, (2) provide a description of the implementation of each program, (3) determine the feasibility of including each of these 14 drug court programs in a national impact evaluation, and (4) develop a viable design strategy for evaluating program impact and the success of the 14 drug court programs. The design strategy was to be presented in the form of a written proposal for a supplemental noncompetitive phase II grant.

The actual impact evaluation and an assessment of the success of the drug court programs were to be completed during phase II of the study using a design strategy resulting from phase I.

Various Administrative and Research Factors Hampered Completion of the National Impact Evaluation

NIJ’s two-phase national impact evaluation was originally planned for completion by June 30, 2001. Phase I was awarded for up to 24 months and was scheduled to conclude no later than June 30, 2000. However phase I was not completed until September 2001—15 months after the original project due date. Phase II, which NIJ expected to award after the satisfactory submission of a viable design strategy for completing an impact evaluation, has since been discontinued. Various administrative and research factors contributed to delays in the completion of phase I and DOJ’s subsequent decision to discontinue the evaluation. The factors included (1) DCPO’s delay in notifying its grantees of RAND’s plans to conduct site visits; (2) RAND’s lateness in meeting task milestones; (3) NIJ’s multiple grant extensions to RAND that extended the timeframe for completing phase I and further delayed NIJ’s subsequent decision to discontinue phase II; and (4) the inability of the phase I efforts to produce a viable design strategy that was to be used to complete a national impact evaluation in phase II.

Administrative Delay in Notifying Grantees

Phase I of the NIJ-sponsored study was initially hampered by DCPO’s delay in notifying its grantees of plans to conduct the national impact evaluation. In November 1998, DCPO agreed to write a letter notifying its grantees of RAND’s plan to conduct the national evaluation. The notification letters were sent in March 1999. As a result, drug court program site visits, which RAND had originally planned to complete by February 1999, were not completed until July 1999.

Lateness in Meeting Task Milestones

Although RAND completed most of the tasks associated with the national evaluation phase I objectives, it was generally late in meeting task milestones. The conceptual framework for the evaluation of 14 DCPO- funded drug court programs, which RAND was originally scheduled to complete by September 1999, was submitted to NIJ in May 2000—8 months after the original task milestone. This timeframe, according to RAND, was impacted by the delay in DOJ’s initiation of site visits. NIJ officials said that RAND also did not deliver a complete description and analysis of drug court implementation issues to NIJ, which was also due in September 1999, until it received the first draft of RAND’s report in March 2001. The feasibility study, which was originally scheduled to be completed by RAND in September 1999, was provided to NIJ in November 1999. This study informed NIJ of RAND’s concerns with the evaluability of some of the 14 selected DCPO sites. The viable design strategy proposal for evaluating program impact at each of the 14 drug court programs, which RAND was originally expected to complete by May 1999, was not completed. In addition, as discussed below and detailed in appendix VI, RAND was consistently late in meeting the extended milestones for delivery of the final product for phase I.

Multiple Grant Extensions

Although RAND raised concerns in November 1999 regarding the feasibility of completing a national impact evaluation at some of the 14 selected DCPO sites, NIJ continued to grant multiple no-cost extensions that further extended the completion of phase I. The first no-cost grant extension called for phase I of the project to end by September 30, 2000; the second no-cost extension called for phase I to end by December 31, 2000; and the final extension authorized completion of phase I by May 31, 2001. Despite the multiple extensions and RAND’s repeated assurances that the phase I report was imminent, a final phase I report was not completed until September 18, 2001—21 months after the original milestone for completion of phase I. NIJ officials said that, in retrospect, they should have discontinued this effort sooner. Appendix VI provides additional details on the phase I delays in the NIJ-sponsored effort to complete a national impact evaluation.

Lack of a Viable Design Strategy

Phase I of the NIJ-sponsored national impact evaluation did not produce a viable design strategy that would enable an impact evaluation to be completed during phase II using the selected DCPO-funded drug court programs. RAND did offer an alternative approach. However, this approach did not address the original objective—to conduct a national impact evaluation. During its feasibility study, RAND rated the evaluability of the 14 program sites as follows: 4 - poor or neutral/poor, 5 - neutral, and 5 - neutral/good or good. In response, NIJ and DCPO asked RAND to consider completing the evaluation using those DCPO-funded program sites that were deemed somewhat feasible. RAND, however, was not receptive to this suggestion and did not produce a viable design strategy based on the 14 DCPO-funded programs or the subset of DCPO-funded programs that were deemed feasible to use in phase II to evaluate the impact of federally funded drug court programs. As a result, DOJ continues to lack a design strategy for conducting a national impact to enable it to address the impact of federally funded drug court programs in the near term.

DOJ’s Alternative Plan for Completing a National Evaluation Will Not Provide Near-Term Answers

To address the need for the completion of a national impact evaluation, DCPO and NIJ are considering plans to complete a longitudinal study of drug-involved offenders in up to 10 drug court program jurisdictions. The DCPO Director said that the study would be done at a national level, and the scope would include comparison groups and the collection of individual level and post-program recidivism data. DOJ expects that this project, which is in its formative stage, if implemented, will take up to 4 years to complete—with results likely in year 2007. We recognize that it would take time to design and implement a rigorous longitudinal evaluation study and that if properly implemented, such an effort should better enable DOJ to provide information on the overall impact of federally funded drug court programs. However, its year 2007 completion timeframe will not enable DOJ to provide the Congress and other stakeholders with near-term information on the overall impact of federally funded drug court programs that has been lacking for nearly a decade.

Conclusions

Despite a significant increase in the number of drug court programs funded by DCPO since 1997 that are required to collect and maintain performance and outcome data, DOJ continues to lack vital information on the overall impact of federally funded drug court programs. Furthermore, the agency’s alternative plan for addressing the impact of federally funded drug court programs will not offer near-term answers on the overall impact of these programs. Improvements in DCPO’s management of the collection and utilization of performance and outcome data from federally funded drug court programs are needed. Additionally, more immediate steps from NIJ and DCPO to carry out a methodologically sound national impact evaluation could better enable DOJ to provide Congress and other drug court program stakeholders with more timely information on the overall impact of federally funded drug court programs. Until DOJ takes such actions, the Congress, public, and other stakeholders will continue to lack sufficient information to (1) measure long-term program benefits, if any; (2) assess the impact of federally funded drug court programs on the criminal behavior of substance abuse offenders; or (3) assess whether drug court programs are an effective use of federal funds.

Recommendations for Executive Action

To improve the Department of Justice’s collection of data on the performance and impact of federally funded drug court programs, we recommend that the Attorney General develop and implement a management information system that is able to track and readily identify the universe of drug court programs funded by DCPO; take steps to ensure and sustain an adequate grantee response rate to DCPO’s data collection efforts by improving efforts to notify and remind grantees of their reporting requirements; take corrective action towards grantees who do not comply with DOJ’s data collection reporting requirements; reinstate the collection of post-program data in DCPO’s data collection effort, selectively spot checking grantee responses to ensure accurate reporting; analyze performance and outcome data collected from grantees and report annually on the results; and consolidate the multiple DOJ-funded drug court program-related data collection efforts to better ensure that the primary focus is on the collection and reporting of data on DCPO-funded drug court programs.

To better ensure that needed information on the impact of federally funded drug court programs is made available to the Congress, public, and other drug court stakeholders as early as possible, we also recommend that the Attorney General take immediate steps to accelerate the funding and implementation of a methodologically sound national impact evaluation and to consider ways to reduce the time needed to provide information on the overall impact of federally funded drug court programs. Furthermore, we recommend that steps be taken to implement appropriate oversight of this evaluation effort to ensure that it is well designed and executed, and remains on schedule.

Agency Comments and Our Evaluation

We requested comments on a draft of this report from the Attorney General. We also requested comments from RAND on a section of the draft report pertaining to its efforts to complete phase I of NIJ’s national evaluation effort.

On April 3, 2002, DOJ provided written comments on the draft report (see app. VII). The Assistant Attorney General for the Office of Justice Programs noted that we made several valuable recommendations for improving the collection of data on the performance and impact of federally funded drug court programs and outlined steps DOJ is considering to address two of the six recommendations we make for improving its collection of data on the performance and impact of federally funded drug court programs. However, concerning the remaining four recommendations for improving DOJ’s data collection effort, DOJ does not specifically outline any plans (1) for taking corrective action towards grantees who do not comply with DCPO’s data collection reporting requirements; (2) to reinstate the collection of post program data in DCPO’s data collection effort, despite the evidence cited in our report supporting the feasibility of collecting post program data; (3) to analyze and report results on the performance and outcome of DCPO grantees; and (4) to consolidate the multiple DOJ-funded drug court program-related data collection efforts to ensure that the primary focus of any future efforts is on the collection and reporting of data on DCPO-funded programs.

Although DOJ points out in its comments that a number of individual program evaluation studies have been completed, no national impact evaluation of these programs has been done to date. We continue to believe that until post-program follow-up data on program participants are collected across a broad range of programs and also included within the scope of future program and impact evaluations (including nonprogram participant data), it will not be possible to reach firm conclusions about whether drug court programs are an effective use of federal funds or whether different types of drug court program structures funded by DCPO work better than others. Also, unless these results are compared with those on the impact of other criminal justice programs, it will not be clear whether drug court programs are more or less effective than other criminal justice programs. As such, these limitations have prevented firm conclusions from being drawn on the overall impact of federally funded drug court programs.

With respect to our recommendations for improving DOJ’s drug court program-related impact evaluation efforts, DOJ, in its comments, outlines steps it is taking to complete a multisite impact evaluation and its plans to monitor the progress of this effort and to provide interim information during various intervals. As discussed on page 18 of this report, this effort is intended to be done at a national level, and the scope is to include comparison groups and the collection of individual-level and post-program recidivism data.

On April 1, 2002, RAND provided written comments on the segment of the draft report relating to DOJ’s efforts to complete a national impact evaluation (see app. VIII). In its comments, RAND, as we do in our report, acknowledges the need for improvements in the data collection infrastructure for DCPO-funded drug court programs. RAND notes its rationale for why it views the deliverables associated with phase I of the NIJ-sponsored national impact evaluation as being timely and notes that researchers generally have discretion to revise timelines and scopes of work, with the agreement of the client. However, as we point out in our report (pp. 17-18 and app. VI), RAND requested several no-cost extensions to complete the deliverables for various task milestones and did not produce a viable design strategy for addressing the impact of DCPO-funded drug court programs. In addition, NIJ officials said that RAND also did not deliver a complete description and analysis of drug court implementation issues to NIJ until it received the first draft of RAND’s report in March 2001. The deliverable RAND refers to in its comment letter was a paper that RAND had prepared for the National Institute on Drug Abuse, which NIJ never considered to be a product under the grant to evaluate the impact of DCPO-funded drug court programs. As we also pointed out in our report (p. 17 and app. VI), NIJ was not amenable to RAND changing the scope or methodology of the national impact evaluation effort. In addition, RAND commented that a “simple” evaluation design was expected. NIJ’s original objective, however, never called for a simple evaluation design, but rather a viable design strategy involving the use of comparison groups and the collection of post-program data.

We conducted our work at DOJ headquarters in Washington, D.C., between March 2001 and February 2002 in accordance with generally accepted government auditing standards.

As agreed with your offices, unless you publicly release its contents earlier, we plan no further distribution of this report until 30 days from its issue date. At that time, we will provide copies of this report to the Attorney General, the Director of the Office of Management and Budget, and other interested parties. We will also make copies available to others upon request.

If you or your staff have any questions about this report, please contact Daniel C. Harris or me at (202) 512-2758 or at ekstrandl@gao.gov. Key contributors to this report are acknowledged in appendix IX.

Objectives, Scope, and Methodology

Our overall objective for this review was to assess how well the Department of Justice (DOJ) has implemented efforts to collect performance and impact data on federally funded drug court programs. We specifically focused on DOJ’s (1) Drug Courts Program Office’s (DCPO) efforts to collect performance and outcome data from federally funded drug court programs and (2) National Institute of Justice’s (NIJ) efforts to complete a national impact evaluation of federally funded drug court programs.

While there are drug court programs that receive funds from other federal sources, our review focused on those programs receiving federal funds from DCPO, which is DOJ’s component responsible for administering the federal drug court program under the Violent Crime Act. The scope of our work was limited to (1) identifying the processes DCPO used to implement its semiannual data collection effort; (2) determining DCPO grantees' compliance with semiannual data collection and reporting requirements; (3) determining what action, if any, DCPO has taken to monitor and ensure grantee compliance with the data collection reporting requirements; (4) identifying factors and barriers that may have contributed to a grantee's nonresponse and to delays in and the subsequent discontinuation of the NIJ-sponsored national evaluation of DCPO-funded programs; and (5) identifying improvements that may be warranted in DOJ's data collection efforts.

To assess how well DCPO has implemented efforts to collect performance and outcome data from federally funded drug court programs, we (1) interviewed appropriate DOJ officials and other drug court program stakeholders and practitioners; (2) reviewed DCPO program guidelines to determine the drug court program grantee data collection and reporting requirements; (3) analyzed recent survey data collected by DCPO and the Drug Court Clearinghouse and Technical Assistance Project (Drug Court Clearinghouse) to obtain information on the number of drug court programs that have been able to provide outcome data; and (4) conducted structured interviews with a statistically valid probability sample of DCPO-funded drug court programs to determine (a) the programs' ability to comply with DCPO's data collection requirements, (b) whether the programs had complied with the data collection requirements, and (c) for those programs that did not comply with the data collection requirements, why they did not comply and what action, if any, DCPO had taken.

For our structured interviews, we selected a stratified, random sample of 112 DCPO-funded drug court programs from a total of 315 drug court programs identified by DOJ as DCPO grantees in 2000. We stratified our sample into two groups based on whether the programs were listed in DCPO's database as respondents or nonrespondents to the required DCPO semiannual data collection survey in year 2000. To validate the accuracy of the list provided by DCPO, we compared the listing of 315 drug court programs identified as required to comply during a year 2000 reporting period with information on drug court program-related grant awards made by DCPO that was provided by OJP’s Office of the Comptroller to determine if the program was a DCPO grantee during the year 2000 reporting period. We defined a respondent as any drug court program grantee that was identified in DCPO's database as having responded to the DCPO survey during each applicable year 2000 reporting period. We defined a nonrespondent as a drug court program grantee that was identified in DCPO's database as not having responded to the DCPO survey during any applicable year 2000 reporting period. We used a structured data collection instrument to interview grantees. We interviewed 73 nonrespondents and 39 respondents. All results were weighted to represent the total population of drug court programs operating under a DCPO grant in year 2000.

All statistical samples are subject to sampling errors. Measures of sampling error are defined by two elements, the width of the confidence intervals around the estimate (sometimes called the precision of the estimate) and the confidence level at which the intervals are computed. Because we followed a probability procedure based on random selections, our sample is only one of a large number of samples that we might have drawn. As each sample could have provided different estimates, we express our confidence level in the precision of our sample results as a 95 percent confidence interval. This is the interval that would contain the actual population value for 95 percent of the samples we could have drawn. As a result, we are 95 percent confident that each of the confidence intervals based on the structured interviews will include the true value in the study population. All percentage estimates from the structured interviews have sampling errors of plus or minus 10 percentage points or less unless otherwise noted. For example, this means that if a percentage estimate is 60 percent and the 95 percent confidence interval is plus or minus 10 percentage points, we have 95 percent confidence that the true value in the population falls between 50 percent and 70 percent.

We performed limited verification of the drug court programs in our sample that were identified as non-respondents in DCPO’s database to determine whether they were actually DCPO grantees in 2000. Data obtained from the drug court programs was self-reported and, except for evidence obtained to confirm grantee compliance with DCPO's year 2000 reporting requirements, we generally did not validate their responses. We also did not fully verify the accuracy of the total number of drug court programs, or universe of drug court programs, provided to us by DCPO and the Drug Court Clearinghouse.

To assess DOJ's efforts to complete a national impact evaluation of federally funded drug court programs, we interviewed officials from (1) NIJ, who were responsible for DOJ's national evaluation effort; (2) DCPO, who were responsible for administering the federal drug court program under the Violent Crime Act; and (3) RAND, who were awarded the NIJ grant to complete phase I of the national evaluation effort. To identify the various administrative and research factors that hampered the completion of DOJ's national impact evaluation, we (1) interviewed NIJ and RAND officials who were responsible for the research project; (2) reviewed project objectives, tasks, and milestones outlined in NIJ's original solicitation and the NIJ approved RAND proposal and grant award; (3) reviewed correspondence between NIJ and RAND from 1998-2001; and (4) reviewed various project documents, including (a) RAND's evaluability assessment, (b) progress reports submitted to NIJ, (c) RAND's requests for no-cost extensions, (d) NIJ grant adjustment notices, (e) RAND's phase I draft report, and (f) RAND's phase I final report. Additionally, we compared project task milestones included in the NIJ approved RAND proposal with the actual project task completion dates.

To determine the universe and DCPO funding of drug court programs, we (a) interviewed appropriate DOJ officials and other drug court program stakeholders and practitioners; (b) reviewed and analyzed grant information obtained from DOJ's Office of Justice Programs grant management information system and DCPO; (c) reviewed and analyzed information on the universe of drug court programs maintained by the Drug Court Clearinghouse; and (d) reviewed congressional appropriations and DOJ press releases.

We attempted to verify information on the universe of DCPO-funded drug court programs, but as the findings in our report note, we were unable to do so due to inefficiencies in DOJ's drug court-related grant information systems. We were able to validate and correct some of the information provided by the various sources noted above through a comparison of the various databases noted and the primary data we had collected from drug court programs during our 1997 review and during our year 2001 follow-up structured interviews with a stratified, random sample of DCPO-funded drug court programs.

We conducted our work at DOJ headquarters in Washington, D.C., between March 2001 and February 2002 in accordance with generally accepted government auditing standards.

Operating Drug Court Programs by Location as of December 31, 2001

Based on information available as of December 31, 2001, drug court programs were operating in 48 states, the District of Columbia, and Puerto Rico. New Hampshire and Vermont were the only states without an operating drug court program but both have programs being planned. Guam also has programs being planned. California, Florida, Louisiana, Missouri, New York, and Ohio account for 344, or almost 44 percent, of the 791 operating drug courts. Figure 4 shows the number of operating drug court programs in each jurisdiction.

Drug Court Programs by Target Population as of December 31, 2001

Populations targeted by U.S. drug court programs included adults, juveniles, families, and Native American tribes. Table 2 shows the breakdown by target population of operating and planned drug court programs.

Status and DCPO Support of Drug Court Programs by Jurisdiction as of December 31, 2001

As Table 3 shows, drug court programs in the United States vary by target population and program status and have received various types of grants from the DOJ Drug Courts Program Office (DCPO).

Number, Amount, and Type of DCPO Grants Awarded to Drug Court Programs

Table 4 shows the number and total amount of DCPO grants awarded to plan, implement, or enhance U.S. drug court programs from fiscal years 1995 through 2001.

Timeline of NIJ’s Effort to Complete National Drug Court Impact Evaluation

NIJ issues solicitation for national evaluation of drug court programs NIJ awards grant to RAND RAND requests DCPO to write letters to 14 DCPO-funded sites regarding site visits for the national evaluation RAND submits written progress report to NIJ (no problems or changes were noted)

Scheduled milestone for completion of site visits RAND informs NIJ that it was still awaiting DCPO introductory letter to 14 DCPO-funded sites DCPO sent letter notifying 14 sites of the national evaluation Scheduled milestone for completion of phase II design strategy Written progress report submitted by RAND (no problems or changes were noted)

Scheduled milestone for completion of conceptual framework RAND provides evaluability assessment of 14 sites to NIJ noting feasibility concerns RAND requests conference with NIJ to discuss evaluability assessment NIJ informs RAND that DCPO still wants impact evaluations on some of the 14 sites RAND submits conceptual framework for 14 sites to NIJ NIJ and DCPO review the conceptual framework NIJ informs RAND that the report on the results of phase I must be submitted prior to the submission of a phase II proposal DCPO requests findings from RAND RAND requests guidance about conceptual framework paper RAND requests the first no-cost extension through September 30, 2000 NIJ informed RAND that phase I findings should be submitted in writing before RAND submits a proposal for phase II. RAND informed NIJ that a report on phase I findings would be completed by November 2000 RAND submits written progress report to NIJ noting their findings, an alternative strategy, and their request for a no-cost extension to enable RAND to bridge the time period between phase I and phase II NIJ grants RAND its first no-cost extension through September 30, 2000 DCPO and NIJ inquire about the status of the phase I draft report. NIJ reminds RAND of the original project requirements for an impact evaluation in phase II RAND inquired about whether the phase I grant would be extended beyond September 30, 2000 NIJ asked RAND to complete the phase I report by September 30, 2000, and reiterated to RAND that any proposals for phase II should address original solicitation objectives NIJ gives RAND the option to (1) let the phase I grant end and prepare the phase II proposal for a new grant or (2) extend the phase I project timeline to allow time for review of a phase II proposalRAND requested second no-cost extension NIJ grants no-cost extension to RAND extending completion of phase I until December 31, 2000. NIJ also inquires about status of draft and reminds RAND that draft must be submitted before a phase II proposal is accepted. RAND agreed RAND presented results from phase I at American Society of Criminology Conference noting that the phase I report would be available by the end of December In response to an NIJ inquiry, RAND informs NIJ that a phase I draft report would be completed by the end of January 2001 (NIJ did not extend the grant)

In response to an NIJ inquiry, RAND informs NIJ that the phase I draft report would be completed in February 2001 Written progress report submitted by RAND noting that a draft report will be submitted to NIJ in February 2001 (no problems were noted)

RAND informs NIJ that a draft phase I report will be completed in March 2001. NIJ grants third no-cost, extension to RAND extending completion of phase I until May 31, 2001 to allow for peer review of the forthcoming draft report NIJ receives draft phase I report and submits draft to peer reviewers NIJ informs RAND that phase II plans are uncertain NIJ sends peer review results to RAND and inquires as to when final report could be expected. NIJ provides RAND with specific instructions to eliminate the alternative phase II proposal from the finalphase I report noting that RAND's alternative proposal was so different from the project objective that it would be inappropriate to continue the effort RAND meets with NIJ to discuss phase I effort and completion of final report. RAND informs NIJ that the final report will be completed by the end of July 2001 Written progress report submitted by RAND (no problems or changes noted)

Comments from the Department of Justice

The following are GAO comments on DOJ’s letter of April 3, 2002.

GAO Comments

1.

In his reviews, Dr. Belenko noted that the long-term post-program impact of drug courts on recidivism and other outcomes are less clear—pointing out that the measurement of post-program outcomes other than recidivism remains quite limited in the drug court evaluation literature. He also noted that the evaluations varied in quality, comprehensiveness, use of comparison groups, and types of measures used and that longer follow-up and better precision in equalizing the length of follow-up between experimental and comparison groups are needed. 2. Dr. Belenko noted that the evaluations reviewed were primarily process, as opposed to impact, evaluations. He also noted that a shortcoming of some of the drug court evaluations was a lack of specificity about data collection time frames—pointing out that several studies lacked a distinction between recidivism that occurs while an offender is under drug court supervision and recidivism occurring after program participation.

Comments from RAND

GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

Charles Michael Johnson, Nettie Y. Mahone, Deborah L. Picozzi, Jerome T. Sandau, David P. Alexander, Douglas M. Sloane, and Shana B. Wallace made key contributions to this report.

GAO’s Mission

The General Accounting Office, the investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs