Drug Courts: Better DOJ Data Collection and Evaluation Efforts Needed to Measure Impact of Drug Court Programs
Published: 2002-04-18T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: In exchange for the possibility of dismissed charges or reduced sentences, defendants with substance abuse problems agree to be assigned to drug court programs. In drug courts, judges generally preside over the proceedings; monitor the progress of defendants; and prescribe sanctions and rewards in collaboration with prosecutors, defense attorneys, and treatment providers. Most decisions about drug court operations are left to local jurisdictions. Although programs funded by the Drug Court Program Office (DCPO) must collect and provide performance measurement and outcome data, the Department of Justice (DOJ) has not effectively managed this effort because of (1) its inability to readily identify the universe of DCPO-funded drug court programs, including those subject to DCPO's data collection reporting requirements; (2) its inability to accurately determine the number of drug court programs responding to DCPO's semiannual data collection survey; (3) inefficiencies in the administration of DCPO's semiannual data collection effort; (4) the elimination of post-program impact questions from the data collection survey effort; and (5) the lack of use of the Drug Court Clearinghouse. Various administrative and research factors have also hampered DOJ's ability to complete the two-phase National Institute of Justice-sponsored national impact evaluation study. As a result, DOJ continues to lack vital information needed to determine the overall impact of federally funded programs and to assess whether drug court programs use federal funds effectively.

section_title: Letter
paragraphs: This report responds to your request that we assess the Department of  Justice’s (DOJ) efforts to collect data on the performance and impact of  federally funded drug court programs. The main purpose of a drug court  program is to use the authority of the court to reduce crime by changing  defendants’ substance abuse behavior. Under this concept, in exchange for  the possibility of dismissed charges or reduced sentences, defendants are  diverted to drug court programs in various ways and at various stages in  the judicial process. Judges generally preside over drug court proceedings;  monitor the progress of defendants; and prescribe sanctions and rewards  as appropriate in collaboration with prosecutors, defense attorneys,  treatment providers, and others. While some basic requirements are set at  the federal level, most decisions about how a drug court operates are left to  local jurisdictions.
This report follows up on our 1997 report, which concluded that (1) many  drug court programs were not maintaining follow-up data on program  participants’ criminal recidivism or drug use relapse after they have left the  program and (2) differences and limitations in the objectives, scope, and  methodology of existing evaluation studies, among other things, did not  permit firm conclusions to be made on the overall impact or effectiveness  of drug court programs. In our 1997 report, we recommended that DOJ- funded drug court programs be required to collect and maintain post- program follow-up data on program participants’ criminal recidivism and,  to the extent feasible, post-program follow-up data on drug use relapse. To  improve the methodological soundness of future federally funded impact  evaluations, we recommended that these impact evaluations include post- program data and comparison groups within their scope. In 1998, DOJ  implemented our recommendations and, beginning in 1999, required its  DCPO-funded programs to periodically provide performance and outcome  data on their drug court programs. In addition, in 1998, DOJ undertook an  effort, through NIJ, to conduct a national impact evaluation using 14 DCPO- funded drug court programs. These efforts included the collection of post- program data within their scope.
As agreed with your staffs, this report focuses on DOJ’s (1) Drug Courts  Program Office’s (DCPO) efforts to collect performance and outcome data  from federally funded drug court programs and (2) National Institute of  Justice’s (NIJ) efforts to complete a national impact evaluation of federally  funded drug court programs. To achieve these objectives, among other  things, we (1) interviewed appropriate DOJ officials and other drug court  program researchers, stakeholders, and practitioners; (2) reviewed DCPO  program guidelines to determine grantee data collection and reporting  requirements; (3) conducted structured interviews with a representative  sample of DCPO-funded drug court programs; and (4) analyzed data from  recently completed surveys conducted by other drug court community  stakeholders. A more detailed description of our scope and methodology is  contained in appendix I.
subsections: section_title: Background
paragraphs: Since 1989, when the first drug court program was established, the number  of drug court programs has increased substantially. In addition, DCPO’s  oversight responsibilities and funding to support the planning,  implementation, and enhancement of these programs have increased.
As shown in figure 1, the number of operating drug court programs has  more than tripled since our prior report from about 250 in 1997 to almost  800 in 2001 based on information available as of December 31, 2001.
The number of operating programs that received DCPO funding, and thus  were subject to its oversight, has also grown—from over 150 in fiscal year  1997 to over 560 through fiscal year 2001.
As shown in figure 2, the number of drug court programs started by  calendar year since our prior report has also increased. Although the  number of drug court programs started in 2001 dropped, over 450  additional programs have been identified as being planned based on  information available as of December 31, 2001.
Based on information available as of December 31, 2001, drug court  programs were operating in 48 states, the District of Columbia, and Puerto  Rico. Only New Hampshire and Vermont had no operating drug court  programs. Six states (California, Florida, Louisiana, Missouri, New York,  and Ohio) accounted for over 40 percent of the programs. Appendix II  provides information on the number of operating drug court programs in  each state.
Although there are basic elements common to many drug court programs,  the programs vary in terms of approaches used, participant eligibility and  program requirements, type of treatment provided, sanctions and rewards,  and other practices. Drug court programs also target various populations  (adults, juveniles, families, and Native American tribes). Appendix III  provides details on the number of drug court programs by targeted  population, and appendix IV provides details on the drug court programs  by jurisdiction and the types of funding, if any, the programs have received  from DCPO.
Federal funding for drug court programs has also continued to increase. As  shown in table 1, congressional appropriations for the implementation of  DOJ’s drug court program has increased from about $12 million in fiscal  year 1995 to $50 million in fiscal years 2001 and 2002. Since fiscal year 1995,  Congress has appropriated about $267 million in Violent Crime Act related  funding to DOJ for the federal drug court program. DCPO funding in direct  support of drug court programs has increased from an average of about $9  million in fiscal years 1995 and 1996 to an average of about $31 million for  fiscal years 1997 through 2001. Between fiscal years 1995 and 2001, DCPO  has awarded about $174.5 million in grants to fund the planning,  implementation, and enhancement of drug court programs. About $21.5  million in technical assistance, training, and evaluations grants were  awarded. About $19.6 million were obligated for management and  administration purposes and to fund nongrant technical assistance,  training, and evaluation efforts. Since the inception of the DCPO drug court  program, a total of $3 million in prior year recoveries have been realized.  About $4.5 million through fiscal year 2001 had not been obligated.  Congress appropriated an additional $50 million for fiscal year 2002. At the  time of our review, DCPO was in the process of administering the fiscal  year 2002 grant award program.
Appendix V provides details on the number, amount, and types of grants  DCPO awarded since the implementation of the federal drug court  program.
section_title: DOJ Has Not Sufficiently Managed the Collection and Utilization of Performance and Outcome Data Collected from Federally Funded Drug Court Programs
paragraphs: Since 1998, DCPO implementation and enhancement grantees have been  required to collect, and starting in 1999, to submit to DCPO, among other  things, performance and outcome data on program participants. DCPO  collects these data semiannually using a Drug Court Grantee Data  Collection Survey. This survey was designed by DCPO to ensure that  grantees were collecting critical information about their drug court  programs and to assist in the national evaluation of drug court programs. In  addition, DOJ intended to use the information to respond to inquiries  regarding the effectiveness of drug court programs. However, due to  various factors, DCPO has not sufficiently managed the collection and  utilization of these data. As a result, DOJ cannot provide Congress, drug  court program stakeholders, and others with reliable information on the  performance and impact of federally funded drug court programs.
subsections: section_title: Factors Contributing to Insufficiencies in DOJ’s Management
paragraphs: Various factors contributed to insufficiencies in DOJ’s drug court program  data collection effort. These factors included (1) inability of DOJ to readily  identify the universe of DCPO-funded drug court programs, including those  subject to DCPO’s data collection reporting requirements; (2) inability of  DOJ to accurately determine the number of drug court programs that  responded to DCPO’s semiannual data collection survey; (3) inefficiencies  in the administration of DCPO’s semiannual data collection effort; (4) the  elimination of post-program impact questions from the scope of DCPO’s  data collection survey effort; and (5) the insufficient use of the Drug Court  Clearinghouse.
subsections: section_title: DOJ Has Been Unable to Readily Identify the Universe of Drug Court Programs It Has Funded
paragraphs: DOJ’s grant management information system, among other things, tracks  the number and dollar amount of grants the agency has awarded to state  and local jurisdictions and Native American tribes to plan, implement, and  enhance drug court programs. This system, however, is unable to readily  identify the actual number of drug court programs DCPO has funded.  Specifically, the system does not contain a unique drug court program  identifier, does not track grants awarded to a single grantee but used for  more than one drug court program, and contains data entry errors that  impact the reliability of data on the type of grants awarded. For example, at  the time of our review, the system contained some incorrectly assigned  grant numbers, did not always identify the type of grant awarded, and  incorrectly identified several grantees as receiving a planning,  implementation, and enhancement grant in fiscal year 2000. These factors  made it difficult for DCPO to readily produce an accurate universe of the  drug court programs that had received DCPO funding and were subject to  DCPO’s data collection reporting requirement.
Although DOJ has been able to provide information to enable an estimate  of the universe of DCPO-funded drug court programs to be derived, the  accuracy of this information is questionable because DCPO has relied on  the Drug Court Clearinghouse to determine the number of DCPO-funded  drug court programs and their program implementation dates. One of the  Drug Court Clearinghouse’s functions has been to identify DCPO-funded  drug court programs. However, the Drug Court Clearinghouse has only  been tasked since 1998 with following up with a segment of DCPO grantees  to determine their implementation date. Thus, the information provided to  DCPO on the universe of DCPO-funded drug court programs is at best an  estimate and not a precise count of DCPO drug court program grantees.  Noting that its current grant information system was not intended to  readily identify and track the number of DCPO-funded drug court  programs, DCPO officials said that they plan to develop a new management  information system that will enable DOJ to do so. Without an accurate  universe of DCPO-funded drug court programs, DCPO is unable to readily  determine the actual number of programs or participants it has funded or,  as discussed below, the drug court programs that should have responded to  its semiannual data collection survey.
section_title: DOJ Has Been Unable to Accurately Determine Response Rates for Its Data Collection Survey
paragraphs: According to DCPO officials, grantee response rates to DCPO’s semiannual  survey have declined since DCPO began administering the survey in 1998.  As shown in figure 3, the information in DCPO’s database indicated that  grantee response rates declined from about 78 percent for the first survey  reporting period (July to Dec. 1998) to about 32 percent for the July to  December 2000 reporting period. However, results from our follow-up  structured interviews with a representative sample of the identifiable  universe of drug court programs that were DCPO grantees during the 2000  reporting periods revealed that DCPO did not have an accurate account of  grantees’ compliance with its semiannual data collection survey.
Based on our structured interviews, we estimate that the response rate to  the DCPO data collection survey for the January to June 2000 reporting  period was about 60 percent in contrast to the 39 percent response rate  DCPO reported. Similarly, the response rate to the DCPO survey for the  July to December 2000 reporting period was about 61 percent in contrast to  the 32 percent response rate DCPO reported. The remaining programs did  not respond or were uncertain as to whether they responded to DCPO’s  data collection survey for each of the reporting periods in 2000. DOJ  officials said that some of the surveys they did not receive may have been  mailed to an incorrect office within DOJ. DCPO officials acknowledged  that this type of error could be mitigated if DCPO routinely followed up  with the drug court programs from which they did not receive responses.
Furthermore, based on our follow-up structured interviews with a  representative sample of DCPO-funded drug court programs that were  listed as nonrespondents in DCPO’s database, we estimate that about 61  percent had actually responded to DCPO’s survey for the January to June  2000 reporting period. About two-thirds of these programs could produce  evidence that they responded. For the July to December 2000 reporting  period, we estimate that about 51 percent of the DCPO-funded drug court  programs that were listed as nonrespondents in DCPO’s database had  actually responded to the survey. About two-thirds of these programs  could produce evidence that they responded.
section_title: Inefficiencies Existed in the Administration of DCPO’s Semiannual Data Collection Effort
paragraphs: The requirement for grantees to submit DCPO’s semiannual survey is  outlined in DOJ’s grant award notification letter that drug court program  grantees receive at the beginning of their grant period. In addition, the  survey is made available in the grantee application kit as well as on DCPO’s  website. However, other than these steps, DCPO has not consistently  notified its drug court program grantees of the semiannual reporting  requirements nor has it routinely forwarded the survey to grantees. At the  time of our review, DCPO had taken limited action to improve grantees’  compliance with the data collection survey requirements. DCPO officials  said that they generally had not followed up with drug court program  grantees that did not respond to the survey and had not taken action  towards the grantees that did not respond to the semiannual data collection  reporting requirement. Results from our follow-up structured interviews  showed that DCPO had not followed up to request completed surveys from  about 70 percent of the drug court program grantees that were  nonrespondents during the January to June 2000 reporting period and from  about 76 percent of the nonrespondents for the July to December 2000  reporting period.
DCPO has had other difficulties managing its data collection effort.  Specifically, (1) DCPO inadvertently instructed drug court program  grantees not to respond to questions about program participants’ criminal  recidivism while in the program; (2) confusion existed between DCPO and  its contractor, assigned responsibility for the semiannual data collection  effort, over who would administer DCPO’s data collection survey during  various reporting periods; and (3) some grantees were using different  versions of DOJ’s survey instruments to respond to the semiannual data  collection reporting requirement.
section_title: DCPO Eliminated Post-Program Data from Its Data Collection Effort
paragraphs: The overall success of a drug court programs is dependent on whether  defendants in the program stay off drugs and do not commit more crimes  when they complete the program. In our 1997 report we recommended that  drug court programs funded by discretionary grants administered by DOJ  collect and maintain follow-up data on program participants’ criminal  recidivism and, to the extent feasible, follow-up data on drug use relapse.  In 1998, DCPO required its implementation and enhancement grantees to  collect and provide performance and outcome data on program  participants, including data on participants’ criminal recidivism and  substance abuse relapse after they have left the program. However, in 2000,  DCPO revised its survey and eliminated the questions that were intended to  collect post-program outcome data.
The DCPO Director said that DCPO’s decision was based on, among other  things, drug court program grantees indicating that they were not able to  provide post-program outcome data and that they lacked sufficient  resources to collect such data. DCPO, however, was unable to produce  specific evidence from grantees (i.e., written correspondence) that cited  difficulties with providing post-program outcome data. The Director said  that difficulties have generally been conveyed by grantees, in person,  through telephone conversations, or are evidenced by the lack of responses  to the post-program questions on the survey.
Contrary to DCPO’s position, evidence exists that supports the feasibility of  collecting post-program performance and outcome data. During our 1997  survey of the drug court programs, 53 percent of the respondents said that  they maintained follow-up data on participants’ rearrest or conviction for a  nondrug crime. Thirty-three percent said that they maintained follow-up  data on participants’ substance abuse relapse.
Recent information collected from DCPO grantees continues to support the  feasibility of collecting post-program performance and outcome data. The  results of structured interviews we conducted in the year 2001 with a  representative sample of DCPO-funded drug court programs showed that  an estimated two-thirds of the DCPO-funded drug court programs  maintained criminal recidivism data on participants after they left the  program. About 84 percent of these programs maintained such data for 6  months or more. Of the remaining one-third that did not maintain post- program recidivism data, it would be feasible for about 63 percent to  provide such data. These estimates suggest that about 86 percent of DCPO- funded drug court programs would be able to provide post-program  recidivism data if requested.
The results of structured interviews we conducted in the year 2001 with a  representative sample of DCPO-funded drug court programs also showed  that about one-third of the DCPO-funded drug court programs maintained  substance abuse relapse data on participants after they have left the  program. About 84 percent of these programs maintained such data for 6  months or more. Of the estimated two-thirds that did not maintain post- program substance abuse relapse data, it would be feasible for about 30  percent to provide such data. These estimates suggest that about 50  percent of DCPO-funded drug court programs would be able to provide  post-program substance abuse data if requested.
According to survey results collected by the Drug Court Clearinghouse in  2000 and 2001, a significant number of the drug court programs were able  to provide post-program outcome data. For example, about 47 percent of  the DCPO-funded adult drug court programs that responded to the Drug  Court Clearinghouse’s 2000 operational survey reported that they  maintained some type of follow-up data on program participants after they  have left the program. Of these drug court programs, about 92 percent said  that they maintained follow-up data on recidivism and about 45 percent  said that they maintained follow-up data on drug usage.
Of the DCPO-funded adult and juvenile drug court programs operating for  at least a year that responded to the Drug Court Clearinghouse’s annual  survey that was published in 2001, about 56 percent were able to provide  follow-up data on program graduates’ recidivism and about 55 percent  were able to provide follow-up data on program graduates’ drug use  relapse.
section_title: DCPO Has Not Sufficiently Utilized the Drug Court Clearinghouse’s Data Collection Efforts
paragraphs: Operating under a cooperative agreement with DCPO, the Drug Court  Clearinghouse has successfully collected performance and outcome data  through an annual survey of all operating adult, juvenile, family, and tribal  drug court programs, including those funded by DCPO. In addition, as  previously noted, the Drug Court Clearinghouse has generally administered  an operational survey to adult drug court programs every 3 years, including  those funded by DCPO. The Drug Court Clearinghouse annually  disseminates the results from its annual survey and has periodically  published comprehensive drug court survey reports that provide detailed  operational, demographic, and outcome data on the adult drug court  programs identified through its data collection efforts. Although funded by  DOJ, the Drug Court Clearinghouse has not been required to primarily  collect and report separately on the universe of DCPO-funded programs. In  addition, no comprehensive or representative report has been produced by  DCPO or the Drug Court Clearinghouse that focuses primarily on the  performance and outcome of DCPO-funded drug court programs. Instead,  DCPO instructed the Drug Court Clearinghouse, in July 2001, to eliminate  recidivism data from its survey publications. Although the Drug Court  Clearinghouse has developed and implemented survey instruments to  periodically collect and disseminate recidivism and relapse data, the DCPO  Director had concerns with the quality of the self-reported data collected  and the inconsistent time frames for which post-program data were being  collected by drug court programs.
section_title: DOJ’s Effort to Complete a National Impact Evaluation of DCPO-Funded Drug Court Programs Has Fallen Short of Its Objective
paragraphs: In response to recommendations in our 1997 report, DOJ undertook,  through NIJ, an effort to conduct a two-phase national impact evaluation  focusing on 14 selected DCPO-funded drug court programs. This effort  was intended to include post-program data within its scope and to involve  the use of nonparticipant comparison groups. However, various  administrative and research factors hampered DOJ’s ability to complete the  NIJ-sponsored national impact evaluation, which was originally to be  completed by June 30, 2001. As a result, DOJ fell short of its objective,  discontinued this effort, and is considering an alternative study that, if  implemented, is not expected to provide information on the impact of  federally funded drug court programs until year 2007. Unless DOJ takes  interim steps to evaluate the impact of drug court programs, the Congress,  the public, and other drug court stakeholders will not have sufficient  information in the near term to assess the overall impact of federally  funded drug court programs.
subsections: section_title: The Objectives of DOJ’s National Evaluation Effort
paragraphs: The overall objective of the NIJ-sponsored national evaluation was to study  the impact of DCPO-funded drug court programs using comparison groups  and studying, among other things, criminal recidivism and drug use relapse.  This effort was to be undertaken in two phases and to include the  collection of post-program outcome data. The objectives for phase I, for  which NIJ awarded a grant to RAND in August 1998, were to (1) develop a  conceptual framework for evaluating the 14 DCPO-funded drug court  programs, (2) provide a description of the implementation of each  program, (3) determine the feasibility of including each of these 14 drug  court programs in a national impact evaluation, and (4) develop a viable  design strategy for evaluating program impact and the success of the 14  drug court programs. The design strategy was to be presented in the form  of a written proposal for a supplemental noncompetitive phase II grant.
The actual impact evaluation and an assessment of the success of the drug  court programs were to be completed during phase II of the study using a  design strategy resulting from phase I.
section_title: Various Administrative and Research Factors Hampered Completion of the National Impact Evaluation
paragraphs: NIJ’s two-phase national impact evaluation was originally planned for  completion by June 30, 2001. Phase I was awarded for up to 24 months and  was scheduled to conclude no later than June 30, 2000. However phase I  was not completed until September 2001—15 months after the original  project due date. Phase II, which NIJ expected to award after the  satisfactory submission of a viable design strategy for completing an  impact evaluation, has since been discontinued. Various administrative and  research factors contributed to delays in the completion of phase I and  DOJ’s subsequent decision to discontinue the evaluation. The factors  included (1) DCPO’s delay in notifying its grantees of RAND’s plans to  conduct site visits; (2) RAND’s lateness in meeting task milestones; (3)  NIJ’s multiple grant extensions to RAND that extended the timeframe for  completing phase I and further delayed NIJ’s subsequent decision to  discontinue phase II; and (4) the inability of the phase I efforts to produce a  viable design strategy that was to be used to complete a national impact  evaluation in phase II.
subsections: section_title: Administrative Delay in Notifying Grantees
paragraphs: Phase I of the NIJ-sponsored study was initially hampered by DCPO’s delay  in notifying its grantees of plans to conduct the national impact evaluation.  In November 1998, DCPO agreed to write a letter notifying its grantees of  RAND’s plan to conduct the national evaluation. The notification letters  were sent in March 1999. As a result, drug court program site visits, which  RAND had originally planned to complete by February 1999, were not  completed until July 1999.
section_title: Lateness in Meeting Task Milestones
paragraphs: Although RAND completed most of the tasks associated with the national  evaluation phase I objectives, it was generally late in meeting task  milestones. The conceptual framework for the evaluation of 14 DCPO- funded drug court programs, which RAND was originally scheduled to  complete by September 1999, was submitted to NIJ in May 2000—8 months  after the original task milestone. This timeframe, according to RAND, was  impacted by the delay in DOJ’s initiation of site visits. NIJ officials said that  RAND also did not deliver a complete description and analysis of drug  court implementation issues to NIJ, which was also due in September 1999,  until it received the first draft of RAND’s report in March 2001. The  feasibility study, which was originally scheduled to be completed by RAND  in September 1999, was provided to NIJ in November 1999. This study  informed NIJ of RAND’s concerns with the evaluability of some of the 14  selected DCPO sites. The viable design strategy proposal for evaluating  program impact at each of the 14 drug court programs, which RAND was  originally expected to complete by May 1999, was not completed. In  addition, as discussed below and detailed in appendix VI, RAND was  consistently late in meeting the extended milestones for delivery of the  final product for phase I.
section_title: Multiple Grant Extensions
paragraphs: Although RAND raised concerns in November 1999 regarding the feasibility  of completing a national impact evaluation at some of the 14 selected  DCPO sites, NIJ continued to grant multiple no-cost extensions that further  extended the completion of phase I. The first no-cost grant extension called  for phase I of the project to end by September 30, 2000; the second no-cost  extension called for phase I to end by December 31, 2000; and the final  extension authorized completion of phase I by May 31, 2001. Despite the  multiple extensions and RAND’s repeated assurances that the phase I  report was imminent, a final phase I report was not completed until  September 18, 2001—21 months after the original milestone for completion  of phase I. NIJ officials said that, in retrospect, they should have  discontinued this effort sooner. Appendix VI provides additional details on  the phase I delays in the NIJ-sponsored effort to complete a national impact  evaluation.
section_title: Lack of a Viable Design Strategy
paragraphs: Phase I of the NIJ-sponsored national impact evaluation did not produce a  viable design strategy that would enable an impact evaluation to be  completed during phase II using the selected DCPO-funded drug court  programs. RAND did offer an alternative approach. However, this approach  did not address the original objective—to conduct a national impact  evaluation. During its feasibility study, RAND rated the evaluability of the  14 program sites as follows: 4 - poor or neutral/poor, 5 - neutral, and 5 -  neutral/good or good. In response, NIJ and DCPO asked RAND to consider  completing the evaluation using those DCPO-funded program sites that  were deemed somewhat feasible. RAND, however, was not receptive to this  suggestion and did not produce a viable design strategy based on the 14  DCPO-funded programs or the subset of DCPO-funded programs that were  deemed feasible to use in phase II to evaluate the impact of federally  funded drug court programs. As a result, DOJ continues to lack a design  strategy for conducting a national impact to enable it to address the impact  of federally funded drug court programs in the near term.
section_title: DOJ’s Alternative Plan for Completing a National Evaluation Will Not Provide Near-Term Answers
paragraphs: To address the need for the completion of a national impact evaluation,  DCPO and NIJ are considering plans to complete a longitudinal study of  drug-involved offenders in up to 10 drug court program jurisdictions. The  DCPO Director said that the study would be done at a national level, and  the scope would include comparison groups and the collection of  individual level and post-program recidivism data. DOJ expects that this  project, which is in its formative stage, if implemented, will take up to 4  years to complete—with results likely in year 2007. We recognize that it  would take time to design and implement a rigorous longitudinal evaluation  study and that if properly implemented, such an effort should better enable  DOJ to provide information on the overall impact of federally funded drug  court programs. However, its year 2007 completion timeframe will not  enable DOJ to provide the Congress and other stakeholders with near-term  information on the overall impact of federally funded drug court programs  that has been lacking for nearly a decade.
section_title: Conclusions
paragraphs: Despite a significant increase in the number of drug court programs funded  by DCPO since 1997 that are required to collect and maintain performance  and outcome data, DOJ continues to lack vital information on the overall  impact of federally funded drug court programs. Furthermore, the agency’s  alternative plan for addressing the impact of federally funded drug court  programs will not offer near-term answers on the overall impact of these  programs. Improvements in DCPO’s management of the collection and  utilization of performance and outcome data from federally funded drug  court programs are needed. Additionally, more immediate steps from NIJ  and DCPO to carry out a methodologically sound national impact  evaluation could better enable DOJ to provide Congress and other drug  court program stakeholders with more timely information on the overall  impact of federally funded drug court programs. Until DOJ takes such  actions, the Congress, public, and other stakeholders will continue to lack  sufficient information to (1) measure long-term program benefits, if any;  (2) assess the impact of federally funded drug court programs on the  criminal behavior of substance abuse offenders; or (3) assess whether drug  court programs are an effective use of federal funds.
section_title: Recommendations for Executive Action
paragraphs: To improve the Department of Justice’s collection of data on the  performance and impact of federally funded drug court programs, we  recommend that the Attorney General  develop and implement a management information system that is able  to track and readily identify the universe of drug court programs funded  by DCPO;  take steps to ensure and sustain an adequate grantee response rate to  DCPO’s data collection efforts by improving efforts to notify and remind  grantees of their reporting requirements;  take corrective action towards grantees who do not comply with DOJ’s  data collection reporting requirements;  reinstate the collection of post-program data in DCPO’s data collection  effort, selectively spot checking grantee responses to ensure accurate  reporting;  analyze performance and outcome data collected from grantees and  report annually on the results; and  consolidate the multiple DOJ-funded drug court program-related data  collection efforts to better ensure that the primary focus is on the  collection and reporting of data on DCPO-funded drug court programs.
To better ensure that needed information on the impact of federally funded  drug court programs is made available to the Congress, public, and other  drug court stakeholders as early as possible, we also recommend that the  Attorney General take immediate steps to accelerate the funding and  implementation of a methodologically sound national impact evaluation  and to consider ways to reduce the time needed to provide information on  the overall impact of federally funded drug court programs. Furthermore,  we recommend that steps be taken to implement appropriate oversight of  this evaluation effort to ensure that it is well designed and executed, and  remains on schedule.
section_title: Agency Comments and Our Evaluation
paragraphs: We requested comments on a draft of this report from the Attorney  General. We also requested comments from RAND on a section of the draft  report pertaining to its efforts to complete phase I of NIJ’s national  evaluation effort.
On April 3, 2002, DOJ provided written comments on the draft report (see  app. VII). The Assistant Attorney General for the Office of Justice Programs  noted that we made several valuable recommendations for improving the  collection of data on the performance and impact of federally funded drug  court programs and outlined steps DOJ is considering to address two of the  six recommendations we make for improving its collection of data on the  performance and impact of federally funded drug court programs.  However, concerning the remaining four recommendations for improving  DOJ’s data collection effort, DOJ does not specifically outline any plans (1)  for taking corrective action towards grantees who do not comply with  DCPO’s data collection reporting requirements; (2) to reinstate the  collection of post program data in DCPO’s data collection effort, despite  the evidence cited in our report supporting the feasibility of collecting post  program data; (3) to analyze and report results on the performance and  outcome of DCPO grantees; and (4) to consolidate the multiple DOJ-funded  drug court program-related data collection efforts to ensure that the  primary focus of any future efforts is on the collection and reporting of data  on DCPO-funded programs.
Although DOJ points out in its comments that a number of individual  program evaluation studies have been completed, no national impact  evaluation of these programs has been done to date. We continue to believe  that until post-program follow-up data on program participants are  collected across a broad range of programs and also included within the  scope of future program and impact evaluations (including nonprogram  participant data), it will not be possible to reach firm conclusions about  whether drug court programs are an effective use of federal funds or  whether different types of drug court program structures funded by DCPO  work better than others. Also, unless these results are compared with those  on the impact of other criminal justice programs, it will not be clear  whether drug court programs are more or less effective than other criminal  justice programs. As such, these limitations have prevented firm  conclusions from being drawn on the overall impact of federally funded  drug court programs.
With respect to our recommendations for improving DOJ’s drug court  program-related impact evaluation efforts, DOJ, in its comments, outlines  steps it is taking to complete a multisite impact evaluation and its plans to  monitor the progress of this effort and to provide interim information  during various intervals. As discussed on page 18 of this report, this effort  is intended to be done at a national level, and the scope is to include  comparison groups and the collection of individual-level and post-program  recidivism data.
On April 1, 2002, RAND provided written comments on the segment of the  draft report  relating to DOJ’s efforts to complete a national impact  evaluation (see app. VIII). In its comments, RAND, as we do in our report,  acknowledges the need for improvements in the data collection  infrastructure for DCPO-funded drug court programs. RAND notes its  rationale for why it views the deliverables associated with phase I of the  NIJ-sponsored national impact evaluation as being timely and notes that  researchers generally have discretion to revise timelines and scopes of  work, with the agreement of the client.  However, as we point out in our  report (pp. 17-18 and app. VI), RAND requested several no-cost extensions  to complete the deliverables for various task milestones and did not  produce a viable design strategy for addressing the impact of DCPO-funded  drug court programs. In addition, NIJ officials said that RAND also did not  deliver a complete description and analysis of drug court implementation  issues to NIJ until it received the first draft of RAND’s report in March 2001.  The deliverable RAND refers to in its comment letter was a paper that  RAND had prepared for the National Institute on Drug Abuse, which NIJ  never considered to be a product under the grant to evaluate the impact of  DCPO-funded drug court programs. As we also pointed out in our report  (p. 17 and app. VI), NIJ was not amenable to RAND changing the scope or  methodology of the national impact evaluation effort.  In addition, RAND  commented that a “simple” evaluation design was expected. NIJ’s original  objective, however, never called for a simple evaluation design, but rather a  viable design strategy involving the use of comparison groups and the  collection of post-program data.
We conducted our work at DOJ headquarters in Washington, D.C., between  March 2001 and February 2002 in accordance with generally accepted  government auditing standards.
As agreed with your offices, unless you publicly release its contents earlier,  we plan no further distribution of this report until 30 days from its issue  date. At that time, we will provide copies of this report to the Attorney  General, the Director of the Office of Management and Budget, and other  interested parties. We will also make copies available to others upon  request.
If you or your staff have any questions about this report, please contact  Daniel C. Harris or me at (202) 512-2758 or at ekstrandl@gao.gov. Key  contributors to this report are acknowledged in appendix IX.
section_title: Objectives, Scope, and Methodology
paragraphs: Our overall objective for this review was to assess how well the  Department of Justice (DOJ) has implemented efforts to collect  performance and impact data on federally funded drug court programs. We  specifically focused on DOJ’s (1) Drug Courts Program Office’s (DCPO)  efforts to collect performance and outcome data from federally funded  drug court programs and (2) National Institute of Justice’s (NIJ) efforts to  complete a national impact evaluation of federally funded drug court  programs.
While there are drug court programs that receive funds from other federal  sources, our review focused on those programs receiving federal funds  from DCPO, which is DOJ’s component responsible for administering the  federal drug court program under the Violent Crime Act.  The scope of our  work was limited to (1) identifying the processes DCPO used to implement  its semiannual data collection effort; (2) determining DCPO grantees'  compliance with semiannual data collection and reporting requirements;  (3) determining what action, if any, DCPO has taken to monitor and ensure  grantee compliance with the data collection reporting requirements;  (4) identifying factors and barriers that may have contributed to a grantee's  nonresponse and to delays in and the subsequent discontinuation of the  NIJ-sponsored national evaluation of DCPO-funded programs; and  (5) identifying improvements that may be warranted in DOJ's data  collection efforts.
To assess how well DCPO has implemented efforts to collect performance  and outcome data from federally funded drug court programs, we  (1) interviewed appropriate DOJ officials and other drug court program  stakeholders and practitioners; (2) reviewed DCPO program guidelines to  determine the drug court program grantee data collection and reporting  requirements; (3) analyzed recent survey data collected by DCPO and the  Drug Court Clearinghouse and Technical Assistance Project (Drug Court  Clearinghouse) to obtain information on the number of drug court  programs that have been able to provide outcome data; and  (4) conducted structured interviews with a statistically valid probability  sample of DCPO-funded drug court programs to determine (a) the  programs' ability to comply with DCPO's data collection requirements,  (b) whether the programs had complied with the data collection  requirements, and (c) for those programs that did not comply with the data  collection requirements, why they did not comply and what action, if any,  DCPO had taken.
For our structured interviews, we selected a stratified, random sample of  112 DCPO-funded drug court programs from a total of 315 drug court  programs identified by DOJ as DCPO grantees in 2000. We stratified our  sample into two groups based on whether the programs were listed in  DCPO's database as respondents or nonrespondents to the required DCPO  semiannual data collection survey in year 2000. To validate the accuracy of  the list provided by DCPO, we compared the listing of 315 drug court  programs identified as required to comply during a year 2000 reporting  period with information on drug court program-related grant awards made  by DCPO that was provided by OJP’s Office of the Comptroller to  determine if the program was a DCPO grantee during the year 2000  reporting period. We defined a respondent as any drug court program  grantee that was identified in DCPO's database as having responded to the  DCPO survey during each applicable year 2000 reporting period. We  defined a nonrespondent as a drug court program grantee that was  identified in DCPO's database as not having responded to the DCPO survey  during any applicable year 2000 reporting period. We used a structured data  collection instrument to interview grantees. We interviewed 73  nonrespondents and 39 respondents. All results were weighted to represent  the total population of drug court programs operating under a DCPO grant  in year 2000.
All statistical samples are subject to sampling errors. Measures of sampling  error are defined by two elements, the width of the confidence intervals  around the estimate (sometimes called the precision of the estimate) and  the confidence level at which the intervals are computed. Because we  followed a probability procedure based on random selections, our sample  is only one of a large number of samples that we might have drawn. As each  sample could have provided different estimates, we express our confidence  level in the precision of our sample results as a 95 percent confidence  interval. This is the interval that would contain the actual population value  for 95 percent of the samples we could have drawn. As a result, we are 95  percent confident that each of the confidence intervals based on the  structured interviews will include the true value in the study population. All  percentage estimates from the structured interviews have sampling errors  of plus or minus 10 percentage points or less unless otherwise noted. For  example, this means that if a percentage estimate is 60 percent and the 95  percent confidence interval is plus or minus 10 percentage points, we have  95 percent confidence that the true value in the population falls between 50  percent and 70 percent.
We performed limited verification of the drug court programs in our sample  that were identified as non-respondents in DCPO’s database to determine  whether they were actually DCPO grantees in 2000. Data obtained from the  drug court programs was self-reported and, except for evidence obtained  to confirm grantee compliance with DCPO's year 2000 reporting  requirements, we generally did not validate their responses. We also did not  fully verify the accuracy of the total number of drug court programs, or  universe of drug court programs, provided to us by DCPO and the Drug  Court Clearinghouse.
To assess DOJ's efforts to complete a national impact evaluation of  federally funded drug court programs, we interviewed officials from  (1) NIJ, who were responsible for DOJ's national evaluation effort;  (2) DCPO, who were responsible for administering the federal drug court  program under the Violent Crime Act; and (3) RAND, who were awarded  the NIJ grant to complete phase I of the national evaluation effort. To  identify the various administrative and research factors that hampered the  completion of DOJ's national impact evaluation, we (1) interviewed NIJ  and RAND officials who were responsible for the research project;  (2) reviewed project objectives, tasks, and milestones outlined in NIJ's  original solicitation and the NIJ approved RAND proposal and grant award;  (3) reviewed correspondence between NIJ and RAND from 1998-2001; and  (4) reviewed various project documents, including (a) RAND's evaluability  assessment, (b) progress reports submitted to NIJ, (c) RAND's requests for  no-cost extensions, (d) NIJ grant adjustment notices, (e) RAND's phase I  draft report, and (f) RAND's phase I final report. Additionally, we compared  project task milestones included in the NIJ approved RAND proposal with  the actual project task completion dates.
To determine the universe and DCPO funding of drug court programs, we  (a) interviewed appropriate DOJ officials and other drug court program  stakeholders and practitioners; (b) reviewed and analyzed grant  information obtained from DOJ's Office of Justice Programs grant  management information system and DCPO; (c) reviewed and analyzed  information on the universe of drug court programs maintained by the Drug  Court Clearinghouse; and (d) reviewed congressional appropriations and  DOJ press releases.
We attempted to verify information on the universe of DCPO-funded drug  court programs, but as the findings in our report note, we were unable to  do so due to inefficiencies in DOJ's drug court-related grant information  systems. We were able to validate and correct some of the information  provided by the various sources noted above through a comparison of the  various databases noted and the primary data we had collected from drug  court programs during our 1997 review and during our year 2001 follow-up  structured interviews with a stratified, random sample of DCPO-funded  drug court programs.
We conducted our work at DOJ headquarters in Washington, D.C., between  March 2001 and February 2002 in accordance with generally accepted  government auditing standards.
section_title: Operating Drug Court Programs by Location as of December 31, 2001
paragraphs: Based on information available as of December 31, 2001, drug court  programs were operating in 48 states, the District of Columbia, and Puerto  Rico. New Hampshire and Vermont were the only states without an  operating drug court program but both have programs being planned.  Guam also has programs being planned. California, Florida, Louisiana,  Missouri, New York, and Ohio account for 344, or almost 44 percent, of the  791 operating drug courts. Figure 4 shows the number of operating drug  court programs in each jurisdiction.
section_title: Drug Court Programs by Target Population as of December 31, 2001
paragraphs: Populations targeted by U.S. drug court programs included adults,  juveniles, families, and Native American tribes. Table 2 shows the  breakdown by target population of operating and planned drug court  programs.
section_title: Status and DCPO Support of Drug Court Programs by Jurisdiction as of December 31, 2001
paragraphs: As Table 3 shows, drug court programs in the United States vary by target  population and program status and have received various types of grants  from the DOJ Drug Courts Program Office (DCPO).
section_title: Number, Amount, and Type of DCPO Grants Awarded to Drug Court Programs
paragraphs: Table 4 shows the number and total amount of DCPO grants awarded to  plan, implement, or enhance U.S. drug court programs from fiscal years  1995 through 2001.
section_title: Timeline of NIJ’s Effort to Complete National Drug Court Impact Evaluation
paragraphs: NIJ issues solicitation for national evaluation of drug court programs NIJ awards grant to RAND  RAND requests DCPO to write letters to 14 DCPO-funded sites regarding site visits for the national evaluation RAND submits written progress report to NIJ (no problems or changes were noted)
Scheduled milestone for completion of site visits RAND informs NIJ that it was still awaiting DCPO introductory letter to 14 DCPO-funded sites DCPO sent letter notifying 14 sites of the national evaluation Scheduled milestone for completion of phase II design strategy Written progress report submitted by RAND (no problems or changes were noted)
Scheduled milestone for completion of conceptual framework RAND provides evaluability assessment of 14 sites to NIJ noting feasibility concerns RAND requests conference with NIJ to discuss evaluability assessment NIJ informs RAND that DCPO still wants impact evaluations on some of the 14 sites RAND submits conceptual framework for 14 sites to NIJ NIJ and DCPO review the conceptual framework NIJ informs RAND that the report on the results of phase I must be submitted prior to the submission of   a phase II proposal DCPO requests findings from RAND  RAND requests guidance about conceptual framework paper RAND requests the first no-cost extension through September 30, 2000 NIJ informed RAND that phase I findings should be submitted in writing before RAND submits a proposal for   phase II.  RAND informed NIJ that a report on phase I findings would be completed by November 2000 RAND submits written progress report to NIJ noting their findings, an alternative strategy, and their request for a no-cost extension to enable RAND to bridge the time period between phase I and phase II NIJ grants RAND its first no-cost extension through September 30, 2000 DCPO and NIJ inquire about the status of the phase I draft report. NIJ reminds RAND of the original project  requirements for an impact evaluation in phase II RAND inquired about whether the phase I grant would be extended beyond September 30, 2000  NIJ asked RAND to complete the phase I report by September 30, 2000, and reiterated to RAND that   any proposals for phase II should address original solicitation objectives NIJ gives RAND the option to (1) let the phase I grant end and prepare the phase II proposal for a new grant or (2) extend the phase I project timeline to allow time for review of a phase II proposalRAND requested second no-cost extension  NIJ grants no-cost extension to RAND extending completion of phase I until December 31, 2000.  NIJ also  inquires about status of draft and reminds RAND that draft must be submitted before a phase II proposal is  accepted.  RAND agreed RAND presented results from phase I at American Society of Criminology Conference noting that the   phase I report would be available by the end of December In response to an NIJ inquiry, RAND informs NIJ that a phase I draft report would be completed by the   end of January 2001 (NIJ did not extend the grant)
In response to an NIJ inquiry, RAND informs NIJ that the phase I draft report would be completed in  February 2001 Written progress report submitted by RAND noting that a draft report will be submitted to NIJ in February 2001 (no problems were noted)
RAND informs NIJ that a draft phase I report will be completed in March 2001. NIJ grants third no-cost,  extension to RAND extending completion of phase I until May 31, 2001 to allow for peer review of the forthcoming draft report NIJ receives draft phase I report and submits draft to peer reviewers NIJ informs RAND that phase II plans are uncertain NIJ sends peer review results to RAND and inquires as to when final report could be expected. NIJ provides RAND with specific instructions to eliminate the alternative phase II proposal from the finalphase I report noting that RAND's alternative proposal was so different from the project objective that it would be inappropriate to continue the effort RAND meets with NIJ to discuss phase I effort and completion of final report.  RAND informs NIJ that the   final report will be completed by the end of July 2001 Written progress report submitted by RAND (no problems or changes noted)
section_title: Comments from the Department of Justice
paragraphs: The following are GAO comments on DOJ’s letter of April 3, 2002.
subsections: section_title: GAO Comments
paragraphs: 1.
In his reviews, Dr. Belenko noted that the long-term post-program  impact of drug courts on recidivism and other outcomes are less  clear—pointing out that the measurement of post-program outcomes  other than recidivism remains quite limited in the drug court evaluation  literature. He also noted that the evaluations varied in quality,  comprehensiveness, use of comparison groups, and types of measures  used and that longer follow-up and better precision in equalizing the  length of follow-up between experimental and comparison groups are  needed. 2. Dr. Belenko noted that the evaluations reviewed were primarily  process, as opposed to impact, evaluations. He also noted that a  shortcoming of some of the drug court evaluations was a lack of  specificity about data collection time frames—pointing out that several  studies lacked a distinction between recidivism that occurs while an  offender is under drug court supervision and recidivism occurring after  program participation.
section_title: Comments from RAND
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: Charles Michael Johnson, Nettie Y. Mahone, Deborah L. Picozzi, Jerome T.  Sandau, David P. Alexander, Douglas M. Sloane, and Shana B. Wallace made  key contributions to this report.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the investigative arm of Congress, exists to  support Congress in meeting its constitutional responsibilities and to help improve  the performance and accountability of the federal government for the American  people. GAO examines the use of public funds; evaluates federal programs and  policies; and provides analyses, recommendations, and other assistance to help  Congress make informed oversight, policy, and funding decisions. GAO’s  commitment to good government is reflected in its core values of accountability,  integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents is through the  Internet. GAO’s Web site (www.gao.gov) contains abstracts and full-text files of  current reports and testimony and an expanding archive of older products. The  Web site features a search engine to help you locate documents using key words  and phrases. You can print these documents in their entirety, including charts and  other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs