Adult Drug Courts: Studies Show Courts Reduce Recidivism, but DOJ Could Enhance Future Performance Measure Revision Efforts
Released: 2011-12-09T00:00:00

Letter

Background

Drug court programs are designed to address the underlying cause of an offender’s behavior—alcohol, drug addiction, and dependency problems. Drug court programs share several general characteristics but vary in their specific policies and procedures because of, among other things, differences in local jurisdictions and criminal justice system practices. In general, judges preside over drug court proceedings, which are called status hearings; monitor offenders’ progress with mandatory drug testing; and prescribe sanctions and incentives as appropriate in collaboration with prosecutors, defense attorneys, treatment providers, and others. Drug court programs vary in terms of the substance-abuse treatment required. However, most programs offer a range of treatment options and generally require a minimum of 1 year of participation before an offender completes the program.

Practices for determining defendants’ eligibility for drug court participation vary across drug court programs, but typically involve screening defendants for their criminal history, current case information, whether they are on probation, and their substance use, which can include the frequency and type of use, prior treatment experiences, and motivation to seek treatment. In 2005, we reported that based on literature reviewed, eligible drug-court program participants ranged from nonviolent offenders charged with drug-related offenses who had substance addictions, to relatively medium risk defendants with fairly extensive criminal histories and who had failed prior substance-abuse-treatment experiences. Appendix IV presents additional information about the general characteristics of drug court programs. As shown in appendix V, BJA, in collaboration with the National Association of Drug Court Professionals (NADCP), identified The Key Components, which describes the basic elements that define drug courts and offers performance benchmarks to guide implementation.

BJA administers the Adult Drug Court Discretionary Grant Program to provide financial and technical assistance to states, state courts, local courts, units of local government, and Indian tribal governments to develop and implement drug treatment courts. Through the Adult Drug Court Discretionary Grant Program, BJA offers funding in four broad drug- court grant categories. See appendix VI for a more detailed discussion on each of the following grant categories.

Implementation grants: Available to jurisdictions that have completed a substantial amount of planning and are ready to implement an adult drug court.  Enhancement grants: Available to jurisdictions with a fully operational (at least 1-year) adult drug court.  Statewide grants: Available for two purposes: (1) To improve, enhance, or expand drug court services statewide through activities such as training and/or technical assistance programs for drug court teams and (2) To financially support drug courts in local or regional jurisdictions that do not currently operate with BJA Adult Drug Court Discretionary Grant Program funding.

Joint grants: In fiscal year 2010, BJA, in collaboration with the Department of Health and Human Services, Substance Abuse and Mental Health Services Administration (SAMHSA), offered a joint grant program for the enhancement of adult drug court services, coordination, and substance-abuse treatment capacity.

From fiscal years 2006 through 2010, Congress appropriated about $120 million for DOJ’s administration of all drug court programs. Of this amount, $76 million was used for the Adult Drug Court Discretionary Grant Program, which includes funding provided to grantees through the previously mentioned grant categories. The grant award totals for the Adult Drug Court Discretionary Grant Program increased from $2 million in fiscal year 2006 to $29 million in fiscal year 2010. Correspondingly, the number of Adult Drug Court Discretionary Grant Program awards increased from 16 in fiscal year 2006 to 110 in fiscal year 2010—an increase of 588 percent, as shown in figure 1.

With regard to drug courts’ effectiveness, however, drug courts have been difficult to evaluate because they are so varied, and the resources required to conduct a study that would allow conclusions about the effectiveness of drug courts can be substantial. In particular, while drug courts generally adhere to certain key program components, drug courts can differ in factors including admission criteria, type and duration of drug treatment, degree of judicial monitoring and intervention, and application of sanctions for noncompliance. In February 2005, we studied drug courts and reported that in most of the 27 drug-court program evaluations we reviewed, adult drug court programs led to recidivism reductions during periods of time that generally corresponded to the length of the drug court program. Several syntheses of multiple drug court program evaluations, conducted in 2005 and 2006, also concluded that drug courts are associated with reduced recidivism rates, compared to traditional correctional options. However, the studies included in these syntheses often had methodological limitations, such as the lack of equivalent comparison groups and the lack of appropriate statistical controls.

BJA Is Expanding Use of Grantee Performance Data but Could Enhance Processes as It Continues to Refine Performance Measures

BJA’s Ongoing Data Collection Efforts

BJA collects an array of performance data from its adult drug court grantees through its Performance Measurement Tool (PMT) and OJP’s Grants Management System (GMS). Since fiscal year 2008, BJA has required grantees to submit quantitative performance data on a quarterly basis and qualitative performance information on a semi-annual basis. The quantitative information grantees submit to BJA varies depending on the type of grant awarded. For example, information that BJA can calculate based on what Implementation grantees have been required to submit quarterly includes “the percent of drug court participants who exhibit a reduction in substance use during the reporting period,” “the percent of program participants who re-offended while in the drug court program,” and “the number and percent of drug court graduates.” Information that BJA can calculate based on what Enhancement grantees have been required to submit includes “the increase in units of substance- abuse treatment services” and “the percent increase in services provided to participants.” In addition to the quarterly reporting of quantitative performance data, all adult drug court grantees must submit progress reports semi-annually. As part of these progress reports, grantees provide qualitative or narrative responses to seven questions. Table 1 shows the seven questions to which grantees must submit narrative responses when completing their semi-annual reports.

BJA officials told us that grant managers regularly review individual grantees’ quarterly performance data and semi-annual progress reports and use this information to determine whether additional training or technical assistance could improve their performance. However, according to BJA officials, resource constraints in the past had prevented staff from fully analyzing the performance data BJA collects from all adult drug court grantees—specifically the analysis of grantees’ answers to the seven narrative questions—to identify more effective program approaches and processes to share with the drug court community. In early fiscal year 2011, BJA officials initiated a new process called GrantStat to maximize the use of performance information by leveraging the resources of other BJA divisions, BJA’s training and technical assistance partners, its contractor, and other key stakeholders. GrantStat provides an analytical framework to assess grantee performance data and other relevant information on a semi-annual basis to determine the effectiveness of the grant programs in BJA’s portfolio.

In September 2011, BJA officials applied GrantStat to a review of the Adult Drug Court Discretionary Grant Program. As part of the process, they collected, reviewed, and analyzed performance data and other relevant information from a cohort of Implementation grantees to determine the overall effectiveness of the adult drug court program and to identify grantees that might need additional technical assistance to improve their outcomes. BJA officials told us that as part of the GrantStat review, they and their technical-assistance provider’s staff reviewed selected Implementation grantees’ responses to the seven narrative questions and discussed common issues they each identified. For example, BJA identified that a number of grantees had lower-than- expected capacity because drug court stakeholders (e.g., district attorneys) were referring fewer drug-involved defendants to these drug courts. BJA also reported reviewing and discussing other qualitative information, such as the training and technical assistance provider’s site- visit reports, to determine grantees’ fidelity to the 10 key components. BJA officials acknowledged that prior to GrantStat, they had not leveraged the summary data that its technical assistance providers had previously compiled from grantees’ narrative responses to these seven questions and indicated that future iterations of GrantStat would continue to include both qualitative and quantitative performance data reviews.

Our prior work has emphasized the importance of using performance data to inform key decisions and underscored that performance measures can be used to demonstrate the benefits of a program or identify ways to improve it. In addition, we also have reported that effective performance measurement systems include steps to use performance information to make decisions. In doing so, program managers can improve their programs and results. Recognizing that BJA is working through GrantStat to improve its use of performance data in managing the drug court program, we identified six management activities for which performance information can be most useful to decision makers and benchmarked BJA’s practices against them. The six activities are: (1) setting program priorities, (2) allocating resources, (3) adopting new program approaches, (4) identifying and sharing with stakeholders more effective program processes and approaches, (5) setting expectations for grantees, and (6) monitoring grantee performance. See appendix VII for the definition of the six management activities. As illustrated in table 2, BJA has current and planned efforts underway across all six activities.

According to BJA officials, after the GrantStat review, they identified trends and developed several potential findings and action items for program design changes. However, BJA officials added that since the action items originated from GrantStat’s first review, they are not implementing them immediately. Instead, BJA plans to evaluate the action items over the next 6 months to ensure they are feasible and effective alternatives for improving grantee outcomes. We are encouraged by BJA’s recent efforts to regularly analyze grantee performance data to determine whether the program is meeting its goals. We also are encouraged that BJA is using this information to better inform its grant-related management activities, such as setting program priorities, identifying and sharing effective processes and approaches, and setting expectations for grantees.

BJA Recently Revised Its Drug Court Performance Measures

During the course of our review, BJA revised its adult drug court program performance measures to improve their reliability and usefulness. BJA provided us with the revised measures on October 28, 2011. According to BJA officials, unclear definitions of some of the previous measures confused grantees about what data elements they were expected to collect. For example, officials told us that grantees may have been confused with how to measure “the number of participants admitted” and “the number of drug court participants.” Specifically, BJA officials added that their analysis of several years of data shows that some grantees reported the same number for these two measures, some grantees reported a higher number than were admitted, a few grantees reported a lesser number for the number of participants than the number admitted, and some grantees reported these two measures in each of these three ways over multiple reporting periods. According to BJA officials, such a wide degree of variability made these measures unreliable, and BJA was thus hindered from comparing grantee performance data across grantee cohorts.

BJA’s performance measure revisions resulted in the following:  All grantees are required to report on “participant level” measures. Examples of these measures include the demographic make-up of their drug court participant populations, the amount of service provided to their participants, and the geographic location of their drug courts;  Enhancement, Joint, and Statewide grantees are required to report on participant level outcomes, such as graduation rates, to ensure consistency with measures BJA collects from Implementation grantees;  Measures previously excluded from the PMT, such as retention rates and outcomes of participants once they complete the drug court program, are now included;  BJA has established two sets of benchmarks as points of reference against which to gauge grantees’ performance. The first set of benchmarks requires a comparison of grantees’ performance against averages of drug court performance derived from research. The second set of benchmarks requires a comparison of grantees’ performance to historical performance data reported to BJA by adult drug court grantees; and  BJA revised the descriptions and the definitions of the measures to help ensure their clarity.

To revise the performance measures, BJA officials consulted with technical assistance providers and a drug court researcher to discuss possible improvements to the performance measures, reviewed drug court literature, and reviewed and analyzed BJA grantees’ clarification and information requests to identify the most common problems adult drug court grantees historically experienced submitting performance information to BJA. In addition, BJA obtained comments on the proposed measures from BJA staff and other DOJ stakeholders, as well as Enhancement, Implementation, Joint, and Statewide grantees. BJA officials also invited all current grantees to participate in four teleconferences to obtain their feedback on the feasibility of collecting and reporting the new measures and their suggestions to improve the clarity of the measures’ definitions and descriptions. BJA officials finalized the new measures in October 2011 and plan to closely monitor grantees’ performance data submissions to ensure the reliability and usefulness of the measures and then revise as necessary after the first reporting period. BJA officials also stated that they expected to review the measures’ overall reliability and validity after the first reporting period— October 1, 2011, through December 30, 2011.

BJA officials reported that the revised measures will strengthen the reliability and improve the usefulness of grantee performance data in making grant-related decisions. For example, BJA officials stated that reliable and useful data would help them to identify the most effective grantees and common characteristics these courts share to inform the types of drug courts the officials choose to fund in future grant solicitations. BJA officials also reported that as a result of the revision, they expect to be able to conduct more sophisticated analyses using GrantStat that are needed to inform grant-related decisions. For example, BJA officials told us that implementing benchmarks and participant level measures will enable the agency to compare similar drug courts (e.g., large-urban jurisdictions of similar size, demographic make-up, and geographic context) to one another and across jurisdictions, thereby improving BJA’s understanding of grantees’ impact on the populations they serve.

BJA Could Enhance Two Key Practices as It Continues to Review and Revise Its Adult Drug Court Performance Measures

BJA’s process to revise its performance measures generally adhered to some of the key practices that we have identified as important to ensuring that measures are relevant and useful to decision-making. These key practices included obtaining stakeholder involvement and ensuring that the measures have certain key attributes, such as clarity. The key practices also describe the value of testing the measures to ensure that they are credible, reliable and valid and documenting key steps throughout the revision process. However, BJA could take actions to improve its efforts in these two areas. For instance, BJA officials told us that after the grantees’ first reporting period concludes, they plan to assess the data that grantees submitted to ensure that the measures produce reliable and useful data over at least the first quarter of fiscal year 2012. They stated that if necessary, at that point they will then further revise the measures. Nevertheless, BJA officials have not documented how they will determine if the measures were successful or whether changes would be needed. In addition, BJA officials did not record key methods and assumptions used to guide their revision efforts, such as the feedback stakeholders provided and BJA’s disposition of these comments. For example, BJA officials provided a document generally showing the original performance measure; whether it was removed, revised or replaced; and BJA’s justification for the action, but this document did not demonstrate how BJA had incorporated the stakeholder feedback it considered when making its decisions. The document also did not include a link to a new performance measure in instances where an older one was being replaced. Further, BJA’s justification did not include the rationale for the changes it made to 22 of the 51 performance measures. According to BJA officials, they did not document their decisions in this way because of the rapid nature of the revision process and limited staff resources. They also told us that maintaining such documentation and providing it to stakeholders held little value.

Our previous work has shown the importance of documentation to the successful development of effective performance measures. In the past, we have reported that revising performance measures involves a number of aspects needing to be carefully planned and carried out and that by documenting the steps undertaken in developing and implementing the revised measures, agencies can be better assured their revisions result in effective performance measures. In addition, academic literature on the best practices for developing effective performance measures states that agencies should develop products to document and guide their revision efforts. These products, among other things, can include plans for ensuring the quality and integrity of the data for full-scale implementation of the measures. Further, Standards for Internal Control in the Federal Government call for clear documentation of significant events, which can include assumptions and methods surrounding key decisions, and this documentation should be readily available for examination. As BJA moves forward in assessing the revised measures and implementing additional changes, if it deems necessary, BJA could better ensure that its efforts result in successful and reliable metrics and are transparent by documenting key methods used to guide revision efforts and an assessment of its measures. This would also help bolster the integrity of its decisions.

Drug Courts Were Associated with Lower Recidivism and Relapse Rates for Program Participants Than Criminal Courts

In the evaluations we reviewed, adult drug-court program participation was generally associated with lower recidivism. Our analysis of evaluations reporting recidivism data for 32 programs showed that drug court program participants were generally less likely to be re-arrested than comparison group members drawn from the criminal court system, although the differences in likelihood were reported to be statistically significant in 18 programs. Across studies showing re-arrest differences, the percentages of drug court program participants rearrested were lower than for comparison group members by 6 to 26 percentage points. One program did not show a lower re-arrest rate for all drug-court program participants relative to the comparison group within 3 years of entry into the program, although that study did show a lower re-arrest rate for drug court participants who had completed the program than for members of the comparison group. In general, the evaluations we reviewed found larger differences in re-arrest rates between drug-court program completers and members of the comparison group than between all drug-court program participants and the comparison group members. The rearrest rates for program completers ranged from 12 to 58 percentage points below those of the comparison group. The completion rates reported in the evaluations we reviewed ranged from 15 percent to 89 percent.

Included among the evaluations we reviewed was the MADCE, a 5-year longitudinal process, impact, and cost evaluation of adult drug courts. The MADCE reported a re-arrest rate for drug court participants that was 10 percentage points below that of the comparison group; specifically, 52 percent of drug court participants were re-arrested after the initiation of the drug court program, while 62 percent of the comparison group members were re-arrested. However, the 10 percentage point difference between these rearrest rates for the samples of drug court participants and comparison group members was not statistically significant. The MADCE study also reported that drug court participants were significantly less likely than the comparison group to self-report having committed crimes when they were interviewed 18 months after the baseline (40 percent vs. 53 percent), and drug court participants who did report committing crimes committed fewer than comparison group members.

We assigned a numerical rating to each evaluation to reflect the quality of its design and the rigor of the analyses conducted. Our methodology for rating the evaluation studies is detailed in appendix III. After assigning the rating, we grouped the studies into two tiers. Tier 1 studies were the most carefully designed and incorporated substantial statistical rigor in their analyses. Tier 2 studies, while still meeting our basic criteria for methodological soundness, were relatively less rigorous in their design and analyses. Both tier 1 and tier 2 studies reported differences between drug court participants and comparison group members and both sets of studies found that some but not all differences were statistically significant.

Table 3 shows whether a difference in recidivism rates was reported for each program—expressed as the difference in the rate of re-arrest between all drug court program participants and the comparison group. In some cases the difference in recidivism was reported as something other than a difference in the re-arrest rate, such as a difference in the number of arrests or the relative odds of an arrest. In those cases, table 3 notes that a difference was reported, but does not include the difference in re- arrest rates. For example, the evaluation of the Queens Misdemeanor Treatment Court reported that the re-arrest rate for program participants was 14 percentage points lower than the re-arrest rate of comparison group members up to 2 years after participants entered into the program, and 10 percentage points lower at 3 or more years after entry. Similarly, the evaluation of the Hillsborough County Adult Drug Court reported a statistically significant difference in the relative odds of an arrest after drug court program enrollment but did not report the difference in rearrest rates, therefore table 3 indicates a statistically significant reduction in rearrest rates but does not show the difference in rates.

The evaluations we reviewed showed that adult drug-court program participation was also associated with reduced drug use. Our analysis of evaluations reporting relapse data for eight programs showed that drug court program participants were less likely than comparison group members to use drugs, based on drug tests or self-reported drug use, although the difference was not always significant. This was true for both within-program and post-program measures, and whether drug use was reported as the difference in the frequency of drug use or the proportion of the treatment and comparison groups who used drugs.

The MADCE concluded drug courts produce significant reductions in drug relapse. Specifically, MADCE reported that “drug court participants were significantly less likely than the comparison group to report using all drugs (56 vs. 76 percent) and also less likely to report using ‘serious’ drugs (41 vs. 58 percent), which omit marijuana and ‘light’ alcohol use (fewer than four drinks per day for women or less than five drinks per day for men). On the 18-month oral fluids drug test, significantly fewer drug court participants tested positive for illegal drugs (29 vs. 46 percent). Further, among those who tested positive or self-reported using drugs, drug court participants used drugs less frequently than the comparison group.” Regarding post-drug court program relapses, the MADCE concluded that participation in drug court—along with less frequent drug use among offenders prior to arrest, and the absence of mental health problems— were the strongest predictors of success against relapses. Table 4 summarizes the results of drug-use relapse reported in the evaluations we reviewed.

Of the studies we reviewed, 11 included sufficient information to report a net benefit figure. Of these studies, the net benefit ranged from positive $47,852 to negative $7,108 per participant. The net benefit is the monetary benefit of reduced recidivism accrued to society from the drug court program through reduced future victimization and justice system expenditures, less the net costs of the drug court program—that is, the cost of the program less the cost of processing a case in criminal court. A negative net benefit value indicates that the costs of the drug court program outweigh its estimated benefits and that the program was not found to be cost beneficial. Eight of the studies reported positive net benefits—the benefits estimated to accrue from the drug court program exceeded the program’s net costs. Three of the 11 studies reported negative net benefits. We did not attempt to determine whether the differences in the reported values were because of differences in study methodology or the attributes of the drug courts themselves. The environment in which the drug court operates may also be important. For example, the largest net benefit reported was for Kings County, in which members of the comparison group were incarcerated, in contrast to other programs in which members of the comparison group were given probation, which is less costly. The more costly the alternative, such as incarceration, the more likely a drug court will have positive net benefits. In this case, the study reported that society would accrue $47,852 in benefits relative to conventional court processing.

Table 5 below shows whether, based on the available information, the study was shown to be cost beneficial. It also shows the net benefits per participant of the drug court study. For example, MADCE found that the drug court participants led to a net benefit of $6,208 per participant— within the range of the other studies. The MADCE analysis of costs and benefits is discussed further in appendix II.

Conclusions

During the course of our review, BJA made strides in managing its adult drug court program, including implementation of the GrantStat process and recent revisions to the grantee performance measures. Given that BJA has committed to testing its new measures during this first grantees’ reporting period, enhancements could be made to facilitate this assessment. By documenting how it plans to assess the measures and determine any changes that may be needed and providing the rationale for future revisions, BJA could bolster the transparency and integrity of its decisions. Doing so could also improve the reliability of the data it collects, its usefulness to managers in guiding the program, and the success of its measures.

Recommendation for Execution Action

Recognizing that BJA has recently revised the adult drug-court performance measures and has plans to assess their utility, we recommend that BJA’s Director take the following action to ensure that its revision process is transparent and results in quality and successful metrics to inform management’s key decisions on program operations:  Document key methods used to guide future revisions of its adult drug-court program performance measures. This documentation should include both a plan for how BJA will assess the measures after conclusion of the grantees’ first reporting period and a rationale for why each measure was refined, including a discussion of the scope and nature of any relevant stakeholder comments.

Agency Comments

We provided a draft of this report to DOJ for review and comment. On December 1, 2011, we received written comments on the draft report from DOJ, which are reproduced in full in appendix VIII. DOJ concurred with our recommendation and described actions under way or planned to address the recommendation. DOJ also provided technical comments, which we incorporated as appropriate.

DOJ stated that BJA will continue to document grantee feedback and will ensure that revisions to the measures are documented in accordance with GAO’s best practices standards. In particular, DOJ stated that BJA will document (1) whether the name and definition of the measure is consistent with the methodology used to calculate it; (2) whether the measure is reasonably free from bias; (3) whether the measure meets the expectation of the program; and (4) its rationale for why each performance measure was refined, including the scope and nature of any relevant stakeholder comments. We believe that such actions would improve the reliability of the information collected, its usefulness to managers in making key decisions on program operations, and the success of its measures.

We are sending copies of this report to the Attorney General and interested congressional committees. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

Should you or your staff have any questions concerning this report, please contact me at (202) 512-9627 or by e-mail at maurerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IX.

Appendix I: DOJ Has Fully Implemented Most of Our 2002 Recommendations and Plans to Address the Remaining One

The following provides the current status of the seven recommendations we made in 2002—which have since been closed—on DOJ’s collection of performance data. Specifically, DOJ has fully implemented six of them and partially implemented one. DOJ has plans to fully address the remaining recommendation related to analyzing performance and outcome data collected from grantees and reporting annually on the results. Table 6 reflects this status.

Appendix II: MADCE Is the Most Comprehensive Study of Drug Courts to Date, but Generalizability of Findings May Be Limited

NIJ’s MADCE was conducted by the Urban Institute, Center for Court Innovation, and Research Triangle Institute. Data were collected from 1156 drug court participants in 23 different drug courts in 7 geographic clusters and from a comparison group of 625 drug-involved offenders in six different sites in four geographic clusters. Data collected included: three waves of interviews; drug tests; administrative records on treatment, arrests, and incarceration; court observation and interviews with staff and other stakeholders; and budget and other cost information. The evaluation was designed to address the following four questions: (1) Do drug courts reduce drug use, criminal behavior, and other associated offender problems? (2) Do drug courts generate cost savings for the criminal justice system and other public institutions? (3) Are drug courts especially effective or less effective for certain categories of offenders or program characteristics? (4) Which drug court policies and offender perceptions explain their overall impact?

MADCE Found Reductions in Recidivism and Relapse, but Generalizability May Be Limited

The MADCE’s major findings can be summarized as follows:  Drug courts produce statistically significant reductions in self-reported crime. While both the drug court participants and comparison group participants reported large numbers of crimes in the year preceding the 18-month follow-up, drug court participants reported statistically significantly fewer than the comparison group members. Drug court participants were less likely than members of the comparison group to report committing any crimes (40 percent vs. 53 percent) and drug court participants reported committing fewer crimes in the preceding 12 months than comparison group members (43 criminal acts vs. 88 criminal acts). The difference between the two groups in the probability of an official re-arrest over 24 months was not statistically significant, though the percentage of individuals rearrested was lower for the drug court group than the comparison group (52 percent vs. 62 percent), as was the average number of re-arrests (1.24 vs. 1.64).  Drug courts produce statistically significant reductions in drug use.

Drug court participants were less likely than members of the comparison group to report using any drugs (56 percent vs. 76 percent) and any serious drugs (41 percent vs. 58 percent), and less likely to test positively for drugs at the 18-month follow-up (29 percent vs. 46 percent). Furthermore, the large difference in self-reported relapse rates is evident at 6 months (40 percent vs. 59 percent), so the impact of drug courts on alcohol and other drug use is sustained. The interview data also indicate that among the drug court participants and comparison group members that were using drugs, the drug court participants, on average, were using them less frequently.  Drug court participants reported some benefits, relative to comparison group members, in other areas of their lives. At 18 months, drug court participants were statistically significantly less likely than comparison group members to report a need for employment, educational, and financial services, and reported statistically significantly less family conflict. However, there were modest, non-significant differences in employment rates, income, and family emotional support, and no differences found in experiencing homelessness or depression.  Regardless of background, most offenders who participated in drug courts had better outcomes than offenders who were in the comparison programs. However, the impact of drug courts was greater for participants with more serious prior drug use and criminal histories, and the impact was smaller for participants who were younger, male, African-American, or who had mental health problems.  While the treatment and service costs were higher for drug court participants than treatment and service costs associated with the alternative “business-as-usual” comparison programs, drug courts save money through improved outcomes, according to the researchers, primarily through savings to victims resulting from fewer crimes and savings resulting from fewer re-arrests and incarcerations.

The authors of the study assert that their findings have strong internal validity—that is, that the findings were actually produced by the drug court programs—and external validity—that is, that the findings can be generalized to the population of all drug court participants and potential comparison group members. The claim to strong internal validity is not without merit, given the high response rates, low attrition, propensity score adjustments, and conservative estimates produced by the hierarchical models used. The claim of high internal validity is also supported by the sensitivity analyses undertaken for several outcomes using other models and methods of adjustments that produced little or no change in conclusions. The claim to strong external validity, which relates to the generalizability of the results beyond the sample of courts and comparison sites and specific offenders considered, may be somewhat overstated. The authors note that the 23 drug courts included in the study represent “a broad mix of urban, suburban, and rural courts from 7 geographic clusters nationwide,” but that doesn’t assure that, collectively, the drug courts that were included resemble the hundreds of drug courts that were not included, especially since they were not chosen at random. It also seems unlikely that the six comparison sites from four states are representative of all potential controls, or all alternative programs in all states, and it is potentially problematic that all of the selected sites, including drug court and comparison sites, were alike in their willingness and interest in participating. Those concerns notwithstanding, this is the broadest and most ambitious study of drug courts to date; it is well done analytically; and the results, as they relate to the impact of drug courts, are transparent and well described.

MADCE’s Cost Benefit Analysis Focused on Individuals

The MADCE cost benefit analysis approach differed from most of the other studies we reviewed. In most of the other studies, the average cost and benefit of a drug court participant was compared to the average cost and benefit of normal court processing. In contrast, the MADCE obtained a separate net benefit figure for each individual. The net benefit was obtained by tracking each individual’s use of resources, such as hearings or meetings with case managers, and program outcomes like use of public assistance. The MADCE also tracked each individual’s rates of re- arrest, number of crimes, and time of incarceration. The crimes are multiplied by cost to victims per crime to obtain the cost to society. The difference between the net benefits of the drug court participants and the comparison group were obtained using a hierarchical model similar to the one used for program outcomes. After applying the method, the MADCE found that the drug court participants led to a net benefit of $6,208 to society per participant, as compared to the comparison group. However, due to the variability in the estimate, the study did not find that the net benefits were statistically significant.

The lack of a statistically significant difference may be because of greater variability in the MADCE approach than the approach used in other studies. Specifically, the MADCE did not assume identical costs for each participant. As a result, costs may be higher for individuals who have lower rates of re-arrest, perhaps because those individuals received more treatment. According to the study’s authors, by assuming identical costs for each participant, the standard approach understates the variance in the computed net benefit figure by not including the variability in cost. However, the MADCE authors assumed that the prices of services were consistent across sites by using a weighted average. In contrast, some studies generate site-specific cost figures. In this way, the MADCE approach did exclude one source of variation that is present in some other studies.

In addition to tracking costs and benefits at the individual level, the MADCE also included some effects of drug court participation that some other studies omit. This is consistent with OMB guidance that states that studies should be comprehensive in the benefits and costs to society considered. One of the benefits considered by the MADCE, sometimes omitted elsewhere, is the estimated earnings of the drug court participant. However, it is unclear that the full value of earnings should have been considered a net benefit to society. For example, to be comprehensive, a study should also consider the cost to society of providing that benefit. The net benefit would account for the value of production from this employment less the wages paid. Although in this case, it is unlikely that this would affect the result of the analysis, as the earnings are similar for drug court participants and the comparison group.

Appendix III: Objectives, Scope, and Methodology

To determine what data DOJ collects on the performance of federally funded adult drug courts and to what extent DOJ has used this data in making grant-related decisions, we analyzed the reporting guidance and requirements that BJA provided in fiscal years 2007 through 2011 to grantees applying for Adult Drug Court Discretionary Grant Program funds; BJA-generated grantee performance data reports from October to December 2010; and BJA’s guides for managing grants and enforcing grantee compliance that were issued in fiscal year 2011. We selected 2007 as the starting point for our review because BJA implemented its Performance Measurement Tool (PMT)—an online reporting tool that supports BJA grantees’ ability to collect, identify, and report performance- measurement data activities funded by the grantees’ awards—in fiscal year 2007. We also reviewed our prior reports and internal control standards as well as other academic literature regarding effective performance-management practices. We then used this information and BJA officials’ statements to identify and define six management activities for which performance information can be most useful in making grant- related decisions. Further, we interviewed cognizant BJA officials about the extent to which they use grantees’ performance data when engaging in these management activities, any challenges faced with ensuring grantee compliance, ongoing efforts to revise program performance metrics, and the extent to which BJA’s revisions incorporate best practices we previously identified.

To determine what is known about the effectiveness of adult drug courts in reducing recidivism and substance-abuse relapse rates and what the costs and benefits of adult drug courts are, we conducted a systematic review of evaluations of drug-court program effectiveness issued from February 2004 through March 2011 to identify what is known about the effect of drug court programs on the recidivism of and relapse of drug- involved individuals as well as the costs and benefits of drug courts. We also reviewed DOJ’s NIJ-funded MADCE, a 5-year longitudinal process, impact, and cost evaluation of adult drug courts that was issued in June 2011. We identified the universe of evaluations to include in our review using a three-stage process. First, we (1) conducted key-word searches of criminal justice and social science research databases; (2) searched drug court program-related Web sites, such as those of BJA and NADCP; (3) reviewed bibliographies, meta-analyses of drug court evaluations, and our prior reports on drug court programs; and (4) asked drug court researchers and DOJ officials to identify evaluations. Our literature search identified 260 documents, which consisted of published and unpublished outcome evaluations, process evaluations, commentary about drug court programs, and summaries of multiple program evaluations. Second, we reviewed the 260 documents our search yielded and identified 44 evaluations that reported recidivism or substance use relapse rates using either an experimental or quasi-experimental design, or analyzed program costs and benefits. Third, we used generally accepted social science and cost benefit criteria to review the 44 evaluations.

To assess the methodological quality of evaluations that reported on recidivism or relapse rates, we placed each evaluation into one of five categories, with category 1 evaluations being the most rigorous and category 5 the least, as outlined in table 7.

We excluded studies that were placed in category 5 in the table above or studies in which the comparison group was not drawn from a criminal court. We were left with 33 studies, plus the MADCE, that reported on the effectiveness of 32 drug court programs or sets of programs. As noted in our report, we then grouped the 34 studies, including the MADCE, into two tiers according to their quality category, Tier 1 studies were those that fell into categories 1 or 2, Tier 2 studies were those that fell into categories 3 or 4.

Observed differences in recidivism could arise from measured and unmeasured sources of variation between drug court participants and comparison group members. If comparison group members differed systematically from drug court participants on variables that are also associated with recidivism, such as the degree of their substance-abuse addiction problem and these variables were not accounted for by the design or analysis used in the evaluation, then the study could suffer from selection bias wherein observed differences in recidivism could be because of these sources of variation rather than participation in the drug court program. As indicated in table 7, our evaluation of the methods used to deal with selection bias was reflected in the quality categorization of each study.

To assess the methodological quality of evaluations that reported on drug court program costs and benefits, we assessed them according to the five criteria we developed and outlined in table 8 below.

We determined that an essential criterion for reporting a net benefit of drug courts was that the costs of the drug court were assessed against a baseline (i.e., “business-as-usual” or traditional court processing). Eleven studies met this essential standard and were used to report on program costs and benefits. We excluded other studies not meeting this standard even though they may have met others.

To obtain information on our outcomes of interest—that is, recidivism, substance use relapse, and costs and benefits—we used data collection instruments to systematically collect information about the methodological characteristics of each evaluation, the drug court participants and comparison group members studied, and the outcomes of the participants and other comparable groups reported. Each evaluation was read and coded by a senior social scientist, statistician, or economist with training and experience in evaluation research methods. A second senior social scientist, statistician, or economist then reviewed each completed data collection instrument to verify the accuracy of the information included. Part of our assessment also focused on the quality of the data used in the evaluations as reported by the researchers and our observations of any problems with missing data, any limitations of data sources for the purposes for which they were used, and inconsistencies in reporting data. We incorporated any data problems that we noted in our quality assessments.

We selected the evaluations in our review based on their methodological strength; therefore, our results cannot be generalized to all drug court programs or their evaluations. Although the findings of the evaluations we reviewed are not representative of the findings of all evaluations of drug court programs, the evaluations consist of those evaluations we could identify that used the strongest designs to assess drug-court program effectiveness.

To identify the extent to which DOJ has addressed the recommendations that we made in 2002 regarding drug court programs, we interviewed cognizant DOJ officials and obtained and reviewed documentation (e.g., drug-court program grant solicitations and grantee-performance reporting guidance) on the actions taken to address and implement each of our prior recommendations. We conducted this performance audit from November 2010 through December 2011 in accordance with generally accepted government-auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our objectives.

Appendix IV: Overview of Drug Court Program Characteristics

This appendix provides a general description of drug court program components (see table 9). Drug court programs rely on a combination of judicial supervision and substance-abuse treatment to motivate defendants’ recovery. Judges preside over drug court proceedings, which are called status hearings; monitor defendants’ progress with mandatory drug testing; and prescribe sanctions and incentives, as appropriate in collaboration with prosecutors, defense attorneys, treatment providers, and others. Drug court programs can vary in terms of the substance-abuse treatment required. However, most programs offer a range of treatment options and generally require a minimum of about 1 year of participation before a defendant completes the program.

Appendix V: Ten Key Components of a Drug Court— Developed by BJA in Collaboration with the National Association of Drug Court Professionals

Appendix V: Ten Key Components of a Drug Court— Developed by BJA in Collaboration with the National Association of Drug Court Professionals 1. Integration of substance-abuse treatment with justice system case processing. 2. Use of a non-adversarial approach, in which prosecution and defense promote public safety while protecting the right of the participant to due process. 3. Early identification and prompt placement of eligible participants. 4. Access to continuum of treatment, rehabilitation, and related services. 5. Frequent testing for alcohol and illicit drugs. 6. A coordinated strategy governs drug court responses to participants’ compliance. 7. Ongoing judicial interaction with each participant. 8. Monitoring and evaluation to measure achievement of program goals and gauge effectiveness. 9. Continuing interdisciplinary education to promote effective planning, implementation, and operation. 10. Forging partnerships among drug courts, public agencies, and community-based organizations generates local support and enhances drug court program effectiveness.

Appendix VI: BJA Offers Solicitations in Four Broad Drug-Court Grant Categories— Implementation, Enhancement, Statewide, and Joint

As mentioned, the Adult Drug Court Discretionary Grant Program provides financial and technical assistance to states, state courts, local courts, units of local government, and Indian tribal governments to develop and implement drug treatment courts. There are four different types of awards that BJA makes to adult drug-court grantees through the program. Table 11 provides a description of the grant types.

Appendix VII: Key Management Activities Identified for Which Performance Information Can Be Most Useful

How performance information may be used to support the activity Performance information is used to set priorities in budgeting and to target resources. Agencies can also use this information to identify priorities on which to focus their efforts. For example, targeting grants to address “underserved” client groups.

Performance information is used to compare results of agencies’ programs with goals and to identify where program resources should be targeted to improve performance and achieve goals. When faced with reduced resources, such analyses can assist agencies’ efforts to minimize the impact on program results.

Performance information is used to assess the way a program is conducted and the extent to which a program’s practices and policies have or have not led to improvements in outcomes. Such information is used to identify problems and consider alternative approaches and processes in areas where goals are not being met and to enhance the use of program approaches and processes that are working well.

Performance information is used to identify and increase the use of program approaches that are working well and share these effective processes and approaches with stakeholders.

Performance information is used to establish the targets and goals that grantees are expected to achieve. These targets and goals can be used as the basis for corrective action (e.g., technical assistance, freezing of funds) or to reward high performing grantees.

Performance information is used to compare grantees’ performance results with established targets and goals to determine the extent to which grantees have met them and, if necessary, target program resources (e.g., technical assistance) to improve grantees’ performance.

Appendix VIII: Comments from the Department of Justice, Bureau of Justice Assistance

Appendix IX: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the contact named above, Joy Booth, Assistant Director and Frederick Lyles, Jr., Analyst-in-Charge, managed this assignment. Christoph Hoashi-Erhardt, Michael Lenington, and Jerry Seigler, Jr., made significant contributions to the work. David Alexander, Benjamin Bolitzer, Michele Fejfar, and Doug Sloane assisted with design and methodology. Pedro Almoguera, Carl Barden, Harold Brumm, Jr., Jean McSween, Cynthia Saunders, Jeff Tessin, Susan B. Wallace, and Monique Williams assisted with evaluation review. Janet Temko provided legal support, and Katherine Davis provided assistance in report preparation.

Bibliography

Bouffard, Jeffrey A., and Katie A. Richardson. “The Effectiveness of Drug Court Programming for Specific Kinds of Offenders: Methamphetamine and DWI Offenders Versus Other Drug-Involved Offenders.” Criminal Justice Policy Review, 18(3) (September 2007): 274-293.

Carey, Shannon M., and Michael W. Finigan. Indiana Drug Courts: St. Joseph County Drug Court Program Process, Outcome and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2007.

Carey, Shannon M., and Michael W. Finigan. Indiana Drug Courts: Vanderburgh County Day Reporting Drug Court Process, Outcome and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2007.

Carey, Shannon M., and Michael W. Finigan. “A Detailed Cost Analysis in a Mature Drug Court Setting: A Cost-Benefit Evaluation of the Multnomah County Drug Court.” Journal of Contemporary Criminal Justice, 20(3) (August 2004): 315-338.

Carey, Shannon M., Michael W. Finigan, et. al. Indiana Drug Courts: Monroe County Drug Treatment Court Process, Outcome and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2007.

Carey, Shannon M., Michael Finigan, Dave Crumpton, and Mark S. Waller. “California Drug Courts: Outcomes, Costs and Promising Practices: An Overview of Phase II in a Statewide Study.” Journal of Psychoactive Drugs (November 2006).

Carey, Shannon M., Lisa M. Lucas, Mark S. Waller, Callie H. Lambarth, Robert Linhares, Judy M. Weller, and Michael W. Finigan. Vermont Drug Courts: Rutland County Adult Drug Court Process, Outcome, and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2009.

Carey, Shannon, and Gwen Marchand. Marion County Adult Drug Court Outcome Evaluation-Final Report. Portland, OR: NPC Research, 2005.

Carey, Shannon M. and Mark S. Waller. Oregon Drug Court Cost Study: Statewide Costs and Promising Practices-Final Report. Portland, OR: NPC Research, 2010.

Carey, Shannon M., and Mark Waller. California Drug Courts: Costs and Benefits-Phase III. Portland, OR: NPC Research, 2008.

Carey, Shannon M., and Mark S. Waller. Guam Adult Drug Court Outcome Evaluation-Final Report. Portland, OR: NPC Research, 2007.

Dandan, Doria Nour. Sex, Drug Courts, and Recidivism. University of Nevada, Las Vegas: 2010.

Ferguson, Andrew, Birch McCole, and Jody Raio. A Process and Site- Specific Outcome Evaluation of Maine’s Adult Drug Treatment Court Programs. Augusta, ME: University of Southern Maine, 2006.

Finigan, Michael W., Shannon M. Carey, and Anton Cox. Impact of a Mature Drug Court Over 10 Years of Operation: Recidivism and Costs (Final Report). Portland, OR: NPC Research, 2007.

Gottfredson, Denice C., Brook W. Kearley, Stacy S. Najaka, and Carlos M. Rocha. “How Drug Treatment Courts Work: An Analysis of Mediators.” Journal of Research in Crime and Delinquency, 44(1) (February 2007): 3- 35.

Gottfredson, Denice C., Brook W. Kearley, Stacy S. Najaka, and Carlos M. Rocha. “Long-term effects of participation in the Baltimore City drug treatment court: Results from an experimental study.” Journal of Experimental Criminology, 2(1) (January 2006): 67-98.

Gottfredson, Denice C., Brook W. Kearley, Stacy S. Najaka, and Carlos M. Rocha. “The Baltimore City Drug Treatment Court: 3-Year Self-Report Outcome Study.” Evaluation Review, 29(1) (February 2005): 42-64.

Krebs, C.P., C.H. Lindquist, W. Koetse, and P.K. Lattimore. “Assessing the long-term impact of drug court participation on recidivism with generalized estimating equations.” Drug and Alcohol Dependence, 91(1) (November 2007): 57-68.

Labriola, Melissa M. The Drug Court Model and Chronic Misdemeanants: Impact Evaluation of the Queens Misdemeanor Treatment Court. New York, NY: Center for Court Innovation, 2009.

Latimer, Jeff, Kelly Morton-Bourgon, and Jo-Anne Chrétien. A Meta- Analytic Examination of Drug Treatment Courts: Do They Reduce Recidivism? Ottawa, Ontario: Department of Justice Canada, 2006.

Listwan, Shelley Johnson, James Borowiak, and Edward J. Latessa. An Examination of Idaho’s Felony Drug Courts: Findings and Recommendations-Final Report. Kent State University and University of Cincinnati: 2008.

Logan, T. K., William H. Hoyt, Kathryn E. McCollister, Michael T. French, Carl Leukefeld, and Lisa Minton. “Economic evaluation of drug court: methodology, results, and policy Implications.” Evaluation and Program Planning, 27 (2004):381–396.

Loman, Anthony L. A Cost-Benefit Analysis of the St. Louis City Adult Felony Drug Court. Institute of Applied Research. St. Louis, MO: 2004.

Lowenkamp, Christopher T., Alexander M. Holsinger, Edward J. Latessa. “Are Drug Courts Effective: A Meta-Analytic Review.” Journal of Community Corrections. (Fall 2005): 5-28.

Mackin, Juliette R., Shannon M. Carey, and Michael W. Finigan. Harford County District Court Adult Drug Court: Outcome and Cost Evaluation. Portland, OR: NPC Research, 2008.

Mackin, Juliette R., Shannon M. Carey, and Michael W. Finigan. Prince George’s County Circuit Court Adult Drug Court: Outcome and Cost Evaluation. Portland, OR: NPC Research, 2008.

Mackin, Juliette R., Lisa M. Lucas, Callie H. Lambarth, Mark S. Waller, Shannon M. Carey, and Michael W. Finigan. Baltimore City Circuit Court Adult Drug Treatment Court and Felony Diversion Initiative: Outcome and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2009.

Mackin, Juliette R., Lisa M. Lucas, Callie H. Lambarth, Mark S. Waller, Theresa Allen Herrera, Shannon M. Carey, and Michael W. Finigan. Howard County District Court Drug Treatment Court Program Outcome and Cost Evaluation. Portland, OR: NPC Research, 2010.

Mackin, Juliette R., Lisa M. Lucas, Callie H. Lambarth, Mark S. Waller, Theresa Allen Herrera, Shannon M. Carey, and Michael W. Finigan. Montgomery County Adult Drug Court Program Outcome and Cost Evaluation. Portland, OR: NPC Research, 2010.

Mackin, Juliette R., Lisa M. Lucas, Callie H. Lambarth, Mark S. Waller, Theresa Allen Herrera, Shannon M. Carey, and Michael W. Finigan. Wicomico County Circuit Court Adult Drug Treatment Court Program Outcome and Cost Evaluation. Portland, OR: NPC Research, 2009.

Mackin, Juliette R., Lisa M. Lucas, Callie H. Lambarth, Mark S. Waller, Judy M. Weller, Jennifer A. Aborn, Robert Linhares, Theresa L. Allen, Shannon M. Carey, and Michael W. Finigan. Baltimore City District Court Adult Drug Treatment Court: 10-Year Outcome and Cost Evaluation. Portland, OR: NPC Research, 2009.

Marchand, Gwen, Mark Waller, and Shannon M. Carey. Barry County Adult Drug Court Outcome and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2006.

Marchand, Gwen, Mark Waller, and Shannon M. Carey. Kalamazoo County Adult Drug Treatment Court Outcome and Cost Evaluation-Final Report. Portland, OR: NPC Research, 2006.

Marinelli-Casey, Patricia, Rachel Gonzales, Maureen Hillhouse, Alfonso Ang, Joan Zweben, Judith Cohen, Peggy Fulton Hora, and Richard A. Rawson. “Drug court treatment for methamphetamine dependence: Treatment response and posttreatment outcomes.” Journal of Substance Abuse Treatment, 34(2) (March 2008): 242-248.

Mitchell, Ojmarrh, and Adele Harrell. “Evaluation of the Breaking the Cycle Demonstration Project: Jacksonville, FL and Tacoma, WA.” Journal of Drug Issues, 36(1) (Winter 2006): 97-118.

Piper, R. K., and Cassia Spohn. Cost/Benefit Analysis of the Douglas County Drug Court. Omaha, NE: University of Nebraska at Omaha, 2004.

Rhodes, William, Ryan Kling, and Michael Shively. Suffolk County Drug Court Evaluation. Abt. Associates, Inc., 2006.

Rhyne, Charlene. Clean Court Outcome Study. Portland, OR: Multnomah County Department of Community Justice, 2004.

Rossman, S., M. Rempel, J. Roman, et.al. The Multi-Site Adult Drug Court Evaluation: The Impact of Drug Courts. Washington, D.C.: Urban Institute, 2011.

Shaffer, Deborah K., Kristin Bechtel, and Edward J. Latessa. Evaluation of Ohio’s Drug Courts: A Cost Benefit Analysis. Cincinnati, OH: Center for Criminal Justice Research, University of Cincinnati, 2005.

Wilson, David B., Ojmarrh Mitchell, and Doris L. Mackenzie. “A systematic review of drug court effects on recidivism.” Journal of Experimental Criminology, 2(4) (2006): 459-487.

Zarkin, Gary A., Lara J. Dunlap, Steven Belenko, and Paul A. Dynia. “A Benefit-Cost Analysis of the Kings County District Attorney’s Office Drug Treatment Alternative to Prison (DTAP) Program.” Justice Research and Policy, 7(1) (2005).