Drug Abuse Treatment: Data Limitations Affect the Accuracy of National and State Estimates of Need
Released: 1998-09-15T00:00:00

Letter

Background

In fiscal year 1998, authorized federal funding for drug treatment programs totaled approximately $3.2 billion, with the Department of Health and Human Services (HHS) receiving $1.7 billion. SAMHSA received more than half ($944 million) of HHS’ drug treatment budget. Approximately 80 percent of SAMHSA’s total budget, which includes funding for both drug prevention and treatment, is distributed to states through block grants and formula grant programs. SAMHSA also supports activities that include the administration of NHSDA and STNAP.

Since 1972, NHSDA has provided national estimates of the prevalence of drug use in the U.S. civilian noninstitutionalized population aged 12 years and older. NHSDA, administered by OAS, is an ongoing survey of members of households in the United States on their use of illicit drugs, their nonmedical use of prescription drugs, and their use of alcohol and tobacco products. NHSDA is currently the nation’s most comprehensive survey of drug use. It provides annual information on national trends in the use of substances and data that can be used to analyze patterns of substance use, the size and characteristics of substance use among various special populations, and the populations needing treatment.

To determine the need for treatment, SAMHSA combines various measures of symptoms, problems, and patterns of use included in the NHSDA questionnaire. This information is intended to approximate clinical criteria for drug dependence and to supplement it with other data that indicate treatment need. SAMHSA calculates the number of persons in need of treatment as those who met at least one of the following criteria in the past year: dependence on any illicit drug; heavy drug use (that is, used heroin at least once, used marijuana daily, or frequent use of some other drug); injection drug use of heroin, cocaine, or stimulants; or received drug abuse treatment.

States are also expected to develop estimates of treatment need on a statewide and local basis and report them to CSAT in their block grant applications and through STNAP. Under the 1992 Alcohol, Drug Abuse, and Mental Health Administration Reorganization Act (P.L. 102-321), states are required to use needs assessment data in developing and implementing the plans submitted as part of their block grant applications. Specifically, states are required to develop and report in their block grant applications estimates of treatment need by age, sex, and race or ethnicity for the state as a whole and for each substate planning area. Through STNAP, CSAT provides states with funding and technical assistance to conduct studies to determine the need and demand for substance abuse treatment in relation to the states’ resource availability.

The Government Performance and Results Act was enacted in 1993 in part to improve performance measurement by federal agencies. It requires agencies to set goals, measure performance, and report on their accomplishments. The legislation was enacted to increase program effectiveness and public accountability by having federal agencies focus on results and service quality. SAMHSA developed several performance goals as part of HHS’ 1999 Results Act performance plan. These goals include providing estimates of the prevalence of substance abuse in each of the 50 states and the District of Columbia and increasing to 80 percent the proportion of block grant applications that include needs assessment data developed from STNAP.

NHSDA Limitations Affect the Accuracy of National Estimates of Drug Abuse Treatment Need

Although OAS relies primarily on NHSDA to make national estimates of drug abuse treatment need for the general population and certain subpopulations, the survey has limitations that can lead to underestimates of treatment need. These limitations include the survey’s use of self-reported data; the exclusion of certain high-risk populations; and a sample for some subpopulations, such as pregnant women, that is too small to produce valid estimates. To improve the accuracy of its estimates, OAS adjusted the NHSDA data with data from other sources that are presumed to be more reliable. For example, with this adjustment, OAS estimates of treatment need in 1995 increased by nearly a third. This adjusted estimate, however, is still considered conservative and does not provide subpopulation estimates of treatment need. OAS plans to expand NHSDA (effective in 1999) to further improve the accuracy of drug use and treatment need estimates.

NHSDA Data Can Result in Underestimates of Treatment Need

Several limitations of NHSDA can result in underestimates of treatment need for the general population and subpopulations, such as pregnant women. HHS and the Institute of Medicine have reported on a number of these limitations. For example, NHSDA data are based on self-reports, which rely on respondents’ truth and memory. Although NHSDA procedures were designed to encourage honesty and improve recall, SAMHSA and others assume some degree of underreporting; however, SAMHSA has not adjusted NHSDA data to account for this limitation.

NHSDA also excludes certain populations at high risk for drug use. NHSDA was initially designed as a survey to determine the rate of drug use within U.S. households and as such has excluded drug use by individuals in institutional settings, such as prisons and residential treatment centers, and by those with no permanent residence, including homeless and transient people. As a result, the survey does not include population groups known to have high rates of drug use who are often not in a household environment.

In addition, NHSDA’s sample size for some subpopulations is too small to produce valid estimates. For example, for the 1994-95 survey, OAS reported that the number of women who were pregnant at the time of the interview—770—and reported using illicit drugs—28—was too small to make certain estimates.

OAS’ Adjusted NHSDA Data Provide a Higher Estimate of Treatment Need by Compensating for Some Undercounting

To partially account for NHSDA’s undercoverage of hard-to-reach populations and underreporting of drug use by survey respondents, OAS developed a methodology that substitutes data from sources presumed to be more reliable. Using this methodology, OAS estimated that in 1995, about 8.9 million people in the United States needed drug abuse treatment compared with the 6.9 million estimate—including 2.6 million women—derived solely from NHSDA. While this adjustment results in a treatment need estimate that is about 29 percent higher than the estimate based on only NHSDA data, it still results in conservative estimates of treatment need.

In addition, while OAS’ ratio adjustment was designed to improve the national estimate of treatment need for the general population, it does not estimate treatment need for women and other subpopulations. The ratio adjustment replaces some NHSDA data with information from Uniform Crime Reports (UCR) and the National Drug and Alcohol Treatment Unit Survey (NDATUS), now known as the Uniform Facility Data Set, to estimate treatment need. These data sources provide information on the number of persons arrested, treated for drugs, or both and are presumed to be more reliable. UCR, compiled by the Federal Bureau of Investigation from administrative records of police departments nationally, contains information on arrests and is adjusted for nonresponse and underreporting. NDATUS is a 1-day annual census of all specialty drug abuse and alcohol treatment units nationally. To obtain data on persons treated for drug abuse, approximately 11,800 specialty providers are surveyed on the number and type of patients treated and services received.

This adjustment categorizes NHSDA responses into one of four arrest and treatment groups: arrested and treated, treated but not arrested, arrested but not treated, and not arrested and not treated. According to OAS, the NHSDA estimates appear to significantly underestimate the number in each of the first three categories; to compensate, numbers from UCR and NDATUS are substituted for NHSDA data. The methodology provides only a partial adjustment because any underreporting in the not arrested and not treated category is not affected by the adjustment. Also, the adjustment is still subject to NHSDA limitations. Therefore, according to OAS, the ratio-adjusted estimates represent improved, but conservative estimates of treatment need.

Expanded NHSDA Expected to Produce More Accurate National and State Estimates of Treatment Need

OAS is expanding NHSDA’s sample from 18,000 to 70,000 respondents each year and modifying its methodology to obtain state-level data and better national and subpopulation prevalence estimates. The expanded NHSDA will capture larger samples of youth, racial and ethnic minorities, pregnant women, and hard core drug users, which are expected to result in more accurate subpopulation estimates. The expanded NHSDA is expected to produce comparable state estimates of need annually; however, the sample sizes are not large enough to produce annual substate estimates. According to SAMHSA officials, it will be possible to generate substate estimates by combining multiple years of NHSDA data. While the additional data are expected to result in more precise estimates, treatment need will likely still be underestimated due to the survey’s continued exclusion of certain high-risk populations and reliance on self-reported data.

A major component of the expansion is to allow for estimates for each of the 50 states and the District of Columbia. A regression model OAS developed in 1996 uses NHSDA sample data and local area indicators to estimate state-level drug prevalence and treatment need. However, because of sample size requirements, this methodology only generated estimates for 26 states and 25 metropolitan areas. (See the appendix for a description of this methodology and individual state and metropolitan area estimates.) The expanded sample uses a similar methodology but has been designed to produce direct estimates for the 8 most populous states with smaller samples drawn for the other 42 states and the District of Columbia. The smaller samples will support model-based estimates that use information from the national sample, local indicators derived from the Census Bureau and other sources, and state samples.

The method for collecting information and the content of the NHSDA questionnaire will also be modified under the expansion. Specifically, NHSDA will employ computer-assisted interviewing in 1999, which is expected to minimize respondent errors and partially increase the reliability of self-reporting by building in greater privacy for the respondent. The content of the questionnaire will also be augmented to obtain income and insurance data, national mental health statistics, data on treatment and prevention, and information on crime and other deviant behaviors. The projected annual cost for the expansion is $34 million.

According to SAMHSA officials, the expanded NHSDA will help them identify states with serious drug abuse problems and help target technical assistance and discretionary funds. SAMHSA expects the expanded NHSDA to improve its prevalence estimates of drug abuse in the 50 states and the District of Columbia—one of the goals included in its 1999 performance plan. SAMHSA officials also said that the expanded NHSDA will provide data to monitor the performance of various federal and state agencies engaged in efforts to reduce the supply and demand of illicit drugs. For example, the expanded NHSDA is expected to allow for measurement of the national goal of reducing past month use of illicit drugs among 12- to 17-year-olds by 35 percent by the end of year 2002.

Some experts question the additional cost associated with expanding NHSDA’s sample size to provide state-level estimates. They state that less costly alternatives using modeling techniques that rely on currently available estimates, such as synthetic estimation, could achieve similar goals at a significantly reduced cost. However, SAMHSA officials believe that the approach used in the expanded NHSDA will result in more accurate estimates than those produced using a purely synthetic estimation methodology. They also pointed out that the methodology used for the expanded NHSDA has been tested and validated. However, SAMHSA officials and other experts believe that more validation is needed overall in the methods used to estimate drug abuse treatment need.

State-Reported Data Are Incomplete and of Variable Quality

SAMHSA collects state and local treatment need data through state block grant applications and state reports required under STNAP. Through STNAP, CSAT has provided financial and technical assistance to states to conduct needs assessments. However, while SAMHSA is overseeing state efforts to develop and report estimates of treatment need, not all states have produced such estimates. In addition, CSAT’s monitoring and review of states’ block grant reporting does not ensure the data are complete, accurate, and consistently reported. Our review of needs assessment information in states’ fiscal year 1997 block grant applications found the data to be incomplete and of questionable quality.

While data developed under STNAP have been used as a state resource and planning tool, the program has been limited in developing state in-house capacity and improving states’ reporting in block grant applications, as intended. One of SAMHSA’s goals is to increase the proportion of state block grant applications that include needs assessment data developed under STNAP. However, HHS’ performance plan did not include any information on how SAMHSA will accomplish its goal of increasing state reporting or how it would improve the accuracy of the data reported by states. Further, SAMHSA’s oversight of STNAP does not encourage coordination among CSAT staff providing oversight and technical assistance or strict monitoring of states’ compliance with program requirements.

State Block Grant Reporting of Treatment Need Data Is Incomplete and in Some Cases Inaccurate

More than $1 billion in block grant funds are distributed to states for planning, carrying out, and evaluating activities to prevent and treat substance abuse. States report, as part of their annual block grant applications, information on intended use of federal funds for drug treatment. They are asked to report information on populations, areas, and localities with the greatest need for treatment services and information on the state’s capability to provide treatment. This information is collected to provide SAMHSA with information on how states are using block grant funds and assist states in identifying gaps in services and targeting resources. Although states are required by federal law to report needs assessment information in block grant applications, the data reported does not affect their block grant awards.

Our review of needs assessment information in fiscal year 1997 block grant applications found the data to be incomplete, inaccurate, and inconsistently reported. According to SAMHSA, this is due, in part, to states’ lack of sufficient data and resources to complete the extensive amount of data required in block grant applications. While some states have reported complete information, our review showed that about 25 percent (14 states) did not report on the total population needing treatment and about a third (18 states) did not report information on the total population seeking treatment. In addition, a number of states did not provide information on subpopulations. For example, about 25 percent of states did not report information on women needing treatment, and almost 60 percent did not report information on children and adolescents aged 17 and under needing treatment. We also found inaccuracies in the data reported by states. For example, the number of males and females under age 11 reported needing treatment in one state was greater than the state’s entire population.

Our review of 1997 applications also revealed inconsistencies in states’ reporting of needs assessments, both within a state and across states. For example, some states’ reporting of total women needing treatment on one of the forms in the application was inconsistent with the reporting of that same information—disagreggated by age, sex, and race or ethnicity—on another form in the application. States’ reporting of information is also not consistent across states. States define need differently and employ different methods and databases to estimate need.

Due to the lack of quality of needs assessment data reported in block grant applications, the data have limited use in determining gaps between needs and services available and assuring federal officials that federal funds are being used for the purposes intended. Under block grant regulations, states are required to submit the best available needs assessment data. According to agency officials, the phrase “best available” leaves the agency little basis on which to challenge the data submitted by states in block grant applications. While SAMHSA has not taken the initiative to ensure that accurate, complete, and consistent information is reported in the applications—nor has it validated state estimates or reviewed the methodologies used to develop them—SAMHSA officials expressed concerned about the quality of the data and are in the process of addressing these concerns.

STNAP Has Helped Some States Improve Resource Allocation but Has Been Less Successful in Meeting Its Other Goals

In 1992, CSAT developed STNAP to help states produce better estimates of treatment need and develop plans for use of treatment resources. Between 1992 and 1996, CSAT awarded STNAP contracts to 53 states and territories totaling $59 million. As of June 1998, 23 states and territories had been awarded new contracts, totaling approximately $24 million, to continue activities under a second round of contract awards.

STNAP was designed to develop and maintain a data collection and analysis infrastructure to assist states in surveillance, planning, budgeting, and policy development. STNAP has three primary objectives: (1) assist states in better allocating treatment funds, (2) enhance and sustain states’ in-house capabilities to assess need, and (3) improve states’ reporting in block grant applications. The program has had limited success in meeting its objectives.

According to some state officials, STNAP has been useful in helping states target resources and enhance service delivery. For example, New Jersey reported using prevalence estimates, developed from an STNAP contract, in its allocation formula for distributing alcohol treatment money to better reflect the distribution of need at the county level. Iowa reported using its results to allocate funds based on objective estimates of need, which helped them target outreach efforts that offer the most potential for success. Iowa officials also reported that they used STNAP data to redesign the state’s approach for providing tailored outreach and treatment services for women. Data generated in New Mexico were reportedly used to initiate substance abuse recognition and counseling training in public health offices and create specialized counseling for health care providers to create smoking and alcohol cessation programs for pregnant women.

However, states have been slow in developing in-house capacity to assess need—one of STNAP’s objectives. According to CSAT, most states have been unable to develop sufficient capacity due to inadequate state-level resources and expertise and, as a result, have relied on outside consulting firms, local universities, or both. CSAT officials characterize these relationships as mixed and said that effective contracts with consultants and universities is dependent on the quality of state oversight. While contracts with consultants and universities can limit the development of in-house expertise and result in a lack of continuity and a sustained data infrastructure, they have allowed some states to establish and maintain a knowledge base and network. For example, while Texas and South Carolina used universities for data collection, they used in-house staff expertise for analyses and reporting. To further assist states in developing their in-house capacity to assess need, CSAT contracted with the National Technical Center (NTC) at Harvard Medical School to provide technical assistance.

According to CSAT officials, states’ reporting of results developed under STNAP in block grant applications—the third objective of the program—has not yet been fully realized because most states have not completed their planned data collection and analyses. As of February 1998, 19 states have contracts that have been completed or allowed to expire, with some work remaining on final reports. Although states were initially awarded 3-year contracts, most states received unfunded contract extensions and are taking, on average, 5 years to finish. SAMHSA requires states to incorporate needs assessments developed under STNAP in block grant applications, but SAMHSA has not enforced this requirement for those states that have completed their contracts. Although one of SAMHSA’s performance goals is to increase to 80 percent the proportion of state applications that include STNAP needs assessment data, SAMHSA did not provide any information in the performance plan on how it will increase state reporting or verify the data reported by states in block grant applications.

Individual state estimates developed under STNAP were also originally intended to be used as a basis for developing national estimates of need. However, this goal has been dropped by SAMHSA because of data incomparability across states. Specifically, while states are required to assess need for a core set of abused drugs using clinical definitions of dependence, states have overall flexibility in designing their studies. As a result, states employed different survey instruments and sample sizes that affect the resulting estimates’ comparability.

CSAT’s Management of STNAP Does Not Ensure State Compliance With Some Program Requirements

CSAT’s oversight of STNAP has not ensured timely completion of the contract or compliance with some program reporting requirements. Of the 19 states that completed the contract, only 8 (42 percent) did so within the original 3-year time frame. According to a former state official, the complex data collection and analysis procedures and unrealistic expectations about response rates developed under CSAT’s contract attributed to delays in contract completion. CSAT officials stated that the extended time necessary for states to complete the contract is an indication of a need for more program direction. States are also required to report findings to CSAT—through monthly, annual, and final reports—as part of their contracts and to report STNAP-collected data in block grant applications. Our review found that only 11 states have submitted final reports, and CSAT could only locate 6 of the 11 reports. Further, some states completed the project but did not report data in their block grant applications.

CSAT has not consistently communicated STNAP objectives and requirements to states. Also, CSAT project officers acknowledge that their review of state contracts has been inconsistent and there is little coordination among them. Specifically, CSAT project officers responsible for STNAP oversight and state technical assistance have not coordinated their efforts or taken advantage of experiences and lessons learned from their involvement with different states. CSAT officials acknowledge that stricter monitoring of states’ compliance with program requirements is needed. According to SAMHSA, some changes have been instituted to improve monitoring; however, specific plans of action to achieve these goals have not yet been fully developed.

Conclusions

Reliable assessments of treatment need—at national, state, and local levels and for specific population groups—are an essential component to accurately target treatment services. While SAMHSA has efforts under way to improve its national estimates through the expansion of NHSDA, the survey is still likely to result in an underestimate of treatment need. Also, STNAP’s goals to help states develop estimates of treatment need and improve state reporting of need data have not been fully accomplished. Even though states are required to provide estimates of treatment need as part of their block grant applications, not all states report this information and some of the data reported are inaccurate. SAMHSA recognizes the need to increase state reporting and has set a target for increasing the number of states that provide the information. It also recognizes that the overall quality of the data reported is problematic. However, SAMHSA has not indicated how it will increase state reporting or improve the quality of the data reported by states in block grant applications.

Recommendation

In keeping with its goal of improving state reporting, we recommend that the Administrator of SAMHSA develop an action plan for how the agency will increase states’ reporting of accurate, complete, and consistent treatment need data in block grant applications and include a summary of these actions in HHS’ year 2000 performance plan.

Agency and Other Comments

We provided copies of a draft of this report to SAMHSA and others for review. SAMHSA generally agreed with the report’s findings and with the need for an action plan aimed at improving state reporting of treatment need data as we recommended. While SAMHSA recognized the need for an action plan, it stated that it would be inappropriate to include in a performance plan the level of detail required for an action plan. We did not intend to imply that the performance plan should include extensive detail; however, it should include a discussion of strategies the agency will use to achieve its goals. Accordingly, we modified our recommendation to clarify how action plan information should be reflected in the performance plan. SAMHSA also provided a number of technical comments, which we incorporated where appropriate. We also obtained comments from researchers and experts in the field who were knowledgeable about these issues and incorporated their comments where appropriate.

We are sending copies of this report to the Secretary of HHS, the Administrator of SAMHSA, officials of state substance abuse agencies, appropriate congressional committees, and other interested parties. We will make copies available to others upon request.

Please contact me at (202) 512-7119 or James O. McClyde, Assistant Director, at (202) 512-7152, if you or your staff have any questions. Other major contributors to this report were Ann Calvaresi Barr and Janina Johnson.

OAS’ Regression Analyses and Estimates of Treatment Need for 26 States and 25 Metropolitan Areas

In 1996, OAS developed models for estimating state-level treatment need that use regression analyses combining NHSDA data with local area indicators—such as drug-related arrests, alcohol-related death rates, and Census Bureau data—that were found to be associated with substance abuse. The models produce estimates that are a weighted average of an indirect synthetic regression estimate and a direct survey estimate. Therefore, the models require at least some NHSDA sample data for each area under consideration. A total of 26 states and 25 metropolitan areas met the sample size criteria (at least 300 interviews) required for estimation using these models.

According to OAS, the analysis applies a consistent methodology across states; however, the estimates produced are subject to many of the limitations of NHSDA national estimates. OAS has developed state and selected metropolitan area estimates using this regression analysis for 1991 through 1993. (See tables 1 and 2.) According to an OAS official, OAS is developing state estimates using 1994 through 1996 NHSDA data.

Number (in thousands)

Anaheim-Santa Ana, Calif.

Atlanta, Ga.

Baltimore, Md.

Boston, Mass.

Chicago, Ill.

Dallas, Tex.

Denver, Colo.

Detroit, Mich.

El Paso, Tex.

Houston, Tex.

Los Angeles, Calif.

Miami-Hialeah, Fla.

Minneapolis-St. Paul, Minn.

Nassau-Suffolk, N.Y.

New York, N.Y.

Newark, N.J.

Oakland, Calif.

Philadelphia, Pa.

Phoenix, Ariz.

San Antonio, Tex.

San Bernardino, Calif.

San Diego, Calif.

St. Louis, Mo.

Tampa-St. Petersburg, Fla.

Washington, D.C.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.