Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

SAMHSA’s 

Treatment Improvement 

Protocols 

JUNE GIBBS BROWN 

Inspector General 

MARCH 1998 

OEI-07-96-00130 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them This statutory mission is carried out through 

a nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

The OEI’s Kansas City Regional Office prepared this report under the direction of James H. 

Wolf, Regional Inspector General. Principal OEI staff included: 

REGION 

Ray Balandron, Project Leader 

Perry Seaton, Program Analyst 

Deborah Walden, Program Analyst 

Dennis Tharp, Program Analyst 

Janet Miller, Program Analyst 

HEADQUARTERS 

Al Levine, Program Specialist 

Mark Krushat, Director, 

Research and Special Projects 

To obtain copies of this report, please call the Kansas City Regional Office at (8 16) 426-3697. 

Reports are also available on the World Wide Web at our home page address: 

http://www.dhhs.gov/progorg/oei 
EXECUTIVE SUMMARY 

PURPOSE 

To determine the extent of dissemination, and practitioner’s awareness and use of certain 

protocols for the treatment of individuals with alcohol and other drug abuse problems. 

BACKGROUND 

Treatment Improvement Protocols 

Treatment Improvement Protocols (TIPS) are consensus-based “best practice” guidelines 

developed for the Substance Abuse and Mental Health Services Administration (SAMHSA) for 

use in the treatment of individuals with alcohol and other drug problems. Since 1993, 23 TIPS 

have been developed and issued at an estimated average cost of approximately $300,000 each. 

Dissemination of TIPS 

After each publication, SAMHSA disseminates a small number of TIPS to all State Alcohol and 

Substance Abuse Directors, to Addiction Technology Transfer Centers, to special/demonstration 

funded grantees that exist, and to individuals within the Department of Health and Human 

Services. It also provides some at conferences and has made some available on the Internet. 

However, the vast majority are made available through the National Clearinghouse for Alcohol 

and Drug Information (NCADI). SAMHSA provides the bulk of its inventory to the 

clearinghouse which provides them to others by request only. 

Generally speaking, SAMHSA does not undertake any proactive advertising campaign covering 

availability of the TIPS. Except for the inclusion in the NCADI catalog and order form, they are 

not advertised in SAMHSA newsletters, professional publications/brochures, national/state 

medical associations’ publications/journals, etc. 

Methodology 

Our study focused on five specific TIPS selected based on discussions and concurrence with 

SAMHSA policy and executive staff. These five covered topics were methadone treatment, 

pregnant substance-using women, alcohol and other drug abusing adolescents, screening for 

infectious diseases among abusers, and assessing and treating patients with coexisting mental 

illness and alcohol and other drug abuse. 

To determine the extent of dissemination, we chose to survey a random sample of 770 health care 

providers representing a broad range of provider types. Our goal was to get an extended view of 

the knowledge, use of, and interest in TIPS. In addition, we wanted to provide SAMHSA with 

baseline information by which the agency could measure the extent or success of future 

dissemination efforts. We identified four provider groups to survey: (1) SAMHSA funded 

grantee service providers, the “target audience” for which TIPS are developed; 

i 
(2) narcotic/methadone treatment providers which administer methadone maintenance treatment 

and must be registered with the Food and Drug Administration (FDA); (3) treatment providers at 

Community Mental Health Centers (CMHCs) which, besides being Federally funded, offer a 

broad range of services including substance abuse counseling and treatment; and (4) physicians 

(hereinafter referred to as “customary providers”) which billed the Civilian Health and Medical 

Program of the Uniformed Services (CHAMPUS) for alcohol and other drug abuse problems. 

This latter group of physicians serve a universal broad population over and above Medicare, 

including children, adults, and adolescents as well as elderly. Our sample sizes were based on 

certain factors which included the expected proportion of providers who would know about TIPS, 

a precision factor and confidence interval, the effective populations of each group, and an 

expected response rate. 

FINDINGS 

Thirty-two percent of the SAMHSA funded grantees reported they were aware of at least one 

of the five TIPS referenced in the survey. 

Of the 137 SAMHSA funded grantees responding to the survey, 44 reported they were familiar 

with at least 1 of the 5 TIPS. Since SAMHSA considers these grantees their “target audience,” it 

was expected that this group would exhibit a much greater awareness of the protocols. 

Eighty-two percent of the grantees aware of TIPS indicated they were also using them in their 

practice. Of the 93 SAMHSA funded grantees that were not aware of TIPS, 74 percent reported 

that on becoming aware of them they believe they could be useful in their practice. Generally, 

the positions held by the respondents were of six types. These included 97 Program Directors, 

12 Program Coordinators, 10 Counselor/Registered Nurses, 4 Administrative Directors, 

3 President/Vice President/Owners, 2 Team Leader/Interns, and 9 who did not provide their 

position. 

Eighty-six percent of the FDA narcotic/methadone treatment providers responded that they 

were aware of at least one of the five TIPS referenced in the survey. 

Sixty of the 70 FDA narcotic/methadone treatment providers responding to the survey reported 

that they were aware of at least 1 of the 5 TIPS. The broad familiarity was with the single 

protocol covering methadone treatment. Such a high level of awareness may well be due to the 

fact that these providers administer strictly methadone treatment. Also, 52 of the 60 (87 percent) 

indicated they were using this protocol in their practice. The ten providers not aware of any TIPS 

reported that on becoming aware of them they believe they could be useful in their practice. 

Thirty-two percent of Community Mental Health Centers reported they were aware of at least 

one of the five TIPS referenced in the survey. 

Of the 125 CMHCs responding to the survey, 40 reported they were aware of at least 1 of the 

5 TIPS as well as others in general. Thirty-six (90 percent) of them responded that they were also 

using them in their practice. Eight-one percent (69) of the 85 centers that were not aware of TIPS 

indicated that on becoming aware of them they believe they could be useful in their practice. 

ii 
This high percentage of potential interest in TIPS by CMHCs shows that there are alcohol and 

other drug abuse treatment providers outside SAMHSA’s “target audience” that, if aware, would 

make use of them 

Four percent of the “customary provider” group responded that they were aware of at least 

one of the five TIPS referenced in the survey. 

Only 6 of the 142 “customary provider” group responding to the survey reported that they were 

aware of at least 1 of the 5 protocols as well as TIPS in general. Since these physicians serve a 

general broad population and provide a wide range of health care services, it was not surprising 

that they would be generally unaware of them Of the 6 aware of TIPS, 2 indicated that they were 

using them as part of their practice. Of the 136 “customary providers” unaware of TIPS, 44 

(32 percent) expressed an interest in TIPS after learning of them 

RECOMMENDATIONS 

SAMHSA should take a more proactive approach to advertising the availability of all past and 

future TIPS. 

Since less than half of the SAMHSA funded grantees that responded were aware of TIPS, this 

indicates that greater effort needs to be made to raise their level of awareness of them However, 

their usefulness is supported by our data which shows that 84 percent of all survey respondents 

that were aware of any of the five TIPS were using them in their practice. While SAMHSA has 

made positive strides in disseminating the protocols, such as putting some on the Internet, 

distributing them at alcohol and other drug abuse conferences/seminars, and providing them to 

all State Alcohol and Substance Abuse Directors, there are additional awareness mechanisms that 

could be considered. Examples are advertising TIPS in pertinent publications, brochures, and 

professional journals, etc. They could also be noted in periodic newsletters/catalogs published by 

SAMHSA. 

SAMHSA should consider expanding their “target audience.” 

In the process of expanding efforts in advertising TIPS, SAMHSA should look beyond their 

“target audience.” It should consider including at minimum CMHCs as they develop future TIPS 

and advertise its current ones. Our survey results show that 81 percent of CMHCs that were 

previously not aware of any of the 5 TIPS, now had an interest in using them in their practice. 

This compares to 74 percent of SAMHSA funded grantees. Also, 32 percent of “customary 

providers” reported now that they are aware of TIPS they believe the TIPS could be useful. We 

believe this information infers that providers not commonly considered alcohol and other drug 

abuse treatment providers do have an interest in them and their application within their practice. 

. . . 

111 
AGENCIES COMMENTS AND OIG RESPONSE 

We received formal written comments from the SAMHSA Administrator and Assistant Secretary 

for Planning and Evaluation (ASPE) who agreed with our findings and recommendations. 

However, SAMH SA did raise several questions about our sampling methodology. 

In response to SAMHSA’s concern about our sampling technique, we note that our samples were 

scientifically drawn and at the sizes we selected, our precision is within six percent of each of the 

estimates. We have modified our report to include the confidence intervals for each of our main 

point estimates. 

SAMHSA also raised a concern about including treatment programs in our population that were 

never targeted as the primary audience for the TIPS. The comments refer to a list of providers 

provided by SAMHSA’s Office of Applied Studies. It was this office that had recommended that 

we use this list as it represented the “target population” for dissemination of TIPS. Nevertheless, 

the concern raised seemed important enough to review. We followed up on this by re-examining 

respondents information provided in our survey. Out of 137 respondents, all but 12 appeared to 

be potential users of the TIPS. The 12 possible exceptions identified themselves as prevention 

rather than treatment programs. In fact, we called six of these prevention respondents and three 

indicated an interest in TIPS. Overall, we believe the list was an appropriate one to have used for 

our analysis. 

We recognize that the agency is undertaking its own evaluation and, towards that effort, we hope 

that our results will be helpful in planning and implementation. However, we believe that our 

recommendations of wider advertising and expansion of the target audience could be 

implemented before the larger study is completed. 

We have attached actual comments from SAMHSA and ASPE in Appendix B. 

iv 
PAGE 

TABLE OF CONTENTS 

EXECUTIVE SUMMARY 

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 

FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...6 

SAMHSAfundedgrantees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...6 

FDA narcotic/methadone treatment providers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 

Community Mental Health Centers ............................................ 6 

CHAMPUS “customary providers” ........................................... .7 

RECOMMENDATIONS .................................................... 8 

AGENCIES COMMENTS AND OIG RESPONSE .............................. 10 

APPENDICES 

NonrespondentAnalysis.. .............................................. ..A- 1 

Agencies Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B- 1 
INTRODUCTION 

PURPOSE 

To determine the extent of dissemination, and practitioner’s awareness and use of certain 

protocols for the treatment of individuals with alcohol and other drug problems. 

BACKGROUND 

Treatment Improvement Protocols 

Treatment Improvement Protocols (TIPS) are consensus-based “best practice” guidelines 

developed by the Quality Assurance and Evaluation Branch of the Substance Abuse and Mental 

Health Services Administration’s (SAMHSA) Center for Substance Abuse Treatment (CSAT). 

They are developed to promote the transfer of state-of-the-art protocols and guidelines for the 

treatment of alcohol and other drug abuse from acknowledged clinical, research, and 

administrative experts to the Nation’s alcohol and other drug abuse treatment resources. 

The process begins with the recommendation of an alcohol and other drug abuse problem topic 

for consideration by a panel of experts including clinicians, researchers, and program managers, 

as well as professionals in such related fields as social services or criminal justice. Following the 

selection of a topic, CSAT forms a Federal resource panel to review the state-of-the-art in 

treatment and program management. Recommendations from this panel are then transferred to a 

second panel consisting of non-Federal experts who are very familiar with the topic. This group, 

known as a non-Federal consensus panel makes recommendations, defines protocols, and arrives 

at agreement on protocols. Its members represent alcohol and other drug abuse treatment 

programs, hospitals, community health centers, counseling programs, criminal justice and child 

welfare agencies, and private practitioners. The panel chairperson is charged with the 

responsibility for ensuring the resulting protocol reflects true group consensus. 

Next, an evaluation of the proposed guidelines and protocol must be made by a third group 

whose members serve as expert field reviewers. Following review of their recommendations and 

responses, the document is approved for publication. The published TIP reflects alcohol and 

other drug abuse treatment guidelines to be used for provision of high quality and innovative 

treatment in public and private programs. 

Since 1993,23 TIPS have been developed and issued. SAMHSA estimates the average cost for 

developing each TIP to be approximately $300,000, not including printing costs. This study 

focused on five specific TIPS. The five TIPS were selected based on discussions and concurrence 

with SAMHSA policy and executive staff. The five TIPS and year of publication are listed in the 

following table. 

1 
FIVE TIPS REFERENCED IN SURVEY 

1 State Methadone Treatment Guidelines 1993 II 

2 Pregnant, Substance-Using Women 1993 

4 Guidelines for the Treatment of Alcohol and Other Drug Abusing 1993 

Adolescents 

6 Screening for Infectious Diseases Among Substance Abusers 1993 

9 Assessment and Treatment of Patients with Coexisting Mental 1994 

Illness and Alcohol and Other Drug Abuse 

Besides being some of the earliest TIPS published, it is believed that these particular protocols 

cover a wide range of treatment types as well as provider types and should have a better chance 

of being known by alcohol and other drug treatment providers. 

Dissemination of TIPS 

After each publication, SAMHSA disseminates approximately 300 TIPS to a number of 

individuals or components within the Department of Health of Health and Human Services, all 

State Alcohol and Substance Abuse Directors, the Addiction Technology Transfer Centers, and 

to any special or demonstration funded grantees that exist. There is no direct mailing of TIPS to 

the approximate 13,000 SAMHSA block grant/State funded grantees. The SAMHSA also 

provides TIPS at conferences and seminars and has made several of the TIPS available on the 

Internet. In the future it is their intent to have all of them on the Internet. All remaining 

inventory of TIPS is forwarded to the National Clearinghouse for Alcohol and Drug Information 

(NCADI) which is responsible for all other distribution of the protocols, though done by request 

only. The NCADI does include TIPS in it’s inventory catalog and in their brochure/order form 

which are also available upon request. The SAMHSA pays approximately one million dollars 

per year to the clearinghouse for this effort. 

The SAMHSA does not undertake any proactive advertising campaign covering availability of 

the TIPS. Except for the inclusion in the NCADI catalog and it’s order folm, TIPS are not 

referenced in periodic SAMHSA newsletters, professional publication/brochures, national/state 

medical associations’ publications/journals, etc. At the time this study was initiated, there was an 

ample supply of the five protocols included in our study as well as the other TIPS. For the 

specific five protocols focused on in this study, inventory as of October 1997 ranged from TIP #l 

with 13,000 copies to TIP #4 with 55,500 copies. The table below shows the quantity printed 

and the inventory available in October 1997 for each TIP in the study. 

2 
TIPS INVENTORY 

TIP # Quantity Printed No. Available in October 1997 

TIP # 1 51,095 13,000 

TIP # 2 99,852 41,208 

TIP # 4 101,170 55,500 

TIP # 6 99,816 5 1,405 

TIP # 9 98,806 32,150 

THE OBJECTIVES OF THIS REPORT 

The purpose of this report is to provide SAMHSA with a better understanding of the extent to 

which TIPS are actually disseminated throughout the alcohol and other drug abuse treatment 

health care community and other providers by determinin g the level of awareness of these 

protocols among the provider population. In addition, we want to advise SAMHSA to what 

extent the TIPS are being incorporated into providers’ practices. We have not evaluated the 

effectiveness of the guidelines used by any of the providers or distinguished the strengths and 

weaknesses of the individual TIPS. 

Another objective of this report is to identify the potential for providers’ interest after being made 

aware of them through this study. Additionally, we are identifying information by which 

SAMHSA can measure or evaluate the necessity for the expansion of future activities through 

expanded dissemination and advertising their availability. 

METHODOLOGY 

To determine the extent of dissemination and SAMHSA’s advertising techniques, we chose to 

survey a broad sample of health care providers to get an extended view of the knowledge, use of, 

and interest in TIPS for treatment of alcohol and other drug abuse problems. We identified four 

provider groups to survey: (1) SAMHSA funded grantee service providers, the “target audience” 

for which TIPS are developed; (2) narcotic/methadone treatment providers which administer 

methadone maintenance treatment and must be registered with the Food and Drug 

Administration (FDA); (3) treatment providers at Community Mental Health Centers (CMHCs) 

which offer a broad range of services including substance abuse counseling and treatment; and 

(4) physicians, hereinafter referred to as “customary providers,” which had billed the Civilian 

Health and Medical Program of the Uniformed Services (CHAMPUS) for alcohol and other drug 

abuse problems. These “customary providers” serve a universal broad population over and above 

Medicare, including children, adults, and adolescents as well as elderly. 

3 
Some providers were of a specialty that would unquestionably treat alcohol and other drug abuse 

problems whereas others were of an unrelated specialty but received some payment for treating 

these type problems. Therefore, the extent to which these four provider types treated alcohol and 

other drug abuse problems could vary substantially. We believe that these four provider types 

ensured we had the broadest possible representative sample groups of alcohol and other drug 

abuse service providers. 

The SAMHSA funded grantee listing, which numbered 13,394, was provided to us by 

SAMHSA’s Office of Applied Studies. Since this group of alcohol and other drug abuse 

treatment providers is SAMHSA’s “target audience” when developing TIPS, SAMHSA strongly 

suggested that they be included in our study. We randomly sampled 198 of these providers. 

The FDA narcotic treatment providers which administer strictly methadone maintenance 

treatment must register with the FDA. Additionally, because of the specificity of the treatment 

provided by this group, they were also suggested by SAMHSA to be included in our study. 

While these narcotic treatment providers are not specifically a subset of the SAMHSA funded 

grantees, there is a substantial amount of overlap of these groups. Of the 70 FDA narcotic 

treatment providers responding to the survey, 58 (74 percent) were also included in the 

SAMHSA funded grantees total listing. Our universe of FDA narcotic treatment providers 

numbered 862 and we randomly sampled 110 of them 

Community Mental Health Centers were selected because of their Federal funding and broad 

range of services, including individual and group counseling, psycho social rehabilitation, 

therapeutic education, life skills training, day programs, and substance abuse counseling and 

treatment. Our universe included 549 CMHC’s, of which we randomly sampled 163. We were 

particularly interested in CMHCs’ possible awareness of and interest in TIPS. 

Our fourth group, which included a listing of 8,486 “customary providers” from CHAMPUS was 

selected to determine if they were being overlooked as health care providers having a possible 

interest in TIPS. According to CHAMPUS, these physicians had submitted at least one claim for 

diagnostic codes related to alcohol and other drug abuse treatment. We randomly sampled 

299 physicians from this “customary provider” group. 

Each of these 770 health care providers was mailed an individual survey. Overall, 62 percent of 

our sample responded to the survey. We were unable to perform a nonrespondent analysis for the 

CMHC, FDA, or “customary provider” groups. We did, however, perform a nonrespondent 

analysis for the SAMHSA funded grantee provider group and concluded that there is minimal 

effect due to the nonresponders (see appendix A for details on our analysis). The number of 

surveys mailed to each provider type and the number of responses received is shown in the 

following table. 

4 
Number of TIPS Surveys Mailed 

and Responses Received 

by Provider Type 

Provider Type No. Mailed No. Responded Percentage 

SAMHSA 198 137 69% 

FDA 110 70 64% 

CMHC 163 125 77% 

Total 770 474 62% 

Our sample sizes were based on certain factors which included the expected proportion of 

providers who would know about TIPS, a precision factor and confidence interval, the effective 

populations of each provider type, and an expected response rate of sixty percent. Our survey 

document included both closed- and open-ended questions which were primarily specific to the 

five protocols selected for the study. A follow-up survey was mailed to each provider not 

responding to our initial mailing. 

We conducted our review in accordance with the Quality Standards for Inspections issued by 

the President’s Council on Integrity and Efficiency. 

5 
FINDINGS 

Thirty-two percent of the SAMHSA funded grantees reported they were aware of at least one 

of the five TIPS referenced in the survey. 

Of the 137 SAMHSA grantees responding to the survey, 44 reported they were familiar with at 

least 1 of the 5 TIPS referenced in the survey. At the 90 percent confidence interval, this 

produced an actual value estimated to lie between 25 and 39 percent. This is increased to 

36 percent if we include those that reported they were aware of them in general. Since SAMHSA 

considers these grantees as their “target audience,” it was expected that this group of providers 

would exhibit a much greater awareness of the protocols. Eighty-two percent of the grantees 

aware of TIPS reported they were also using them in their practice. Of the 93 SAMHSA grantee 

respondents that were not aware of any TIPS, 74 percent reported that on becoming aware of 

them through our survey, they believe they could be useful in their practice. Generally, the 

positions held by the respondents were of six types. These included 97 Program Directors, 

12 Program Coordinators, 10 Counselors/Registered Nurses, 4 Administrative Directors, 

3 President/Vice President/Owners, 2 Team Leader/Interns, and 9 who did not provide their 

position. 

Eighty-six percent of the FDA narcotic/methadone treatment providers responded that they 

were aware of at least one of the five TIPS referenced in the survey. 

Sixty of the 70 FDA narcotic/methadone treatment providers responding to the survey reported 

that they were aware of at least 1 of the 5 TIPS referenced in the survey, with familiarity being 

specifically with protocol #1 which covers methadone treatment guidelines. At the 90 percent 

confidence interval, the actual value was estimated to lie between 78 and 93 percent. We believe 

that this high level of awareness of TIPS may well be due to the fact that these narcotic providers 

specifically administer methadone treatment and would therefore be aware of the TIP covering 

this topic. 

Fifty-two of the 60 (87 percent) FDA narcotic/methadone treatment providers aware of TIP # I 

indicated that they were using it in their practice. Although not specifically included in the 

SAMHSA “target audience,” both groups are using TIPS at similar levels. 

The 10 FDA narcotic/methadone treatment providers not aware of TIPS reported that on 

becoming aware of them through the survey, they believe they could be useful in their practice. 

Thirty-two percent of community mental health centers reported they were aware of at least 

one of the five TZPs referenced in the survey. 

Of the 125 CMHCs responding to the survey, 40 reported that they were aware of at least 1 of the 

5 TIPS referenced in the survey as well as other TIPS in general. At the 90 percent confidence 

interval for this sample, the actual value was estimated to lie between 26 and 38 percent. Of the 

40 respondents aware of the 5 TIPS, 36 (90 percent) were using them in their practice. 

Eighty-one percent of the 85 CMHCs that reported they were not aware of TIPS indicated that on 

6 
becoming aware of them through the survey, they believe TIPS could be useful in their practice. 

This high percentage of interest in these protocols by CMHCs shows that there are alcohol and 

other drug abuse treatment providers outside SAMHSA’s “target audience” that would use them 

if they were aware of them 

Four percent of the “customary provider” group responded that they were aware of at least 

one of the five TIPS referenced in the survey. 

Six of the 142 “customary provider” group responding to the survey reported that they were 

aware of at least 1 of the 5 protocols referenced in the survey as well as TIPS in general. At the 

90 percent confidence interval, the actual value was estimated to lie between 1 and 7 percent. 

Since these physicians serve a general broad population and provide a broad range of health care 

services, it was expected that they would be generally unaware of them Of the 6 “customary 

providers” aware of TIPS, 2 (33 percent) indicated that they were using them as part of their 

practice. Also, of the 136 “customary providers” reporting they were unaware of TIPS, 

44 (32 percent) expressed an interest in TIPS after learning of them 

7 
RECOMMENDATIONS 

We conclude from survey results that there is a need to raise the level of awareness of TIPS 

throughout the alcohol and other drug abuse treatment health care community. Our data shows a 

potential use of them by other than the SAMHSA funded grantees and a substantial interest in 

TIPS by the various provider types once they become aware of them In addition, our survey has 

provided baseline information by which SAMHSA can measure the extent or success of future 

dissemination efforts. Overall, we would conclude that the ample inventory of TIPS which 

currently exists permits SAMHSA the potential to look at various alternatives to enhance 

awareness of providers who serve the alcohol and other drug abuse patient. 

SAMHSA should take a more proactive approach to advertising all past and future TIPS. 

SAMHSA should take a more proactive approach to making all possible alcohol and other drug 

abuse treatment providers aware of TIPS. Since less than half of the SAMHSA targeted provider 

group were aware of them this indicates that greater effort needs to be made to raise their 

awareness level. This is also supported by our data showing that 84 percent of all survey 

respondents that were aware of any of the five TIPS were also using them in their practice. 

It is understood that direct mailings to all possible alcohol and other drug abuse treatment 

providers is not feasible simply because demand would be greater than supply. However, 

inventories are certainly large enough to send them to all SAMHSA grantee service providers, its 

principal “target audience.” Furthermore, with a proactive information campaign many of those 

unaware of TIPS could learn about them and where applicable to their practice, could request 

them Some respondents unaware of TIPS indicated that our survey served as a good mechanism 

to make them aware of the protocols. Suggestions made by respondents included advertising 

TIPS through pertinent publications, brochures, professional journals, etc. TIPS could be 

referenced in periodic newsletters and/or catalogs published by SAMHSA. It could also advise 

the various pertinent national/state medical associations for their notification to members. Such 

an expanded communication network should build upon SAMHSA’s efforts to produce a 

valuable treatment tool and should raise providers’ awareness of the treatment protocols. 

SAMHSA should also continue to expand its efforts in such areas as putting TIPS on the Internet 

and distributing them at National and Regional alcohol and other drug abuse 

conferences/seminars. 

SAMHSA should consider expanding their “target audience. )’ 

In the process of expanded efforts in advertising TIPS, SAMHSA should look beyond their 

“target audience.” It should consider including at minimum CMHCs as they develop future TIPS 

and advertise its current ones. Our survey results show that 81 percent of CMHCs that were 

previously not aware of any of the 5 TIPS, now had an interest in using them in their practice. 

8 
This compares to 74 percent of SAMHSA funded grantees. Also, 32 percent of the “customary 

physicians” reported now that they are aware of TIPS they believe they could be useful. We 

believe the data clearly supports that other health care providers not commonly considered 

alcohol and other drug abuse treatment providers do have an interest in them and their 

application within their practice. 

9 
AGENCIES COMMENTS AND OIG RESPONSE 

We received formal written comments from the SAMHSA Administrator and Assistant Secretary 

for Planning and Evaluation (ASPE) who agreed with our findings and recommendations. 

However, SAh4HSA did raise several questions about our sampling methodology. 

OZG Response to SAMHSA Comments 

In response to SAMHSA’s concern about our sampling technique, we note that our samples were 

scientifically drawn and at the sizes we selected, our precision is within six percent of each of the 

estimates. We have modified our report to include the confidence intervals for each of our main 

point estimates. 

SAMHSA also raised a concern about including treatment programs in our population that were 

never targeted as the primary audience for the TIPS. The comments refer to a list of providers 

provided by SAMHSA’s Office of Applied Studies. It was this office that had recommended that 

we use this list as it represented the “target population” for dissemination of TIPS. Nevertheless, 

the concern raised seemed important enough to review. We followed up on this by re-examining 

respondents’ information provided in our survey. Out of 137 respondents, all but 12 appeared to 

be potential users of TIPS. The 12 possible exceptions identified themselves as prevention rather 

than treatment programs. In fact, we called six of these prevention respondents and three 

indicated an interest in TIPS. Overall, we believe the list was an appropriate one to have used for 

our analysis. 

We recognize that the agency is undertaking its own evaluation and, towards that effort, we hope 

that our results will help in planning and implementation. However, notwithstanding the agency 

evaluation, we believe that our specific recommendations of wider advertising and expansion of 

the target audience could be implemented before the larger study is completed. 

We have attached actual comments from SAh4HSA and ASPE in Appendix B. 

10 
APPENDIX A 

NONRESPONDENT ANALYSIS 

We conducted an analysis to determine if the survey’s nonrespondents are different from the 

respondents. If the nonrespondents are significantly different, bias may be introduced into the 

results. We used data file information on the 198 facilities included in the SAMHSA funded 

grantee sample group. We analyzed the facilities with respect to their number of active clients, 

whether they provide substance abuse treatment, and whether they provide methadone to their 

clients, all as of October 1, 1995. These are all categorical variables and were tested using the 

&i-square test statistic. 

We found the results are not biased with respect to the size of the facility or whether the facility 

provides substance abuse treatment. The &i-square test statistic was not significant in the 

analysis of these two variables. 

The methadone facilities are under represented being only approximately 5 percent of the total 

surveys and 10 percent of the nonrespondents. This difference is statistically significant. 

However, we believe that this difference does not materially affect our conclusions. 

The following tables demonstrate the results of our analysis. Included in the tables are the 

number of respondents and nonrespondents that fall within each of the variable categories. The 

&i-squared test statistic and the associated degrees of freedom are noted on each table. 

SIZE OF FACILITY 

I

Size of Facility Respondents % Nonrespondents % Total 

0 - 18 Active Clients 30 27% 10 19% 40 

19 - 39 Active Clients 28 25% 13 25% 41 

40 - 108 Active Clients 31 27% 12 22% 43 

109 - 109+ Active Clients 24 21% 18 34% 42 

Total 113 100% 53 100% 166 

(Xi-squared test statistic=3.512 

Degrees of freedom=3 

p-value=0.3 19 

A-l 
SUBSTANCE ABUSE TREATMENT 

Facility Provides 

Substance Abuse Respondents % Nonrespondents % Total 

Yes 

No 22 16% 9 14% 31 

Tntnl l?h inn% 62 1I)l)% 198 

Chi-squared test statistic=0.089 

Degrees of freedom= 1 

p-value=0.766 

METHADONE TREATMENT 

Facility Provides 

Methadone Treatment Respondents % Nonrespondents % Total 

Yes 4 3% 6 10% 10 

No 

Don’t know 

Non-treatment 21 16% 5 8% 26 

Tntfll 134 100% 62 100% 196 

(X-squared test statistic=16.560 

Degrees of freedom=3 

p-value<O.OO 1 

A-2 
APPENDIX B 

AGENCIES COMMENTS 

B-l 
B-2 
B-3 
DEPARTMENT OF HXAL,TH A r$UtlAN .WWKXS Public Health SWVIC~ 

Substance Abuse and Mental 

JAN I 3 I998 Hsakh Sewicm Administration 

Rockville MD 20857 

To: June Gibbs 3rown 

Inspector General 

From: Administrator, SAMHSA 

Subject: ~~,~7~$rt~~A’s Treatment Improvement Protocols” 

Thank you for t@eopportunityo comment on the subject drafl repo.rt. The SubstanceAbuse and

Mental Health Services Administration (SAMHSA), through its Cenkr for Substance Abuse 

Treatment(CSA‘I’),is proudof its ruIe in the dcvcIapment of the Rcatment Improvement 

Protocolsms) that have,oyer the last five-years, produced 24 consens~b~ doments 

relakd to improvingpublkly-fbndedsubstanceabuse treatment servicesand systems. 

With regard to the report developed by the Office of the ‘Spector GenemI (OIG),SAMHSA 

expressed its concerns pi& to the study, and reiterates again, that the primary sanlple of tided 

grantees drawn to determinethe impactand usefihxss of the TIPS was extrao~y small 

(198of 13,349prhidcra), withmanyof theSBLY#Aprogramsconsistingof alcohol-only, 

prevention-only, ormistxllaneous potential ;rec@ients (e.g., Boy’s Clubs/Gjri’s Clubs, a 

penitentiary, a medical school, a research institute, and a church group) which were IXYQ 

targeted as the primary audience for tliese treatmentkiented protocols. The result was a lower 

than anticipakd positive response (32 percent) to the TIPS. Converdy, where CSAT 

specifically directed a TIP on methadone setices to national narcotic treatment programs, over 

86 percent of responding units were aware of this protocol. 

Overall, SkWiSA accepts the conclusions and recommendations of the OIGreport. In fact, 

even prior to the beginning of the study, CSAT had committed its&. to conductkig, and has 

subsequently tiplemented, a 4-year, $32 million study to improve the development and 

dissemination of the ‘llPs. This study will be conducted with attention to appropriate axed 

rigoroussampling, both quantitative and qualitative, with the intention of improving the 

viability, dissemination, and use of these important guidelines and materials. Central to this 

studywillbethe effortto improvethetazgetingof the TIPStQspcciaIizcdpcrpulatioasNIB!!x 

expansion of the target audier~ces,as recommended by the OIG report. 

Thank you again for the opportunity to comment. We look forward to receiving your final 

report. 

Nelba Chqvez, Ph.D. 

B-2 
DEPARTMENT OF HEALTH & HUMAN SERVICES otfrce of the secrsrary 

Was5inglOn, D.C. 20201 

TO: June Gibbs Brown 

Inspector General, DHHS 

FROM: CheqI Austein7?@fG 

Division Director 

Public He&h Policy, ASPE 

DATE: Febrq 6,1998 

SUBJECT: Clearax: of OIG DrafI Report “SAMHSA’s Treatmcxt Improvenient Protocols” 

(OEI-O7-96-00130) 

Thank you for the opporznity to review a draft version of the report evaluating S&HSA’s 

Treatment Improvement Protocol (TIP) series. A mcmbcr of my staff familiar with SMSA 

programs has exzmined the report and concurs with its publication, In ,oencraI, rnv s+kzf!ffound 

the methodoIogicaI and sampling approach reasonable, if not e&rely representative, The 

report’s conclusions and recommendations appear justified and b&d firmIy upon the study 

fmdings. Moreover, my staff is aware that SAMHSA accepts the report’s conclusions and 

rewmmcndations, and & currently implemem?ing a more extensive study of the TLFseries in an 

attempt to improve tic viability, dissemination, and use of these guidelines. it is likely that the 

recommendations or”the report Tom your of&e wiIl provide important inform&n to 

SAMHSA’s cumnt evaluation eEo& in this area 

B-3