Department  of Health  and  Human  Services 
OFFICE  OF 
INSPECTOR  GENERAL 
SAMHSAs 

Treatment  Improvement 

Protocols 

JUNE  GIBBS  BROWN 
Inspector  General 
MARCH  1998 
OEI-07-96-00130  OFFICE  OF INSPECTOR  GENERAL 
The mission  of the Office  of Inspector  General  (OIG),  as mandated  by Public  Law  95452,  is to 
protect  the integrity  of the Department  of Health  and Human  Services  programs  as well as the 
health  and welfare  of beneficiaries  served  by them  This  statutory  mission  is carried  out  through 
a nationwide  program  of audits,  investigations,  inspections,  sanctions,  and fraud  alerts.  The 
Inspector  General  informs  the Secretary  of program  and management  problems  and recommends 
legislative,  regulatory,  and operational  approaches  to correct  them 
Office  of Evaluation  and Inspections 
The  Office  of Evaluation  and Inspections  (OEI)  is one  of several  components  of the Office  of 
Inspector  General.  It conducts  short-term  management  and program  evaluations  (called 
inspections)  that  focus  on issues  of concern  to the Department,  the Congress,  and the public.  The 
inspection  reports  provide  findings  and recommendations  on the efficiency,  vulnerability,  and 
effectiveness  of departmental  programs. 
The  OEIs Kansas  City Regional  Office  prepared  this report  under  the direction  of James  H. 
Wolf,  Regional  Inspector  General.  Principal  OEI  staff included: 
REGION  HEADQUARTERS 
Ray Balandron,  Project  Leader  Al Levine,  Program  Specialist 
Perry  Seaton,  Program Analyst  Mark  Krushat,  Director, 
Deborah  Walden,  Program Analyst  Research  and Special Projects 
Dennis  Tharp,  Program Analyst 
Janet  Miller,  Program Analyst 
To  obtain  copies  of this report,  please  call the Kansas  City Regional  Office  at (8 16) 426-3697. 
Reports  are also available  on the World  Wide  Web  at our  home  page  address: 
http://www.dhhs.gov/progorg/oei  EXECUTIVE  SUMMARY 
PURPOSE 
To  determine  the extent  of dissemination,  and practitioners  awareness  and use  of certain 
protocols  for the treatment  of individuals  with  alcohol  and other  drug  abuse  problems. 
BACKGROUND 
Treatment  Improvement  Protocols 
Treatment  Improvement  Protocols  (TIPS)  are consensus-based  best practice  guidelines 
developed  for the Substance  Abuse  and Mental  Health  Services  Administration  (SAMHSA)  for 
use in the treatment  of individuals  with  alcohol  and other  drug  problems.  Since  1993, 23 TIPS 
have  been  developed  and issued  at an estimated  average  cost  of approximately  $300,000  each. 
Dissemination  of TIPS 
After  each  publication,  SAMHSA  disseminates  a small number  of TIPS to all State  Alcohol  and 
Substance  Abuse  Directors,  to Addiction  Technology  Transfer  Centers,  to special/demonstration 
funded  grantees  that  exist,  and to individuals  within  the Department  of Health  and Human 
Services.  It also provides  some  at conferences  and has made  some  available  on the Internet. 
However,  the vast majority  are made  available  through  the National  Clearinghouse  for Alcohol 
and Drug  Information  (NCADI).  SAMHSA  provides  the bulk  of its inventory  to the 
clearinghouse  which  provides  them  to others  by request  only. 
Generally  speaking,  SAMHSA  does  not  undertake  any proactive  advertising  campaign  covering 
availability  of the TIPS.  Except  for the inclusion  in the NCADI  catalog  and order  form,  they  are 
not  advertised  in SAMHSA  newsletters,  professional  publications/brochures,  national/state 
medical  associations  publications/journals,  etc. 
Methodology 
Our  study  focused  on five specific  TIPS selected  based  on discussions  and concurrence  with 
SAMHSA  policy  and executive  staff.  These  five covered  topics  were  methadone  treatment, 
pregnant  substance-using  women,  alcohol  and other  drug  abusing  adolescents,  screening  for 
infectious  diseases  among  abusers,  and assessing  and treating  patients  with  coexisting  mental 
illness  and alcohol  and other  drug  abuse. 
To  determine  the extent  of dissemination,  we chose  to survey  a random  sample  of 770 health  care 
providers  representing  a broad  range  of provider  types.  Our goal  was to get  an extended  view  of 
the knowledge,  use  of, and interest  in TIPS.  In addition,  we wanted  to provide  SAMHSA  with 
baseline  information  by which  the  agency  could  measure  the extent  or success  of future 
dissemination  efforts.  We identified  four  provider  groups  to survey:  (1) SAMHSA  funded 
grantee  service  providers,  the target  audience  for which  TIPS are developed; 
i  (2) narcotic/methadone  treatment  providers  which  administer  methadone  maintenance  treatment 
and must  be registered  with  the Food  and Drug  Administration  (FDA);  (3) treatment  providers  at 
Community  Mental  Health  Centers  (CMHCs)  which,  besides  being  Federally  funded,  offer  a 
broad  range  of services  including  substance  abuse  counseling  and treatment;  and (4) physicians 
(hereinafter  referred  to as customary  providers)  which  billed  the Civilian  Health  and Medical 
Program  of the Uniformed  Services  (CHAMPUS)  for alcohol  and other  drug  abuse  problems. 
This  latter  group  of physicians  serve  a universal  broad  population  over  and above  Medicare, 
including  children,  adults,  and adolescents  as well as elderly.  Our  sample  sizes were  based  on 
certain  factors  which  included  the expected  proportion  of providers  who  would  know  about  TIPS, 
a precision  factor  and confidence  interval,  the effective  populations  of each  group,  and an 
expected  response  rate. 
FINDINGS 
Thirty-two  percent  of the SAMHSA  funded  grantees  reported  they  were aware  of at least  one 
of the five  TIPS referenced  in the survey. 
Of the  137 SAMHSA  funded  grantees  responding  to the  survey,  44 reported  they  were  familiar 

with  at least  1 of the 5 TIPS.  Since  SAMHSA  considers  these  grantees  their  target  audience,  it 

was expected  that  this group  would  exhibit  a much  greater  awareness  of the protocols. 

Eighty-two  percent  of the grantees  aware  of TIPS indicated  they  were  also using  them  in their 

practice.  Of the 93 SAMHSA  funded  grantees  that  were  not  aware  of TIPS,  74 percent  reported 

that  on becoming  aware  of them  they  believe  they  could  be useful  in their  practice.  Generally, 

the positions  held  by the respondents  were  of six types.  These  included  97 Program  Directors, 

12 Program  Coordinators,  10 Counselor/Registered  Nurses,  4 Administrative  Directors, 

3 President/Vice  President/Owners,  2 Team  Leader/Interns,  and 9 who  did not  provide  their 

position. 

Eighty-six  percent  of the FDA  narcotic/methadone  treatment  providers  responded  that  they 
were aware  of at least  one of the five  TIPS referenced  in the survey. 
Sixty  of the 70 FDA  narcotic/methadone  treatment  providers  responding  to the  survey  reported 

that  they  were  aware  of at least  1 of the 5 TIPS.  The  broad  familiarity  was with  the  single 

protocol  covering  methadone  treatment.  Such  a high  level of awareness  may  well be due  to the 

fact that  these  providers  administer  strictly  methadone  treatment.  Also,  52 of the 60 (87 percent) 

indicated  they  were  using  this protocol  in their  practice.  The  ten providers  not  aware  of any TIPS 

reported  that  on becoming  aware  of them  they  believe  they  could  be useful  in their  practice. 

Thirty-two  percent  of Community  Mental  Health  Centers  reported  they  were aware  of at least 
one of the five  TIPS referenced  in the survey. 
Of the  125 CMHCs  responding  to the  survey,  40 reported  they  were  aware  of at least  1 of the 
5 TIPS as well as others  in general.  Thirty-six  (90 percent)  of them  responded  that  they  were  also 
using  them  in their  practice.  Eight-one  percent  (69) of the  85 centers  that  were  not  aware  of TIPS 
indicated  that  on becoming  aware  of them  they believe  they  could  be useful  in their  practice. 
ii  This high  percentage  of potential  interest  in TIPS by CMHCs  shows  that  there  are alcohol  and 
other  drug  abuse  treatment  providers  outside  SAMHSAs  target  audience  that,  if aware,  would 
make  use  of them 
Four percent  of the  customary  provider  group  responded  that  they  were aware  of at least 
one of the five  TIPS referenced  in the survey. 
Only 6 of the  142 customary  provider  group  responding  to the  survey  reported  that  they  were 
aware  of at least  1 of the 5 protocols  as well as TIPS in general.  Since  these  physicians  serve  a 
general  broad  population  and provide  a wide  range  of health  care  services,  it was not  surprising 
that  they  would  be generally  unaware  of them  Of the 6 aware  of TIPS,  2 indicated  that  they  were 
using  them  as part  of their  practice.  Of the  136 customary  providers  unaware  of TIPS, 44 
(32 percent)  expressed  an interest  in TIPS after learning  of them 
RECOMMENDATIONS 
SAMHSA  should  take  a more proactive  approach  to advertising  the availability  of all past  and 
future  TIPS. 
Since  less than half of the SAMHSA  funded  grantees  that responded  were  aware  of TIPS, this 
indicates  that  greater  effort  needs  to be made  to raise  their  level of awareness  of them  However, 
their  usefulness  is supported  by our  data  which  shows  that  84 percent  of all survey  respondents 
that  were  aware  of any of the five TIPS were  using  them  in their  practice.  While  SAMHSA  has 
made  positive  strides  in disseminating  the protocols,  such  as putting  some  on the Internet, 
distributing  them  at alcohol  and other  drug  abuse  conferences/seminars,  and providing  them  to 
all State  Alcohol  and Substance  Abuse  Directors,  there  are additional  awareness  mechanisms  that 
could  be considered.  Examples  are advertising  TIPS in pertinent  publications,  brochures,  and 
professional  journals,  etc.  They  could  also be noted  in periodic  newsletters/catalogs  published  by 
SAMHSA. 
SAMHSA  should  consider  expanding  their  target  audience. 
In the process  of expanding  efforts  in advertising  TIPS,  SAMHSA  should  look  beyond  their 
target  audience.  It should  consider  including  at minimum  CMHCs  as they  develop  future  TIPS 
and advertise  its current  ones.  Our  survey  results  show  that  81 percent  of CMHCs  that  were 
previously  not  aware  of any of the 5 TIPS, now  had  an interest  in using  them  in their  practice. 
This  compares  to 74 percent  of SAMHSA  funded  grantees.  Also,  32 percent  of customary 
providers  reported  now  that  they  are aware  of TIPS they believe  the TIPS could  be useful.  We 
believe  this information  infers  that  providers  not  commonly  considered  alcohol  and other  drug 
abuse  treatment  providers  do have  an interest  in them  and their  application  within  their  practice. 
. . . 
111  AGENCIES  COMMENTS  AND  OIG  RESPONSE 
We received  formal  written  comments  from  the SAMHSA  Administrator  and Assistant  Secretary 

for Planning  and Evaluation  (ASPE)  who  agreed  with  our  findings  and recommendations. 

However,  SAMH  SA did raise  several  questions  about  our  sampling  methodology. 

In response  to SAMHSAs  concern  about  our  sampling  technique,  we note  that  our  samples  were 

scientifically  drawn  and at the  sizes we selected,  our  precision  is within  six percent  of each  of the 

estimates.  We have  modified  our  report  to include  the confidence  intervals  for each  of our main 

point  estimates. 

SAMHSA  also raised  a concern  about  including  treatment  programs  in our  population  that  were 

never  targeted  as the primary  audience  for the TIPS.  The  comments  refer  to a list of providers 

provided  by SAMHSAs  Office  of Applied  Studies.  It was this office  that had  recommended  that 

we use this list as it represented  the target  population  for dissemination  of TIPS.  Nevertheless, 

the concern  raised  seemed  important  enough  to review.  We followed  up on this by re-examining 

respondents  information  provided  in our  survey.  Out of  137 respondents,  all but  12 appeared  to 

be potential  users  of the TIPS.  The  12 possible  exceptions  identified  themselves  as prevention 

rather  than  treatment  programs.  In fact,  we called  six of these  prevention  respondents  and three 

indicated  an interest  in TIPS.  Overall,  we believe  the list was an appropriate  one  to have  used  for 

our  analysis. 

We recognize  that  the agency  is undertaking  its own  evaluation  and,  towards  that  effort,  we hope 

that  our  results  will be helpful  in planning  and implementation.  However,  we believe  that  our 

recommendations  of wider  advertising  and expansion  of the target  audience  could  be 

implemented  before  the larger  study  is completed. 

We have  attached  actual  comments  from  SAMHSA  and ASPE  in Appendix  B. 

iv  TABLE  OF  CONTENTS 
PAGE 

EXECUTIVE  SUMMARY 
INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 
FINDINGS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...6 
SAMHSAfundedgrantees  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...6 
FDA  narcotic/methadone  treatment  providers  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 
Community  Mental  Health  Centers  ............................................  6 
CHAMPUS  customary  providers  ...........................................  .7 
RECOMMENDATIONS  ....................................................  8 
AGENCIES  COMMENTS  AND  OIG  RESPONSE  ..............................  10 
APPENDICES 
NonrespondentAnalysis..  ..............................................  ..A- 1 
Agencies  Comments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B-1  INTRODUCTION 
PURPOSE 
To  determine  the extent  of dissemination,  and practitioners  awareness  and use  of certain 
protocols  for the treatment  of individuals  with  alcohol  and other  drug  problems. 
BACKGROUND 
Treatment  Improvement  Protocols 
Treatment  Improvement  Protocols  (TIPS)  are consensus-based  best practice  guidelines 
developed  by the  Quality  Assurance  and Evaluation  Branch  of the Substance  Abuse  and Mental 
Health  Services  Administrations  (SAMHSA)  Center  for Substance  Abuse  Treatment  (CSAT). 
They  are developed  to promote  the transfer  of state-of-the-art  protocols  and guidelines  for the 
treatment  of alcohol  and other  drug  abuse  from  acknowledged  clinical,  research,  and 
administrative  experts  to the Nations  alcohol  and other  drug  abuse  treatment  resources. 
The  process  begins  with  the recommendation  of an alcohol  and other  drug  abuse  problem  topic 
for consideration  by a panel  of experts  including  clinicians,  researchers,  and program  managers, 
as well as professionals  in such  related  fields  as social  services  or criminal  justice.  Following  the 
selection  of a topic,  CSAT  forms  a Federal  resource  panel  to review  the  state-of-the-art  in 
treatment  and program  management.  Recommendations  from  this panel  are then  transferred  to a 
second  panel  consisting  of non-Federal  experts  who  are very  familiar  with  the topic.  This  group, 
known  as a non-Federal  consensus  panel  makes  recommendations,  defines  protocols,  and arrives 
at agreement  on protocols.  Its members  represent  alcohol  and other  drug  abuse  treatment 
programs,  hospitals,  community  health  centers,  counseling  programs,  criminal  justice  and child 
welfare  agencies,  and private  practitioners.  The  panel  chairperson  is charged  with  the 
responsibility  for ensuring  the resulting  protocol  reflects  true  group  consensus. 
Next,  an evaluation  of the proposed  guidelines  and protocol  must  be made  by a third  group 
whose  members  serve  as expert  field reviewers.  Following  review  of their  recommendations  and 
responses,  the document  is approved  for publication.  The  published  TIP  reflects  alcohol  and 
other  drug  abuse  treatment  guidelines  to be used  for provision  of high  quality  and innovative 
treatment  in public  and private  programs. 
Since  1993,23  TIPS have  been  developed  and issued.  SAMHSA  estimates  the  average  cost  for 
developing  each  TIP  to be approximately  $300,000,  not  including  printing  costs.  This  study 
focused  on five specific  TIPS.  The  five TIPS were  selected  based  on discussions  and concurrence 
with  SAMHSA  policy  and executive  staff.  The  five TIPS and year of publication  are listed  in the 
following  table. 
1 
 FIVE  TIPS REFERENCED  IN SURVEY 

1  State  Methadone  Treatment  Guidelines  1993  II 
2  Pregnant,  Substance-Using  Women  1993 
4  Guidelines  for the Treatment  of Alcohol  and Other  Drug  Abusing  1993 
Adolescents 
6  Screening  for Infectious  Diseases  Among  Substance  Abusers  1993 
9  Assessment  and Treatment  of Patients  with  Coexisting  Mental  1994 
Illness  and Alcohol  and Other  Drug  Abuse 
Besides  being  some  of the earliest  TIPS published,  it is believed  that  these  particular  protocols 
cover  a wide  range  of treatment  types  as well as provider  types  and should  have  a better  chance 
of being  known  by alcohol  and other  drug  treatment  providers. 
Dissemination  of TIPS 
After  each  publication,  SAMHSA  disseminates  approximately  300 TIPS to a number  of 
individuals  or components  within  the Department  of Health  of Health  and Human  Services,  all 
State  Alcohol  and Substance  Abuse  Directors,  the Addiction  Technology  Transfer  Centers,  and 
to any special  or demonstration  funded  grantees  that  exist.  There  is no direct  mailing  of TIPS to 
the approximate  13,000  SAMHSA  block  grant/State  funded  grantees.  The  SAMHSA  also 
provides  TIPS at conferences  and seminars  and has made  several  of the TIPS available  on the 
Internet.  In the future  it is their  intent  to have  all of them  on the Internet.  All remaining 
inventory  of TIPS is forwarded  to the National  Clearinghouse  for Alcohol  and Drug  Information 
(NCADI)  which  is responsible  for all other  distribution  of the protocols,  though  done  by request 
only.  The  NCADI  does  include  TIPS in its inventory  catalog  and in their  brochure/order  form 
which  are also available  upon  request.  The  SAMHSA  pays  approximately  one million  dollars 
per year to the clearinghouse  for this effort. 
The  SAMHSA  does  not  undertake  any proactive  advertising  campaign  covering  availability  of 
the TIPS.  Except  for the inclusion  in the NCADI  catalog  and its order  folm,  TIPS are not 
referenced  in periodic  SAMHSA  newsletters,  professional  publication/brochures,  national/state 
medical  associations  publications/journals,  etc.  At the time  this study  was initiated,  there  was an 
ample  supply  of the five protocols  included  in our  study  as well as the other  TIPS.  For  the 
specific  five protocols  focused  on in this study,  inventory  as of October  1997 ranged  from  TIP  #l 
with  13,000  copies  to TIP  #4 with  55,500  copies.  The  table below  shows  the quantity  printed 
and the inventory  available  in October  1997 for each  TIP  in the  study. 
2 
 TIPS INVENTORY 

TIP #  Quantity  Printed  No. Available  in October  1997 
TIP  # 1  51,095  13,000 
TIP  # 2  99,852  41,208 
TIP  # 4  101,170  55,500 
TIP  # 6  99,816  5 1,405 
TIP  # 9  98,806  32,150 
THE  OBJECTIVES  OF THIS  REPORT 

The  purpose  of this report  is to provide  SAMHSA  with  a better  understanding  of the extent  to 
which  TIPS are actually  disseminated  throughout  the alcohol  and other  drug  abuse  treatment 
health  care  community  and other  providers  by determinin  g the level  of awareness  of these 
protocols  among  the provider  population.  In addition,  we want  to advise  SAMHSA  to what 
extent  the TIPS are being  incorporated  into  providers  practices.  We have  not  evaluated  the 
effectiveness  of the guidelines  used  by any of the providers  or distinguished  the  strengths  and 
weaknesses  of the individual  TIPS. 
Another  objective  of this report  is to identify  the potential  for providers  interest  after being  made 
aware  of them  through  this  study.  Additionally,  we are identifying  information  by which 
SAMHSA  can measure  or evaluate  the necessity  for the expansion  of future  activities  through 
expanded  dissemination  and advertising  their  availability. 
METHODOLOGY 
To  determine  the extent  of dissemination  and SAMHSAs  advertising  techniques,  we chose  to 
survey  a broad  sample  of health  care providers  to get  an extended  view  of the knowledge,  use of, 
and interest  in TIPS for treatment  of alcohol  and other  drug  abuse  problems.  We identified  four 
provider  groups  to survey:  (1) SAMHSA  funded  grantee  service  providers,  the  target  audience 
for which  TIPS are developed;  (2) narcotic/methadone  treatment  providers  which  administer 
methadone  maintenance  treatment  and must  be registered  with  the Food  and Drug 
Administration  (FDA);  (3) treatment  providers  at Community  Mental  Health  Centers  (CMHCs) 
which  offer  a broad  range  of services  including  substance  abuse  counseling  and treatment;  and 
(4) physicians,  hereinafter  referred  to as customary  providers,  which  had billed  the Civilian 
Health  and Medical  Program  of the Uniformed  Services  (CHAMPUS)  for alcohol  and other  drug 
abuse  problems.  These  customary  providers  serve  a universal  broad  population  over  and above 
Medicare,  including  children,  adults,  and adolescents  as well as elderly. 
3 
 Some  providers  were  of a specialty  that  would  unquestionably  treat  alcohol  and other  drug  abuse 
problems  whereas  others  were  of an unrelated  specialty  but received  some  payment  for treating 
these  type  problems.  Therefore,  the extent  to which  these  four  provider  types  treated  alcohol  and 
other  drug  abuse  problems  could  vary  substantially.  We believe  that  these  four  provider  types 
ensured  we had  the broadest  possible  representative  sample  groups  of alcohol  and other  drug 
abuse  service  providers. 
The  SAMHSA  funded  grantee  listing,  which  numbered  13,394,  was provided  to us by 
SAMHSAs  Office  of Applied  Studies.  Since  this group  of alcohol  and other  drug  abuse 
treatment  providers  is SAMHSAs  target  audience  when  developing  TIPS,  SAMHSA  strongly 
suggested  that  they be included  in our  study.  We randomly  sampled  198 of these  providers. 
The  FDA  narcotic  treatment  providers  which  administer  strictly  methadone  maintenance 
treatment  must  register  with  the FDA.  Additionally,  because  of the  specificity  of the treatment 
provided  by this group,  they  were  also  suggested  by SAMHSA  to be included  in our  study. 
While  these  narcotic  treatment  providers  are not  specifically  a subset  of the  SAMHSA  funded 
grantees,  there  is a substantial  amount  of overlap  of these  groups.  Of the 70 FDA  narcotic 
treatment  providers  responding  to the  survey,  58 (74 percent)  were  also included  in the 
SAMHSA  funded  grantees  total  listing.  Our universe  of FDA  narcotic  treatment  providers 
numbered  862 and we randomly  sampled  110 of them 
Community  Mental  Health  Centers  were  selected  because  of their  Federal  funding  and broad 
range  of services,  including  individual  and group  counseling,  psycho  social  rehabilitation, 
therapeutic  education,  life skills training,  day programs,  and  substance  abuse  counseling  and 
treatment.  Our universe  included  549 CMHCs,  of which  we randomly  sampled  163.  We were 
particularly  interested  in CMHCs  possible  awareness  of and interest  in TIPS. 
Our fourth  group,  which  included  a listing  of 8,486  customary  providers  from  CHAMPUS  was 
selected  to determine  if they  were  being  overlooked  as health  care providers  having  a possible 
interest  in TIPS.  According  to CHAMPUS,  these  physicians  had  submitted  at least  one  claim for 
diagnostic  codes  related  to alcohol  and other  drug  abuse  treatment.  We randomly  sampled 
299 physicians  from  this customary  provider  group. 
Each  of these  770 health  care providers  was mailed  an individual  survey.  Overall,  62 percent  of 
our  sample  responded  to the  survey.  We were  unable  to perform  a nonrespondent  analysis  for the 
CMHC,  FDA,  or customary  provider  groups.  We did, however,  perform  a nonrespondent 
analysis  for the SAMHSA  funded  grantee  provider  group  and concluded  that  there  is minimal 
effect  due  to the nonresponders  (see  appendix  A for details  on our  analysis).  The  number  of 
surveys  mailed  to each  provider  type  and the number  of responses  received  is shown  in the 
following  table. 
4 
 Number  of TIPS Surveys  Mailed 
and  Responses  Received 
by Provider  Type 
Provider  Type  No. Mailed  No. Responded  Percentage 
SAMHSA  198  137  69% 
FDA  110  70  64% 
CMHC  163  125  77% 
Total  770  474  62% 
Our  sample  sizes were  based  on certain  factors  which  included  the expected  proportion  of 
providers  who  would  know  about  TIPS,  a precision  factor  and confidence  interval,  the effective 
populations  of each  provider  type,  and an expected  response  rate  of sixty percent.  Our  survey 
document  included  both  closed- and open-ended  questions  which  were  primarily  specific  to the 
five protocols  selected  for the  study.  A follow-up  survey  was mailed  to each  provider  not 
responding  to our  initial mailing. 
We conducted  our  review  in accordance  with  the Quality  Standards  for  Inspections  issued  by 
the Presidents  Council  on Integrity  and Efficiency. 
5 
 FINDINGS 

Thirty-two  percent  of the SAMHSA  funded  grantees  reported  they  were aware  of at least  one 
of the five  TIPS referenced  in the survey. 
Of the  137 SAMHSA  grantees  responding  to the  survey,  44 reported  they  were  familiar  with  at 
least  1 of the 5 TIPS referenced  in the  survey.  At the 90 percent  confidence  interval,  this 
produced  an actual  value  estimated  to lie between  25 and 39 percent.  This  is increased  to 
36 percent  if we include  those  that  reported  they  were  aware  of them  in general.  Since  SAMHSA 
considers  these  grantees  as their  target  audience,  it was expected  that  this group  of providers 
would  exhibit  a much  greater  awareness  of the protocols.  Eighty-two  percent  of the grantees 
aware  of TIPS reported  they  were  also using  them  in their practice.  Of the 93 SAMHSA  grantee 
respondents  that  were  not  aware  of any TIPS, 74 percent  reported  that  on becoming  aware  of 
them  through  our  survey,  they believe  they  could  be useful  in their  practice.  Generally,  the 
positions  held  by the respondents  were  of six types.  These  included  97 Program  Directors, 
12 Program  Coordinators,  10 Counselors/Registered  Nurses,  4 Administrative  Directors, 
3 President/Vice  President/Owners,  2 Team  Leader/Interns,  and 9 who  did not  provide  their 
position. 
Eighty-six  percent  of the FDA  narcotic/methadone  treatment  providers  responded  that  they 
were aware  of at least  one of the five  TIPS referenced  in the survey. 
Sixty  of the 70 FDA  narcotic/methadone  treatment  providers  responding  to the  survey  reported 
that  they  were  aware  of at least  1 of the 5 TIPS referenced  in the  survey,  with  familiarity  being 
specifically  with  protocol  #1 which  covers  methadone  treatment  guidelines.  At the 90 percent 
confidence  interval,  the  actual  value  was estimated  to lie between  78 and 93 percent.  We believe 
that  this high  level  of awareness  of TIPS may  well be due  to the fact  that  these  narcotic  providers 
specifically  administer  methadone  treatment  and would  therefore  be aware  of the TIP  covering 
this topic. 
Fifty-two  of the 60 (87 percent)  FDA  narcotic/methadone  treatment  providers  aware  of TIP  # I 
indicated  that  they  were  using  it in their  practice.  Although  not  specifically  included  in the 
SAMHSA  target  audience,  both  groups  are using  TIPS at similar  levels. 
The  10 FDA  narcotic/methadone  treatment  providers  not  aware  of TIPS reported  that  on 
becoming  aware  of them  through  the  survey,  they believe  they  could  be useful  in their practice. 
Thirty-two  percent  of community  mental  health  centers  reported  they  were aware  of at least 
one of the five  TZPs referenced  in the survey. 
Of the  125 CMHCs  responding  to the survey,  40 reported  that  they  were  aware  of at least  1 of the 

5 TIPS referenced  in the  survey  as well as other  TIPS in general.  At the 90 percent  confidence 

interval  for this sample,  the actual  value  was estimated  to lie between  26 and 38 percent.  Of the 

40 respondents  aware  of the 5 TIPS,  36 (90 percent)  were  using  them  in their  practice. 

Eighty-one  percent  of the  85 CMHCs  that reported  they  were  not  aware  of TIPS indicated  that  on 

6  becoming  aware  of them  through  the  survey,  they  believe  TIPS could  be useful  in their  practice. 
This high  percentage  of interest  in these  protocols  by CMHCs  shows  that  there  are alcohol  and 
other  drug  abuse  treatment  providers  outside  SAMHSAs  target  audience  that  would  use them 
if they  were  aware  of them 
Four percent  of the  customary  provider  group  responded  that  they  were aware  of at least 
one of the five  TIPS  referenced  in the survey. 
Six of the  142 customary  provider  group  responding  to the  survey  reported  that  they  were 
aware  of at least  1 of the 5 protocols  referenced  in the  survey  as well as TIPS in general.  At the 
90 percent  confidence  interval,  the actual  value  was estimated  to lie between  1 and 7 percent. 
Since  these  physicians  serve  a general  broad  population  and provide  a broad  range  of health  care 
services,  it was expected  that  they  would  be generally  unaware  of them  Of the  6 customary 
providers  aware  of TIPS, 2 (33 percent)  indicated  that  they  were  using  them  as part  of their 
practice.  Also,  of the  136 customary  providers  reporting  they  were  unaware  of TIPS, 
44 (32 percent)  expressed  an interest  in TIPS after learning  of them 
7 
 RECOMMENDATIONS 
We conclude  from  survey  results  that  there  is a need  to raise the level  of awareness  of TIPS 
throughout  the  alcohol  and other  drug  abuse  treatment  health  care  community.  Our data  shows  a 
potential  use  of them  by other  than  the SAMHSA  funded  grantees  and a substantial  interest  in 
TIPS by the various  provider  types  once  they become  aware  of them  In addition,  our  survey  has 
provided  baseline  information  by which  SAMHSA  can measure  the extent  or success  of future 
dissemination  efforts.  Overall,  we would  conclude  that  the ample  inventory  of TIPS which 
currently  exists  permits  SAMHSA  the potential  to look  at various  alternatives  to enhance 
awareness  of providers  who  serve  the alcohol  and other  drug  abuse  patient. 
SAMHSA  should  take  a more proactive  approach  to advertising  all past  and future  TIPS. 
SAMHSA  should  take  a more  proactive  approach  to making  all possible  alcohol  and other  drug 

abuse  treatment  providers  aware  of TIPS.  Since  less than half of the  SAMHSA  targeted  provider 

group  were  aware  of them  this indicates  that  greater  effort  needs  to be made  to raise  their 

awareness  level.  This  is also supported  by our  data  showing  that  84 percent  of all survey 

respondents  that  were  aware  of any of the five TIPS were  also using  them  in their  practice. 

It is understood  that  direct  mailings  to all possible  alcohol  and other  drug  abuse  treatment 

providers  is not  feasible  simply  because  demand  would  be greater  than  supply.  However, 

inventories  are certainly  large  enough  to send  them  to all SAMHSA  grantee  service  providers,  its 

principal  target  audience.  Furthermore,  with  a proactive  information  campaign  many  of those 

unaware  of TIPS could  learn  about  them  and where  applicable  to their  practice,  could  request 

them  Some  respondents  unaware  of TIPS indicated  that  our  survey  served  as a good  mechanism 

to make  them  aware  of the protocols.  Suggestions  made  by respondents  included  advertising 

TIPS through  pertinent  publications,  brochures,  professional  journals,  etc.  TIPS could  be 

referenced  in periodic  newsletters  and/or  catalogs  published  by SAMHSA.  It could  also advise 

the various  pertinent  national/state  medical  associations  for their notification  to members.  Such 

an expanded  communication  network  should  build upon  SAMHSAs  efforts  to produce  a 

valuable  treatment  tool  and  should  raise providers  awareness  of the treatment  protocols. 

SAMHSA  should  also continue  to expand  its efforts  in such  areas  as putting  TIPS on the Internet 

and distributing  them  at National  and Regional  alcohol  and other  drug  abuse 

conferences/seminars. 

SAMHSA  should  consider  expanding  their  target  audience.  ) 
In the process  of expanded  efforts  in advertising  TIPS, SAMHSA  should  look  beyond  their 
target  audience.  It should  consider  including  at minimum  CMHCs  as they  develop  future  TIPS 
and advertise  its current  ones.  Our  survey  results  show  that  81 percent  of CMHCs  that  were 
previously  not  aware  of any of the 5 TIPS, now  had  an interest  in using  them  in their  practice. 
8 
 This  compares  to 74 percent  of SAMHSA  funded  grantees.  Also,  32 percent  of the customary 
physicians  reported  now  that  they  are aware  of TIPS they believe  they  could  be useful.  We 
believe  the data  clearly  supports  that  other  health  care providers  not  commonly  considered 
alcohol  and other  drug  abuse  treatment  providers  do have  an interest  in them  and their 
application  within  their  practice. 
9 
 AGENCIES  COMMENTS  AND  OIG  RESPONSE 
We received  formal  written  comments  from  the SAMHSA  Administrator  and Assistant  Secretary 
for Planning  and Evaluation  (ASPE)  who  agreed  with  our  findings  and recommendations. 
However,  SAh4HSA  did raise  several  questions  about  our  sampling  methodology. 
OZG Response  to SAMHSA  Comments 
In response  to SAMHSAs  concern  about  our  sampling  technique,  we note  that  our  samples  were 
scientifically  drawn  and at the  sizes we selected,  our precision  is within  six percent  of each  of the 
estimates.  We have  modified  our  report  to include  the confidence  intervals  for each  of our main 
point  estimates. 
SAMHSA  also raised  a concern  about  including  treatment  programs  in our  population  that  were 
never  targeted  as the primary  audience  for the TIPS.  The  comments  refer  to a list of providers 
provided  by SAMHSAs  Office  of Applied  Studies.  It was this office  that  had recommended  that 
we use  this list as it represented  the target  population  for dissemination  of TIPS.  Nevertheless, 
the concern  raised  seemed  important  enough  to review.  We followed  up on this by re-examining 
respondents  information  provided  in our  survey.  Out of  137 respondents,  all but  12 appeared  to 
be potential  users  of TIPS.  The  12 possible  exceptions  identified  themselves  as prevention  rather 
than  treatment  programs.  In fact,  we called  six of these  prevention  respondents  and three 
indicated  an interest  in TIPS.  Overall,  we believe  the list was  an appropriate  one  to have  used  for 
our  analysis. 
We recognize  that  the agency  is undertaking  its own  evaluation  and,  towards  that  effort,  we hope 
that  our results  will help  in planning  and implementation.  However,  notwithstanding  the agency 
evaluation,  we believe  that  our  specific  recommendations  of wider  advertising  and expansion  of 
the target  audience  could  be implemented  before  the larger  study  is completed. 
We have  attached  actual  comments  from  SAh4HSA  and ASPE  in Appendix  B. 
10 
 APPENDIX  A 
NONRESPONDENT  ANALYSIS 
We conducted  an analysis  to determine  if the surveys  nonrespondents  are different  from  the 
respondents.  If the nonrespondents  are significantly  different,  bias may be introduced  into  the 
results.  We used  data  file information  on the  198 facilities  included  in the  SAMHSA  funded 
grantee  sample  group.  We analyzed  the facilities  with  respect  to their  number  of active  clients, 
whether  they  provide  substance  abuse  treatment,  and whether  they  provide  methadone  to their 
clients,  all as of October  1, 1995.  These  are all categorical  variables  and were  tested  using  the 
&i-square  test  statistic. 
We found  the results  are not  biased  with  respect  to the  size of the facility  or whether  the facility 
provides  substance  abuse  treatment.  The  &i-square  test  statistic  was not  significant  in the 
analysis  of these  two  variables. 
The  methadone  facilities  are under  represented  being  only  approximately  5 percent  of the total 
surveys  and  10 percent  of the nonrespondents.  This  difference  is statistically  significant. 
However,  we believe  that  this difference  does  not  materially  affect  our  conclusions. 
The  following  tables  demonstrate  the results  of our  analysis.  Included  in the  tables  are the 
number  of respondents  and nonrespondents  that  fall within  each  of the variable  categories.  The 
&i-squared  test  statistic  and the  associated  degrees  of freedom  are noted  on each  table. 
SIZE  OF FACILITY 
I
Size  of Facility  Respondents  %  Nonrespondents  %  Total 
0 - 18 Active  Clients  30  27%  10  19%  40 
19 - 39 Active  Clients  28  25%  13  25%  41 
40 - 108 Active  Clients  31  27%  12  22%  43 
109 - 109+ Active  Clients  24  21%  18  34%  42 
Total  113  100%  53  100%  166 
(Xi-squared  test  statistic=3.512 
Degrees  of freedom=3 
p-value=0.3  19 
A-l  SUBSTANCE  ABUSE  TREATMENT 
Facility  Provides 
Substance  Abuse  Respondents  %  Nonrespondents  %  Total 
Yes 

No  22  16%  9  14%  31 

Tntnl  l?h  inn%  62  1I)l)%  198 

Chi-squared  test  statistic=0.089 

Degrees  of freedom=  1 

p-value=0.766 

METHADONE  TREATMENT 
Facility  Provides 
Methadone  Treatment  Respondents  %  Nonrespondents  %  Total 
Yes  4  3%  6  10%  10 

No 
Dont  know 

Non-treatment  21  16%  5  8%  26 

Tntfll  134  100%  62  100%  196 

(X-squared  test  statistic=16.560 

Degrees  of freedom=3 

p-value<O.OO 1 

A-2 
 APPENDIX  B 

AGENCIES  COMMENTS 

B-l  B-2 
 B-3 
 DEPARTMENT  OF  HXAL,TH  A  r$UtlAN  .WWKXS  Public Health SWVIC~ 
Substance  Abuse  and  Mental 
JAN  I  3  I998  Hsakh Sewicm  Administration 
Rockville MD 20857 
To: 	 June Gibbs 3rown 
Inspector  General 
From:  Administrator,  SAMHSA 
Subject:  ~~,~7~$rt~~As  Treatment  Improvement Protocols 
Thank you for t@eopportunittoy  c
omment on the subject drafl repo.rt. The SubstanceAbuse and
Mental Health Services Administration (SAMHSA), through its Cenkr  for Substance Abuse 

Treatment(CSAI)i,s proudof its ruIe in the dcvcIapment of the Rcatment  Improvement 

Protocolsms)  that have,oyer the last five-years, produced 24 consens~b~  doments 

relakd to improvingpublkly-fbndedsubstanceabuse treatment servicesand systems. 
With regard to the report developed by the Office of the Spector  GenemI (OIG),SAMHSA 
expressed its concerns  pi&  to the study,  and reiterates  again,  that the primary sanlple of tided 
grantees drawn to determinethe impactand usefihxss  of the TIPS was extrao~y  small 
(198of 13,349prhidcra), withmanyof theSBLY#Aprogramsconsistingof alcohol-only, 
prevention-onloyr,mistxllaneous  potential ;rec@ients (e.g., Boys Clubs/Gjris Clubs, a 
penitentiary, a medical  school, a research institute, and a church group) which were IXYQ 
targeted as the primary audience for tliese treatmentkiented  protocols. The result was a lower 
than anticipakd positive response (32 percent) to the TIPS.  Converdy,  where CSAT 
specifically directed a TIP on methadone setices  to national  narcotic treatment programs, over 
86 percent of responding units were aware of this protocol. 
Overall, SkWiSA  accepts the conclusions and recommendations  of the OIGreport. In fact, 
even prior to the beginning of the study, CSAT had committed  its&. to conductkig, and has 
subsequently tiplemented,  a 4-year, $32 million  study to improve the development and 
dissemination of the llPs.  This study will be conducted  with attention to appropriate axed 
rigoroussampling, both quantitative and qualitative,  with the intention of improving the 
viability, dissemination, and use of these important guidelines  and materials.  Central to this 
studywillbethe effortto improvethetazgetingof the TIPStQspcciaIizcdpcrpulatioaNsIB!!x 
expansion of the target audier~ces,as recommended by the OIG report. 
Thank you again for the opportunity to comment.  We look  forward to receiving your final 
report. 
Nelba Chqvez, Ph.D. 
B-2 
 DEPARTMENT  OF HEALTH  & HUMAN  SERVICES  otfrce of the secrsrary 
Was5inglOn,  D.C. 20201 
TO: 	 June Gibbs Brown 
Inspector  General, DHHS 
FROM: 	 CheqI  Austein
7?@fG 
Division Director 
Public He&h  Policy,  ASPE 
DATE:  Febrq  6,1998 
SUBJECT: 	 Clearax:  of OIG DrafI Report  SAMHSAs  Treatmcxt  Improvenient  Protocols 
(OEI-O7-96-00130) 
Thank  you for the opporznity  to review  a draft  version  of the report  evaluating  S&HSAs 
Treatment  Improvement  Protocol  (TIP) series.  A mcmbcr  of my staff familiar  with SMSA 
programs  has exzmined the report and concurs  with  its publication,  In ,oencraI, rnv s+kzff!ofund 
the  methodoIogicaI  and sampling  approach  reasonable,  if not e&rely  representative,  The 
reports  conclusions  and recommendations  appear  justified  and b&d  firmIy  upon the study 
fmdings.  Moreover,  my staff is aware that  SAMHSA  accepts  the reports  conclusions  and 
rewmmcndations,  and & currently  implemem?ing  a more  extensive  study  of the TLFseries in an 
attempt  to improve  tic viability, dissemination,  and  use  of these  guidelines.  it is likely that the 
recommendations  orthe report Tom  your  of&e  wiIl  provide  important  inform&n  to 
SAMHSAs  cumnt  evaluation eEo&  in this  area 
B-3