Independent Evaluation of the Substance 

Abuse Prevention and Treatment Block 

Grant Program 

Final Evaluation Report 

U.S. Department of Health and Human Services 

Substance Abuse and Mental Health Services Administration 

Center for Substance Abuse Treatment 

Center for Substance Abuse Prevention 

http://www.samhsa.gov
Independent Evaluation of the Substance 

Abuse Prevention and Treatment Block Grant 

Program 

Final Evaluation Report 

U.S. Department of Health and Human Services 

Substance Abuse and Mental Health Services Administration 

Center for Substance Abuse Treatment 

Center for Substance Abuse Prevention 

1 Choke Cherry Road 

Rockville, Maryland 20857 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

iii

Acknowledgements 

This report was prepared for the Substance Abuse and Mental Health Services 

Administration (SAMHSA) of the U.S. Department of Health and Human Services (DHHS) 

by Altarum Institute under Task Order No. 280-03-3501. The tasks for this Independent 

Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program were 

added through a Task Order modification to an existing Task Order for the Center for Mental 

Health Services’ Independent Evaluation of the Community Mental Health Services Block 

Grant Program. Both evaluations were conducted under the same Task Order number (280-

03-3501). This report is the result of substantial contributions by numerous individuals. Eric 

Gelman served as the Project Director, and Jessica McDuff was the Project Manager. Dr. 

Theresa Mitchell-Hampton served as the Task Order Officer (TOO) for the Center for 

Substance Abuse Treatment (CSAT), with Sherrye Fowler as Alternate Task Order Officer 

(ATOO) for CSAT, and Dr. John Park served as the TOO for the Center for Substance Abuse 

Prevention (CSAP). Jessica McDuff, Laura McGovern, Sarah Lifsey, Sally Holthouse, and 

Halima Ahmadi of Altarum Institute contributed significantly to the development of key 

findings and the writing of this report. Anne Herron, John Campbell, and Theresa MitchellHampton of CSAT and Namratha Swamy of Altarum Institute guided the work and provided 

many helpful comments and suggestions. The Evaluation Advisory Workgroup (named in 

Appendix B) provided many insights on the evaluation design, data collection protocols and 

procedures, and policy relevance of the results. 

Disclaimer 

The content of this report does not necessarily reflect the views or policies of SAMHSA or the 

DHHS, nor does it necessarily reflect the views of any of the Evaluation Advisory Workgroup 

members. The authors are solely responsible for the content of this publication. 

Public Domain Notice 

All material appearing in this report is in the public domain and may be reproduced or copied 

without permission from the Substance Abuse and Mental Health Services Administration. 

Citation of the source is appreciated. 

Recommended Citation 

Center for Substance Abuse Treatment and Center for Substance Abuse Prevention. 

Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant 

Program: Final Evaluation Report. Rockville, MD: Substance Abuse and Mental Health 

Services Administration, 2009. 

Electronic Access 

This publication can be accessed electronically through the following Internet World Wide 

Web connection: http://www.tie.samhsa.gov. 

Originating Offices 

Division of State and Community Assistance, Center for Substance Abuse Treatment, and 

Division of Systems Development, Center for Substance Abuse Prevention, Substance 

Abuse and Mental Health Services Administration, 1 Choke Cherry Road, Rockville, MD 

20857.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

iv

Table of Contents

Executive Summary ............................................................................................................................. 1

Challenges and Recommendations for Program Improvement .......................................................... 4

Background .......................................................................................................................................... 8

Evaluation Purpose and Design ......................................................................................................... 9

Purpose ............................................................................................................................................. 9

Design ............................................................................................................................................. 10

Methods .......................................................................................................................................... 12

Limitations ....................................................................................................................................... 13

Key Findings ...................................................................................................................................... 14

SAMHSA’s National Outcome Measures (NOMs) ........................................................................... 15

Key Finding 1. ................................................................................................................................. 16

Key Finding 2 .................................................................................................................................. 21

Key Finding 3. ................................................................................................................................. 26

Key Finding 4 .................................................................................................................................. 28

Key Finding 5 .................................................................................................................................. 31

Key Finding 6. ................................................................................................................................. 36 

Program Challenges and Recommendations for Improvement ..................................................... 39

Conclusion ......................................................................................................................................... 44

Appendices 

Appendix A: Legislative Goals of the SAPT BG Program 

Appendix B: Evaluation Advisory Workgroup Membership 

Appendix C: Program Logic Model and Evaluation Framework 

Appendix D: Evaluation Instruments 

Appendix E: Complete Discussion of Evaluation Methods and Limitations 

Appendix F: Secondary Analysis of the National Outcome Measures (NOMs) 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

v

Tables and Figures

Table 1. Summary of Data Collection Methods.................................................................................... 12

Table 2. SAMHSA’s National Outcome Measures ............................................................................... 15

Figure 1-1. Percent of Clients Abstinent from Alcohol 

from Admission to Discharge, FY 2006–2008 ...................................................................................... 17

Figure 1-2. Percent of Clients Abstinent from Drug Use 

from Admission to Discharge, FY 2006–2008 ...................................................................................... 17

Figure 1-3. Percent of Clients Employed/Attending School 

from Admission to Discharge, FY 2007–2008 ...................................................................................... 18

Figure 1-4. Percent of Clients with Stable Housing from 

Admission to Discharge, FY 2006–2008 .............................................................................................. 18

Figure 1-5. Percent of Clients Experiencing Social Support 

and Social Connectedness at Admission and Discharge, FY 2006–2008 ............................................ 19

Figure 1-6. Percent of Client Arrests in the Past 30 Days 

from Admission to Discharge, FY 2006–2008 ...................................................................................... 20

Table 3. Length of Stay (LOS) in Days by Treatment Modality, FY 2006–2008 ................................... 20

Figure 1-7. Women's Treatment Services Expenditures for All BG-Funded 

Entities: 1994 (Baseline) and 2005–2007............................................................................................. 22

Table 4. Examples of Evidence-based Practices Implemented with SAPT BG Funds ......................... 24

Table 5. TA and Training Topics and Outcomes.................................................................................. 31

Table 6. Interagency and Subrecipient Linkage Agreements............................................................... 35

Table 7. Average Age of First Use of Substances among Youth Aged 12 to 17 Years, FY 2008......... 37

Table 8. Perception of Risk/Harm from Substance Use among Youth Aged 12 to 17, FY 2008.......... 38

Table 9. Perception of Substance Use among Youth Aged 12 to 17 Years, FY 2008 .......................... 39

Table 10. SAPT BG Program Challenges and Recommendations for Improvement............................ 40

Table 11. Achievement of the 17 Legislative Program Goals............................................................... 45
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

1

Executive Summary

The Substance Abuse Prevention and Treatment Block Grant (SAPT BG) Program provides funds to 

States, Territories, the Pacific Jurisdictions, and one Native American Tribe to plan, carry out, and 

evaluate activities to prevent and treat substance abuse. The SAPT BG Program, legislated by Congress 

in 1981, is administered by the Center for Substance Abuse Prevention (CSAP) and Center for 

Substance Abuse Treatment (CSAT) of the Substance Abuse and Mental Health Services Administration 

(SAMHSA) and represents the largest source of Federal funding to States for the prevention and 

treatment of substance use disorders. It constitutes a substantial amount of all States’ budgets for 

substance abuse programming and serves an average of 2 million individuals each year. States have 

flexibility in determining how funds should be allocated to address local needs; however, to receive 

funding, States must meet specific set-aside and maintenance of effort (MOE) requirements and conduct 

activities designed to achieve the 17 legislative goals of the Program.

In 2005, CSAT and CSAP contracted with Altarum Institute under Task Order Number 280-03-3501 to 

conduct the first independent evaluation of the SAPT BG Program. The purpose of the evaluation is to 

assess the extent to which the SAPT BG Program is effective, functioning as intended, and achieving 

desired outcomes. 

The evaluation has five main objectives:

To examine the processes and activities by which States implement the legislative and policy 

requirements (e.g., 17 goals) of the SAPT BG Program

To assess activities associated with the Federal administration of the SAPT BG Program and 

how they support Program implementation and accountability

To examine State system processes and capacity regarding the collection and submission of data 

on BG-funded activities

To assess specified outcomes associated with States’ treatment and prevention services 

To explore unique ways in which States use and leverage SAPT BG funds.

To examine the effectiveness and impact of the SAPT BG Program, the evaluation employed a 

multimethod design that incorporated process and outcome evaluation strategies. Quantitative and 

qualitative data collection, analysis, and triangulation were necessary to obtain a complete picture of 

Program strengths, effects, areas for improvement, and client- and State system-level outcomes. Specific 

data collection methods included review of Program documents, secondary analysis of data collected for 

the Program’s National Outcome Measures (NOMs), and the development and administration of 

interview protocols with State and Federal staff and Web-based surveys for Program reviewers. 

The independent evaluation of the SAPT BG Program resulted in six key findings about the outcomes 

and effects of the Program, pointing to Program successes as well as to areas for improvement in 

processes and implementation. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

2

Key Finding 1: The SAPT BG Program has demonstrated a positive effect 

on the health and lives of individuals with substance use disorders.

Secondary analysis of data collected for the NOMs demonstrated positive client outcomes in all six 

treatment domains: alcohol and drug abstinence, employment/school participation, stable housing, social 

connectedness, criminal justice involvement, and retention in treatment. 

Key Finding 2: The SAPT BG Program has acted as a major impetus for 

improving State prevention and treatment systems’ infrastructure and 

capacity.

SAPT BG Program emphasis on demonstrating the effectiveness of BG-funded programs and services to 

reduce substance abuse and to improve the lives of those affected by it has driven State system 

infrastructure development and capacity improvements and resulted in the following outcomes:

Increased availability of services for diverse and underserved populations

Increased development and implementation of evidence-based practices (EBPs)

Improved development and collection of specific outcome measures

Increased development and maintenance of State data management systems.

Key Finding 3: States have leveraged SAPT BG Program requirements, 

resources, and Federal guidance to sustain and improve their State systems.

States have used the BG requirements and funding to go beyond the intended and expected outcomes of 

the Program, leveraging BG resources to sustain and improve State substance abuse prevention and 

treatment systems. Although not a requirement for BG funding, State leveraging of SAPT BG Program

requirements, resources, and Federal guidance demonstrates the importance of the BG in the 

development of State systems. States leveraged BG funds to:

Prevent harm to the service system resulting from State legislature reductions in funds for 

prevention and treatment and advocate for additional State funding. In some States, prevention 

activities would not exist without SAPT BG support.

Provide BG funds as seed money for new programs that other public and private organizations 

have subsequently funded.

Set State policies and priorities based on Federal leadership and development of national policies 

and priorities.

Key Finding 4: Through a standard system of communication, monitoring, 

and reporting, CSAT, CSAP, and the States effectively and efficiently 

manage the SAPT BG Program.

CSAP and CSAT have developed several successful management strategies to steer States as they work 

toward the 17 legislative goals. Management involves the following crucial activities:

Communicating Program goals and activities 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

3

Monitoring and oversight to facilitate open communication and ensure compliance

Leading complex data collection and reporting activities

Providing technical assistance (TA) and training to aid States in meeting their goals.

Key Finding 5: The SAPT BG Program has contributed to the development 

and maintenance of successful State collaborations with other agencies and 

stakeholders concerned with preventing substance abuse and treating 

substance use disorders.

State substance abuse agencies have increased the development and maintenance of collaborative 

working relationships with a variety of other Federal, State, and local agencies and providers. States 

fostered many of these partnerships as they worked to accomplish the Program’s 17 legislative goals. 

These State collaborations served five critical roles:

To increase achievement of Synar Program goals and objectives

To improve the coordination of prevention services

To improve the coordination of treatment services with public and private health insurers

To expand services and programs available through joint funding initiatives

To increase the ability to address statewide critical public health or safety issues.

Key Finding 6: Although baseline data support the need for prevention 

services and activities, the use of national survey State estimates data alone 

to assess the NOMs limits CSAP’s ability to attribute changes in the NOMs 

to SAPT BG-funded prevention services and activities. 

To reduce the data collection burden for State and local prevention agencies, CSAP uses data from the 

National Survey on Drug Use and Health (NSDUH) to fulfill NOMs data requirements, including 30-

day substance use, perceived risk or harm from use, age of first use, perception of disapproval/attitude 

towards substance use, and perception of workplace policy. Two significant difficulties are inherent in 

this strategy: conclusions about NOMs changes as a result of BG-funded prevention services and 

activities cannot be made based primarily on the results of national survey State estimates that do not 

identify individuals or groups who may have been affected by BG-funded activities, and the NSDUH is 

limited by small sample sizes in many States, which leads to under coverage of some populations. 

Additional data are needed to link changes in NOMs measures to interaction with BG-funded prevention

services and activities.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

4

Challenges and Recommendations for Program Improvement

In addition to its strengths and accomplishments, the large and complex SAPT BG Program experiences 

a variety of challenges. By addressing and resolving these challenges, CSAT and CSAP can improve 

Program effectiveness and efficiency and increase the quality of Program services provided to 

individuals and communities.

Challenge: Need for improved communication and a consistent message from Federal to State 

staff about some Program goals and requirements. States commented about the lack of unified, 

consistent messages from CSAT and CSAP in some specific areas, including:

NOMs data definitions and standard data collection processes

The reporting of financial information and MOE compliance 

Expectations for BG applications 

Acceptable fulfillment of the 17 legislative goals.

Challenge: Need for clarification about the roles and responsibilities of Federal, State, and 

contractor staff related to CSAT Core Technical Reviews and Technical Assistance (TA). Although 

CSAT State Project Officers (SPOs) work with States to refine their BG applications and to address the 

17 goals, CSAT SPOs are not involved in substantive aspects of State Core Technical Reviews. To 

ensure an objective assessment of compliance, CSAT SPOs typically attend the Core Technical Review

but remain in the background, which is confusing for States. In addition, States perceive that decisionmaking authority for CSAT TA requests lies in the hands of the TA contractor, the TA contract

Recommendations for Program Improvement:

Improve Program Communication and Guidance

Clarify Program data definitions and requirements, including ―what counts‖ for achievement 

of the 17 goals, MOE and other financial calculations, and the NOMs data elements

Develop unified Federal guidance about Program requirements and expectations and a data 

dictionary with uniform and realistic definitions 

Continue to seek State input and develop better definitions for required outcome data 

elements

Provide opportunities for internal communication within CSAT and CSAP, training and 

mentoring staff to ensure that consistent guidance is provided to States

Strengthen ongoing communication between State Project Officers and their assigned states 

via devoted resources for knowledge management

Recommendations for Program Improvement:

Clarify Roles and Responsibilities

Clarify the role of the CSAT SPO to avoid confusion among State and Federal staff

Assign CSAT SPOs a more substantive role in Program monitoring and TA provision to take 

advantage of SPO State-specific expertise and to improve State satisfaction with monitoring 

and support 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

5

Government Project Officer (GPO), and the Branch Chief, rather than with the CSAT SPOs, who 

typically are most informed about individual State issues and TA needs. 

Challenge: High level of burden on States to provide information for the Program. Each year, 

States are required to produce an SAPT BG application that describes the activities conducted to achieve 

the 17 legislative goals. The length of the report reflects the collection of information for reports to 

Congress. The majority of States spend 6 to 9 months each year gathering information for and 

developing the BG application, using staff resources that States argue could be better spent on TA for 

providers and other BG subrecipients. States also are asked to provide a large amount of background 

information in preparation for Program monitoring reviews – information that States say can be obtained 

from their applications and online resources. 

Challenge: Limited utility of Program monitoring reports and recommendations for some States.

The time lag associated with the finalization of monitoring reports (6 to 12 months) makes some 

recommendations obsolete by the time they reach the States. Delays are most pronounced when the 

review contractor, SPO, and States disagree about review findings, which must be resolved before a 

report can be finalized. In addition, some report recommendations are too general and do not account for 

the unique combination of political, social, and economic forces that affect the State’s prevention and 

treatment system. Some States report that TA recommendations in the treatment review reports seem 

motivated by contractor interests and skills rather than what is in the best interest of the State. 

Recommendations for Program Improvement:

Reduce State Administrative Burden

Implement a multiple-year application cycle (every 3 to 5 years) that would require States to 

submit a multiple-year plan and provide annual progress reports based on plan objectives

Revisit the primary purpose of the BG application and eliminate questions or areas that do not 

address it

Encourage Program monitors and reviewers to obtain information from WebBGAS or online 

resources to reduce State administrative burden

Recommendations for Program Improvement:

Improve Utility of Program Monitoring Reports and Recommendations

Expedite the report review process by instituting a process through which States are able to 

contest review findings without delaying the finalization of the review report

Set and enforce deadlines for submission of report comments and report revisions so that 

reports are not delayed by any one individual

Select and train reviewers to ensure that they possess a comprehensive understanding of the 

State systems that they may be assigned to review 

Further involve CSAT SPOs in reviewing TA recommendations to ensure that they address 

State concerns and are not motivated by contractor interests
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

6

Challenge: Unmet TA and training needs. Many States do not know how to make formal requests for 

Federal TA and training and are not aware of the potential areas in which TA is available. In addition, 

the treatment and prevention system reviews produce some useful TA recommendations, but the TA is 

either not approved or not conducted to the satisfaction of the State. Finally, there is a disconnect 

between Federal TA providers and CSAT SPOs; several States reported receiving support for a TA 

request from their CSAT SPO only to have it denied by the CSAT TA contract GPO or Branch Chief.

Challenge: Limited ability to demonstrate some individual-level outcomes and system-level 

outcomes. NSDUH data used for prevention NOMs do not account for interaction with BG-funded 

prevention services and activities. Thus, Federal and State staff cannot claim that changes in the NOMs 

are due in part to BG-funded prevention services and activities. In addition, use of the Treatment 

Episode Data Set (TEDS) for the treatment NOMs presents the following difficulties: Not all States 

participate in the TEDS initiative and questions remain about the consistency of data collection and data 

quality procedures across States. Finally, the Program needs some system-level outcome measures 

related to infrastructure development, collaboration with other State agencies and organizations, and 

effectiveness of TA and other Program support activities.

Recommendations for Program Improvement:

Improve and Expand Provision of TA

Clarify the TA and training request process and regularly inform States about the process 

Increase efforts to market TA and training to the States so that they understand what is 

available

Expand the scope of Federal TA provided to States to include additional TA designed to 

identify and meet the needs of diverse populations, address EBP implementation challenges, 

and assist with State infrastructure enhancements

Clarify the role of the CSAT SPO in TA provision and encourage direct communication 

between States and CSAT TA decisionmakers in order to improve responsiveness to States; 

promote a team approach

To improve TA access and satisfaction, provide additional resources for the following TA and 

training formats:

o Guidelines and support for train-the-trainer models, regular training cycles, and a 

helpdesk

o Distance learning, Web-based trainings, and online tutorials

o Peer-to-peer TA at workshops, conferences, and regional meetings

o Wider dissemination of ―off-the-shelf‖ TA tools and materials
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

7

Challenge: Need for additional resources to further improve State data infrastructures. Many 

States are still struggling to develop data collection, information, and monitoring systems that will 

enable them to track outcomes effectively. States have received modest funding to create new or 

overhaul existing data collection and reporting systems; however, they need additional resources to 

continue improvements and to foster data literacy. States also cited a need for improved networking 

opportunities among States so that they can learn from each other and not have to ―reinvent the wheel.‖

Recommendations for Program Improvement:

Improve Data Collection Strategies and Processes

Develop prevention outcome measures that assess attitudes and behaviors pre- and postinteraction with BG-funded services and activities when the prevention strategy supports this 

evaluation design.

Compare NSDUH results on the NOMs for respondents who did and did not report 

experience with prevention services and messages to determine whether exposure to 

prevention services or messages can be associated with more desirable NOMs results.

Strongly encourage States to participate in TEDS and to use TEDS data definitions to 

improve the reliability of NOMs data 

Develop materials, host Web-based and in-person trainings, and provide onsite TA to States 

and subrecipients to ensure that data definitions are being interpreted correctly and 

consistently

Continue close collaboration with State substance abuse and other appropriate State agencies 

(e.g., data and statistics, corrections) to develop more valid and effective outcome measures 

Develop 2 to 4 system-level indicators to demonstrate Program effects on State systems 

development and enhancement

Recommendations for Program Improvement:

Invest Additional Resources to Improve State Data Infrastructures

Invest additional resources to help States complete the development and maintenance of data 

collection systems that will increase their ability to demonstrate SAPT BG Program outcomes 

and make data-driven decisions 

Create more opportunities for State-to-State TA regarding the development and maintenance 

of State data collection and reporting systems. Additional opportunities for State-to-State TA, 

regional trainings, and networking at conferences will help more States develop and maintain 

effective data systems
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

8

Background

Substance abuse is one of our Nation’s most pervasive and devastating public health problems. 

According to the 2007 results of the National Survey on Drug Use and Health (NSDUH), more than 22 

million Americans have been classified with substance dependence or abuse based on criteria outlined in 

the Diagnostic and Statistical Manual of Mental Disorders, 4th edition (DSM-IV).1Families and 

communities suffer, too; the consequences of substance abuse include increased rates of unemployment, 

homelessness, child abuse and neglect, domestic violence, HIV/AIDS and other sexually transmitted 

diseases, motor vehicle accidents, sexual assault, and homicide.2Thus, successful nationwide efforts to 

prevent and treat substance abuse are critical to mitigating the effects of the disease and improving the 

quality of life for individuals, families, and communities.

Congress has played a key role in publicly funded efforts to prevent and treat substance abuse through 

the authorization of the Substance Abuse Prevention and Treatment (SAPT) Block Grant (BG) Program. 

Administered by the Center for Substance Abuse Treatment (CSAT) and the Center for Substance 

Abuse Prevention (CSAP) of the Substance Abuse and Mental Health Services Administration 

(SAMHSA), the Program provides funds to States, Territories, the Pacific Jurisdictions, and one Native 

American Tribe3to plan, carry out, and evaluate activities to prevent and treat substance abuse. The 

SAPT BG Program, the largest source of Federal funding to States for the prevention and treatment of 

substance use disorders, constitutes a substantial amount of all States’ budgets for substance abuse 

services and activities and serves an average of 2 million individuals each year.

As intended by the original legislation4, States have flexibility in determining how funds should be 

allocated to address local needs; however, to receive funding, States must meet specific set-aside and 

maintenance of effort (MOE) requirements. These requirements, introduced by the Alcohol, Drug 

Abuse, and Mental Health Administration (ADAMHA) Reorganization Act of 1992 and amended in the 

Children’s Health Act of 2000, are:

Expend no less than 20 percent of BG funding on primary prevention programs

Expend no less than the amount spent in fiscal year (FY) 1994 on services for pregnant women 

and women with dependent children

Expend no less than the average level of expenditures for FY 1991 and FY 1992 for the 

provision of tuberculosis services

Expend no more than a 5 percent increase over the State allotment in FY 1991 for HIV services 

in designated States5

Enforce the Synar Amendment against the sale of tobacco to underage individuals 

Maintain annual State expenditures for prevention and treatment services at no less than the 

average level of expenditures for the 2 years preceding the current fiscal year

Limit State administrative expenses to 5 percent of the annual SAPT BG allocation. 

The legislation further requires States to conduct activities to achieve the 17 SAPT BG Program 

legislative goals. These goals aim to enhance and improve: State systems’ infrastructure, services 

capacity, access to services for underserved populations, and adherence to Federal laws and regulations 

that affect the provision of substance abuse prevention and treatment services. A complete description of 

the 17 Program goals is located in Appendix A.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

9

Through the establishment of Block Grants during the Reagan Administration, Congress intended to 

transfer decisionmaking authority for public health and social programs from the Federal Government to 

the States. As such, Congress did not require the attainment of specific State- or client-level outcomes. 

However, by the 1990s, Congress, the White House, and Federal agencies realized the need to collect 

and analyze performance data to determine which taxpayer-funded programs were effective and which 

needed improvements to realize program goals. 

In 2002, the Bush Administration’s Office of Management and Budget (OMB) implemented the 

Program Assessment Rating Tool (PART) to evaluate every Federally-funded program’s purpose, 

design, planning, management, results, and accountability to determine its overall effectiveness. The 

SAPT BG Program underwent the PART process in FY 2003 and received a rating of Ineffective. The 

Program received high scores on three of four PART areas, including Program Purpose and Design, 

Strategic Planning, and Program Management. It received a low rating in a fourth area, Program 

Results/Accountability; OMB analysts concluded that ―no independent evaluation of the Program has 

been completed‖ to establish that the SAPT BG Program was, in fact, effective and fulfilling its 

legislative mandates.

To this end, CSAT and CSAP implemented a multi-pronged approach to look at outcomes, implement 

performance improvement, and conduct an independent evaluation. In 2000, building on work from 

CSAT’s two Treatment Outcomes and Performance Pilot Studies (TOPPS), State reporting on admission 

and discharge cohorts was piloted. In 2004, States and SAMHSA agreed on the domains of the National 

Outcome Measures, and States were asked to voluntarily report outcomes in the BG application. In 

2007, OMB approval was received to incorporate mandatory reporting of the NOMs in the BG 

application. Further, in 2005, CSAT and CSAP contracted with Altarum Institute (then Health Systems 

Research, Inc.) under Task Order Number 280-03-3501 to conduct the first independent evaluation of 

the SAPT BG Program. Through this comprehensive national evaluation, CSAT and CSAP sought to 

address the concerns of the OMB PART process and to demonstrate the strengths, challenges, outcomes, 

and impacts of the SAPT BG Program. This report, the primary product of that 3-year independent 

evaluation, is based on an extensive array of quantitative and qualitative evaluation strategies, including 

interviews with dozens of Federal and State staff and an analysis of thousands of pages of Program

materials. The report is presented in the following sections:

Evaluation Purpose and Design

Key Findings

Program Challenges and Recommendations for Improvement

Conclusion

Evaluation Purpose and Design

Purpose

This independent evaluation of the SAPT BG Program seeks to assess the extent to which the Program

is effective, functioning as intended, and achieving desired outcomes. 

The evaluation has five main objectives:
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

10

To examine the processes and activities by which States implement the legislative and policy 

requirements (e.g., 17 goals) of the SAPT BG Program

To assess activities associated with the Federal administration of the SAPT BG Program and 

how they support Program implementation and accountability

To examine State system processes and capacity regarding the collection and submission of data 

on BG-funded activities

To assess specified outcomes associated with States’ treatment and prevention services 

To explore unique ways in which States use and leverage SAPT BG funds.

Design

To thoroughly and objectively examine the effectiveness and impact of the SAPT BG Program, the 

evaluation employed a multimethod design that incorporated process and outcome evaluation strategies. 

Quantitative and qualitative data collection, analysis, and triangulation were necessary to obtain a 

complete picture of Program strengths, effects, areas for improvement, and client- and State systemlevel outcomes. Secondary analysis of CSAT’s Office of Applied Studies (OAS) Drug and Alcohol 

Services Information System (DASIS) Treatment Episode Data Set (TEDS) client-level data at program 

admission and discharge allowed for the use of a pre- and post-test design to examine client-level 

outcomes associated with participation in SAPT BG-funded treatment programs and changes in these 

outcomes over time. The assessment of State system-level outcomes and Federal and State management 

processes related to the Program was conducted using a post-test design. 

Evaluation Advisory Workgroup. An Evaluation Advisory Workgroup (EAW) was convened at the 

outset of the evaluation and maintained throughout the project. The EAW included individuals within 

State and county substance abuse agencies, evaluation experts, and members of national advocacy 

groups. The EAW provided guidance and feedback during all phases of the evaluation, including 

evaluation design, instrument development, data collection, data analysis, and final report development. 

A list of EAW members who provided their time and expertise for the evaluation is located in Appendix 

B.

Logic Model and Evaluation Framework. The evaluation team first developed a logic model for the 

SAPT BG Program; this logic model formed the basis for development of the evaluation framework, 

which contained main areas of inquiry and specific questions to be addressed. The EAW assisted in the 

development of logic model components, including Program inputs, activities, outputs, outcomes, and 

impacts. Once the logic model had been finalized, evaluation questions were developed to assess Federal 

and State Program implementation according to the logic model’s activities and outputs and desired 

outcomes and impacts, which included the following: 

Activities and Outputs

Application template and guidance development and distribution

Application development, review, and approval

Allocation and distribution of SAPT BG funds

Program monitoring

Data collection, reporting, and analysis

Technical assistance (TA), training, and other Program support
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

11

Federal leadership related to the Program and the field 

State infrastructure development 

State leveraging of Program policies and funds. 

Desired Outcomes and Impacts 

Improved Federal and State communication and information exchange 

Increased ability to demonstrate Program outcomes to stakeholders 

Improved State compliance with legislative requirements (i.e., 17 goals) 

Increased access to services for target populations 

Increased use of evidence-based practices (EBPs) in treatment and prevention programs 

Improved substance abuse systems of care within and across States 

Improved client status on indicators of abstinence, employment status, criminal justice system 

involvement, housing stability, and social connectedness. 

Appendix C includes the SAPT BG Program logic model and evaluation framework. 

Data Sources and Instrumentation. The evaluation used a variety of primary and secondary 

quantitative and qualitative data sources to obtain a complete picture of the complex SAPT BG Program 

and how it is implemented across its 60 recipients. Data on processes and outcomes were needed for a 

thorough and objective evaluation of Program accomplishments, challenges, and positive and negative 

effects; and for the development of useful recommendations for improvement. Qualitative and 

quantitative secondary data sources used for the evaluation included: 

State SAPT BG applications submitted through the Web-based Block Grant Application System 

(WebBGAS) 

State Technical Review (TR) and State Prevention and Synar System Review (SPSSR) 

monitoring reports 

SAPT BG Program National Outcome Measures (NOMs) data 

NSDUH data 

Reports on implementation of the NOMs and specific SAMHSA initiatives, including CSAP’s 

Strategic Prevention Framework. 

These data provided descriptions of BG activities in several areas, including compliance with the 17 

legislative goals, financial data, baseline and outcomes data for performance measurement, and 

descriptions of Federal and State efforts to improve substance abuse prevention and treatment systems. 

However, the data did not provide information related to State and Federal perspectives on Program 

management or system-level outcomes. To assess these areas, the following primary data collection 

instruments were developed in collaboration with the EAW: 

Federal Staff Interview Protocol 

State Staff Interview Protocol 

Web-based Surveys for TR and SPSSR Reviewers. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

12

The interview protocols were designed to collect information about areas of the evaluation framework 

that could not be obtained from secondary data sources. The Web-based surveys were developed to elicit 

specific information related to the conduct of TRs and SPSSRs from reviewers. The three instruments 

developed for the evaluation were submitted for Federal OMB clearance and approved prior to 

administration. Copies of the instruments are located in Appendix D. 

Methods 

The evaluation relied on a number of methods to collect and analyze information to ensure the 

achievement of evaluation objectives. Data collection methods are summarized in Table 1, followed by a 

description of data storage, cleaning, and analysis. A more detailed discussion of evaluation methods is 

included in Appendix E. 

Table 1. Summary of Data Collection Methods 

Data Collection 

Method 

Data Sources Participants Topics Covered 

Review of Program 

Documents 

Secondary qualitative 

data; included FY 2007 

BG applications, TR, 

SPSSR, and 

CSAP/CSAT internal 

reports 

States Achievement of 17 goals 

Program expenditures 

Program monitoring 

Analysis of Client/ 

Participant Level 

NOMs Data 

Secondary quantitative 

data retrieved from 

Web-based data 

collection system for 

FYs 2004–2008 

States, BG-funded 

programs, program 

participants/clients 

Abstinence 

Employment status 

Criminal justice system involvement 

Housing stability 

Social connectedness 

Analysis of State 

System-Level 

NOMs Data 

Secondary quantitative 

data retrieved from 

Web-based data 

collection system for 

FYs 2004–2008

States, BG-funded 

programs 

Access to services 

Retention in services 

Cost effectiveness 

Use of EBPs 

Interviews with 

State Staff 

Primary data; purpose 

of site visits was to 

administer semistructured State staff 

group interview 

protocol 

21 States selected to 

ensure diverse 

sampling of 

geography, population 

size, target populations 

served, and degree of 

dependence on BG 

funding 

Application development, review, and 

approval 

Allocation and distribution of BG 

funds 

Program monitoring and support 

Data collection, reporting, and analysis 

Federal leadership 

State infrastructure development 

State leveraging of Program policies 

and funds 

Interviews with 

Federal Staff 

Primary data; semistructured interviews 

occurred in person at 

Federal offices 

28 State Project 

Officers (SPOs), 

Government Project 

Officers (GPOs), and 

SAPT BG Federal 

management staff 

Application template and guidance 

development 

Application review and approval 

Program monitoring and support 

Data collection, reporting, and analysis 

Federal leadership 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

13

Web-based Surveys 

for TR and SPSSR 

Reviewers 

Primary data; 

respondents completed 

surveys online 

6 TR reviewers and 4 

SPSSR reviewers 

Preparation for reviews 

Identification of compliance issues 

Communication about compliance 

issues 

Development of review reports and 

recommendations 

Strengths of the review process 

Challenges and areas for improvement 

Data storage, cleaning, and analysis. Quantitative data for the evaluation, including NOMs data, 

financial data, and responses to the Web-based surveys, initially were stored and cleaned in Microsoft 

Excel® spreadsheets and subsequently imported to SPSS 16.0 software for statistical analysis. 

Frequencies and percentages were calculated for all data, and trend analysis was conducted for the 

NOMs. Qualitative data, including interview responses and abstracted Program information, initially 

were stored and cleaned in Microsoft Word® and subsequently imported to NVivo 7 software for 

content analysis. The data were analyzed for key themes and differences in responses across 

respondents. Specific qualitative examples that illustrated Federal and State accomplishments and 

outcomes for the SAPT BG Program also were stored in NVivo; many are included in this final 

evaluation report. 

Limitations 

Every effort was made to ensure evaluation rigor and objectivity; however, due to time, funding, and 

data constraints, the evaluation does have the following limitations: 

Generalizability of findings. The flexible nature of the SAPT BG Program allows for the creation of 60 

different State approaches for using BG funds to support prevention and treatment systems of care. The 

time and resources available for the evaluation limited the number of site visits and State interviews that 

could be conducted. Efforts were made during site visit selection to ensure that general characteristics of 

States were represented in the evaluation; however, the States selected for site visits may not be 

representative of all States and are not representative of the Territories and Jurisdictions receiving SAPT 

BG funds. The EAW provided feedback about themes that emerged as a result of data analysis to help 

ensure that no major themes related to Program activities, strengths, areas for improvement, or outcomes 

were missed. However, it remains possible that additional interviews conducted with all SAPT BG 

recipients might lead to slightly different themes and conclusions. 

In addition, for the analysis of the NOMs, generalizability is limited by varying total numbers of 

reporting States (Ns) across years. There is a marked increase in State participation in the collection and 

reporting of NOMs data beginning in FY 2008. Generalizability of NOMs results for earlier years is 

limited by the lower participation rate of States (less than half of funded States for many NOMs 

measures). 

Snapshot analysis of Program documents. The time and funding available for the evaluation, 

combined with the sheer volume of Program documentation in the form of 60 State BG applications, TR 

and SPSSR reports, and other internal reports, limited the scope of document review and analysis to 1 

year (FY 2007 for BG applications and the most recent year for TR, SPSSR, and other Program reports). 

This design did not allow for measurement of change from year to year related to the quality of 

information contained in Program documents or their usefulness to State and Federal stakeholders. In 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

14

addition, conclusions based in part on the analysis of Program documents might have differed slightly 

had multiple years been examined. 

Comparability of outcomes data. States are required to use prescribed data definitions for treatment 

outcomes or to provide documentation of different data definitions for State-collected data. However, 

not all States provided assurances that data definition rules were followed. In addition, during their 

interviews, State staff expressed concern that States continue to include different types and levels of data 

in the NOMs data set. 

Use of self-reported data. Virtually all of the data collected for the evaluation, including State BG 

applications, interviews and surveys, and NOMs data, were the self-reported perceptions and 

experiences of Program participants and managers. Social desirability may have affected participant and 

manager responses, particularly during the collection of NOMs data and in the State staff group 

interview, and respondents may have replied in a manner that would be viewed favorably by others. 

Key Findings 

The independent evaluation of the SAPT BG Program resulted in a number of key findings about the 

outcomes and impacts of the Program as well as successes and areas for improvement in Program 

processes and implementation. The six key findings to be discussed in this section include the following: 

The SAPT BG Program has demonstrated a positive effect on the health and lives of individuals 

with substance use disorders 

The SAPT BG Program has acted as a major impetus for improving State prevention and 

treatment systems’ infrastructure and capacity 

States have leveraged SAPT BG Program requirements, resources, and Federal guidance to 

sustain and improve their State systems 

Through a standard system of communication, monitoring, and reporting, CSAT, CSAP, and the 

States effectively and efficiently manage the SAPT BG Program 

The SAPT BG Program has contributed to the development and maintenance of successful State 

collaborations with other agencies and stakeholders concerned with preventing and treating 

substance abuse 

Although baseline data support the need for prevention services and activities, the use of national 

survey State estimates data alone to assess the NOMs limits CSAP’s ability to attribute changes 

in the NOMs to SAPT BG-funded prevention services and activities. 

Before beginning the discussion about specific key findings, the following section provides information 

about SAMHSA’s NOMs, which were used as part of the effort to determine SAPT BG Program 

effectiveness and impact. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

15

SAMHSA’s National Outcome Measures (NOMs) 

In an effort to demonstrate improvements in SAMHSA-funded substance abuse services access, 

capacity, quality, and positive effect on clients and State systems, SAMHSA collaborated with 

representatives of State and local substance abuse prevention and treatment agencies to develop the 

NOMs. The goal was to select outcome measures that could be used to manage and measure 

performance and to determine whether SAMHSA’s vision of ―a life in the community for everyone‖ is 

being achieved.6

 NOMs development represented decades of work with the Performance Data 

Workgroup of the National Association of State Alcohol and Drug Abuse Directors (NASADAD). Prior 

to NOMs, State outcomes monitoring studies included the Treatment Outcomes & Performance Pilot 

Studies 1997–1999 (TOPPS I) and the Treatment Outcomes & Performance Pilot Studies Enhancement 

1998–2001 (TOPPS II). 

Ten States participated in TOPPS I, which was designed to support a collection of distinctive State 

outcomes monitoring studies. The purpose of TOPPS II, funded in 19 States, was to demonstrate 

effective methods for collecting and analyzing treatment effectiveness information. TOPPS II was 

designed to pilot States’ development or enhancement of management information systems (MIS) and 

outcomes monitoring systems (OMS) for evaluating client outcomes within States’ substance abuse 

treatment systems. TOPPS II added a consensus-developed set of common data elements for the 

coordinated measurement of outcomes by all participating States. TOPPS II demonstrated that a core 

data set could be collected across individual States and also enabled States to pilot similar and divergent 

analytic and data utilization approaches. Many of the lessons learned from the TOPPS pilots informed 

the consensus process as SAMHSA, NASADAD, and State substance abuse agencies developed the 

NOMs. 

NOMs were developed to measure program effects on clients and participants and on the development 

of State and community prevention and treatment systems. The complete list of client and system level 

NOMs is presented in Table 2.7

 

Table 2. SAMHSA’s National Outcome Measures

Client/Participant-Level NOMs System-Level NOMs 

Decreased use of or continued abstinence from 

drugs and alcohol 

Increased or retained employment or school 

participation 

Decreased criminal justice involvement 

Increased stability in housing 

Increased social support and connectedness 

Increased access to services 

Increased retention in treatment 

Improved client perception of care 

Improved cost effectiveness 

Increased use of EBPs 

Collection of outcome and NOMs data was voluntary beginning in 2004 with not all States participating. 

Required collection and reporting of the NOMs for the SAPT BG Program began in FY 2007, and the 

State participation rate in the collection and reporting of NOMs data increased significantly in FY 2008. 

Interpretation of NOMs results for years prior to FY 2007 should consider the much lower participation 

of States (less than half of States reported data for most of the NOMs measures prior to FY 2007). To 

ensure transparency, NOMs data and analysis results are published on the SAMHSA website at 

http://www.nationaloutcomemeasures.samhsa.gov. 

To decrease data collection burden on SAPT BG recipients, CSAT and CSAP allowed States to use data 

already collected through the TEDS and the NSDUH to fulfill NOMs data reporting requirements. Data 

from these two sources were imported into the Program’s Web-based data collection system 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

16

(WebBGAS). TEDS data were used for client-level treatment outcomes, and NSDUH data were used for 

prevention outcomes. For treatment, States had the option either to accept the imported TEDS data to 

populate their treatment NOMs or to submit their own State data as long as the TEDS data definitions 

were used. For prevention, States also could choose to submit their own outcomes data as long as the 

same measures were used. Information about how data are collected for the TEDS and NSDUH and 

limitations of these data is located in Appendix F. 

Key Finding 1: The SAPT BG Program has demonstrated a positive effect 

on the health and lives of individuals with substance use disorders. 

Through participation in treatment programs funded by the SAPT BG, individuals with substance use 

disorders have demonstrated positive outcomes in all six client-level NOMs domains: alcohol and drug 

abstinence, employment/school participation, stable housing, social connectedness, criminal justice 

involvement, and retention in treatment. States report treatment NOMs either through submission of 

their own data collected from BG-funded providers or through pre-population of NOMs data fields with 

TED’s data. Regardless of which data collection and reporting method States choose, treatment NOMs 

are collected using the following TEDS data definitions: 

Client: An individual who has an alcohol or drug related problem, has completed the screening and 

intake process, has been formally admitted for treatment or recovery service, and has his or her 

client record 

Admission: The first date of service, prior to which no service has been received for 30 days 

Discharge: The last date of service, subsequent to which no service has been received for 30 days. 

Limitations to TED’s data collection and analysis influence the conclusions drawn about client 

outcomes: 

It is impossible to determine the reasons why individuals were discharged from treatment. 

Discharges include program completers, but also include people who leave against medical advice 

and people who leave for personal or family reasons. If data were collected and analyzed with 

respect to length of stay or degree of program completion, it is likely that client progress would be 

more pronounced for program completers and near-completers. 

Currently, when an individual moves from one treatment modality to another, for the purposes of 

TEDS collection, he/she is discharged from the first modality and ―newly‖ admitted to the next one. 

This makes the average percent change from admission to discharge smaller than it would be if 

admission were considered to be the beginning of service provision and discharge were the 

conclusion of treatment services, regardless of modality. 

Specific results and discussion of each client-level treatment NOM follow. 

Outcome 1. Increased Abstinence from Alcohol and Other Drugs 

Results of the abstinence measures demonstrated significant client improvement within and across years 

and the success of BG-treatment programs in assisting clients to reduce substance use. For this measure, 

abstinence is defined as no use of alcohol or other drugs in the past 30 days. As shown in Figures 1-1 

and 1-2, across all years, clients showed an average increase in alcohol and drug abstinence from 

admission to discharge (alcohol: 44 percent at admission versus 64 percent at discharge; other drugs: 40 

percent at admission versus 59 percent at discharge). For alcohol abstinence, there also was a positive 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

17

trend in percent change from year to year, which indicates that an increasing number of clients are 

abstinent from alcohol use after participating in BG-funded programs. 

Figure 1-1. Percent of Clients Abstinent from Alcohol from Admission 

to Discharge, FY 2006–2008 

3 4%

4 3%

5 6% 5 4%

6 2%

7 3%

0%

1 0%

2 0%

3 0%

4 0%

5 0%

6 0%

7 0%

8 0%

9 0%

100%

2006

(N=3 4) 

2007

(N=3 7)

2008

(N=4 9)

Discharge

Admission

(%) Percent Change

4 4% Admission - average

6 4% Discharge - average

(3 7% )

(4 4% ) (3 2% )

Figure 1-2. Percent of Clients Abstinent from Drug Use from Admission 

to Discharge, FY 2006–2008 

3 8% 3 4%

4 8%

5 4% 5 6%

6 8%

0%

1 0%

2 0%

3 0%

4 0%

5 0%

6 0%

7 0%

8 0%

9 0%

100%

2006

(N=3 2)

2007

(N=3 6)

2008

(N=4 9)

Discharge

Admission

(%) Percent Change

4 0% Admission - average

5 9% Discharge - average

(4 2% ) (6 5% )

(4 2% )

Outcome 2. Increased Employment and School Participation 

Figure 1-3 demonstrates that individuals reported an increase in employment and school participation 

after participating in BG-funded treatment programs (36 percent admission average versus 44 percent 

discharge average). These numbers show a positive effect within years and from FY 2007 to FY 2008; 

however, a lack of data from earlier years precludes drawing any conclusions about trends. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

18

Figure 1-3. Percent of Clients Employed/Attending School from Admission 

to Discharge, FY 2007–2008 

3 1%

4 1%

4 1% 4 6%

0%

1 0%

2 0%

3 0%

4 0%

5 0%

6 0%

7 0%

8 0%

9 0%

100%

2007

(N=4 0)

2008

(N=5 1)

Discharge

Admission

(%) Percent Change

4 4% Discharge-average

3 6% Admission-average

(1 0% ) (1 2% )

Outcome 3. Increased Stable Housing 

According to data presented in Figure 1-4, individuals participating in SAPT BG-funded programs 

demonstrated a slight increase in stable housing between admission and discharge (92 percent admission 

average versus 94 percent discharge average). For this NOM, individuals are asked whether they are 

homeless, and those who indicate that they are not homeless are counted as having stable housing. 

Because it is possible for one to have an unstable housing situation without being homeless, the question 

for this measure might assess stable housing more accurately if individuals were provided a definition of 

stable housing and asked whether their current housing situation met that definition. 

Figure 1-4. Percent of Clients with Stable Housing from Admission 

to Discharge, FY 2006–2008 

9 0% 9 4% 9 2% 9 5% 9 3% 9 4%

0%

1 0%

2 0%

3 0%

4 0%

5 0%

6 0%

7 0%

8 0%

9 0%

100%

2006

(N=3 2)

2007

(N=3 6)

2008

(N=4 7)

(4% ) (3% ) (1% )

9 2% Admission-average

9 4% Discharge-average

Discharge

Admission

(%) Percent Change
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

19

Outcome 4. Increased Social Support and Social Connectedness 

Individuals participating in BG-funded programs reported a significant increase in social support and 

social connectedness from admission to discharge, as shown in Figure 1-5. Social support and 

connectedness related to recovery are defined as attending self-help groups (either secular or faith 

affiliated), attending meetings of other recovery-oriented organizations, or interacting with family 

members or friends supportive of recovery. Percent change within years has been consistently large (50 

percent in FY 2007 and 49 percent in FY 2008), and there is a 2-year average increase from admission 

to discharge (43 percent versus 64 percent). These results suggest that participation in BG-funded 

treatment programs leads individuals to participate in community-based recovery activities and forge 

and renew relationships with friends and family members who are supportive of recovery. The total 

number of States reporting data for this measure is comparatively small, particularly in 2007 (N=11). 

Steady growth in the number of States reporting data for this measure will facilitate increased reliability 

and consistency of results for the social support and social connectedness measure. 

Figure 1-5. Percent of Clients Experiencing Social Support 

and Social Connectedness at Admission and Discharge, FY 2007–2008 

46%

39%

69%

58%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2007

(N=11)

2008

(N=31)

Discharge

Admission

(%) Percent Change

6 4% Discharge -average

4 3% Admission -average

(5 0% ) (4 9% )

Outcome 5. Decreased Criminal Justice Involvement 

Criminal justice involvement is measured by the number of arrests for any charge in the past 30 days. 

Data for the criminal justice measure presented in Figure 1-6 show a marked decrease in the percent of 

individuals arrested in the past 30 days from admission to discharge. Percent change was consistently 

large for all three years, indicating that participation in BG-funded substance abuse treatment programs 

is associated with a decrease in client arrests. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

20

Figure 1-6. Percent of Client Arrests in the Past 30 Days from Admission 

to Discharge, FY 2006–2008 

4 2%

3 7%

2 0%

1 3% 1 1% 1 1%

0%

1 0%

2 0%

3 0%

4 0%

5 0%

6 0%

7 0%

8 0%

9 0%

100%

2006

(N=2 4)

2007

(N=2 9)

2008

(N=4 4)

1 2% Discharge - average

3 3% Admission - average

Discharge

Admission

(%) Percent Change

(-6 9% ) (-7 0% )

(-4 8% )

Outcome 6. Increased Retention in Treatment for Long-Term Residential and 

Outpatient Services; Inconclusive Results for Short-Term Residential and 

Detoxification Services 

The retention domain measures length of stay in days of detoxification and treatment of individuals 

according to service modality. Research has shown that longer lengths of stay are correlated with 

improved treatment outcomes; thus, retention in treatment is perceived as a positive outcome for BGfunded detoxification and treatment programs8

. Length of stay data indicate an average increase in 

length of stay in long-term residential, outpatient, and intensive outpatient treatment programs (Table 3). 

Trend analysis for short-term residential treatment was inconclusive, as the number of days in treatment 

increased from FY 2006 to FY 2007, but decreased from FY 2007 to FY 2008. Data for additional years 

are needed to determine stronger trends. Results also are inconclusive for ambulatory detoxification 

services (which do not include inpatient detoxification services that are not funded through the SAPT 

BG Program). Although the numbers for FY 2006 and FY 2008 are consistent, in FY 2007, there was a 

large increase in number of days spent in detoxification programs. Further study is needed to determine 

the reasons for this increase. Data from future years also will help to form conclusions about trends in 

length of stay in detoxification programs. 

Table 3. Length of Stay (LOS) in Days by Treatment Modality, FY 2006–2008 

Treatment Modality FY 

2006 (N=29) 

Average 

LOS 

Treatment Modality FY 

2007 (N=33) 

Average 

LOS 

Treatment Modality FY 

2008 (N=49 

Average 

LOS 

Short-term residential 28 Short-term residential 33 Short-term residential 31

Long-term residential 85 Long-term residential 85 Long-term residential 90

Outpatient 102 Outpatient 115 Outpatient 119

Intensive outpatient 81 Intensive outpatient 103 Intensive outpatient 107

Ambulatory 

detoxification

40 Ambulatory 

detoxification

80 Ambulatory 

detoxification

40

Results from the secondary analysis of NOMs data suggest that the SAPT BG Program is successful in 

assisting individuals with substance use disorders to improve their health and quality of life related to 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

21

abstinence, criminal justice involvement, employment/school participation, housing stability, and social 

support and connectedness. The Program also has been successful in increasing length of stay in longterm residential and outpatient programs. Further study is needed to address the inconclusive results for 

client retention in short-term residential and ambulatory detoxification programs. Due to the mandatory 

NOMs reporting requirement that became official in FY 2007, States are steadily improving their 

reporting on client-level NOMs, which should enable SAMHSA to improve its ability to demonstrate 

Program effects on the lives of individuals with substance use disorders. 

Key Finding 2: The SAPT BG Program has acted as a major impetus for 

improving State prevention and treatment systems’ infrastructure and 

capacity. 

SAPT BG Program emphasis on demonstrating the effectiveness of BG-funded programs and services to 

reduce substance abuse and to improve the lives of those affected by it has driven State system 

infrastructure development and capacity improvements. BG requirements have incentivized States to 

address the effects of substance abuse in at-risk populations by using innovative evidence-based 

strategies and have laid the foundation necessary to build comprehensive client data systems. The ability 

of States to collect consistent and representative outcome measures has not only improved the extent to 

which programs and services may be described, but has served as the catalyst for data-driven assessment 

and decisionmaking at the State level. Discussion of the effect of the SAPT BG Program on State 

systems is based on the results of quantitative and qualitative data analysis and is organized according to 

the following outcomes: 

Increased availability of services for diverse and underserved populations 

Increased development and implementation of EBPs 

Improved development and collection of specific outcome measures 

Increased development and maintenance of State data management systems. 

Outcome 1. Increased Availability of Services for Diverse and Underserved 

Populations 

SAPT BG requirements proved effective in ensuring the availability of substance abuse prevention and 

treatment services to diverse and underserved populations. States relied on dedicated BG set-aside 

funding to provide the necessary resources for clients who may have otherwise struggled to obtain 

needed treatment and prevention services. Among all States, dedicated funding for pregnant women and 

their dependents, intravenous drug users (IVDUs), and HIV intervention services was unique to the BG 

Program. 

Women and Children 

Goals 3 and 9 of the SAPT BG Program require States to set aside a portion of their grants to serve 

pregnant women and women with dependent children and to ensure that pregnant women seeking or 

referred to services are given preference in admissions to appropriate treatment facilities. Goal 3 also 

requires States to provide childcare and prenatal care to women undergoing substance abuse treatment. 

Single State Agencies (SSAs) for substance abuse adhered to these requirements by making contractual 

agreements with and conducting compliance reviews and onsite monitoring of BG funding providers 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

22

and other subrecipients. SSAs allotted TA and training to subrecipients to ensure that set-asides were 

met in accordance with BG requirements. 

In addition, some States dedicated their BG set-aside funding to providers specializing in women’s 

services, while other States required that subrecipients use a portion of their contracts to serve women 

and their dependent children. The BG requirement for women’s treatment has resulted in States going 

beyond what was required in 1994 to continue to increase the breadth and quality of services to women. 

As Figure 1-7 shows, women’s treatment services expenditures for all BG-funded entities have grown 

by more than 50% since the beginning of legislation requiring BG recipients to set aside funds for 

women’s services. 

Figure 1-7. Women's Treatment Services Expenditures for All BG-Funded 

Entities: 1994 (Baseline) and 2005–2007 

$ 0

$50,000,000

$100,000,000

$150,000,000

$200,000,000

$250,000,000

$300,000,000

$350,000,000

1994 2005 2006 2007

In order to develop a complete 

picture of how services are being 

provided to a key target population, 

one State generates detailed 

monthly reports about its women’s 

programs. Women’s treatment 

providers in the State also meet 

quarterly as a part of a practice 

improvement collaborative. The 

collaborative provides a rich source 

of information that is then used to 

identify women’s treatment needs. 

BG funding for women’s services served as the impetus for 

further funding from other sources to offer continued 

support for existing programs and to develop new and 

innovative women’s programs to fill gaps in treatment 

services. Policymakers and legislators, in particular, have 

become increasingly aware of and responsive to the needs 

of underserved and diverse populations such as pregnant 

women. For example, one State expanded halfway house 

services for reuniting previously incarcerated women with 

their children by leveraging an existing BG-funded 

program, and another State directed Medicaid resources 

into the foster care system for children of women with 

substance use disorders. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

23

IV Drug Users

A majority of States reported offering comprehensive services for IVDUs in accordance with SAPT BG 

set-aside requirements. Methadone programs were the most frequently reported service provided, 

followed closely by outreach initiatives. SSAs collaborated with agencies such as public health and 

social service agencies, State and local health departments, prisons, faith-based programs, consumer 

advocacy groups, and law enforcement to disseminate information about the availability and range of 

IVDU service offerings. SSAs also provided training and TA to providers about IVDU services. States 

described the following types of outreach activities: 

Substance abuse screening and referrals

Health screenings for sexually-transmitted diseases (STDs), Hepatitis B and C, and 

Tuberculosis

Public education and community presentations

Skill-building for IVDUs and their families

Street outreach

Media campaigns and literature distribution

Help lines.

In addition to conducting outreach activities, a number of States used BG set-asides to provide 

detoxification services such as residential programs (including short-term, long-term, and transitional 

treatment), hospital-based programs, emergency services, social treatment programs, and medically 

managed detoxification. 

HIV Services

As a requirement of the SAPT BG Program, States whose rate of AIDS cases is 10 or more per 100,000 

individuals are mandated to expend a portion of their BG funds on HIV early intervention services. 

These States described a wide variety of HIV early intervention and monitoring services available 

through BG-funded substance abuse treatment programs. Many States provided a range of services, 

including:

HIV testing

HIV/AIDS/STD education, including risk and prevention education 

Individual and group counseling, before and after HIV testing 

Case management services. 

To ensure that subrecipients and providers were complying with BG requirements and that clients were 

receiving appropriate HIV intervention services, a number of States provided TA and training to 

subrecipients, which went beyond the scope of typical monitoring activities. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

24

“EBPs instill intellectual vitality 

into the service system and serve 

as reminder to professionals that 

there is a science-based 

methodology for measuring 

outcomes. As a result of the BG 

emphasis on EBPs, there has 

been an increasing awareness 

about current EBPs in treatment 

and prevention, and while this 

shift may have occurred without 

Federal leadership, it has 

happened a lot faster with the 

impetus of the BG.” 

– State Training Coordinator

Outcome 2. Increased Development and Implementation of Evidence-Based 

Practices (EBPs) 

From FY 2005 to FY 2007, States demonstrated a substantial increase in the number of EBPs in 

SAMHSA’s National Registry of Effective Practices and Programs (NREPP) used with clients. States 

implemented 22 NREPP EBPs in FY 2005 and 119 EBPs in FY 2007. Prevention and parent education 

EBPs showed the greatest increase in use from FY 2005 to FY 2007. 

State staff reported that the number of EBPs and innovative 

services available to consumers increased as a result of BG 

requirements, TA and training, and Federal support. States 

reported that prevention and treatment systems are now built 

around EBPs, which have been incorporated into a range of 

services and subrecipient and provider trainings. 

States have used Federal leadership to advocate for a shift 

toward EBPs. State policymakers and legislators have become 

open to the incorporation of EBP requirements into licensure 

regulations, policies, and systems related to youth access to 

tobacco, gender issue awareness, and women-specific 

treatment standards. Most States noted that all prevention 

activities are based on EBPs. One State’s prevention 

coordinator stated that the development and broad 

implementation of prevention EBPs ―would not have 

happened without the funding and the synergy of the BG.‖

Specific NREPP EBPs implemented by States to expand service capacity are listed in Table 4. 

Table 4. Examples of Evidence-based Practices Implemented with SAPT BG Funds 

Prevention EBPs Treatment EBPs

Al’s Pals: Kids Making Healthy Choices

CASASTART (Striving Together to Achieve Rewarding 

Tomorrows) 

Communities Mobilizing for Change on Alcohol

Creating Lasting Family Connections

Dare to be You

Family Matters

Keepin’ It Real

Parenting Wisely

Project Northland

Project SUCCESS (Schools Using Coordinated 

Community Efforts to Strengthen Students)

Project ALERT

Strengthening Families Program

Assertive Community Treatment

Motivational Interviewing 

Matrix Model

Recovery Support Services

Community Reintegration Programs

HIV Rapid Testing

Wraparound Services 

Family Behavior Therapy

Multisystemic Therapy for Juvenile Offenders

Pathways’ Housing First Program

Seeking Safety

Trauma Recovery and Empowerment Model

Outcome 3. Improved Development and Collection of Specific Outcome Measures 

SAPT BG data collection and set-aside requirements have helped States improve their prevention and 

treatment delivery systems in ways that otherwise would have been impossible. States have leveraged 

SAPT BG Program requirements and resources to require performance data from subrecipients that are 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

25

“The data collection process 

has been a very powerful one 

for the development of our 

system because of the almost 

philosophical position in 

regards to the use of data in 

decision-making. People at all 

levels recognize that the data 

do say something significant 

about the efficiencies and the 

effectiveness of programs.” 

– State interviewee

subsequently used for program improvement and for the provision of comprehensive prevention and 

treatment services. For example:

BG data collection and set-aside requirements have served as the impetus for uniform, 

consistent data collection. Several States noted that one of the results of Federally-required BG 

data collection is that it provides more consistent data across and within States. Additionally, as 

States continue to collect data, comparisons and trend data may be used to gauge State progress 

toward meeting BG objectives. 

Collection of outcome measures offers States a profile of activities oriented toward prevention 

and treatment. The data collection helps States to present a profile of their activities and provides a 

―snapshot‖ as to what a State’s population looks like as it relates to a particular field. States then use 

the data collection requirement to provide increased 

feedback to providers, which ensures that the information 

the State collects is useful to providers and other 

subrecipients.

Development and collection of data focuses on outcomes.

States appreciated that data collection is moving from an 

emphasis on process to outcomes data, which has allowed 

States to be able to promote, leverage, and justify 

requirements for outcomes data. Having to collect these 

measures has ―forced necessary improvements‖ in State 

systems. One State noted that data collection activities 

required by the BG have propelled the State away from a 

cost-driven system and toward an outcomes-driven system.

Outcome 4. Increased Development and Maintenance of State Data Management 

Systems

The BG requirement that States collect and report outcome measures has catalyzed the development of 

State data collection systems. States have dedicated BG resources that might otherwise have been used 

for direct services to developing or improving their capacity to collect and report BG-related data. A 

majority of States have developed or honed their Web-based reporting systems, and many others use 

automated systems. Two States mentioned that while their data reporting systems were not yet Webbased, they were in planning stages to implement statewide Web portals to allow for instantaneous data 

accessibility. One State noted that improving data collection systems ultimately improves 

implementation, stating that ―data collection drives systems planning and program policies, which will 

change practice and improve implementation.‖

State data management systems necessary to collect required outcome measures for the BG Program

have catalyzed service integration and coordination across and within States, led to the development of 

data infrastructures such as electronic health records (EHRs), and provided the ability to report 

outcomes within and across States. In addition, development and maintenance of State data management 

systems have enabled Federal and State partners to:

Gain a better understanding of State systems and activities

Identify service gaps and address emerging needs

Respond to stakeholder inquiries about Program achievements and effectiveness
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

26

One State noted that 

while it may be difficult 

sometimes to come up 

with the necessary level 

of State funds each year, 

“without the MOE, there 

would be even less State 

funding available.”

Identify provider TA and training needs

Ensure accountability and compliance with BG legislative requirements

Demonstrate outcomes and justify funding requests

Improve the quality of data collected and submitted as the data management systems become more 

sophisticated. 

Key Finding 3: States have leveraged SAPT BG Program requirements, 

resources, and Federal guidance to sustain and improve their State systems.

States have used the BG requirements and funding to go beyond the intended and expected outcomes of 

the Program, leveraging BG resources to sustain and improve State substance abuse prevention and 

treatment systems. Although not a requirement for BG funding, State leveraging of SAPT BG Program

requirements, resources, and Federal guidance demonstrates the importance of the BG in the 

development of State systems. States have leveraged BG resources for many reasons: to advocate for 

continued and/or additional funding, implement changes to the prevention and treatment system through 

new programs and services, and spur State policy changes. 

Outcome 1. Prevented Harm to the Service System Resulting from State Legislature 

Reductions in Funds for Prevention and Treatment and Advocated for Additional 

Funding

Using the legislative requirements included in the application template 

and guidance, States have gained leverage with their legislatures and 

other funders of substance abuse prevention and treatment services. 

This is an unanticipated, yet positive, outcome of the process. In this 

way, BG requirements, especially the MOE, offer SSAs a certain level 

of budgetary ―protection‖ and are helpful and effective in keeping 

substance abuse services available. A few States discussed using the 

guidance document and language about the State spending MOE 

requirement to convince their State legislatures not to reduce funding 

for substance abuse services. Other States noted that the SSA for Substance Abuse Services requirement 

helps to insulate the SSA from the constant realignment and disbanding that can occur as a result of 

changing political environments or budgetary constraints. 

States also report using BG data to advocate for additional funding from State and other sources. For 

example, positive BG outcomes data have led to an increase in one State’s general revenue funding. 

Another State has used its data and research to demonstrate that EBPs for prevention and treatment have 

resulted in fewer arrests and reduced demand for inpatient or medical care; because it has documented 

and verified BG Program outcomes, it experiences ―much less push-back‖ in the budgeting process. This 

State notes that the devotion of State dollars to the treatment and prevention of substance abuse is no 

longer a partisan issue within the legislature, and attributes this to the impact of the BG. 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

27

Outcome 2. Set State Policies and Priorities Based on Federal Leadership and 

Development of National Policies and Priorities

States have used BG resources to drive State policies and priorities, most notably in the areas of 

prioritizing women’s services, IVDU services, and HIV services; developing prevention services and 

implementing the Strategic Prevention Framework; and requiring performance data from subrecipients. 

States also have used BG funding and its requirements to advocate for other policy changes. One State 

noted that its policymakers have become more responsive to the issue of children of women with 

substance abuse issues being moved into foster care, directing State Medicaid and BG resources into the 

foster care system. Others described drug enforcement policies; policy changes at the State level about 

mental health and substance abuse services for the criminal justice population; the incorporation of EBP 

requirements into licensure regulations; policies and systems related to youth access to tobacco; gender 

issue awareness and women-specific treatment standards; and the successful passage of a State Clean 

Indoor Air Act, that ―wouldn’t have happened without the funding and the synergy of working together 

and with the community.‖

One State noted that SAMHSA has given States the ―right incentive‖ with BG funding, and the State has 

used these guidelines and resources in meaningful and sustainable ways. This State noted that three 

funding streams in the State pay for substance abuse services. SAMHSA funding ―trumps them all‖ and 

is the entity that truly sets policy, practice, and requirements.

Prioritization of women’s, IVDU, and HIV services

Most Federal staff felt that BG resources had been used to leverage policy changes. They noted that the 

MOE requirements and the requirements to address the needs of particular groups (such as women and 

children) give the SSAs the leverage to implement policies that States might not otherwise accept. Many 

State staff echoed this belief, citing as examples the emphasis on women’s services and the HIV setaside. One State prevention group used 20 percent of its BG funds to partner with four other 

departments, helping to coordinate prevention efforts statewide. Because of this partnership, the group 

successfully lobbied to have a portion of State revenues dedicated specifically to women’s treatment 

services and to increase women’s Medicaid benefits from 2 months to 12.

Several States noted that they also have expanded services for IVDUs and have used Federal leadership 

to advocate for a shift toward EBPs. One State has used BG funding to push approval packages through 

its legislature to support a peer recovery network of services. Another State described an initiative in 

which the SSA is using NSDUH to identify areas of the State that need, but are not receiving, alcohol 

and drug services. 

Prevention services and implementation of the Strategic Prevention Framework 

BG requirements also allow States to advocate for substance abuse strategies to be built into systems 

that otherwise might not address the problem at all. CSAP respondents mentioned that BG funding is 

central to prevention because it provides much--and in some States and Jurisdictions, close to all--of the 

State’s prevention budget. As a result, they said, CSAP plays a central leadership role in prevention. 

They pointed to CSAP’s development of the Strategic Prevention Framework (SPF), which is guiding 

many States’ and Jurisdictions’ approach to prevention.

Requiring performance data from subrecipients

States also noted that BG data collection requirements have given them the leverage necessary to require 

their subrecipients to report data as part of BG contractual requirements. One State noted that ―data 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

28

collection is an expected expense, and it is expected of the subrecipients to be timely and accurate in 

their reporting.‖ Data collection and reporting have served as the impetus for providers and counties to 

review and revise their own data collection practices. States then use the data collection requirement to 

ensure that the information the State collects is useful to providers and other subrecipients. 

Some States noted that the BG data collection requirement has driven a focus on the quality of the data 

they collect. One State noted that they ―go back to the providers to get them to pay more attention to the 

quality of the discharge data instead of just plugging in admission data for discharge.‖ Another State 

noted that the data collection requirement has compelled the State to train its providers. This State found 

that some providers ―don’t take the time to discharge clients and the data collection requirements force 

the State to remind providers to discharge clients and make a complete record.‖

Outcome 3. Used BG Funds as Seed Money for New Programs that Other Public 

and Private Organizations Have Subsequently Funded

More than half of States interviewed (12) reported having used SAPT BG funds to initiate programs that 

have subsequently continued with State and other funding sources. The remainder of States interviewed 

(9) noted that although no programs have been completely shifted from BG funding, most now receive a 

combination of State, Federal, and other funding sources; many programs that began with only BGfunding have expanded their funding base. 

Programs for women’s services were the most frequently cited example of such expanded funding. For 

example, one State described its ―Healthy Beginnings‖ program, a treatment program for pregnant 

women that includes housing and treatment for mothers and their children. The program began as a 

CSAT demonstration project, then was funded through the BG; because the program has continued to 

demonstrate healthy birth outcomes, the State legislature and general revenue now fund it. Another State 

used BG funds to initiate its Drug Court program, which the legislature now funds. Other examples 

include criminal justice programs, programs for indigent individuals convicted of driving under the 

influence (DUI), older adult services, and HIV services. States with such programs often described using 

BG funding as ―seed money‖ and sought to fund programs with a high likelihood of sustainability. 

States that have not shifted programs entirely from BG funding 

to other sources stated that most of their programs receive a 

combination of funding from sources that remain fairly stable. 

However, several States described ways in which BG funding 

has been the catalyst for the development of a variety of 

programs, which are then leveraged to obtain additional funding. 

For example, one State has a perinatal set-aside program that 

began as a pilot project and has since developed into a much 

larger Perinatal Treatment Expansion Program that receives $25 

million in State and Federal funds. BG funding also ―helped get 

the ball rolling‖ on another State’s Elder Service Providers program, which has grown from 4 to 21 

programs that now serve approximately 150,000 of the State’s population aged 60 and older. Several 

States noted that receiving BG funding makes their providers more competitive when seeking additional 

funding sources. 

One State noted that their 

competitive request for 

proposal (RFP) process for 

providers to receive BG 

funding “will necessarily drop 

some providers, so they know 

the money is more of a 

developmental grant, and they 

will have to find other support.”
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

29

Key Finding 4: Through a standard system of communication, monitoring, 

and reporting, CSAT, CSAP, and the States effectively and efficiently 

manage the SAPT BG Program.

A nationwide grant program as complex as the SAPT BG Program requires effective and efficient 

management to coordinate a myriad of Program activities across diverse States. CSAP and CSAT have 

developed several successful management strategies to steer States as they work toward the 17 

legislative goals. Management involves the following crucial activities:

Communication of Program goals and activities to create a standardized system

Monitoring and oversight to facilitate open communication

Leading of complex data collection and reporting activities

Providing TA and training to ensure compliance and to aid States in meeting their goals.

Outcome 1. Communicated Program Goals and Activities to Create a Standardized 

System

CSAT and CSAP communicate Program goals and expectations to the States in ways that lead to 

uniform documentation and assessment of State BG activities. SAPT BG legislation requires that CSAT 

and CSAP establish standards for the acceptance of the BG application and that they disseminate this 

guidance to the States. This annual process begins with the dissemination of the BG application 

guidance and template, which conveys the proper structures for reporting past, present, and planned 

activities related to the State’s compliance with the Program’s 17 legislative goals. The application 

template guides the States in the creation of a single document that includes an annual report, a progress 

report, and a State plan. 

The BG application is a standard document that enables CSAT and CSAP to obtain the information 

needed to assess and monitor compliance with Program requirements, learn about States’ intended use 

of SAPT BG funds, and document the progress States have made in using those funds to achieve their 

goals. Federal staff interviewed agreed that, for the most part, it is a consistent and standardized 

structure that leaves little room for misinterpretation and provides SPOs with a ―snapshot‖ of State 

activity. 

Technical reviews and SPSSRs are the final step for communicating Program goals and activities. These 

reviews allow direct and face-to-face communication between State and Federal staff and result in 

guidance documents with recommendations for improvement and suggestions for TA. They allow 

Federal staff to be sure that States have the capacity to adhere to BG requirements and can collect, 

report, and use performance data in ways intended by Program specifications.

Outcome 2. Provided Monitoring and Oversight to Facilitate Open Communication

Monitoring and oversight activities—the application review and approval, TRs, SPSSRs, and Federal 

data collection activities—encourage open and flexible communication among Federal, State and 

subrecipient levels. Application review and approval activities are a major source of communication that 

leads to a longer-term collaborative Federal-State relationship. The majority of the States interviewed 

described various ways of providing feedback and seeking clarification, including:

Communication with SPOs
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

30

“The review process provides 

SAMHSA with a better reality 

base for the state of the 

service system and gives us 

an opportunity to showcase 

our efforts to the Federal 

funding entities.” 

– State Director

Annual NASADAD meetings and surveys

Comments through the Federal Register 

E-mail list serve messages and standard mail communications with SAMHSA.

The SPO is the linchpin of the relationship, providing the initial 

review of the application and working with the State via WebBGAS, 

e-mail, and phone to clarify issues and discuss difficulties. The SPO 

manages communication between the State and the subsequent layers 

of review, including the Team Leader, the Branch Chiefs, the Synar 

Coordinator, and the Office of Grants Management. These multiple 

reviews ensure that monitoring and oversight are thorough, unbiased, 

and most likely to identify emerging State needs. This, in turn, 

fosters State-to-subrecipient feedback in identifying areas for 

improvement, determining need among the target populations, identifying methods for collecting and 

reporting data, and recognizing emerging TA needs.

Technical reviews and SPSSRs are other venues in which communication has improved. The reviews 

allow Federal staff to see firsthand how the State is using its BG money and allow a dialogue between 

the SPO and the State. Two-thirds of the States agreed that the TRs improve communication with the 

Federal staff and that the SPSSR process improves Federal and State information exchange by allowing 

for better clarification of expectations and communication of what is pending at the Federal level. On 

the Federal side, the reviews enable the SPOs to better understand the State’s system, identify strengths 

and areas for improvement, and make suggestions for TA.

Outcome 3. Led Complex Data Collection and Reporting Processes

CSAT and CSAP developed the NOMs and began requiring States to collect and submit data focused on 

client- and system-level outcomes in response to an OMB PART recommendation. States were asked to 

provide feedback regarding what they would need to implement the requirements; how to conceptualize 

the demographic and service content to be collected through NOMs; the data variables; and specific 

measures, such as the social connectedness NOM. States also provided and received feedback with 

regard to technical issues related to submitting data, particularly around issues of State and Federal data 

system incompatibility, technical issues about transmitting data, and the ―nuts and bolts‖ of data.

Federal data collection helps provide a ―big picture view‖ of outcome measures that can contribute to 

change at the State and local level. CSAT and CSAP are using NOMs in particular to document Program

outcomes. Other outcomes of the management of the Federal data collection and reporting activities 

include:

Development and improvement of State data collection capacity

Identification of State and subrecipient TA needs

Monitoring and accountability

Feedback to stakeholders on all levels

Identification of trends within and among States

Catalyst for States to focus on data quality.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

31

Outcome 4. Provided TA and Training to Ensure Compliance and to Aid States in 

Meeting their Goals 

The Federal TA and training process has several strengths, which include that it provides access to 

skilled experts, involves responsive SPOs, features a user-friendly TA tracking system and consultant 

database, provides access to high-quality data collection and management training, and offers a range of 

TA topics to accommodate different State needs. The TA and training process is so responsive in part 

because States are involved in developing TA and training plans through the SPSSR and TR program 

monitoring processes. State plans describe and prioritize challenge areas, identify TA support sources, 

and develop timelines with action items. Several State prevention and treatment systems improved as a 

result of data-related TA recommendations, and program monitoring reports were used to identify gaps 

in the uniformity of data collection and to evaluate readiness to collect data elements required by the 

NOMs. 

Successful organization of TA and training can allow States with tight budgets to provide TA and 

training to providers and encourages the development of the infrastructure necessary for implementing 

BG activities such as CSAP’s SPF. SAMHSA supports States by providing access to online training and 

TA and by providing flexible training and TA to allow for a range of State circumstances and need. 

States can request TA either during reviews or directly from SPOs. Subrecipient TA and training varies 

from State to State, but is most commonly generated through a needs assessment. Table 5 demonstrates 

the broad array of TA that is provided to States and subrecipients, as well as TA outcomes. 

Table 5. TA and Training Topics and Outcomes 

Federal-to-State TA and Training Topics Federal-to-State TA and Training Outcomes 

Data collection and management systems

Readiness to collect and report NOMs

Strategic planning

Standard operating procedures

Fiscal measures and compliance

Cross-training for collaborating agencies

Disaster readiness

EBPs and programs

Improved capacity and capabilities of data collection 

and management systems

Improved ability to analyze and use Program data

Improved connectivity between SSA and other 

stakeholders

Increased capability to develop and enforce standard 

treatment protocols

Increased implementation of EBPs throughout the State

State-to-Subrecipient TA and Training Topics State-to-Subrecipient TA and Training Outcomes 

EBPs and programs 

Standardized assessment tools

Provider ethics

Cross-agency collaboration

Community-based prevention networks

Data collection, management, and utilization

Increased development of provider skills in using EBPs

Enhanced capacity and ability to collect and report data

Improved ability to engage and retain clients

Improved provider attitudes toward becoming clientfocused and recovery-oriented 

Improved ability to identify and apply for funding 

opportunities

Key Finding 5: The SAPT BG Program has contributed to the development 

and maintenance of successful State collaborations with other agencies and 

stakeholders concerned with preventing substance abuse and treating 

substance use disorders. 

SSAs have increased the development and maintenance of collaborative working relationships with a 

variety of other Federal, State, and local agencies and providers. SSAs fostered many of these 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

32

“At first, everyone was 

grumbling about Synar and 

the extra work it meant for 

our staff. We didn’t know how 

to go about all of the Synar 

activities. But we got a lot of 

great TA from SAMHSA, and 

we eventually made the 

connections we needed with 

the other State agencies [to 

implement the Synar 

program]. Now, Synar is one 

of our greatest success 

stories because we have 

hard data to show for our 

work.”

– State BG Coordinator

partnerships as they worked to accomplish the Program’s 17 legislative goals. These State collaborations 

serve five critical roles:

To increase achievement of Synar Program goals and objectives

To improve the coordination of prevention services

To improve the coordination of treatment services with public and private health insurers

To expand services and programs available through joint funding initiatives

To increase the ability to address statewide critical public health or safety issues.

Outcome 1. Increased Achievement of Synar Program Goals and Objectives

The purpose of the Synar Program is to reduce the extent to which tobacco products are available to 

individuals under age 18 by enforcing laws that prohibit manufacturers, retailers, and distributors of 

tobacco products from selling or distributing these products to any individual under the age of 18. 

Successful implementation of the Synar Program is required under Goal 8 of the SAPT BG Program, 

and States that are not achieving specified vendor compliance rates can be subject to a strict withholding 

penalty of up to 40 percent of the State’s SAPT BG funding award. States pursue the following key 

objectives for the Synar Program: 

Conduct annual random, unannounced inspections of retail 

tobacco outlets (also known as the Synar survey). This includes 

developing a random sample, maintaining lists of tobacco 

merchants, developing Synar protocols, and conducting the 

actual compliance checks

Educate tobacco vendors about Synar-related laws and 

regulations

Analyze Synar data to determine compliance at local, regional, 

and statewide levels and note regions that need to improve 

Synar compliance

Develop and submit an annual report to SAMHSA that details 

State progress in reducing tobacco availability to youth, the 

methods used to identify outlets, inspection procedures, and 

plans for enforcing the law in the next fiscal year.

To realize Synar program goals, SSAs collaborated with a wide variety of State agencies, advisory 

committees, tobacco retailers, and research universities. SSAs reported that the Synar program fostered 

much collaboration that otherwise would not have been initiated or maintained with the following types 

of State agencies and local organizations:

Department of Health or Public Health: Involved in oversight of all Synar activities

Division of Health Statistics or Program Evaluation: Tasked with drawing the random sample of 

tobacco merchants and administering the survey

Tobacco and alcohol control agencies: Maintained lists of tobacco merchants in the State, and 

conducted merchant compliance checks
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

33

Synar for Underage Alcohol 

Sales

One SSA collaborates with 

the State’s alcoholic 

beverage control agency to 

use the Synar process and 

merchant lists for conducting 

compliance checks on 

alcohol vendors. The State is 

beginning to see modest 

decreases in underage 

alcohol sales as a result of 

Synar-like monitoring and 

enforcement.

Addressing Substance Abuse, 

Violence, and School Dropout

One SSA contracts with the State 

Board of Education to administer 

a prevention and early 

intervention program in local 

school districts that aims to 

prevent substance abuse, 

decrease school violence and 

dropout rates, and increase 

academic achievement. The

program is co-funded by the 

SAPT BG and the Federal 

Department of Education Safe 

and Drug Free Schools initiative.

State and local law enforcement agencies: Assisted in conducting merchant compliance checks and 

any subsequent actions taken in violation cases 

Research universities: Analyzed Synar data to identify trends and compliance issues 

Synar workgroups or advisory committees: Provided feedback 

and guidance to the SSA 

Comptrollers and public accountants: Tracked tobacco sales 

Prevention providers: Assisted in the development of Synar 

protocols 

Subrecipient coordinating agencies: Submitted information for 

the Annual Synar Report. 

In addition to performing tasks required by the Synar regulation, 

collaborating State agencies and organizations worked together to 

use and disseminate Synar findings. Examples of these activities 

included: 

Disseminating the Synar report to inform the legislature, media, 

and the public 

Presenting Synar results at national conferences (e.g., National Synar Conference, Liquor Control 

Board Conference) 

Planning future strategies to further decrease youth access to tobacco 

Involving additional stakeholders in decreasing youth tobacco use (e.g., tribal organizations, faithbased organizations, advocacy groups). 

Outcome 2. Improved Coordination of Prevention Services 

SSAs coordinated with a variety of agencies and organizations to ensure appropriate and effective 

statewide prevention services. Prevention education and community-based initiatives were among the 

prevention strategies that most commonly involved such collaboration. 

Prevention Education

Prevention education directed at school-aged children and 

youth comprised a large proportion of SSA collaborative 

efforts. Working with State departments of education, public 

health, alcohol control, and public safety, SSAs and their 

subrecipients conducted the following types of activities: 

Offering classroom presentations focused on the 

characteristics and effects of substances and available 

prevention and treatment services 

Developing and producing grade-specific health education 

curricula on alcohol, tobacco, and other drugs 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

34

Teen Institute

A community-based partnership in 

one large State comprised of 

prevention and treatment 

providers, educators, parents, and 

students sponsors an annual 

Teen Institute, which brings

together hundreds of teens, 

parents, and educators for one 

week of learning and planning for 

school-based, peer-implemented 

prevention programs.

Developing educational materials to reach students K-12 and their parents on issues such as 

underage drinking and media literacy 

Conducting workshops, guest speaker presentations, and other schoolwide activities during Red 

Ribbon Week (an annual drug prevention campaign in schools) 

Providing peer leadership trainings to middle and high school students 

Offering training on drug prevention strategic planning and campus-based social marketing at 

colleges and universities. 

Community-based Initiatives 

Support of community coalitions and initiatives was another 

significant area for collaboration and coordination in 

prevention activities. Efforts were designed to increase the 

scope and efficacy of community-based activities and the 

influence of community coalitions in preventing substance 

abuse. Examples of community-based collaboration include: 

Working with smaller communities to build their capacity 

to develop, implement, and sustain comprehensive 

prevention programs 

Developing and using standardized coalition planning 

processes 

Implementing prevention EBPs at the community level 

Using a train-the-trainer approach to provide community coalitions with knowledge to implement 

science-based prevention education programs 

Training faith-based organizations and religious leaders to help them address prevention issues and 

build capacity to work effectively in their communities on youth substance use prevention 

Offering TA to help community coalitions more effectively address the needs of individuals who are 

deaf or hard of hearing. 

Outcome 3. Improved Coordination of Treatment Services 

More than half of the States interviewed reported that coordination of State substance abuse treatment 

services and programs has improved significantly as a result of SAPT BG Program activities. States 

credited the BG and SAMHSA’s oversight for a ―long history‖ of collaboration among programs and 

State agencies. To achieve the treatment-related legislative goals of the SAPT BG Program, SSAs have 

developed interagency agreements (including memoranda of understanding (MOUs)) and required 

subrecipients to establish linkage agreements with one another. 

The majority of States initiated interagency agreements, including formal MOUs, to improve the 

continuum of care for treatment program clients. They established interagency agreements to ensure that 

clients were provided services such as prenatal and child care; tuberculosis and HIV screening, 

education, and treatment; and vocational rehabilitation. In addition, these formal contractual 

relationships were critical to achieving high-level coordination among State agencies that interact 

frequently with individuals with substance use disorders. Linking agencies helps SSAs to ensure that 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

35

organizations provide comprehensive and holistic services to assist clients in all areas of their lives. 

SSAs have developed and maintained a large number of formal interagency and subrecipient agreements 

with several types of State and local agencies, as shown in Table 6. 

Table 6. Interagency and Subrecipient Linkage Agreements 

State Agencies with Formal Interagency 

Agreements 

Local Agencies and Providers with Subrecipient 

Linkage Agreements

Health or Public Health

Mental Health

Children and Families

Criminal Justice

Medicaid

Education

Adult and Child Protective Services

Social Services

Communicable Diseases

Rehabilitation

Disabilities

Tribal Health

Public Safety

Governor’s Office

Child Protective Services

Corrections/Drug Courts

Local Public Health Departments and Hospitals

Temporary Assistance to Needy Families (TANF) 

Offices

Domestic Violence Agencies

Aging Services

Mental Health Providers

Social Services

Several SSAs met periodically with subrecipients and agencies participating in statewide referral 

networks to review procedures to ensure thorough assessments, appropriate treatment planning, services 

related to treatment goals, and accurate data collection and documentation of services provided. 

Outcome 4. Expanded Services and Programs Available through Joint Funding 

Initiatives 

In addition to improving the coordination of treatment and prevention services, SSAs collaborated with 

other State agencies to expand access to substance abuse services through joint funding of programs and 

initiatives. Using SAPT BG funds, SSAs co-sponsored a variety of services and programs, including 

treatment programs for women, HIV services, and prevention services. Specific examples of successful 

collaboration include the following: 

 Cosponsoring additional case management services for pregnant women with the State’s Healthy 

Start program 

 Developing a family drug court to address family and parenting issues among women who have 

substance use problems and their dependent children 

 Collaborating with the State Medicaid office to change policies to increase the use of Medicaid 

funds for women’s services

 Coordinating with the Department of Health to publish an RFP and subsequently select HIV 

service providers 

 Funding regional grants with the Department of Education to support school-based education 

programs and peer leadership trainings targeted to middle and high school-aged youth 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

36

Reducing Substance Abuse 

on College Campuses

One SSA co-sponsored a 

summit with the State Attorney 

General’s office, the 

Department of Public Health, 

and a consortium of college 

presidents and administrators 

to discuss and plan strategies 

to decrease underage alcohol 

use and abuse on college 

campuses.

Alleviating Prison Overcrowding

Members of one SSA participated on 

a legislatively-created workgroup on 

prison overcrowding. Based on this 

work, legislation was enacted to 

alleviate overcrowding and provide 

early release to treatment facilities for 

offenders with mental health or 

substance use disorders. These 

efforts led to future cooperation 

between the SSA, the Department of 

Corrections, and the Judicial Branch.

 Awarding competitive subcontracts for the provision of parent education with funding from the Safe 

and Drug Free Schools and Communities Governor’s Grant.

Outcome 5. Increased Ability to Address Critical Public Health or Safety Issues 

Statewide 

SSA collaborations with other State agencies and local 

stakeholders enabled them to address specific public health or 

public safety related concerns. A number of SSAs reported 

participation on task forces, interagency workgroups, and 

legislative committees designed to plan and implement strategic 

activities to mitigate public health problems. Specific examples of

these types of collaboration include:

Participation on the Governor’s task force for preventing 

substance abuse-related criminal activity

Completion of a 5-year Governor’s prevention initiative for 

youth designed to reduce substance use among 12 to 17 yearolds and improve coordination among organizations and agencies that work with them

Participation in and leadership of interagency workgroups developed to address:

Fetal alcohol spectrum disorders

Treatment needs for children and adolescents

Alcohol misuse and abuse on college campuses

Co-occurring mental and substance use disorders

Inhalant abuse

Prescription drug abuse

Housing and homelessness

Tribal health

Substance-related crime

Participation in legislative committees and workgroups 

designed to educate policymakers and advise on 

substance abuse related legislation and policy, 

including:

Drug demand reduction

Mandated use of EBPs for substance abuse 

treatment

Alternatives to incarceration

Underage drinking
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

37

Driving under the influence 

Tobacco access. 

Key Finding 6: Although baseline data support the need for prevention 

services and activities, the use of national survey State estimates data alone 

to assess the NOMs limits CSAP’s ability to attribute changes in the NOMs 

to SAPT BG-funded prevention services and activities. 

As part of the NOMs initiative to increase accountability and improve the ability to demonstrate 

program outcomes, CSAP collaborated with national, State, and local prevention organizations to 

develop outcome measures. These measures examine attitudes toward substance use, abstinence, school 

or employment participation, criminal justice involvement, and social connectedness. To reduce the data 

collection burden for State and local prevention agencies (which receive only 20 percent of SAPT BG 

funds), CSAP uses data from the NSDUH to fulfill NOMs data requirements. There are two significant 

difficulties inherent in this strategy: 

Conclusions about NOMs changes as a result of BG-funded prevention services and activities 

cannot be made based primarily on the results of national survey State estimates that do not 

identify individuals or groups who may have been affected by BG-funded activities. Additional 

data are needed to link changes in NOMs measures to interaction with BG-funded prevention 

services and activities. 

The NSDUH is limited by small sample sizes in many States, which leads to under coverage of 

some populations, including individuals and groups who have been affected by prevention 

services and activities. 

States may request to substitute data instead of utilizing the NSDUH survey State estimates. States also 

provide their own specific data on EBP implementation and the number of persons served. CSAP has 

convened an expert panel that has provided recommendations on revising the NOMs data collection and 

analysis strategy in order to demonstrate the effectiveness of SAPT BG-funded prevention activities. In 

FY 2007, CSAP began requiring the collection and reporting of prevention NOMs. Results of the 

baseline analysis for the following outcome measures are presented in this section: age of first substance 

use, perception of risk or harm from substance use, disapproval or perceived disapproval of substance 

use, and attitude toward random drug and alcohol testing. Analysis of data from future years will enable 

the assessment of general population change related to these measures; however, without a data source 

more proximal to prevention interventions, it will not be possible to determine the extent to which 

changes can be attributed to BG-funded prevention services and activities. 

Measure 1. Age of First Substance Use 

This measure assesses age of first substance use by youth aged 12 to 17 years. Table 7 shows that young 

ages of first use were reported for all substances, supporting the need for prevention programs to delay 

first use of substances. 

Table 7. Average Age of First Use of Substances among Youth Aged 12 to 

17 Years, FY 2008 

Substance Age of First Use 

Cigarettes 12.4
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

38

Measure 2. Perception of Risk/Harm from Substance Use 

This prevention measure assesses the perception of risk or harm from substance use among 12 to 17 

year-olds. As shown in Table 8, more than three-quarters of survey respondents reported perceiving risk 

or harm from the use of cigarettes, marijuana, and alcohol. The percent reporting moderate to great risk 

was significantly higher for cigarettes (93 percent) than for alcohol (83 percent) and marijuana (76 

percent). However, the perception of risk or harm from substance use may not lead to delayed first use 

of substances (shown in Table 7). These results indicate the need for prevention strategies that go 

beyond education to delay first use of substances. 

Table 8. Perception of Risk/Harm from Substance Use 

among Youth Aged 12 to 17, FY 2008 

Measure 3. Disapproval or Perceived Disapproval of Substance Use 

This measure assesses youth disapproval or perception of others’ disapproval of youth substance use. As 

Table 9 shows, more than 80 percent of youth aged 12 to 17 years reported disapproval or perceived 

disapproval associated with youth use of cigarettes, marijuana, and alcohol. However, as with the results 

for perception of risk or harm, an increase in disapproval or perceived disapproval may not be leading to 

an increase in the average age of first substance use. Prevention approaches are needed that will delay 

age of first use and in turn lead to lower rates of substance misuse and abuse and related deleterious 

consequences. 

Alcohol 13.0

Tobacco products other than cigarettes 13.3

Marijuana or hashish 13.6

Other illegal drugs 12.7

Perception of Risk/Harm from Substance Use Percent Reporting 

Moderate or great risk from cigarettes 93.1

Moderate or great risk from marijuana 82.7

Moderate or great risk from alcohol 76.3
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

39

Table 9. Perception of Substance Use among Youth 

Aged 12 to 17 Years, FY 2008 

Measure 4. Attitude toward Random Drug and Alcohol Testing 

This measure assesses youth perception of workplace policies related to alcohol and other drug use. At 

baseline, 22 percent of 15 to 17 year-olds reported that they would be more likely to work for an 

employer who randomly conducted drug and alcohol tests. Questions have been raised regarding the 

validity of this measure for addressing the school/employment participation domain because many 15 to 

17 year-olds lack a frame of reference for judging employers and workplace policies. Measures of 

school attendance, as well as suspensions and expulsions related to substance use, would be more 

relevant measures for this age group. 

Program Challenges and Recommendations for 

Improvement 

The complexity of the SAPT BG Program presents many challenges to its implementation and ability to 

demonstrate outcomes. Interviews with Federal and State staff pointed to many issues that present 

obstacles to achieving Program goals. Program challenges and recommendations for improvement are 

separated into the following overarching areas: 

Communication and management processes 

Data collection strategies and processes 

Improvement of State data infrastructures. 

Specific challenges and recommendations for improvement in each of these areas are presented in Table 

10. Recommendations are based on the results of this independent evaluation and the incorporation of 

Federal and State staff reflections and insights. 

Perception by Substance Percent Reporting

Disapproval of cigarette use 88.2

Perception of disapproval of cigarette use 85.4

Disapproval of alcohol use 85.1

Disapproval of using marijuana regularly 81.6

Disapproval of using marijuana experimentally 80.7
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

40

Table 10. SAPT BG Program Challenges and Recommendations for Improvement 

Communication and Management Processes

Challenges Recommendations for Improvement

Need for improved communication and a consistent 

message from Federal to State staff about some 

Program goals and requirements. States commented 

about the lack of unified, consistent messages from 

CSAT and CSAP in some specific areas, including:

NOMs data definitions and standard data collection 

processes

The reporting of financial information and MOE 

compliance 

Expectations for BG applications 

Acceptable fulfillment of the 17 legislative goals.

Clarify Program data definitions and requirements, 

including ―what counts‖ for achievement of the 17 

goals, MOE and other financial calculations, and the 

NOMs data elements.

Develop unified Federal guidance about Program 

requirements and expectations and a data dictionary 

with uniform and realistic definitions. 

Offer more comprehensive training for States, such 

as online tutorials and regional meetings.

Continue to seek State input and develop better 

definitions for required outcome data elements.

Provide opportunities for internal communication 

within CSAT and CSAP, training, and mentoring to 

ensure that consistent guidance is provided to States.

Strengthen ongoing communication between State 

Project Officers and their assigned states via devoted 

resources for knowledge management.

Need for clarification about the roles and 

responsibilities of Federal, State, and contractor staff 

related to CSAT Core Technical Reviews and 

Technical Assistance (TA).

Although CSAT State Project Officers (SPOs) work 

with States to refine their BG applications and address 

the 17 goals, it appears that CSAT SPOs are not 

involved in substantive aspects of State Core 

Technical Reviews. To ensure an objective assessment 

of compliance, CSAT SPOs typically attend the Core 

Technical Review but remain in the background, 

which is confusing for States. 

In addition, States perceive that decision-making 

authority for CSAT TA requests lies in the hands of 

the TA contractor, the TA contract Government 

Project Officer (GPO), and the Branch Chief, rather 

than with the CSAT SPOs, who typically know the 

most about individual State issues and TA needs.

Explain the role of the CSAT SPO in the Technical 

Review process (presite and onsite) to avoid 

confusion on the part of State and Federal staff.

Explain the TA review and approval process to State 

and Federal staff and regularly reiterate 

communication about the TA process to ensure that 

State staff have the needed information regardless of 

State staff turnover.

Include CSAT SPO State-specific expertise in the 

decision-making process.

Provide thorough explanations to States when TA is 

reduced or disapproved.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

41

Communication and Management Processes

Challenges Recommendations for Improvement

High level of burden on States to provide information 

for the Program. 

Each year, States are required to produce an SAPT 

BG application that describes the activities conducted 

to achieve the 17 legislative goals. The length of the 

report reflects the collection of information for 

reports to Congress. Federal and State staff report 

that while existing questions or areas in the 

application are rarely deleted, new questions or areas 

are frequently added. This leads to an application that 

is duplicative and cumbersome. 

The majority of States spend 6 to 9 months each year 

gathering information for and developing the BG 

application, using staff resources that States argue 

could be better spent on TA for providers and other 

BG subrecipients. 

States also are asked to provide a large amount of 

background information in preparation for the TRs 

and SPSSRs – information that States say can be 

obtained from their applications and online 

resources.

Consider implementing a multiple-year application 

cycle (every 3 to 5 years) that would require States 

to submit a multiple-year plan and provide annual 

progress reports based on plan objectives.

Revisit the primary purpose of the BG application 

and eliminate questions or areas that do not address 

it.

Encourage TR and SPSSR reviewers to obtain 

information from WebBGAS or online resources to 

reduce State administrative burden.

Limited utility of Program monitoring reports and 

recommendations for some States. 

The time lag associated with the finalization of TR 

and SPSSR monitoring reports (6 to 12 months) 

makes information obsolete by the time it reaches the 

States. Delays are most pronounced when the 

TR/SPSSR contractor, SPO, and States disagree 

about review findings, which must be resolved before 

a report can be finalized. The time lag forces many 

States to move forward with system enhancements 

and decisions that cannot wait for several months 

while reports are being vetted. 

Some States claim that TR and SPSSR report 

recommendations are too general because reviewers 

do not always consider the unique combination of 

political, social, and economic forces that affect the 

State’s prevention and treatment system. 

Some States report that TA recommendations in the 

TR reports seem motivated by contractor interests 

and skills rather than what is in the best interest of 

the State. 

Consider expediting the report review process by 

instituting a process through which States are able 

to contest review findings without delaying the 

finalization of the review report. In this way, 

disagreement about one area of the report will not 

delay the development and implementation of TA 

activities to address uncontested areas. 

Set and enforce deadlines for submission of report 

comments and report revisions so that reports are 

not delayed by any one individual. 

Select and train reviewers to ensure that they

possess a comprehensive understanding of the State 

systems that they may be assigned to review.

Further involve CSAT SPOs in reviewing TA 

recommendations to ensure that they address State 

concerns and are not motivated by contractor 

interests.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

42

Communication and Management Processes

Challenges Recommendations for Improvement

Unmet TA needs. 

Many States do not know how to make formal 

requests for Federal TA and training and are not 

aware of the potential areas in which TA is available. 

Several States believe that only TA requests related 

to NOMs data collection and reporting will be 

approved, so they do not submit TA requests in other 

areas. 

There is a disconnect between Federal TA providers 

and CSAT SPOs; several States reported receiving 

approval for a TA request from their CSAT SPO 

only to have it denied by the CSAT TA contract GPO 

or Branch Chief. 

TRs and SPSSRs produce useful TA 

recommendations, but the TA is either not approved 

or not conducted to the satisfaction of the State. 

Clarify the TA and training request process and 

regularly inform States about the process. With the 

high turnover rate for State staff, frequent 

communication is needed to ensure that all States 

understand the availability and process for 

requesting TA. 

Increase efforts to market TA and training to the 

States so that they understand what is available.

Expand the scope of Federal TA provided to States. 

States reported that they would benefit from 

Federal TA designed to: identify and meet the 

needs of diverse populations, address EBP 

implementation challenges, and assist with State 

infrastructure enhancements.

Clarify the role of the CSAT SPO in TA provision 

and encourage direct communication between 

States and CSAT TA decisionmakers in order to 

improve responsiveness to States. Promote a team 

approach.

To improve TA access and satisfaction, provide 

additional resources for the following TA and 

training formats:

Guidelines and support for train-the-trainer 

models, regular training cycles, and a helpdesk

Distance learning, Web-based trainings, and 

online tutorials

Peer-to-peer TA at workshops, conferences, 

and regional meetings

Wider dissemination of ―off-the-shelf‖ TA 

tools and materials.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

43

Data Collection Strategies and Processes 

Challenges Recommendations for Improvement

Limited ability to demonstrate the effect of the 

Program in some areas. 

NSDUH data used for prevention NOMs do not 

account for interaction with BG-funded 

prevention services and activities. Thus, Federal 

and State staff cannot claim that changes in the 

NOMs are due in part to BG-funded prevention 

services and activities. 

Use of TEDS for the treatment NOMs presents

the following challenges: Not all States 

participate in the TEDS initiative and questions 

remain about the consistency of data collection

and data quality procedures across States. 

The Program needs some system-level outcome 

measures in order to demonstrate infrastructure 

development, collaboration with other State 

agencies and organizations, and effectiveness of 

TA and other program support activities.

Develop prevention outcome measures that assess 

attitudes and behaviors pre- and post-interaction with 

BG-funded services and activities when the prevention 

strategy supports this evaluation design.

Compare NSDUH results on the NOMs for 

respondents who did and did not report experience with 

prevention services and messages to determine whether 

exposure to prevention services or messages can be 

associated with more desirable NOMs results.

Strongly encourage States to participate in TEDS and 

use TEDS data definitions to improve the reliability of 

NOMs data. 

Conduct monitoring and TA activities to ensure that 

interpretation of TEDS questions is uniform across 

States and providers. Develop materials, host Webbased and in-person trainings, and provide onsite TA to 

States and subrecipients to ensure that data definitions 

are being interpreted correctly and consistently.

Continue close collaboration with State substance 

abuse and other appropriate State agencies (e.g., data 

and statistics, corrections) to develop more valid and 

effective outcome measures 

Because one of the key purposes of the SAPT BG 

Program is to improve State systems, consider the 

development of a few system-level indicators to 

demonstrate Program effects on State systems 

development and enhancement.

Improvement of State Data Infrastructures

Challenges Recommendations for Improvement

Need for additional resources to further improve 

State data infrastructures. 

Many States are still struggling to develop data 

collection, information, and monitoring systems 

that will enable them to track outcomes 

effectively.

States have received modest funding to create 

new or overhaul existing data collection and 

reporting systems; however, in the words of one 

State Director, ―These funds are a drop in the 

bucket compared to what is needed for 

infrastructure development.‖ 

States cited a need for improved networking 

opportunities among States so that States can 

learn from each other and not have to ―reinvent 

the wheel.‖

Invest additional resources to help States complete the 

development and maintenance of sophisticated data 

collection systems that will increase their ability to 

demonstrate SAPT BG Program outcomes and make 

data-driven decisions at the State and Federal levels.

Create more opportunities for State-to-State TA 

regarding the development and maintenance of State 

data collection and reporting systems. States 

experiencing difficulties developing functional data 

systems can learn from States that have already 

developed systems that fulfill their data collection, 

reporting, and dissemination needs. Additional 

opportunities for State-to-State TA, regional trainings, 

and networking at conferences will help more States 

develop and maintain effective data systems.
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

44

“Because of the women’s treatment 

requirement, the treatment field has 

made so much progress in 

successfully treating women. We’ve 

had excellent leadership from CSAT 

in terms of [funding] model programs 

and [identifying] best practices, and 

at the State level, we’ve trained many 

practitioners in women’s treatment 

approaches. I don’t think we would 

have come so far so fast without the 

Block Grant requirement.”

– Women’s Treatment Coordinator

Conclusion

The SAPT BG Program has had a positive effect on the health and well-being of individuals and 

communities it serves, demonstrating real improvements in participant- and client-level outcomes. 

Under the effective management of CSAP and CSAT, the Program has achieved its purpose of 

providing flexible funds to States to improve and enhance their substance abuse prevention and 

treatment systems, and it has contributed significantly to State progress in infrastructure and capacity 

development, implementation of EBPs, leveraging of BG resources, fostering of collaborations, and 

performance measurement and improvement. Without the SAPT BG Program, many communities 

would lack the resources to sustain evidence-based treatment programs or implement any prevention 

services, and the ability of our Nation to address the issue and negative consequences of substance abuse 

would be severely compromised. 

The Program has led States to develop specific outcome measures and comprehensive data collection 

and reporting systems to assess access to services, service quality, and client status. Absent the 

leadership of the SAPT BG Program, the collection and reporting of outcomes data related to States’ 

substance abuse treatment systems would not be occurring at its current level. Program requirements 

also have spurred the development and maintenance of State data systems that enable State and

subrecipient partners to identify needs, measure outcomes, and make data-driven decisions.

Authorizing legislation for the SAPT BG Program 

prescribed 17 different requirements related to State 

spending of BG funds on prevention and treatment 

services. These requirements became known as the ―17 

goals,‖ which represent the SAPT BG Program’s main 

activities and its expectations of States9. Accomplishing 

these goals reflects the Program’s ability not only to carry 

out its legislative mandate, but to touch upon and improve 

the lives of people with substance abuse problems and the 

communities in which they live. This evaluation found that 

States are overwhelmingly meeting these legislative 

requirements; in fact, for 15 of the 17 goals, they have met 

or exceeded expectations. This achievement reflects the 

remarkable efforts of individuals and organizations 

nationwide to prevent, treat, and reduce the problems of substance use and abuse.

Federal, State, and subrecipient staff described a variety of activities pursued in their work to 

accomplish the 17 legislative goals. These programs reflect the resources and imagination of State 

program planners, who have developed programs that range from school-based prevention education to 

housing services for pregnant women and their children. 

Table 11 reflects the level at which States achieved the 17 goals. To be categorized as having exceeded 

expectations for accomplishing one of the 17 goals, Federal, State, and subrecipient staff were required 

to meet all of the following criteria:

A high level of Federal and State leadership and assistance was provided 

States and subrecipients planned and conducted activities that met the specific language of the 

legislative requirement
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report 

45

State staff reported significant improvements in State system capacities and capabilities due to 

goal achievement. 

For two of the goals, additional information is needed from States to determine the degree of goal 

achievement. For Goal 10, States were asked to describe improvements made to their referral processes 

and procedures. Most States described their existing referral processes but did not elaborate in BGAS 

about improvements. For Goal 15, States were asked to describe how their independent peer review 

process contributed to the improvement of the quality and appropriateness of treatment services 

provided. Instead, most States described the independent peer review process without providing much 

detail about either peer review findings or how these findings were being used to improve the quality of 

services provided. Further information that directly addresses the goal statements will be helpful in 

determining the degree to which Goals 10 and 15 are being achieved in States. 

The fact that so many States are meeting or exceeding the requirements for the 17 goals reflects the 

degree to which Federal and State leaders have worked together with subrecipients to develop and 

promote programs that meet the multiple needs of diverse individuals and communities. 

Table 11. Achievement of the 17 Legislative Program Goals 

Goal Short Title

Degree of Goal Achievement

Exceeded Met Additional Info 

Needed

Goal 1: Continuum of substance abuse treatment services X

Goal 2: Spending on primary prevention programs X

Goal 3: Spending on services for pregnant women and women 

with dependent children X

Goal 4: Treatment for intravenous drug abusers X

Goal 5: Tuberculosis services for people in substance abuse 

treatment X

Goal 6: Early intervention services for HIV for people in 

substance abuse treatment X

Goal 7: Group homes for recovering substance abusers X

Goal 8: State efforts to reduce the availability of tobacco 

products X

Goal 9: Preferential admission of pregnant women to substance 

abuse treatment X

Goal 10: Improved process for referring individuals to substance 

abuse treatment X

Goal 11: Continuing education X

Goal 12: Coordination of services X

Goal 13: Needs assessment by State and locality X

Goal 14: Ensuring that needles and syringes are not provided for 

illegal drug use X

Goal 15: Improving the quality and appropriateness of treatment 

services through independent peer review X

Goal 16: Protecting patient records from inappropriate disclosure X

Goal 17: Adherence to Charitable Choice Provisions and 

Regulations X

A program as complex as the SAPT BG Program encounters a variety of challenges to its successful 

implementation and management. This evaluation identified challenges that include: need for improved 

communication from Federal to State staff about Program goals and requirements; administrative burden 
Independent Evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

Final Evaluation Report

46

on States; need for improved timeliness and utility of Program monitoring reports and 

recommendations; unmet TA and training needs; limited ability to demonstrate system-level outcomes; 

and a lack of resources to further improve State data infrastructures.

By addressing these challenges, Federal and State staff will be able to improve the effectiveness and 

efficiency of Program communication and management; demonstrate a wider array of participant- and 

system-level Program outcomes; make data-driven decisions related to services and systems; and 

increase the implementation and dissemination of EBPs and other successful approaches for substance 

abuse prevention and treatment. These improvements will lead to increased quality and effectiveness of 

Program services provided to individuals and communities and will enable SAMHSA to make further 

progress toward its vision of ―a life in the community for everyone‖ affected by substance abuse. 

 

1 Substance Abuse and Mental Health Services Administration. (2008). Results from the 2007 National Survey on Drug Use 

and Health: National Findings. http://www.oas.samhsa.gov/nsduh/2k7nsduh/2k7Results.pdf

2 U.S. Department of Health and Human Services. (2006). Healthy People 2010: Midcourse Review. 

http://www.healthypeople.gov/Data/midcourse/pdf/FA26.pdf

3

For simplicity, we will use the term ―State‖ within this document to refer to States, Territories, the District of Columbia, 

Pacific Jurisdictions, and the American Indian Tribe.

4 Omnibus Budget Reconciliation Act of 1981 (Public Law 97-35)

5

States whose rate of AIDS cases is 10 or more per 100,000 individuals as confirmed by the Centers for Disease Control and 

Prevention. For legislation, refer to Title XIX, Part B, Subparts II, Section 1924, 300(b)(4)(A)(i), 300(b)(4)(A)(ii), and 

300(b)(4)(B).

6

http://www.samhsa.gov/about/. Accessed December 1, 2008.

7

http://www.nationaloutcomemeasures.samhsa.gov. Accessed December 1, 2008.

8Gottheil E, McLellan AT, Druley KA. (1992.) Length of Stay, Patient Severity and Treatment Outcome: Sample Data from 

the Field of Alcoholism. Journal of Studies on Alcohol, 53: 69-75.

Greenfield L, Burgdorf K, Chen X, Porowski A, Roberts T, Herrell J. (2004). Effectiveness of Long-term Residential 

Substance Abuse Treatment for Women: Findings from Three National Studies. The American Journal of Drug and Alcohol 

Abuse, 30(3): 537-550.

McCusker J, Bigelow C, Vickers-Lahti M, Spotts D, Garfield F, Frost R. (1997). Planned Duration of Residential Drug 

Abuse Treatment: Efficacy versus Effectiveness. Addiction, 92(11), 1467-1478.

9

42 U.S.C. 300x-21–300x-31 and 300x-53; 45 C.F.R. 96.122–96.136; 42 C.F.R. part 2; and 42 C.F.R. part 54
Appendix A 

Legislative Goals of the SAPT BG Program
A- 2 

The 17 Legislative Goals of the SAPT BG Program 

Goal Short Title Goal Description

Goal 1: Continuum of substance abuse 

treatment services 

Maintain a continuum of substance abuse treatment services that meets 

the needs for services identified by the State 

Goal 2: Spending on primary prevention 

programs 

Spend not less than 20 percent on primary prevention programs for 

individuals who do not require treatment for substance abuse and specify 

activities proposed for each of the six prevention strategies 

Goal 3: Spending on services for pregnant Est

women and women with dependent 

children 

ablish new programs or expand the capacity of existing programs to 

make available treatment services designed for pregnant women and 

women with dependent children 

Goal 4: Treatment for intravenous drug 

abusers 

Provide treatment to intravenous drug abusers and fulfill requirements of 

90 percent capacity reporting, 14- to 120-day service provision, interim 

services, outreach activities, and monitoring 

Goal 5: Tuberculosis services for people in 

substance abuse treatment 

Make tuberculosis services available routinely to each individual 

receiving treatment for substance abuse and monitor such service 

delivery 

Goal 6: Early intervention services for 

HIV for people in substance abuse 

treatment 

For designated States, make available within existing programs early 

intervention services for HIV in areas of the State that have the greatest 

need for such services and monitor such service delivery 

Goal 7: Group homes for recovering 

substance abusers 

Provide for and encourage the development of group homes for 

individuals recovering from substance abuse through the operation of a 

revolving loan fund (this goal is voluntary) 

Goal 8: State efforts to reduce the 

availability of tobacco products 

Reduce the extent to which tobacco products are available to individuals 

under age 18 through enforcement of the Synar amendment 

Goal 9: Preferential admission of 

pregnant women to substance abuse 

treatment 

Ensure that each pregnant woman be given preference in admission to 

treatment facilities and, when a facility has insufficient capacity, ensure 

that the pregnant woman be referred to the State, which will refer the 

woman to a facility that does have the capacity to admit the woman or, if 

no such facility has the capacity to admit the woman, will make available 

interim services within 48 hours 

Goal 10: Improved process for referring 

individuals to substance abuse treatment 

Improve the process in the State for referring individuals to the treatment 

modality that is most appropriate for the individual 

Goal 11: Continuing education Provide continuing education for the employees of facilities that provide 

prevention activities or treatment services 

Goal 12: Coordination of services Coordinate prevention activities and treatment services with the provision 

of other appropriate services 

Goal 13: Needs assessment by State and 

locality 

Submit an assessment of the need for both treatment and prevention in 

the State for authorized activities, both by locality and by the State in 

general 

Goal 14: Ensuring that needles and 

syringes are not provided for illegal drug 

use 

Ensure that no program funded through the block grant will use funds to 

provide individuals with hypodermic needles or syringes so that such 

individuals may use illegal drugs 

Goal 15: Improving the quality and 

appropriateness of treatment services 

through independent peer review 

Assess and improve, through independent peer review, the quality and 

appropriateness of treatment services delivered by providers that receive 

funds from the block grant. 
A- 3 

The 17 Legislative Goals of the SAPT BG Program 

Goal Short Title Goal Description 

Goal 16: Protecting patient records from 

inappropriate disclosure 

Ensure that the State has in effect a system to protect patient records from 

inappropriate disclosure (Compliance with the Health Insurance 

Portability and Accountability Act) 

Goal 17: Adherence to Charitable Choice 

Provisions and Regulations 

For States that distribute SAPT BG funds to faith-based or charitable 

organizations, ensure that the State has in effect a system to comply with 

42 C.F.R. part 54 

(Sources: 42 U.S.C. 300x-21 to 35 and 300x-51 to 66 and Uniform Application, FY 2008, Substance Abuse 

Prevention and Treatment Block Grant, Substance Abuse and Mental Health Services Administration.)
Appendix B 

Evaluation Advisory Workgroup Membership 
B-2 

The Independent Evaluation of the SAPT BG Program benefitted from the knowledge and 

experience of a variety of State substance abuse agency representatives, treatment and prevention 

specialists, and evaluation experts. SAMHSA would like to acknowledge and thank the 

following Evaluation Advisory Workgroup members who provided their time and expertise for 

the evaluation: 

Teresa E. Anderson, Ph.D. 

Associate Director for Outcomes 

 Measurement 

Commonwealth Medicine 

Center for Health Policy and Research 

Shrewsbury, MA 

Theodora Binion-Taylor 

Director 

Division of Alcohol and Substance Abuse 

Illinois Department of Human Services 

Chicago, IL 

Maria Canfield 

Former Chief 

Nevada Bureau of Alcohol and Drug Abuse 

Carson City, NV 

Barbara Cimaglio 

Deputy Commissioner 

Alcohol and Drug Abuse Programs 

Vermont Department of Health 

Burlington, VT 

Patrick J. Fleming, MPA, LSAC 

Division of Substance Abuse Services 

Salt Lake County Government Center 

Salt Lake City, UT 

Robert Johnson 

Former Senior Deputy Director 

Substance Abuse Services 

Addiction Prevention and Recovery 

 Administration 

Washington, DC 

Michael Magnusson 

Former Chief 

Division of Prevention Services 

Ohio Department of Alcohol and Drug 

 Addiction Services 

Columbus, OH

Howard Shapiro, Ph.D. 

Former Executive Director 

State Associations of Addiction Services 

Washington, DC 

Gilbert Sudbeck, M.S.W. 

Director 

Division of Alcohol and Drug Abuse 

South Dakota Department of Human 

 Services 

Pierre, SD 

Debbie Synhorst 

National Prevention Network President 

Iowa Department of Public Health 

Division of Behavioral Health 

Des Moines, IA 

Don Wright 

Former Unit Manager 

Substance Abuse Services 

North Dakota Department of Human 

 Services 

Bismarck, ND
Appendix C 

Program Logic Model and Evaluation Framework 
LOGIC MODEL FOR THE SUBSTANCE ABUSE PREVENTION AND TREATMENT BLOCK GRANT PROGRAM EVALUATION

Federal Activities 

 Funding distribution 

 Development of 

application template and 

guidance for States 

 Application review and 

approval 

 Use of application 

information 

 Program monitoring 

 Technical reviews 

 Prevention SA 

 Synar reviews 

 Program development and 

support 

 TA and Training 

 Data collection, analysis, 

and dissemination 

Federal Outputs 

 Funds (allocations to 

States) 

 Application instructions 

and guidance 

 Application approval 

 BGAS Reports 

 TR reports 

 SPSR reports 

 Synar reports 

 Other data reports 

 Provision of TA to States 

 National conferences/ 

meetings 

 Science-based publications 

Federal Outcomes 

Short-term 

 Improved Federal/State 

information exchange 

 State compliance with legislation 

requirements 

 Improved ability to describe State 

BG program outcomes 

Long-term 

 Improved capability to respond to 

Congressional and public 

information requests 

 Improved administration and 

management of BG program 

 Improved ability to provide 

leadership to States for improving 

substance abuse prevention and 

treatment systems 

Impacts 

Federal Impacts 

 Improved system of care for 

substance abuse treatment 

nationwide 

 Improved system for 

prevention of substance abuse 

nationwide 

 Increased Congressional 

support for substance abuse 

prevention and treatment 

services 

State Impacts 

 Improved system of care for 

substance abuse treatment 

Statewide 

 Improved system of substance 

abuse prevention Statewide 

Societal Impacts 

 Improved access to recovery 

for individuals in need 

 Increased number of 

individuals in recovery from 

substance addiction 

 Decreased initiation of 

substance use 

 Decreased number of 

individuals developing 

substance addiction 

Federal Resources 

 Funding 

(Congressional 

appropriation) 

 Federal staff 

 Federal contractors 

State Resources 

 Funding (State 

appropriation) 

 State staff 

 Local behavioral 

health authorities 

and other 

intermediaries 

 Service providers 

State Outcomes 

Short-term 

 Improved documentation of State 

BG activities 

Long-term 

 Increased coordination of 

substance abuse prevention 

services and programs 

 Increased coordination of 

substance abuse treatment services 

and programs 

 Increased number of EBPs, 

innovative services available 

 Improved quality of services 

 Decreased unmet treatment need 

 Increased frequency of programs 

initiated with BG funds continuing 

by using State and other funding 

sources (leveraging) 

 Leveraging BG resources to 

develop policy changes 

State Activities 

 Application development 

and submission 

 Needs assessment 

 State funding allocation 

and distribution 

 Programs/services that 

address 17 goals, 

including set-aside 

requirements 

 State-level administrative 

needs/initiatives 

 Program development 

 TA and training 

 Evaluation of programs 

and services funded 

through the BG 

 Monitoring 

State Outputs 

 State BG application 

 Needs assessment report 

 Funds (allocations to local 

orgs/agencies) 

 Programs/services 

delivered 

 EBPs and innovative 

programs 

 Number of providers 

 Number of people served 

 Targeted populations 

served (17 goals) 

 Availability of data 

 Provision of TA to 

local orgs 

 Number of orgs trained 

 Number of training 

events 
Appendix C

Evaluation Framework 
Appendix C: Evaluation Framework 1

Independent Evaluation of the Substance Abuse Prevention and 

Treatment Block Grant Program 

Evaluation Framework 

SAPT BG Program Implementation 

Federal Implementation 

Federal Funding Distribution 

What is the process by which the Federal government allocates BG funds to States? 

What Federal activities are supported by the administrative set aside? 

Development of Application Template and Guidance for States 

What is the process for the development of the application template and guidance for States? 

Who is involved? 

Is BGAS used in this process? If so, how? 

If there are Federal staff members who are not involved, how are changes to the template 

and guidance communicated to them? 

Is there a process for Federal staff to obtain feedback about the application template and 

guidance? If so, what is it? 

What is involved in order to change the application template and guidance? (e.g., the 

process for obtaining OMB approval) 

How are any revisions to the application template and guidance communicated to States? 

What is the timeframe for the development and distribution of the application template 

and guidance for each year? (e.g., Are there challenges related to keeping the timeframe?) 

What is the intended purpose of the Application Template and Guidance (e.g., to 

guide the States’ planning processes? To establish federal expectations of the States’ 

performance?) 

What are the strengths of the process for the development of the application template and 

guidance for States? 

What are the challenges of the process for the development of the application template and 

guidance for States? 

What are supports to the process for the development of the application template and 

guidance for States? 

What are the barriers to the process for the development of the application template and 

guidance for States? 

What are recommended changes to the process for the development of the application 

template and guidance for States? 
Appendix C: Evaluation Framework 2

Application Review and Approval 

What is the process by which applications are reviewed and approved? 

What role do State Project Officers play in the review of applications? 

Is BGAS used in this process? If so, how? 

How is approval status communicated to other Federal staff and to the States? 

What are the strengths of the process by which applications are reviewed and approved? 

What are the challenges of the process by which applications are reviewed and approved? 

What are recommended changes to the process for the development of the application 

template and guidance for States? 

Use of Application Information 

How do Federal staff use States’ application information? 

How is BGAS used in this process? 

What reports are generated from BGAS on a regular basis? 

What reports are generated from BGAS on a one-time basis? 

Other than the SPOs, who requests information that can be obtained from States’ applications? 

Program Monitoring 

What are the goals of program monitoring for the SAPT BG program? 

How do CSAT and CSAP monitor State compliance to the BG requirements (legislation and 

CSAT/CSAP policies)? 

Are States’ intended use plans compared to States’ annual reports and progress reports to 

assess the degree to which planned activities were implemented? If so, how? 

How are potential issues with State compliance identified? 

Who decides what potential issues require Federal or State action? 

How are issues that require action communicated to the States? 

How often do SPOs visit States to monitor compliance to the BG program? What 

activities do they conduct onsite? 

What are the strengths of the process by which CSAT and CSAP monitor State compliance 

with the BG program? 

What are the weaknesses of the process by which CSAT and CSAP monitor State compliance 

with the BG program? 

What are recommendations for improving the process by which CSAT and CSAP monitor 

compliance with the BG program? 

How useful is the monitoring process to the State? How could it be more useful? 

What are unintended positive or negative results of the monitoring process? 

Technical Reviews 

What role do the technical reviews play in the verification of State compliance with the BG 

program? 

Are there policies governing the technical review process? If so, what are they? 

Who conducts the technical reviews? 

How are the reviewers identified? 
Appendix C: Evaluation Framework 3

Do the reviewers receive training about how to conduct the technical reviews? If so, what 

training do they receive? 

What is the timeframe for this training (e.g., how long before the actual technical reviews 

does the training occur)? 

What guidance does Federal staff provide to States concerning the technical review process? 

What materials are provided? 

What instructions are provided? 

What products result from the technical reviews? (e.g., technical review report and 

recommendations) 

What is the timeliness of the submission of technical review products? 

How do Federal staff (program staff and grants management) use technical review products? 

Do States receive the technical review reports? If so, what do States do with the reports? 

How do States use the technical review process to improve their BG program 

implementation? (e.g., request TA, obtain policy guidance) 

What are the strengths of the technical review? 

What are the weaknesses of the technical review? 

What are recommendations for improving the technical review? 

To what extent does the technical review process improve Federal/State information exchange? 

What are any unintended positive or negative results of the technical review? 

State Prevention and Synar System Reviews (SPSSRs) 

What role do the SPSSRs play in the verification of State compliance with the BG program? 

Are there policies governing the SPSSR process? If so, what are they? 

Who conducts the SPSSRs? 

How are the reviewers identified? 

Do the reviewers receive training about how to conduct the SPSSRs? If so, what training do 

they receive? What is the timeframe for this training (e.g., how long before the actual SPSSR 

does the training occur)? 

What guidance does Federal staff provide to States concerning the SPSSR process? 

What materials are provided? 

What instructions are provided? 

What products result from the SPSSR? (e.g., SPSSR report and recommendations) 

What is the timeliness of the submission of products? 

Do States receive the SPSSR reports? If so, what do States do with the reports? 

How do Federal staff (program staff and grants management) use SPSSR products? 

How do States use the SPSSR process to improve their BG program implementation? (e.g., 

request TA, obtain policy guidance) 

What are the strengths of the SPSSR? 

What are the weaknesses of the SPSSR? 

What are recommendations for improving the SPSSR? 

To what extent does the SPSSR process improve Federal/State information exchange? 

What are any unintended positive or negative results of the SPSSR? 
Appendix C: Evaluation Framework 4

Grants Management 

What role does SAMHSA Grants Management staff play in the monitoring of State 

compliance with the BG program? 

Are there grants management policies that govern monitoring of State compliance with the 

BG program? If so, what are they? How are they enforced? 

BG Development and Support 

How does SAMHSA provide BG program development and support to States? 

What type of support is provided? 

What resources are available (e.g., TA, training)? 

Who provides BG program development and support (including TA, training)? 

SPOs? If so, in what areas? 

Federal contractors? If so, through what vehicles and in what areas? 

What are the strengths of BG program development and support provided by CSAT and 

CSAP to States? 

What are the weaknesses of BG program development and support provided by CSAT and 

CSAP to States? 

What are recommendations for improving BG program development and support provided 

by CSAT and CSAP to States? 

Data Collection, Analysis, Reporting, and Dissemination (e.g., TEDS, NOMs) 

How do CSAT and CSAP collect data on the BG program? 

What types of data are collected? 

Do CSAT and CSAP solicit feedback from States about BG data collection? If so, what 

are some examples of State feedback? 

How do CSAT and CSAP analyze data on the BG program? 

Who analyzes data on the BG program? 

To what extent were BG data used to improve Federal administration and management of 

the BG program? 

How do CSAT and CSAP report data on the BG program? 

What are examples of reports that are developed on BG program data? 

Who are the audiences for these reports on BG program data? 

How are SAPT BG program data disseminated? 

How do Federal staff use BG program data? 

How do State staff use Federally-disseminated SAPT BG program data? 

What are the strengths of SAPT BG data collection, analysis, and reporting? 

What are the weaknesses of SAPT BG data collection, analysis, and reporting? 

What are recommendations for improving SAPT BG data collection, analysis, and reporting? 

What are unintended positive or negative results of SAPT BG data collection, analysis, 

reporting, and dissemination? 
Appendix C: Evaluation Framework 5

State Implementation: For all State implementation categories: How do State 

implementation activities fulfill the SAPT BG legislative requirements? 

Application Development and Submission 

What is the process for the development and submission of the State BG application 

(including the intended use plan, progress report, and annual report)? 

Who is involved? 

What is the timeframe? 

Who approves the application on the State level? 

How are modifications made? 

What is the process for conducting a State needs assessment required by the BG program? 

Who is involved? 

How are State needs assessments funded? 

How often do States conduct needs assessments? 

How do States use needs assessment information? 

Is the State BG application used for any purpose other than obtaining Federal BG funds? If 

so, what are the other uses? 

What are the strengths of the process for developing the State BG application? 

What are the weaknesses of the process for developing the State BG application? 

What are supports that facilitate the SAPT BG State application and development process? 

What are barriers to the SAPT BG State application and development process? 

What are recommendations for improving the process for developing the State BG 

application? 

What are unintended positive or negative results of the SAPT BG application development 

and submission process? 

Annual Synar Report 

What is the process for the development and submission of the annual Synar report? 

Is the annual Synar report used for any purpose other than obtaining Federal SAPT Block 

Grant funds? 

If so, what are the other uses? 

What are the benefits of the annual Synar report process to States? 

What are the weaknesses of the annual Synar report process? 

What are recommendations for improving the process for developing the annual Synar 

report? 

What are the unanticipated positive or negative results from producing the annual Synar 

report? 

State Funding Allocation and Distribution 

What is the process by which States allocate BG funds? 

Who is involved? 

What is the timeframe? 

How do States allocate funds to meet the BG set-aside requirements? 
Appendix C: Evaluation Framework 6

How do States allocate funds to meet the 17 goals of the BG program? 

Are there State laws that impact how BG funds are allocated? 

Who approves the allocation of funds on the State level? 

How do States distribute BG funds to subrecipients? 

By which mechanisms? 

How frequently are funds distributed? 

How many subrecipients receive BG funds? What are their funding allocations? 

What are the strengths of the process for allocating and distributing BG funds? 

What are the weaknesses of the process for allocating and distributing BG funds? 

What are recommendations for improving the process for allocating and distributing BG funds? 

How does the allocation of BG funds affect the way that other funds are distributed in the 

State? 

Programs and Services 

What service modalities are funded through the SAPT BG? 

What types of programs are funded through the SAPT BG? 

What target populations do they serve? 

What types of EBPs and innovative programs are funded through the SAPT BG? 

How many individuals receive services funded through the BG? 

What are the issues involved in knowing this information? 

Have any programs developed and/or supported by BG funds been funded 

subsequently by other means? 

State-level Administrative Needs and Initiatives 

What State administrative activities are supported by the SAPT BG? 

Program Development 

Do States use BG resources to provide program development and support to subrecipients? If 

so, how? 

How are needs identified? 

Who provides BG program development and support (including TA, training)? 

State staff? If so, in what areas? 

State contractors? If so, in what areas? 

How many TA and training events occurred in the past year? 

How many subrecipient organizations received State TA and training in the past year? 

What are some examples of changes that have been made as a result of BG-related TA? 

What are the strengths of BG program development and support provided by States to 

subrecipients? 

What are the weaknesses of BG program development and support provided by States to 

subrecipients? 

What are recommendations for improving BG program development and support provided 

by States to subrecipients? 

What are unintended positive or negative results of BG program development/support 

provided by States to subrecipients? 
Appendix C: Evaluation Framework 7

Evaluation of Programs and Services Funded through the BG 

How do States collect data on the BG program from subrecipients? 

What types of data are collected? 

Do States solicit feedback from subrecipients about data collection? If so, what are 

examples of this feedback? 

Do States analyze data on the BG program? If so, for what purposes? 

Do States develop reports using data on the BG program? If so, what are examples of this? 

Who are the audiences for these reports? 

What are the strengths of State BG data collection, analysis, and reporting? 

What are the weaknesses of State BG data collection, analysis, and reporting? 

What are recommendations for improving State BG data collection, analysis, and reporting? 

What are unintended positive or negative results of State BG data collection, analysis, and 

reporting? 

SAPT BG Program Outcomes 

Federal Outcomes 

Short-term 

To what extent do States submit complete applications, State plans, and implementation 

reports? 

To what extent is the technical review process useful for Federal program monitoring? To 

what extent does the technical review process improve Federal/State information exchange? 

To what extent is the SPSSR process useful for Federal program monitoring? To what extent 

does the SPSSR process improve Federal/State information exchange? 

To what extent is there State compliance with statutory requirements? 

As a result of the data collection activities, to what extent does the Federal government have 

an improved ability to describe State BG program outcomes? 

To what extent were BG program data used to improve Federal administration and 

management of the BG program? 

Long-term 

As a result of BG data collection and analysis activities, to what extent does the Federal 

government have an improved capability to respond to Congressional information requests? 

To what extent were BG data used to make major improvements in Federal administration 

and management of the BG program? 

To what extent and how have BG resources been used to leverage Federal policy and priority 

changes? 

Do CSAT and CSAP provide leadership to States related to the SAPT BG program? If so, 

how? 

Through the SAPT BG program, do CSAT and CSAP play a national leadership role in 

improving the substance abuse prevention and treatment system? If so, how? 
Appendix C: Evaluation Framework 8

State Outcomes 

Short-term 

To what extent has the target population specified in the legislation been served using BG 

funds? 

As a result of BG activities, to what extent have States improved their documentation of State 

prevention and treatment activities? 

Has there been an increase in positive client perceptions of care? 

Long-term 

As a result of BG activities, to what extent have States improved their coordination of State 

substance abuse prevention and treatment services/programs? 

To what extent has there been an increase in the number of EBPs and innovative services 

available because of the BG program? 

To what extent has the BG program contributed to improving the quality of States’ substance 

abuse prevention and treatment services? 

To what extent has the BG program contributed to a decrease in unmet treatment need? 

To what extent have programs initiated with SAPT BG funds been continued using State and 

other funding sources (e.g., leveraging)? 

To what extent have States leveraged SAPT BG resources to implement policy change? 

Client Outcomes 

To what extent have clients achieved abstinence from drug and alcohol use due to 

participation in BG-funded programs? 

To what extent have clients increased their employment and school participation due to BGfunded programs? 

To what extent have clients decreased their criminal justice involvement due to participation 

in BG-funded programs? 

To what extent have clients increased their housing stability due to participation in BGfunded programs? 

To what extent have clients increased their social support and connectedness due to 

participation in BG-funded programs?

(treatment only) 
Appendix D 

Evaluation Instruments 
OMB #0930-0291 

Expiration Date: 3/31/2011 

1

INTERVIEW GUIDE FOR FEDERAL STAFF INVOLVED WITH THE SAPT BG PROGRAM

Estimates of Burden for the Collection of Information. 

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of 

information unless it displays a currently valid OMB control number. The OMB control number 

for this project is 0930-0291. Public reporting burden for this collection of information is 

estimated to average 90 minutes per interview, including the time for reviewing instructions, 

searching existing data sources, gathering and maintaining the data needed, and completing and 

reviewing the collection of information. Send comments regarding this burden estimate or any 

other aspect of this collection of information, including suggestions for reducing this burden, to 

SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, 

Maryland, 20857. 

Introduction 

Thank you so much for taking the time to participate in this interview. We know that you are 

extremely busy, and we greatly appreciate your input. As you know, the Centers for Substance 

Abuse Treatment (CSAT) and Prevention (CSAP) have contracted with Altarum Institute to 

conduct an evaluation of the Substance Abuse Prevention and Treatment Block Grant Program 

(SAPT BG). The purpose of our discussion today is to learn how the SAPT BG is implemented 

at the Federal level and to understand the impact of the SAPT BG within States. 

Your agency’s name and location and your general job title (e.g., Public Health Advisor) may be 

identified in reports prepared for this study and in data files provided to CSAT and CSAP. 

However, none of your responses during the interview will be released in a form that identifies 

you or any other Federal staff member by name. Please remember that this study is not part of an 

audit or management review of Federal operations. Your participation in the interview is 

completely voluntary. 

The estimated total time to complete this interview is 90 minutes. 

Do you have any questions before we begin? 

Background

1. What is your title, and how long have you been in this position? 

2. Briefly describe your responsibilities with regard to the SAPT BG. 

Federal Funding Distribution 

3. How do CSAT and CSAP allocate SAPT BG funds to States? 

Probes: 

a) Is there an allocation formula? If so, on what is the allocation formula based? 

b) Who is involved? What roles do they play? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

2

c) What is the time frame for the allocation process? 

d) What role do State Project Officers (SPOs) play in the allocation of SAPT BG 

funds? 

e) Do you feel the allocation formula can be improved? If so, in what ways? 

 

4. What Federal activities are supported by the SAPT BG 5 percent administrative setaside? 

Development of Application Template and Guidance for States

5. What is the intended purpose of the application template and guidance? 

6. What is the process for the development of the application template and guidance for 

States? 

 Probes: 

a) Who is involved? 

b) How are changes to the template and guidance communicated to stakeholders 

(e.g., other Federal staff members, State stakeholders)? 

c) What is involved in order to make changes to the application template and 

guidance? 

d) What have been some of the most recent changes, and why? What future 

changes are anticipated? 

e) What is the time frame for the development and distribution of the application 

template and guidance each year? 

f) Are there challenges related to adhering to the time frame? If so, please 

describe. 

7. What are the strengths of the application template and guidance document? 

8. What are the weaknesses of the application template and guidance document? 

9. Is the application template and guidance used by States beyond its intended purpose? If 

so, what are other uses? 

10. How would you improve the application template and guidance document? 

11. How would you improve the process of developing the application template and 

guidance? 

Application Review and Approval 

12. How are SAPT BG applications reviewed and approved? 

 Probes: 

a) What role do SPOs play in the review and approval of applications? 

b) Is the Block Grant Application System (BGAS) used in this process? If so, 

how? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

3

c) How is approval status communicated to other Federal staff members and to 

the States? 

13. What are the strengths of the application review and approval process? 

14. What are the weaknesses of the application review and approval process? 

15. How would you improve the application review and approval process? 

16. Have there been any unintended positive or negative results of the application review 

and approval process? If so, please describe. 

Use of Application Information 

 

17. How do you use States’ application information? 

 

18. How is BGAS used in this process? 

 

19. What reports are generated from BGAS: 

a) On a regular basis? 

b) On a one-time basis? 

 

20. Other than the SPOs, who requests information that can be obtained from States’ 

applications? 

Annual Synar Report (ASR) 

21. What is the intended purpose of the ASR? 

22. What is the process by which guidance is given to States on developing their ASR? 

23. How is the ASR reviewed and approved? 

24. How are the ASRs used? 

25. Other than SPOs, who requests information obtained from the ASRs? 

26. What are the strengths of the ASR process? 

27. What are the weaknesses of the ASR process? 

28. How would you improve the ASR process? 

29. Have there been any unintended positive or negative results of the ASR process? If so, 

please describe. 
OMB #0930-0291 

Expiration Date: 3/31/2011 

4

Program Monitoring 

Technical Reviews 

 

[Interviewers: The following questions should be administered to Federal staff members who 

work in substance abuse treatment. If you are not speaking with any treatment staff involved 

with CSAT Technical Reviews, skip to the next section.] 

30. Please describe your role in the Technical Reviews. 

 Probes: 

a) What is your role in the Core Elements Technical Review? 

b) What is your role in the State Requested Technical Review? 

31. What is the purpose of the Technical Reviews? 

 Probes: 

a) What is the purpose of the Core Elements Technical Review? 

b) What is the purpose of the State Requested Technical Review? 

32. Are there policies governing the technical review process? If so, what are they? 

 Probes: 

a) How are States selected for Core Elements Technical Reviews each year? 

b) What is the process for handling a State-Requested Technical Review? Who 

decides what issues will be addressed? 

 

33. Who conducts the technical reviews? 

 

34. How are the reviewers identified? 

 

35. Do the reviewers receive training about how to conduct the technical reviews? If so, 

what training do they receive? What is the time frame for this training (e.g., how long 

before the actual technical reviews does the training occur)? 

 

36. What guidance do you provide to States concerning the technical review process? 

 Probes: 

a) What materials are provided? 

b) What instructions are provided? 

37. What products result from the technical reviews (e.g., technical review report and 

recommendations)? 

38. What is the timeliness of the submission of technical review products? 

 

39. How do you (program staff and grants management) use technical review products? 

 
OMB #0930-0291 

Expiration Date: 3/31/2011 

5 

40. Do States receive the technical review reports? If so, do you know what States do with 

the reports? 

 

State Prevention and Synar Systems Reviews \ 

[Interviewers: The following questions should be administered to Federal staff members who 

work in substance abuse prevention. If you are not speaking with any prevention staff, skip to 

the Grants Management section.] 

41. Please describe your role in the State Prevention and Synar System Reviews. 

42. What is the purpose of the State Prevention and Synar System Reviewss? 

43. Are there policies governing the State Prevention and Synar System Review process? 

If so, what are they? 

Probe: 

a) How are States selected for State Prevention and Synar System Reviews each 

year? 

 

44. Who conducts the State Prevention and Synar System Reviews? 

 

45. How are the reviewers identified? 

 

46. Do the reviewers receive training about how to conduct the State Prevention and Synar 

System Reviews? If so, what training do they receive? What is the time frame for this 

training (e.g., how long before the actual State Prevention and Synar System Review 

does the training occur)? 

 

47. What guidance do you provide to States concerning the State Prevention and Synar 

System Review process? 

 Probes: 

a) What materials are provided? 

b) What instructions are provided? 

 

48. What products result from the State Prevention and Synar System Reviews (e.g., State 

Prevention and Synar System Review report and recommendations)? 

 

49. What is the timeliness of the submission of products? 

 

50. How do you (program staff and grants management) use State Prevention and Synar 

System Review products? 

51. Do States receive the State Prevention and Synar System Review reports? If so, do you 

know what States do with the reports? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

6 

Grants Management 

 

52. What role does Grants Management play in monitoring compliance with the SAPT BG 

program? 

 Probes: 

a) Are there specific grants management policies that govern the monitoring of 

compliance with the SAPT BG program? 

b) If so, what are they? 

c) How are they enforced? 

53. How would you improve the services provided by Grants Management to States? 

 

Other Program Monitoring 

54. In what other ways do CSAT and CSAP monitor State compliance with the SAPT BG 

requirements? 

55. How are potential issues with State compliance identified? 

 Probes: 

a) Are States’ intended use plans compared to States’ annual reports and 

progress reports to assess the degree to which planned activities were 

implemented? If so, please describe this process. 

b) Who decides what potential issues require Federal or State action? 

c) How are issues that require action communicated to States? 

d) Is there followup from CSAT and CSAP to determine if potential issues have 

been addressed? 

56. What are the strengths of SAPT BG program monitoring? 

57. What are the weaknesses of SAPT BG program monitoring? 

58. How would you improve SAPT BG program monitoring? 

59. Have there been any unintended positive or negative results of SAPT BG program 

monitoring? If so, what are they? 

SAPT BG Development and Support 

60. How do CSAT and CSAP provide SAPT BG-related support (e.g., training, technical 

assistance) to States? 

 Probes: 

a) What types of support are provided? 

b) Who provides SAPT BG-related support to States? 

c) If SPOs, in what areas do they provide support? 

d) If contractors, in what areas and through what vehicles do they provide 

support? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

7 

61. What are the strengths of the SAPT BG-related support that CSAT and CSAP provide 

to States? 

62. What are the weaknesses of the SAPT BG-related support that CSAT and CSAP 

provide to States? 

63. How would you improve the SAPT BG-related support that CSAT and CSAP provide 

to States? 

Data Collection (e.g., Treatment Episode Data Set, Voluntary Prevention and Treatment 

Measures, National Outcome Measures), Analysis, and Dissemination 

64. How do CSAT and CSAP collect data on the SAPT BG program, and for what 

purposes? 

 Probes: 

a) What types of data are collected? 

65. Do CSAT and CSAP solicit feedback from States about SAPT BG data collection? If 

so, how? 

66. Do CSAT and CSAP incorporate State feedback about SAPT BG data collection? If 

so, please provide examples of State feedback that CSAT and CSAP have 

incorporated. 

67. How do CSAT and CSAP analyze data on the SAPT BG program? 

 Probe: 

a) Who analyzes data on the SAPT BG program? 

 

68. How do CSAT and CSAP disseminate data on the SAPT BG program? 

 Probes: 

a) What are examples of reports that are developed using SAPT BG program 

data? 

b) Who are the audiences for these reports on SAPT BG program data? 

c) Do CSAT and CSAP share SAPT BG data with States? If so, how? 

d) What other stakeholders receive SAPT BG program data, and for what 

purposes? 

69. Do you use SAPT BG program data? If so, in what ways (e.g., Federal administration 

and management)? 

70. What are the strengths of SAPT BG data collection, analysis, and dissemination? 

71. What are the weaknesses of SAPT BG data collection, analysis, and dissemination? 

72. How would you improve SAPT BG data collection, analysis, and dissemination? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

8 

73. Have SAPT BG program data been used for purposes other than those originally 

intended? If so, please describe. 

74. As a result of data collection, analysis, and dissemination activities, to what extent has 

the Federal Government improved its ability to describe State BG program outcomes? 

75. Have SAPT BG data been used to make improvements in Federal administration and 

management of the SAPT BG program? If so, please describe the changes. 

76. Have there been any unintended positive or negative results of SAPT BG data 

collection, analysis, and dissemination? If so, please describe.

Federal-level SAPT BG Outcomes 

77. Do the Technical Reviews and State Prevention and Synar System Reviews improve 

State and Federal communication and information exchange? Please explain. 

78. Have SAPT BG resources been used to leverage Federal policy and priority changes? 

If so, please describe. 

79. Do CSAT and CSAP play a leadership role in improving the substance abuse 

prevention and treatment system? In guiding the States in the SAPT BG? If yes, 

please describe. 

80. In what other ways has the SAPT BG had an impact at the Federal level? 

Closing 

Thank you very much for your time. Your participation is greatly appreciated. If you think of 

anything else you would like to add, please contact me by phone or e-mail. 
OMB #0930-0291 

Expiration Date: 3/31/2011 

1

INTERVIEW GUIDE FOR STATE STAFF INVOLVED WITH THE SAPT BLOCK GRANT PROGRAM

OMB Clearance Number: 0930-0291 Expiration Date: 3/31/2011

Estimates of Burden for the Collection of Information. 

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a 

collection of information unless it displays a valid Office of Management and Budget 

(OMB) control number. The valid OMB control number for this collection is 0930-0291. 

The time required to complete this information collection is estimated to average 3.5 hours 

per interview, including the time to review instructions, search existing data resources, 

gather the data needed, and complete and review the information collected. 

INTRODUCTION

Thank you so much for taking the time to participate in this interview. We know that you are 

extremely busy, and we greatly appreciate your input. As you know, the Centers for Substance 

Abuse Treatment (CSAT) and Prevention (CSAP) have contracted with Altarum Institute to 

conduct an evaluation of the Substance Abuse Prevention and Treatment Block Grant Program. 

The purpose of our discussion today is to learn how SAPT Block Grant activities are 

implemented and to understand the impact of the SAPT Block Grant in your State. 

As described in the letter we sent you earlier, your agency’s name and location and your general 

job title (e.g., State Mental Health Commissioner, State Planner) may be identified in reports 

prepared for this study and in data files provided to CSAT and CSAP. However, none of your 

responses during the interview will be released in a form that identifies you or any other State 

staff member by name. Please remember that this study is not part of an audit or management 

review of State operations. Your participation in the interview is completely voluntary. Failure to 

complete the interview will not affect your State’s SAPT Block Grant in any way.

The estimated total time to complete this interview is 3.5 hours. When we are halfway through 

the interview protocol, we will have a 10-minute break. It will be important to keep the break to 

10 minutes in order to complete the interview within the allotted time. In addition, if we are 

spending too long on any given section of the protocol, I will interrupt gently to move us forward 

so that we can complete the interview within the allotted timeframe. We greatly appreciate your 

detailed feedback; however, we want to be respectful of your busy schedules. 

Do you have any questions before we begin? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

2

BACKGROUND

1. What is your title, and how long have you been in this position? 

2. Briefly describe your responsibilities with regard to the SAPT Block Grant. 

[Interviewers: Please be sure to gather this information from all State participants.] 

FEDERAL ACTIVITIES

Application Template and Guidance for States 

3. Is there a formal mechanism for your State to provide feedback on the application 

guidance and template? 

4. In past years, there were changes to both the application template and the application 

guidance. Was there any official notification regarding these changes prior to the release 

of the application template and the guidance? 

5. How far in advance of the application deadline did your State receive the template and 

the guidance? Were you satisfied with this time frame? 

6. How would you improve the application template and guidance? Their dissemination? 

Application Review and Approval 

7. What are the benefits to the State of the application review and approval process? 

8. What are the weaknesses of the application review and approval process? 

9. How would you improve the application review and approval process? 

Program Monitoring 

Technical Reviews 

[Interviewers: The following questions should be administered to State staff members who 

work in substance abuse treatment. If you are not speaking with any State staff involved with 

CSAT Technical Reviews, skip to the next section.] 

Core Elements Technical Reviews 

10. When was your State’s most recent Federally required Core Elements Technical Review 

site visit? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

3

11. Prior to a Core Elements Technical Review site visit, does your State receive guidance 

from Federal staff about expectations for the site visit (for example, materials that should 

be prepared)? If so, please describe. 

12. What products does your State receive after a Core Elements Technical Review site visit? 

Approximately how long after a site visit do you receive the products? Are you satisfied 

with this time frame? 

13. What changes, if any, have been made as a result of a Core Elements Technical Review 

site visit and the subsequent products (e.g., report, recommendations)? 

14. How does your State use the Core Elements Technical Review reports? 

15. What are the benefits of the Core Elements Technical Review? Are these reviews useful 

to your State? If so, how? 

16. What are the weaknesses of the Core Elements Technical Review? 

17. How would you improve the Core Elements Technical Review? How could these reviews 

be more useful to your State? 

18. Have there been any unintended positive or negative results of the Core Elements 

Technical Review? If so, please describe. 

19. Do the Core Elements Technical Reviews improve State and Federal communication and 

information exchange? Please explain. 

State-Requested Technical Reviews 

20. Has your State participated in a State-Requested Technical Review? [Interviewer: If yes, 

continue. If no, skip to the next section.] 

21. Please describe the most recent State-Requested Technical Reviews in which your State 

has participated. 

Probes: 

a) Why did your State request a technical review? 

b) What issues did your State wish to address through the technical review? 

c) What was the process for requesting and receiving a State-requested site visit? 

22. What were the results of any State-Requested Technical Reviews? Was your State 

satisfied with the requested technical reviews? Please explain. 

23. How could the State-Requested Technical Reviews have been more helpful to your State? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

4

24. What products did your State receive after a State-Requested Technical Review site visit? 

Approximately how long after the site visit did you receive the products? Were you 

satisfied with this time frame? 

25. What changes, if any, have been made as a result of State-Requested Technical Review 

site visits and the subsequent products (for example, reports or recommendations)? 

26. Were there any unintended positive or negative results of the State-Requested Technical 

Reviews? If so, please describe. 

27. Do the State-Requested Technical Reviews improve State and Federal communication 

and information exchange? Please explain. 

[FACILITATORS: IF MORE THAN 1 HOUR HAS PASSED, SPEED UP THE 

INTERVIEW.] 

State Prevention and Synar System Reviews 

[Interviewers: The following questions should be administered to State staff members who 

work in substance abuse prevention. If you are not speaking with any prevention staff, skip to 

the next section.] 

28. When was your State’s most recent State Prevention and Synar System Review 

conducted? 

29. Prior to a State Prevention and Synar System Review site visit, does your State receive 

guidance from Federal staff about expectations for the site visit (for example, materials 

that should be prepared)? If so, please describe. 

30. What products does your State receive after a State Prevention and Synar System Review 

site visit? Approximately how long after a site visit do you receive the products? Are you 

satisfied with this time frame? 

31. What changes, if any, have been made as a result of a State Prevention and Synar System 

Review site visit and the subsequent products (for example, report or recommendations)? 

32. How does your State use the State Prevention and Synar System Review reports? 

33. What are the benefits of the State Prevention and Synar System Reviews? Are the State 

Prevention and Synar System Reviews useful to your State? If so, how? 

34. What are the weaknesses of the State Prevention and Synar System Reviews? 

35. How would you improve the State Prevention and Synar System Reviews? How could 

the State Prevention and Synar System Reviews be more useful to your State? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

5

36. Are there any unintended positive or negative results of the State Prevention and Synar 

System Reviews? If so, please describe. 

37. Do the State Prevention and Synar System Reviews improve State and Federal 

communication and information exchange? Please explain. 

Grants Management 

 

38. What role does Grants Management play in monitoring compliance with the SAPT Block 

Grant program? 

 Probes: 

d) Are there specific grants management policies that govern the monitoring of 

State compliance with the SAPT Block Grant program? 

e) If so, what are they? 

f) How are they enforced? 

39. How would you improve the services provided by Grants Management to States? 

Other Program Monitoring 

40. What is the purpose of any other Federal program monitoring (e.g., compliance, program 

improvement)? 

41. How are potential issues with State compliance identified? 

 Probes: 

e) Who decides what potential issues require Federal or State action? 

f) How are issues that require action communicated to States? 

g) Is there followup from CSAT and CSAP to determine if potential issues have 

been addressed? 

42. What are the strengths of any other SAPT Block Grant program monitoring? 

43. What are the weaknesses of any other SAPT Block Grant program monitoring? 

44. How would you improve any other SAPT Block Grant program monitoring? 

45. Have there been any unintended positive or negative results of any other SAPT Block 

Grant program monitoring? If so, what were they? 

SAPT Block Grant Technical Assistance (TA) and Training (Federal to State) 

[Interviewer: Make sure to ask about both prevention and treatment TA and training, depending 

on the composition of the interview group.] 

46. In the past few years, has your State received TA and training through Federal SAPT 

Block Grant resources? If yes, in what areas? In what formats? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

6

47. Was your State satisfied with the TA and training received through Federal SAPT Block 

Grant resources? Why or why not? 

48. In the past few years, has your State utilized any off-the-shelf Federal TA products 

supported by the SAPT Block Grant? If yes, which products? 

49. What, if any, specific changes has your State made as a direct result of Federal TA or 

training? Were they were made as a result of in person TA delivered by consultants or 

as a result of the Federal TA products? 

50. How would you improve consultant-based Federal TA and training to States? 

51. How would you improve the Federal TA products currently available? 

[FACILITATORS: TAKE A 10-MINUTE BREAK NOW; IF MORE THAN 2 HOURS 

HAVE PASSED, BE PREPARED TO SPEED UP THE SECOND HALF OF THE 

INTERVIEW] 

Data Collection (e.g., Treatment Episode Data Set, National Outcome Measures), Analysis, 

and Dissemination 

52. Do CSAT and CSAP solicit feedback from the States about Federally required data 

collection? If so, how? 

53. Has your State ever provided feedback – either officially or unofficially – about SAPT 

Block Grant data collection for the Treatment Episode Data Set and National Outcome 

Measures? If so, please describe. Were CSAT and CSAP responsive to your feedback? 

Please explain. 

54. How prepared is your State to respond to NOMS? What is the current status of your 

State’s capacity to collect data for NOMS?

55. Does your State receive Federal reports based on data from the SAPT Block Grant 

Program? If so, how does your State use these reports? 

56. Have there been any unanticipated positive or negative results from complying with 

SAPT Block Grant data collection, analysis, and reporting? If so, please describe. 

57. What are the strengths of the Federally required SAPT Block Grant data collection? 

58. What are the weaknesses of the Federally required SAPT Block Grant data collection? 

59. How would you improve the data collection process? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

7

STATE ACTIVITIES 

SAPT Block Grant Application Development 

60. Please describe the SAPT Block Grant application development process. 

 Probes: 

a) Who is involved (roles rather than names)? 

b) How long does the process take? 

c) What feedback is sought? 

61. What is the process for conducting the State needs assessment required by the SAPT 

Block Grant program? 

Probes: 

a) Who is involved? 

b) How are your State’s needs assessments funded?

c) How often does your State conduct needs assessments? 

d) How does your State use needs assessment information? 

 

62. Is your State Block Grant application used for any purpose other than obtaining Federal 

SAPT Block Grant funds? If so, what are the other uses? 

 

63. What are the benefits of the application process to your State? 

64. What are the weaknesses of the application process? 

65. How would you improve the SAPT Block Grant application process? 

66. Have there been any unanticipated positive or negative results from producing the SAPT 

Block Grant application? If so, please describe. 

Annual Synar Report 

67. Please describe the annual Synar report development process. 

 Probes: 

a) Who is involved (roles rather than names)? 

b) How long does the process take? 

c) What feedback is sought? 

68. Is your annual Synar report used for any purpose other than obtaining Federal SAPT 

Block Grant funds? If so, what are the other uses? 

 

69. What are the benefits of the annual Synar report process to your State? 

70. What are the weaknesses of the annual Synar report process? 

71. How would you improve the annual Synar report process? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

8

72. Have there been any unanticipated positive or negative results from producing the annual 

Synar report? If so, please describe. 

State Funding Allocation 

73. What is the process by which your State allocates SAPT Block Grant funds (e.g., 

allocation formula)? 

 Probes: 

a) Who is involved (roles rather than names)? 

b) How long does the process take? 

74. Does the allocation of SAPT Block Grant funds affect the way that other funds are 

distributed in the State? If so, please describe. 

 

75. What are the advantages of your State’s SAPT Block Grant funding allocation process? 

76. What are the disadvantages of your State’s SAPT Block Grant funding allocation 

process? 

77. How would you improve your State’s process for allocating SAPT Block Grant funds?

[FACILITATORS: IF MORE THAN 3 HOURS HAVE PASSED, MOVE THE 

INTERVIEWEES ALONG MORE QUICKLY IN THE LAST QUARTER OF THE 

INTERVIEW.] 

Programs and Services Funded Through the SAPT Block Grant 

78. Are there any programs that have been developed or supported originally using SAPT 

Block Grant funds that have been continued using other means of support? If so, please 

describe the program and the other means of support. 

79. Are there any State-level administrative activities that are directly supported by SAPT 

Block Grant funds? If so, please describe. 

TA and Training Provided to Subrecipients 

[Interviewer: Make sure to ask about prevention, treatment, and Synar TA and training, 

depending on the composition of the interview group.] 

80. In the past few years, has your State used SAPT Block Grant resources to provide TA or 

training to subrecipients (organizations that receive SAPT Block Grant funds)? If so, 

please describe the types of TA and training that your State has provided. 
OMB #0930-0291 

Expiration Date: 3/31/2011 

9

81. What is the process for deciding what TA and training should be offered to SAPT Block 

Grant subrecipients? 

 Probe: 

a) Who identifies training or TA needs? 

82. Who has provided TA or training? State staff, contractors, other? 

83. Using your best estimate, how many TA and training events were conducted in the past 

year? 

84. Using your best estimate, how many different subrecipients or providers participated in 

the TA and training events in the past year? 

85. To the extent that you are aware, please describe any programmatic changes that have 

occurred as a result of receiving TA or training. 

86. How would you improve TA and training to subrecipients? 

87. Have there been any unanticipated positive or negative results from providing TA and 

training to subrecipients? If so, please describe. 

Performance Monitoring of Programs and Services that Receive SAPT Block Grant 

Funding 

 

88. How does your State collect prevention and treatment data from subrecipients? 

Probe: 

a) Does your State provide data collection forms or templates to subrecipients? If 

so, please describe. 

b) Does your State conduct onsite monitoring site visits? If so, please describe. 

c) How does your State ensure that providers and subrecipients are collecting 

data in accordance with Block Grant requirements (for example, using TEDS 

definitions for admission and discharge)? 

d) Do State staff review data submitted by providers before passing the data on 

to CSAT? If so, please describe the review process. 

89. How does the State analyze data collected for the SAPT Block Grant Program? 

90. How does your State use the data provided by SAPT Block Grant subrecipients (e.g., 

produce SAPT Block Grant State application, the annual Synar report, and other reports)? 

91. Is there a formal process for subrecipients to provide feedback to the State about the data 

collection for the SAPT Block Grant? If so, please describe the process. 

92. What types of feedback have subrecipients provided to the State about data collection? 
OMB #0930-0291 

Expiration Date: 3/31/2011 

10

93. Has this feedback been incorporated? If so, please provide examples of incorporated 

feedback. 

94. Have there been any unanticipated positive or negative results from collecting, analyzing, 

and reporting subrecipient SAPT Block Grant data? If so, please describe. 

95. How would you improve the subrecipient-to-State data collection process? 

SAPT BLOCK GRANT PROGRAM OUTCOMES

Federal Outcomes 

96. Do CSAT and CSAP play a leadership role in improving the substance abuse prevention 

and treatment system? In guiding the States in the SAPT Block Grant? If yes, please 

describe. 

State Outcomes 

97. As a result of SAPT Block Grant activities, has your State improved its coordination of 

substance abuse prevention and treatment services and programs? If so, please describe 

any improvements and how the SAPT Block Grant contributed to them. 

98. Has there been an increase in the number of evidence-based practices and innovative 

services available because of the SAPT Block Grant program? If so, please describe the 

newer services and how the SAPT Block Grant contributed to their availability. 

99. Has your State leveraged SAPT Block Grant requirements and resources to implement 

State policy or funding changes? If so, please provide examples. 

CLOSING

Thank you very much for your time. Your participation is greatly appreciated! If you think of 

anything else you would like to add, please contact me by phone or e-mail. 
OMB #0930-0291

Expiration Date: 3/31/2011

The Independent Evaluation of the SAPT BG Program

Estimates of Burden for the Collection of Information.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of 

information unless it displays a currently valid OMB control number. The OMB control number for this 

project is 0930-0291, which expires March 31, 2011. Public reporting burden for this collection of 

information is estimated to average 60 minutes per response, including the time for reviewing instructions, 

searching existing data sources, gathering and maintaining the data needed, and completing and reviewing 

the collection of information. Send comments regarding this burden estimate or any other aspect of this 

collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance 

Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

Dear Technical Reviewer:

As you know, the Substance Abuse Prevention and Treatment Block Grant (SAPT BG) Program was 

authorized by Congress to provide funds to States, Territories, and one Indian Tribe for the purpose of 

planning, implementing, and evaluating activities to prevent and treat substance abuse and is the largest 

Federal program dedicated to improving substance abuse prevention and treatment systems. The sponsors 

of the program, the Substance Abuse and Mental Health Services Administration’s Center for Substance 

Abuse Prevention and Center for Substance Abuse Treatment, have contracted with Health Systems 

Research, Inc., a public policy consulting firm in Washington D.C., to conduct an independent evaluation 

of this program. 

We are soliciting feedback about the SAPT BG from key program stakeholders. As a past Technical 

Reviewer, you have important insights and views about the intent, implementation, and impact of the 

SAPT BG from your site visits to States. We would greatly appreciate your assistance with the evaluation 

through the completion of this survey. Most of the questions are closed-ended questions where you will 

be asked to check the appropriate answer or answers. In addition, there are several open-ended questions 

where you have the opportunity to comment. We urge you to be as honest and thorough as possible. 

Thank you very much for taking the time to participate. 

Please be assured that your answers will be strictly confidential. We will only report aggregated responses 

to the questions, and we will never attribute specific comments to particular individuals. Your responses 

will not be used by CSAP to assess State compliance with the requirements and will not have any 

repercussions for any particular State; they will be used solely for the purpose of evaluating the overall 

SAPT BG Program.

SAPT Block Grant Survey Technical Reviewer Page 1 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

1) Have you participated in Core Elements Technical Reviews? [If yes, continue. If no, skip 

to question 28.]

Yes

No

Cannot answer

2) If yes, which of the following was your role? (Check all that apply)

3) What is the purpose of the Core Elements Technical Review?

Management reviewer

Clinical reviewer

Fiscal Team Leader

Other (please describe):

Team Leader

4) How often does a State representative identify compliance issues for the reviewers 

during the Core Elements Technical Review site visit? 

5) When conducting Core Elements Technical Review site visits, how often do you identify 

potential issues for Federal action? 

Never

Rarely

Sometimes

Usually 

Always

Cannot answer

Never

Rarely

Sometimes

Usually 

Always

Cannot answer

SAPT Block Grant Survey Technical Reviewer Page 2 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

6) When conducting Core Elements Technical Review site visits, how often do you identify 

potential issues for State action? 

Never

Rarely

Sometimes

Usually 

Always

Cannot answer

7) How are issues that require action communicated to the States? (Check all that apply)

Site visit report

Other site visit products

(please describe)

Other (please describe)

E-mail contact

Phone contact

Exit interview at the conclusion of site visit

8) How are issues that require action communicated to CSAT? (Check all that apply)

Email contact

Phone contact

Exit interview at the conclusion of site visit

Site visit monitoring report

Other site visit products

(please describe)

Other (please describe)

9) What are the strengths of the Core Elements Technical Review process? 

SAPT Block Grant Survey Technical Reviewer Page 3 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

10) What are the weaknesses of the Core Elements Technical Review process? 

11) To what extent do you agree that the Core Elements Technical Review site visits are 

useful to States?

Strongly disagree

Somewhat disagree

Neither disagree nor agree

Somewhat agree

Strongly agree

Cannot answer

12) Please list any ways in which you feel the Core Elements Technical Review site visits 

could be more useful to States.

13) What types of training, if any, did you receive to prepare you for conducting the Core 

Elements Technical Review site visits? (Check all that apply)

Other (please describe)

In-person training

Written instructions

No training received

Shadowing of experienced site visitors

SAPT Block Grant Survey Technical Reviewer Page 4 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

14a) If you received in-person training, please rate how useful you feel the training you 

received was.

14b) If you received in-person training, how long before the actual site visits began did the 

training occur? (Check one)

15a) Do you have any recommendations for improving the training and preparation for the 

site visits to make you a more effective reviewer? 

15b) If yes, what are they?

16a) If you received written instructions, please rate how useful they were.

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

Yes

No

SAPT Block Grant Survey Technical Reviewer Page 5 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

16b) If you received written instructions, how long before the actual site visits began did you 

receive them? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

16c) How useful was the Core Element Review Protocol in helping to gather the information 

needed to prepare the report?

Not useful

A little useful

Somewhat useful 

Mostly useful

Extremely useful

Cannot answer

17a) Were you provided any other materials in preparation for the site visits?

Yes

No

17b) If yes, what other materials?

18a) Do you think that there are requirements of the SAPT BG program that are not 

addressed adequately in the protocols for the Core Elements Technical Reviews?

Yes

No

SAPT Block Grant Survey Technical Reviewer Page 6 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

18b) If yes, please discuss.

19a) Did you receive information about the State’s responsibilities in the Core Elements 

Technical Review site visit? 

Yes

No

Cannot answer

19b) If yes, what information did you receive?

20) On average, how prepared are the States that you have visited on Technical Review site 

visits?

Very unprepared

Somewhat unprepared

Minimally prepared

Very prepared

Completely prepared

Cannot answer

21) What products result from the Core Elements Technical Review site visits to States? 

Other (please describe)

Site visit report and recommendations

Technical assistance plan

Strategic plan

SAPT Block Grant Survey Technical Reviewer Page 7 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

22) How long after a site visit do you typically submit your drafts of site visit products to 

CSAT? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

23) How long after the submission of the technical review report draft to CSAT do States 

typically receive a copy of the site visit report? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

24a) Do you know if Federal program staff and grants management use site visit products? 

Yes

No

Don't know

24b) If yes, how?

25a) Do you know if States use site visit products? 

Yes

No

Don't know

SAPT Block Grant Survey Technical Reviewer Page 8 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

25b) If yes, how?

26a) Do you have any recommendations for improving the dissemination of Core Elements 

Technical Review site visit products?

26b) If yes, what are they?

27a) Do you have any recommendations for more effective uses of the Core Elements 

Technical Review site visit products? 

27b) If yes, what are they?

28) Have you participated in State-Requested Technical Reviews? [If yes, continue. If no or 

cannot answer, the survey is complete.]

Yes

No

Don't know

Yes

No

Don't know

Yes

No

Cannot answer

SAPT Block Grant Survey Technical Reviewer Page 9 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

29) What is the purpose of the State-Requested Technical Review?

30) For what issues have you conducted a State-Requested Technical Review?

31) What have been the results of the State-Requested Technical Reviews in which you have 

participated?

32) What are the strengths of the State-Requested Technical Reviews? 

33) What are the weaknesses of the State-Requested Technical Reviews? 

SAPT Block Grant Survey Technical Reviewer Page 10 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

34) To what extent do you agree that the State-Requested Technical Review site visits are 

useful to States?

35) Please list any ways in which you feel the State-Requested Technical Review site visits 

could be more useful to States.

36) What types of training, if any, did you receive to prepare you for conducting the StateRequested Technical Review site visits? (Check all that apply)

37a) If you received in-person training, please rate how useful you feel the training you 

received was.

Strongly disagree

Somewhat disagree

Neither disagree nor agree

Somewhat agree

Strongly agree

Cannot answer

In-person training

Written instructions

No training received

Shadowing of experienced sit

Other (please describe)

e visitors

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

SAPT Block Grant Survey Technical Reviewer Page 11 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

37b) If you received in-person training, how long before the actual State-requested site visits 

began did the training occur? (Check one)

38a) If you received written instructions, please rate how useful they were.

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

38b) If you received written instructions, how long before the actual State-requested site 

visits began did you receive them? (Check one)

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

39a) Were you provided any other materials in preparation for the State-requested site 

39b) If yes, what other materials?

Yes

No

SAPT Block Grant Survey Technical Reviewer Page 12 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

40) Please rate how prepared the last State you visited was for the State-requested site visit.

41) What products resulted from the State-Requested Technical Review site visits to States? 

(Check all that apply)

42) How long after a State-requested site visit do you typically submit your drafts of site 

visit products to CSAT? (Check one)

43) How long after the submission of the draft State-requested technical review report to 

CSAT do States typically receive a copy of it? (Check one)

Very unprepared

Somewhat unprepared

Minimally prepared

Very prepared

Completely prepared

Cannot answer

Site visit report and recommendations

Technical assistance plan

Strategic plan

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

Other (please describe)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

SAPT Block Grant Survey Technical Reviewer Page 13 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

44a) Do you know if Federal program staff and grants management use State-requested site 

visit products? 

Yes

No

Don't know

44b) If yes, how?

45a) Do you know how States use State-Requested Technical Review products? 

Yes

No

Don't know

45b) If yes, how?

46a) Do you have any recommendations for improving the dissemination of State-Requested 

Technical Review site visit products?

46b) If yes, what are they?

Yes

No

Don't know

SAPT Block Grant Survey Technical Reviewer Page 14 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

47a) Do you have any recommendations for more effective uses of the State-Requested 

Technical Review site visit products? 

47b) If yes, what are they?

Yes

No

Don't know

SAPT Block Grant Survey Technical Reviewer Page 15 of 15
OMB #0930-0291

Expiration Date: 3/31/2011

The Independent Evaluation of the SAPT BG Program

SAPT Block Grant Survey Prevention and Synar Reviewer Page 1 of 9

Estimates of Burden for the Collection of Information.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of 

information unless it displays a currently valid OMB control number. The OMB control number for this 

project is 0930-0291, which expires March 31, 2011. Public reporting burden for this collection of 

information is estimated to average 60 minutes per response, including the time for reviewing 

instructions, searching existing data sources, gathering and maintaining the data needed, and completing 

and reviewing the collection of information. Send comments regarding this burden estimate or any other 

aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA 

Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

Dear State Prevention Systems Assessment Reviewer:

As you know, the Substance Abuse Prevention and Treatment Block Grant (SAPT BG) Program was 

authorized by Congress to provide funds to States, Territories, and one Indian Tribe for the purpose of 

planning, implementing, and evaluating activities to prevent and treat substance abuse and is the largest 

Federal program dedicated to improving substance abuse prevention and treatment systems. The sponsors 

of the program, the Substance Abuse and Mental Health Services Administration’s Center for Substance 

Abuse Prevention and Center for Substance Abuse Treatment, have contracted with Health Systems 

Research, Inc., a public policy consulting firm in Washington D.C., to conduct an independent evaluation 

of this program. 

We are soliciting feedback about the SAPT BG from key program stakeholders. As a past State 

Prevention and Synar System Reviewer, you have important insights and views about the intent, 

implementation, and impact of the SAPT BG from your site visits to States. We would greatly appreciate 

your assistance with the evaluation through the completion of this survey. Most of the questions are 

closed-ended questions where you will be asked to check the appropriate answer or answers. In addition, 

there are several open-ended questions where you have the opportunity to comment. We urge you to be 

as honest and thorough as possible. 

Please be assured that your answers will be strictly confidential. We will only report aggregated 

responses to the questions, and we will never attribute specific comments to particular individuals. Your 

responses will not be used by CSAP to assess State compliance with the requirements and will not have 

any repercussions for any particular State; they will be used solely for the purpose of evaluating the 

overall SAPT BG Program.

Thank you very much for taking the time to participate. 
OMB #0930-0291

Expiration Date: 3/31/2011

1) What is the purpose of the State Prevention and Synar System Review?

2) In what roles did you serve on the State Prevention and Synar System Reviews? (Check 

all that apply)

State Prevention and Synar System Review Lead

Prevention Consultant

Synar Consultant

Other (please describe)

3) How often does a State representative identify compliance issues for the State 

Prevention and Synar System Reviewers during the visits? 

Never

Rarely

Sometimes

Usually 

Always

Cannot answer

4) When conducting State Prevention and Synar System Review site visits, how often do 

you identify potential issues for Federal action? 

Never

Rarely

Sometimes

Usually 

Always

Cannot answer

SAPT Block Grant Survey Prevention and Synar Reviewer Page 2 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

5) When conducting State Prevention and Synar System Review site visits, how often do 

you identify potential issues for State action? 

Never

Rarely

Sometimes

Usually 

Always

Cannot answer

6) How are issues that require action communicated to CSAP? (Check all that apply)

E-mail contact

Phone contact

In-person meeting

Site visit report

Other (please describe)

7) How are issues that require action communicated to States? (Check all that apply)

E-mail contact

Phone contact

Exit interview at the conclusion of site visit

Site visit report

Other (please describe)

8) What are the strengths of the current State Prevention and Synar System Review 

process? 

9) What are the weaknesses of the current State Prevention and Synar System Review 

process? 

SAPT Block Grant Survey Prevention and Synar Reviewer Page 3 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

10) To what extent do you agree that the State Prevention and Synar System Review site 

visits are useful to States?

Strongly disagree

Somewhat disagree

Neither disagree nor agree

Somewhat agree

Strongly agree

Cannot answer

11) Please list any ways in which you feel the State Prevention and Synar System Review 

site visits could be more useful to States.

12) What types of training, if any, did you receive to prepare you for conducting the State 

Prevention and Synar System Review site visits? (Check all that apply)

In-person training

Written instructions

Shadowing of experienced site visitors

No training received

Other (please describe)

13a) If you received in-person training, please rate how useful you feel the training you 

received was.

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

SAPT Block Grant Survey Prevention and Synar Reviewer Page 4 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

13b) If you received in-person training, how long before the actual site visits began did the 

training occur? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

14a) Do you have any recommendations for improving the training and preparation for the 

site visits to make you a more effective reviewer? 

Yes

No

14b) If yes, what are they?

15a) If you received written instructions, please rate how useful they were.

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

SAPT Block Grant Survey Prevention and Synar Reviewer Page 5 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

15b) If you received written instructions, how long before the actual site visits began did you 

receive them? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

16) How useful was the State Prevention and Synar System Review Guide in helping to 

gather the information needed to prepare the State Prevention and Synar System 

Review site visit report? 

Not useful

A little useful

Somewhat useful

Mostly useful

Extremely useful

Cannot answer

17a) Were you provided any other materials in preparation for the site visits?

Yes

No

17b) If yes, what other materials?

18a) Do you have any suggestions for improving the State Prevention and Synar System 

Review Guides?

Yes

No

SAPT Block Grant Survey Prevention and Synar Reviewer Page 6 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

18b) If yes, please discuss.

19a) Did you receive information about the State’s responsibilities for the State Prevention 

and Synar System Review site visit? 

Yes

No

Cannot answer

19b) If yes, what information did you receive?

20) Please rate how prepared the last State you visited was for the State Prevention and 

Synar System Review site visit.

Very unprepared

Somewhat unprepared

Minimally prepared

Very prepared

Completely prepared

Cannot answer

21) What products result from the State Prevention and Synar System Review site visits to 

States? (Check all that apply)

State Prevention and Synar System Review report and recommendations

Technical assistance plan

Strategic plan

Other (please describe)

SAPT Block Grant Survey Prevention and Synar Reviewer Page 7 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

22) How long after a site visit do you typically submit your drafts of site visit products to 

CSAP? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

23) How long after the submission of the draft site visit report do States typically receive a 

copy of it (after CSAP staff edits are made)? (Check one)

Within 1 week

Within 2-3 weeks

Within 1 month

Within 2 months

Longer than 2 months

Other (please describe)

24a) Do you know if Federal program staff and grants management use site visit products? 

Yes

No

Don't know

24b) If yes, how?

25a) Do you know if States use site visit products? 

Yes

No

Don't know

SAPT Block Grant Survey Prevention and Synar Reviewer Page 8 of 9
OMB #0930-0291

Expiration Date: 3/31/2011

25b) If yes, how?

26a) Do you have any recommendations for improving the dissemination of State Prevention 

and Synar System Review site visit products?

Yes

No

Don't know

26b) If yes, what are they?

27a) Do you have any recommendations for more effective uses of the State Prevention and 

Synar System Review site visit products? 

Yes

No

Don't know

27b) If yes, what are they?

Thank you very much for your participation.

SAPT Block Grant Survey Prevention and Synar Reviewer Page 9 of 9
Appendix E 

Complete Discussion of Evaluation Methods and Limitations
E-2

To collect and analyze information for the independent evaluation of the Substance Abuse 

Prevention and Treatment (SAPT) Block Grant (BG) Program, the following methods were 

developed and implemented:

Review of SAPT BG Program documents. Information related to the achievement of the 17 

legislative goals, Program expenditures, and systems improvement efforts was abstracted from 

States’ Federal Fiscal Year (FFY) 2007 BG applications in the Web-based Block Grant 

Application System (WebBGAS). An abstraction form was developed and used to standardize 

the collection of information about each State’s progress towards achieving the 17 goals. Data 

collection staff were trained in data abstraction processes used during the review of Program 

documents. For each State, a team of two staff members abstracted data individually, then 

compared and combined their abstractions to ensure thorough and accurate information for each 

State. Completed abstractions were reviewed by senior evaluation staff to ensure quality and 

consistent application of data abstraction techniques. Technical Review (TR), State Prevention 

and Synar System Review (SPSSR), and Center for Substance Abuse Treatment (CSAT)/Center 

for Substance Abuse Prevention (CSAP) internal reports were examined for common themes 

across States, which were documented and incorporated into the analysis of TR, SPSSR, and 17-

goal activities. 

Collection of NOMs/TEDS/NSDUH data. SAPT BG regulations require that States collect and 

report data for the National Outcome Measures (NOMs) into WebBGAS. The vast majority of 

the 60 SAPT BG recipients use pre-populated Treatment Episode Data Set (TEDS) data (for 

treatment measures) and National Survey of Drug Use and Health (NSDUH) data (for prevention 

measures) to fulfill the NOMs requirements. The WebBGAS contractor uploads these States’ 

TEDS data into the WebBGAS system. A few BG recipients use their own State-collected data 

to adhere to the NOMs requirements and are required to use TEDS data definitions to ensure 

valid and consistent data collection. States that do not use TEDS data definitions for all measures 

are required to document their data definitions in WebBGAS. In early 2008, the WebBGAS 

contractor provided all of the NOMs data contained in WebBGAS for BG applications for FYs 

2004 to 2008 as well as related documentation to the evaluation team.

Interviews with State staff. In-person interviews were conducted with 21 groups of State staff 

during site visits to State offices in May through July 2008. The 99-question State staff interview 

protocol was initially administered in three States to assess its validity and reliability and to 

determine a timeframe for interview completion. The interview was originally designed to be 

administered via telephone to individual State employees familiar with different areas of BG 

implementation. However, during the initial site visits to test the protocol, it was determined that 

one 3-hour group interview conducted in person would be the best way to ensure accurate and 

thorough responses to interview questions. The group interviews also mitigated the impact of 

State staff turnover on the ability to provide information – in a group setting, there was at least 

one State employee who had been with the State through at least two BG cycles. 

Directors of the Single State Authorities (SSAs) formally responsible for SAPT BG Program 

administration were asked to select State staff who were the most knowledgeable about the 

Program to participate in the interviews. Size of interview groups ranged from 4 to 12 State staff 

members and always included the SSA Director. Roles of other State staff who participated in 
E-3

the group interviews included the Treatment Supervisor/Lead, Prevention Supervisor/Lead, BG 

Coordinator/completer of the SAPT BG application, Lead Data Analyst, Program 

Evaluator/Monitor, and Technical Assistance (TA) and Training Manager. Prior to the site visits, 

data collection team members were provided with thorough training about interview techniques 

and each question in the interview protocol. Site visit teams consisted of two data collection team 

members, at least one of whom was a senior evaluator.

Site visit selection. Due to time constraints and costs associated with site visits, a sample of 21 

SAPT BG recipients was selected for onsite interviews. To narrow the number of potential States 

to be visited, Alaska, Hawaii, and the Territories/Jurisdictions were omitted due to the high costs 

and logistical burden associated with their visits. To select 21 of the remaining States for site 

visits, the States were first separated into five regions: Midwest, Northeast, Southeast, 

Southwest, and West. Each region contained approximately the same number of States. Then

the following criteria were applied for each region to select representative States:

At least one large State with a metropolitan area

At least one rural State with a relatively small population

At least one State with racial/ethnic diversity in population

At least one State where the SAPT BG comprises a large percentage (>60 percent) of 

total State funds for substance abuse services

At least one State where the SAPT BG comprises a small percentage (<40 percent) of 

total State funds for substance abuse services.

At least four States were selected in each region with the exception of the West region, which 

had three States selected as representatives. The West region contains all of the Pacific 

Jurisdictions, Alaska, and Hawaii.

Interviews with Federal staff. Federal staff associated with the SAPT BG Program were 

interviewed individually at their offices by a team of two data collection staff members, at least 

one of whom was a senior evaluator. All State Project Officers (SPOs), Government Project 

Officers (GPOs), and SAPT BG Federal management staff were selected for in-person 

interviews, which occurred in May 2008. The Federal staff interview protocol consisted of 79 

open-ended questions and took an average of 90 minutes to complete. Prior to the site visits, data 

collection team members were provided with thorough training about interview techniques and 

each question in the interview protocol. A total of 28 Federal staff were interviewed for the 

evaluation.

Web-based surveys. To collect information from TR and SPSSR reviewers in a convenient 

manner, a web-based survey and survey platform was developed and launched for the evaluation. 

The web-based survey system contained a user-friendly interface and a secure e-mail function 

that alerted and reminded the reviewers to access the survey. The virtual survey system was 

thoroughly tested to ensure optimal functioning and compliance to SAMHSA DMS-IT and other 

Federal guidelines. A total of 6 TR reviewers completed the 47-question TR survey, and 4 

SPSSR reviewers completed the 27-question SPSSR survey. Each survey contained closed and 

open-ended questions and took less than 1 hour to complete. Multiple attempts were made to 
E-4 

increase the pool of respondents for the survey; however, only 10 reviewer names were provided 

to the web-based survey team. Surveys were completed in June through August of 2008. 

Data storage, cleaning, and analysis. Quantitative data for the evaluation, including NOMs 

data, financial data, and responses to the web-based surveys, initially were stored and cleaned in 

Microsoft Excel® spreadsheets and subsequently imported to SPSS 16.0 software for statistical 

analysis. Frequencies and percentages were calculated for all data, and trend analysis was 

conducted for the NOMs. 

Qualitative data, including interview responses and abstracted program information, initially 

were stored and cleaned in Microsoft Word and subsequently imported to NVivo 7 software for 

content analysis. The data from were analyzed for key themes and differences in responses 

across respondents. Themes were aggregated across all interviews, surveys, and program 

information, and themes mentioned most frequently were reported. Specific qualitative examples 

that illustrated Federal and State accomplishments and outcomes for the SAPT BG Program also 

were stored in NVivo; many are included in this final evaluation report. 
Appendix F 

Secondary Analysis of the National Outcome Measures 

(NOMs) 
F-2

In an effort to address issues of access, capacity, service quality, client satisfaction, and objective 

outcomes, the Substance Abuse and Mental Health Services Administration (SAMHSA)

collaborated with the National Association of State Alcohol/Drug Abuse Directors (NASADAD) 

to develop the National Outcome Measures (NOMs). The goal was to select outcome measures 

that could be utilized to manage and measure performance and to determine whether the 

agency’s vision is being achieved.

1

Specific NOMs were identified by SAMHSA for substance abuse treatment and prevention. 

These measures include the following “domains” or target areas: Abstinence from Alcohol and 

Other Drugs, Employment/Education, Crime and Criminal Justice, Retention, Social 

Support/Social Connectedness, Cost Effectiveness, and Use of Evidence-Based Practices. These 

NOMs are related to services provided to youth aged 12 to 17 and adults aged 18 and older. 

SAMHSA required Block Grant (BG) funding recipients to report data for the NOMs starting in

FY 2008; States that are not in compliance may lose up to 5 percent of their BG funding.

Purpose

The purpose of this analysis was to measure the impact of the Substance Abuse Prevention and 

Treatment (SAPT) BG program on access to services, service quality and client satisfaction, and 

determine the extent to which the program is achieving its goals.

Methodology

Data utilized to pre-populate NOMs were drawn primarily from SAMHSA’s Treatment Episode 

Data Set (TEDS) and the National Survey on Drug Use and Health (NSDUH); these data were 

accessed through the SAPT Web-based Block Grant Application System (WebBGAS).

Treatment Episode Data Set (TEDS). The TEDS is an administrative data system that provides 

descriptive information about admissions to providers of substance abuse treatment. TEDS is 

sponsored by the Office of Applied Studies (OAS) at SAMHSA. The TEDS system includes 

records for approximately 1.5 million substance abuse treatment admissions annually. TEDS 

consists of data that are routinely collected by States in monitoring their individual substance 

abuse treatment systems. In general, facilities reporting TEDS data are those that receive State 

alcohol and/or drug agency funding, including Federal Block Grant funds, for the provision of 

substance abuse treatment. The TEDS system excludes facilities that are operated by private for 

profit agencies, hospitals, and the State correctional system, if they are not licensed through the 

State substance abuse agency.

National Survey of Drug Use and Health (NSDUH). The NSDUH is an annual survey of the 

civilian, noninstitutionalized population of the United States aged 12 or older. This survey is 

planned and managed by SAMHSA’s OAS, and data collection is conducted under a Federal 

contract.2

 

1

https://www.nationaloutcomemeasures.samhsa.gov

2

https://nsduhweb.rti.org/
F-3

The NSDUH is the primary source of information on the use of illicit drugs, alcohol, and tobacco 

in residents of households (e.g., living in houses/townhouses, apartments, condominiums);

persons in noninstitutional group quarters (e.g., shelters, rooming/boarding houses, college 

dormitories, migratory workers' camps, halfway houses); and civilians living on military bases. 

The survey is conducted with approximately 67,500 persons 12 years of age and older each year.

Analysis

Quantitative analysis of secondary data was conducted using the previously described data 

sources. Simple descriptive statistics such as frequencies, percent changes, and averages were 

calculated for each NOM. Percent changes were calculated for NOMs treatment domains to 

understand changes in values from admission to discharge. Calculations were made by 

subtracting the admission value from the discharge value, then dividing by the admission value 

and multiplying the result by 100. Analysis of trends within and across years also was conducted.

Data Definitions and Assumptions

Treatment NOMs are collected using the following TEDS data definitions:

Client: An individual who has an alcohol or drug related problem, has completed the 

screening and intake process, has been formally admitted for treatment or recovery 

service, and has his or her client record

Admission: The first date of service, prior to which no service has been received for 30 

days

Discharge: The last date of service, subsequent to which no service has been received for 

30 days.

Data for the criminal justice involvement domain is based on clients arrested for any 

charge in the past 30 days at admission versus discharge. The question to address this 

data element in FY 2007 was framed as: “What is your current involvement with the 

legal system (police, court or jail)?” Possible responses included:

o In lock-up facility, mandatory hospitalization involuntary commitment, or youth 

facility

o On probation or parole, felony charges pending or conviction, awaiting 

sentencing, in a half-way house, contested divorce or custody issues

o Misdemeanor charges pending or conviction, court-ordered outpatient treatment, 

in detention

o Non-criminal problems, informal probation, truancy, minor litigation, mutually 

agreeable divorce/custody issues, no threat of jail

o No legal involvement at all.

The employment and school attendance domain was developed to measure the 

employment status of persons treated in the States’ substance abuse treatment systems. 

This domain is measured by the change in all clients receiving treatment who reported 

being employed (including part-time) at admission and discharge.
F-4

The question phrased in FY 2007 was a measure of productive use of time, not paid 

employment: “How often do you do activities such as hunting, fishing, berry picking, 

work, school, sports, or treatment activities?” A traditional paid employment question 

was added in FY 2008.

Limitations

The following limitations to TEDS and NSDUH data collection and analysis should be taken into 

account before drawing conclusions about outcomes:

TEDS

The number and client mix of TEDS admissions does not represent the total demand for 

substance abuse treatment nor the prevalence of substance abuse in the general 

population.

Because States were not required to report on NOMs prior to FY 2008, data for several 

domains are missing from FY 2004 to FY 2006.

It is impossible to determine the reasons why individuals were discharged from 

treatment. Discharges include program completers, but also include people who leave 

against medical advice and people who leave for personal or family reasons. If data were 

collected and analyzed with respect to length of stay or degree of program completion, it 

is likely that client progress would be more pronounced for program completers and nearcompleters.

Currently, when a client moves from one treatment modality to another, for the purposes 

of TEDS collection, he/she is discharged from the first modality and “newly” admitted to 

the next one. This makes the average percent change from admission to discharge smaller 

than it would be if admission were considered to be the beginning of service provision 

and discharge were the conclusion of treatment services, regardless of modality.

NSDUH

The NSDUH is utilized as a data source to populate a number of NOMs prevention 

domains. The main limitations of using NSDUH data to determine prevention outcomes 

are small sample sizes in many States, which lead to underreporting and under coverage 

of some populations, and the inability of the survey to identify participants in BG-funded 

prevention programs. This precludes drawing the conclusion that NOMs changes are 

associated with participation in BG-funded prevention services and activities.

Across different measures, years, and States, a large number of data fields were missing.

From FY 2004 to FY 2008, NSDUH data were collected differently for some prevention 

domains, including perceptions of risk or harm from substance use and attitudes about 

substance use at work or school. Therefore, it was not appropriate to conduct an analysis

of these outcomes across years.