# Presentation: 671732

## How CSAP’s Service to Science Initiative Improves:The Evaluation Capacity of Promising Interventions:

**National Prevention Network****21****st**** Annual Prevention Research Conference****Prevention Research: Driving Successful Outcomes**
**August 25, 2008, Indianapolis, Indiana**

- How CSAP’s Service to Science Initiative Improves:The Evaluation Capacity of Promising Interventions:
- A View from Provider and Participant Perspectives

## Multiple perspectives about STS

**Multiple perspectives about STS**

**Federal Funder / Planner**
- Carol McHale, SAMHSA/CSAP
- 				**CAPT Evaluator**
- Wayne Harding, NECAPT
**STS Clients & Their CAPT TA Provider**
*Zero Adult Provider Coalition*
- Robyn Weisman, Director/Developer
- Rebecca Sechrist, Evaluator
- Mary Ellen Murphy, CAPT TA Provider
*Youth Action for Prevention Program*
- Marlene Berg, Director/Developer
- Emil Cohen, Evaluator
- Kim Dash, CAPT TA Provider

## Agenda

**Agenda**

- Overview of Service to Science (Carol McHale)
- Characteristics of Participant Programs and Program Outcomes (Wayne Harding)
- Brief Descriptions of the Programs and their Experience with STS (*Zero Adult Provider Coalition, Youth Action for Prevention Program)*
- Discussion with You (Moderated by Wayne Harding)

## Overview of Service to ScienceInitiative

## SAMHSA’s Strategic Prevention Framework Steps

- Assessment

- Implementation

- Sustainability &
- Cultural Competence

**Profile population needs, resources, and readiness to address needs and gaps**

**Monitor, evaluate, sustain, and improve or replace those that fail**

**Implement evidence-based prevention programs and activities**

**Develop a Comprehensive Strategic Plan**

**Mobilize and/or build capacity to address needs**

- Capacity

- Planning

- Evaluation

**SAMHSA’s ****Strategic Prevention Framework Steps**

## Service-to-Science Goals

**Service-****to-Science Goals**

- To help innovative, field-grown prevention interventions that address substance abuse (or related issues) develop and document evidence of their effectiveness.
- To increase the pool of evidence-based
- 	interventions.

## Guiding Principles for  Service-to-Science 

**Guiding Principles for **** Service-****to-Science **

- Meet programs “where they are” along a continuum of evidence of effectiveness.
- Participating programs have a common direction but different final destinations.

## Service-to-Science Activities

**Service-****to-Science Activities**

- CAPTs solicit prevention program nominations from State prevention officials.
- CAPTs assess the readiness of candidate programs.
- CAPTs conduct regional Service to Science Academies.
- CAPTs provide follow-up technical assistance.

## Service-to-Science Activities (Continued)

**Service-****to-Science Activities ****(Continued)**

- Service to Science Academy participants – eligible to compete for “mini-subcontracts” to further enhance evaluation capacity through the CRP, Inc. Service to Science Contract.
- CAPTs provide limited technical assistance post “mini-subcontract” award.

## Service-to-Science Activities (Continued)

**Service-****to-Science Activities ****(Continued)**

- Beginning in 2008, Service to Science has enhanced its outreach activities to:
- 	 1. Pacific Jurisdictions
- 2. Native Alaskans
- 3. American Indians, with CSAP’s   Native American Center for Excellence (NACE)

## Number of Programs Served Through Service to Science 

**Number of Programs Served Through Service to Science **

- As of June 2008, the CAPTs had provided 239 prevention programs with technical assistance.
- Since 2006, 79 programs have received subcontracts of up to $30,000.

## Defining Features of Service to Science TA

**Defining Features of Service to Science TA**

- Customized
- Long-term
- Relationship-based
- Flexible

## SAMHSA/CSAP’s Service-to-Science Initiative 

**SAMHSA/CSAP’s ****Service-to-Science Initiative**** **

**Published in**
**Peer Reviewed**
**Journal**

**“****Documented**
**Effectiveness”**

**Evidence-Based Review/Rating Processes**

**Applicant**
**Pool**

- CAPTs
- Regional
- S-S
- Academies
- and
- Follow-up
- Technical Assistance

- CAPTs Support
- “Mini
- Subcontracts”
- Evaluation
- Capacity
- (1 year)

- CAPTs
- Technical Assistance
- Post Award Follow-up

**SAMHSA’s**
**NREPP**

**Other Registries & Federal**
**Lists**

- State Prevention Officials
- (National Prevention Network)
**Nomination and Recruitment**

- Centers for the
- Application of Prevention
- Technologies
- (CAPTs)
**Pre-assessment**

## Selected Evaluation Findings 

## Methods

**Methods**

- Pilot study (2006) interviews and/or online survey with program Directors.  Target sample of 79 programs; 58 (73%) responded.
- Six case studies of 2005 STS participants and extended follow-up case studies on 5 that received mini-subcontracts. Six case studies of 2006 participants.
- 2007 online survey targeted 142 programs that had been participants for at least 15 months.  93 responded (a 77% rate after removing 21 programs for which respondents were no longer available).

## Participant ProgramCharacteristics 

## Geographic Distribution (N=230)

**Geographic Distribution ****(N=230)**

## Program Type (IOM classification)

**Program Type (IOM classification)**

|  | Percent (a) |
| --- | --- |
|  Universal | 52% |
|  Selected  | 51% |
|  Indicated | 22% |

- (a) Multiple responses allowed. N=93.

## Program Duration and FTEs

**Program Duration and FTEs**

|  | Length of Program Existence(a) | Number of 
Full-Time Employees(b) |
| --- | --- | --- |
|  Mean | 10.6 years | 4.6 FTEs |
|  Median | 8 | 3 |
|  Range | 2 - 37 | 0 - 47 |

- (a) N=79, (b) N=89

## Primary Behaviors Targeted for Change

**Primary Behaviors Targeted for Change**

|  | Percent (a) |
| --- | --- |
|  Alcohol Use/Abuse | 80% |
|  Illicit Drug Use/Abuse | 60% |
|  Tobacco Use | 58% |
|  Violence (including bullying) |  37% |
|  HIV/AIDS |   8% |

- (a) Multiple responses allowed. N=93.

## Ages of Population(s) Served

**Ages of Population(s) Served**

|  | Percent (a) |
| --- | --- |
|  Adolescents (ages 13-17) | 71% |
|  Childhood (ages 6-12) |  51% |
|  Young Adults (ages 18-25) |  19% |
|  Early Childhood (ages 0-5) |   10% |
|  Adults (ages 26-55) |   10% |
|  Older Adults (ages >55) |    7% |
|  No Distinction by Age |    3% |

- (a) Multiple responses allowed. N=93.

## Race/Ethnicity of Population(s) Served

**Race/Ethnicity of Population(s) Served**

|  | Percent (a) |
| --- | --- |
|  No Distinction | 85% |
|  Hispanic or Latino |   9% |
|  American Indian or Native Alaskan |   5% |
|  Black or African American |   4% |
|  White |   4% |
|  Asian American |   3% |
|  Native Hawaiian or Other Pacific Islander |   3% |

- (a) Multiple responses allowed. N=93.

## Strategies Used

**Strategies Used**

|                                      Percent (a)                      Percent |  |  |  |
| --- | --- | --- | --- |
|  Improve Knowledge /   Awareness | 90% |  Mobilize Community | 27% |
|  Improve Skills | 85% |  Improve Identification and   Referral | 25% |
|  Healthy Alternatives | 79% |  Improve Policies | 17% |
|  Improve Norms | 59% |  Improve Laws | 12% |
|  Improve Mental / Physical Health | 54% | Improve Access to and   Quality of Care | 11% |
|  Build Collaboration | 36% |  Improve Enforcement |   8% |

- (a) Multiple responses allowed. N=93.

## Evaluation Design Prior to STS

**Evaluation Design Prior to STS**

|  | Percent (a) |
| --- | --- |
|  No Formal Evaluation | 13% |
|  Posttest Only Single Group |    4% |
|  Pretest-Posttest Single Group |   42% |
|  Pretest-Posttest with Comparison Group |   14% |
|  Pretest-Posttest with Control Group |     9% |
|  Pretest-Posttest with Control Group and Replication(s) Evaluated by the Program |     5% |
|  Pretest-Posttest with Control Group and Replication(s) Evaluated by Independent |     4% |
|  Other / Unsure |     9% |

- (a) Multiple responses allowed. N=93.

## Service To ScienceOutcomes 

## Value of STS Services

**Value of STS Services**

|  | Percent (a) |
| --- | --- |
|  Satisfied with CAPT TA(Very Satisfied or Somewhat Satisfied vs. Somewhat Dissatisfied or Very Dissatisfied) | 72% |
|  CAPT TA Useful (A Little, Somewhat, or Very Useful vs. Not at All Useful) | 91% |
|  Would Recommend STS to Similar   Programs                                         (Definitely Yes or Probably Yes vs. Probably No or Definitely No)                                                                        | 91% |

- (a) N=93.

## Factors That Facilitated Progress

**Factors That Facilitated Progress**

-    When asked to discuss factors that facilitated progress on their evaluation the largest percentage of clients (79%) identified TA from the CAPTs.

- (a) From the 2006 pilot study. N=58.

## 5 Most Common TA Issues

**5 Most Common TA Issues**

|  | Percent (a) |
| --- | --- |
|  Improving understanding of  program rating systems such as NREPP and NASADAD | 87% |
|  Improving understanding of evaluation | 83% |
|  Developing and/or improving a logic model | 76% |
|  Developing and/or implementing a more rigorous evaluation design | 72% |
|  Improving organizational commitment to evaluation | 68% |

- (a) Multiple responses allowed. N=93.

## TA Contributed to Program Capacity(a) in 5 Most Common Issues

**TA Contributed to Program Capacity****(a)**** in 5 Most Common Issues**

|  | Percent (b) |
| --- | --- |
|  Improving understanding of  program rating systems such as NREPP & NASADAD | 95% |
|  Improving understanding of evaluation | 95% |
|  Developing and/or improving a logic model | 92% |
|  Developing and/or implementing a more rigorous evaluation design | 95% |
|  Improving organizational commitment to evaluation | 84% |

- (a) Contributed “A Little,” “Somewhat,” or “A Great Deal” vs. “Did Not Contribute at All.”  (b)  N=93.

## Plan To Seek Recognition from NREPP or NASADAD

**Plan To Seek Recognition from NREPP or NASADAD**

|  | Percent (a) |
| --- | --- |
|  Have not considered seeking formal recognition  | 15% |
|  Considered but decided not to seek formal recognition  | 28% |
|  Plans to seek or are seeking formal recognition  | 57% |

- (a) N=91, 2 programs that sought recognition prior to STS assigned to missing