WEBINAR #5: Student Assistance Programs (SAP):

Program Evaluation and Improvement

“Talk. They Hear You.”® Campaign SAP Webinar Series

March 30, 2022, 3 p.m. ET
2

Featured Experts

Robert M. Vincent, M.S.Ed.

Associate Administrator for Alcohol 

Prevention and Treatment Policy

SAMHSA

Ellen Morehouse, 

LCSW, CASAC, CPP

Executive Director

Student Assistance 

Services Corporation

Perri Rosen, Ph.D., NCSP

Consulting Psychologist 

Pennsylvania Office of Mental 

Health and Substance Abuse 

Services

Michael L. Dennis, Ph.D.

Lighthouse Institute Director and 

Senior Research Psychologist

Chestnut Health Systems
3

Featured Experts

Emily M. Eckert, LPCC-S

Assistant Director, Office 

of Whole Child Supports 

Ohio Department of 

Education

Valerie Connolly-Leach, OCPS

Behavioral Health/Prevention 

Administrator, Office of Prevention 

and Wellness

Ohio Department of Mental 

Health & Addiction Services

Sara Ellsworth

Regional Administrator, 

Behavioral Health and 

Student Supports

Capital Region ESD 113,

Washington

Amity Noltemeyer, Ph.D., 

NCSP

Associate Dean of the 

Graduate School, Professor of 

Educational Psychology 

Miami University

Erin L. Wick, M.B.A., SUDP

Senior Director, Behavioral Health 

and Student Supports 

Capital Region ESD 113,

Washington
4

Today’s Session Objectives

Today’s session will cover:

• Program evaluation best practices

• Program performance and monitoring examples

• Making improvements to SAPs
5

SAP Webinar Series Overview

 Webinar #1: Student Assistance Programs

 Webinar #2: Standing Up SAP Services

 Webinar #3: Assessing Needs and Building Capacity

 Webinar #4: Planning and Implementation

• Webinar #5: Program Evaluation and Improvement
6

Polling Questions

This presentation includes five short polls to help us learn more 

about today’s webinar participants.

• All answers will be presented as totals or percentages, and

no one will be able to tell how specific individuals

answered.

• By answering the questions, you agree to have your

answers anonymously included in the totals as

percentages.
SAMHSA Strategic Prevention Framework: 5-Step Planning Process

There are many ways to get 

started depending on your 

existing infrastructure, 

resources, need, etc. 

7
8

Review: 9 Critical Components of Effective SAPs

1. School Board Policy

2. Staff Development

3. Program Awareness

4. Internal Referral Process

5. Problem-solving Team and Case Management

6. Direct Services to Students

7. Cooperation and Collaboration

8. Integration with Other School-based Programs

9. Program Evaluation and Improvement
9

Difference Between Program Management & Evaluation

• Goals of Program Management: To use limited 

resources to effectively implement and sustain

program components that achieve agency goals.

• Goals of Program Evaluation: To provide useful 

feedback about the program to a variety of 

audiences. Most often, feedback is perceived as 

useful if it aids in decision-making.
10

Common Questions in Evaluation

• Who is being served, and what are their needs? 

• How are services triaged or matched to those in need? 

• To what extent is a process or intervention being implemented as planned? 

• Is there evidence of health disparities, service gaps, or policies that can be 

addressed to improve implementation? 

• How is the degree of implementation related to outcomes? 

• What does the intervention cost to implement well? 

• Which of two or more approaches are more effective? 

• Which of two or more approaches are more cost-effective? 
11

Needs Assessment and Approach Selection

• What are the needs of the target population? 

• How are services being targeted by need and 

severity? 

– Who needs suicide prevention? (someone who is suicidal) 

– Who needs prevention, early intervention, or treatment for 

alcohol use? (frequency/frequency of use, substance use disorder)

• How well targeted are the services provided/planned to the needs of the students or 

other populations? 

– What are the “evidence-based” practices that have worked better than average before? 

– To what degree are services already targeted at this population in terms of intellectual 

development, demographics, or other context?

• Given constraints of the policy, funding, staffing, and school setting, what is feasible? 

– Which approach is more likely to be implemented well? 
12

Implementation/Process Improvement Evaluation

• How well is an approach, process, or intervention being implemented? 

– Can you make a service cascade of the desired steps and identify where there are 

large gaps or dropoffs? 

• Examine potential reasons for gaps/dropoffs like:

– Check for health disparities by gender, ethnicity,

race, and age.

– Check for variation by site or clinical profile.

– Identify long gaps between steps.

– Check for policies that may limit or delay access to services. 

– Check for possible misfits between original materials with the population or context.

• Examine whether the degree of implementation is related to outcomes:

– Test the presumed causal mechanism.

• Consider possible revisions to the original needs assessment.
13

Comparative Effectiveness and Cost-Effectiveness

• Only of interest when there is “uncertainty” about which 

of two or more approaches/interventions is more effective 

or cost-effective. 

• Only of value if the intervention was implemented well 

and outcomes/costs were reliably measured.

• Can compare them on the basis of changes over time, matching, analysis of 

covariance, needs-based assignment (aka regression discontinuity), and 

random assignment.

• Costs may vary by perspective (to program, to school, to society).

• Cost-effectiveness usually focuses on the cost per the main clinical outcome(s).
14

Student Assistance Services Corp., New York
15

Evaluation to Demonstrate Effectiveness for 

Program Sustainability

Monthly, Semi-annual, and Annual Monitoring

• Sources of referrals

Students, parents, administrators, faculty, support staff, and community members

• Numbers of students seen

Individual, group, and family sessions

• Consultations and meetings with school staff and community providers

• Referrals for additional services

Private practitioners, mental health services, substance use disorder treatment, 

relationship abuse, etc.

• Schoolwide awareness activities

Drug Facts Week, Great American Smokeout, National COA Awareness Week, etc.
16

Evaluation to Demonstrate Effectiveness for 

Program Sustainability

Semi-annual and Annual Monitoring

• Demographics

• Identified issues

Annual Monitoring: Pre/Post

• Perceived risk of harm

• Maintenance of no use

• 30-day use

• Disciplinary actions

• Bonding to school
17

Evaluation Challenges

• Multi-component program

• Workforce

• Situations (COVID-19, natural disasters, other disasters)

• Experimental design

Evaluation to Demonstrate Effectiveness for 

Program Sustainability
2018 –19 2019–20 2020–21 2021–22 Control 2021–22 Significance

Alcohol 1.9–2.2 1.8–2.2 1.97–2.21 2.25 1.92 < .001

Marijuana 1.8–2.1 1.8–2.1 2.00–2.14 2.15 2.05 .064

Rx Drugs 2.4–2.5 2.4–2.6 2.56–2.64 2.65 2.34 < .001

Binge Drinking 2.5–2.7 2.5–2.7 2.59–2.70 2.70 2.51 < .001

Mean Pre/Post Change in Perception of Risk: 

Moderate Risk (2) Great Risk (3)

18
• Conduct focus groups.

• Train staff to increase effectiveness.

• Tailor materials to different populations.

• Use commercial marketing strategies.

19
20

Student Assistance Program, Pennsylvania
Program Evaluation Efforts — Pennsylvania

Team Level

• Implementation 

and fidelity 

checklists

• Satisfaction surveys

• Team member 

survey for 

improvement of 

team functioning

County Level

• County 

coordination 

meetings

• Cross-system 

information-sharing 

and problemsolving

• Multiple sources of 

data, including SAPrelated data and 

other data

Statewide

• School data reports

• Agency data reports

• Monitoring of SAP 

trainers

• SAP team and 

liaison agency 

surveys

• Academic 

partnership to 

evaluate trends

21
Data and Reporting — Pennsylvania

PDE4092 (School Report)

• Reported by school entities to the 

Pennsylvania Department of Education 

(PDE) annually

• Data provided:

– Demographics

– Referral reason

– Recommendations for school and/

or community services

– Access to school and/or community 

services

– Factors that impacted referral outcomes

Joint Quarterly Reporting System (Agency 

Report)

• Reported by counties to the Pennsylvania 

Department of Human Services and the 

Department of Drug and Alcohol Programs 

quarterly

• Data provided:

– Participation in team meetings, parent 

meetings, and consultations

– Demographics for screenings and/or 

assessments 

• Number of students screened or 

assessed for suicide risk

– Outcomes for screenings and/or assessments

22
Pennsylvania Statewide Data Infographics

23
SAP Evaluation with Penn State PRO Wellness

Collaborative analysis of SAP data to:

1) Identify trends in referral data

2) Highlight program successes and

challenges

3) Disseminate findings to

Pennsylvania schools and

communities

24
Pennsylvania Network for Student Assistance Services 

(PNSAS) website: https://pnsas.org/ 

Additional PA Data and Evaluation Resources: 

•

Implementation Checklist for New SAP Teams

• SAP Best Practice Guidelines for Fidelity

• SAP Satisfaction Surveys - for staff, students, and

parents/caregivers

• PNSAS Data - infographics, data summaries, and data

reporting forms

Substance Abuse and Mental Health 

Services Administration 

Pennsylvania SAP Resources

25
Ohio School Wellness Initiative

26
Program Evaluation Example

Goal→ Schools should plan for evaluating, 

sustaining, and innovating their SAP practices. 

Implementation of SAP will be less impactful 

when plans have not been made to ensure that 

its long-term goals can be met through its 

continuation and improvement in the face of 

potential changes and obstacles.

27
Framework for Evaluation and Improvement

28
Program Evaluation

Best Practices

• The SAP has a formal process for 

reviewing and evaluating practices 

and procedures.

• SAP satisfaction and stakeholder 

input information is solicited and 

utilized.

• Program evaluation includes input

from administrators, staff, 

students, parents, and community 

members. Priorities?

Area for 

improvement?

Going well?

Self-Assessment

29
• Develop and implement a clear evaluation plan from the outset.

• Evaluate process variables (e.g., implementation) and outcome variables (e.g., 

mental health and substance use) (Virginia Department of Education, 2013).

• Obtain feedback and data from diverse stakeholders, using multiple methods of 

data collection.

• Common evaluation questions (Center on Positive Behavioral Interventions and 

Supports, 2020):

Plan and Implement

30
Gaining quick knowledge gives you the ability to pivot your efforts (when things are not working) 

and dig deeper (when things are working) to increase the positive outcomes expenienced by your 

school community. This is especially important when we consider the mental health and behavioral 

health concerns facing our students. 

Determine 

priorities shared 

across stakeholder 

Determine 

improvements to be 

made 

Objective data should 

be given to staff when 

necessary with helping 

the student 

groups

Use evaluation 

data to inform 

decision-making 

Schools. courts. and social 

------1 service agencies should develop 

policies that allow them to 

exchange sensitive information 

while respecting the needs and 

rights of their students 

Substance Abuse and Mental Health 

Services Administration 

Share, Reflect, and Improve

31
Washington State Evaluation System — ESD-113

32
33

Goals:

• Monitor the effectiveness and quality of programs. 

• Ensure compliance with state and federal regulations.

• Identify service gaps and prioritize improvements.

• Inform training and education for staff.

Program Evaluation and Improvement
Program Evaluation Example — ESD-113

34
Washington State Evaluation System — ESD-113

Universal Activity Reports

• Prevention Activities Summary

• Prevention Activities Summary for OSPI

• Groups Conducted Summary

Indicated Student Reports

• Service Summary

• Student Progress

• Student Outcomes

• Substance Use Change

• Substance Use Change Categories

• Case Management Referral Status by 

Agency

Data Monitoring Reports

• Quality Control

• Quality Control Summary

• Program Evaluation Form Status

• Follow-up (grades) Status Report

• Other Peer Support Group Referrals

Staffing Reports

• Staff Listing Report

• Staffing Summary Report

• School List Report

35
SAP Services Impact

Student Behaviors

• Decreased substance use

• Alcohol—average 37% reduction

• Binge drinking—average 44% reduction

• Marijuana—average 41% reduction

• Decreased problem behaviors

• School suspensions—average 38% reduction

• Arrests—average 29% reduction

36
SAP Services Impact at Capital Region ESD 113

Student Behaviors

• Increased school attendance

– Average of 39% of participants reported they were 

more likely to attend school because of SAP services

• Satisfaction with services

– Average of 92% reported they were glad the 

participated in SAP services

37
38

Program Improvement Strategies

• Develop and improve outreach efforts.

• Monitor data collection.

• Consistently share data/program outcomes with 

stakeholders.

• Promote staff training and education. 

• Implement evidence-based practices. 
39

Q&A
40

Student Assistance: A Guide for School Administrators

Purpose: This document provides school leaders 

and administrators with key information about 

student assistance services for substance use and 

mental health initiatives.
41

Student Assistance Resources Guide

Purpose: This document describes some student 

assistance resources and provides tangible ways 

school administrators and student assistance 

teams can implement the strategies.

It also assists school leaders and student 

assistance teams with increasing their awareness 

of student assistance services by providing 

guidance on resources available for this critically 

important work.
42

Student Assistance Discussion Starter Video

Purpose: This video describes some student assistance resources and provides tangible 

ways school administrators and student assistance teams can implement the strategies.

It also assists school leaders and student assistance teams with increasing their awareness 

of student assistance services by providing guidance on resources available for this 

critically important work.
43

SAP Webinars 1–4

Access and view previous SAP webinars!

Webinar #1: Student Assistance Programs

Webinar #2: Standing Up SAP Services

Webinar #3: Assessing Needs and Building Capacity

Webinar #4: Planning and Implementation

“Talk. They Hear You.”® Campaign SAP Resources
44

Thank You

Sign up for the monthly

“Talk. They Hear You.” campaign 

e-newsletter!

• Pre-crafted social media messages and 

graphics

• Featured resources and prevention tips

• Campaign news

• Partner spotlights

http://eepurl.com/grCRjD

Engage with SAMHSA on 

social media:

@samhsa

@samhsagov

Substance Abuse and Mental 

Health Services Administration 

(SAMHSA)

@samhsagov

SAMHSA