# Presentation: 089358

## Methods and Tools for 
Self-Evaluation and Accountability

- Self-Evaluation and Accountability

- Abraham Wandersman, Ph.D., *University of South Carolina*
- Pamela Imm, Ph.D., *University of South Carolina*
- Matthew Chinman, Ph.D., *The Consultation Center, *
*	**Yale University School of Medicine*

## What is Getting To Outcomes?

- A system based on ten empowerment evaluation and accountability questions that contain elements of successful programming

- A.1

## Step 1: Identify Goals and Desired Outcomes (PPIII)

*Program Type*
***Part A: Make a list of the primary goals of the program.***
- Ask yourself: “What were we trying to accomplish?”  Of the goals listed, check the ones that apply to your program and add any others on the lines provided.
- _______            _________________________________ _______            _________________________________ _______            _________________________________ _______            _________________________________ _______            _________________________________ _______            _________________________________

- A.2

## PP III Step 1: Identify Goals and Desired Outcomes

- What you want to accomplish? (Goals)
- For whom? (Target Population)
- What are the desired outcomes? (e.g., changes)

- A.3

## PP III Step 2: Process Assessment

- What you actually are doing?
- How frequently program activities take place?
- What target group you are reaching/ Who are you missing?
- What feedback from participants can be used to improve the program in the future?

- A.4

## PP III Step 3: Outcome Assessment

- Select appropriate measures of goal attainment
- Administer the measure preferably before and after the program to determine the amount of change in the target population due to the intervention

- A.5

## PP III Step 4: Impact Assessment

- Measure community indicators (e.g., youth AOD use, DUI arrests, AOD-related crashes
- Preferably use a comparison community

- A.6

## United WaySummary of Program Outcome Model

- INPUTS

- ACTIVITIES

- OUTPUTS

- OUTCOMES

- Resources dedicated to or consumed by the program

- What the program does with the inputs to fulfill its mission

- The direct products of program activities

- Benefits for participants during and after program activities

## Getting To Outcomes

**Tools for Bridging the Gap Between Science and Practice**
- Manual & Training
- Draft Version
- Final Version
- Website

- A.7

## Accountability

- Defined as...
***The systematic inclusion of critical elements of program planning, implementation, and evaluation in order to achieve results.***

- A.8

## Features of the Getting To Outcomes Accountability System

- 1. A comprehensive process that systematically incorporates the critical elements of effective programming.

- A.9

## Features of the Getting To Outcomes Accountability System

- 2. _**Getting To Outcomes**_ can be useful at any stage of your work.

- A.10

## Features of the Getting To Outcomes Accountability System

- 3.  _**Getting To Outcomes**_ promotes cultural competence in programming.
- Planning Stage
- Implementation Stage
- Evaluation Stage

- A.11

## Features of the Getting To Outcomes Accountability System

- 4.  _**Getting To Outcomes**_ encourages the use of logic models to ensure a conceptual link between the identified problem and the potential solutions.

- A.12

## Drug-Free Communities Program Evaluation Framework

- Baseline		Activities(Processes)	     Outcomes	          Impact

- Goal:
- Reduction in Substance Abuse

- Community Risk and Protective Factors

- Incidence and Prevalence of Substance Abuse

- Enhanced/ Strengthened Coalitions:
- Increased Coalition Capabilities
- Increased Citizen Participation
- Increased Interagency and Inter-governmental Collaboration
- Collaborative, Data-Driven Prevention Planning

- Enhanced Prevention Efforts:
- Comprehensive, Research-Based Strategies that Address Identified Needs

- Improved Immediate Outcomes of Prevention Efforts

- Reduced Risk Factors and Increased Protective Factors

- Reduced Substance Abuse

- Contextual/Environmental Factors

## Target Audience

- The _**Getting to Outcomes**_ system will be useful to many:
- Direct Service Providers
- Program Directors
- State-level Personnel
- Evaluators
- Funders

- A.13

## General format for each question in Getting To Outcomes

- Definition of the Question’s Program Element
- Importance of the Question
- Addressing the Question
- A Checklist for each Question

- A.14

## Empowerment Evaluation and Accountability Questions

- A.15

## The Getting To Outcomes Process

- #1
- Needs/
- Resources

- #2
- Goals

- #3
- Best
- Practices

- #4
- Fit

- #5
- Plan

- #6
- Capacities

- #7
- Process
- Evaluation

- #8
- Outcome
- Evaluation

- #9
- Improve

- #10
- Sustain

- A.16

## The Program’s Logic Model

- The Comprehensive
- Definition and Framing
- of the Issue/Problem
- or Condition

- The Goals and
- Objectives

- The Implementation
- Plan

- The Program’s
- Prevention and/or
- Intervention
- Strategy

- Measurable Indicators
- Of Immediate,
- Short Term and Long
- Term Outcomes

- The Organizational
- Capacities

- A.17

## Logic Model

- Defined as...
*A series of connections that link problems and/or needs with the actions taken to achieve goals.*

- A.18

## Accountability Question 1 NEEDS/RESOURCES

- What are the underlying needs and conditions that must be addressed?

- 1.1

## Needs Assessment

- Defined as...
* **A systematic process of gathering information about the current conditions of a targeted area that underlie the “need” for an intervention.*

- 1.2

## Why Conduct a Needs/Resource Assessment?

- Conducting a needs and resources assessment begins the accountability process by determining what objective data is available to prioritize the needs to be addressed.

- 1.3

## Purposes of a Needs/Resource  Assessment

- _Once these factors are identified:_
- Programs can be designed and implemented to address these factors.
- Programs can be implemented for special groups or populations that have special circumstances.

- 1.4

## Needs Assessment (cont.)

- _If done well, a needs assessment can identify_:
- Baseline data showing particular problems/needs
- Where (e.g., school, neighborhood) the particular problem(s) is most prevalent
- For which group of people (potential target population) the particular problem is most prevalent

- 1.5

## Needs Assessment Methods

- Community Forums and Hearings
- Case Studies
- Social Indicators

- 1.6

## Needs Assessment Methods (cont.)

- Service Provider Survey
- Key Informant Survey
- Target Population Surveys

- 1.7

## Why Risk/Protective Factor Model?

- Individual level of risk and protection make a difference
- The more risk factors present, the greater chance of problems
- The more protective factors, the less likelihood of problem behaviors

- 1.8

## Why Risk/Protective Factor Model?(cont.)

- Risk and protective factors predict diverse behavior problems
- Risk and protective factors show consistency across race and cultures

- 1.9

## Risk Factors: Domains

- Individual/Peer Risk Factors
- Family Risk Factors
- School Risk Factors
- Community Risk Factors

- 1.10

## Individual/Peer Risk Factors

- Rebelliousness
- Friends who engage in the problem behavior
- Favorable attitudes toward the problem behavior
- Early initiation of the problem behavior

- 1.11

## School Risk Factors

- Early and persistent antisocial behavior
- Academic failure beginning in early elementary school
- Low commitment to school

- 1.12

## Family Risk Factors

- A family history of high-risk behavior
- Family management problems
- Family conflict
- Favorable parental attitudes and involvement in the problem behavior

- 1.13

## Community Risk Factors

- Availability of drugs
- Community laws and norms favorable toward drug use
- Transition and mobility
- Low neighborhood attachment and community disorganization
- Extreme economic and social deprivation

- 1.14

## Protective Factors

- Opportunities for prosocial involvement
- Rewards/recognition for prosocial involvement
- Healthy beliefs and clear standards for behavior
- Bonding
- Skill-building

- 1.15

## Accountability Question 1 Checklist

*Make sure that you have...*
- Selected a target area in which to do a needs assessment
- Examined rates of alcohol and other drug abuse related incidents in your target area
- Clearly identified a potential target population from within the target area whose behavior needs to be changed

- 1.16

## Accountability Question 1 Checklist (cont.)

- Compiled baseline data for the target population and a comparison population (if available)
- Clearly articulated the causes and underlying risk factors within your target area showing the factors most likely contributing to the problem
- Assessed the risk and protective factors of participants in the area
- Conducted a resource or asset assessment

- 1.17

## WINNERS

- Using the _Getting To Outcomes_ Accountability System

- 1.18

## Analysis of existing (archival) data

- Decline in overall grade point average for 6th and 7th graders
- Increased rates of disciplinary referrals
- 40% of students come from single parent families or have four or more siblings
- 50% of school-aged children living at or below the poverty level & 70% receiving subsidized lunches

- 1.19

## Community Town Meeting (Forum)

- Parents care deeply about their children and are feeling overwhelmed by various challenges
- The welfare to work initiative has placed many of the parents in jobs resulting in less structure/supervision for the children
- Parents noted that their kids had little contact with adults, especially male, role models

- 1.20

## Parental Survey

- Parents reported subjective feelings of being overwhelmed
- Parents reported many financial problems
- Parents reported concern about their childrens’ negative behavior (e.g., stealing, lying, fighting)
- Parents reported suspicions of alcohol and other drug use
- Increasing rates of smoking were also mentioned

- 1.21

## Resource Assessment

- Schools
- Community
- Public/Private Partnerships

- 1.22

## Accountability Question 2 GOALS

- What are the goals, target population, and objectives (i.e., desired outcomes)?

- 2.1

## 2.1A

## Goals

- Defined as...
*Broad, measurable statements that describe the desired longer-term impacts of what you want to accomplish*

- 2.2

## Target Population

- Defined as...
*Participants your program is designed to change*

- 2.3

## Objectives (i.e., Desired Outcomes)

- Defined as...
*The specific changes expected in your target population(s) as a result of your program*

- 2.4

## Goal Statements

- Describe, using summary goal statements, the long-term impacts you wish to accomplish
**1____________________________**
**2____________________________**
**3____________________________**

- 2.5

## Objectives

- Develop outcomes statements that address the following
**What **will change?**_____________________**
- For **whom?__________________________**
- By** how much?_______________________**
**When **will the change occur?**____________**
**How **will it be measured?**_______________**

- 2.6

## Accountability Question 2 Checklist

*Make sure that you have...*
- Program goal(s) that are clearly stated
- Program goal(s) that are realistic and measurable
- Clearly defined the target population(s)
- Objectives (e.g., desired outcomes) that are linked to your goals

- 2.7

## Accountability Question 2 Checklist (cont.)

- Specified the amount of change expected in your desired outcomes
- Specified by when you expect the desired outcomes to occur
- Clarity about how the desired outcomes will be measured
- Access to the information needed to measure the goals and desired outcomes

- 2.8

## ACCOUNTABILITY QUESTION #2: Goal Statements

- GOALS:
- Briefly describe what you want to accomplish (long-term results):_To increase the age of initiation of smoking and drinking from age 11 to age 14; To improve school performance in targeted youth; To improve the “character” and discipline of youth in school.             ._
- Always check to make sure that your goal statements:
- _ ____ _Are realistic
- _ ____ _Are clearly stated
- _ ____ _Are measurable

- 2.9

## ACCOUNTABILITY QUESTION #2: Target Population

- Ask Yourself:
- 	•Whom are we trying to reach?
- _	__Describe Target Population_
- _	__5th grade students in Yourtown Elementary School _
- _	___________________________________________
- _	__________________________________________                                                                             _
- 	• How many do you want to involve?
- 	_50 5th grade students                                        _	__________________________________

- 2.10

## ACCOUNTABILITY QUESTION #2: Objectives (i.e., desired outcomes)

- 	Briefly describe what changes in your target 		population you expect as a result of your 		program.
- 	Consider:
- 		**What** will change? _Grade point average_			For **Whom**? _50 students                        _			By **how much**? _10% increase                _			**When** will the change occur?_After 1 year_			**How** will it be measured? _Report cards _

- 2.11

## ACCOUNTABILITY QUESTION #2: Objectives (i.e., desired outcomes) (cont.)

- Always check to make sure you outcome statements are:
- _  ____ _Clearly stated
- _  ____ _Obtainable
- _  ____ _Linked to a program goal
- _  ____ _Results for which the program should 	be held accountable

- 2.12

## ACCOUNTABILITY QUESTION #2: Objectives (i.e., desired outcomes)

- 	Briefly describe what changes in your target 		population you expect as a result of your 		program.
- 	Consider:
- 		**What** will change?_Disciplinary referrals_ 			For **Whom**?_ 50 students                          _			By **how much**?_ 25% decrease                 _			**When** will the change occur?_After 1 year_			**How** will it be measured?_Office records  _

- 2.13

## ACCOUNTABILITY QUESTION #2: Objectives (i.e., desired outcomes) (cont.)

- Always check to make sure you outcome statements are:
- _  ____ _Clearly stated
- _  ____ _Obtainable
- _  ____ _Linked to a program goal
- _  ____ _Results for which the program should 	be held accountable

- 2.14

## ACCOUNTABILITY QUESTION #2: Objectives (i.e., desired outcomes)

- 	Briefly describe what changes in your target 		population you expect as a result of your 		program.
- 	Consider:
- 		**What** will change? _Character            _ 			For **Whom**? _50 students                    __	__	__	__	_By **how much**? _25% increase            _			**When** will the change occur?_After 1 year_			**How** will it be measured? _Survey scores_

- 2.15

## ACCOUNTABILITY QUESTION #2: Objectives (i.e., desired outcomes) (cont.)

- Always check to make sure you outcome statements are:
- _  ____ _Clearly stated
- _  ____ _Obtainable
- _  ____ _Linked to a program goal
- _  ____ _Results for which the program should 	be held accountable

- 2.16

## Accountability Question 3  BEST PRACTICE

- Which science (evidence) based models and best practice programs can be useful in reaching the goals?

- 3.1

## Definition of Science Based

*In general, the term, science-based, refers to a process in which experts, using commonly agreed upon criteria for rating research interventions, come to a consensus that evaluation research findings are credible and can be substantiated (Northeast CAPT, 1999)*

- 3.2

## Several examples of these criteria may include:

- The degree to which the program is based on a well-defined theory or model
- The degree to which the target population received sufficient intervention (i.e., dosage)
- The degree to which there is strong evidence of a cause and effect relationship (i.e., a high likelihood that the **program** caused or strongly contributed to the **desired outcomes**)

- 3.3

## Best Practice

- Defined as...
*Knowledge that is developed from the field that has demonstrated lessons learned to use or to avoid (e.g., mistakes)*

- 3.4

## Principles of Effectiveness

- 1)  Select and clearly define a target population.
- 2)  Address the major forms of drug abuse.
- 3)  Address the major risk and protective factors.
- 4)  Intervene in families.
- 5)  Intervene in major community institutions as well.
- 6)  Intervene early enough.

- Source:  Office of National Drug Control Policy *Evidence-Based Principles and Guidelines for Substance Abuse Prevention and Management (September 9, 1999)*

- 3.5

## Principles of Effectiveness (cont.)

- 7)  Intervene often enough.
- 8)  Address availability and marketing.
- 9)  Share information.
- 10) Strengthen anti-drug use attitudes and norms.
- 11) Strengthen life skills and drug-refusal skills.
- 12) Consider alternative activities.
- 13) Use interactive techniques.

- Source:  Office of National Drug Control Policy *Evidence-Based Principles and Guidelines for Substance Abuse Prevention and Management (September 9, 1999)*

- 3.6

## Prevention Principles

- Prevention programs should be designed to enhance “protective factors.”
- Prevention programs should target all forms of drug abuse.
- Prevention programs should include skills.
- Prevention programs for adolescents should include interactive methods.
- Prevention programs should include a parents’ or caregivers’ component.

- Source: National Institute on Drug Abuse, *Preventing Drug Use Among Children and Adolescents (1997)*

- 3.7

## Prevention Principles (cont.)

- Prevention programs should be long-term.
- Family-focused prevention efforts have a greater impact.
- Community programs that include media campaigns and policy changes are more effective accompanied by school and family interventions.
- Community programs need to strengthen norms against drug use.

- Source: National Institute on Drug Abuse, *Preventing Drug Use Among Children and Adolescents (1997)*

- 3.8

## Prevention Principles (cont.)

- Schools offer opportunities to reach all populations.
- Prevention programming should be adapted to the local community.
- The higher the level of risk of the target population, the more intensive the prevention effort must be.
- Effective prevention programs are cost-effective.

- Source: National Institute on Drug Abuse, *Preventing Drug Use Among Children and Adolescents (1997)*

- 3.9

## Addressing Accountability Question 3

- Examine what science-based and best practice sources/resources are available in your content area
- Determine how the results of the science-based/best practice program fit with the goals and objectives already identified in Accountability Question 2
- Select the program

- 3.10

## Accountability Question 3 Checklist

*Make sure that you have...*
- Examined what science-based sources/ resources are available in your content area.
- Determined how the results of the science-based/best practice program fit with your goals and objectives.
- Determined if the results of the science-based/best practice program are applicable to your target population (e.g., same age, similar characteristics).

- 3.11

## Accountability Question 3 Checklist (cont.)

- Included the evidence-based Principles of Effectiveness if you are adapting a science-based program or developing a best practice program.
- Constructed a logic model of your program to foster clarity of purpose, buy-in from stakeholders, and a rationale for program selection.

- 3.12

## WINNERS (BEST PRACTICE)

- Could not find a science-based program
- Adapted a curriculum
- Included a mentoring component
- Principles of effectiveness

## An Example of a Logic Model Based on the WINNERS Program

- Conditions

- Outcomes

- Activities

- Goals

- Increasing Rates of Drug, Alcohol and Tobacco Use Among Youth

- Poor Academic Achievement

- Increasing Incidents of Conduct Disorder Problems

- Goal: To increase the age of initiation of smoking and drinking from age 11 to age 14

- Goal: To improve school performance in targeted youth.

- Goal: To improve the “character” and discipline of youth in school

- A Mentorship Program Providing Positive Adult Role Models who Act as Companions and Tutors

- Implementation of a character-based program focusing on issues important to key stakeholders

- Decreased favorable attitudes toward problem behaviors (alcohol and tobacco use, particularly)
- Surveys

- Improved Commitment to School
- Attendance/truancy
- Grades
- Behavioral reports

- Decreased Rebelliousness
- Behavioral reports
- Observations
- Truancy

## Accountability Question 4 FIT

- What actions need to be taken so the selected program “fits” the community context?

- 4.1

## Program “Fit”

- Defined as...
*The degree to which a selected science-based/best practice program fits the community context, the philosophy of the organization, etc. *

- 4.2

## Consider how the proposed program will fit with

- Values and practices of the community
- The philosophical mission of the agency/organization
- The level of community readiness for the prevention/intervention

- 4.3

## Addressing Accountability Question 4

- The targeted population’s cultural context.
- Local programs already offered to your target population.
- The programs of other community agencies already addressing the identified needs.

- 4.4

## Accountability Question 4 Checklist

*Make sure that you have...*
- Determined how your program will fit with existing programs offered to the same target population
- How your program will fit with existing programs to meet larger community goals
- Examined how the program will fit within your agency/organizational structure

- 4.5

## WINNERS(FIT)

- Examined existing programs for the target population
- Examined fit within school
- Examined fit within the community context

## Accountability Question 5 PLAN

- What is the plan for this program?

- 5.1

## Importance of Planning

- Lewis Carroll
- 1872
- _Through the Looking Glass_

**Importance of Planning**

- Alice: Which way should I go?
- Cat: That depends on where you are going.
- Alice: I don’t know where I’m going?
- Cat: Then it doesn’t matter which way you go!!

- 5.2

##  Plan

- Defined as...
*A roadmap for your activities that will facilitate the systematic implementation of your program*

- 5.3

## Why is planning important?

- Cannot remember everything
- Worksheets serve as prompts to remember all the details
- Planning improves implementation
- Better implementation leads to better outcomes

- 5.4

## Addressing Accountability Question 5

- List all activities, large and small (recruitment, training)
- For each activity:
- Link to desired outcomes

- For each activity:
- when scheduled
- how many expected
- expected duration
- who is responsible
- resources needed/available
- location

- 5.5

## Key Activities

- 5.6

## Ensuring Cultural Competence

- Ensure that the program is culturally relevant for your audience.
- Use the following checklist as a beginning DRAFT.

- 5.7

## 5.7A

## DRAFT Cultural Competence Checklist

- 5.8

## DRAFT Cultural Competence Checklist (cont.)

- 5.9

## Program Fidelity

- Define as...
*How closely your program was implemented to what the creators of the program intended*

- 5.10

## Program Fidelity

- Why is it important?
- The program is based on a theory
- How to ensure fidelity?
- Program authors for checklists/instruction manuals
- Develop own checklists
- Fidelity vs. costs & local context

- 5.11

## Accountability Question 5 Checklist

*Make sure that you have...*
- Specific, well-planned activities to reach your goals.
- A realistic, timeline for the completion of each activity.
- Identified those who can be responsible for each activity.

- 5.12

## Accountability Question 5 Checklist (cont.)

- A budget that outlines the funding amounts for each activity.
- Identified facilities/locations available for each activity.
- Identified resources available for each activity.
- Identified resources needed for each activity.

- 5.13

## Accountability Question 6 CAPACITIES

- What organizational capacities are needed to implement the program?

- 6.1

## Organizational Capacities

- Defined as...
*The capacities the organization brings to direct and sustain the prevention program.*

- 6.2

## Types of Capacities

- Human capacities
- Technical capacities
- Fiscal capacities

- 6.3

## Human Capacities

- Credential and experience of the staff.
- Staff training is likely to be required.
- There must be leadership and commitment to the program.

- 6.4

## Technical Capacities

- Determine that what you need to implement the program will be available
- If unavailable, how will you get the technical resources that are required?
- Specific technical skills (e.g., evaluation) may also be required

- 6.5

## Fiscal Capacities

- Adequate funding to implement a prevention program
- Creative ways to obtain non-monetary resources

- 6.6

## Accountability Question 6 Checklist

*Make sure that you have...*
- Staff with appropriate credentials and experience
- Adequate number of staff hired
- Clearly defined roles for staff members
- Determined that the leaders have an adequate understanding of the program

- 6.7

## Accountability Question 6 Checklist (cont.)

- Strong leadership for the program
- Strong commitment for the program
- Adequate technical resources (or a plan to get them)
- Adequate funding for the program to be implemented as planned

- 6.8

## WINNERS(CAPACITIES)

- Assessed staffing requirements
- Assessed technical capacities
- Assessed fiscal requirements

## Accountability Question 7 PROCESS EVALUATION

- Is the program being implemented with quality?

- 7.1

## 7.1A

## Process Evaluation

- Defined as...
*An assessment of activities were implemented, the quality of the implementation, and the strengths and weaknesses of the implementation.*

- 7.2

## Why is a process evaluation important?

- Generates information that can be used for:
- ongoing program refinement
- replicating the program in the future (what worked what didn’t)
- demonstrating activity prior to the collection of outcome data

- 7.3

## General Elements of a process evaluation?

- What activities were implemented?
- How many activities were implemented?
- What were the strengths and weaknesses of the implementation?
- Timeliness
- Were all aspects of the plan implemented?

- 7.4

## Typical Process Indicators

- program attendance
- number of sessions /other activities
- number of planning meetings
- amount of materials used
- actual date of implementation (vs scheduled)

- 7.5

## Process Evaluation: Three Areas

- Planning Process
- Program Implementation
- Post Program Implementation

- 7.6

## Process Evaluation:  Planning Process

- attendance
- discussion topics (linked to outcomes)
- actions decided upon
- persons responsible
- target date

- report on actions taken from previous meetings
- persons responsible
- target date
- Master List of Actions for group

- Meeting Form

- 7.7

## Sample Instruments to Assess Planning Meetings

- Meeting Form R
- Meeting Effectiveness Inventory

- 7.8

## Process Evaluation:  Implementation

- What you planned VS what happened
- Indicators to track implementation
- actual date/timeliness
- actual attendance (% attendance goal achieved)
- actual duration (% duration goal achieved)
- materials distributed
- resource adequacy

- 7.9

## Implementation: Part 2

- 7.10

## Resources and Timeliness: Part 3

- 7.11

## Open Ended Questions: Part 4

- Program:
- What was not implemented that was planned?
- Why?
- What was implemented that was not planned?
- Why?
- Who was missing?
- What led to their absence?

- 7.12

## Process Evaluation:  Post Implementation

- “Midcourse corrections” and
- When you repeat the program
- information from the planning process assessments
- information about implementation assessment
- Satisfaction surveys - participants
- Project Insight Form (barriers, facilitators) - program staff

- 7.13

## The Project Insight Form can be used to track lessons learned

- Helps to identify...
- which factors were barriers to program implementation
- which factors facilitated program implementation

- 7.14

## Accountability Question 8  OUTCOME EVALUATION

- How well is the program working?

- 8.1

## Sample Individual-level Outcomes

**Changes in...**
_**knowledge**_**:** what people learn about a subject
_**attitudes**_**:** how people feel toward a subject
_**skills**_**:** the knowhow people acquire that enables people to make a problem better for themselves
_**behaviors**_**:** this is a change in how people actually act

- 8.2

## Sample outcomes pertaining to community-wide interventions

- Community awareness and mobilization
- Changes in policies and laws to control drinking and drug use (e.g., DUI laws)
- Increased cooperation and collaboration among community agencies

- 8.3

## Intermediate Outcomes

- Typically address changes in the risk and protective factors _associated_ with behaviors (e.g., attitudes toward drug use).

- 8.4

## Why measure outcomes?

- Answers: Did it work?
- Can use the outcome information to improve the program
- Can use outcome information to decide between different programs
- IF YOU DO NOT:  Positive process evaluations (implementation & satisfaction) may be misleading

- 8.5

## Aim for Achieving Outcomes that:

- Affect _actual behaviors_
- Affect _larger groups of people_
- _Are longer lasting_

- 8.6

## Steps Suggested for Outcome Evaluation

- Decide what you want to assess
- Select an evaluation design to fit your program
- Choose methods of measurement
- Decide who you will assess

- 8.7

## Steps Suggested for Outcome Evaluation (cont.)

- Determine when you will conduct the assessment
- Gather the data
- Analyze the data
- Interpret the data

- 8.8

## Evaluation Designs

- Post Only Design
- Pre-post design - Target Group
- Pre-Post Design (with Comparison Group)
- Pre-Post Design with  Control Group

- 8.9

## Outcome Evaluation:  Choosing a Design

- Structure of the outcome evaluation
- Measures change
- Confidence that your program was responsible

- 8.10

## Data Collection Methods:Quantitative techniques

- answer who, what, where, how much
- target larger numbers of people
- more structured and standardized

- mostly deal with numbers that are analyzed with statistics
- Types
- Surveys
- face to face
- phone
- self-administered
- Archival data

- 8.11

## Data Collection Methods: Qualitative Techniques

- answer why and how
- usually involve talking to or observing people
- target fewer people

- deal in words organized into themes
- Types
- Interviews
- Focus groups
- Open-ended questions
- Participant observation

- 8.12

## Data Collection Issues

- Consent
- Confidentiality
- Anonymity

- 8.13

## Analysis:  Descriptive

- Mean (average) = sum of the ratings divided by the total number
- Standard Deviation = measure of spread
- Median = the rating that splits the group in half (50% above and below)
- Mode = the most common rating in the group
- Frequencies = % of each category

- 8.14

## Accountability Question 8 Checklist

*Make sure that you have...*
- Decided what you want to assess
- Selected an evaluation design to fit your program
- Chosen methods for measurement
- Decided who you will assess

- 8.15

## Accountability Question 8 Checklist (cont.)

- Determined when you will conduct the assessment
- Gathered the data
- Analyzed the data
- Interpreted the data

- 8.16

## Character Scores

**Character Scores**

- 8.17

## Referrals

**Referrals**

- 8.18

## Grades

**Grades**

- 8.19

## WINNERS(OUTCOME EVALUATION)

- Overhead of results

## Accountability Question 9  IMPROVE

- How will continuous quality improvement strategies be included?

- 9.1

## Continuous Quality Improvement

- Defined as...
*CQI involves the systematic assessment and feedback of evaluation information about planning, implementation and outcomes to improve programs.*

- 9.2

## A Straightforward CQI Process: Summarize main points from 1-8

- 1) Have the needs of the target group changed?
- 2) Have the goals/ outcomes changed?
- 3) Are there better best practices available
- 4) Does the program still fit with your agency’s goals?

- 5) How well did you plan?
- 6) Have any organizational capacities changed?
- 7) How well was the program implemented?
- 8) What were the conclusions of the outcome evaluation?

- 9.3

## WINNERS(CQI)

- Needs/Resources
- Reexamination of question 1-8
- Related changes to program

## Accountability Question 10  SUSTAIN

- If the program is successful, how will it be sustained?

- 10.1

## 10.1A

## Sustainability

- Defined as...
*refers to the continuation of program after the initial funding has ended*

- 10.2

## Why is sustainability important?

- A waste when good programs end and the need still exists
- New programs are expensive to START
- Ending a successful program can generate resistance to (funders, staff, community) new ones

- 10.3

## Addressing Accountability Question 10

- Program Negotiation Process
- Program Effectiveness
- Program Financing
- Training

- Institutional Strength
- Integration with Existing Programs/Services
- Program Champions

- 10.4

## Accountability Question 10 Checklist

*Make sure that you have...*
- Started discussions early with community members about sustaining the program
- Ensured that the needs of the community are driving the program
- Developed a consensus-building process to reach a compromise for addressing different stakeholder (community, funder, technical experts) needs.

- 10.5

## Accountability Question 10 Checklist (cont.)

- Ensured that the program is achieving the desired outcomes
- Begun an assessment of the community’s local resources to identify potential “homes” for the program
- Considered options (e.g., scaled-down version of the program) to discuss with those who may sustain the program.

- 10.6

## Accountability Question 10 Checklist (cont.)

- Clear strategies in place for gradual financial self-sufficiency
- There is a strong organizational base for the program
- The program can be integrated with other programs
- The goals of the program are adapted to the local population needs

- 10.7

## Accountability Question 10 Checklist (cont.)

- The program is compatible with the mission and activities of the host organization
- There is a respected program champion
- The program is endorsed from the top of the organization

- 10.8

## WINNERS(SUSTAIN)

- Effectiveness
- Funding Sources
- Program Champion
- “Home” for the Program
- Community Buy-in

## Training & TA

- Resource Development

- Report Writer

- Needs Assessment

- Community Resources

- Logic Model

- Program Selection

- Implement Program

- Process MIS

- Outcome Measures Core + Unique

- Data Analysis

**Decision Support System (CSAP)**

## Training & TA

- Resource Development

- Report Writer

- Needs Assessment

- Community Resources

- Logic Model

- Program Selection

- Implement Program

- Process MIS

- Outcome Measures Core + Unique

- Data Analysis

- (Needs/Resources)

- (Needs/Resources)

- (Goals)

- (Best Practice)

- (Plan/ Process)

- (Process Evaluation)

- (Outcome Evaluation)

- ROGG

- (Outcome Evaluation)