# Presentation: 506632

## Evaluation Design, and 

- Director of Projects
- Social Science Research & Evaluation, Inc.
- 21-C Cambridge Street, Burlington, MA 01803
- 781-273-4206, wharding@ssre.org
- Prepared for:
**CSAP’s Northeast Center for the Application **
**of Prevention Technologies (CAPT)**
- Health and Human Development Programs at
- Education Development Center, Inc. (EDC), Newton, MA
- www.Captus.org

**Evaluation Design, and **
**Measurement of Outcomes**

- © 2000 EDC, Inc.

## Workshop Objectives

- Increase understanding of evaluation design and its relationship to certainty and cost.
- Increase understanding of what outcomes are (and are not).
- Identify key issues in measuring outcomes.
- USED’s expectation concerning evaluation design and outcomes, particularly GPRA* indicators.
- * Government Performance Results Act of 1993.

- © 2000 EDC, Inc.

## What is Evaluation?

- Evaluation is a systematic attempt to determine whether (or to what degree) a program (policy, procedure, activity) produces its stated outcomes (short and/or long term).

- © 2000 EDC, Inc.

## Four Evaluation Designs

**1. One-group posttest design**
**	****	****	****X****	****posttest**
**2. One-group pretest-posttest design**
**	****pretest ****	****X****	****posttest**
**3. Pretest-posttest comparison-group design**
**	****pretest****	****	****X****	****posttest****	****pretest****	****	****O****	****posttest**
**4. Pretest-posttest control-group design**
**	****pretest****	****	****X****	****posttest****	****random assignment****	****pretest****	****	****O****	****posttest**

- © 2000 EDC, Inc.

## Multiple Choice:  What Is A Quark?

- A unit of currency in Peru.
- The sound a duck makes.
- A sub-atomic particle.
- A sedan made by Toyota.
- A candy bar made by Nestle.

- © 2000 EDC, Inc.

## Five Issues

- More complex evaluations increase certainty about the results but they also cost more.
- More complex evaluation designs are more difficult to implement.
- There are many more evaluation designs than the four we just reviewed.
- No evaluation design is immune from threats to its validity.
- The list of possible complications is long.

- © 2000 EDC, Inc.

## GPRA* Indicators

-  1. The percentage of grantees whose target students show a measurable decrease in binge drinking.
-    2. The percentage of grantees that show a measurable increase in the percentage of target students who believe that alcohol abuse is harmful to their health.
-    3. The percentage of grantees that show a measurable increase in the percentage of target students who disapprove of alcohol abuse.
- 	*(Government Results Reporting Act, see USED, 2005 for GRAA indicators.)

- © 2000 EDC, Inc.

## Measurement of GPRA Indicators

- An ***appropriate ***measure must be chosen for each of the 3 GPRA indicators.
- You will need a measure for binge drinking, for perception of harm, and for disapproval of alcohol abuse.
- For example to measure binge drinking you might ask:  “On how many occasions during the last 30 days (if any) have you been drunk or very high from drinking alcoholic beverages?” *
- * SAMHSA, 2005.

- © 2000 EDC, Inc.

**Notes:**
 Working with your evaluator you will need to chose an appropriate measure for each indicator:  a measure for binge drinking, one for perception of harm, and one for disapproval of alcohol abuse. 

 USED will supply specific measures for these shortly. Meanwhile, to get an idea, of what these measures may look like, you might want to look at the GPRA measures SMHSA is recommending – the web address is on the referencec page of your handout.

 FYI,  I have provided you with a handout that defines some of the characteristics of “appropriate” measures that were used by CSAP (Center for Substance Abuse Prevention) in its core measure initiative.

## Measuring Outcomes

**	**Next to choosing an overall evaluation design, specifying the outcomes to be achieved is probably the single most important step in evaluation.
** **

- © 2000 EDC, Inc.

## What Outcomes Are

- Outcomes are the things you
- expect to occur as the result
-  of an activity or intervention.

- © 2000 EDC, Inc.

## What Outcomes Are Not

-   Outcomes are not goals.** **
- Goals are broader, less precise
- statements about the impact a program intends to have on its target population.

- © 2000 EDC, Inc.

## If the program’s goal is to reduce student substance abuse, does this mean it will have succeeded:

- When a smaller number of students try drugs than tried them before?
- When students who were using drugs stop using them?
- When both of the above occur?
- When one less student is using drugs than before?
- When the percentage of drug-using students ahs been reduced by 1% or more?
- When fewer students use all substances – alcohol, tobacco, marijuana, and other drugs – or when fewer use any one type of drug?

- © 2000 EDC, Inc.

## What Outcomes Are Not

-   Outcomes are not Process Objectives.
- Process Objectives are statements
-  about planned program activities

- © 2000 EDC, Inc.

## Examples of Process Objectives

- By month 3, to develop a six-session parent training program for Jonestown adolescent mothers and fathers that is designed to improve the physical and psychological health of both teen parents and their children.
- By month 6, to pilot test the program on a convenience sample of 20 Jonestown teen parents and revise the program based on (1) participants’ performance on standardized pretests and posttests measuring parenting skills, and (2) participants’ opinions about the course.
- By month 12, to deliver the revised program to 50 volunteer teen parents from the Jonestown area.

- © 2000 EDC, Inc.

## Outcome Objectives

- who is expected to change
- what is expected to change (e.g., knowledge, attitudes, skills, behaviors)
- how much change is expected (e.g., a percentage change in teen alcohol use)
- when the change will occur (e.g., at the end of the program)
- *Chinman et al., 2004.

**Outcome Objectives**

- © 2000 EDC, Inc.

## Sample Outcome

**Sample Outcome**

- © 2000 EDC, Inc.

## Another Sample Outcome 

- In addition, we expect there will be a statistically significant difference in the pre- to posttest performance of these teens as compared to a control group that will not receive the training.** **

**Another Sample Outcome**

- © 2000 EDC, Inc.

## Who Wins?

- 	There are 12 runners sprinting around a course. One team of 6 runners wears blue shirts, one team of six wears green shirts.  How might we measure or define which team wins.

- © 2000 EDC, Inc.

## Dimensions of Outcomes

- Who or what is going to change?
- How much change is going to take  place?
- How much change do you expect?
- Change compared to what?
- What will the changes cost?
- When will the changes occur?
- How many outcomes are there and how are they related?
- Were there any unintended outcomes?

- © 2000 EDC, Inc.

## Who or what is going to change?

- Describe:
- Basic demographic characteristics.
- Number of people exposed to the program (or the number who will change if this is different from the number who will be exposed).
- Specific characteristics or needs of the population, including risk and protective factors (e.g., teen mothers, or pregnant teens, or those at high-risk for substance abuse)

- © 2000 EDC, Inc.

## How much change is 

**going to take place?**

- Absolute numbers of people who changed?
- Percentage of people who changed?
- Big changes for some people, or some change for all?

- © 2000 EDC, Inc.

## Real Change:  An Example

- Population of drug users
- Percentage of youth drug users

- 4000		4500
- 2000		2250
-   50%		 45%

- At baseline     Three years later

**Real ****Change****:  ****An Example**

- © 2000 EDC, Inc.

## How much change do you expect?

- Be informed about what is reasonable to expect.
- When in doubt, be cautious about your predictions.

- © 2000 EDC, Inc.

## Change compared to what?

- Experiment – compare the outcomes for people exposed to your program to the outcomes for people who receive a traditional program.
- Experiment – compare your outcomes to those for people who get no program.
- Compare your outcomes to rates reported in a state or national survey.
- Compare your outcomes to those reported in literature for the same or similar programs.

- © 2000 EDC, Inc.

## What will the changes cost? 

- The same outcomes at less cost.

- © 2000 EDC, Inc.

## When will the changes occur? 

- When is it reasonable to see effects?
- How long will people participate in the program?
- How long will the effects last?
- Do the effects evolve or improve over time?

- © 2000 EDC, Inc.

## How many outcomes are there and how are they related?

- Develop a measurable outcome specific to each of the changes.
- Identify short- and long-term outcomes.
- Investigate how outcomes might interact.

- © 2000 EDC, Inc.

## Were there any unintended outcomes? 

- Positive?
- Negative?

- © 2000 EDC, Inc.

## References

- Chinman, M., Imm, P. & Wandersman, A. *Getting To Outcomes* 2004. http://www.rand.org/pubs/technical_reports/TR101/index.html, accessed February 2006.
- Executive Office of the President, Office of Management and Budget,  Government Performance Results Act of 1993. www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html, accessed 2/2/06.
- U.S. Department of Education, Office of Safe and Drug-Free Schools. Overview Information Grants to Reduce Alcohol Abuse; Notice Inviting Applications for New Awards for Fiscal Year (FY) 2005.  Information and Application Procedures, FY 2005. Federal Register: February 9, 2005 Volume 70, Number 26: 6843-6846, February 2005. (see: Federal Register Online via GPO, www.wais.access.gpo.gov, accessed 2/2/06.
- SAMHSA. SAMHSA Performance Measurement/GPRA Tools. http://www.samhsa.gov/grants06/tools.aspx, accessed 2/6/06.

- © 2000 EDC, Inc.