**HSA 2558 HEALTH PROGRAM EVALUATION**

**SPRING TERM 2007 (2074)**

**January - April, 2007**

**Monday 1:00-3:50 p.m.**

**Room A215 Parran Hall**

**Kenneth J. Jaros, Ph.D. and Diane J. Abatemarco, Ph.D.**

**210 Parran Hall A223 Crabtree Hall**

**624-3102 383-9629**

**kjaros@pitt.edu dja17@pitt.edu**

**[Course Objectives and Outline]{.underline}**

**I.** **GENERAL PURPOSE AND GOALS**

This course is designed to give students skills which will enable them
to more effectively review, monitor, design and manage program
evaluations in the health and public service sector. The course will
include lectures, class exercises and discussions, and other
presentations which address practical evaluation issues of importance to
administrators, program managers, planners, and researchers. Students
will be introduced to the basic concepts and methods of program
evaluation, and will practice the application of these methods. The
examples and models presented and discussed will be primarily from the
field of health care, although examples from other disciplines will also
be included. The course is not intended to be an introductory class in
research methods or statistics - some basic knowledge of these subjects
is assumed and they will be discussed in the context of evaluation
strategies.

It will also not be possible to cover every aspect of the extensive
field of evaluation research. The primary goal is to give students a
basic understanding of program evaluation methods and a set of skills to
assist in practical program management, and project assessment.

II. **COURSE OBJECTIVES**

The course will attempt to give students:

-   an understanding of program evaluation in the context of basic
    research methods and policy development;

```{=html}
<!-- -->
```
-   a knowledge and understanding of the primary evaluation methods used
    in the assessment of public funded programs;

-   skills to comprehend and assess evaluation reports, and proposals
    for evaluation;

-   skills to formulate, design, and present evaluation projects;

-   the ability to recognize and address key issues in the
    implementation and management of evaluations; and

```{=html}
<!-- -->
```
-   an understanding of the means for using evaluation results for
    program and policy changes.

**III. EXPECTATIONS AND ASSIGNMENTS**

Readings will be assigned for each session and students will be
responsible for this material, as well as that which is presented in
class. In-class lectures and presentations are designed to broaden,
expand, and clarify the student\'s understanding of program evaluation
issues and methods - in other words, lectures will not necessarily be a
review of the assigned readings.

Students will also be expected to participate in class discussions and
exercises and participation will be taken into account in the final
grade.

Grades will be determined based on the following criteria:

Homework Assignments (2x25%) 50%

Final Paper 40%

Class participation 10%

**IV. READINGS AND REFERENCES**

Although we will generally be following the outline of the Rossi, Lipsey
and Freeman text, several other readings have been assigned. Other
topical readings may also be assigned during the course of the term.

**[Text Book:]{.underline}**

Peter H. Rossi, Mark W. Lipsey and Howard E. Freeman, Evaluation -- A
Systematic Approach, SAGE Publications, 2004.

**[Additional Assigned and Suggested Readings:]{.underline}**

Beck, K. H., Rauch, W. J., Baker, E. A. and Williams, A. F. (1999)
Effects of ignition interlock license restrictions on drivers with
multiple alcohol offenses: a randomized trial in Maryland. [American
Journal of Public Health, 89]{.underline}, 1696-1700.

Bonaguro, J.A., Rhonehouse, M. and Bonaguro, E. W. Effectiveness of four
school health education projects upon substance use, self-esteem, and
adolescent stress. [Health Education Quarterly, 15,]{.underline} 81-92.

Bybee, D., Mowbray, C. T. and Cohen, E. H. Evaluation of a homeless
mentally ill outreach program: differential short-term effects.
[Evaluation and Program Planning, 18,]{.underline} 13-24.

Daltroy, L. H., Phillips, C. B., Eaton, H. M., Larson, M. G., Partridge,
A. J., Logigian, M. and Liang, M. H. (1995) Objectively measuring
physical ability in elderly persons: the physical capacity evaluation.
[American Journal of Public Health*,*85]{.underline}, 558-560.

Dennis, M. L., Fetterman, D. M. and Sechrest, L. (1994) Integrating
qualitative and quantitative evaluation methods in substance abuse
research. [Evaluation and Program Planning, 17]{.underline}, 419-427.

Deuson, R. R., Hoekstra, E. J., Sedjo, R., Bakker, G., Melinkovich, P.,
Daeke, D., Hammer, A. L., Goldsman, D. and Judson, F. N. (1999) The
Denver school-based adolescent hepatitis B vaccination program: a cost
analysis with risk simulation. [American Journal of Public Health,
89]{.underline}, 1722-1727.

Foshee, V. A., Bauman, K. E., Arriaga, X. B., Helms, R. W., Koch, G. G.
and Linder, G. F. (1998) An evaluation of safe dates, an adolescent
dating violence prevention program. [American Journal of Public Health,
88]{.underline}, 45-50.

Sallis, J. F., McKenzie, T. L., Alcaraz, J. E., Kolody, B., Faucette, N.
and Hovell, M. F. (1997) The effects of a 2-year physical education
program (SPARK) on physical activity and fitness in elementary school
students. [American Journal of Public Health, 87]{.underline},
1328-1334.

Stevens, S. A., Cockburn, J., Hirst, S. and Jolley, D. (1997) An
evaluation of educational outreach to general practitioners as part of a
statewide cervical screening program. [American Journal of Public
Health, 87]{.underline}, 1177-1181.

Trussell, J., Koenig, J., Ellertson, C. and Stewart, F. (1997)
Preventing unintended pregnancy: the cost-effectiveness of three methods
of emergency contraception*.* [American Journal of Public Health,
87]{.underline}, 932-937.

Whittaker, J., Overstreet, E. J., Grasso, A., Tripodi, T. and Boylan, F.
(1988) Multiple indicators of success in residential youth care and
treatment. [American Journal of Orthopsychiatry, 58]{.underline},
143-147.

Wills, T. A. and Cleary, S. D. (1997) The validity of self-reports of
smoking: analyses by race/ethnicity in a school sample of urban
adolescents. [American Journal of Public Health, 87]{.underline}, 56-61.

**V. CLASS SESSIONS**

**[January 8]{.underline} Session 1**

Introduction to course, putting evaluation in perspective, definitions,
historical factors, purpose of evaluations

*Learning Objectives:*

-   Define health programs

-   Define program evaluation

-   Understand the purpose and focus of program evaluation

-   Recognize the importance of program evaluation

-   Recognize different approaches to program evaluation

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapters 1 & 2

**[January 15]{.underline} No Class - Martin Luther King Day**

**[January 22]{.underline} Session 2**

Planning the evaluation, types and categories of evaluations, goal
setting, formulating the evaluation question, defining the program
causal linkages; introduction to the "logic model"

*Learning Objectives:*

-   Identify the components of the evaluation process

-   Recognize the different types of evaluations

-   Understand and identify program goals and objectives

-   Conceptualize and develop program causal linkages

-   Understand the importance of the evaluation planning process

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapters 2 & 3

> \(2\) Cramer M E, Mueller K J, & Harrrop D, Comprehensive Evaluation
> of a Community Coalition: A Case Study of Environmental Tobacco Smoke
> Reduction. Public Health Nursing 20(6), 464-477

**[January 29]{.underline} Session 3**

Establishing goals and objectives, defining variables, basic evaluation
startegies

*Learning Objectives:*

-   Understand basic models for conducting social science research, and
    the application of these models and methods to program evaluation
    questions

-   Be able to diagram these designs and explain how the evaluation
    questions will be addressed

-   Specifically to differentiate between experimental,
    quasi-experimental, and qualitative approaches to evaluation, and to
    be able to apply these approaches to appropriate evaluation
    questions

-   Understand the concept and definition of dependent and independent
    variables, and how these terms and definitions relate to program
    evaluation

-   Understand the basic strategies for collecting information for
    research and evaluation purposes

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapters 3 & 5

**[February 5]{.underline} Session 4**

Measurement of objectives and data collection

*Learning Objectives:*

-   Understand the various levels and types of measurement typically
    employed in program evaluation studies

-   Understand the concept of validity and reliability of measurements,
    and to critically assess measures on these criteria

-   Gain ability through literature searching to locate exiting scales
    and psychometric measure that may be applicable for use in program
    evaluations

-   Have a basic understanding of principles of questionnaire
    construction

-   Gain basic skills in developing valid and reliable measures for use
    in program evaluations

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapter 6 & 7

\(2\) Daltroy, et al.

\(3\) Wills & Cleary

\(4\) Whittaker et al.

\(5\) Campbell H S, Phaneuf MR, Deane K. Cancer per support programs --
do they work?

**[February 12]{.underline} Session 5**

Needs assessment, diagnostic procedures, use of existing data sources

Systems and methods for program monitoring and accountability

*Learning Objectives:*

-   Define and conceptualize needs assessment

```{=html}
<!-- -->
```
-   Recognize the importance of needs assessment in the evaluation
    process

-   Identify the components of a needs assessment

-   Identify sources of information/data for a needs assessment

-   Understand the purpose and importance of program monitoring

-   Recognize the different purposes of program monitoring

-   Identify data sources for program monitoring

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapter 4

**[February 19]{.underline} Session 6 (Homework #1 Assigned)**

Introduction to basic evaluation designs; consideration of ethical and
values issues

*Learning Objectives:*

-   Gain greater familiarity with and understanding of the concepts and
    issues regarding evaluation designs and measurement that were
    presented in Sessions 3 and 4

-   Develop an improved understanding of the most appropriate types of
    evaluation designs to apply to various program evaluation questions

-   Expand skills in recognizing strengths and weaknesses of measures
    employed in program evaluations.

-   Gain understanding of concept of threats to internal validity in
    evaluation designs

-   Gain a general understanding of the types of statistical analyses
    used in program evaluation

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapter 8

**[February 26]{.underline} Session 7** **(Homework #1 Due)**

Evaluations using randomized and non-randomized designs

*Learning Objectives:*

-   Recognize the most common types of evaluation designs not employing
    random assignment to groups, and understand their applicability to
    various program evaluation strategies

-   Gain experience in critiquing evaluation designs, with particular
    emphasis on limitations of the methodology and measurements being
    employed

-   Understand practical problems associated with conducting evaluation
    studies in community- based service delivery settings

-   Gain ability to articulate evaluation designs and rationale to
    stakeholders

-   Develop appropriate data collection and data analysis strategies for
    various evaluation methodologies

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapters 8 & 9

\(2\) Bonaguro, et al.

\(3\) Bybee, et al.

\(4\) Stevens, et al.

**[March 5]{.underline} Session 8**

Review of evaluation strategies; discussion of examples

*Assigned Readings:* (1) Sallis, et al.

\(2\) Beck, et al.

\(3\) Foshee, et al.

**[March 12]{.underline} No Class - Spring Break**

**[March 19]{.underline} Session 9**

Issues of ethics and cultural competency in Community-Based evaluation

*Guest Lecturer:*

*Assigned Readings:* (1) Rossi, Lipsey & Freeman, Chapters 12

**[March 26]{.underline} Session 10**

Examples of approaches to qualitative evaluation

*Guest Lecturer:*

*Assigned Readings*: (1) Review Rossi, Lipsey, & Freeman, Chapter 4

\(2\) Dennis, et al.

**[April 2]{.underline} Session 11** **(Homework #2 Assigned)**

Practical evaluation approaches using multiple methods

*Guest Lecturer:*

**[April 9]{.underline}** **Session 12** **(Homework #2 Due)**

Analyzing Effects

*Assigned Readings*: (1) Rossi, Lipsey & Freeman, Chapter 10

**[April 16]{.underline}** **Session 13**

**[April 23]{.underline} Session 14 (FINAL PAPER DUE)**

Student Presentations & class discussion