Program Evaluation, Another Tool for Managers

“We learn about the quality of a decision by observing its results. We learn most about the quality of

a decision if we deliberately, systematically assess its impact or outcomes.”

John R. Feussner, M.D.

Chief Research and Development Officer, Veterans Health Administration.

What is program evaluation?

 Program evaluation is a method used to provide specific

information about a clinical or administrative initiative’s

activities, outcomes, costs and effectiveness in meeting

goals. A “program” is a set of activities developed to

accomplish one or more goals. Program evaluations vary in

the technical complexity and sophistication of their research

designs. While some evaluations require complex technologies and research methodologies such as those conducted by HSR&D, other evaluations may require management expertise.

In theory, all programs are candidates for evaluation,

although any number of barriers can undermine the utility of

an evaluation’s findings. These barriers may include: the

political environment, the cost of the evaluation (relative to

its potential benefit), the time involved and the study design.

Why is program evaluation important to

managers?

Ideally, program evaluation determines if an initiative is

meeting its stated objectives and, if relevant, those of the

larger organization. Program evaluations may also assess

the feasibility and implementation of proposals for new,

similar programs. As programs mature, occasional or

ongoing evaluation can identify potential adjustments for

managers.

Potential limitations that managers should be aware of

regarding program evaluations include: conditional conclusions, timeliness, bias, corruption of measures, applicability,

invalid and unreliable data and unintended effects.

When and how often should program evaluation

be done?

 In an ideal world, program evaluation should be an

ongoing process. At a minimum, managers should make

evaluation decisions on three occasions during a program’s

life span, when:

M A N A G E M E N T B R I E F

Number 7 April 1998

• the program is being designed and first implemented;

• the program or its environment is scheduled to change in

some significant way; and when

• alternative programs show promise for achieving better

results (outcomes or cost), or when the program’s

effectiveness is being questioned.

What are the steps in planning a program

evaluation?

While there is no cookbook approach, several steps are

common to program evaluations. Evaluation plans generally include these elements:

1. Identify the key user of the evaluation. Identifying

decision maker(s), purposes and applications of the

evaluation plus formulating evaluation question(s) at

evaluation outset are critical to success. Well constructed

research questions address project needs, processes, and

effectiveness, as well as cost effectiveness.

2. Judge the value of a full assessment by conducting a brief

pre-evaluation assessment that will clarify potential

quality and utility of a program evaluation.

3. Conduct a review of the relevant refereed literature,

contact administrative peers for opinion, then determine

whether this information answers the preliminary

questions.

4. Determine the evaluation design based on the questions

being asked and the degree of methodological rigor

required to draw valid inferences.

5. Develop the interim communication process, scope and

depth of the final report plus the ultimate dissemination

plan.

Created by the Management Decision & Research Center, HSR&D, VA Office of Research & Development.
What resources are needed to conduct a program

evaluation?

• Evaluation team: A team contains internal or external

staff, or a combination overseen by a principal investigator, or one or more mid-level coordinators. The range

of research disciplines and questions asked will determine the exact makeup of the evaluation team.

• Data: Program evaluation requires data, whether

qualitative or quantitative, pre-program baseline data

greatly improves the evaluators’ ability to draw valid

conclusions.

• Time: The evaluation plan must realistically identify

start dates, interim milestones, and end dates.

• Funding: Evaluation costs vary. Using outside evaluators or purchased review data can affect the evaluation

budget. Regardless of costs, evaluators must explicitly

identify and separate evaluation costs from the costs of

running the program.

Research findings:

The projects and their impacts described below exemplify the range of recent and ongoing HSR&D program

evaluations.

• Evaluating and Improving Pressure Ulcer Care.

This project evaluated the use of the Patient Assessment

File, a VA administrative database, as a quality improvement tool for preventing pressure ulcers among patients

in long-term care facilities. Using this database, researchers developed a model for a successful quality

improvement program that has substantially lowered the

incidence of pressure ulcer development at a number of

VA long-term care facilities. Berlowitz DR , Halpern J.

Evaluating and improving pressure ulcer care: the VA

experience with administrative data. Joint Commission

Journal on Quality Improvement 1997; 23: 424-33.

• Twelve-Step and Cognitive-Behavioral Treatment

for Substance Abuse. This comparative evaluation

provided important evidence on the effectiveness of 12-

step treatment for substance abuse, compared with

cognitive-behavioral treatment. Nearly 3,700 patients

from 15 VA inpatient programs were involved in this

study, which showed that patients treated through 12-

step programs fared as well as those who received

cognitive-behavioral treatment. Ouimette PC, Finney

JW, Moos RH. Journal of Consulting and Clinical Psychology

1997; 65(2): 230-240.

• Evaluation of the Comprehensive Women Veterans

Health Centers. Since their establishment in 1994,

women veterans health centers have succeeded in

providing comprehensive and needed physical and

mental health care services to women veterans, this

evaluation concluded. Using a structured instrument to

collect data on the centers’ operations, the program

evaluators noted that many centers experienced substantial growth during the two-year observation period and

became highly visible in the communities they served.

They described the centers as a “strong representation of

VA’s commitment to equal treatment for women veterans.” Weiss TW, Wray NP, Mansyur C. Final Report:

Women Veterans Health Program. Houston Center for

Quality of Care and Utilization Studies. VA HSR&D Field

Program: Houston, July 15, 1996.

• Evaluation of VA Major Depressive Disorder Clinical

Guidelines Implementation. This ongoing evaluation

is assessing the effectiveness of VA’s implementation of

clinical practice guidelines for major depression and the

impact on patient outcomes, quality of care and resource

use. Post-implementation data will be compared with

pre-implementation data from four VAMCs. This study

holds promise for improving the processes of care used

to treat depression and for increasing the use of guidelines by physicians.

Selected sources for program evaluation

information:

Rossi PH, Freeman HE. Evaluation: a systematic approach, 5th

Edition. Beverly Hills: Sage Publications, 1993.

Shadish WR, Cook TD, Leviton LC. Foundations of program

evaluation: theories of practice. Newbury Park, CA: Sage

Publications, 1991.

Yin RK. Case study research: design and methods, 2nd Edition.

Thousand Oaks: Sage Publications, 1994.

For more information or to send comments,

contact:

Gerry McGlynn, M.Ed, MDRC, Manager, Information

Dissemination Program, MDRC 152-M, 150 South

Huntington Ave., Boston, MA. FTS 700-839-4433 or

617-278-4433, fax 617-278-4438. E-mail address:

geraldine.mcglynn@med.va.gov

The Research and Development web page contains a

downloadable PDF version of the primer: Program Evaluation for Managers, by William Yeaton, et al. Boston, MA:

The Management Decision & Research Center, 1997. 22

Pages. http://www.va.gov/resdev/prt

This is our seventh MANAGEMENT BRIEF. Please give us your suggestions;

Topics for future BRIEFS: ___________________________________

______________________________________________________

______________________________________________________

Comments on the MANAGEMENT BRIEF:

______________________________________________________

______________________________________________________

Name: _________________________________________________

Title: __________________________________________________

Phone: ________________________ Fax: ____________________

E-mail: ________________________________________________

Address:________________________________________________

______________________________________________________

Please send me the MANAGEMENT BRIEF by: E-mail ___ or Fax ___

The MANAGEMENT BRIEF is produced by the Management Decision and

Research Center, a program within the VA’s Office of Research &

Development, Health Services Research and Development Service.

The purpose of the MANAGEMENT BRIEF is to provide VA senior

managers with a concise and timely overview of a specific health care

topic that includes topic definition, benefits, VA activities, and resources

for further information.

M A N A G E M E N T B R I E F Number 7, April 1998, Page 2