# Presentation: 589743

## Program Evaluation 101

**Juliana M. Blome, Ph.D. , MPH**
**Chief, Office of Program Analysis and Evaluation**
**National Institutes of Health, USDHHS**
**Bethesda, MD**
**April 18, 2008**

**Bridges to the Baccalaureate **
**Technical Assistance Workshop**
**National Institute of General Medical Sciences**

## “Program Evaluation...!?”

**“****Program Evaluation...!?”**

## Evaluation: What is it?   

**   **
**	****Program evaluations are ****individual****, ****systematic ****studies that use ****objective**** ****measurement and analysis to answer specific ****questions**** about how well a program is working.**
**	****	****	****          ****	****	*****- #GAO/GGD-00-204 Program Evaluation***

## Program Evaluation Answers Questions Like.... 

**Does it work?**
**How well does it work?**
**Does it do what we want it to? **
**Does it work for the reasons we think it does? **
**Is it cost effective? **
**Are the benefits worth it?  **
**What are the unintended consequences? **

## Why bother?  

**Supports continuous program improvement**
**Increases understanding of the program – how are activities and strategies linked to results? **
**Leads to improved planning and management**
**Provides shared understanding of program**

## Guidelines for Conducting Successful Evaluations

**Invest heavily in planning early on**
**Use knowledgeable, experienced evaluators**
**Integrate evaluation into ongoing activities of the program**
**Select key questions**

**P****5**

## Typical Evaluations

_**Needs Assessment**_
**What is nature & extent of the issues program should address?**
**Planning phase**
_**Process Evaluation**_
**Is program being conducted & producing output as planned?  **
**How can process can be improved?**
_**Outcome Evaluation**_
**Extent to which a program’s goals have been met? **

**Notes:**
Needs Assessment – Usually looking at needs of stakeholders, developing approp programs goals and how design or modify a program to achieve those goals  / Usually a tool for strategic planning and priority setting

Feasibility study – What’s the best way to evaluate the program? Is this the right time to conduct an evaluation? Can it be conducted at a reasonable cost? Is the program mature enough?—reasonable to expect outcomes at this time? Determine which evaluation design and data collection strategies can and should be used

Process Evaluation – Looking at program operations to determine if being conducted as planned, or whether output being produced, or how processes can be improved.  Often looking at comparison group, a recognized standard of operations 

Outcome evaluation – examine program accomplishments and effects to determine if program meeting intermediate and long-term goals. Often comparing current program performance against prior program performance, comparable control group, or recognized standards of performance

## Sample Study Questions

| Needs 
Assessment | What problem is the program attempting to address?
Whom does this program serve; to what extent are their needs met?
What should be the documented goals of the program?
 |
| --- | --- |
| Process Evaluation | Is the program being implemented as planned?
  If not, why?
How could the program’s processes be improved?
Has the program achieved recognized standards of       performance?
 |

**Sample Study Questions**

**Notes:**
Study Questions are the key questions your evaluation seeks to answer.  They are crucial – what you want to answer determines the rest of your study design, including what data you will need and then how best to collect and present that data.  Study questions should be clear, specific, measurable and answerable.   

In devising study questions, it’s important to note that the type of evaluation corresponds with the type of study question.  For instance, the study questions for a feasibility study will focus on specific ways to determine whether a formal evaluation should take place and how it should be designed.      

## Sample Study Questions

| Outcome Evaluation | To what extent has the program achieved its goals?
Is the current performance different from the past?
Has the program been more successful than a comparable program?
Which characteristics/ activities are most related to success?
What are the intended/ unintended effects of the program? |
| --- | --- |

**Sample Study Questions**

**Notes:**
On the other hand, the study questions for a process evaluation will focus on areas of program management and process; and the study questions for an outcome evaluation will naturally focus on measuring the outcomes from a program.  

##  Why should you care? 

**If you don’t know where you’re going, any road will take you there.**
*	**	**	**     **- Lewis Carroll, Alice in Wonderland*

- -Illustration by Sir John Tenniel, eBooks@ Adelaide, 2004

## Evaluation Standards

**      **_**Utility**_
- 	**Evaluations should serve the practical information needs of a given audience**
_**Feasibility**_
**	****Evaluations take place in the field and should be realistic, prudent, diplomatic and frugal**
_**Propriety**_
**	****The rights of individuals affected by evaluations should be protected**
_**Accuracy**__** **_
**	****Evaluations should produce and convey accurate information about a program’s merit and/or worth**

- For more information on standards, see Joint Cmte on Standards on Educational Evaluation at http://www.wmich.edu/evalctr/jc/

## Key Steps in Evaluation

**Engage stakeholders**
**Describe the program**
**Focus the evaluation design**
**Gather credible evidence**
**Justify conclusions (present data, analysis used, and findings)**
**Ensure use and share lessons**

## 1. Engage Stakeholders

***Who are the stakeholders?***
**	****	**
**	****	**
**	****	****Those involved in program ****	****operations, those affected by the ****	****program operations, and primary ****	****users of evaluation results**

## “I think you should be more explicit here in Step Two.”

**“****I think you should be more explicit here in Step Two.”**
- By Sidney Harris, Copyright    2007, The New Yorker

## 2. Describe the program

**What is the goal and purpose of the program? **
**What problem or need is it designed to address? **
**What are the objectives? What are the strategies to achieve the objectives? **
**What are the expected effects? **
**What are the resources and activities? **
**How is the program supposed to work? **

## 
Activity ≠ Program
You cannot evaluate a program by assessing only an activity

**Activity ****≠**** Program**
**You cannot evaluate a program by assessing only an activity**

**Remember:  **

**Notes:**
What do you really want to know?  

## Model of a Training Program   

**Resources             Activities             Impact**

**Research base**

**Workshops & **
**Seminars**

**Mentoring by **
**faculty member**

**Training in **
**scientific **
**methods**

_**Short term **_
**Knowledge**
**Skills **
**Attitudes**
_**Intermediate**_
**Behaviors**
**Practices **
_**Long term**_
**Enter PhD Program**

-  **Faculty & Staff**

**Money**

**Equipment & Technology**

**What is invested?**

-  **(Inputs)                         (Outputs)                      (Outcomes)**

**What is invested? **

**What is done? **

**What are the changes or benefits? **

## 3. Focus the evaluation design

**What do you want to know? ****(key questions)**
**Who will be involved in or affected by the evaluation or use the findings? ****(stakeholders)****  **
**To focus an evaluation, consider its purpose, uses, questions, methods, roles, budgets, deliverables etc. **
**An evaluation cannot answer **_**all**_** questions **
**for **_**all**_** stakeholders. **

**Notes:**
What do you really want to know?  

## 4. Gather credible evidence

** **

***Evidence must be ***
***believable, trustworthy, and relevant***** **
**Select methodological approach & data collection instruments**
**Determine who is studied and when  **

**Notes:**

What counts as evidence?
Must have evidence seen as trustworthy & relevant 
Depends on questions asked and stakeholders’ views 

## Methodology & Sample

** **

**Qualitative **
**Quantitative**
**Mixed method **
** ****Consider..... **
**sampling design**
**use of comparison groups**
**timing and frequency of data collection**
**issues of bias (sample and respondent)**

**Notes:**

What counts as evidence?
Must have evidence seen as trustworthy & relevant 
Depends on questions asked and stakeholders’ views 

## 5. “Justify” conclusions

**Consider data: **
**Analysis and synthesis  **
**	****- determine findings**
**Interpretation **
**	****- what do findings mean?**
**Judgments**
** ****	****- what is the value of findings based on  accepted standards?**
**Recommendations – **
**	****- what claims can be made?**
**	****- what are the limitations of your design? **

**Notes:**
Linked to evidence  & consistent with agreed upon values or standards

## An evaluation plan should include: 

- Program Description (with goals)
- Purpose of and rational for evaluation
- Evaluation Design (study questions, pop’n, key variables, conceptual framework)
- Data Collection & Analyses (include ethical issues and limitations of data)
- Products of evaluation and their use
- Project Management (implementation, advisory committees, timeline)
- Budget estimate

## If you remember nothing else...

- Evaluation is a _**tool **_to help you make decisions about program management