Basic Steps in Conducting an Evaluation

1 
Learning objectives

By the end of this presentation, you will be able to:

• Describe the basic steps for conducting an evaluation 

• Plan for an evaluation

• Identify the key components of an evaluation plan

• Identify approaches for collecting and analyzing data

• Understand how to communicate and apply findings for 

program improvement

2 
Building evidence of effectiveness

3 
Evaluation cycle – Four phases

Planning

Development

Implementation

Action and 

Improvement

4 
Basic steps for conducting an evaluation

Phase Step

Planning Step 1: Build (or Review) a Program Logic Model 

Step 2: Define Purpose and Scope 

Step 3: Budget for an Evaluation 

Step 4: Select an Evaluator

Development Step 5: Develop an Evaluation Plan

Implementation Step 6: Collect Data

Step 7: Analyze Data

Action and 

Improvement

Step 8: Communicate Findings

Step 9: Apply Findings and Feedback for Program 

Improvement

5 
Planning phase steps

Planning 

Build a Program 

Logic Model

Budget for an 

Evaluation 

Define Purpose 

and Scope

Select an Evaluator

6 
Step 1: Build a program logic model

• A logic model can serve as a framework for your written 

evaluation plan. 

• It can help you focus your evaluation by identifying: 

– Questions want/need answered 

– Aspects of program to evaluate 

– Type of evaluation design

– Information to collect 

– Measures and data collection methods 

– Evaluation timeframe

For more information on logic models, CNCS grantees can refer to the module, “How to 

Develop a Program Logic Model” located on the Knowledge Network. 

7 
Step 2: Define purpose and scope

Each evaluation should have a primary purpose 

around which it can be designed and planned.

• Why is the evaluation being done? What do you want to 

learn? 

• How will the results be used? By whom? 

• Additional things to consider: 

– Specific program requirements

– Resources available to carry out the evaluation

8
Step 2: Define purpose and scope

9 
Group exercise: Develop research questions 

for a veterans job readiness program

Exercise 

The hypothetical veterans program is designed to address 

unemployment among veterans and their spouses as well 

as their transition into civilian work and community life. 

Using the logic model developed for the veterans 

program, what might be some potential research 

questions? 
Hypothetical AmeriCorps Veterans Program

 Process Outcomes

INPUTS ACTIVITIES OUTPUTS Outcomes

Short-Term Medium-Term Long-Term

What we 

invest

What we do Direct products from 

program activities

Changes in knowledge, 

skills, attitudes, opinions

Changes in behavior or 

action that result from 

participants’ new 

knowledge

Meaningful changes, 

often in their 

condition or status in 

life

Funding 

Staff

100 

AmeriCorps 

State and 

National

members

50 nonAmeriCorps 

volunteers

Research

Conduct job 

readiness 

workshops

Provide job search 

assistance

Provide peer 

counseling services 

Provide referrals to 

transitioning 

services 

Educate potential 

employers

# individuals 

participating in 

workshops

# individuals 

receiving job search 

assistance 

# individuals 

receiving counseling 

services 

# families receiving 

referrals 

# employers 

receiving education

Increased confidence in 

gaining employment

Increase in job 

readiness skills

Increased knowledge of 

effective job search 

strategies 

Increased knowledge of 

community services

Increased employer 

knowledge of hiring 

benefits

Increase in job 

placement

Increased capacity of 

families to manage 

transition from military 

to civilian work and 

family life

Increased adoption of 

military-friendly 

practices by 

employers

Individuals maintain 

stable employment

Increased family 

well-being

Employers routinely 

hire veterans and 

military spouses
Step 3: Budget for an evaluation 

Common cost categories:

• Staff time 

• Materials, equipment, and supplies

• Travel

• Data collection

12
Step 3: Budget for an evaluation

Consider questions of:

• Who will conduct it? 

– If external evaluator, consider what services are and are not 

included in their cost

– If own staff, consider cost of time spent on evaluation relative to 

programmatic tasks

• What will it include and how will it be conducted?

• Will it involve new data collection? 

– If so, at what time points and where?

• Who will manage it? 

13
Step 4: Select an evaluator

• An evaluator is an individual or team of people 

responsible for leading the evaluation.

• Potential options for an evaluator include:

– An external source (e.g., consulting firm, college or university 

personnel, independent consultant)

– An internal source - program staff member(s)

14
Step 4: Select an evaluator

A key decision is whether to use an internal staff 

member or to rely on an external evaluator. 

• Factors to consider when making this decision:

– Purpose of the evaluation

– Staff workload and expertise

– Program resources (e.g., financial, necessary computer software, 

etc.)

– Specific program requirements (e.g., AmeriCorps grantees ≥ 

$500,000 are required to conduct an independent evaluation)

15
Step 4: Select an evaluator

Evaluator’s independence: 

• No conflicts of interest related to the evaluation

• Able to provide an unbiased assessment of the program’s 

outcomes/impacts 

16
Step 4: Select an evaluator

How do you find an external evaluator? 

• Academic settings

– Contact individuals at your local college or university

• Professional settings

– American Evaluation Association (AEA) website, click on “Find an 

Evaluator” tab (http://www.eval.org)

• Ask others in your network

17
Step 4: Select an evaluator

Consider whether your potential evaluator has - 

• Formal training in evaluation studies

• Experience evaluating similar programs/interventions 

• Experience that matches the design, methods, and/or 

approach of your planned evaluation 

• Capacity to handle the scale of your planned evaluation

• Personal style that fits your program staff or organization

18
Development step: Developing an evaluation 

plan 

Development Develop an 

Evaluation Plan

19
What is an evaluation plan?

An evaluation plan is a written document that 

describes how you will evaluate your program:

• Explains the program model being evaluated

• Provides detailed instructions for the evaluation 

• Describes and justifies the evaluation approach selected 

20
Why develop an evaluation plan?

• Clarifies what direction the evaluation should take based 

on priorities, resources, time, and skills 

• Creates shared understanding of the purpose and use of 

evaluation results

• Fosters program transparency to stakeholders and 

decision makers

• Helps identify whether there are sufficient program 

resources to carry out the evaluation 

• Facilitates smoother transition when there is staff turnover

21
Step 5: Develop an evaluation plan

What should your evaluation plan include?

• I. Introduction

• II. Program background

• III. Research questions

• IV. Evaluation design

• V. Sampling methods, measurement tools, and data 

collection procedures

• VI. Analysis plan

• VII. Reporting results approach

• VIII. Timeline, budget, evaluator qualifications

See the Frequently Asked Questions: Evaluation document on the Knowledge Network 

for more details.

22
Step 5: Develop an evaluation plan

I. Introduction

The introduction is intended to establish the 

context of your planned evaluation. 

It should explain:

• The problem/issue addressed by the program

• Your program’s theory of change

• Purpose of the planned evaluation

• General approach for planned evaluation

23
Step 5: Develop an evaluation plan

II. Program background

This section should provide detail about your 

program model:

It should include: 

• Your program’s theory of change

• Existing research supporting your program’s theory of 

change

• Logic model

• Outcomes of interest that your evaluation will assess

24
Step 5: Develop an evaluation plan

III. Key evaluation research question(s)

Your evaluation plan should list each of your 

research question(s) that will be investigated.

Your research question(s) should be:

• Clearly stated

• Measurable

• Align with your program’s theory of change and logic 

model

25
Step 5: Develop an evaluation plan 

IV. Evaluation design

Your plan should detail your selected evaluation 

design and a rationale for why it will be used.

• When selecting a specific design, consider the following: 

– Which design will provide desired information and/or fulfill program 

requirements?

– How feasible is each option?

– Are there any ethical concerns to choosing a design?

– What are the costs associated with each design option?

26
Step 5: Develop an evaluation plan

Two common types of evaluation designs: 

• Process/Implementation design: 

– Examines how well the program matches its theoretical model

– Confirms what the program actually does

• Outcome/Impact design: 

– Addresses how a program’s activities relate to changes in 

participants or beneficiaries

– Provides evidence as to whether the program causes observed 

changes

27
Step 5: Develop an evaluation plan 

V. Sampling methods, measurement tools, and 

data collection procedures

This section should detail how you will collect or 

compile data for your evaluation by describing:

• What/who are the sources of data

• Types of data to be collected/compiled (e.g., surveys, 

interviews, administrative data)

• Sampling methods (if any)

• When the data will be collected and by whom

• How the data will be analyzed

28
Implementation steps: Collecting and 

analyzing data

Implementation

Collect Data Analyze Data

29
Step 6: Collect data

• Existing data 

– Internal program data 

– External datasets or program/administrative data

• New data

– Develop data collection instruments (interview protocols and/or 

questionnaires)

– Conduct interviews

– Field surveys

30
Step 7: Analyze data

• Quantitative data

– Statistical analysis (mean, median, chi-square, t-test, ANOVA, 

regression, etc.) 

• Qualitative data

– Content analysis (cross-site analysis, theme identification, case 

study descriptions)

31
Step 7: Example data collection and analysis 

crosswalk

Process Evaluation of a Job Readiness Program for Veterans

Research 

question

Indicators What is collected and 

how?

From whom / data 

sources? 

When collected and by 

whom?

How will you analyze the data?

Is the job 

readiness 

program 

being 

implemented 

as designed? 

a) Member use 

of program 

curriculum 

during 

workshops 

b) Duration of 

workshops

c) Participant 

workshop rates

a - c) Members 

report details 

about workshops 

in logs with predefined categories 

of reporting

a - b) observations 

of workshops

a - c) Members 

a - b) Evaluator 

observes participants 

in workshops

a - c) External 

evaluator collects the 

workshop logs 

quarterly

a) Quarterly 

observations by the 

evaluator(s) using 

structured 

observation protocols

a - c) Generate 

frequencies on use of 

curriculum; average 

duration of workshops; and 

average rate of workshop 

attendance

c) Generate frequencies 

and averages on 

quantitative data (e.g., 

ratings scales, frequency 

scales) and thematically 

code and analyze openended comments/notes

32
Step 7: Example data collection and analysis 

crosswalk

Impact Evaluation of a Job Readiness Program for Veterans

Research 

question

Outcome of 

interest

What is collected and 

how?

From whom / data 

sources? 

When collected and by 

whom?

How will you analyze the 

data?

What impact 

does the job 

readiness 

intervention 

have on 

veterans’ 

ability to 

secure and 

maintain 

employment 

relative to a 

comparison 

group?

Veterans’ 

employment 

status

Veterans’ 

employment status 

is measured with a 

survey. 

33
Step 7: Example data collection and analysis 

crosswalk

Impact Evaluation of a Job Readiness Program for Veterans

Research 

question

Outcome of 

interest

What is collected and 

how?

From whom / data 

sources? 

When collected and by 

whom?

How will you analyze the 

data?

What impact 

does the job 

readiness 

intervention 

have on 

veterans’ 

ability to 

secure and 

maintain 

employment 

relative to a 

comparison 

group?

Veterans’ 

employment 

status

Veterans’ 

employment status 

is measured with a 

survey. 

Veterans 

participating in the 

program serves as 

the intervention 

group.

Veterans receiving 

no job assistance 

services serve as 

the comparison 

group. 

The evaluator 

administers the 

survey at two time 

points:

- before the job 

readiness program 

begins

- 1 year after the job 

readiness program 

is implemented 

Calculate the difference in 

average outcome in the 

intervention group minus 

the difference in average 

outcome in the comparison 

group before and after 

treatment (difference in 

differences method)

34
Step 7: Analyze data

Consider two questions:

• What conclusions about the research questions can be 

drawn from the data that have been analyzed? 

• What does the data suggest about the program’s theory 

of change?

35
Action and improvement steps: Reporting 

and utilizing results

Action and 

Improvement

Apply Findings and 

Feedback for Program 

Improvement 

Communicate 

Findings 

36
Step 8: Communicate findings

Who are the potential target audiences? 

• Program staff, agency personnel, stakeholders, 

beneficiaries, funders, etc. 

What are potential tools for communicating 

findings? 

• Formal report, shorter memos, PowerPoint briefings, etc. 

37
Step 8: Communicate findings

What is an evaluation report?

• Key product resulting from evaluation 

• A written document that objectively describes: 

– Program background

– Evaluation purpose, methods, procedures, and limitations

– Evaluation results

– Conclusions and recommendations

– Lessons learned 

– Questions for future research 

38
Step 8: Communicate findings

When reporting findings, it is important to: 

• Report positive, as well as negative findings

• Present results that are not necessarily conclusive, but 

show promise and warrant further examination 

• Be careful not to overstate your findings

39
Step 8: Communicate findings

Other Useful Products for Communication:

• Executive summary of final report (5-10 pages)

• Short research briefs (2-4 pages)

– Graphics and pictures

– Bulleted information

• Non-technical memos

40
Step 9: Apply findings and feedback for 

program improvement 

Evaluation findings can support decisions and 

actions with respect to:

• Program design, implementation and effectiveness

• Program improvement

• Implementing change

41
Basic steps for conducting an evaluation

Phase Step

Planning Step 1: Build (or Review) a Program Logic Model 

Step 2: Define Purpose and Scope 

Step 3: Budget for an Evaluation 

Step 4: Select an Evaluator

Development Step 5: Develop an Evaluation Plan

Implementation Step 6: Collect Data

Step 7: Analyze Data

Action and 

Improvement

Step 8: Communicate Findings

Step 9: Apply Findings and Feedback for Program 

Improvement

42
Resources on evaluation

Go to the National Service Knowledge Network for 

more information:

https://www.nationalserviceresources.gov/evaluati

on-americorps 

Other courses available or coming soon:

• How to Develop a Program Logic Model

• Overview of Evaluation Designs

• How to Write an Evaluation Plan

• Budgeting for Evaluation

• And more in the coming months!

43
Step 7: Example data collection and analysis 

crosswalk

Process or Impact Evaluation of Your AmeriCorps Program

Research 

question

Indicators or 

Outcome of 

interest

What is collected and 

how?

From whom / data 

sources? 

When collected and by 

whom?

How will you analyze the 

data?

44