NORC at the University of Chicago

Basic Steps in Conducting 

an Evaluation

1
By the end of this presentation, you will be able to:

• Describe the basic steps for conducting an evaluation 

• Plan for an evaluation

• Identify the key components of an evaluation plan

• Identify approaches for collecting and analyzing data

• Understand how to communicate and apply findings for program 

improvement

Learning Objectives
What are the basic steps for conducting an evaluation?

• Planning phase

• Development phase

• Implementation phase

• Action and improvement 

Overview of Presentation

3
Evaluation Cycle—Four Phases

4

Planning

Development

Implementation

Action and 

Improvement
Basic Steps for Conducting an Evaluation

5

Phase Step

Planning Step 1: Build (or Review) a Program Logic Model 

Step 2: Define Purpose and Scope 

Step 3: Budget for an Evaluation

Step 4: Select an Evaluator

Development Step 5: Develop an Evaluation Plan

Implementation Step 6: Collect Data

Step 7: Analyze Data

Action and 

Improvement

Step 8: Communicate Findings

Step 9: Apply Findings and Feedback for Program 

Improvement
Planning Phase Steps

6

Build a 

Program 

Logic Model

Define 

Purpose and 

Scope

Select an 

Evaluator

Budget for an 

Evaluation Planning
• A program logic model is a detailed visual representation of a program and its

theory of change.

• Communicates how a program works by depicting the intended relationships 

among program components: 

• Inputs or resources

• Activities

• Outputs

• Outcomes

Step 1: Build a Program Logic Model

7

Inputs Activities Outputs Short Medium Long

Outcomes
• Logic models read from left to right

• There are two “sides” to a logic model - a process side and an 

outcomes side.

Step 1: Build a Program Logic Model 

8
Step 1: Example Logic Model 

9

INPUTS ACTIVITIES OUTPUTS Outcomes

Short-Term Medium-Term Long-Term

What we invest What we do Direct products from 

program activities

Changes in knowledge, skills, 

attitudes, opinions

Changes in behavior or 

action that result from 

participants’ new knowledge

Meaningful changes, 

often in their condition or 

status in life

Funding 

Staff

100 

AmeriCorps 

State and 

National

members

50 nonAmeriCorps 

volunteers

Research

Conduct job 

readiness workshops

Provide job search 

assistance

Provide peer 

counseling services 

Provide referrals to 

transitioning services

Educate potential 

employers

# individuals 

participating in 

workshops

# individuals receiving

job search assistance

# individuals receiving

counseling services

# families receiving 

referrals 

# employers receiving 

education

Increased confidence in 

gaining employment

Increase in job readiness 

skills

Increased knowledge of 

effective job search 

strategies

Increased knowledge of 

community services

Increased employer 

knowledge of hiring 

benefits

Increase in job 

placement

Increased capacity of 

families to manage 

transition from military to 

civilian work and family 

life

Increased adoption of 

military-friendly practices 

by employers

Individuals maintain 

stable employment

Increased family wellbeing

Employers routinely 

hire veterans and 

military spouses

Process Outcomes
Why develop a logic model?

• Generate clear and shared understanding of how program works

• Support program planning and improvement

• Serve as foundation for evaluation 

For an overview of logic models, AmeriCorps grantees can refer to the module, “How 

to Develop a Program Logic Model” located on the Evaluation Resources webpage. 

Step 1: Build a Program Logic Model 

10
• A logic model can serve as a framework for your written evaluation plan. 

• It can help you focus your evaluation by identifying: 

• Questions want/need answered 

• Aspects of program to evaluate 

• Type of evaluation design

• Information to collect 

• Measures and data collection methods 

• Evaluation timeframe

For more information on logic models, AmeriCorps grantees can refer to the module, 

“How to Develop a Program Logic Model” located on the Evaluation Resources 

webpage. 

Step 1: Build a Program Logic Model 

11
Each evaluation should have a primary purpose around which it can 

be designed and planned.

• Why is the evaluation being done? What do you want to learn? 

• How will the results be used? By whom? 

• Additional things to consider: 

• Specific program requirements

• Resources available to carry out the evaluation

Step 2: Define Purpose and Scope

12
Step 2: Define Purpose and Scope

13
Exercise

The hypothetical veterans program is designed to address unemployment among 

veterans and their spouses as well as their transition into civilian work and community life. 

Using the logic model developed for the veterans program, what might be some potential 

research questions? 

Group Exercise: Develop Research Questions for 

a Veterans Job Readiness Program

14
Common cost categories:

• Staff time 

• Materials, equipment, and supplies

• Travel

• Data collection

Step 3: Budget for an Evaluation

15
Consider questions of:

• Who will conduct it?

• If external evaluator, consider what services are and are not included in their cost

• If own staff, consider cost of time spent on evaluation relative to programmatic tasks

• What will it include and how will it be conducted?

• Will it involve new data collection? 

• If so, at what time points and where?

• Who will manage it? 

Step 3: Budget for an evaluation

16
• An evaluator is an individual or team of people responsible for leading 

the evaluation.

• Potential options for an evaluator include:

• An external source (e.g., consulting firm, college or university personnel, 

independent consultant)

• An internal source - program staff member(s)

Step 4: Select an Evaluator

17
A key decision is whether to use an internal staff member or to rely on an 

external evaluator. 

• Consider what expertise is needed

• Scope and complexity may demand expertise outside of your program staff

• Check your funder’s requirements 

• Funders may require an external evaluation to ensure objectivity (e.g., Required for 

AmeriCorps ASN grantees receiving >$500,000 annually)

• Consider financial and staff time implications

• Both types of evaluations have cost implications, whether financial or staff time

Step 4: Select an Evaluator

18
• Certain types of AmeriCorps grantees (i.e., AmeriCorps State and 

National grantees) may be interested in achieving a specific evidence 

tier.

• Preliminary evidence- internal or external

• Moderate evidence- external

• Strong evidence- external

• Consider taking a hybrid approach

• Program and external evaluator share evaluation responsibilities

For more information about these requirements and evidence levels, visit: 

https://americorps.gov/grantees-sponsors/evaluation-resources

Step 4: Select an Evaluator

19
How do you find an external evaluator? 

• Academic settings

• Contact individuals at your local college or university who are not affiliated with the 

program

• Professional settings

• American Evaluation Association (AEA) website, click on “Find an Evaluator” tab 

(https://www.eval.org)

• Personal Networks

• Ask your funder

Step 4: Select an Evaluator

20
Consider whether your potential evaluator has -

• Formal training in evaluation studies

• Experience evaluating similar programs/interventions 

• Experience that matches the design, methods, and/or approach of your planned 

evaluation 

• Capacity to handle the scale of your planned evaluation

• Personal style that fits your program staff or organization

• Ability to conduct an objective and unbiased evaluation (e.g., no conflicts of 

interest, nor appearance of conflicts).

• Experience working with/evaluating AmeriCorps programs specifically 

Step 4: Select an Evaluator

21
Development Step: Developing an Evaluation Plan

22

Development Develop an 

Evaluation Plan
• Details the program model being evaluated

• Describes and justifies the evaluation approach selected

• Provides instructions for the evaluation / a guide for each step of the 

evaluation process

What is an Evaluation Plan?

23
• Helps decide what information is needed to address the evaluation 

objectives

• Helps identify methods for getting the needed information

• Helps determine a reasonable and realistic timeline for the evaluation

• Creates a shared understanding between stakeholders (e.g., the grantee 

staff, evaluator, AmeriCorps staff)

Purpose of an Evaluation Plan

24
Step 5: Develop an Evaluation Plan

25

What should your evaluation plan include?

I. Theory of change

II. Scope of the evaluation

III. Evaluation outcome(s) of 

interest

IV. Research questions

V. Evaluation design

VI. Sampling methods

VII. Data collection procedures, 

data sources, and measurement 

tools

VIII. Analysis plan

IX. Evaluator qualifications

X. Timeline

XI. Budget
I. Theory of Change

• Describe how the activities undertaken by your program contribute to 

a chain of results that lead to the intended outcomes.

Step 5: Develop an Evaluation Plan

26

Theory of Change

elements

Program

context

Sequence of

required events

Underlying

assumptions

Logic

model

Short-term

outcomes

Intermediate

outcomes

Long-term

outcomes
II. Scope of the Evaluation

• State the goal(s) of the evaluation and specify which program 

activities will be assessed

• Programs are not expected to evaluate every component of their 

logic model. Instead, the evaluation may focus on a sub-set of 

program activities.

Step 5: Develop an Evaluation Plan

27
III. Outcome(s) of interest

• Describe what outcomes your evaluation will measure

• Process / implementation outcomes or outputs

• Program beneficiary outcomes

• Member outcomes 

• Your outcomes of interest should be:

• Part of your program’s theory of change and included in your logic model

• Feasible for your program to measure given the source(s) of data needed and level of 

effort required

Step 5: Develop an Evaluation Plan

28
IV. Research Questions

• One or more questions that define exactly what your evaluation intends to 

accomplish

• The following are characteristics of a good research question:

• Clearly stated and specific

• Aligns with your theory of change / logic model

• Connect to the outcomes of interest

• Measurable and feasible to answer 

• Aligns with your chosen evaluation design

Step 5: Develop an Evaluation Plan 

29
V. Evaluation Design 

• Description of general categories of evaluation designs:

Step 5: Develop an evaluation plan

30

Process Evaluation Outcome Evaluation Impact Evaluation

• Examines the extent to which 

a program is operating as 

intended by assessing 

ongoing program operations 

and determining whether the 

target population is being 

served

• Results may be used to 

determine what changes 

and/or improvements should 

be made to the program’s 

operations

• Measures changes in 

knowledge, attitude(s), 

behavior(s) and/or 

condition(s) among 

program beneficiaries or 

other stakeholder groups

• Results may demonstrate 

what the program has 

achieved 

• Measures changes in 

program beneficiaries or 

stakeholders relative to a 

reasonably similar 

comparison/ control group

• Results are an estimate of 

the program’s impact on 

beneficiaries or other 

stakeholder groups
Step 5: Develop an Evaluation Plan

31

AmeriCorps Approved Evaluation Designs

Type of Design Category Details needed on evaluation design

Experimental 

design/Randomized 

Controlled Trial (RCT)

Impact • Description of the random assignment procedures that will 

be used to form treatment and control groups

• Eligibility criteria for inclusion in the study

Quasi-experimental

Design (QED)

Impact • Description of the approach for identifying a reasonably 

similar comparison group (e.g., propensity score matching, 

difference in difference analysis) 

• List of variables (covariates) to be used to statistically equate 

treatment and comparison groups at baseline

Non-experimental 

Design

Outcome • Description of whether pre- AND post-test measurements OR 

post-only measurements will be used

Process Process • Description of the methods that will be used (i.e., qualitative 

only, quantitative only, or mixed methods) 
Step 5: Develop an Evaluation Plan

32

Alternative Evaluation Approach (AEA)

AEA Grantee Justification

Funding threshold Large - Grantees who receive an average of less than $1 million per year 

can request to be exempt from the large grantee requirements 

and conduct an internal non-impact evaluation.

Previous impact 

evaluation

Large - Previously conducted an impact evaluation with demonstrated 

evidence of effectiveness (i.e., Strong or Moderate evidence).

AmeriCorps 

National Evaluation

Large or 

Small

- Grantees participating in an AmeriCorps’ Office of Research and 

Evaluation national evaluation (i.e., bundled evaluation or Return 

on Investment) that will not be completed during current grant 

cycle

- Large grantees can also request this if the national evaluation’s 

design does not fulfill the requirements for a large grantee.
Step 5: Develop an evaluation plan

33

Alternative Evaluation Approach (AEA) - continued

AEA Grantee Justification

Structure of 

program or 

grantee 

organization

Large - Insurmountable challenges forming a comparison group.

- Significant changes to program design.

Replication Large - Implementing an evidence-based intervention with fidelity in a 

new setting.

- A grantee’s application must be assessed at the Strong or 

Moderate evidence level.

Timing Large or 

Small

- Evaluation will not be completed by end of current grant cycle.

- AEA approval required only if an interim evaluation report will not 

meet evaluation requirements.
Implementation Steps: Collecting and Analyzing Data

34

Implementation

Collect Data Analyze 

Data
What type of data meets your evaluation needs?

• Existing data (i.e., secondary data)

• Internal program data (e.g., participant records, program logs, performance 

measurement data)

• External datasets / administrative data (e.g., student records, test scores, medical 

records, test scores, Census data, unemployment insurance claims)

• New data (i.e., primary data)

• Data from surveys, assessments, interviews, and observations

Step 6: Collect Data

35
Step 6: Collect Data

36

Quantitative Methods Qualitative Methods

Scope Less in-depth data across a larger 

number of study subjects

More in-depth data on fewer study subjects

Data 

collection

Standardized instruments with mainly 

closed-ended questions (i.e., 

questions with pre-defined response 

options) such as surveys and multiple 

choice assessments/tests

Standardized instruments and semi-structured 

interview guides mainly with open-ended 

questions (i.e., questions with no pre-defined 

response options) and can be used for 

interview, focus group, and observation 

protocols

Data format Numeric Narrative
Step 7: Analyze Data

37

Quantitative Methods Qualitative Methods

Data 

analysis

Statistical approaches are used to 

summarize the data (frequencies, 

means, crosstabs, regression ) 

Content analysis is often used in which 

themes/patterns in the data are identified, 

categorized, coded, and summarized

Results Results can be compared, and 

generalized to a larger population;

May provide statistical evidence of

program impact

Results provide meaning, illustrative 

explanation, and views of study subject(s); 

NOT able to provide statistical evidence of 

program impact
Step 7: Example Data Collection and Analysis Crosswalk

38

Process Evaluation of a Job Readiness Program for Veterans

Research 

question

Indicators What is collected 

and how?

From whom / data 

sources? 

When collected and 

by whom?

How will you analyze the 

data?

Is the job 

readiness 

program being 

implemented 

as designed? 

a) Member use of 

program 

curriculum during 

workshops 

b) Duration of 

workshops

c) Participant 

workshop rates

a - c) Members 

report details 

about workshops 

in logs with predefined 

categories of 

reporting

a - b) observations 

of workshops

a - c) Members 

a - b) Evaluator 

observes 

participants in 

workshops

a - c) External 

evaluator collects the 

workshop logs 

quarterly

a) Quarterly 

observations by the 

evaluator(s) using 

structured 

observation protocols

a - c) Generate 

frequencies on use of 

curriculum; average 

duration of workshops; 

and average rate of 

workshop attendance

c) Generate frequencies 

and averages on 

quantitative data (e.g., 

ratings scales, frequency 

scales) and thematically 

code and analyze openended comments/notes
Step 7: Example Data Collection and Analysis Crosswalk

39

Impact Evaluation of a Job Readiness Program for Veterans

Research 

question

Outcome of 

interest

What is collected 

and how?

From whom / data 

sources? 

When collected and 

by whom?

How will you analyze the 

data?

Do veterans 

who receive 

the job 

readiness 

intervention 

have a higher 

average 

employment 

rate compared 

to a similar 

group of 

veterans who 

did not receive 

the 

intervention?

Veterans’ 

employment 

status

Veterans’ 

employment status 

is measured with a 

survey. 

Veterans 

participating in the 

program serves as 

the intervention 

group.

Veterans receiving 

job assistance 

services from 

another program 

serve as the 

comparison group. 

The evaluator 

administers the survey 

at two time points:

- before the job 

readiness program 

begins

- 1 year after the job 

readiness program 

is implemented 

Calculate the difference in 

average employment rate 

in the intervention group 

minus the difference in 

average employment rate 

in the comparison group 

before and after 

treatment (difference in 

differences method)
Consider two questions:

• What conclusions about the research questions can be drawn from 

the data that have been analyzed?

• What does the data suggest about the program’s theory of change?

Step 7: Analyze Data

40
Action and Improvement Steps: Reporting and Utilizing 

Results

41

Action and 

Improvement

Communicate 

Findings

Apply Findings and 

Feedback for Program 

Improvement
Who are the potential target audiences? 

• Program staff, agency personnel, stakeholders, beneficiaries, funders, etc. 

What are potential tools for communicating findings? 

• Formal report, shorter memos, PowerPoint briefings, etc. 

Step 8: Communicate Findings

42
• AmeriCorps uses evaluation reports to:

• Identify training and technical assistance needs for grantees

• Identify and share promising practices and models for replication

• Strengthen the evidence base for national service

• Grantees are encouraged to use their evaluation report to:

• Create awareness of and demonstrate success (or lessons learned), and promote 

sustainability

• Identify opportunities for program improvement, adjustment, and future action

• Guide the development of an evaluation plan for upcoming grant cycle

• Support the evidence base for future grant applications

Step 8: Communicate Findings

43
What is an evaluation report?

• A written document that presents the evaluation’s methods and findings 

• Provides a transparent basis for: 

• Understanding the program’s alignment with its theory of change

• Decision-making on policies and programs 

• Drawing lessons for program improvement

Step 8: Communicate Findings

44

Evaluation 

planning 

(Define the purpose 

and methods to be 

used)

Evaluation 

implementation

(Collect and analyze 

data)

Evaluation 

reporting 

(Summarize and 

interpret results)
Step 8: Communicate Findings

45

What are the key sections of an evaluation report?

Executive Summary

Background and Purpose

Evaluation Methods

Results/Findings

Conclusions & Recommendations

References & Appendices
When reporting findings, it is important to: 

• Report positive, as well as negative findings

• Present results that are not necessarily conclusive, but show promise 

and warrant further examination

• Be careful not to overstate your findings

Step 8: Communicate Findings

46
Tips to consider: 

• Develop an outline for your report before starting the writing process 

• Assume the reader has no prior knowledge 

• Make the information visually appealing and easy to read 

• Use diagrams, graphs or charts to highlight central findings

• Take steps to ensure the credibility of your evaluation report

• Proofread your report

Step 8: Communicate Findings

47
Other Useful Products for Communication:

• Executive summary of final report (5-10 pages)

• Short research briefs (2-4 pages)

• Graphics and pictures

• Bulleted information

• Non-technical memos

Step 8: Communicate Findings

48
Evaluation findings can support decisions and actions with respect to:

• Program design, implementation and effectiveness

• Program improvement

• Implementing change

Step 9: Apply Findings and Feedback for Program 

Improvement

49
Please visit ASN’s Evaluation Resources for more information:

• https://www.nationalservice.gov/resources/evaluation/evaluation-resources

ASN’s Evaluation Resources include course on many topics, such 

as: 

• Developing a Logic Model

• Developing Research Questions

• Designing an Evaluation

• Managing an External Evaluation

AmeriCorps Evaluation TA Portal: 

https://americorpsevaluationta.norc.org/

Resources on Evaluation

50
Questions?
Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and 

Evaluation: evaluation@cns.gov

Thank you!