Best Practices for Writing an Evaluation Plan

November 7th, 2019 
Presenters

• Evaluation Technical Assistance Team

• Carrie Markovitz, PhD

• Kristen Neishi, MA

• Kim Nguyen, PhD
Learning objectives

• Understand what an evaluation plan is and the purpose of developing one

• Identify key sections of an evaluation plan

• Understand what information to include in an evaluation plan
What is an evaluation plan?

• Details the program model being evaluated

• Describes and justifies the evaluation approach selected

• Provides instructions for the evaluation / a guide for each step of the evaluation 

process
Purpose of an evaluation plan

• Helps decide what information is needed to address the evaluation objectives

• Helps identify methods for obtaining and analyzing the needed information

• Helps anticipate potential challenges 

• Helps determine a reasonable and realistic timeline for the evaluation

• Creates a shared understanding between stakeholders (e.g., the grantee staff, 

evaluator, CNCS staff)
Key components of a plan

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
I. Theory of change

• Describe how the activities 

undertaken by your program 

contribute to a chain of results that 

lead to the intended outcomes

• Your evaluation plan must align with 

your theory of change

Program

context

Issue/Problem

Strategies/ 

Activities

Long-term

outcomes Intermediate

outcomes

Short-term

outcomes

Logic

model Theory of Change

elements
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
II. Outcome(s) of interest

• Describe what outcomes your evaluation will measure

• Process / implementation outcomes

• Program beneficiary outcomes

• Member outcomes 

• Your outcomes of interest should be:

• Part of your program’s theory of change

• Feasible for your program to measure given the source(s) of data needed and level of effort 

required
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
III. Research questions

• One or more questions that define exactly what your evaluation intends to 

accomplish

• Characteristics of a good research question:

• Clearly stated and specific

• Aligned with your theory of change / logic model

• Measurable and feasible to answer
III. Research questions

Research questions are worded differently depending on their focus:
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
IV. Evaluation design

Description of the type of evaluation design that will be used to answer your 

research questions

Process Evaluation Outcome/Impact Evaluation 

• Examines the extent to which a program is operating as 

intended by assessing ongoing program operations and 

determining whether the target population is being 

served

• Results may be used to determine what changes and/or 

improvements should be made to the program’s 

operations

• Measures changes in knowledge, attitude(s), behavior(s) 

and/or condition(s) that may be associated with or 

caused by the program

• Results may demonstrate what the program has 

achieved and/or its outcome or impact on beneficiaries or 

other stakeholder groups
IV. Evaluation design

Type of design Details needed on evaluation design

Experimental 

design/Randomized Controlled 

Trial (RCT)

• Description of the eligibility criteria, random assignment procedures, and monitoring 

procedures for the treatment and control groups

Quasi-experimental design 

(QED)

• Description of the approach for identifying a reasonably similar comparison group (e.g., 

propensity score matching) 

• List of variables (covariates) to be used to statistically equate treatment and comparison 

groups at baseline

Non-experimental design • Description of whether pre- AND post-test measurements OR post-only measurements will 

be used

Process • Description of the methods that will be used (i.e., qualitative only, quantitative only, or 

mixed methods) 
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
V. Sampling methods

For each data source, describe the sample or the target population for the 

evaluation, including: 

• Eligibility criteria that limits the sample or population (e.g., participation level, site/location, 

age or grade level) 

• Sampling procedures (e.g., random, purposeful, or convenience sampling) 

• Expected size of the sample or population 

• Rationale for sample size (e.g., power analysis)
V. Sampling methods

• Power analysis is used to determine:

• How large a sample is needed to enable statistical judgments that are accurate and reliable 

(i.e., required minimum sample size)

• How likely your statistical test will be to detect effects of a given size in a particular situation 

• Your plan must include the results of a power analysis IF:

• An impact evaluation design (i.e. experimental/RCT or QED) 

• Your analysis involves statistical significance testing
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
VI. Data

• Provide a detailed description of the data that will be collected or extracted to 

answer the research questions:

• Who/what will be the source of the data?

• What tools/instruments will be used to collect data? 

• What is the plan for accessing administrative/extant data?

• What information will be collected/compiled? 

• When and how often data will data be collected?

• Ensure that the data are adequate for addressing all of the study's research 

questions
VI. Data

Research Question Data Collection 

Tool

Data Sources Data 

Instruments

Frequency of 

Data Collection

Topics for Data Collection

Do students who participate in the AmeriCorps

intervention demonstrate gains in math and reading 

scores, compared to students who do not 

participate in the intervention?

Administrative Data 

(existing data)

Department of 

Education

Assessment/Tests Fall, Winter, 

Spring

Student math scores

Student reading scores

What factors affect family involvement in the 

AmeriCorps intervention?

Focus Groups Parents Interview Protocol Fall/Spring Family involvement in 

academic activities; 

challenges and facilitators to 

program participation 

Does participation in the AmeriCorps intervention 

improve students’ behavior (social/emotional as 

well as attendance)?

Phone Interviews

Program data 

(existing data)

Teachers Interview Protocol Fall, Spring Student behavior; attendance 

records

Does participation in the AmeriCorps program 

increase members’ professional skills and interest 

in continued civic engagement?

Survey AmeriCorps

Members

Online Survey Fall, Spring Professional skills; future 

goals; program satisfaction

How is the AmeriCorps intervention being 

implemented across sites?

Focus Groups

Phone Interviews

Survey

Parents, Teachers, 

AmeriCorps 

Members

Interview 

Protocols, Online 

survey

Fall Implementation; challenges 

and facilitators
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
VII. Analysis plan

Explain how each data source will be analyzed to produce findings that address the 

evaluation's research questions

Type of design Details needed on analysis

Non-experimental / 

Process evaluation 

design

• The quantitative data analysis techniques that will be used to produce the study findings 

(e.g., Chi-square, t-tests, frequencies, means)

• The qualitative data analysis techniques that will be used to produce the study findings 

(e.g., content analysis, thematic coding) 
VII. Analysis plan

Chi-square tests and t-tests are not adequate for conducting a QED analysis. 

Instead, a multivariate regression model (e.g., ANOVA) is preferred, so covariates 

(e.g., pre-test measures and other variables that may affect the outcome of interest) 

can be controlled for in the analysis. 

Type of design Details needed on analysis

Impact design (RCT

or QED)

• The statistical test/model that will be used to compare outcomes for the treatment and 

comparison groups

• Plans to assess baseline equivalency of the treatment and comparison groups and any 

statistical adjustments to be used (if necessary)
What to include on...

I. Theory of change

II. Outcome(s) of interest

III. Research questions

IV. Evaluation design

V. Sampling methods

VI. Data collection procedures, data 

sources, and measurement tools

VII. Analysis plan

VIII. Timeline

IX. Evaluator qualifications

X. Budget
VIII-X. Timeline, Evaluator, & Budget

• Provide a detailed timeline of when the major evaluation activities will occur (e.g., 

finalize evaluation plan, hire evaluator, develop instruments, collect data, analyze 

data, write report)

• Helps determine if the evaluation is on track to be completed on time (i.e., before the next 

GARP cycle)

• Describe the evaluator(s) who will be carrying out the evaluation activities, 

including

• Whether they are internal or external to the program; and 

• Qualifications for conducting the evaluation

• Specify the budget allotted for the evaluation
General guidelines to follow

• Know what type of evaluation you must complete

• Small vs large grantee requirements

• Fully describe each component of the plan

• Ensure that your description of each of the components aligns with one another 

(i.e., interrelated)

• Know where to go for help

• CNCS’s website

• External evaluator

• Technical assistance portal

• CNCS contact/State Commission representative
Evaluation resources

• CNCS Research and Evaluation general link: 

https://nationalservice.gov/impact-our-nation/research-evaluation

• CNCS Evaluation Policy: 

https://www.nationalservice.gov/resources/evaluation/cncs-evaluation-policies

• CNCS TA portal link:

https://americorpsevaluationta.norc.org/
Questions?