Laying the Groundwork 

Before Your First Evaluation

1

AmeriCorps and NORC at the University of Chicago
• Understand five critical activities for your first grant cycle

• Recognize how these activities lay the foundation for future evaluation work

• Learn how to plan ahead to ensure that your program achieves key first-cycle 

milestones

Learning Objectives
• Do everything you can to set your program up for success in Year 1 

• Think through risks/threats to implementation and mitigate them

• Begin to develop data collection systems

• Begin to develop performance measurement plans

Pre-Award Time Period
1. Program design and implementation

2. Build and refine data collection systems

3. Performance measurement

4. Staff capacity and responsibilities

5. Evaluation planning

6. Become a learning organization

Key Foundational Elements
Your First Three Years: Program Activities Over Time
Building evidence of effectiveness Building Evidence of Effectiveness

Stage 1:

Identify a strong 

program design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess program 

outcomes

Stage 2:

Ensure effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based
• Goal: Improve program design and ensure effective implementation

• Key components

• Refine logic model

• Assess implementation

• Conduct process evaluation 

• Adjust program model as needed

1. Program Design And Implementation
• Make sure your “model” is actually a model

• Achieve shared agreement about what the model is and is not

• Ensure all pieces of the model are plausibly connected

Refine Your Logic Model
Assessing Your Theory Of Change

Indicate to what extent each statement is true.

Not at all 

true

Somewhat 

true

True Not 

applicable

Theory of Change

There is a coherent, logical program theory. Strategies and activities are designed to address a 

clearly identified and defined problem or need. There is a logical connection between the 

program strategies and activities and the intended outcomes or desired changes. Goals and 

objectives are articulated and attainable with the available resources. (The program has a logic 

model.)

❑ ❑ ❑ ❑

Program participation is clearly defined and distinguishable from nonparticipation. There is no 

ambiguity about who is in the program and who is not.

❑ ❑ ❑ ❑

There is a shared understanding among program leadership and staff about the core elements 

of the program and the context in which the program operates. 

❑ ❑ ❑ ❑

There is agreement across the program leadership and staff as to what the expected program 

outcomes are.

❑ ❑ ❑ ❑

Clear Time Frame for the Program

The intervention has a clearly defined timeframe. ❑ ❑ ❑ ❑

There is a reasonable and shared expectation around the timeframe for when 

observable/measurable outcomes in the short, intermediate or long term will occur. 

❑ ❑ ❑ ❑

Impact Evaluability Assessment Tool. Source: 

https://www.americorps.gov/sites/default/files/document/2015_09_03_ImpactEvaluabilityAssessmentTool_ORE.pdf, “Conducting a Needs or 

Readiness Assessment”. 
• Implementing your program may lead to changes:

• In your logic model

• Some parts may not be plausible

• Some outcomes may not be realistic

• In how your program operates

• Application logic model may be ideal → program model may change as it is 

implemented

• Important to document those changes so that you understand what you are actually 

doing

• Document any variation in implementation

Assess Implementation
Assessing Implementation

Indicate to what extent each statement is true.

Not at all 

true

Somewhat 

true

True Not 

applicable

Program Implementation

If the program is based on a model or logical program theory, it is implemented with fidelity to that 

model and has a well-planned sequence of activities. 

❑ ❑ ❑ ❑

If the program is currently being adapted, it is being adapted using theory/systematically-obtained 

field-based knowledge, and along lines that can be quantified and documented.

❑ ❑ ❑ ❑

Staff members are qualified and properly trained to operate the program. There are enough 

qualified staff members on site to implement the planned activities. 

❑ ❑ ❑ ❑

Frontline workers who deliver the services provided by the program have sufficient qualifications to 

execute the program. There are enough qualified frontline workers on site to successfully execute 

the program.

❑ ❑ ❑ ❑

There are systems in place to track program implementation:

• There are procedures in place to determine if the target population is being served (referral 

system, intake process).

• Data that track service usage is collected (attendance lists, case management logs). 

• Input is sought on a regular basis to understand how participants experience the services and 

to identify and address any problems in a timely manner.

❑

❑

❑

❑

❑

❑

❑

❑

❑

❑

❑

❑

Impact Evaluability Assessment Tool. Source: http://www.nationalservice.gov/resources/evaluation/planningevaluation, “Conducting a Needs or Readiness Assessment”. 
• Goal: Build systems to collect and manage high-quality program data

• Key components

• Select data collection instruments

• Build data collection system

• Build data management system

• Obtain access to administrative data (if applicable)

• Ensure data quality

2. Build and Refine Data Collection Systems
• First, examine logic model and think through data you need to collect

• Outputs

• Outcomes

• Implementation

• Other data beyond performance measures

• Then, develop and test instruments

• Utilize existing resources

• Work with a professional to develop instruments that are valid and reliable

• Good instruments can also be used later for evaluation!

Data Collection System
• Data management system should help you to: 

• Store data

• Access data

• Analyze data

• Use data

• Build (or buy) system to accomplish those goals

• Examples include: Efforts to Outcomes, Design Data, Outcomes Results System...

Data Management System
• Elements of data quality: 

• Valid: data mean what they are supposed to mean

• Complete: everyone reports full set of data

• Consistent: everyone uses same data collection methods

• Accurate: data are free from errors

• Verifiable: everyone follows standard practices and checks their data

• Ensuring data quality means that you describe your program’s 

achievements in a trustworthy manner

Ensuring Data Quality 
• Goal: Collect and report performance measures; use results for 

program improvement

• Key components

• Select performance measures

• Determine measurement frequency

• Adjust performance measures

• Execute performance management processes

• Connect performance management to evaluation 

3. Performance Measurement
• Select measures that reflect what your program is actually doing

• Select measures you can use to understand how your program is 

working

• Select targets that are ambitious yet achievable 

• Adjust measures annually if needed

Measure Selection and Adjustment

Measures you 

collect and 

report for 

AmeriCorps 

grant

Other 

measures

Good 

performance 

measurement 

system!
• Do not just collect and report your 

performance measurement data – you need 

to use it!

• Are goals being met?

• Use data for program improvement

• Continuous feedback loop

• Programmatic decisions are

• grounded in data 

• Strengthen implementation

• Performance Measures Core Curriculum: 

http://www.nationalservice.gov/resources/p

erformance-measurement

Performance Management

Select measures 

and set targets

Collect 

performance 

measures data

Analyze data 

Identify 

programmatic 

improvements

Strengthen 

implementation
• AmeriCorps housing assistance program

• O5: Number of economically disadvantaged individuals, including homeless individuals, 

receiving housing services [Target = 500 calls]

• O11: Number of economically disadvantaged individuals, including homeless individuals, 

transitioned into safe, healthy, affordable housing [Target = 600]

• Actuals: O5 = 900, O11 = 300

• What’s going on? 

• Individuals are calling the hotline multiple times

• Placement process takes longer than anticipated

• Program response? 

• Adjust call process – create a second number for repeat clients

• Adjust target for O11 to reflect reality of placement process

Performance Management Example
• Good performance measurement can be the basis for future 

evaluation activities

• Outcome measures may be extended

• Instruments can be utilized 

• Data can suggest interesting research questions

• Performance measurement does not stop once evaluation begins

Connection to Evaluation
• Goal: Build staff capacity to collect, manage, and utilize data

• Key components

• Develop internal staff skills

• Assign responsibilities 

• Engage external experts

• Become a learning organization

4. Staff Capacity and Responsibilities 
• Develop staff skills in data collection, management, and analysis

• Know enough to know what you know, and know what you don’t 

know

• Staff do not have to be experts, but they need to be educated and 

informed consumers

• Assign responsibilities – someone has to take ownership of key tasks

Internal Staff Capacity
• Utilize external experts for

• Data collection instruments

• Complicated analysis

• Questions about measurement

• Evaluation 

• Consider consultants, pro bono experts, cooperative extension, 

universities

• Talk to other programs and stakeholders

Identify and Work with External Experts
• This checklist is a tool designed to help organizations assess and discuss a program’s 

readiness to participate in a rigorous impact evaluation. It can be used for 

assessment, planning, and communication purposes. 

• The focus areas covered in this checklist are organized into three broad content 

categories: 

• Organization Readiness 

• Program Readiness

• Evaluation Readiness 

• Each category addresses key elements of readiness. Readiness in all three areas 

covered is important for successful planning and implementation of an evaluation.

Exercise: Evaluability Assessment Checklist
• Goal: Develop a plan for your first evaluation, which will occur in your 

second grant cycle (if not earlier).

• Key components

• Select research questions

• Budget for evaluation 

• Develop evaluation plan

• Make design decisions (process or impact, internal or external) based on grantee 

size

5. Evaluation Planning
• Research questions are a list of questions grantees want to learn about 

their program

• Research questions should be: 

• Clear, specific, and well-defined

• Focus on a program or program component

• Measureable by the evaluation

• Appropriate for the design proposed

• Aligned with your logic model

Develop Research Questions
• In general, evaluation budgets should be: 

• Commensurate with stakeholder expectations and involvement 

• Appropriate for the research design used and key questions to be answered 

• Adequate for ensuring quality and rigor 

• In line with the level of program and organizational resources available 

• Budgeting for evaluation is a smart strategic investment

Budget For Evaluation
• Written document that details all he evaluation steps and activities you plan to conduct

• Dynamic tool (i.e., a living document) that should be continually updated as you plan 

and develop each aspect of the evaluation

• Evaluation Plan Outline

I. Theory of change

II. Scope of the evaluation

III. Evaluation outcome(s) of interest

IV. Research questions

V. Evaluation design 

VI. Sampling methods

VII. Data collection

VIII.Analysis plan 

IX. Evaluator qualifications

X. Timeline

XI. Budget 

Write An Evaluation Plan
• A learning organization:

• Reflects on past successes and challenges

• Uses data to inform decision making

• Makes adjustments to programs and processes based on data

• Is not afraid to question assumptions

• Thinks strategically about how to improve

• Builds evidence of effectiveness

• Sets a research agenda for the future

6. Becoming A Learning Organization
• By the end of your first grant cycle, you should have:

• Refined your program and ensured effective implementation

• Built and refined data collection systems

• Utilized accurate performance measures

• Built staff capacity and defined responsibilities

• Prepared a plan for your first evaluation

• Become a learning organization

• And hopefully you’ve also had some fun!

Milestones
Exercise: Laying the Foundation for Your Program
• Go to the Evaluation Resources page for more information:

• https://americorps.gov/grantees-sponsors/evaluation-resources

• Other courses available:

• How to Develop a Program Logic Model

• Overview of Evaluation Designs

• How to Write an Evaluation Plan

• Budgeting for Evaluation

• Data Collection for Evaluation

• Asking the Right Research Questions

• Managing an External Evaluation

• And more!

Resources on Evaluation

32
Questions?
To contact the Office of Research and 

Evaluation: evaluation@cns.gov

Thank you!