NORC at the University of Chicago

Developing a Long-Term 

Research Agenda

1
By the end of this presentation, you will be able to:

• Recognize the importance of building a long-term research agenda

• Identify the various stages in building evidence of a program’s effectiveness

• Understand the key questions to consider prior to developing a long-term research 

agenda for your program

Learning Objectives
PART 1

Defining a Long-term 

Research Agenda
• A long-term research agenda is a series of intentional or planned program 

evaluations and research tools that build towards addressing a research goal.

• Similar to a strategic plan, a research agenda generally spans over several years.

• A research agenda is unique and should be tailored to each individual program.

• A research agenda is a dynamic tool (i.e., a living document) that should be 

revised/updated based on new evidence, shifts in program direction, etc.

What is a Long-term Research Agenda?
Long-term Research Agenda

Research Goal

Program evaluation

Instrument and data collection systems development

Performance measurement
• A research agenda sets clear goals for what program stakeholders want or need 

to know about the program years into the future

• A research agenda defines your destination, then identifies the supporting steps 

that will get you there

• A research agenda continues to build evidence of program effectiveness

• A research agenda demonstrates strategic investment of funds in evaluation 

activities 

Why is it Important to Have a Long-term Research 

Agenda?
• What does a long-term research agenda look like?

• What do we want to have learned 5 years from now? 10 years from now?

• Work backwards: define your destination, then name the supporting steps that will 

get you there

• Each evaluation should build on what you learned previously

• If you invest evaluation money strategically, scarce resources can have a big 

impact

Build a Long-term Research Agenda

7
• AmeriCorps Housing Assistance Program

• Goal: Demonstrate that the program has a positive impact on beneficiaries 

via an impact evaluation or quasi-experimental evaluation.

• Step 1: Collect program data, routinely, on family background characteristics and number 

of families served. 

• Step 2: Process study: Is the program being implemented with fidelity to the model? 

• Step 3: Collect pre/post outcome data each year via annual survey.

• Step 4: In addition to data collected from Steps 1&2, collect long-term outcomes data via 

follow-up survey (1 year post- program) 

• Step 5: Local demand for program services exceeds the services provided by the program, 

so identify other local public and private sources providing services to similar families to serve 

as a comparison group and conduct a quasi-experimental design (QED study).

Example of a Long-term Research Agenda

8
Collect routine program data

Process evaluation

Pre-post outcome data via survey (e.g., nonexperimental outcome evaluation)

Long term outcome data via 

follow-up survey

Quasi-experimental 

design (QED)

Example: Stages in a Long-term Research Agenda

• Program: AmeriCorps program provides housing assistance for low-income families.

• Research Goal: Demonstrate that the program has a positive impact on beneficiaries via a quasiexperimental design (QED)
• Program maturity

• How long the program has been in operation

• Existing evidence base

• Evidence that has already been generated on the program that the long-term research 

agenda should build off

• Funder requirements and other stakeholder needs

• AmeriCorps has evaluation requirements for some of its grantees, and those requirements 

should affect a program’s decisions about its long-term research agenda

• Sometimes the same evaluation can meet the needs and requirements of multiple funders 

What to Consider When Developing a Long-term Research 

Agenda
• Long-term program goals

• A long-term research agenda should be designed to systematically provide information 

that supports a program’s long-term strategic goals 

• Long-term research goals

• Programs should have long-term research goals that relate to building evidence of 

effectiveness over time 

• Evaluation budget

• The amount of the program’s funding base that will set aside for evaluation activities each 

year or each grant period

What to Consider When Developing a Long-term 

Research Agenda
Exercise Part I: Key Considerations in Developing a Long-term 

Research Agenda for Your AmeriCorps Program

Your AmeriCorps program

Program 

maturity 

Existing 

evidence

Funder

requirements

Long-term

program goals

Long-term 

research goals

Evaluation

budget
PART 2

Building Evidence of 

Effectiveness
Stage 1:

Identify a strong 

program design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess program 

outcomes

Stage 2:

Ensure effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Building evidence of effectiveness

Evidence 

Informed

Evidence 

Based

Building Evidence of Effectiveness
Stage 1: What is the Program Model and What is it 

Supposed to do?

Stage 1:

Identify a 

strong 

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes

Stage 2:

Ensure 

effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based

Stage 1:

Identify a

strong 

program

design

Gather evidence to support program design 

(e.g., conduct a literature review and/or needs 

assessment)

Develop logic model

Pilot program
Stage 1: What is the Program Model and What is it 

Supposed to do?

Stage 2:

Ensure 

effective 

implementatio

n

Output performance measurement

Process evaluation

Stage 1:

Identify a 

strong 

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes

Stage 2:

Ensure 

effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based
Stage 1: What is the Program Model and What is it 

Supposed to do?

Stage 3:

Assess

program 

outcomes

Measure program outcomes (e.g., nonexperimental evaluation design) 

Stage 1:

Identify a 

strong 

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes

Stage 2:

Ensure 

effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based
Stage 1: What is the Program Model and What is it 

Supposed to do?

Stage 4:

Obtain

evidence of 

positive

program 

outcomes

Impact evaluation (e.g., quasi-experimental 

design, experimental design)

Stage 1:

Identify a 

strong 

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes

Stage 2:

Ensure 

effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based
Stage 1: What is the Program Model and What is it 

Supposed to do?

Stage 5:

Attain causal 

evidence of 

positive

program 

outcomes

Outcome evaluation

Stage 1:

Identify a 

strong 

program 

design

Stage 5:

Attain causal 

evidence of 

positive 

program 

outcomes

Stage 3:

Assess 

program 

outcomes

Stage 2:

Ensure 

effective 

implementation

Stage 4:

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based
Building Evidence of Effectiveness: 

Stages 1-5

Output 

Performance 

Measurement

Process evaluation

Outcome 

Performance 

Measurement 

Gather evidence that 

supports program 

design.

Develop logic model

Pilot program

Outcome 

Evaluation

Impact 

Evaluation

Stage 1: 

Identify a 

strong 

program 

design

Stage 5: Attain 

causal evidence 

of positive 

program 

outcomes Stage 3: 

Assess 

program 

outcomes Stage 2: Ensure 

effective 

implementation

Stage 4: 

Obtain 

evidence of 

positive 

program 

outcomes

Evidence 

Informed

Evidence 

Based
Stage 1: 

Identify a 

strong 

program 

design

Stage 5: Attain 

strong evidence 

of positive 

program 

outcomes Stage 3: 

Assess 

program 

outcomes Stage 2: Ensure 

effective 

implementation

Stage 4: 

Obtain 

evidence of 

positive 

program 

outcomes

Exercise Part II: Building evidence of effectiveness 

for your AmeriCorps program

Evidence 

Informed

Evidence 

Based

Exercise Part II: Building Evidence of Effectiveness 

for Your AmeriCorps Program
PART 3

Example Scenarios
Stage 1: 

Identify a 

strong 

program 

design

Stage 5: Attain 

strong evidence 

of positive 

program 

outcomes Stage 3: 

Assess 

program 

outcomes Stage 2: Ensure 

effective 

implementation

Stage 4: 

Obtain 

evidence of 

positive 

program 

outcomes

Scenario 1: Building a long-term research 

agenda for a new Senior Companion Program

Evidence 

Informed

Evidence 

Based

Scenario 1: Building a Long-term Research Agenda for 

a New Senior Companion Program
Scenario 1: Logic Model for a Senior Companion 

Program (SCP)

Process Outcomes

INPUTS ACTIVITIES OUTPUTS

OUTCOMES

SHORT-TERM MEDIUM-TERM LONG-TERM

What we invest What we do Direct products from 

program activities

Changes in knowledge, 

skills, attitudes, opinions

Changes in behavior or 

action that result from 

participants’ new knowledge

Meaningful changes, often in 

their condition or status in life

Funding 

4 FT Staff

30 SCP 

volunteers

Training

Provide independent 

living services 

(transportation, 

nutrition/food 

support, assistance 

with medical 

appointments, etc.)

Provide 

companionship 

services

50 older adults received 

transportation services

45 older adults received 

nutrition/food support,

30 older adults received 

assistance with medical 

appointments 

30 older adults received 

companionship services

Older adults experience 

improved capacity for 

independent living.

Older adults experience 

increases in social 

support.

SCP volunteers 

experience increased 

feelings of socialconnectedness.

Older adults experience 

increased social and 

instrumental support.

Older adults experience 

decreases in stress, 

anxiety, and depression.

SCP volunteers 

experience feelings of 

making a positive impact.

Older adults experience 

greater levels of 

independence.

Older adults experience 

increases in overall 

health and well-being.

SCP volunteers 

experience increases in 

overall health.
Scenario 1: Key Considerations in Developing 

a Long-term Research Agenda

New, Senior Companion Program

Program maturity AmeriCorps grantee with no prior years of program implementation. Operating in 

only one community site.

Existing evidence The program’s evidence falls in the first stage on the continuum as it has 

conducted a literature review to determine best practices for implementing core 

service activities. No evaluations have been conducted on the program.

Funder

requirements

AmeriCorps Seniors grantees are required to report national performance 

measures; grantees have no explicit evaluation requirements. 

Long-term

program goals

Achieve full program operation with efficiency and fidelity to the program’s 

central model. Realize all expected program outcomes. 

Long-term 

research goals

Generate data to facilitate program improvements and ensure an efficient, full 

operation of the program’s service activities. Generate data on the program’s 

short- and medium-term outcomes (see logic model). 

Evaluation budget 10% of the program’s annual funding has been set aside for evaluation activities.
Scenario 1: Long-term Research Agenda for a 

Senior Companion Program (SCP)

Evaluation activities Stage of evidence

1 Develop a logic model and a detailed program implementation 

plan. 

1: Identify strong program 

design

2 Create a data system to routinely collect performance 

measurement data and background data on program 

beneficiaries and SCP volunteers. Program staff and SCP 

volunteers begin routine data collection activities.

2: Ensure effective 

implementation and develop 

data collection plan

3 Develop a survey to collect short-term outcome data, focusing 

on beneficiaries’ capacity for independent living and increases in 

social support. SCP volunteers administer pre/post surveys to 

program beneficiaries and analyze data. 

3: Assess program outcomes

4 Conduct an internal process evaluation to determine if the 

program is being implemented with fidelity to the central model. 

Make data-driven adjustments to the program’s implementation 

as needed.

2: Ensure effective 

implementation

5 Conduct a non-experimental outcome evaluation using an 

external evaluator, measuring both short-term and medium-term 

outcomes. 

3: Assess program outcomes
Stage 1: 

Identify a 

strong 

program 

design

Stage 5: Attain 

strong evidence 

of positive 

program 

outcomes Stage 3: 

Assess 

program 

outcomes Stage 2: Ensure 

effective 

implementatio

n

Stage 4: 

Obtain 

evidence of 

positive 

program 

outcomes

Scenario 2: Building a long-term research agenda 

for a large, established AmeriCorps program

Evidence 

Informed

Evidence 

Based

Scenario 2: Building a Long-term Research Agenda 

for a Large, Established AmeriCorps Program
Scenario 2: Example Logic Model for Large, 

Established, Environmental Restoration Program 

INPUTS ACTIVITIES OUTPUTS Outcomes

Short-Term Medium-Term Long-Term

What we invest What we do Direct products from 

program activities

Changes in knowledge, 

skills, attitudes, opinions

Changes in behavior or action 

that result from participants’ new 

knowledge

Meaningful changes, often in 

their condition or status in life

Funding 

Staff

200 

AmeriCorps 

State and 

National

members

200 nonAmeriCorps 

volunteers

Research

Conduct forest

enhancement and 

restoration

Complete up-keep 

activities to enable 

native plants to 

survive

Install 100,000 

native trees and 

shrubs on public

land

Remove 50% of 

invasive plant

species on 10 

forest sites

Increase diversity and 

coverage of native plant 

species 

Reduce presence of 

invasive plant species

Improve habitat spaces for 

wildlife

Increase survival rate of 

native plant species and 

wildlife

Maintain conservation of 

healthy, productive, 

sustainable ecosystems
Scenario 2: Key Considerations in Developing a Longterm Research Agenda

Large, Established ASN Environmental Restoration Program

Program maturity AmeriCorps grantee in its second three-year AmeriCorps grant cycle. Already operating in 

multiple sites and expects to add additional service sites.

Existing evidence Established data collection processes to collect performance measurement output and 

outcome data. Conducted internal process evaluation yielding evidence that the program is 

being implemented with fidelity in most service sites. 

Funder

requirements

Large grantees must conduct an external impact evaluation by the end of their second grant 

cycle. Large grantees are required to submit an impact evaluation report AND an evaluation 

plan for a future evaluation with their re-compete application after completing two or more 

three-year cycles. 

Long-term program 

goals

Achieve and maintain fidelity of program implementation across all existing sites and any new 

service sites. Build stronger evidence of effectiveness to support future requests for higher levels 

of funding to expand program operations.

Long-term research 

goals

Conduct an external impact evaluation to assess the program’s short- and medium-term 

outcomes. Four to six years is the minimum amount of time for program outcomes to be 

realized . For this reason, the grantee will submit a request for an alternative evaluation 

approach for timing considerations.

Evaluation budget 15% of the grantee’s annual funding has been set aside for evaluation activities. Grantee is 

seeking additional outside funding for the impact evaluation.
Scenario 2: Long-term research agenda for large, 

established environmental restoration program

Evaluation activities Stage of 

evidence

Grant cycle

1 Conduct a quasi-experimental design (QED) study using an 

external evaluator, measuring all short- and medium-term 

outcomes over a six-year time frame and relative to a matched 

comparison group of sites (i.e., adjacent non-serviced areas that 

are similar to the pre-restoration conditions at the treatment 

sites). 

5: Obtain 

evidence of 

positive 

program 

outcomes

2+3

2 Continue to collect and analyze output and outcome 

performance measurement data on an annual basis. 

3: Assess 

program 

outcomes

2, 3, 4, etc.

3 Conduct an internal process evaluation focusing on new service 

sites to determine if the program’s new restoration projects are 

being implemented with fidelity to the central model. Make 

data-driven adjustments to the program’s implementation as 

needed.

2: Ensure 

effective 

implementati

on

2
Part 4

Exercise
Stage 1: 

Identify a 

strong 

program 

design

Stage 5: Attain 

strong evidence 

of positive 

program 

outcomes Stage 3: 

Assess 

program 

outcomes Stage 2: Ensure 

effective 

implementatio

n

Stage 4: 

Obtain 

evidence of 

positive 

program 

outcomes

Exercise: Building evidence of effectiveness for 

your AmeriCorps program

Evidence 

Informed

Evidence 

Based

Exercise: Building Evidence of Effectiveness for 

Your AmeriCorps Program
Exercise Part I: Key Considerations in Developing a Long-term 

Research Agenda for Your AmeriCorps Program

Your AmeriCorps program

Program maturity 

Existing evidence

Funder

requirements

Long-term

program goals

Long-term 

research goals

Evaluation budget
Stage 1: 

Identify a 

strong 

program 

design

Stage 5: Attain 

strong evidence 

of positive 

program 

outcomes Stage 3: 

Assess 

program 

outcomes Stage 2: Ensure 

effective 

implementatio

n

Stage 4: 

Obtain 

evidence of 

positive 

program 

outcomes

Exercise Part II: Building evidence of effectiveness 

for your AmeriCorps program

Evidence 

Informed

Evidence 

Based

Exercise Part II: Building Evidence of Effectiveness for 

Your AmeriCorps Program
Exercise Part III: Long-term Research Agenda for Your 

AmeriCorps Program

Evaluation activities Stage of 

evidence

Grant cycle

1 

2

3

4
• A long-term research agenda is a developmental approach to 

evaluation whereby evidence of effectiveness is built over time.

• A long-term research agenda is unique and should be tailored to fit 

each individual program.

• There is value to building evidence at all stages along the 

continuum.

• A long-term research agenda should reflect an iterative process 

where evidence is built gradually over time. 

Important Points to Remember
Key Points to Consider When Developing a Longterm Research Agenda
• AmeriCorps Resources

https://www.americorps.gov/grantees-sponsors/evaluation-resources

• The American Evaluation Association: http://www.eval.org

• The Evaluation Center: http://www.wmich.edu/evalctr/

• Innovation Network’s Point K Learning Center: 

http://www.innonet.org

Resources
Questions?
Instrument and data collection systems development

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and 

Evaluation: evaluation@cns.gov

Thank you!