NORC at the University of Chicago

How to Develop the Right Research 

Questions for Program Evaluation

1
By the end of this presentation, you will be able to:

• Understand the importance of research questions

• Understand the four basic steps for developing research questions

• Write research questions for different types of evaluation designs (i.e., process 

evaluation and outcome evaluation)

Learning Objectives

2
• Foundation of a successful 

evaluation

• Define the topics the evaluation 

will investigate

• Guide the evaluation planning 

process

• Provide structure to evaluation 

activities

Why are Research Questions Important?

Research 

Questions

Data 

Collection

Analysis

Conclusions

Questionnaire 

Design

3
• Step 1: Develop a logic model to clarify program design and theory of 

change

• Step 2: Define the evaluation’s purpose and scope

• Step 3: Determine the type of evaluation design: process, outcome, or 

impact

• Step 4: Draft and finalize evaluation’s research questions

Steps for Developing Research Questions

4
• A logic model is a graphic “snapshot” of how a program works (its theory 

of change); it communicates the intended relationships among program 

components.

• Inputs, activities, and outputs on the left side of the logic model depict a program’s 

processes/implementation

• Changes that are expected to result from these processes are called outcomes and are 

depicted on the right side of the logic model

• Research questions should test some aspect of the program’s theory of 

change as depicted in a logic model.

Step 1: Develop a Logic Model to Clarify the 

Program Design

5
Example Logic Model for Health Literacy Program

INPUTS ACTIVITIES OUTPUTS Outcomes

Short-Term Medium-Term Long-Term

What we invest What we do Direct products from program 

activities

Changes in knowledge, 

skills, attitudes, opinions

Changes in behavior or 

action that result from 

participants’ new knowledge

Meaningful changes, often 

in their condition or status 

in life

Funding 

4 FT staff

100 AmeriCorps 

members serve as 

health care 

advisors

10 partnerships 

with communitybased

organizations 

Member training

Develop and 

disseminate accurate, 

accessible, and

actionable health and 

safety information

Conduct health 

literacy workshops

Provide individualized

health literacy 

sessions

500 health and safety 

education materials 

disseminated

4 half-day workshop sessions

(at least 20 residents per 

session; 80 total)

100 individual and small 

group health literacy sessions 

(60 mins each) serving 300 

people

Increase in residents’ 

understanding of 

prevention and selfmanagement

of conditions

Increase in residents’ 

motivation to adopt good 

health practices

Increase in residents’ 

ability to search for and 

use health information 

Increase in residents’ 

adoption of healthy 

behaviors and 

recommendations of 

the program (such as 

getting necessary 

medical tests)

Improved health and 

wellness status and 

quality of life for 

residents in the area

6
As you define the evaluation’s purpose and scope, the following 

questions should be considered:

• Why is the evaluation being done? What information do stakeholders need or hope 

to gain from the evaluation?

• What requirements does the evaluation need to fulfill? 

• Which components of the program are the strongest candidates for evaluation? 

• How does the evaluation align with the long-term research agenda for your 

program? 

• What resources (budget, staff, time) are available for the evaluation?

Step 2: Define the Evaluation’s Purpose and Scope

7
• Why is the evaluation being done? What information do stakeholders 

need or hope to gain from the evaluation?

• Each evaluation should have a primary purpose

• What requirements does the evaluation need to fulfill? 

• Funders may have specific expectations

• Which components of the program are the strongest candidates for 

evaluation?

• You do not need to evaluate your whole program at once 

Step 2: Define the Evaluation’s Purpose and Scope

8
• How does the evaluation align with the long-term research agenda for 

your program?

• What do you want to know in 5 or 10 years? 

• What resources (budget, staff, time) are available for the evaluation?

• Evaluation’s scope should align with resources

Step 2: Define the Evaluation’s Purpose and Scope

9
Performance 

Measures - Outputs

Process Evaluation

Performance Measures 

– Outcomes

Process and/or 

Outcome Evaluation

Identify and 

pilot implement 

a program 

design

Attain strong 

evidence of positive 

program outcomes

Assess 

program’s 

Ensure effective outcomes

full 

implementation

Obtain 

evidence of 

positive 

program 

outcomes

Building Evidence of Effectiveness

Evidence 

Informed

Evidence 

Based

- Gather evidence 

that supports 

program design

- Develop logic 

model

- Pilot 

implementation

Outcome Evaluation

Impact Evaluation

Building Evidence of Effectiveness

10
Step 3: Determine Type of Evaluation: Process, 

Outcome, or Impact

Process Evaluation Outcome Evaluation Impact Evaluation

• Goal is generally to inform 

changes or improvements 

in the program’s 

operations

• Documents what the 

program is doing and to 

what extent and how 

consistently the program 

has been implemented as 

intended

• Does not require a 

comparison group

• Includes qualitative and 

quantitative data 

collection

• Goal is to identify the results 

or effects of a program on 

its beneficiaries

• Measures program 

beneficiaries' changes in 

knowledge, attitude(s), 

behavior(s) and/or 

condition(s) that result from 

a program

• May include a comparison 

group

• Typically requires 

quantitative data and may 

use either descriptive or 

advanced statistical 

methods

• Goal is to estimate the 

impact of a program on 

beneficiaries relative to a 

control or comparison 

group that receives no or 

an alternative program

• Measures both the changes 

in program beneficiaries' 

knowledge, attitude(s), 

behavior(s) and/or 

condition(s) and those 

same changes in a 

comparison or control 

group

• Requires quantitative data 

and advanced statistical 

methods

11
One or more questions that reflect what your study is trying to understand 

and/or assess about your program.

The following are characteristics of a good research question:

Clearly stated and specific

Aligns with your theory of change/logic model

Connects to one or more outcomes of interest

Measurable and feasible to answer

Aligns to your chosen evaluation design and scope of evaluation

Step 4: Draft and Finalize Evaluation’s Research 

Questions

12
Research questions are worded differently depending on their 

focus: 

Basic Principles in Designing Research Questions

13
• Research questions for a process evaluation should: 

• Focus on the program or a program component

• Ask who, what, where, when, why, or how?

• Use exploratory verbs, such as report, describe, discover, seek, or 

explore

Basic Principles in Designing Research Questions 

for a Process Evaluation

14
Examples of general process questions (more specific subquestions may be developed):

• How is the program being implemented?

• How do program beneficiaries describe their program experiences?

• What resources are being described as needed for implementing the 

program?

Template for Developing General Research 

Questions: Process Evaluation

[Who, what, where, when, why, how] is the [program, 

model, component] for [evaluation purpose]?

15
Examples of Research Questions for a Process 

Evaluation

Broad to More Specific

How is the program 

being implemented?

Are staff implementing the program within the 

same timeframe? 

Are staff implementing the program with the same 

intended target population?

What variations in implementation, if any, occur by 

site? Why are variations occurring? Are they likely 

to effect program outcomes?

Are there unique challenges to implementing the 

program by site?

16
Examples of Research Questions for a Process 

Evaluation

Broad to More Specific

How do program 

beneficiaries describe 

their program 

experiences?

What are the benefits for program beneficiaries?

Are there any unintended consequences of 

program participation? 

What resources are 

being described as 

needed for 

implementing the 

program?

What recommendations do program staff offer for 

future program implementers?

17
Research Questions Checklist

Clearly stated and specific

Aligns with your theory of change/logic model

Connects to the outcomes of interest

Measurable and feasible to answer

Aligns with your chosen evaluation design and 

scope of evaluation

18
General research question: Is the program being implemented as 

intended?

Assess whether each of the following is a good sub-question for the 

process evaluation:

• Are all AmeriCorps members engaged in delivering health literacy activities?

• To what extent are AmeriCorps members receiving the required training and supervision?

• Are program participants more likely to adopt preventive health practices than nonparticipants?

• To what extent are community partners faithfully replicating the program in other states?

Exercise #1: Assessing Potential Research 

Questions for a Process Evaluation 

19
• Are all AmeriCorps members engaged in delivering health literacy activities?

• Too vague

• Better: To what extent are AmeriCorps members consistently implementing the program with 

the same target population across all sites?

• To what extent are AmeriCorps members receiving the required training and 

supervision?

• Good question, assuming required training and supervision are defined

• Are program participants more likely to adopt preventive health practices than nonparticipants?

• This is not appropriate for a process evaluation 

• To what extent are community partners faithfully replicating the program in other states?

• Not aligned with program logic model

• Better: What variations in community partners’ participation, if any, occur by site?

Exercise #1: Suggested Answers

20
Research questions for an outcome evaluation should: 

• Be direct and specific as to the theory or assumption being tested (i.e., 

program effectiveness or impact)

• Examine changes, effects, or impacts

• Specify the outcome(s) to be measured aligned to the theory of 

change

Basic Principles in Designing Research Questions 

for an Outcome Evaluation

21
Did [model, program, program component] have a [change, 

effect] on [outcome(s)] for [individuals, groups, or 

organizations]?

Examples:

• Did program beneficiaries change their (knowledge, attitude, 

behavior, or condition) after program completion?

• Did all types of program beneficiaries benefit from the program or 

only specific subgroups?

Template for Developing Research Questions: 

Outcome Evaluation

22
Did [model, program, program component] have an [impact]

on [outcome(s)] for [individuals, groups, or organizations]

relative to a comparison group?

• Comparison between those receiving intervention being studied 

(program/treatment group) and those not receiving the intervention 

(comparison/control group)

• Compare data on outcomes of interest for both groups at two 

different time points

Template for Developing Research Questions: 

Impact Evaluation

23
Example:

• Are there differences in outcomes for program participants compared to 

those not in the program?

Template for Developing Research Questions: 

Impact Evaluation

Pre-test Treatment Post-test

Intervention 

Group X X X

Comparison 

Group X O X

24
(1) Among parents with a history of involvement with Child 

Protective Services, were parents who received program 

services less likely to experience new allegations of 

maltreatment than parents who did not receive these 

services? 

(2)Do parents who receive the new, more intensive 

program services have a lower likelihood of new 

allegations of maltreatment than parents who receive 

only the basic program services? 

Impact Evaluation Research Questions

25
• Do students who participate in the tutoring program 

achieve grade-level reading proficiency at higher rates 

relative to a comparison group of students who did not 

participate in the tutoring program? 

• Are wellness program participants more likely to participate 

in weekly mindfulness practices compared to similar 

individuals who did not participate in the wellness program? 

Additional Impact Evaluation Research Question 

Examples

26
For this exercise, use the program’s logic model to identify 

which outcome(s) to include in the evaluation.

Consider the following:

• Which outcome(s) can be achieved within the timeframe of the 

evaluation (covering at least one year of program activities)?

• Which outcomes are feasible to measure? 

• What data are already available?

Exercise #2: Developing Research Questions for an 

Outcome or Impact Evaluation

27
Outcome evaluation: 

• Did [model, program, program component] have a [change, effect]

on [outcome(s)] for [individuals, groups, or organizations]?

Impact evaluation:

• Did [model, program, program component] have an [impact] on 

[outcome(s)] for [individuals, groups, or organizations] relative to a 

comparison group?

Exercise #2: Developing Research Questions for an Outcome 

or Impact Evaluation

28
Research Questions Checklist

Clearly stated and specific

Aligns with your theory of change/logic model

Connects to the outcomes of interest

Measurable and feasible to answer

Aligns with your chosen evaluation design and 

scope of evaluation

29
For outcome evaluations that do not include a comparison 

group:

• Did program participants increase their understanding of prevention 

after program completion?

• Did program participants feel more confident in the self-management of 

their pre-existing conditions after program completion?

• Did program participants improve their skills in searching for and using 

health information after program completion?

• Were program participants more likely to search for and use health 

information on their own after program completion?

Exercise #2: Suggested Answers

30
For impact evaluations that include a comparison group:

• Are program participants more likely to adopt healthy behaviors 

compared to similar individuals who did not participate in the 

program?

• Are program participants more likely to obtain medical tests and 

procedures compared to similar individuals who did not participate in 

the program?

• Does the impact of the program vary by program participants’ age, 

gender, or pre-existing medical condition?

Exercise #2: Suggested Answers

31
Consider the following:

• Do the research question(s) fit with the goals for the evaluation?

• Do the research question(s) align with the program’s logic model and the 

components of the program that will be evaluated?

• Are these questions aligned with your funder's requirements?

• What kinds of constraints (costs, time, personnel, etc.) are likely to be 

encountered in addressing these research question(s)?

• Do the research questions fit into the program’s long-term research 

agenda?

Step 4: Draft and Finalize Evaluation’s Research 

Questions

32
• Research questions are the keystone in an evaluation from which all other 

activities evolve

• Research questions vary depending on whether you will conduct a 

process vs an outcome evaluation 

• Prior to developing research questions, define the evaluation’s purpose 

and scope and decide the type of evaluation design – process, outcome, 

or impact.

• Research questions should be clear, specific, and well-defined

• Research questions should be developed in consideration of your longterm research agenda

Important Points to Remember

33
• ASN’s Evaluation Resources

• https://www.nationalservice.gov/resources/evaluation/evaluation-resources

• The American Evaluation Association

• https://www.eval.org/

• The Evaluation Center

• https://wmich.edu/evaluation

• The Community Tool Box

• https://ctb.ku.edu/en/evaluating-community-programs-and-initiatives

• Choosing the Right Research Questions

• https://www.wcasa.org/resources/evaluation/before-you-begin/

• Technical Assistance Portal

• https://americorpsevaluationta.norc.org/

Resources

34
Questions and Answers

35
Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and 

Evaluation: evaluation@cns.gov

Thank you!