How to Develop the Right Researc

Questions for Program Evaluation
Learning objectives

By the end of this presentation, you will be • Understand the importance of research questions• Understand the four basic steps for developing re

questions

• Write research questions for different types of ev

designs (i.e., process evaluation and outcome ev
Why are research questions import

• Foundation of a successful 

evaluation

• Define the topics the 

evaluation will investigate

• Guide the evaluation planning 

process

• Provide structure to evaluation

Research 

Questions

Data 

Collection

Conclusions

Questionnaire

Design

activities
Steps for developing research ques

• Step 1: Develop a logic model to clarify program 

and theory of change

• Step 2: Define the evaluation’s purpose and scop• Step 3: Determine the type of evaluation design: 

or outcome

• Step 4: Draft and finalize evaluation’s research q
Step 1: Develop a logic model to cla

the program design

• A logic model is a graphic “snapshot” of how a pro

works (its theory of change); it communicates the relationships among program components.– Inputs, activities, and outputs on the left side of the logic m

a program’s processes/implementation

– Changes that are expected to result from these processes 

outcomes and are depicted on the right side of the logic m

• Research questions should test some aspect of th

program’s theory of change as depicted in a logic 
Outcomes INPUTS ACTIVITIES OUTPUTS Short-Term Medium-Term 

What we invest What we do Direct products from program Changes in knowledge, Changes in behav

activities skills, attitudes, action that result 

opinions participants’ ne

knowledge

Funding Develop and 500 health and safety Increase in Increase in 

disseminate education materials residents’ residents’ adop

4 FT staff accurate, disseminated understanding of of healthy 

accessible, and prevention and self- behaviors and 

100 actionable health 4 half-day workshop management recommendatio

AmeriCorps and safety sessions (at least 20 of conditions of the program 

members serve information residents per session; (such as getting 

as health care 80 total) Increase in necessary med

advisors Conduct health residents’ tests)

literacy workshops 100 individual and small motivation to adopt

10 partnerships group health literacy good health 

with Provide sessions (60 mins each) practices

community- individualized serving 300 people

based health literacy Increase in organizations sessions residents ability to 

search for and use 

Member health information training

Example logic model for health literprogram
Step 2: Define the evaluation’s purp

and scopeAs you define the evaluation’s purpose and the following questions should be considere• Why is the evaluation being done? What informa

stakeholders need or hope to gain from the evalu

• What requirements does the evaluation need to f• Which components of the program are the strong

candidates for evaluation? 

• How does the evaluation align with the long-term 

agenda for your program? 

• What resources (budget, staff, time) are available 

evaluation?
• Why is the evaluation being done? What info

do stakeholders need or hope to gain from thevaluation?

– Each evaluation should have a primary purpose

• What requirements does the evaluation need 

fulfill? 

– Funders may have specific expectations

• Which components of the program are the st

candidates for evaluation?

– You do not need to evaluate your whole program at 

Step 2: Define the evaluation’s purpand scope
Step 2: Define the evaluation’s purp

and scope• How does the evaluation align with the long-t

research agenda for your program?

– What do you want to know in 5 or 10 years? 

• What resources (budget, staff, time) are avai

the evaluation?

– Evaluation’s scope should align with resources
Building Evidence of Effectiveness

Performance Measures‐ OutputsProcess Evaluation

Performance Measures– OutcomesProcess and/orOutcome Evaluation

Identify and pilot

implement a

program design

Assess

program’s

Ensure effective outcomes

full

implementation

Obtain evidence

of positive

program

outcomes

EvidenceInformed

‐ Gather evidence thatsupports program design

‐ Develop logic model‐ Pilot implementation

Outcome Evaluation
Step 3: Determine type of evaluatio

process or outcome

Process Evaluation Outcome Evalua

• Goal is generally to inform

• Goal is to identify the r

changes or improvements in the effects of

a program

program’s operations

• Measures program ben

• Documents what the program is changes in knowledge, 

doing and to what extent and behavior(s) and/or con

how consistently the program that result from

a prog

has been implemented as

intended

• May include

a comparis

(impact evaluation)

• Does not require

a comparison

group

• Typically require quant

and advanced statistica

• Includes qualitative and

quantitative data collection
Step 4: Draft and finalize evaluation

research questionsResearch questions are a list of questions tanswered at the end of the evaluation.Research questions should be: • Clear, specific, and well-defined• Focus on a program or program component• Measureable by the evaluation• Aligned with your logic model
Basic principles in designing resea

questions

Differences in research questions for proceoutcome evaluations

Changes?Effects?Impacts?

Research questions for process-focused evaluations ask:

Who?What?When?Where?Why?How?

About:

Inputs/resourcesProgram activitiesOutputsStakeholder views

Research questions for outcome-focused evaluations ask about: 

In:(Short-term)KnowledgeSkillsAttitudesOpinions

(Medium-term)BehaviorsActions

(LoCoSta
Basic principles in designing resea

questions for a process evaluationResearch questions for a process evaluatioshould: 

– Focus on the program or a program componen– Ask who, what, where, when, why, or how?– Use exploratory verbs, such as report, describe

discover, seek, or explore
Template for developing general res

questions: process evaluation[Who, what, where, when, why, how] is th[program, model, component] for [evalua

purpose]

?

Examples:

• How is the program being implemented?• How do program beneficiaries describe their prog

experiences?

• What resources are being described as needed f

implementing the program?
Examples of research questions for a 

process evaluation

Broad to More Specific

How is the program

being implemented?

Are staff implementing the prograsame timeframe?

Are staff implementing the progra

same intended target population?What variations in implementatio

occur by site? Why are variations 

Are they likely to effect program o

Are there unique challenges to imthe program by site?
Broad to More Specific

How do program What are the benefits for program beneficiaries?beneficiaries describe

their program

experiences? Are there any unintended consequ

program participation?

What resources are

being described as

needed forimplementing the

What recommendations do prografor future program implementers?

program?Examples of research questions for a process evaluation
Research Questions Checklist
Exercise #1: Assessing potential res

questions for a process evaluation 

General research question: Is the program implemented as intended?Assess whether each of the following is a gsub-question for the process evaluation:• Are all AmeriCorps members engaged in deliveri

literacy activities?

• To what extent are AmeriCorps members receivin

required training and supervision?

• Are program participants more likely to adopt pre

health practices than non-participants?

• To what extent are community partners faithfully 

replicating the program in other states?
Exercise #1: Suggested answers

• Are all AmeriCorps members engaged in delivering health 

activities?

– Too vague– Better: To what extent are AmeriCorps members consistently im

the program with the same target population across all sites?

• To what extent are AmeriCorps members receiving the req

training and supervision?

– Good question, assuming required training and supervision are 

• Are program participants more likely to adopt preventive he

practices than non-participants?

– This is not appropriate for a process evaluation 

• To what extent are community partners faithfully replicating 

program in other states?

– Not aligned with program logic model– Better: What variations in community partners’ participation, if a

by site?
Basic principles in designing resea

questions for an outcome evaluatioResearch questions for an outcome evaluashould: • Be direct and specific as to the theory or assump

being tested (i.e., program effectiveness or impac

• Examine changes, effects, or impacts• Specify the outcome(s) to be measured
Template for developing research 

questions: outcome evaluationDid [model, program, program componehave a [change, effect] on [outcome(s)] f[individuals, groups, or organizations]?Examples: Did program beneficiaries change their (knowled

attitude, behavior, or condition) after program completion?

 Did all types of program beneficiaries benefit fro

program or only specific subgroups?
Template for developing research 

questions: impact evaluationDid [model, program, program componehave an [impact] on [outcome(s)] for [individuals, groups, or organizations] reto a comparison group?Example: Are there differences in outcomes for program 

participants compared to those not in the progra
Exercise #2: Developing research que

for an outcome or impact evaluation

For this exercise, use the program’s logic midentify which outcome(s) to include in the evaluation.Consider the following:• Which outcome(s) can be achieved within the tim

of the evaluation (covering at least one year of practivities)?

• Which outcomes are feasible to measure? • What data are already available?
Exercise #2: Developing research que

for an outcome or impact evaluation

Outcome evaluation: Did [model, program, program componehave a [change, effect] on [outcome(s)] f[individuals, groups, or organizations]

?

Impact evaluation:Did [model, program, program componehave an [impact] on [outcome(s)] for [individuals, groups, or organizations] reto a comparison group?
Research Questions Checklist
Exercise #2: Suggested answers

For outcome evaluations that do not include a comparison group:• Did program participants increase their understanding of prevention after 

completion?

• Did program participants feel more confident in the self-management of th

conditions after program completion?

• Did program participants improve their skills in searching for and using he

after program completion?

• Were program participants more likely to search for and use health inform

own after program completion?

For impact evaluations that include a comparison group:• Are program participants more likely to adopt healthy behaviors compare

individuals who did not participate in the program?

• Are program participants more likely to obtain medical tests and procedur

similar individuals who did not participate in the program?

• Does the impact of the program vary by program participants’ age, gende

medical condition?
Step 4: Draft and finalize evaluation

research questionsConsider the following:• Do the research question(s) fit with the goals for the 

evaluation?

• Do the research question(s) align with the program’s 

model and the components of the program that will be evaluated?

• Are these questions aligned with your funder's require• What kinds of constraints (costs, time, personnel, etc

likely to be encountered in addressing these research question(s)?

• Do the research questions fit into the program’s long-

research agenda?
Important points to remember

• Research questions are the keystone in an evalu

from which all other activities evolve

• Research questions vary depending on whether 

conduct a process vs an outcome evaluation 

• Prior to developing research questions, define th

evaluation’s purpose and scope and decide the tevaluation design – process or outcome.

• Research questions should be clear, specific, and 

defined

• Research questions should be developed in 

consideration of your long-term research agenda
Resources

• CNCS’s Knowledge Network

– http://www.nationalservice.gov/resources/americorps/e

resources-americorps-state-national-grantees

• The American Evaluation Association

– http://www.eval.org

• The Evaluation Center

– http://www.wmich.edu/evalctr/

• The Community Tool Box

– http://ctb.ku.edu/en/table-of-contents/evaluate/evaluat

community-interventions/choose-evaluation-questions

• Choosing the Right Research Questions

– http://www.wcasa.org/file_open.php?id=1045
Questions and Answers