NORC at the University of Chicago

How to Develop the Right Research Questions for Program Evaluation

1

1

By the end of this presentation, you will be able to:

Understand the importance of research questions

Understand the four basic steps for developing research questions

Write research questions for different types of evaluation designs (i.e., process evaluation and outcome evaluation)

Learning Objectives

  2

Facilitator notes: For this presentation, we have identified a few learning objectives.  

Understand the importance of research questions in program evaluation

Understand the four basic steps for developing research questions 

Write research questions for different types of evaluation designs, specifically process evaluations and outcome evaluations

2

Foundation of a successful evaluation

Define the topics the evaluation will investigate

Guide the evaluation planning process

Provide structure to evaluation activities

Why are Research Questions Important?

Research Questions

Data Collection

Analysis

Conclusions

Questionnaire Design

  3

Facilitator notes: Research questions are often viewed as the central element around which all other evaluation activities are anchored – they are the foundation of a successful evaluation and the key to defining exactly what it is you're trying to better understand. Research questions define the topics an evaluation will investigate and guide the evaluation planning process. Choosing evaluation questions at the start of an evaluation makes clear what your study is trying to understand and/or assess about your program.

During the evaluation, the research questions help provide structure to all other evaluation activities. As depicted in the diagram on this slide, virtually all other evaluation activities should connect back to the research questions. For example, questionnaires are developed and data are collected to answer research questions. Analysis activities should be focused on answering the evaluation’s research questions and conclusions are drawn by directly answering the evaluation’s research questions. 

3

Step 1: Develop a logic model to clarify program design and theory of change

Step 2: Define the evaluation’s purpose and scope

Step 3: Determine the type of evaluation design: process, outcome, or impact

Step 4: Draft and finalize evaluation’s research questions

Steps for Developing Research Questions

  4

Facilitator notes: On this slide, we present four basic steps for developing research questions:

Step 1: Develop a logic model to clarify program design and theory of change

Step 2: Define the evaluation’s purpose and scope

Step 3: Determine the type of evaluation design: process, outcome, or impact

Step 4: Draft and finalize the evaluation’s research questions

4

A logic model is a graphic “snapshot” of how a program works (its theory of change); it communicates the intended relationships among program components.

Inputs, activities, and outputs on the left side of the logic model depict a program’s processes/implementation

Changes that are expected to result from these processes are called outcomes and are depicted on the right side of the logic model

Research questions should test some aspect of the program’s theory of change as depicted in a logic model.

Step 1: Develop a Logic Model to Clarify the Program Design

  5

Facilitator notes: The first step in developing research questions for a program evaluation is to develop a logic model to ensure that there is clarity and a shared understanding of the program design and theory of change among your program’s key stakeholders. Research questions should test some part of the program's theory of change as depicted in a logic model so this is an important first step. Moreover, a well-defined logic model will help you to make decisions about what components of the program  to focus the evaluation on and to select evaluation objectives that are feasible. 

A logic model is a detailed visual representation of your program and its theory of change that communicates how your program works, the resources you have to operate your program, the activities you carry out, and the outcomes you hope to achieve. Your program logic model should clearly communicate how your program works by depicting the intended relationships among program components. Key program components consist of:

Inputs or resources are considered essential for a program’s activities to occur. They may include any combination of human, financial, organizational, and community-based resources that are available to a program and used to carry out a program’s activities. 

Activities are the specific actions that make up your program or intervention. They reflect processes, tools, events, and other actions that are used to bring about your program’s desired changes or results.  

Outputs are what a program’s specific activities will create or produce, providing evidence of service delivery (e.g., the number of beneficiaries served). 

Outcomes are the specific changes that may result from a program’s activities or intervention. A program’s outcomes fall along a continuum, ranging from short- to long-term results (e.g., an increase in knowledge of healthy food choices, a decrease in delinquency rates, or an increase in literacy). A program’s influence is most directly related to short- and medium-term outcomes. It is important to note that all program evaluations are not designed to measure long-term outcomes and AmeriCorps is not expecting programs to measure them in all cases.

Logic models are typically read from left to right, employing an if-then sequence among key components. A generic example is shown here. It reads, if your program has these inputs or resources, then it can carry out these activities. If your program carries out these activities, then it can produce these outputs. If your program has produced these outputs, then it will achieve these outcomes. 

We can think of a logic model as essentially having two “sides”. The process side focuses on how the program’s activities are carried out or its planned work – inputs, activities, and outputs (direct products). The outcomes side of the logic model describes the expected sequence of changes that the program hopes to accomplish, which can be short-term, medium-term, and/or long-term changes. The outcomes side reflects the changes, effects, or impact of the program. 

For more information on developing a program logic model, we encourage you to review the webinar How to Develop a Program Logic Model located on AmeriCorps’s Evaluation Resources page. 

5

Example Logic Model for Health Literacy Program
| INPUTS                                                                                                                                         | ACTIVITIES                                                                                                                                                                 | OUTPUTS                                                                                                                                                                                                                     | Outcomes                                                                                                                                                                                                                         |                                                                                                                                      |                                                                                   |
|------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
|                                                                                                                                                |                                                                                                                                                                            |                                                                                                                                                                                                                             | Short-Term                                                                                                                                                                                                                       | Medium-Term                                                                                                                          | Long-Term                                                                         |
| What we invest                                                                                                                                 | What we do                                                                                                                                                                 | Direct products from program activities                                                                                                                                                                                     | Changes in knowledge, skills, attitudes, opinions                                                                                                                                                                                | Changes in behavior or action that result from participants’ new knowledge                                                           | Meaningful changes, often in their condition or status in life                    |
| Funding  4 FT staff 100 AmeriCorps members serve as health care advisors 10 partnerships with community-based organizations Member training    | Develop and disseminate accurate, accessible, andactionable health and safety informationConduct health literacy workshopsProvide individualized  health literacy sessions | 500 health and safety education materials disseminated4 half-day workshop sessions (at least 20 residents per session; 80 total) 100 individual and small group health literacy sessions (60 mins each) serving 300 people  | Increase in residents’ understanding of prevention and self-managementof conditionsIncrease in residents’ motivation to adopt good health practicesIncrease in residents’ ability to search for and use health information       | Increase in residents’ adoption of healthy behaviors and recommendations of the program  (such as getting necessary medical tests)   | Improved health and wellness status and quality of life for residents in the area |

  6

Facilitator notes: On this slide we present an example of what a logic model might look like for a fictional AmeriCorps health literacy program. This is also in your handout.

The following is a brief narrative description of the program: A disproportionate number of lower socioeconomic and minority groups in an urban region of the state have difficulty understanding and acting upon health information. This issue has negative implications for residents’ healthcare access, costs, quality, and safety. To combat the growing problem of health illiteracy in the county, an AmeriCorps program was created to improve the health literacy and ultimately the health and wellness status and quality of life for residents in the area. 

The logic model we present here is a visual summary of the health literacy program.  We’ll read through this logic model together, starting from left to right. 

On the left side, we begin with the program’s investments, referred to as inputs or resources. For this health literacy program, this includes

Funding

Staff

AmeriCorps members serving as advisors

Partner organizations in the community

Member training

Moving to the next column, if this program has these inputs, then it can carry out a set of its planned activities which include the following:

Develop and disseminate accurate, accessible, and actionable health and safety information

Conduct health literacy workshops

Provide individualized health literacy sessions  

These activities will create or produce the following outputs, which are products or evidence that these activities were carried out:

500 health and safety education materials disseminated

4 half-day workshop sessions ( at least 20 residents per session; 80 total)

100 individual and small group health literacy sessions (60 min each) serving 300 people

As a result of the program’s activities, several changes are expected to occur: 

In the short-term, residents are expected to increase their understanding of prevention and self-management of conditions, their motivation to adopt good health practices, and their ability to search for and use health information.  

These changes in knowledge, attitudes, and skills are expected to lead to an increase in the adoption of healthy behaviors and recommendations of the program  (such as getting necessary medical tests) and ultimately, improved health and wellness status and quality of life for residents in these communities.

We will continue to refer back to this fictional program and logic model throughout this presentation.

6

As you define the evaluation’s purpose and scope, the following questions should be considered:

Why is the evaluation being done? What information do stakeholders need or hope to gain from the evaluation?

What requirements does the evaluation need to fulfill? 

Which components of the program are the strongest candidates for evaluation? 

How does the evaluation align with the long-term research agenda for your program? 

What resources (budget, staff, time) are available for the evaluation?

Step 2: Define the Evaluation’s Purpose and Scope

  7

Facilitator notes: The next step is to use the logic model to set the purpose and scope of the evaluation, which essentially establishes the general parameters for the evaluation. As you define the evaluation’s purpose and scope, the following questions are important to consider:

Why is the evaluation being done? 

What requirements does the evaluation need to fulfill? 

Which components of the program are the strongest candidates for evaluation? 

How does the evaluation align with the long-term research agenda for your program? 

What resources (budget, staff, time) are available for the evaluation?

Next we’ll discuss each of these in more detail.

7

Why is the evaluation being done? What information do stakeholders need or hope to gain from the evaluation?

Each evaluation should have a primary purpose

What requirements does the evaluation need to fulfill? 

Funders may have specific expectations

Which components of the program are the strongest candidates for evaluation?

You do not need to evaluate your whole program at once 

Step 2: Define the Evaluation’s Purpose and Scope

  8

Facilitator notes: 

First, consider: Why is the evaluation being done? What questions do stakeholders need or want answered about your program? 

Each evaluation should have a primary purpose around which it can be designed and planned, although it may have several other purposes. The stated purpose of the evaluation drives the expectations and sets the boundaries for what the evaluation can and cannot deliver. In defining the purpose of the study, it is helpful to identify why the evaluation is being done and how the information collected and reported by the study will actually be used and by whom. For example, are program staff trying to understand how to operate the program more efficiently or identify barriers or constraints to implementation? Or does your program need to produce evidence that it is meeting its intended outcomes? Will the results be used by program staff to make changes to the program’s implementation? Will the results be used by the program’s funder to make decisions about future funding opportunities? In general, defining a specific purpose for your evaluation will allow you to set parameters around the data you collect and methods you will use. 

What requirements does the evaluation need to fulfill? 

Often times, a program’s funder has specific evaluation requirements that must be fulfilled and those requirements may drive the purpose and scope of an evaluation. For example, AmeriCorps State and National grantees that receive an average annual AmeriCorps grant of $500,000 or more must conduct an external evaluation that covers at least one program year. In addition, the evaluation must be designed to provide statistical evidence of the impact of the program compared to what would have happened in the absence of the program through the use of a comparison or control group.

Which components of the program are the strongest candidates for evaluation?  

It is important to remember that you don’t have to evaluate your whole program at once. Instead, you can break a bigger evaluation into smaller components to maximize the resources you have on hand. In deciding which components of the program the evaluation will cover, you should consider:  what parts of your program you want or need to evaluate; whether these potential program components have outcomes you can accurately and reliably measure; what data are already being collected on the program or how feasible it is to collect new data; whether the timeframe for the program aligns with the timeframe for the evaluation; and whether the program model has been fully implemented such that an evaluation is appropriate at this point and time. 

8

How does the evaluation align with the long-term research agenda for your program?

What do you want to know in 5 or 10 years? 

What resources (budget, staff, time) are available for the evaluation?

Evaluation’s scope should align with resources

Step 2: Define the Evaluation’s Purpose and Scope

  9

Facilitator notes: Also consider...

How does the evaluation align with the long-term research agenda for your program?

As you begin to plan your evaluation, step back and put your evaluation in the context of a long term research agenda. Ask program staff and various stakeholders “what does a long-term research agenda look like for this organization?” Figuring out what you want to know 5 or 10 years in the future will help you spend evaluation money more strategically by laying out studies that build upon one another, and will allow you to create complementary resources that can be deployed multiple times. AmeriCorps’s approach to a long-term research agenda emphasizes that evidence of effectiveness is built over time. We discuss this point further on the next slide.

And finally, consider: What resources, that is, budget, staff, and time, are available for the evaluation?

It is important that your evaluation’s purpose and scope aligns with the resources your program is able to set aside for the evaluation. For more information on how to budget for an evaluation, review the webinar “Budgeting for Evaluation” located on AmeriCorps’s Evaluation Resources page.

9

Identify and pilot implement a program design

Attain strong evidence of positive program outcomes

Assess program’s outcomes

Ensure effective full implementation

Obtain evidence  of positive program outcomes

Building Evidence of Effectiveness

Evidence Informed

Evidence Based

- Gather evidence that supports program design

- Develop logic model

- Pilot implementation

Outcome Evaluation

Impact Evaluation

Building Evidence of Effectiveness

  10

Facilitator notes: This diagram illustrates AmeriCorps’s approach to a long-term research agenda which emphasizes a developmental approach to evaluation whereby evidence of effectiveness is built over time. In general, the diagram shows that the evaluation activities associated with a program should be part of a larger, long-term research agenda that spans several years. 

The key building blocks for generating evidence are shown in the diagram. 

The first step is identify and implement a strong program design by gathering evidence that supports the intervention to be used. During this initial process, it is helpful to develop a logic model which clearly communicates the central model of your program. It also is recommended that the program be piloted during this initial step to ensure its effective implementation prior to expanding the program more widely.

Once a strong program design has been identified, the second building block is ensuring the effective full implementation of the program. A process evaluation should be conducted to document program processes, ensure fidelity to the central program model, evaluate program quality and efficiency, and establish continuous process improvement protocols. Much of these activities can be supported through the identification and regular monitoring of performance measures. 

The next level in the continuum is assessing the program’s outcomes. This process involves developing indicators for measuring outcomes, possibly conducting one of the less rigorous outcome evaluation designs, such as a single group pre-post design to measure program outcomes, and conducting a thorough process evaluation. We will discuss what these types of evaluation designs entail later in this presentation. 

One step further in the continuum is obtaining evidence of positive program outcomes by examining the linkages between program activities and outcomes. Programs at this level of the continuum will have performed multiple pre- and post-test evaluations and conducted outcome evaluations using an independent evaluator. 

Finally, the highest level of evidence allows a program to make the claim of being evidence-based by attaining strong evidence of positive program outcomes. At this level, programs have established the causal linkage between program activities and intended outcomes/impacts. Programs at this level have completed multiple independent evaluations using strong study designs, such as a quasi-experimental evaluation using a comparison group or an experimental, random assignment design study. Many of these programs also have measured the cost effectiveness of their program compared to other interventions addressing the same need.

Based on this understanding of a continuum of evidence, AmeriCorps sees value in infusing evaluative thinking and knowledge into every phase of a program’s life cycle – program development, implementation, improvement, and replication/scaling. And AmeriCorps continues to develop a funding strategy that will create a portfolio of programs reflecting a range of evidence levels (e.g., strong, moderate, preliminary) that are appropriate to the program’s life cycle and investment of public dollars. Therefore, in planning for a program’s long-term research strategy, grantees should consider the evaluation activities, design, and research questions that would best correspond to each stage of their program's life cycle. 

10

Performance Measures - Outputs

Process Evaluation

Performance Measures – Outcomes

Process and/or Outcome Evaluation

Step 3: Determine Type of Evaluation: Process, Outcome, or Impact
| Process Evaluation                                                                                                                                                                                                                                                                                | Outcome Evaluation                                                                                                                                                                                                                                                                                                                         | Impact Evaluation                                                                                                                                                                                                                                                                                                                                                                |
|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Goal is generally to inform changes or improvements in the program’s operationsDocuments what the program is doing and to what extent and how consistently the program has been implemented as intendedDoes not require a comparison groupIncludes qualitative and quantitative data collection   | Goal is to identify the results or effects of a program on its beneficiariesMeasures program beneficiaries' changes in knowledge, attitude(s), behavior(s) and/or condition(s) that result from a programMay include a comparison groupTypically requires quantitative data and may use either descriptive or advanced statistical methods | Goal is to estimate the impact of a program on beneficiaries relative to a control or comparison group that receives no or an alternative programMeasures both the changes in program beneficiaries'  knowledge, attitude(s), behavior(s) and/or condition(s) and those same changes in a comparison or control groupRequires quantitative data and advanced statistical methods |

  11

Facilitator notes: Let’s turn back to the steps involved in developing research questions. Once you’ve established the purpose and scope of the evaluation, the next step is to decide which type of evaluation design you will employ. The purpose and scope of your evaluation will determine which type is most appropriate for your study. As you may recall from a previous slide, there are two sides to a logic model – a process side and an outcome side. Similar to what is reflected in the logic model, there are two common types of evaluation designs: a process evaluation design and an outcome evaluation design (which includes impact evaluation designs). Therefore, the side of the logic model that the purpose and scope of your evaluation focuses on will largely determine which type of evaluation design should be used for the evaluation. 

It is important to highlight that process, outcome, and impact evaluations have several notable differences, especially in terms of their goals. The results of a process evaluation are most often used to change or improve the program.  A process evaluation can be used to document what a program is doing and to what extent and how consistently the program demonstrates fidelity to the program’s logic model. An outcome evaluation can be used to determine the results or effects of a program. Outcome evaluations generally measure changes in program beneficiaries' knowledge, attitudes, behaviors, and/or conditions before and after participating in the program. 

To answer the types of research questions associated with a process evaluation generally a comparison group is not necessary. The collection of both qualitative and quantitative data through interviews, surveys, and program administrative data is usually preferred. The more rigorous outcome evaluations include a comparison group against which to measure changes in program beneficiaries. These types of outcome evaluation designs are referred to as impact evaluations. The use of a comparison group provides additional evidence that observed changes in program beneficiaries were due to the program or intervention. Thus, impact evaluations are better able to measure or estimate the impact of the program on beneficiaries relative to a control or comparison group. These types of studies typically require quantitative data collection and often employ advanced statistical methods for analyzing data. 

Remember, AmeriCorps State and National grantees receiving over $500K a year from AmeriCorps are required to conduct an impact evaluation, while those grantees receiving less than $500K a year can conduct either a process evaluation or an impact evaluation. 

Note that this step is about deciding on the basic type of design you will use. After you develop and finalize your research questions, you will determine more specific details associated with your evaluation design. 

11

One or more questions that reflect what your study is trying to understand and/or assess about your program.

The following are characteristics of a good research question:

Clearly stated and specific

Aligns with your theory of change/logic model​

Connects to one or more outcomes of interest

Measurable and feasible to answer​

Aligns to your chosen evaluation design and scope of evaluation

Step 4: Draft and Finalize Evaluation’s Research Questions

  12

Facilitator notes: Once you’ve decided on the type of evaluation design, the next step is drafting and finalizing your research questions. Research questions are a list of questions that you anticipate being able to answer at the end of the evaluation. They reflect what your study is trying to learn about your program. Research questions should be clear, specific, and well-defined so they can be thoroughly answered after completing the other evaluation activities (e.g., questionnaire development, data collection, analysis, etc.). Research questions for a program evaluation should focus on your AmeriCorps program or a component of your program. Again, because the research questions need to eventually be answered, they must be measurable by the evaluation. The research questions also should be aligned with the program’s logic model.  

We will be referring back to these qualities of good research questions at multiple points during the rest of the presentation. 

12

Research questions are worded differently depending on their focus: 

Basic Principles in Designing Research Questions

  13

13

Facilitator notes: The approach for developing research questions differs depending on whether you are conducting a process or an outcome evaluation. As this graphic illustrates, by tying an evaluation’s research questions and design to a program’s logic model, process evaluations address exploratory questions about program operations, processes, and procedures, namely the who, what, when, where, why, and how of program activities and program outputs. On the other hand, outcome evaluations test hypotheses about a program’s changes, effects, or impacts on short-, medium-, or long-term outcomes, ultimately assessing program effectiveness. 

Next, we will provide separate overviews of how to design research questions for a process and an outcome evaluation.

Research questions for a process evaluation should: 

Focus on the program or a program component

Ask who, what, where, when, why, or how?

Use exploratory verbs, such as report, describe, discover, seek, or explore

Basic Principles in Designing Research Questions for a Process Evaluation

  14

Facilitator notes: It is important to remember during the development process that research questions for a process evaluation should: 

Focus on the program or a program component

Ask who, what, where, when, why, or how?

Use exploratory verbs, such as report, describe, discover, seek, or explore

14

Examples of general process questions (more specific sub-questions may be developed):

How is the program being implemented?

How do program beneficiaries describe their program experiences?

What resources are being described as needed for implementing the program?

Template for Developing General Research Questions: Process Evaluation

[Who, what, where, when, why, how] is the [program, model, component] for [evaluation purpose]?

  15

Facilitator notes: For a process evaluation, it is best to start by considering what is the general question you want to answer. General questions should follow from the purpose and goals already identified for the evaluation and ask the who, what, when, where, why, and how associated with program activities and outputs. The questions should focus on studying some topic for general exploration and use neutral language that does not assume a particular outcome from the evaluation.  

Examples of general research questions for a process evaluation are: 

How is the program being implemented?

How do program beneficiaries describe their program experiences?

What resources are being described as needed for implementing the program?

More specific sub-questions may be developed under each of these general research questions. 

Despite the fact that we provide a template here, it is important to keep in mind that there is no specific formula for writing research questions; however, in reviewing and assessing drafts of research questions, it is important to keep in mind the basic requirements described earlier. That is, research questions should be clear and well-defined, focus on a single, distinct program, model, or program component, be measureable, and align with your logic model. Once you have drafted several potential research questions, it is important to ask whether they meet all of these requirements. 

It is often the case that programs are able to identify far more research questions for examination in both process and outcome evaluations than they can explore in a single study. In these circumstances, it is useful to consider how the research questions align with your program’s long-term research agenda, as described earlier. Although you may not be able to study all of your questions about your program at once, it is possible to associate certain questions with later stages of your program development and address these research questions in a future evaluation of your program. 

15

Examples of Research Questions for a Process Evaluation
| Broad                                 | to | More Specific                                                                                                                       |
|---------------------------------------|----|-------------------------------------------------------------------------------------------------------------------------------------|
| How is the program being implemented? |    | Are staff implementing the program within the same timeframe?                                                                       |
|                                       |    | Are staff implementing the program with the same intended target population?                                                        |
|                                       |    | What variations in implementation, if any, occur by site? Why are variations occurring? Are they likely to effect program outcomes? |
|                                       |    | Are there unique challenges to implementing the program by site?                                                                    |

  16

Facilitator notes: Because the general research questions for a process evaluation are often broad, they usually need to be broken down into more specific questions that will then actually guide the other activities of the evaluation toward collecting and analyzing more concrete information or data on the program. Often there is more than one type of evidence that should be considered in answering a general research question, so sub-questions may be written so that they focus on examining one dimension or point of view for informing the general question. Many times these sub-questions form the basis for the actual instruments used to collect data, such as interview guides, surveys, questionnaires, and observational tools. 

Here we provide several examples of sub-questions that can be used to further inform general research questions for a process evaluation. The first example of a general question is “How is the program being implemented?” To further break this very broad question down into the actual areas of focus for the evaluation, several potential sub-questions are offered, including:

Are staff implementing the program within the same timeframe? 

Are staff implementing the program with the same intended target population?

What variations in implementation, if any, occur by site? Why are variations occurring and are they likely to effect program outcomes?

Are there unique challenges to implementing the program by site?

16

Examples of Research Questions for a Process Evaluation
| Broad                                                                      | to | More Specific                                                                |
|----------------------------------------------------------------------------|----|------------------------------------------------------------------------------|
| How do program beneficiaries describe their program experiences?           |    | What are the benefits for program beneficiaries?                             |
|                                                                            |    | Are there any unintended consequences of program participation?              |
| What resources are being described as needed for implementing the program? |    | What recommendations do program staff offer for future program implementers? |

  17

Facilitator notes:  Another example of a general question is “How do program beneficiaries describe their program experiences?” To address this more general question, some potential sub-questions include:

What are the benefits for program beneficiaries?

Are there any unintended consequences of program participation? 

A final example of a general question is “What resources are being described as needed for implementing the program?” A potential sub-question for this general research question is “What recommendations do program staff offer for future program implementers?”

17

Research Questions Checklist
| ​   | Clearly stated and specific​                                                                          |
|-----|-------------------------------------------------------------------------------------------------------|
|     | Aligns with your theory of change/logic model​                                                        |
| ​   | Connects to the outcomes of interest                                                                  |
| ​   | Measurable and feasible to answer​Aligns with your chosen evaluation design and scope of evaluation   |

  18

Facilitator notes:

-Similar to when drafting general research questions, after sub-questions have been drafted, it is important to ask whether the sub-questions meet all of the basic requirements for research questions, namely clearly state and specific, aligns with theory of change and logic model, connects to the outcomes of interest, measurable and feasible to answer, and aligns with your chosen evaluation design and scope of evaluation. 

18

General research question: Is the program being implemented as intended?

Assess whether each of the following is a good sub-question for the process evaluation:

Are all AmeriCorps members engaged in delivering health literacy activities?

To what extent are AmeriCorps members receiving the required training and supervision?

Are program participants more likely to adopt preventive health practices than non-participants?

To what extent are community partners faithfully replicating the program in other states?

Exercise #1: Assessing Potential Research Questions for a Process Evaluation 

  19

Facilitator notes: (Facilitated Exercise) Now that we have presented examples of the types of questions that may be asked for a process evaluation, let’s apply these ideas to a set of potential research questions for the fictional AmeriCorps health literacy program described earlier. Please use the handouts that have been provided on the program’s description and its accompanying logic model as a reference as we walk through some potential research questions for a process evaluation together. 

For this exercise, the fictional AmeriCorps health literacy program is a small AmeriCorps State and National grantee receiving less than $500,000 annually and is required to conduct an evaluation of their program. The grantee has never conducted an evaluation of AmeriCorps-funded activities before and is interested in learning about how their program is working and what can be improved. In order to do so, the program plans to conduct a process evaluation to answer the following general research question: Is the program being implemented as intended?  

Assess whether each of the following is a good sub-question for the program’s process evaluation. For questions that could be improved, what kinds of changes would need to be made? To guide your assessment, consider the following questions discussed earlier:

Are the research questions clear, specific, and well-defined, such that they can be answered at the end of the evaluation?

Do the research questions appear to be measurable and feasible to answer?

Do the research questions focus on the program or a program component that connects to the outcomes of interest? 

Do the research questions align with the program’s logic model and theory of change?

Do the research questions align with your chosen evaluation design and scope of evaluation?

Sub-question 1

Are all AmeriCorps members engaged in delivering health literacy activities? Is this a good research question?  Why or why not?

Discussion: This question is too vague to provide answers to the overarching question of whether the program is being implemented as intended. While all AmeriCorps members may be found to participate in a variety of activities related to health literacy, those activities may or may not be specified in the central program model. Members may also be serving different populations that are not targeted by the program. The question needs to focus on the program itself in order to provide helpful information about the program’s delivery of service activities.  

How can we reformulate this question to answer the broader research question? 

One possibility is to ask instead: To what extent are AmeriCorps members consistently implementing the program with the same target population across all sites? This question is clear, specific, well-defined, and measurable. 

Sub-question 2 

To what extent are AmeriCorps members receiving the required training and supervision? Is this a good research question?  Why or Why not?

Discussion: This is a good research question. The question is clear, specific, well-defined, and measurable. It also addresses the general research question. 

Sub-question 3

Are program participants more likely to adopt preventive health practices than non-participants? Is this a good research question? Why or why not?

Discussion: This is not an appropriate question for a process evaluation for two main reasons. First, this question does not further inform the general research question about fidelity of program implementation that is being asked. Second, as noted earlier, process evaluation questions should use neutral rather than predictive language. This question is more appropriate for an outcome evaluation.  

Sub-question 4

To what extent are community partners faithfully replicating the health literacy program in other states? Is this a good research question? Why or why not?

Discussion: This is not a good research question for this program. Although this question addresses fidelity of implementation, the question is not aligned with the program’s logic model. The goals of the program do not concern replication by community partners. In brainstorming ideas for research questions, sometimes program staff may raise interesting ideas that they are interested in investigating further. These questions may be worth saving for a future, separate evaluation, but not one that focuses on whether the program itself is being executed as planned. This does not mean, however, that questions about community partners cannot be asked at all. The question simply needs to be tailored to the overarching research question.    

How can we reformulate this question? 

One possibility is to ask instead: What variations in community partners’ participation, if any, occur by site? In addition to being clear, specific, and well-defined, answers to this question will also allow the program to learn more about how community partners engage with the program, including any activities related to replication. 

In completing this exercise, you will find that sometimes developing the right research questions simply requires targeting and focusing existing questions that programs may have. By ensuring that research questions are clear, specific, well-defined, measureable, focus on the program, and align with your program logic model as well as your overarching research question, you will be able to obtain answers that can help inform and improve your particular program. 

[Alternate exercise if running short on time – ask “Which of these is a good research question, and why?]

19

Are all AmeriCorps members engaged in delivering health literacy activities?

Too vague

Better: To what extent are AmeriCorps members consistently implementing the program with the same target population across all sites?

To what extent are AmeriCorps members receiving the required training and supervision?

Good question, assuming required training and supervision are defined

Are program participants more likely to adopt preventive health practices than non-participants?

This is not appropriate for a process evaluation 

To what extent are community partners faithfully replicating the program in other states?

Not aligned with program logic model

Better: What variations in community partners’ participation, if any, occur by site?

Exercise #1: Suggested Answers

  20

20

Research questions for an outcome evaluation should: 

Be direct and specific as to the theory or assumption being tested (i.e., program effectiveness or impact)

Examine changes, effects, or impacts

Specify the outcome(s) to be measured aligned to the theory of change

Basic Principles in Designing Research Questions for an Outcome Evaluation

  21

Facilitator notes: Now that we have discussed the process for designing research questions for a process evaluation, next we will discuss the process for designing research questions for an outcome evaluation. For an outcome evaluation, research questions tend to be narrowly focused on confirming a theory or prediction about changes, effects, or impacts of the program on a specified outcome(s). We now will provide a template and examples of research questions for an outcome evaluation. 

21

Did [model, program, program component] have a [change, effect] on [outcome(s)] for [individuals, groups, or organizations]? 

Examples:

Did program beneficiaries change their (knowledge, attitude, behavior, or condition) after program completion?

Did all types of program beneficiaries benefit from the program or only specific subgroups?

Template for Developing Research Questions: Outcome Evaluation

  22

Facilitator notes: For an outcome evaluation, the research questions are used to present a theory about the program’s potential change or effect on a specific outcome or set of outcomes associated with a group of program beneficiaries. Research questions for an outcome evaluation tend to ask whether the model, program, or a specific program component resulted in a predicted outcome for program participants, which is the desired change anticipated by the program.  

As when drafting research questions for a process evaluation, it is important to ask whether research questions for an outcome evaluation meet basic research question requirements, namely are the research questions clear and well-defined, focused on the program or a program component, measureable, and aligned with the program’s logic model?

22

Did [model, program, program component] have an [impact] on [outcome(s)] for [individuals, groups, or organizations] relative to a comparison group?

Comparison between those receiving intervention being studied (program/treatment group) and those not receiving the intervention (comparison/control group)

Compare data on outcomes of interest for both groups at two different time points

Template for Developing Research Questions: Impact Evaluation

  23

Facilitator notes:

Within outcome evaluations are a specific type of evaluation design called impact evaluations. A key element included in the research questions for an impact evaluation is the mention of a comparison group. Impact evaluations include two study groups: the intervention group (i.e., program beneficiaries) and a comparison group (i.e., the study participants that do not receive the intervention or receive alternative program services, and therefore, serve as the basis for comparison when assessing the effects of an intervention on the program beneficiaries). Including a comparison group enables you to answer specific questions related to causality – such as, what would have happened to people if they did not receive the intervention your program offers (i.e., whether the observed changes can be attributed to your intervention). Research questions must compare data on the outcome(s) of interest for both groups (i.e., program/treatment and comparison/control) at two different time points, preferably pre-intervention (baseline) and post-intervention (follow-up).

It is important to note that large AmeriCorps State and National grantees (e.g., those receiving $500k or more per year) are required to complete an impact evaluation on their program. For a more detailed description on differences in types of evaluation design, AmeriCorps State and National grantees can refer to the module, “Overview of Evaluation Design” located on the Evaluation Resources page [previously the Knowledge Network].

23

Example:

Are there differences in outcomes for program participants compared to those not in the program?

Template for Developing Research Questions: Impact Evaluation
|                    | Pre-test | Treatment | Post-test |
|--------------------|----------|-----------|-----------|
| Intervention Group | X        | X         | X         |
| Comparison Group   | X        | O         | X         |

  24

Facilitator notes:

As explained earlier, impact evaluations include two study groups: the intervention group (i.e., program beneficiaries) and a comparison group. Including a comparison group enables you to answer specific questions related to causality – such as, what would have happened to people if they did not receive the intervention your program offers (i.e., whether the observed changes can be attributed to your intervention). 

24

Among parents with a history of involvement with Child Protective Services, were parents who received program services less likely to experience new allegations of maltreatment than parents who did not receive these services? 

Do parents who receive the new, more intensive program services have a lower likelihood of new allegations of maltreatment than parents who receive only the basic program services? 

Impact Evaluation Research Questions

  25

Here we provide some examples of specific research questions that would be most appropriately answered through an impact evaluation because they require closely matched comparison or control groups. [read the examples]

The first question requires a control or comparison group that is external to the program beneficiaries, while the second one involves a comparison between subgroups

25

Do students who participate in the tutoring program achieve grade-level reading proficiency at higher rates relative to a comparison group of students who did not participate in the tutoring program? 

Are wellness program participants more likely to participate in weekly mindfulness practices compared to similar individuals who did not participate in the wellness program? 

Additional Impact Evaluation Research Question Examples

  26

26

For this exercise, use the program’s logic model to identify which outcome(s) to include in the evaluation.

Consider the following:

Which outcome(s) can be achieved within the timeframe of the evaluation (covering at least one year of program activities)?

Which outcomes are feasible to measure? 

What data are already available?

Exercise #2: Developing Research Questions for an Outcome or Impact Evaluation

  27

Facilitator notes: (Facilitated exercise) We are getting near the end of the material, so at this point we will develop together potential research questions for an outcome evaluation of the fictional AmeriCorps health literacy program. Please use the handouts that have been provided on the program’s description and its accompanying logic model as a reference. 

The research questions should be designed to conduct an outcome evaluation of the program to determine whether the program is effective in changing any of its anticipated outcomes, including increasing residents’ understanding of prevention and self-management of conditions, their motivation to change their behavior, their ability to search for and use health information, and their adoption of good health practices. 

For this exercise, the fictional AmeriCorps health literacy program is now a large AmeriCorps State and National grantee receiving more than $500,000 annually and is required to conduct an impact evaluation of their program. To meet funder requirements for this evaluation, data will need to be collected for at least one year. In addition, it is important to note that programs conducting an outcome evaluation do not need to evaluate all program outcomes at one time.

It is helpful to begin by using the logic model of the fictional AmeriCorps health literacy program to determine which outcome(s) will be the focus of the evaluation. As described previously, research questions should test some part of the program's theory of change as depicted in the logic model, so this is an important first step. Since we are designing an outcome evaluation, the logic model is helpful for making decisions about which outcome(s) to focus the evaluation on. In deciding which outcome(s) to include, consider the following:

Of all the outcomes described in the logic model, which outcomes can be achieved within the timeframe of the evaluation, (covering at least one year of program activities)?

Discussion: It is important to consider what questions your evaluation seeks answers to and the timeframe of the evaluation and outcome measures (i.e., the amount of time you have available to collect your data as well as the time you can expect changes or program effects to occur, among other factors). As discussed previously, a program’s influence is most directly related to proximal outcomes, such as short- and medium-term outcomes. Also, all program evaluations are not designed to measure long-term outcomes and AmeriCorps is not expecting programs to measure them in all cases.

For this evaluation then, it would be important to direct our attention to the short- and medium-term outcomes in the logic model. Potential outcomes for the evaluation include: (1) increase in residents’ knowledge of healthy behaviors; (2) increase in motivation to adopt good health practices; (3) increase in ability to search for and use health information; and (4) increase in adoption of healthy behaviors. The long-term outcome of improved health status may take years to realize and would be unrealistic to achieve in the one year timeframe of this evaluation. For that reason, we will not consider the long-term outcome for inclusion in this particular evaluation. 

Which outcomes are feasible to measure?

Discussion: As noted earlier, a logic model describes the underlying theory or logic of how a program is intended to work and how the various components of a program interact to produce results. While the program logic model is designed to help you understand the theory behind this process, not all of the components of a program are easily measured. In the fictional AmeriCorps health literacy program, short- and medium-term outcomes relate to knowledge, motivation, ability, and behavior change. Which of these outcomes would be the most challenging to measure? Of the four, the concept of motivation is an important outcome that contributes to the medium-term change of adopting healthy behaviors; however, motivation itself tends to be a very challenging (but not impossible) concept to measure. Focusing on the other short- and medium-term outcomes that can be more precisely and reliably measured are more likely to strengthen your evaluation’s findings. 

What data are already available?

Discussion: AmeriCorps programs routinely collect data for performance measurement and programs can potentially build on these data for their program evaluation. For example, let’s assume that the fictional AmeriCorps health literacy program already collects performance data indicating whether a change has occurred in participants’ knowledge of healthy behaviors through pre- and post-surveys. For a rigorous impact evaluation, programs can collect the same data from an equivalent comparison group of residents who did not participate in the program. 

In addition to program data, external data such as administrative data may be available and relevant for programs to use. It is important to explore whether information already exists that focus on some of the same short- or medium-term outcomes as the program. It is worth noting that the range of data that are available will likely vary, depending on the program’s focus. For example, while most AmeriCorps programs, including the fictional health literacy program, may look to national or state surveys as possible data sources, education-related programs that operate out of schools may also have school records available to them as an option. 

In all these cases, using existing program or administrative data may also help reduce program evaluation costs, as primary data collection tends to be an expensive evaluation activity.   

27

Outcome evaluation: 

Did [model, program, program component] have a [change, effect] on [outcome(s)] for [individuals, groups, or organizations]?

Impact evaluation:

Did [model, program, program component] have an [impact] on [outcome(s)] for [individuals, groups, or organizations] relative to a comparison group?

Exercise #2: Developing Research Questions for an Outcome or Impact Evaluation

  28

Facilitator notes: Now that we have narrowed possible outcomes to study, our next step is to begin developing potential research questions for an outcome evaluation. You are welcome to divide into small groups for this half of the exercise. 

Here we provide the basic script for developing research questions for an outcome evaluation and for a more rigorous impact evaluation that uses a comparison group. As when drafting research questions for a process evaluation, it is important to ask whether research questions for an outcome evaluation meet basic research question requirements. Again, consider the following when developing your research questions:

Are the research questions clear, specific, and well-defined, such that they can be answered at the end of the evaluation?

Do the research questions appear to be measurable and feasible to answer?

Do the research questions connect to the outcomes of interest? 

Do the research questions align with the program’s logic model and theory of change?

Do the research questions align with the chosen evaluation design and scope of evaluation?

[Optional Facilitator Notes]

Examples of potential research questions for the fictional AmeriCorps health literacy program:

For outcome evaluations that do not include a comparison group:

Did program participants increase their understanding of prevention after program completion?

Did program participants feel more confident in the self-management of their pre-existing conditions after program completion?

Did program participants improve their skills in searching for and using health information after program completion?

Were program participants more likely to search for and use health information on their own after program completion?

For impact evaluations that include a comparison group:

Are program participants more likely to adopt healthy behaviors compared to similar individuals who did not participate in the program?

Are program participants more likely to obtain medical tests and procedures compared to similar individuals who did not participate in the program?

Do program participants that receive more intensive services experience greater impacts compared to a similar group of program participants that receive the general program services?

28

Research Questions Checklist
| ​   | Clearly stated and specific​                                                                          |
|-----|-------------------------------------------------------------------------------------------------------|
|     | Aligns with your theory of change/logic model​                                                        |
| ​   | Connects to the outcomes of interest                                                                  |
| ​   | Measurable and feasible to answer​Aligns with your chosen evaluation design and scope of evaluation   |

  29

Facilitator notes: To review, remember, research questions should be

Clear, specific, and well-defined

Aligned with your logic model and theory of change

Connect to the outcomes of interest

Measurable and feasible to answer

Aligns with your chosen evaluation design and scope of evaluation

29

For outcome evaluations that do not include a comparison group:

Did program participants increase their understanding of prevention after program completion?

Did program participants feel more confident in the self-management of their pre-existing conditions after program completion?

Did program participants improve their skills in searching for and using health information after program completion?

Were program participants more likely to search for and use health information on their own after program completion?

Exercise #2: Suggested Answers

  30

Facilitator notes: Here we present some possible research questions for an outcome evaluation without a comparison group 

30

For impact evaluations that include a comparison group:

Are program participants more likely to adopt healthy behaviors compared to similar individuals who did not participate in the program?

Are program participants more likely to obtain medical tests and procedures compared to similar individuals who did not participate in the program?

Does the impact of the program vary by program participants’ age, gender, or pre-existing medical condition?

Exercise #2: Suggested Answers

  31

Facilitator notes: Here we present some possible research questions for an impact evaluation that includes a comparison group.

[Notes for this last question “Does the impact of the program vary by program participants’ age, gender, or pre-existing medical condition?”  Let’s say your impact evaluation showed that the program is having better results for women than for men. This is an important piece of information, and knowing this you might try to tweak your program model a bit so that you can more effectively serve the male population.]

31

Consider the following:

Do the research question(s) fit with the goals for the evaluation?

Do the research question(s) align with the program’s logic model and the components of the program that will be evaluated?

Are these questions aligned with your funder's requirements?

What kinds of constraints (costs, time, personnel, etc.) are likely to be encountered in addressing these research question(s)?

Do the research questions fit into the program’s long-term research agenda?

Step 4: Draft and Finalize Evaluation’s Research Questions

  32

Facilitator notes: After you have a first draft of your research questions, it is important to ensure that they are the right questions for your evaluative and programmatic goals. It is helpful to conduct a final assessment of your draft research questions by revisiting some important questions that you asked at the beginning of the process for drafting research questions. These questions include:

Do the research question(s) fit with the goals for the evaluation?

Do the research question(s) align with the program’s logic model and the components of the program that will be evaluated?

Are these questions aligned with your funder's requirements?

What kinds of constraints (costs, time, personnel, etc.) are likely to be encountered in addressing these research question(s)?

Do the research questions fit into the program’s long-term research agenda?

32

Research questions are the keystone in an evaluation from which all other activities evolve

Research questions vary depending on whether you will conduct a process vs an outcome evaluation 

Prior to developing research questions, define the evaluation’s purpose and scope and decide the type of evaluation design – process, outcome, or impact.

Research questions should be clear, specific, and well-defined

Research questions should be developed in consideration of your long-term research agenda

Important Points to Remember

  33

Facilitator notes: We’ll end with a few points that are important to remember when developing research questions:

Research questions are the keystone in an evaluation from which all other activities evolve. As discussed earlier, research questions are the central element around which all other evaluation activities are anchored. Research questions should guide the evaluation planning process, help focus and provide structure to an evaluation, and define the issues the evaluation will investigate. 

Research questions vary depending on the evaluation design. The approach for developing research questions differs depending on whether you are conducting a process or an outcome evaluation. Process and outcome evaluations also have different goals and different emphases, which will affect the research questions you draft for your evaluation. For these reasons, it is important to establish which type of design you plan to implement prior to developing your evaluation’s research questions. 

Prior to developing research questions, define the evaluation’s purpose and scope and decide on the type of evaluation design – process, outcome, or impact. Because research questions should test some part of a program's theory of change as depicted in a logic model, there are a number of important steps prior to developing research questions, including using the logic model to define the evaluation’s purpose and scope and determining which type of evaluation design is most appropriate to use (process, outcome, or impact) based on the evaluation’s purpose and scope. It is recommended that these foundational activities be completed prior to developing an evaluation’s research questions. 

Research questions should be clear, specific, and well-defined. Research questions are a list of questions that you anticipate answering at the end of the evaluation, so to the extent possible they should be clear, specific, and well-defined so they can be thoroughly answered after completing the other evaluation activities.

Research questions should be developed in consideration of your long-term research agenda. An important consideration as you finalize your evaluation design and associated research questions is how these evaluation activities best correspond to the current stage of your program's life cycle. As discussed earlier, the evaluation activities associated with a program should be part of a larger, long-term research agenda that spans over several years. Therefore, a program’s evaluation scope, design, and research questions should align with the current stage in which your program operates, such that it helps the program improve and evolve. 

33

ASN’s Evaluation Resources

https://www.nationalservice.gov/resources/evaluation/evaluation-resources

The American Evaluation Association

https://www.eval.org/

The Evaluation Center  

https://wmich.edu/evaluation

The Community Tool Box  

https://ctb.ku.edu/en/evaluating-community-programs-and-initiatives

Choosing the Right Research Questions

https://www.wcasa.org/resources/evaluation/before-you-begin/

Technical Assistance Portal

https://americorpsevaluationta.norc.org/

Resources

  34

Facilitator notes: Here we provide a list of resources on program evaluation and developing research questions that you may find helpful. 

34

Questions and Answers

  35

Facilitator notes: Does anyone have any questions?

35

Carrie E. Markovitz, Ph.D.

NORC at the University of Chicago

markovitz-carrie@norc.org

To contact the Office of Research and Evaluation: evaluation@cns.gov

Thank you!

# image2.emf

# image6.png

# image9.jpeg

# image10.png

# image11.svg

# image12.png

# image3.png

# image4.emf

# image1.png

# image5.png