## Step 1. Tools to Engage Stakeholders

## 1.1 Stakeholder Mapping Exercise

It is suggested that the program enlist the aid of an evaluation stakeholder workgroup**(ESW)** of 8-10 members that represents the stakeholders who have the greatest stake or vested interest in the quitline evaluation (Centers for Disease Control, 2008). These stakeholders or primary intended users will serve in a consultative role on all phases of the evaluation. To begin the process of selecting those members who will best represent your primary intended users, it is suggested that you compile a list of all possible users with corresponding comments about their investment in the quitline evaluation and potential uses for evaluation results. 

**1.1 Stakeholder Mapping**

Now, go back over your list of potential users of the quitline evaluation results and consider their level of priority on the list. For example, providing the information that funders or decision makers need may take a higher priority than other users, even though all users are still very important. You might rate stakeholders in terms of high, medium, or low, or you might rank order them from 1 to n. 

## 1.2 Evaluation Purpose Exercise

Identifying the end users and the evaluation stakeholder workgroup is as important as identifying the purpose of the quitline evaluation. These two aspects of the evaluation serve as a foundation for evaluation planning, focus, design, interpretation, and use of results. The purpose of an evaluation influences the identification of stakeholders for the evaluation, selection of specific evaluation questions, and the timing of quitline evaluation activities. It is critical that the quitline be transparent about intended purposes of the evaluation. If evaluation results will be used to determine whether a quitline component should be added, continued, or eliminated, then stakeholders should know this up front.

To determine the evaluation purpose, the evaluation team should work with those who are requesting the evaluation to identify the possible multiple purposes for the evaluation from multiple sources. The first task is to consider what groups are interested in an evaluation of the quitline. This might include the quitline service provider staff, health department staff, funders, state-level decision makers, and other stakeholders. The second task would be to align the specific group with what they are requesting to be evaluated. The third task would be to ascertain what the potential uses of the evaluation results will be by each group interested in the evaluation. And fourth, the team should develop a purpose statement relevant to each group and evaluation requested. 

**1.2 Evaluation Purpose**

Next, the team should consider each purpose statement for duplication and overlap. What statements could be combined? The final step in the process is to merge the statements into one overall purpose statement.

## 1.3 Stakeholder Inclusion and Communication Plan Exercise

It is important to explore agendas at the beginning of the quitline evaluation process and come to a shared understanding of roles and responsibilities, as well as the purposes of the evaluation. Some stakeholders will be represented on the evaluation stakeholder workgroup (ESW), and some will not. It is important to include a clear communication plan in the evaluation plan to meaningfully engage all appropriate quitline stakeholders and increase participation and buy-in for the evaluation, as well as use of final results.

For each stakeholder relevant to the evaluation, list their appropriate role and how and when they might be engaged in the evaluation. Consider their expertise, level of interest, and availability when developing the communication plan. If there are specific deadlines for information, such as a funding opportunity or quitline contract rebidding process, note those as well. Additional columns could be added for comments.

****

**1.3 Stakeholder Inclusion and Communication Plan**

A note on roles: Stakeholders need not be a member of the evaluation stakeholder workgroup to have a role related to the evaluation. Given a stakeholder’s specific expertise, interest, availability, or intended use of the evaluation results, they may be involved in part or all of the evaluation without being a specific member of the evaluation stakeholder workgroup. Roles might include, but are not limited to the following:

· Development of the quitline evaluation plan.

· Feedback on focusing the evaluation or selecting evaluation questions.

· Needing information about specific quitline evaluation activities or progress of the evaluation.

· Facilitating implementation of specific aspects of the quitline evaluation.

· Included in interpretation meetings.

· Disseminating and promoting use of quitline evaluation results.

## 1.4 Stakeholder Information Needs Exercise

Although focusing the evaluation occurs in Step 3, the groundwork begins with the identification of quitline stakeholders relevant to the evaluation or the primary intended users. Membership in the evaluation stakeholder workgroup is designed to reflect priority information needs, as well as those that will use the evaluation information. However, it is not always possible to include some groups who need information, and it is certainly not possible to include representation from every group that would benefit from evaluation results. This should not prevent evaluation staff and the evaluation stakeholder workgroup from considering all points of view and needs for information when considering how best to focus the quitline evaluation. Therefore, determining stakeholder information needs is both useful for considering membership in the evaluation stakeholder group (Step 1) and focusing the evaluation (Step 3).

From the list of primary intended users (i.e., those who have a stake in the quitline evaluation results), identify what information each stakeholder will use. 

## 1.4 Stakeholder Information Needs 

## Step 2. Tools to Describe the Program

## 2.1 Developing a Quitline Logic Model Exercise

As a logic model for your quitline program is developed, carefully consider the purpose of the quitline evaluation (see 1.2 Evaluation Purpose Exercise). It can be helpful to start with the intended outcomes of the quitline, such as increased calls or referrals, and increased quit attempts. Or it may be easier to start with the inputs (resources) available to the quitline, such as funding amounts or referral networks, as well as quitline activities. In either case, the goal will be to identify as many relevant elements of the context in which the quitline exists that will contribute to or detract from its intended outcomes. 

**Inputs:**Infrastructure and****resources necessary for quitline implementation.

**Activities:**The actual activities conducted by the quitline to achieve its goals.

**Outputs:**Direct products obtained as a result of program activities (e.g., counseling and medication provision).

**Outcomes:**(Short-term; intermediate; long-term; distal) The changes, impacts, or results of quitline implementation (activities and outputs). 

**Environmental Context:** Larger cessation program and policy environment in which the quitline is operating. 

First, identify the elements in the logic model.

Next, go through the items listed above, and start identifying how the inputs, activities, and outputs might link to outcomes (both intended and unintended). Your ESW can help identify some of the links. On the basis of the identified links, place the logic model elements into a logic model diagram. This will form the starting point for a shared understanding of the quitline program and its intended outcomes. An overview of the environmental context can also be highlighted on your logic model (see next).

## 2.2 Describe the Quitline’s Environmental Context Exercise

Developing a clear description of the quitline’s environmental context can be a critical component when determining the factors that can contribute to, and detract from the quitline’s goals. In addition, the environmental context can help identify what evaluation questions are both feasible to answer and most important to answer. 

Start with the quitline’s inputs listed above. For each input identified, transfer them to the tables below. Consider environmental factors that might influence those inputs. The same environmental factor may influence more than one input. Repeat the process for quitline activities, outputs, and outcomes (short-term, intermediate, and long-term). When considering the environmental context, think back to the key stakeholder identification exercise in Step 1. Which individuals and groups are most likely to take an interest in the quitline? Which ones should be more engaged? Which are strong supporters, weak supporters, or have negative views of your quitline? In addition, think about the following questions: 

· What factors are likely to influence your quitline’s available funding capacity?

· What state or national activities, events, or policies might influence your quitline’s activities or outcomes?

· How might the tobacco industry’s promotional efforts affect your quitline’s work?

**2.2 Environmental Context Exercise**

****

## Step 3. Tools to Focus the Design

### Illustrating Decisions Focus to Evaluation 

The following scenarios illustrate how a quitline’s stakeholders, the evaluation purpose, and the quitline components and context interact to help focus a quitline’s evaluation. The types of evaluation questions that can and should be asked will vary on the basis of the length of time a quitline has been in operation, the degree of engagement by stakeholders, and the amount of resources (human and financial) to which a quitline has access, as well as other factors. It may be helpful to share these examples with stakeholders, if necessary, to start the conversation, or to determine if these scenarios apply in your context. Try to discuss what factors apply in your context.

Scenario 1: A New Quitline Service

When technological services (e.g., texting, web services) are added to a quitline, it may not be appropriate to conduct outcome evaluation on quit rates or cost-effectiveness during the first few years. Instead, it may be appropriate to focus on process evaluation questions, such as the number of tobacco users served by each method, how participants who only use technology services differ from those who also use telephone counseling, and the relationship between promotional efforts and reach. Following some basic monitoring and process evaluation and related program improvement, a program should be collecting follow-up data to enable outcome evaluation during subsequent years. It may also want to ask primarily quality-related questions related to whether the quitline program is being delivered as intended, and whether participants are satisfied with their experience.

Scenario 2: Determined Stakeholders

A quitline receives funding by using a state legislative appropriations process. One state legislator continually requests information about how the state quitline compares to quitlines in neighboring states. By inviting her to serve on the evaluation stakeholder workgroup, her larger questions about quitline performance can be addressed through an educational process related to quitline context and the feasibility of producing certain comparative metrics. 

Scenario 3: Limited Funding

A cut in a quitline’s budget can be cause for reconsideration of the relative importance of various evaluation questions. If a quitline has not changed the constellation of service offerings since the last outcome evaluation, it may not be necessary to collect quit outcomes during the period of budget shortfall, and instead, focus on questions related to demonstrating high reach across multiple demographic groups.

Once you have identified the various environmental factors that can both enhance and detract from achievement of your quitline’s goals, it can become easier to determine which evaluation questions are most important to answer.

## 3.1 Focus the Evaluation Exercise

The amount of information you can gather concerning your quitline is potentially limitless. Evaluations, however, are always limited by the number of questions that can be realistically asked, the methods that can actually be employed, the feasibility of data collection, and the available resources. These are the issues at the heart of Step 3 in the CDC framework: focusing the evaluation. The scope and depth of any program evaluation is dependent on program and stakeholder priorities; available resources including financial resources; staff and contractor availability; and the amount of time committed to the evaluation. The quitline staff should work together with the ESW to determine the priority of the questions, the feasibility of answering the questions, and how the results will be used before designing the evaluation plan. 

In this exercise, you will need to consider all the information from previous exercises in Step 1 through Step 2, the logic model, and your stakeholders’ vested interest in the evaluation. 

From the Stakeholder Mapping exercise (1.1), list the stakeholders categorized as high priority for information needs: 

From the Evaluation Purpose Identification exercise, indicate your overall evaluation purpose statement:

Consider each stakeholder’s evaluation needs and the information most appropriate for answering that question (from exercise 1.4):

Given the overall purpose statement, what questions from the high-priority stakeholder group are viable for the current evaluation effort?

Next, the team should consider issues of feasibility related to those evaluation questions that are viable options given the evaluation purpose.

No chart, grid, or exercise can fully answer the question of how best to focus the evaluation. However, the above information should facilitate informed discussions and can help avoid evaluation activities that are misaligned with the quitline purpose or activities, are underfunded, or not of the highest priority for information needs. Additional considerations that might help prioritize your evaluation questions include

· The questions most important to you and your key stakeholders (the “must answer” questions). 

· Questions that provide results that can be used (e.g., for improvement).

· Questions you can answer fully with available or easy to gather data.

· Questions within your resources to answer.

## Step 4. Tools to Gather Credible Evidence

### Matching Measures to Evaluation Questions

In Step 3, we discussed the difference between types of evaluation questions, such as process and outcome questions. For each area of interest for quitlines, we provided sample evaluation questions in Tables 1 and 2. For Step 4, you will need to link each evaluation question to a method and related measures that can help answer it. In Tables 6 and 7 below, a sample of quitline evaluation questions is linked to a measure (or indicator if a measure needs more specification by the program) that can help answer it. Please note that these are examples only, and if your quitline’s evaluation questions are different than the ones listed below, the measures needed to answer them will also be different. 

You may wish to consider these examples with the ESW if they relate to areas of interest in the evaluation plan. It’s important to keep use in mind as you go through this exercise that data should not be collected for their own sake, but because they relay some useful information about the program that will be used for a specific purpose that has been laid out by the ESW.

60 | Page

**Table 6. Sample Process Evaluation Questions and Sample Measures**

**Table 7. Sample Outcome Evaluation Questions and Sample Measures**

## 4.1 Evaluation Methods Checklist Exercise

Here is a checklist of issues based on the evaluation standards that can help the ESW recommend the most appropriate data collection methods: 

## 4.2 Evaluation Methods Grid Exercise

One tool that is particularly useful in your quitline evaluation is an **evaluation methods grid**. This tool is helpful to align evaluation questions with indicators or performance measures and data sources and roles and responsibilities, and it can facilitate advocating for resources for the evaluation. In addition, this tool facilitates a shared understanding of the overall evaluation plan with stakeholders. This tool can take many forms and should be adapted to fit your specific evaluation and context; examples of the forms it takes follow.

**Evaluation Methods Grid Example A**

**Evaluation Methods Grid Example B**

Choose the grid that is most appropriate for your program, and complete it given your chosen evaluation questions from Step 3.

## 4.3 Evaluation Budget Exercise

For this exercise, you will need the Evaluation Methods Grid you completed earlier in Step 4. For this exercise, we have used one grid as an example, but you should use the one you have chosen as most appropriate for your program.

The team should now consider roles and responsibilities, what services might be in-kind, and what activities will cost additional money. Will you need to pay for additional questions on existing surveys, or can you use items that already exist? Are there existing data sources, or will you need to create new ones? Do not forget items such as copying costs for surveys or web services or technology needed in the field, such as recorders or mobile data collection devices.

During this exercise, you may find it necessary to revisit Step 3 or earlier portions of Step 4. Often the budget available doesn’t match the evaluation desired. Either the evaluation scope will need to be reduced or additional resources obtained. It is better to thoroughly consider this now before implementation begins than have to change course mid-implementation cycle.

## Step 5. Tools for Justifying Conclusions

## 5.1 Stakeholder Interpretation Meeting Exercise

Justifying conclusions includes analyzing the information you collect, interpreting what the data mean, and drawing conclusions based on the data. This step is needed to turn the data collected into meaningful, useful, and accessible information. This is often the step in which programs incorrectly assume they no longer need the stakeholder workgroup and that this step is better left to the “experts.” However, including your stakeholder group in this step is directly tied to the previous discussion on credibility and acceptance of data and conclusions. 

Moreover, it is critical that plans include time for interpretation and review from stakeholders, including critics, to increase transparency and validity of the process and conclusions. The emphasis here is on justifying conclusions, not just analyzing data. This step deserves due diligence in the planning process. The propriety standard plays a role in guiding the evaluator’s decisions in how to analyze and interpret data to ensure that all stakeholder values are respected in the process of drawing conclusions (Program Evaluation Standards, 1994). This may include one or more stakeholder interpretation meetings to review interim data and further refine conclusions. A note of caution, as a stakeholder driven process, there is often pressure to reach beyond the evidence when drawing conclusions. It is the responsibility of the evaluator and the evaluation workgroup to ensure that conclusions are drawn directly from the evidence. 

A variety of activities can be included in your evaluation plan to solicit stakeholder input and facilitate interpretation of evaluation data. An example is provided below:

Complete an outline of proposed activities appropriate to the evaluation project to include opportunities for stakeholder interpretation and feedback: 

It is important to consider the time it takes to solicit and incorporate stakeholder feedback in the evaluation project timeline. Check that the budget and timeline created earlier ensures adequate time and funding for the stakeholder inclusion process.

To ensure the stakeholder interpretation meeting is a success, plan for steps to help things run smoothly. Time for these activities needs to be included in the evaluation timeline.

· Send the initial invitation at least 2 months in advance so that stakeholders can plan for the meeting. Remind stakeholders of the overall evaluation purpose and questions.

· Send the preliminary report or PowerPoint presentation with enough lead time before the meeting to allow stakeholders time to review. It is important to remind stakeholders that results are draft and should not be shared outside of the review group.

· Send reminders about the meeting 1 or 2 weeks before the date. Identify any pre-existing documentation that may be useful for understanding context.

· Plan for appropriate technology (and backup) needed, such as recorders, laptop, screen, flipcharts, etc.

· If feasible, use a professional meeting facilitator.

A checklist to facilitate the development of a formal stakeholder interpretation meeting can be found at: http://www.wmich.edu/evalctr/checklists/checklistmenu.htm.

## Step 6. Tools to Ensure Use and Share Lessons Learned

## 6.1 Reporting Checklist

Below is a checklist of items that may be worth discussing during the evaluation planning stage to ensure adequate time and resources are devoted to the implementation and reporting process.

***Tools and Templates: Checklist for Ensuring Effective Evaluation Reports*****

**❏**Provide interim and final reports to intended users in time for use.

**❏**Tailor the report content, format, and style for the audiences by involving audience

members.

**❏**Include an executive summary.

**❏**Summarize the description of the stakeholders and how they were engaged.

**❏**Describe essential features of the program (e.g., in appendices).

**❏**Explain the focus of the evaluation and its limitations.

**❏**Include an adequate summary of the evaluation plan and procedures.

**❏**Provide all necessary technical information (e.g., in appendices).

**❏**Specify the standards and criteria for evaluative judgments.

**❏**Explain the evaluative judgments and how they are supported by the evidence.

**❏**List both strengths and weaknesses of the evaluation.

**❏**Discuss recommendations for action with their advantages, disadvantages, and

resource implications.

**❏**Ensure privacy protections for program clients and other stakeholders.

**❏**Anticipate how people or organizations might be affected by the findings.

**❏**Present minority opinions or rejoinders, where necessary.

**❏**Verify that the report is accurate and unbiased.

**❏**Organize the report logically, and include appropriate details.

**❏**Remove technical jargon.

**❏**Use examples, illustrations, graphics, and stories.

** Adapted from Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). *Program evaluation: Alternative approaches and practical guidelines*(2nd edition). New York, NY: Addison, Wesley Logman, Inc. and presented in Comprehensive Cancer Control Branch Program Evaluation Toolkit, Centers for Disease Control and Prevention, June 2010.

Also visit The Evaluation Center at Western Michigan University online for a free evaluation report checklist:

http://www.wmich.edu/evalctr/checklists/checklistmenu.htm.

## 6.2 Communicating Results Exercise

Evaluation results may not reach the intended audience just because they are published; an intentional communication and dissemination plan should be included in your evaluation plan. As previously stated, the planning stage is the time for the program to begin to think about the best way to share the lessons you will learn from the evaluation. The communication-dissemination phase of the evaluation is a two-way process designed to support use of the evaluation results for program improvement and decision making. To achieve this outcome, a program must translate evaluation results into practical applications and must systematically distribute the information or knowledge through a variety of audience-specific strategies. 

Communicating evaluation results involves sharing information in ways that make it understandable and useful to stakeholders. Successful communication is key to evaluation results being used. You can do this by using a variety of communication formats and channels. A communication format is the actual layout of the communication you will use, such as reports, brochures, one-page descriptions, newsletters, executive summaries, slides, and fact sheets. A communication channel is the route of communication you will use, such as oral presentations, videos, e-mails, webcasts, news releases, and web or phone conferences. Both the formats and channels should consider the needs of different audiences, the type of information you wish to provide, and the purpose of the communication. 

When developing your communication or dissemination strategy, carefully consider the following:

· With which target audiences or groups of stakeholders will you share findings?

· What formats and channels will you use to share findings?

· When and how often do you plan to share findings?

· Who is responsible for carrying out dissemination strategies?

You can plan and track the communication process by using tools similar to the following examples.

****

**6.2 Communicating Results**

** This tool was adapted from DASH’s Communication Matrix in “Using Evaluation to Improve Programs: Strategic Planning” in the Strategic planning kit for school health programs. Available at: http://www.cdc.gov/healthyyouth/evaluation/strategicplan.htm Last accessed 5-19-2014.

This tool can help you track communications with your various audiences, including the communication format(s) (the layout of the communication, such as newsletters) and the communication channel(s) (the route of communication, such as oral presentations), audience feedback on the communication message, and next steps you need to take in response. 

Here are two examples of a communication tracking charts. Select the best items for your context.

## 

94 | Page