Conducting Quitline 

Evaluations 

National Center for Chronic Disease Prevention and Health Promotion 

Office on Smoking and Health

A Workbook for Tobacco 

Control Professionals
The following contributors prepared this publication:

René Lavinghouze, MA Office on Smoking and Health, Centers for Disease Control 

and Prevention

Jessie E. Saul, PhD North American Quitline Consortium

Lei Zhang, PhD Office on Smoking and Health, Centers for Disease Control 

and Prevention

Nicole M. Kuiper, MPH Office on Smoking and Health, Centers for Disease Control 

and Prevention

Henraya McGruder, PhD Office on Smoking and Health, Centers for Disease Control 

and Prevention

The following OSH staff also helped prepare this publication:

Joe Smyser, PhD ORISE Fellow

LaTisha Marshall, MPH Health Scientist

Karen Debrot, DrPH Health Scientist

Suggested Citation

Centers for Disease Control and Prevention. Conducting Quitline Evaluations: A Workbook for 

Tobacco Control Professionals. Atlanta, GA: Centers for Disease Control and Prevention, US 

Dept of Health and Human Services; 2015.
Acknowledgments

This workbook was developed by the Centers for Disease Control and Prevention’s (CDC) 

Office on Smoking and Health (OSH) and the North American Quitline Consortium (NAQC) 

as part of a series of evaluation technical assistance workbooks intended to offer guidance 

and facilitate capacity building on a wide range of evaluation topics. This workbook is a 

guide to facilitate the evaluation of quitlines—one method for helping tobacco smokers 

to quit—and is intended for use by quitline staff and stakeholders, state tobacco control 

program managers, and evaluators. We encourage users to adapt the tools and resources 

in this workbook to meet their program’s evaluation needs.

Quitlines are telephone-based tobacco cessation services that help tobacco users quit. 

Services offered by quitlines include coaching and counseling, referrals, mailed materials, 

training to health care providers, web-based services and, in some instances, free 

medications, such as nicotine replacement therapy (NRT). In the United States, there are 

quitlines in all 50 states, the District of Columbia (DC), Guam, and Puerto Rico. The National 

Network of Tobacco Cessation Quitlines, with its number 1-800-QUIT-NOW, serves as a 

national portal, routing calls to the callers’ state quitline providers. Various stakeholders will 

have differing interests in quitline evaluation topics. For example, some may be interested in 

assessing increases in call volumes as a result of a media campaign, whereas others may 

be interested in assessing quitting success at 7 months for those who enrolled in quitline 

counseling. This guide will facilitate discussing, planning, and conducting evaluations to meet 

these various needs with different stakeholders, as well as assessing what is already being 

implemented and existing data. 

A worksheets and tools section is provided in Part II of this workbook for additional technical 

assistance, and a resource guide is provided for further reading in evaluation and methods.

This workbook (and all the workbooks in this series) applies the CDC Framework for 

Program Evaluation in Public Health (www.cdc.gov/eval). The framework lays out a six-step 

process for the decisions and activities involved in conducting an evaluation. Although the 

framework provides steps for program evaluation, these steps are not always planned or 

implemented in a linear fashion. They often require an iterative process, and some steps 

can be completed concurrently. In some cases, it may make more sense to skip a step 

and return to it later during the planning process. The framework should be used as most 

appropriate within the specific context of your program.

For more information, states are encouraged to contact their OSH Project Officer or 

the Evaluation Team Lead, René Lavinghouze, in OSH at rlavinghouze@cdc.gov. 

In addition, OSH’s surveillance and evaluation resources are accessible at: 

http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluation/index.htm.

Table of Contents

Introduction ..................................................................................................................1

Who is the intended audience for this workbook? .....................................................1

Why evaluate quitlines? ..........................................................................................2

What is an evaluation plan? ....................................................................................2

Why is it important to develop an evaluation plan for your quitline?............................3

CDC’s Framework for Program Evaluation ................................................................4

Part I: Putting the CDC Framework into Action for Your Quitline ..................................7

Step 1: Engage Stakeholders ..................................................................................7

Defining the Purpose in the Plan.........................................................................7

Identifying and Engaging the Evaluation Stakeholder Workgroup (ESW) .................8

Step 2: Describe the Quitline Program ...................................................................15

Develop a Shared Understanding of the Quitline Program...................................15

Linking Activities and Outcomes.......................................................................16

Step 3: Focus the Evaluation ................................................................................19

Developing Evaluation Questions......................................................................19

Types of Quitline Evaluation Questions..............................................................20

Step 4: Gather Credible Evidence...........................................................................27

Choosing the Appropriate Methods...................................................................28

Credible Evidence............................................................................................30

Measurement..................................................................................................30

Data Sources .................................................................................................32

Evaluation Methods Checklist...........................................................................34

Evaluation Methods Grid ..................................................................................34

Budget ...........................................................................................................35
Step 5: Justify Conclusions...................................................................................37

Provide Context: Compare Results to Other Types of Programs...........................38

Provide Context: Present Evaluation Results with Other Data ..............................41

Reporting with Clarity.......................................................................................44

Step 6. Ensure Use of Evaluation Findings and Share Lessons Learned....................47

Developing a Communications Plan..................................................................50

Summary........................................................................................................54

References..................................................................................................................57

Part II. Exercises, Worksheets, and Tools....................................................................59

Step 1. Tools to Engage Stakeholders ....................................................................60

1.1 Stakeholder Mapping Exercise ...................................................................60

1.2 Evaluation Purpose Exercise.......................................................................61

1.3 Stakeholder Inclusion and Communication Plan Exercise .............................62

1.4 Stakeholder Information Needs Exercise .....................................................64

Step 2. Tools to Describe the Program ...................................................................65

2.1 Developing a Quitline Logic Model Exercise.................................................65

2.2 Describe the Quitline’s Environmental Context Exercise................................67

Step 3. Tools to Focus the Design..........................................................................71

Illustrating Decisions Focus to Evaluation .........................................................71

3.1 Focus the Evaluation Exercise ....................................................................72

Step 4. Tools to Gather Credible Evidence ..............................................................77

Matching Measures to Evaluation Questions......................................................77

4.1 Evaluation Methods Checklist Exercise........................................................83

4.2 Evaluation Methods Grid Exercise ...............................................................85

4.3 Evaluation Budget Exercise ........................................................................88

Step 5. Tools for Justifying Conclusions..................................................................89

5.1 Stakeholder Interpretation Meeting Exercise................................................89
Step 6. Tools to Ensure Use and Share Lessons Learned.........................................91

6.1 Reporting Checklist ...................................................................................91

6.2 Communicating Results Exercise................................................................92

6.3 Success Story Examples and Tools.............................................................95

Resource Section........................................................................................................96

Quitline Resources................................................................................................96

Non-OSH Evaluation Resources.............................................................................98

Telling Your Story: Reporting Evaluation Results....................................................101

Qualitative Methods............................................................................................102

Quantitative Methods..........................................................................................102

Evaluation Use....................................................................................................103

OSH Evaluation Resources..................................................................................103
Figures

Figure 1. CDC Framework for Program Evaluation in Public Health.............................4

Figure 2. Sample Quitline Logic Model...................................................................17

Figure 3. Graphic Representation of Quit Rates, Proportion of Medicaid Callers, and 

Proportion of Low Education Callers Over Time.......................................................39

Figure 4. Graphically Demonstrating the Impact of a Quitline’s Media Plan ...............40

Figure 5. Mapping Call Volume to Average Time to Enrollment.................................40

Figure 6. Quitline Call Volume and Phone Counseling Provided, Colorado 

(Jan. – Dec. 2010) ...............................................................................................42

Figure 7. Demographic Summary of New Registrants, Vermont Quit Network: Quit by 

Phone Program, FY 2010......................................................................................43

Figure 8. Examples of Upstream, Midstream, and Downstream Successes with a 

Quitline Media Campaign ......................................................................................52

Tables

Table 1. Process Evaluation Areas of Interest for Quitlines and Sample Monitoring and 

Evaluation Questions ...........................................................................................21

Table 2. Outcome Evaluation Areas of Interest for Quitlines and Sample Outcome 

Evaluation Questions ...........................................................................................23

Table 3. Evaluation Methods Grid, Example A..........................................................35

Table 4. Evaluation Methods Grid, Example B .........................................................35

Table 5. Quitline Evaluation Communication Plan Example.......................................51

Table 6. Sample Process Evaluation Questions and Sample Measures.....................78

Table 7. Sample Outcome Evaluation Questions and Sample Measures....................81
Acronyms

ATS ................................................................................................. Adult Tobacco Survey

BRFSS............................................................. Behavioral Risk Factor Surveillance System

CBPR..................................................................Community Based Participatory Research

CDC................................................................. Centers for Disease Control and Prevention

CMI ............................................................................. Component Model of Infrastructure

ESW ............................................................................. Evaluation Stakeholder Workgroup

FDA.....................................................................................Food and Drug Administration

MDS....................................................................Minimal Data Set for Evaluating Quitlines

NAQC .........................................................................North American Quitline Consortium

NIDRR....................................... National Institute on Disability and Rehabilitation Research

NQDW ........................................................................... National Quitline Data Warehouse

NRT....................................................................................Nicotine Replacement Therapy

OSH.....................................................................................Office on Smoking and Health

PRAMS .....................................................Pregnancy Risk Assessment Monitoring System

STATE System ..............................State Tobacco Activities Tracking and Evaluation System

TUS-CPS...................................Tobacco Use Supplement of the Current Population Survey

YRBSS ................................................................Youth Risk Behavior Surveillance System

1

Introduction

Quitlines are telephone-based tobacco cessation services that help tobacco users quit. 

Services offered by quitlines include coaching and counseling, referrals, mailed materials, 

training to health care providers, web-based services and, in some instances, free 

medications, such as nicotine replacement therapy (NRT).1 Quitlines increase the odds of 

quitting smoking when compared with minimal interventions, self-help, or no counseling.2

In the United States, there are quitlines in all 50 states, the District of Columbia, Guam, and 

Puerto Rico. Quitlines vary in eligibility criteria, counseling, and medication protocols.

To best serve the residents of your state, it is critical to evaluate the quality, effectiveness, 

and impact of quitline services and related interventions designed to drive callers to the 

quitline. This workbook is designed to help you think through key evaluation concepts and 

design quitline evaluations that will improve services and overall cessation efforts.

WHO IS THE INTENDED AUDIENCE FOR THIS WORKBOOK?

This workbook is specifically designed to guide quitline staff, stakeholders, program 

managers, and evaluators in planning, conducting, and interpreting the results of quitline 

evaluation activities. This workbook will help interested parties or stakeholders develop: 

an understanding of what constitutes a quitline evaluation; why a quitline evaluation is 

important; how to develop an effective evaluation plan in the context of the planning 

process; and implementation considerations. 

This workbook was written by the staff of the Office on 

Smoking and Health (OSH) at the Centers for Disease 

Control and Prevention (CDC) and the North American 

Quitline Consortium (NAQC). Part I of this workbook 

defines and describes specific considerations for 

writing an evaluation plan for quitlines, as well as basic 

implementation considerations. It is not intended to 

serve as a standalone resource; rather, it is intended 

to be used with other evaluation resources, such as 

those listed in the resource section of this workbook 

(pages 96-106). In addition, it is recommended that this 

workbook be used with the Developing an Effective 

Evaluation Plan3 and Developing an Effective Evaluation Report4 workbook guides, which 

are also in this series (www.cdc.gov/tobacco/tobacco_control_programs/surveillance_

evaluation). The exercises, worksheets, and tools found in Part II of this workbook are 

An evaluation plan is a written 

document that describes how 

you will monitor and evaluate 

your program so that you will be 

able to describe the What, the 

How, and the Why It Matters for 

your program, and use evaluation 

results for program improvement 

and decision making.3
2 | Conducting Quitline Evaluations 

designed to help you think through the concepts discussed in Part I, and to put them into 

practice. However, these are only examples, and your quitline evaluation will vary on the 

basis of your program, stakeholder priorities, and context. 

WHY EVALUATE QUITLINES?

Evaluation data are vital to engage quitline stakeholders, understand how a quitline 

is functioning, and identify areas for improvement or change. Engaged Data is a core 

component of functioning program infrastructure, which is portrayed in the Component 

Model of Infrastructure.5-7 Functioning program infrastructure is defined as the foundation 

or platform that supports capacity, implementation, and sustainability of quitline initiatives.6

Collecting evaluation data should be an integral component of your quitline, and not just a 

data report at the end of your funding period.

Data gathered during evaluations enable managers, staff, and stakeholders to create the 

best possible programs; to learn from mistakes; to make modifications, as needed; to 

monitor progress towards program goals; to judge the success of the program in achieving 

its short-term, intermediate, and long-term outcomes; and to demonstrate accountability 

for funding. Timely evaluation data that are of use to stakeholders and decision makers 

are critical to engaging key partners working toward program goals and objectives, and 

enabling them to take action in support of the program. 

WHAT IS AN EVALUATION PLAN?

An evaluation plan is a written document that describes how you will monitor and evaluate 

your quitline, as well as how you intend to use evaluation results for program improvement 

and decision making. The evaluation plan clarifies how you will describe the What, the How, 

and the Why It Matters for your quitline. 

 The What reflects the description of your quitline and how its activities are linked with 

its intended effects. It serves to clarify the quitline’s purpose, relationship to other 

components of a comprehensive tobacco control program, and anticipated outcomes. 

 The How addresses the process for implementing a quitline and provides information 

about whether the quitline is operating as it was intended. In addition, the How (or 

process evaluation), with output or short-term outcome information, helps clarify if 

changes should be made to operating procedures or protocols, or other aspects of 

the quitline. 
3

 The Why It Matters provides the rationale for your quitline and the effect it has on 

public health. This is also sometimes referred to as the “so what” question of your 

program. Being able to demonstrate that your quitline has had an impact is critical to 

sustaining its funding. 

An evaluation plan is similar to a roadmap. It clarifies the steps needed to assess the 

processes and outcomes of a program. An effective evaluation plan is more than a column 

of indicators added to your work plan. It is a dynamic tool (i.e., a “living document”) that 

should be updated, as needed, to reflect program changes and priorities over time. A 

quitline evaluation plan serves as a bridge between evaluation and planning by highlighting 

quitline goals, clarifying measurable quitline objectives, and linking quitline activities with 

intended outcomes. 

WHY IS IT IMPORTANT TO DEVELOP AN EVALUATION PLAN 

FOR YOUR QUITLINE?

An evaluation plan can clarify the direction of an evaluation on the basis of priorities, 

resources, time, and skills, and how results will be used to improve quitline access, 

services, customer satisfaction, and outcomes. The process of developing an evaluation 

plan with an evaluation workgroup of stakeholders will foster collaboration and a sense 

of shared purpose. Having a written evaluation plan will foster transparency and ensure 

that stakeholders agree on the purpose of the evaluation and the use and users of the 

evaluation results. Moreover, use of evaluation results must be planned, directed, and 

intentional.8 A written plan is an effective tool and vital part of a functioning program 

infrastructure.5-7

This workbook will not provide a detailed “how to” description for writing an evaluation plan 

for your quitline. Although a few reminders are included in this workbook, more in-depth 

information on writing an evaluation plan can be found in Developing an Effective Evaluation 

Plan.

3 (www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluation)

Part I of this workbook is organized by the six steps of the CDC Framework for Program 

Evaluation.9 Each chapter will introduce the key considerations to be addressed in that step 

related to planning and implementing your quitline evaluation. The main points are also 

illustrated with one or more basic examples. Part II includes exercises, worksheets, tools, 

and a resource section to help program staff and the evaluation stakeholder workgroup 

think through the concepts presented in Part I. 
4 | Conducting Quitline Evaluations 

CDC’S FRAMEWORK FOR PROGRAM EVALUATION

CDC’s Framework for Program Evaluation in Public Health9 (Figure 1) is a guide for 

how to effectively evaluate public health programs and to use the findings for program 

improvement and decision making. Although the framework is described in terms of 

steps, the actions are not always linear and are often completed in an iterative manner 

that’s cyclical in nature. Similar to the framework, the development of an evaluation plan 

is an ongoing process. You may need to revisit a step during the process and complete 

other discrete steps concurrently. Within each step of the framework, there are important 

components that are useful to consider in the creation of a quitline evaluation plan.

Six Steps of the CDC Framework

1. Engage stakeholders.

2. Describe the program.

3. Focus the evaluation design.

4. Gather credible evidence.

5. Justify conclusions.

6. Ensure use and share lessons learned.

Figure 1. CDC Framework for Program Evaluation in Public Health

Over the life of a program, 

any number of evaluations 

may be appropriate, 

depending on the situation. 

There isn’t one right model 

evaluation for all programs.
5

In addition to CDC’s Framework for Program Evaluation in Public Health, there are 

evaluation standards that will enhance the quality of evaluations by guarding against 

potential mistakes or errors in practice. The evaluation standards are grouped around four 

important attributes (Figure 1): 

1. Utility—serve information needs of intended users.

2. Feasibility—be realistic, prudent, diplomatic, and frugal.

3. Propriety—behave legally, ethically, and with due regard for the welfare of those 

involved and those affected.

4. Accuracy—evaluation is comprehensive and grounded in the data.9,10

The underlying logic of the evaluation framework is that good evaluation does not merely 

gather accurate evidence and draw valid conclusions, but produces results that are used 

to make a difference and engage stakeholders. To maximize the chances that evaluation 

results will be used, it’s important to create a market before developing the product (i.e., 

the evaluation). Focusing on the questions that are most salient, relevant, and important to 

your stakeholders helps create the market for your results. You ensure the best evaluation 

focus by understanding where the questions fit into the full landscape of a program, and 

especially by ensuring that you have identified and engaged stakeholders who care about 

these questions and want to take actions on the basis of results. The CDC framework 

approach draws on the fundamental insight that there is no single correct program 

evaluation model that can be applied to all programs. Rather, over the life of a program, 

multiple evaluations may be appropriate, depending on the situation.
6 | Conducting Quitline Evaluations 
7

1 2 3 4 5 6

Part I: Putting the CDC Framework into 

Action for Your Quitline 

STEP 1: ENGAGE STAKEHOLDERS 

Defining the Purpose in the Plan

The purpose of an evaluation influences the identification of stakeholders for the evaluation, 

selection of specific evaluation questions, and the timing of evaluation activities. It is critical 

that the quitline staff be transparent about the intended purposes of the evaluation. If 

evaluation results will be used to determine whether specific quitline features (or the quitline 

itself) should be continued or eliminated, stakeholders should know this up front. The 

stated purpose of the evaluation drives the expectations and sets the boundaries for what 

the evaluation can and cannot deliver. In any single evaluation, and especially in a multiyear 

plan, more than one purpose may be identified; however, the primary purpose can influence 

decisions on resource allocation, use, the selection of stakeholders, and other issues. 

Although there are many ways of stating the identified purpose(s) of the evaluation, they 

generally fall into three categories:8

1. Rendering judgments—accountability.

2. Facilitating improvements—program development.

3. Knowledge generation—transferability.
8 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Identifying and Engaging the Evaluation Stakeholder Workgroup (ESW)

Who are the quitline’s stakeholders?

The first decision to make when initiating a quitline evaluation is which stakeholders to 

include. Stakeholders are consumers of the evaluation results. As the intended consumers, 

users, and beneficiaries of the evaluation results, they will have a vested interest in these 

results.8,11 In general, stakeholders are those who are (1) interested in the quitline and would 

use evaluation results, such as referral sources (e.g., physicians, clinics, health systems, 

health plans, or community groups), government agencies, national partners (e.g., NAQC), 

and decision makers; (2) those who are involved in running the quitline, such as program 

staff, partners, management, the funder, and the service provider; and (3) those who are 

served by the quitline, primarily tobacco users. Other stakeholders may also be included as 

these categories are not exhaustive.

Engaging stakeholders in the evaluation process enhances intended users’ understanding 

and acceptance of the evaluation. Stakeholders are much more likely to buy into and 

support the evaluation if they are involved in the evaluation process from the beginning. 

Moreover, to ensure that the information collected, analyzed, and reported meets the needs 

of the program and its stakeholders, it is best to work with the people who will be using this 

information throughout the entire evaluation process.

Engaging stakeholders in a quitline evaluation can have many benefits. 

Stakeholders can help:

 Determine and prioritize key evaluation questions.

 Pretest data collection instruments.

 Facilitate data collection.

 Implement evaluation activities.

 Increase credibility of analysis and interpretation of evaluation information.

 Ensure that evaluation results are used.

An Evaluation Purpose Exercise worksheet is provided in Part II, Section 1.2 to assist 

you with determining the intended purposes for your quitline’s evaluation.
9

1 2 3 4 5 6

Possible Quitline Stakeholders

Below is a list of possible quitline stakeholders, some of which you may want to engage, 

depending on the specific situation in your state. This list is not exhaustive.

Referral sources:

 Physicians or health care professionals.

 Health educators in hospitals, clinics, etc.

 Employers.

 Health systems (e.g., physicians, other health care providers, clinics, practices, health 

care organizations or systems, health plans or insurers).

 Community-based organizations, including cessation providers.

Federal government agencies and entities:

 US Department of Health and Human Services (HHS).

 Centers for Disease Control and Prevention (CDC).

 Centers for Medicare and Medicaid Services (CMS).

 National Cancer Institute (NCI).

 Veterans Administration (VA).

 Department of Defense (DoD).

State or territorial government agencies and entities:

 State tobacco control program.

 State department of health.

 Women, Infants, and Children (WIC) programs.

 State Medicaid office.

 State agencies conducting surveillance that might collect information on tobacco use 

prevalence or quitline awareness or use (e.g., BRFSS).

 State mental health and substance abuse agencies.

 State agencies on chronic diseases associated with tobacco use.

 Tribal health agencies.

 Local health departments.
10 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Professional organizations and national partners:

 American Cancer Society (ACS).

 American Lung Association (ALA).

 American Heart Association (AHA).

 American Hospital Association (AHA).

 Legacy for Health (Legacy).

 Campaign for Tobacco-Free Kids (TFK).

 North American Quitline Consortium (NAQC).

 National Network Consortium.

 Agencies representing state, territorial, tribal, and local health departments.

Others:

 Employers of major companies or corporations offering quitline services.

 Quitline service providers (staff and management).

 Tobacco users.

Use the evaluation standards to help identify those stakeholders who matter most. Give 

priority to those stakeholders who:

 Will support or authorize changes to the quitline that the evaluation may recommend.

 Are served by the quitline.

 Are responsible for day-to-day implementation of the activities that are part of the 

quitline.

 Can increase the credibility of quitline evaluation efforts.

 Will support, fund, or authorize the continuation or expansion of the quitline.
11

1 2 3 4 5 6

How do I use an Evaluation Stakeholder Workgroup (ESW) to conduct an evaluation?

Stakeholders will often have diverse and, at times, competing interests. Given that a 

single evaluation cannot answer all possible evaluation 

questions raised by diverse groups, it will be critical that the 

prioritization process is outlined in the evaluation plan. 

It is suggested that the program enlist the aid of an 

evaluation stakeholder workgroup (ESW) of 8-10 members 

that represents the stakeholders who have the greatest 

stake or vested interest in the evaluation.12 This workgroup 

of primary intended users will serve in a consultative role 

on all phases of the evaluation. As members of the ESW, 

they will be an integral part of the entire evaluation process 

from the initial design phase to interpretation, dissemination, 

and use of the evaluation. Stakeholders will play a major 

role in the program’s evaluation, including consultation and 

possibly even data collection and decision making on the 

basis of the evaluation results. Stakeholder groups should be 

selected to participate in the ESW on the basis of relevancy 

and feasibility. For example, it is unlikely that a staff member 

from the US Department of Health and Human Services will 

participate on an individual state quitline’s ESW. 

As mentioned previously, stakeholders can have competing 

interests that may come to light in the evaluation planning 

process. It is important to explore agendas in the beginning 

and come to a shared understanding of roles and 

responsibilities, as well as the purposes of the evaluation. It 

is important that both the program and the ESW understand 

and agree to the role of the workgroup.

Stakeholders can be involved in the evaluation at various 

levels. For example, it may be beneficial to include 

referring physicians on a team to develop questions, data 

collection processes, and analysis plans. Consider ways 

to assess partners’ needs and interests in the evaluation, 

develop communication plans for keeping them informed 

of the evaluation’s progress, and integrate their ideas into 

An ESW comprises 

members who have a 

stake or vested interest in 

the evaluation findings and 

can most directly benefit 

from the evaluation. 

These members 

represent the primary 

users of the evaluation 

results and generally 

act as a consultative 

group throughout the 

entire planning and 

implementation of the 

evaluation. In addition, 

members sometimes 

facilitate the dissemination 

of results. Examples 

include promoting 

responses, participation 

in surveys, and in-kind 

support for interviews and 

interpretation meetings. 

The members can identify 

resources to support 

evaluation efforts. The 

exact nature and roles of 

group members is up to 

you, but roles should be 

explicitly delineated 

and agreed to in the 

evaluation plan.
12 | Conducting Quitline Evaluations 

1 2 3 4 5 6

evaluation activities, regardless of their level of involvement. 

A program’s critics should also be engaged in the evaluation. Critics may help identify 

issues around program strategies and evaluation information that could be attacked or 

discredited, thus helping to strengthen the evaluation process. This might be through the 

ESW, or through face-to-face interviewing, depending upon the level of criticism likely. This 

information might also help to understand potential opponent’s rationale for not supporting 

a particular initiative and help engage potential agents of change within the opposition. 

However, it is important to remember that stakeholders will have different viewpoints 

and different information needs. Ascertain their information needs at the beginning of the 

evaluation process to determine how they align with the purpose, design, and use of the 

evaluation. In Step 1: Engage Stakeholders, you will conduct a preliminary assessment of 

information needs. You will pursue this further in Step 2: Describe the Program and Step 3: 

Focus the Evaluation, until you have identified the final evaluation questions. 

Below are examples to illustrate the diversity of the information that may be needed by 

various quitline stakeholders. 

 State governmental agencies, such as state departments of health or state Medicaid 

offices, will be able to use the data to assess the reach of the quitline, or to report on 

the proportion of various population subgroups that use the quitline annually.

 Quitline partners (in particular quitline administrators or funders and service 

providers) can use quitline evaluation data to examine the effect of changes in 

quitline protocols, assess the cost-effectiveness of specific quitline components, and 

identify areas for change or improvement. 

 Voluntary health organizations can use the data collected as part of the quitline 

evaluation to monitor progress in state tobacco control efforts and to highlight the 

importance of the quitline as a component of the state’s comprehensive tobacco 

control plan. 

 Members of the medical community, such as physicians, nurses, and health 

A Stakeholder Inclusion and Communication Plan Exercise worksheet is provided 

in Part II, Section 1.3 to assist with identifying roles for stakeholders for the quitline 

evaluation, as well as modes and timing of communication for each stakeholder group. 
13

1 2 3 4 5 6

educators, may directly use data collected by the quitline to provide up-to-date 

information on quitline services, and to make decisions about whether to refer their 

patients to quitline services.

The process of stakeholder engagement (Step 1) continues throughout the evaluation, 

and stakeholders’ roles should be described in the quitline evaluation plan. The process of 

engaging the ESW continues in the remaining steps, as follows:

Step 2: Describing the program. A shared understanding of the program and what 

the evaluation can and cannot deliver is essential to the successful implementation 

of evaluation activities and the successful use of evaluation results. The program 

and stakeholders must agree on the purpose(s) of the evaluation and on the logic 

model. For example, a good understanding of the goals of the quitline and the 

services provided to quitline callers can help define the program’s proximal and 

distal outcomes. 

Step 3: Focusing the evaluation. Understanding the purpose of the quitline’s 

evaluation and the rationale for prioritization of evaluation questions is critical 

for transparency and acceptance of evaluation findings. It is essential that the 

evaluation address the questions that are of greatest practical importance to the 

quitline and to the priority users of the evaluation. For example, if a state has a 

focus on the high burden of tobacco use among persons insured by Medicaid, 

then some of the high priority evaluation questions should address use of the state 

quitline among this population. 

Step 4: Planning to gather credible evidence. For the evaluation results to be 

accepted and used, stakeholders have to accept that the evaluation methods 

selected are appropriate to answer the questions asked and that the data collected 

are credible. The acceptance of evaluation results begins in the planning phase. 

Stakeholders can guide the selection of appropriate methods. For example, if 

the program focuses on increasing the number of persons from various priority 

populations who use the quitline, then intake data should be collected to allow for 

identification of those populations, and compared with appropriate populationbased surveys that provide both population size and tobacco use prevalence for 

those populations. 

A Stakeholder Information Needs identification exercise is provided in Part II, 

Section 1.4 to assist you with determining stakeholder information needs.
14 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Step 5: Justifying conclusions. Stakeholders should guide the analysis and 

interpretation of evaluation, as well as the process of developing conclusions 

and recommendations. In turn, this will facilitate the acceptance and use of the 

evaluation results by other stakeholder groups. Stakeholders can help determine if 

and when stakeholder interpretation meetings should be conducted. For example, 

if findings indicate that calls to the quitline are lower than expected, it may help to 

involve either health system partners, who refer patients to the quitline, to discuss 

if referrals have declined, or staff or agencies involved in promotional campaigns to 

discuss whether certain promotions have not been as effective as expected. 

Step 6: Ensuring use and sharing of lessons learned. Stakeholders should 

guide the translation of evaluation results into practical applications and actively 

participate in the meaningful dissemination of lessons learned. This will help ensure 

that the results of the quitline evaluation are used. Stakeholders can facilitate the 

development of an intentional, strategic communication and dissemination plan 

as part of the evaluation plan. For example, the results of the quitline evaluation 

could indicate that calls to the quitline are low among Hispanic people. One of the 

lessons learned may be that more promotion of the quitline should be directed to 

this population. 

Evaluation Tools and Resources for Step 1:

Part II 

ü 1.1 Stakeholder Mapping Exercise

ü 1.2 Evaluation Purpose Exercise

ü 1.3 Stakeholder Inclusion and Communication Plan Exercise

ü 1.4 Stakeholder Information Needs Exercise
15

1 2 3 4 5 6

STEP 2: DESCRIBE THE QUITLINE PROGRAM

Develop a Shared Understanding of the Quitline Program

Narrative Description

A narrative description is helpful to ensure a full and complete shared understanding of 

the quitline. A logic model or other type of descriptive graphic may be used to succinctly 

synthesize the main elements of a program. Although a graphic representation of program 

elements is not always necessary, a program narrative is necessary. The program 

description is essential for focusing the evaluation design and selecting appropriate 

methods. Groups can sometimes jump to selecting evaluation methods before they have 

a good grasp of what the program is designed to achieve. The quitline program staff 

and stakeholders must agree upon the program description and the purposes(s) of the 

evaluation. The description will be based on the quitline’s objectives and context, but most 

descriptions include the following at a minimum:

 A statement of need to identify the health issue(s) addressed (e.g., the burden of 

increased morbidity and mortality caused by tobacco use).

 Targets or objectives of quitline activities to ensure that progress is made toward 

program goals (e.g., achieving 8% treatment reach).

 Outcomes of quitline activities (e.g., increased number of calls to the quitline, reach, 

number of quit attempts).

 Inputs or program resources available to support or implement quitline activities 

(e.g., state and federal funding, strategic funding or referral partnerships, quitline 

outreach and promotion).

 Program activities linked to program outcomes through theory or best practice 

program logic to help the intended audience for the program make specified changes 

or take action (e.g., local, state, or national media campaigns, quitline protocols for 

provision of counseling, and medications).

 Environmental context in which a program is implemented (e.g., state-level 

awareness of the quitline, level of support for the quitline by strategic partners, 

degree of integration of the quitline into health systems in the state).
16 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Linking Activities and Outcomes

The program description often includes a graphic representation of quitline activities and 

outcomes to visually show the link between these elements. This clarity can help with both 

strategic planning and program evaluation. It is helpful to review the model with the ESW to 

ensure a shared understanding of the model and to confirm that it is still an accurate and 

complete reflection of your program. The model should identify available resources (inputs), 

what the program is doing (activities), and anticipated achievements (outcomes). You 

should also articulate any challenges you face. 

One example of a graphic representation is a logic model, which is a common tool used 

by evaluators. Logic models are graphic depictions of the relationship between a quitline’s 

infrastructure or inputs, and activities and intended outcomes. As the starting point for 

evaluation and planning, a logic model illustrates the underlying logic behind the program, 

(i.e., why it should work). Over time, evaluation, research, and experience will deepen the 

understanding of what does and does not work, and the model will change accordingly. 

See Figure 2 for a sample quitline logic model.

Logic models typically include the following elements:

Inputs: Infrastructure and resources necessary for program implementation.

Activities: The actual interventions that the program implements to achieve 

outcomes.

Outputs: Direct products created as a result of program activities.

Outcomes: (Short-term; intermediate; long-term); the results of program 

implementation (activities and outputs), changes, and effects.

An exercise for Developing a Quitline Logic Model is included in Part II, Section 2.1 

to assist in identifying the activities, inputs, outputs, and outcomes for your quitline. 

An exercise to Describe the Quitline’s Environmental Context is included in Part 

II, Section 2.2, to assist you with determining what some of the factors are that can 

contribute to and detract from your quitline’s goals. In addition, the environmental 

context can help identify what evaluation questions are both feasible to answer and 

most important to answer. 
17

1

23456

Figure 2. Sample Quitline Logic Model Inputs Activities Outputs Short-term outcomes Intermediate outcomes Long-term outcomes 

Environmental Context 

State Medicaid cessation coverage, state excise tax rates, smokefree policies, media campaigns, state tobacco control funding 

(as a proxy for state tobacco control expenditures) 

Quitline 

promotion and 

outreach 

Quitline funding 

Quitline capacity 

Quitline 

partnerships 

Quitline protocol 

Outreach 

activities 

State / national 

earned media 

State / national 

paid media 

Materials, 

counseling, NRT 

provided 

Completed 

quitline 

promotional 

activities 

Increased 

number of 

tobacco users 

receiving services 

Increased 

number of 

registrants 

Increased number 

of referrals 

Increased quitline 

call volume 

Increased 

awareness of the 

quitline 

Increased number 

of quit attempts 

Increased interest 

in quitting 

Decreased 

tobacco-related 

disparities 

Reduced tobaccorelated medical 

expenditures 

Reduced tobaccorelated morbidity 

& mortality 

Reduced tobacco 

prevalence & 

consumption 

Increased 

cessation 

Targeted to populations with 

tobacco-related disparities 
18 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Evaluation Plan Tips for Step 2

ü A program description will facilitate a shared 

understanding of the program between the quitline 

program staff and the evaluation workgroup. 

ü The description section often includes a logic 

model to visually show the link between activities 

and intended outcomes.

ü The logic model should identify available resources 

(inputs), what the program is doing (activities), 

products of those activities (outputs), and what you 

hope to achieve (outcomes). The environmental 

context should be included, if possible. The level of 

detail should be appropriate for the audience using 

the model.

ü Evaluation results should be incorporated into 

future iterations of quitline programming and 

activities. 

Evaluation Tools and Resources for Step 2:

Part II

ü 2.1 Developing a Quitline Logic Model Exercise

ü 2.2 Describe the Quitline’s Environmental Context 

Exercise
19

1 2 3 4 5 6

STEP 3: FOCUS THE EVALUATION 

The scope and depth of any program evaluation is dependent on program and stakeholder 

priorities; available resources, including financial resources; staff and contractor availability; 

and amount of time committed to the evaluation. The quitline staff should work together with 

the ESW to determine the priority and feasibility of each evaluation question, and identify how 

the results will be used before designing the evaluation. Because resources for evaluation 

are usually limited, this chapter provides a series of decision criteria to help determine the 

most appropriate evaluation focus. These criteria are inspired by the evaluation standards, 

specifically, utility (who will use the results and what information will be most useful to them), 

and feasibility (how much time and resources are available for the evaluation). 

Useful quitline evaluations should focus on the information that will be used by the 

program, stakeholders (including funders), and decision makers to improve the program, 

make decisions, and engage all groups. Establishing the focus of the quitline evaluation 

began with the identification of the primary purposes and the primary intended users of the 

evaluation. This process was further solidified through the selection of the ESW. 

Focus the evaluation design with the ESW on the basis of the identified purposes, program 

context, logic model, and resource limitations. Discuss the priority, feasibility, and efficiency of 

the evaluation with the ESW. Transparency is particularly important in this step. Stakeholders 

and users of the quitline evaluation will need to understand why some questions were 

identified as high priorities, whereas others were 

rejected or delayed. Example scenarios illustrating 

ways that quitlines can focus their evaluations on 

the basis of variations in environmental context are 

presented in Part II, Step 3.

Developing Evaluation Questions

In this step, it is important to solicit evaluation questions from your various quitline 

stakeholder groups on the basis of the stated purposes of the evaluation. The questions 

should be considered in the context of the logic model or program description. Evaluation 

questions should be checked against the logic model, and changes may be made to either 

the questions or the model, thus reinforcing the iterative nature of the evaluation planning 

process. Questions can be prioritized on the basis of the ESW and program information 

needs, as well as feasibility and efficiency issues. Evaluation questions will likely change on 

the basis of the lifecycle of the quitline. For example, newer programs will require different 

questions than well-established programs.

A Focus the Evaluation

exercise is located in Part II, 

Section 3.1 of this workbook.
20 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Types of Quitline Evaluation Questions

An evaluation plan should include both process and outcome measures. In general, process 

evaluation focuses on the first three boxes of the logic model: inputs, activities, and outputs.12

Some process questions serve a monitoring function that help document what is happening 

in the quitline, whereas others are more evaluative in nature and often require additional data 

sources. Outcome evaluation, as the term implies, focuses on the last three outcome boxes of 

the logic model: short-term, intermediate, and long-term outcomes. A single evaluation can and 

should include both process and outcome evaluation questions. As the evaluator and the ESW 

take ownership of the evaluation, honing the evaluation 

focus can help solidify interest in the evaluation among 

quitline stakeholders. Selection of final evaluation questions 

should balance what is most useful to achieving all the 

primary stakeholders’ information needs, as identified 

in Step 1. Having the quitline stakeholders participate in 

the selection of questions increases the likelihood of their 

securing evaluation resources, providing access to data, 

and using the results. This process increases personal 

ownership of the evaluation by the ESW. 

Process Evaluation Questions

Process evaluation questions common in quitline evaluation can document program 

implementation (i.e., monitoring), or can evaluate the effectiveness or impact of quitline 

services. Several monitoring-type questions are whether specified quitline activities are taking 

place, who is conducting the activities, and who the activities are reaching. Questions that 

are more evaluative in nature are whether sufficient infrastructure or capacity is in place to 

conduct quitline activities, or whether sufficient resources have been allocated or mobilized. 

Process evaluations can measure whether actual program implementation was faithful to 

the initial plan, as well as identify unsatisfactory program performance early on to allow 

for corrections to be made in a timely manner.12 Process evaluations can also help identify 

areas where additional training, resources, or technical assistance may be warranted. Table 

1 provides concepts and sample process evaluation questions for quitlines. Depending 

on the context, some of these questions could also be considered outcome evaluation 

indicators (as in the sample logic model in Figure 2). Where something fits in the logic 

model is less important than whether it follows a theoretical pathway of change, and 

represents assessment of program implementation (i.e., process evaluation) or changes in 

knowledge or behavior from program implementation (i.e., outcome evaluation).

Process and Outcome Evaluation 

in Harmony in the Evaluation

A single evaluation can and often 

does include both process and 

outcome evaluation questions. 

Excluding process evaluation 

questions in favor of outcome 

evaluation questions often 

eliminates the understanding of the 

foundation that supports outcomes. 
21

1 2 3 4 5 6

Table 1. Process Evaluation Areas of Interest for Quitlines and Sample Monitoring and Evaluation 

Questions 

Process Evaluation 

Areas of Interest for 

Quitlines

Sample Monitoring 

Questions

Sample Process Evaluation 

Questions

Demand for Quitline 

Services

What is the call volume for the quitline on a 

daily, weekly, monthly, or annual basis?

How does it change over time? 

Why did demand change over time (e.g., 

policy changes, media promotions)?

Quitline Promotion Is promotion of the quitline being conducted 

according to plan and meeting targets? 

How does call volume relate to quitline 

promotional efforts?

Quitline Referral Networks How many referrals are received by the 

quitline? 

What referral sources (e.g., individual 

health care providers, clinics, and health 

systems) are there for the quitline? 

What recruitment strategies or outreach 

activities are being used to add new referral 

sources? 

How does referral type and source effect 

reach?

How do referral sources correlate with 

outreach activities? 

What proportion of persons who were 

referred is successfully contacted by the 

quitline?

What proportion of persons who were 

referred is successfully enrolled by the 

quitline?

Quitline Use How many tobacco users receive services 

(i.e., counseling, medications) from the 

quitline annually? 

What are the characteristics of the callers? 

How does the population of quitline 

participants compare to the population of 

tobacco users in the state?

Are callers representative of the population 

we were trying to reach?

What is the quitline’s reach?

Quality Assurance Are tobacco users receiving proactive calls, 

or is the quitline responding to voicemails 

within the time frame specified by the 

quitline contract? 

Are referrals being processed in a timely 

fashion? 

Are reports accurate and complete?

To what extent are the services provided 

meeting quality standards? 

Is the counseling being provided by using 

evidence-based methods?

Participant Satisfaction What is the level of participant satisfaction 

with services (e.g., counseling, provision of 

medication, educational materials)?

What quitline factors increase participant 

satisfaction? 

What factors decrease participant 

satisfaction?
22 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Process Evaluation 

Areas of Interest for 

Quitlines

Sample Monitoring 

Questions

Sample Process Evaluation 

Questions

Program Intensity What treatment (e.g., number and length 

of proactive calls, type and amount of 

medication) on average are participants 

receiving? 

How do these differ across 

sociodemographic groups, smoking status, 

or whether they were referred or not? 

How do treatments delivered compare with 

program protocol? 

Investment in Services and 

Promotion

What is the annual investment in quitline 

services, medications, promotions, and 

outreach? 

Do we have the right mix of services for our 

investment? 

What could we achieve with more funds?

Quitline Staffing What is the level of supervision for coaches 

or counselors? 

What form does supervision take?

Is the quitline staffed sufficiently to respond 

to all incoming calls and referrals? 

Are the hours of operation adequate to 

meet the demand for services?

Did demand for services exceed capability? 

Are quitline coaches or counselors trained 

appropriately (e.g., language, cultural 

competency trainings)?

Quitline Efficiency How much did the quitline spend per 

enrollee?

How are the quitline’s spending per smoker 

amounts related to reach? 

Did we have the right strategies (e.g., 

promotion, treatment mix) to reach or treat 

our desired population efficiently?

Interaction among 

Cessation Program 

Components

How does the quitline interact (if at all) with 

other cessation program components?

How could components interact more 

effectively and efficiently?

Outcome Evaluation Questions

Outcome evaluation measures changes in knowledge, attitudes, or behaviors that arise 

from your quitline or promotions to the quitline. You may be interested in your quitline 

callers, or the general population, or all the smokers in the state. Depending on the purpose 

of the quitline evaluation, outcome evaluation questions may include some or all of those 

listed in Table 2. 
23

1 2 3 4 5 6

Table 2. Outcome Evaluation Areas of Interest for Quitlines and Sample Outcome Evaluation Questions 

Outcome Areas of Interest 

for Quitlines Sample Outcome Evaluation Questions 

Quitline Awareness What proportion of the state’s adult population (tobacco user and nontobacco user) 

is aware of the quitline? 

Intent to Use Quitline Does the proportion of tobacco users reporting plans to call a quitline for help in 

their next quit attempt change after a media campaign?

Interest in Quitting Has interest in calling the quitline for help with quitting changed among the general 

population, callers, or participants (after a media campaign, for example)?

Changes in Motivation to Quit or 

Confidence in Quitting 

How are participants’ motivations to quit or confidence in quitting changing as 

a result of quitline counseling (especially among those not ready to make a quit 

attempt upon registration)?

Quit Attempts How many or what proportion of tobacco users are making at least one 24-hour 

quit attempt since registering for quitline services?

Long-term Quit Success What proportion of quitline participants report no use of any tobacco product for the 

past 7 (or 30) days at 7-month follow-up? (point prevalence abstinence)1

Reduction in Prevalence How has tobacco use prevalence in the state changed over time? Has the change 

been caused by cessation, prevention, or both?

Economic Evaluations

• Cost-effectiveness2

• Cost-benefit analysis

• Cost-utility analysis

• Return on investment

What is your quitline’s cost per quit, including the cost of promoting the quitline?

How does cost per quit compare with other programs your department, agency, or 

organization provides? How does it compare to other quitlines?

How much does each dollar spent on quitline services save the state (or other 

payers, such as health plans) for prevented medical care costs? 

How many Life Years Saved can be attributed to the quitline? 

What is the cost for each Quality Adjusted Life Year saved?

Attribution of outcomes to the 

quitline

Is there a clear link between outcomes related to your quitline activities, as opposed 

to other events occurring at the same time?

What programmatic or policy changes have occurred in your state or various local 

jurisdictions during the evaluation period in question?

Has the tobacco tax rate increased? 

Have smokefree ordinances gone into effect? 

Have media campaigns promoting the national portal number 1-800-QUIT-NOW 

been in the field?

1 For guidance on calculating quit rates for quitlines, see the NAQC Issue Paper, “Measuring Quit Rates” at http://c.ymcdn.com/sites/www.

naquitline.org/resource/resmgr/docs/naqc_issuepaper_measuringqui.pdf.

2 For more on calculating cost-effectiveness, see the NAQC Issue Paper, “Assessing the Cost Effectiveness of Quitline Programs” at 

http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/issue_papers/assessingcosteffectivenessof.pdf.

13
24 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Feasibility Considerations 

In addition, you must determine whether the intended focus of your quitline evaluation is 

realistic and feasible. Resources and logistics will influence decisions about evaluation 

focus in that some evaluation questions are quicker, easier, and cheaper to answer than 

others. Your feasibility discussions should include the budget and resources (financial and 

human) that can be allocated to the evaluation. The 2014 edition of CDC’s Best Practices 

for Comprehensive Tobacco Control Programs5 recommends that 10% of total annual 

tobacco control program funds be allocated for surveillance and evaluation. The questions 

and subsequent methods selected will have a direct relationship to the financial resources 

available, evaluation team member skills, and environmental constraints. The ESW should 

have a thorough discussion of feasibility and recognition of practical constraints to facilitate 

a shared understanding of what the evaluation can and cannot deliver. Early identification 

of inconsistencies between utility and feasibility is an important part of focusing the 

evaluation. For example, process evaluation questions of a monitoring nature would likely 

be less expensive to collect than the evaluative questions because they often are reported 

by the quitline vendor, and the data need less manipulation from an evaluation standpoint. 

The process of selecting the appropriate methods to answer the priority evaluation 

questions and discussing feasibility and efficiency is iterative. Steps 3, 4, and 5 in the 

evaluation process may be carried out concurrently in a cyclical manner until the group 

comes to consensus.

Even with an established multiyear plan, Step 3 should be revisited with the ESW annually 

(or more often, if needed) to determine if priorities and feasibility issues still hold for the 

planned evaluation activities. This highlights the dynamic nature of the evaluation plan. 

Ideally, your plan should be intentional and strategic by design, and it should generally 

cover multiple years. Moreover, the plan should not be regarded as set in stone. On the 

contrary, it should be flexible and adaptive. It must be flexible because resources and 

priorities change, and it must be adaptive because opportunities and programs change. 

Indeed, the quitline environment will likely change over time with respect to available 

resources, quitline service offerings, and engagement of key partners or referral sources. 

The evaluation questions should change as the quitline environment changes, and as the 

evaluation needs of both the quitline and its stakeholders change. Your evaluation plan 

should be sufficiently flexible and adaptive to accommodate these scenarios, and remain 

focused on the evaluation goals and objectives of the program and the ESW.
25

1 2 3 4 5 6

Evaluation Plan Tips for Step 3

ü It is not possible or appropriate to evaluate every 

aspect or specific initiative of a program every year.

ü Evaluation focus is context-dependent and related 

to the purposes of the evaluation, its primary 

users, the program’s stage of development, the 

logic model, program priorities, and feasibility 

considerations.

ü Evaluation questions should be checked against 

the program model. 

ü The iterative nature of plan development is 

reinforced in this step.

ü Transparency for evaluation choices and priorities 

will be important. Implications of each choice 

should be reviewed and discussed with the ESW.

Evaluation Tools and Resources for Step 3:

Part II

ü Illustrating Decisions to Focus the Evaluation

ü 3.1 Focus the Evaluation Exercise
26 | Conducting Quitline Evaluations 

1 2 3 4 5 6
27

1 2 3 4 5 6

STEP 4: GATHER CREDIBLE EVIDENCE

Once the focus of an evaluation has been solidified, 

and the questions to be answered have been identified, 

it is necessary to select the appropriate methods 

that fit the quitline evaluation questions that have 

been selected. If you have followed the steps in this 

workbook, you have worked with the ESW to select the 

evaluation questions that should provide the necessary 

information to guide program improvement and 

decision making. In addition to selecting the methods, 

it is prudent to identify a timeline and to spell out the 

roles and responsibilities of those overseeing the 

implementation of the evaluation, whether it is program 

staff or stakeholders. This will be documented and 

transparent in the evaluation plan.

To accomplish this step, the following is important:

 Keep in mind the purpose, program description or logic model, evaluation questions, 

and available resources.

 The method(s) need to fit the question(s). There are a multitude of options, including 

but not limited to qualitative, quantitative, mixed methods, multiple methods, 

naturalistic inquiry, experimental, and quasi-experimental.

 Think about what will constitute credible evidence for stakeholders or users.

 Identify sources of evidence (e.g., persons, documents, observations, administrative 

databases, surveillance systems) and appropriate methods for obtaining quality (i.e., 

reliable and valid) data.

 Identify roles and responsibilities along with timelines to ensure that the project 

remains on time and on track.

 Remain flexible and adaptive, and as always, transparent.

Fitting the Method to the 

Evaluation Question(s)

The method (or methods) chosen 

need to fit the evaluation question. 

A poor fit between evaluation 

question and method can and 

often does lead to incomplete 

or inaccurate information. The 

method should be appropriate for 

the question, in accordance with 

the Evaluation Standards.
28 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Choosing the Appropriate Methods

Three general types of research designs are commonly recognized: (1) experimental; 

(2) quasi-experimental; (3) and nonexperimental or observational.14 Although program 

evaluations often apply nonexperimental or observational designs, some evaluation 

questions over the life of the quitline may require experimental or quasi-experimental 

designs. We provide information on each type of design below.

Experimental designs use random assignment to compare the outcome of an intervention 

on one or more groups with an equivalent group or groups that did not receive the 

intervention. For example, you could randomly select a group of quitline registrants, and 

then some could be randomly assigned to be offered an enhanced counseling protocol 

of additional counseling topics or more counseling calls or both, or randomly assigned to 

usual care (serving as controls). All participants have the same chance of being assigned to 

the intervention or control group. Random assignment reduces selection bias, the chance 

that the control and intervention groups vary in any way that could influence differences 

in program outcomes. This allows you to attribute change in outcomes to the changes in 

the quitline protocol. However, it is often difficult to provide an enhanced intervention in a 

quitline setting that will produce an outcome that is different enough from the standard, or 

control, intervention to be measurable without requiring a sample size that is prohibitively 

large. In addition, it is generally considered unethical to provide a less intensive service than 

what is currently offered as standard. 

Quasi-experimental designs make comparisons between nonequivalent groups and do 

not involve random assignment to intervention and control groups. An example would be 

to assess quit rates or other quitting-related activities before and after a change in quitline 

protocol, such as the introduction of free nicotine replacement therapy (NRT). After the 

change, you would assess the same outcomes that were assessed before the change, 

and then you would expect to see a change in outcomes (e.g., quit rates) as a result of 

the introduction of free NRT. Critics could argue that other differences (e.g., seasonal 

differences between the two groups) caused the changes in outcomes, so it is important to 

document that the intervention and comparison groups are similar on key factors, such as 

population demographics and tobacco-use history. 

Related to quasi-experimental design, comparison of outcomes or outcome data among 

states and between one state and the nation as a whole are common ways to evaluate 

public health efforts. Such comparisons can help establish meaningful benchmarks for 

progress. States can compare their progress with that of states with a similar investment 
29

1 2 3 4 5 6

in quitlines, similar quitline services offered, or similar populations served, or they can 

contrast their outcomes with those of states with a larger investment or more intensive 

services offered. Quitline data from the National Quitline Data Warehouse (NQDW) are 

available to access by using several online data applications provided by CDC’s Office on 

Smoking and Health. To access full downloadable data sets of all NQDW data, visit the new 

OSHData tobacco use data portal at the following URL: http://www.cdc.gov/OSHData. To 

access preformatted reports available for each state quitline, including data on services 

provided, reach, population served, and quit rates, visit the STATE System at the following 

URL: http://www.cdc.gov/tobacco/STATESystem.

In addition, NAQC regularly provides reports to quitlines that show de-identified aggregate 

metrics on quitline investment (spending per smoker), quit rates, and reach. Each quitline 

can compare their measures to other similar quitlines, although the identity of the other 

quitlines is not reported by NAQC. Please note that comparisons between quitlines should 

be performed with caution because of variations in the types of services provided and 

populations served by each quitline.

Observational designs include, but are not limited to, time series analysis, cross-sectional 

analysis, and case studies. Periodic cross-sectional surveys (e.g., quitline intake or 7-month 

follow-up surveys) can guide an evaluation and be used to calculate quitline reach and quit 

rates. For quitlines, cross-sectional surveys are the most common form of evaluation data 

collection, given the nature of the intervention. 

Consider the appropriateness and feasibility of nonexperimental designs (e.g., simple 

before–after [pretest–posttest] or posttest-only designs). Depending on a program’s 

objectives and the intended use(s) for the evaluation findings, these designs may be more 

suitable for measuring progress toward achieving quitline goals. In the end, it is important 

to choose a design that will measure what will meet both your immediate and long-term 

needs. 

It is beyond the scope of this workbook to discuss in detail the complexities of what 

appropriate methods to choose. It is important to remember that not all methods fit 

all evaluation questions, and often a mixed-methods approach is the best option for a 

comprehensive answer to a particular evaluation question. At this point in the evaluation, 

it is often when it is crucial to consult with evaluation experts for direction on matching 

method to question. More information can be found through the resources listed in Part II. 
30 | Conducting Quitline Evaluations

1 2 3 4 5 6

Credible Evidence

The evidence gathered to support the answers to your 

evaluation questions should be seen as credible by the 

primary users of the evaluation. The determination of what 

is credible is often dependent on context and can vary 

across programs and stakeholders. The determination 

of credible evidence is naturally tied to the evaluation 

design, implementation, and standards to which the data 

collection, analysis, and interpretation are adhered. Best 

practices for a program area and the evaluation standards 

included in the CDC Framework (Utility, Feasibility, 

Propriety, and Accuracy)9

 will facilitate discussing what 

constitutes credible evidence with an ESW. 

The use of standard data collection instruments, such 

as the National Quitline Data Warehouse Quitline 

Services Survey (administered to quitlines quarterly), 

or the Minimal Data Set for Quitlines on which the NQDW instruments are based, can help 

with issues of credibility with stakeholders. In addition, reporting findings by using standard 

methodologies for outcomes, such as reach and quit rates, can also raise the credibility 

of your evaluation results. For more information on the Minimal Data Set, see http://www.

naquitline.org/mds. For more information on NAQC-recommended methods for calculating 

reach and quit rates, see “Measuring Reach of Quitline Programs”15 and “Calculating Quit 

Rates: 2015 Update.”16

Measurement

Often, there is a wide range of possible measures or indicators that can be selected for any 

one evaluation question. In addition, there may be reporting needs that dictate which 

measures you will want to collect to answer a specific evaluation question. For example, 

the publication Key Outcome Indicators for Evaluating Comprehensive Tobacco Control 

Programs17 includes several outcomes under Goal 3, “Promoting Quitting Among Adults 

and Young People,” that can be reported by using quitline data.17 These include

Outcome 7: Establishment or Increased Use of Cessation Services 

Outcome 11: Increased Number of Quit Attempts and Quit Attempts Using Proven 

Cessation Methods 

Outcome 13: Increased Cessation Among Adults and Young People 

Evaluation Standards from 

the Framework

Utility: Will some methods make 

the data more credible with 

skeptics or key users? 

Feasibility: What methods can 

you afford? 

Propriety: What ethical 

considerations do you need to 

address?

Accuracy: Are the data collection 

methods likely to influence the 

answers given by respondents?
31

1 2 3 4 5 6

If a quitline’s goals include progress toward these or other outcomes, your evaluation 

questions and measures should reflect that. We have provided example questions and 

measures or indicators in Part II, Step 4 of this workbook. For more information on 

selecting valid and reliable indicators and measures, consult the CDC Guide to Key 

Outcome Indicators for Evaluating Comprehensive Tobacco Programs (2005)17 or an 

evaluation consultant. 

In Step 3, we discussed the difference between types 

of evaluation questions, such as process and outcome 

questions. For each area of interest for quitlines, we 

provided sample evaluation questions in Tables 1 and 

2. For Step 4, it is important to link each evaluation 

question to a measure or measures that can help 

answer it. When selecting a measure, evaluators 

should consider whether it is already being collected 

or not, and if not, what the feasibility would be of 

creating a new data stream. It may also be helpful to 

consider whether the measure is one that is commonly 

used by others within the quitline community, and 

whether it can be used for multiple purposes. All 

measures need to be carefully defined in process (e.g., 

what constitutes a call?) and outcome (e.g., how is 

a quit measured?) evaluation; and when measuring 

trends, these definitions should not change over time 

without being explicit.18 Sample evaluation questions 

and measures are included in Part II, Step 4 of this workbook. 

Many measures for process or implementation evaluation and their definitions can be 

found in the NAQC Issue Paper, “Call Center Metrics: Best Practices in Performance 

Measurement and Management to Maximize Quitline Efficiency and Quality.”19 This issue 

paper defines the most critical operational, service performance, and efficiency-related 

call center metrics for establishing and maintaining quitline quality. The paper makes 

recommendations on important metrics to measure, including why they are important for 

funders, service providers, and callers, and how to measure and report these metrics, and 

describes how these reports may be used by both the service providers and funders to 

improve quality.

Considerations for Selecting 

Measures

When selecting measures for 

specific evaluation questions, you 

may want to consider the following:

• Does the measure help 

me answer my evaluation 

questions?

• Is the measure already being 

collected?

• Is it a standard measure?

Sample measures mapped to 

evaluation questions are included in 

Part II, Step 4.
32 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Demographic and tobacco use characteristics of quitline participants can be collected 

through implementation of the National Quitline Data Warehouse (NQDW) surveys. One 

of the surveys, the NQDW Intake Questionnaire, is administered to all callers during their 

first registration call. The questionnaire collects data on demographics (e.g., age, gender, 

race/ethnicity, education), current tobacco use and history, intention to quit smoking 

or using tobacco (if currently using), and how the caller heard about and reached the 

quitline. This questionnaire is adapted from NAQC’s Minimum Data Set (MDS) Intake 

Questions. See http://www.cdc.gov/tobacco/quit_smoking/cessation/nqdw/ and http://

www.naquitline.org/mds for more information, including services, intake, and follow-up 

survey instruments). The NQDW offers a standard approach to collecting measures for 

evaluation of tobacco cessation quitlines. Having a standard set of variables is valuable for 

the following: 

 Establishing commonly defined performance indicators (e.g., quitline reach, quit 

rate) to assist in assessing quitline performance, improving the quality of quitlines, 

identifying knowledge gaps, and designing strategies to fill the gaps.

 Providing a common language allowing for consistent communication with others 

within and external to the quitline community. 

 Establishing quitline performance benchmarks that can be used to identify effective, 

cost-efficient tobacco cessation interventions. 

 Testing and assessing new treatment techniques across large, diverse populations. 

 Collecting consistent data and allowing aggregation of data across quitlines to 

make possible improved analyses of a variety of variables relevant to the success of 

quitlines in the United States. 

Reach calculations and definitions can be found in the NAQC Issue Paper, “Measuring 

Reach of Quitline Programs,”15 which recommends a standard definition of reach and 

provides recommendations for standard measuring and reporting of the reach of quitline 

programs.

Data Sources 

As emphasized already, it is important to select the method or methods that are most

appropriate to answer the evaluation question. The types of data needed should be 

reviewed and considered for credibility and feasibility. On the basis of the methods chosen, 

you may need a variety of input, such as the quitline’s intake and follow-up surveys, quitline 

administrative and utilization data, focus groups, and other quality assurance methods. 
33

1 2 3 4 5 6

You may need to consider multiple data sources and the triangulation of data for reliability 

and validity of the information. Quitline data can be combined and compared with data 

from existing sources (e.g., BRFSS, ATS, TUS-CPS, US Census). The form of the data 

(e.g., quantitative or qualitative) and specifics of how these data will be collected should be 

defined, agreed upon as credible, and the rationale for these choices should be transparent 

to all involved. There are strengths and limitations to various types of data, and they should 

be considered carefully with the help of your ESW. For example, the use of Minimal Data 

Set intake or follow-up survey items may help reduce costs, maximize the use of existing 

information, and facilitate comparability with other programs, but may not provide program 

specificity. Additional questions may need to be added to your quitline’s intake or follow-up 

survey instruments to answer your evaluation questions as identified in Step 3.

All data collected should have a clear link to the associated evaluation question and 

anticipated use to reduce unnecessary burden on the respondent and stakeholders. It is 

important to revisit data collection efforts over the course of a multiyear evaluation plan 

to examine utility against the burden on respondents and stakeholders. Finally, quality 

assurance procedures must be put into place so that data are collected in a reliable way, 

coded and entered correctly, and checked 

for accuracy. Many valid and reliable 

data sources have been consolidated 

and provided through CDC’s OSHData 

(see http://www.cdc.gov/oshdata/) 

and STATE system (see www.cdc.gov/

tobacco/statesystem). OSHData presents 

comprehensive tobacco prevention and 

control data in an online, easy to use, 

interactive data application. The STATE 

System is an interactive application that 

houses and displays current and historical 

state-level data about tobacco use 

prevention and control in preformatted 

reports for easy access.

These and other resources related to data 

sources and measures are available in the 

Resources Section.

Reliability and Validity

When monitoring call volume, ensure you are 

measuring call volumes the same at each time. 

Did these include calls answered live, sent to voice 

mail, hung up or abandoned? You should define 

and use the same measure for reporting purposes 

to reliably assess call volume over time.

Counseling protocols are determined by each 

quitline. When monitoring services are received, 

you want to know whether your data represent 

a valid measure of actual services received. For 

example, are reminder calls being counted as 

counseling calls? Are counselors following the 

counseling protocol? A thorough assessment of 

fidelity to the protocol could demonstrate the 

validity of your treatment data.
34 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Evaluation Methods Checklist

As has been stated above, the methods selected for a quitline evaluation must match 

the evaluation questions identified in Step 3. In addition to considering the specific 

methodology, the evidence that is considered credible by your stakeholders, measures that 

can help answer evaluation questions, and data sources from which to collect measures, 

it may also be beneficial to go back to the Evaluation Framework (Figure 1), and the four 

evaluation standards that form the heart of the Framework: Utility, Feasibility, Propriety, and 

Accuracy. These standards can help focus decisions around data collection more clearly 

and ensure that the results of your quitline’s evaluation will be most useful. An Evaluation 

Methods Checklist exercise has been included in Part II, Section 4.1 of this workbook, 

and can serve as a guide related to each of the evaluation standards when considering 

methods, credibility of evidence, measurement, and data sources.

Evaluation Methods Grid

One tool that is particularly useful in your evaluation activities is an evaluation methods 

grid. This tool is helpful to align evaluation questions with methods, indicators, 

performance measures, data sources, roles, and responsibilities, and it can facilitate a 

shared understanding of the overall evaluation plan with stakeholders. As you develop an 

understanding of the quitline’s components, identify evaluation questions that are important 

to answer and select methods that can help you answer those questions. It will be very 

important to carefully assign roles to all of the parties involved. For example, if you have an 

external evaluator working on the quitline evaluation, which parts of the evaluation method 

selection and data collection will fall under their responsibility? How often will the ESW 

meet? What is the scope of the ESW’s work? The Evaluation Methods Grid can be helpful 

for linking all of the pieces of the evaluation puzzle together that have been identified so 

far. The tool can take many forms and should be adapted to fit your specific evaluation and 

context. Tables 3 and 4 below provide illustrations for methods grids; more examples are 

provided in Part II, Exercise 4.2.

An Evaluation Methods Checklist exercise is located in Part II, Section 4.1 of 

this workbook.
35

1 2 3 4 5 6

Table 3. Evaluation Methods Grid, Example A

Evaluation 

Question

Indicator/ 

Performance 

Measure Method Data Source Frequency Responsibility

What is the 

effect of 

offering free 

NRT through 

the quitline?

Number of calls, 

number of tobacco 

users receiving 

services, 7-month 

quit rate, cost per 

quit.

Cross-sectional 

observational prepost design.

Quitline intake 

data, quitline 

administrative and 

use data, 7-month 

follow-up survey 

data, quitline budget 

data.

Pre and post 

introduction of 

NRT.

Quitline service 

provider (intake, 

administrative 

data); quitline 

evaluator (followup survey and 

analysis).

Table 4. Evaluation Methods Grid, Example B

Evaluation 

Question

Indicators/ Performance 

Measure

Potential Data Source 

(Existing/New) Comments

How effective are 

quitline promotional 

activities at increasing 

demand for quitline 

services?

Description of promotional 

activities (timing, duration, 

content), number of tobacco 

users calling the quitline 

and their demographics, 

referral source, as reported 

by tobacco users, “how heard 

about the quitline?”

TRP and GRP data sources.

Registration/intake data.

Call volume data.

Budget

The evaluation budget discussion was most likely started during Step 3, when the team 

was discussing the focus of the evaluation and feasibility issues. It is now time to develop 

a complete evaluation project budget on the basis of the decisions made about the 

evaluation questions, methods, roles, and responsibilities of stakeholders. A complete 

budget is necessary to ensure that the evaluation project is fully funded and can deliver 

upon promises. The evaluation questions, measures, and priorities may need to be revisited 

as part of this step, depending on how well they match up with available resources.
36 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Evaluation Tools and Resources for Step 4:

Part II

ü Matching Measures to Evaluation Questions

ü 4.1 Evaluation Methods Checklist

ü 4.2 Evaluation Methods Grid Exercise

ü 4.3 Evaluation Budget Exercise

Evaluation Tips for Step 4

ü	Select the method(s) that is best suited to answer 

the evaluation questions. This can often involve a 

mixed methods approach.

ü	Gather evidence that is seen as credible by the 

primary users of the evaluation.

ü	Define implementation roles and responsibilities 

for program staff, evaluation staff, contractors, and 

stakeholders. Ensure there is sufficient expertise 

on the evaluation team to conduct the planned 

evaluation activities and analysis.

ü	Develop an evaluation plan methods grid to facilitate 

a shared understanding of the overall evaluation 

plan, and the timeline and budget for evaluation 

activities.
37

1 2 3 4 5 6

STEP 5: JUSTIFY CONCLUSIONS

Justifying conclusions includes analyzing the information collected, interpreting this 

information, and drawing conclusions from the data. This step is needed to turn the quitline 

data collected into meaningful, useful, and accessible information that can be used to 

engage quitline stakeholders and decision makers. This is often when programs incorrectly 

assume they no longer need the ESW to remain integrally involved in decision making, and 

instead look to the experts to complete the analysis and interpretation. However, engaging 

the ESW in this step is critical to ensuring the meaningfulness, credibility, and acceptance 

of quitline evaluation findings and conclusions. Continuing to consult with stakeholders and 

discussing preliminary findings with them helps to guide the interpretation phase. In fact, 

quitline stakeholders often have novel insights or perspectives that evaluation staff may not 

have, leading to richer interpretation and more fully thought-out conclusions. 

Planning for analysis and interpretation of quitline evaluation data is directly tied to the 

timetable initiated in Step 4. Involving others in Step 5 (analysis, interpretation, and 

justifying conclusions) is a prerequisite to arriving at valid and defensible results, but 

it can take a significant amount of time. Errors or omissions in planning this step can 

create serious delays in completing the final evaluation report, and may result in missed 

opportunities if the report has been timed to correspond with key events or decisions. 

After planning for the analysis of the data, you will prepare to examine the results to 

determine what the data actually say about the program. These results should be 

interpreted in light of the goals of your program, its social or political context, and the 

needs of the stakeholders.

Moreover, it is critical that your plans allow time for interpretation and review by quitline 

stakeholders to ensure the transparency of your process and the validity of the conclusions. 

The emphasis here should be on justifying conclusions, not just analyzing data. This is a 

step that deserves due diligence. It is the responsibility of the evaluator and the ESW to 

ensure that conclusions are drawn directly from the evidence. This is a topic that should 

be discussed with the ESW in the planning stages, along with reliability and validity issues 

and possible sources of bias. If possible and appropriate, triangulation of data should be 

considered, and remedies to threats to the credibility of the data should be addressed as 

early as possible.

A Stakeholder Interpretation Meeting exercise is found in Part II, Section 5.1
38 | Conducting Quitline Evaluations

1 2 3 4 5 6

Provide Context: Compare Results to Other Types of Programs

Evaluation results should always be provided within a specific context, and that context 

should be referenced in the analysis and interpretation of results. Below are some examples 

to help clarify this point.

Example 1: Interpreting Quit Rates Within the Context of Prior Research

One evaluation question that quitlines often ask is how effective their program is at helping 

tobacco users quit. When the results are in, it is important to understand some of the 

basic context before the quit rate for the quitline can be appropriately interpreted. First, 

it is important to recognize that quitlines are 60% more effective at helping people quit 

than no help.2 In addition, providing medications with counseling is 70% more effective 

than providing counseling alone.2 According to a recent literature review, the range of 

reported quit rates was 16%-23% for quitlines providing counseling alone, and 30%-36% 

for quitlines providing medication.20 Recent research showed that the average quit rate for 

quitlines not providing free medications was 24.8% (n = 10), and for quitlines providing free 

medication, 30.3% (n = 33).21

Example 2: Interpreting Quit Rate Changes Over Time

Once results are available, examine them within the context of quit rates for quitlines 

as noted above. When you examine a quitline’s quit rates over time, it’s important to 

maintain a single definition of what constitutes a quit.18 Quitline ‘A’ designed an evaluation 

that included assessing quit rates for all tobacco users served by the quitline. Over the 

course of 3 years, the quit rate has dropped from 28% to 21% (NAQC standard quit rate 

calculation for conventional tobacco only – see NAQC, 201516). The Department of Health 

was concerned about the drop. The ESW included representatives from community clinics 

that provide a large number of referrals to the quitline. During an evaluation meeting, 

they pointed out that the proportion of quitline users with lower education levels and 

Medicaid patients increased dramatically over the same period. The association of low 

socioeconomic status with lower quit rates22 could at least partially explain the observed 

drop in quit rates. The results were graphically represented in an easily interpreted way, as 

follows in Figure 3.
39

1 2 3 4 5 6

Figure 3. Graphic Representation of Quit Rates, Proportion of Medicaid Callers, and Proportion of Low 

Education Callers Over Time

Example 3. Interpreting the Effect of Quitline Promotions

Quitline B was launching a new media campaign and designed an evaluation to assess 

the campaign’s impact on quitline reach, quality, and outcomes. After the campaign had 

aired for 3 months, the ESW met to review preliminary results. Over the 3-month period, 

the number of calls to the quitline had tripled compared with the same 3 months of the year 

prior to the media campaign, thus exceeding the goal for the campaign (Figure 4). 
40 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Figure 4. Graphically Demonstrating the Impact of a Quitline’s Media Plan

However, the ESW included counseling staff from the quitline’s service provider, who called 

attention to the fact that the increase in call volume was making it impossible to connect 

with all callers in a timely fashion, and that it was taking up to several weeks to return calls, 

as shown in Figure 5. 

Figure 5. Mapping Call Volume to Average Time to Enrollment
41

1 2 3 4 5 6

On the basis of these results, and the contextual information provided by the ESW, a 

decision was made to modify the counseling protocol during periods of high call volume to 

limit the number of counseling calls until the backlog of callers had been provided at least 

one counseling call. 

As this example shows, “impact” cannot be appropriately and completely assessed without 

all aspects of the quitline program being taken into consideration. Having representatives 

from all stakeholder groups on the ESW, including those involved in provision of services, 

can be a critical component of interpreting the evaluation results and justifying your 

conclusions.

Provide Context: Present Evaluation Results with Other Data

For some evaluation questions, it is important to present specific measures, and those 

measures alongside other measures that can help provide the appropriate context. 

Example 4. Interpreting the Impact of Quitline Promotions (Part 2)

Colorado was interested in measuring the response to promotions that they had been 

conducting during 2010 by looking at call volume over time. However, they were also 

interested in seeing how higher call volume impacted the overall readiness of callers 

to engage in counseling. To assess this, they looked at the number of calls that were 

answered “live,” or by talking to a quitline staff member, rather than going to voicemail or 

having the person hang up. Further, they recorded the number of callers who registered 

for counseling. They then divided the number of callers registering for counseling by the 

total number of calls that were answered live to arrive at a measure of productive calls, 

or the proportion of calls answered live that resulted in registration for quitline counseling 

services. The graphic representation of this information is provided in Figure 6.
42 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Figure 6. Quitline Call Volume and Phone Counseling Provided, Colorado 

(Jan. – Dec. 2010)

During the year, the number of calls that were answered live increased. At the same time, 

the proportion of calls answered live that resulted in telephone counseling being delivered 

declined slightly during the year. Additional information is needed to determine whether 

the decrease in the number of productive calls resulted from a change in the population 

of tobacco users calling the quitline (e.g., an increase in the number of callers who are not 

prepared to engage in the quitting process), or a decline in service delivery standards (e.g., 

failure to follow up with callers in a timely fashion, resulting in higher rates of lost callers).

Example 5. Interpreting the Demographics of Quitline Callers: How Well Is the Quitline Serving 

Its Intended Population?

Another common evaluation question is, “who is the quitline serving?” It is a straightforward 

matter to report the number and proportion of quitline callers belonging to different age 

groups, genders, and racial or ethnic groups, and reporting different education levels and 
43

1 2 3 4 5 6

patterns of tobacco use (e.g., type of tobacco use, heaviness of use). However, on its own, 

this information does not answer the question of how well a quitline is serving its intended 

population. To answer this question, a quitline must compare the proportion of quitline 

callers in various categories to the proportion of tobacco users in its target population in 

those same categories. The Vermont Quit by Phone program made such a comparison in 

Figure 7:

Figure 7. Demographic Summary of New Registrants, Vermont Quit Network: Quit by Phone Program, 

FY 2010

Source: Vermont Quit by Phone Program, Intake Data for Fiscal Year (FY) 2010, 2008 Vermont Adult Tobacco 

Survey (ATS)

From this graph, the Vermont program could see that they were proportionally serving more 

45-64-year-olds and women than were represented in the general population of Vermont 

smokers. Although this is typical for quitlines nationwide (NQDW, 2013), it may raise a 

question for quitlines: Should they do more to target promotional efforts at younger and 

older smokers, as well as men? 

As these examples illustrate, it is critical to involve all stakeholders in the evaluation 

process from the beginning because it is not always clear what elements of the system will 

be relevant when interpreting results.
44 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Reporting with Clarity

Steps 5 and 6 are about interpretation, engaged data, and ensuring use of evaluation data. 

According to Heath and Heath,23 ideas that “stick” are understandable, memorable, and 

effective in changing thought or behavior. For stakeholders and decision makers to retain 

the knowledge in your evaluation, the data must be presented in simple, clear terms. The 

core message must not be muddied by distractions in the report, and the results must be 

concrete. The evaluation results must be humanized and delivered in terms that are credible 

and actionable.4

To make the information visually appealing and easy to read, consider using graphic design 

best practices or the assistance of a graphic design expert when formatting your evaluation 

report, success stories, or briefing.4 The STATE system provides interactive tools and 

ready-made reports that are easy to read and designed to present data in a graphically 

appealing manner. You can insert the graphs for your state into reports or download the 

Quitline Highlights Report as a PDF. Additional material on reporting with clarity can be 

found in Developing an Effective Evaluation Report.

4
45

1 2 3 4 5 6

Evaluation Tools and Resources for Step 5:

 Part II

ü 5.1 Stakeholder Interpretation Meeting Exercise

ü Checklist for formal stakeholder interpretation meeting

Evaluation Tips for Step 5

ü The justifying conclusions step is needed to translate 

the data collected into meaningful, useful, and 

accessible information for action.

ü Including your stakeholder group in this step is directly 

tied to the previous discussion on the credibility of 

data and conclusions. 

ü The composition of the stakeholder group is key—it is 

not always clear at the outset which perspectives will 

be important with respect to interpreting the findings.

ü Errors in planning in this step can create serious delays 

in completing the final evaluation report and may result 

in missed opportunities if the report has been timed to 

correspond to important events or decisions. 

ü It is critical that your plans build in time for 

interpretation and review by stakeholders, (including 

critics of the program), to increase the transparency of 

your process and the validity of your conclusions.

ü The standards and values of less powerful stakeholders 

and of those stakeholders most directly affected by the 

quitline should be given special weight. Conclusions 

should be fully understandable to stakeholders.

ü Limitations of the evaluation should be examined.
46 | Conducting Quitline Evaluations 

1 2 3 4 5 6
47

1 2 3 4 5 6

STEP 6. ENSURE USE OF EVALUATION FINDINGS AND 

SHARE LESSONS LEARNED

The ultimate purpose of program evaluation is to use the information to improve programs 

and engage stakeholders. The purpose(s) you identified early in the evaluation process 

should guide the use of the evaluation results. The evaluation results can be used to 

demonstrate the effectiveness of your quitline, identify ways to improve your program, 

modify program planning, demonstrate accountability, and justify funding. Step 6 of the 

Evaluation Framework, Ensure Use of Evaluation Findings and Share Lessons Learned, 

should be the ultimate goal of all your evaluation planning and implementation. It is 

sometimes felt that this step is automatic once the report is published. In reality, planning 

for how the evaluation results will be used begins with Step 1 and the consideration of 

stakeholder involvement. Evaluation results are more likely to be used when end-use is 

planned for and built into the six steps in your evaluation plan. Planning for use is directly 

tied to the identified purposes of the evaluation and program and stakeholder priorities. By 

including the ESW in the entire plan development process, you begin building a market for 

your evaluation results and increase the chances that these results will be used for program 

improvement and decision making. This step is directly tied to the utility standard in 

evaluation. Is it ethical to consume program and stakeholder resources if evaluation results 

are never used or are used in a less than optimal way? Use must be planned for, nurtured, 

and included in the evaluation plan from the very beginning.

Five elements are important to ensuring that the findings from an evaluation are used and 

that lessons learned are shared: 

 Preparation 

 Feedback 

 Follow-up 

 Dissemination 

 Making Recommendations 

Use must be planned for, cultivated, and included in the evaluation plan from the 

very beginning.
48 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Preparation 

Preparation refers to the steps taken to use the evaluation findings. Through preparation, 

stakeholders can:

 Strengthen their ability to translate knowledge into appropriate action. 

 Discuss how potential findings might affect decision making. 

 Explore positive and negative implications of potential results, and identify options for 

program improvement. 

Feedback 

Feedback occurs among everyone involved in the evaluation. Feedback, which is 

necessary at all stages of the evaluation process, creates an atmosphere of trust among 

stakeholders. Early in an evaluation, giving and receiving feedback keeps an evaluation on 

track by keeping everyone informed about how the program is being implemented and how 

the evaluation is proceeding. As the evaluation progresses and preliminary results become 

available, feedback helps ensure that primary users and other stakeholders can comment 

on evaluation decisions. Valuable feedback can be obtained by holding discussions and 

routinely sharing interim findings, provisional interpretations, and draft reports. 

Follow-up 

Follow-up refers to the support that users need throughout the evaluation process. In this 

step, it refers to the support users need after receiving evaluation results and beginning to 

reach and justify their conclusions. Active follow-up can achieve the following: 

 Remind users of how you intend to use what you have learned. 

 Help to prevent misuse of results by ensuring that evidence is applied to the 

questions that were the evaluation’s central focus. 

 Prevent lessons learned from becoming lost or ignored in the process of making 

complex or political decisions. 

Dissemination 

Dissemination involves communicating evaluation procedures or lessons learned to 

relevant audiences in a timely, unbiased, and consistent manner. Regardless of how 

communications are structured, the goal of dissemination is to achieve full disclosure and 

impartial reporting. Planning effective communications requires advance discussion of the 
49

1 2 3 4 5 6

reporting strategy with intended users and other stakeholders, and matching the timing, 

style, tone, message source, vehicle, and format of information products to the audience.

Some methods of delivering the information to audiences include the following: 

 Mailings 

 Web sites 

 Community forums 

 Media (television, radio, newspaper, social media) 

 Personal contacts 

 Listservs 

 Organizational newsletters 

How you present your evaluation results will vary depending on how the information will be 

used. This can be specified in the Communications Plan (discussed below).

Making Recommendations

Recommendations are actions to consider as a result of an evaluation. Recommendations 

can strengthen an evaluation when they anticipate and react to what users want to 

know, and may undermine an evaluation’s credibility if they are not supported by enough 

evidence, or are not in keeping with stakeholders’ values. 

Recommendations will depend on the purpose of the evaluation and on your audience. It 

is important to remember that many or all of these key audiences were identified in Step 1, 

and you have engaged many of them throughout the evaluation as stakeholders. Hence, 

you have maximized the chances that the recommendations are relevant and useful to your 

key audiences. You know what information your stakeholders want and what is important to 

them. The feedback they provided early in the evaluation process should make them more 

likely to support the recommendations. 
50 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Developing a Communications Plan

Given that the communication objectives will be tailored to each target audience, it is 

important to consider along with the ESW who the primary audiences are (e.g., the ESW, 

the funding agency, the general public, or some other group). Some questions to ask about 

the potential audiences are:

 Who is a priority?

 What do they already know about the topic? 

 What is critical for them to learn about the evaluation findings?

 How do we want them to use the information? What do we want them to do with it?

 Where do they prefer to receive their information? 

 What is their preferred format? 

 What language level is appropriate?

 Within what timeframe are evaluation updates and reports needed? 

Developing a visual chart of the quitline and the system in which it is embedded (either 

a logic model or a system dynamic map) can help identify places to intervene through 

effective communication strategies with key stakeholders. For example, if poor public 

awareness (or worse, negative public opinion) of the quitline has reduced decision makers’ 

support for continued quitline funding, part of the communications strategy could be to get 

as much media coverage as possible for success stories of “happy quitters.” 

Once the goals, objectives, and target audiences of the communication plan are 

established, you should consider the best way to reach the intended audiences by 

assessing which communication or dissemination channels will best serve your goals 

and objectives. Will the quitline use websites, oral presentations, visual displays, videos, 

storytelling, or press releases? What tools will be used in addition to a final report: 

executive summaries, slide decks, newsletters or fact sheets, infographics, success 

The Communicating Results exercise can be found in Part II Section 6.2 and can assist 

with tracking your audiences and ways to reach them. More information on developing a 

communication and dissemination plan can be found in the Resource Section in Part II 

of this workbook.
51

1 2 3 4 5 6

stories, one-pagers? Carefully consider the best channels and tools to use drawing on 

feedback from your evaluation stakeholder workgroup, target audiences, and from others’ 

experiences. An excellent resource on creative techniques for reporting evaluation results 

is Torres, Preskill, and Pionteck’s (2004) Evaluation Strategies for Communicating and 

Reporting.

24

The following example (Table 5) can help the program chart the written communications plan:

Table 5. Quitline Evaluation Communication Plan Example

Target audience 

(priority) Goals Tools Timetable

Quitline funder Inform them in real time about what’s 

working well and what needs to be 

modified.

Monthly meetings and 

briefing documents

Monthly

Quitline service 

provider

Inform them in real time about what’s 

working well and what needs to be 

modified.

Daily operational 

dashboard, weekly and 

monthly meetings and 

reports

Daily, weekly, monthly

Stakeholders Promote quitline awareness and support. Informational “fact 

sheets” and visits

Annually

Media Promote quitline awareness for the 

general population, create a social norm 

around quitting.

Testimonials, letters 

to the editor, success 

stories

Quarterly, post-campaign, or 

upon evaluation results

It is not necessary or even ideal to wait until the final evaluation report is written to share 

evaluation results. A system for sharing interim results to facilitate program course 

corrections and decision making should be included in the evaluation plan. For example, a 

success story can show movement in your program’s progress over time and demonstrate 

its value and impact. Success stories that focus on upstream, midstream, and downstream 

successes can facilitate program growth and visibility.25 Upstream success involves 

the development of promotional materials before the campaign is launched. Midstream 

success deals with campaign implementation and how it is fielded and received by 

the target audience. And downstream success looks at the intended outcomes of the 

campaign for the target audience. See Figure 8 for more specific examples. Success stories 

can also serve as a vehicle for engaging potential participants, partners, and funders, 

especially when it takes time for a program to mature to long-term outcomes.25-26
52 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Figure 8. Examples of Upstream, Midstream, and Downstream Successes with a Quitline Media 

Campaign

Communicating evaluation results is not enough to ensure use of these results and of 

the associated lessons learned. It is important for the evaluation team and program staff 

to proactively encourage wide dissemination and use of information gleaned through 

the evaluation to accomplish particular evaluation goals. It is helpful to strategize with 

stakeholders early in the evaluation process about how your program will ensure that 

findings are used to support programmatic improvement efforts and guide decision making. 

Example 1. Using Evaluation Results to Improve Quitline Practice

One state quitline was offering 8 weeks of NRT to quitline callers. One of their evaluation 

questions was whether this was an efficient use of resources. To answer the question, the 

quitline evaluator conducted a survey 3 months after registration, and asked about callers’ 

experiences with the quitline, including their receipt and use of NRT. Results showed that 

although an 8-week supply of NRT was sent at one time, on average, participants were 

using just more than 5 weeks of NRT. After reviewing these results, the quitline began 

splitting the shipment of NRT into one 5-week shipment and one 3-week shipment, with 

the second shipment being contingent on completing a counseling call around 3 weeks 

after initial contact with the quitline. Subsequent evaluation results showed that caller 

satisfaction and quit rates remained high, whereas costs decreased. Use of the evaluation 

results was ensured by designing the evaluation questions specifically around how they 

would be used to improve practice.
53

1 2 3 4 5 6

Example 2. Increasing Stakeholder Support for Quitlines

One state quitline was dependent on state legislative funding allocations. To increase 

legislative awareness of the quitline and stakeholder support for continued quitline funding, 

quitline use and quit data were collected and reported on cumulatively and by legislative 

district. Fact sheets were distributed annually to each state legislator, which included districtspecific information about the number of constituents who called the state quitline, received 

quitline services, and successfully quit tobacco use. In addition, testimonials from “happy 

quitters” were collected and included in the fact sheets. Although state legislators were not 

included in this quitline’s ESW, they were identified as a key stakeholder group, and evaluation 

results were disseminated to them in a format they could easily understand and use.

Example 3. Communicating Success Stories

In 2010, the CDC funded quitlines with Communities Putting Prevention to Work (CPPW) 

funding to increase their capacity for serving tobacco users. As part of the evaluation of the 

effect of CPPW funds, CDC produced several success stories with funded communities, 

highlighting the ways that funds were used and the impact those funds had on quitlines and 

tobacco control programs. One success story was an advertising campaign for the Kansas 

Quitline developed by the Kansas Department of Health and Environment’s Tobacco Use 

Prevention Program (TUPP).The summary of the Kansas campaign described the rationale 

for developing the campaign, the content of the two ads that were developed, and provided 

key results of the campaign’s impact (http://www.cdc.gov/tobacco/tobacco_control_

programs/surveillance_evaluation/success-stories/pdfs/kansas.pdf). For more on how to 

create your own success story, see Part II, section 6.3 and CDC, 2007.25

There are several practical steps you can include in your evaluation plan to help ensure that 

the evaluation findings are used. These might include plans to:

 Conduct regularly scheduled meetings with evaluation stakeholders as a forum for 

sharing evaluation findings in real time and developing recommendations for program 

improvement on the basis of evaluation findings (discussed in Step 5). 

 Review evaluation findings and recommendations in regularly scheduled staff 

meetings.

 Engage stakeholders in identifying ways that they can apply evaluation findings to 

improve programs. 
54 | Conducting Quitline Evaluations 

1 2 3 4 5 6

 Coordinate, document, and monitor efforts that program staff and partners are 

making to implement recommendations for program improvement.

 Develop multiple, tailored evaluation reports to address specific stakeholders’ 

information needs.

Summary

The impact of quitline evaluation results can reach far beyond the evaluation report. If 

stakeholders are involved throughout the process, communication and participation may 

also be enhanced. If an effective feedback loop is in place, quitline improvement and 

outcomes may be enhanced. If a strong commitment to sharing lessons learned and 

success stories is in place, then other cessation programs may benefit from the information 

gleaned through the evaluation process. Changes in thinking, understanding, program, and 

organization for all stakeholders may stem from thoughtful evaluative processes.8
55

1 2 3 4 5 6

Evaluation Tips for Step 6

 Planning for use begins with Step 1 and the consideration of 

stakeholder involvement.

ü Use of the evaluation results is most likely to occur when end-use is 

planned for and built into the 6 steps in your evaluation plan. 

ü Use of evaluation findings is most likely to occur when the 

evaluation is collaborative and participatory, a process that begins 

in the planning phase.

ü Your evaluation plan should include a communication and 

dissemination plan. 

ü The planning stage is the time for the program to begin to think 

about the best way to share the lessons you will learn from the 

evaluation.

ü In addition to your final report, you will want to tailor reports that 

highlight specific findings for selected groups of stakeholders. 

Consider the target audience when making decisions about these 

reports’ timing, style, tone, message source, method, and format. 

Stakeholder follow-through should be strategically encouraged as 

you consider these factors. Planning for these reports begins with 

your evaluation plan.

ü Findings should also be made accessible to everyone affected by 

the evaluation. 

Evaluation Tools and Resources for Step 6:

 Part II

ü 6.1 Reporting Checklist Exercise

ü 6.2 Communicating Results Exercise

ü 6.3 Success Stories Examples and Tools
56 | Conducting Quitline Evaluations 

1 2 3 4 5 6
57

References

1. NAQC. What is a Quitline? http://www.naquitline.org/?page=whatisquitline. Accessed June 1, 2015. 

2. Fiore M, Jaen C, Baker T, et al. (2008). A clinical practice guideline treating tobacco use and dependence: 

2008 update. Rockville MD: USDHHS, U.S. Public Health Service. Retrieved from 

http://www.surgeongeneral.gov/tobacco/treating_tobacco_use08.pdf

3. Centers for Disease Control and Prevention. Developing an Effective Evaluation Plan. Atlanta, GA: US Dept 

of Health and Human Services, Centers for Disease Control and Prevention; 2011. http://www.cdc.gov/

tobacco/tobacco_control_programs/surveillance_evaluation/evaluation_plan/index.htm. Accessed May 

21, 2015.

4. Centers for Disease Control and Prevention. Developing an Effective Evaluation Report. Atlanta, GA: US 

Dept of Health and Human Services, Centers for Disease Control and Prevention; 2013. http://www.cdc.

gov/tobacco/tobacco_control_programs/surveillance_evaluation/index.htm. Accessed May 21, 2015.

5. Centers for Disease Control and Prevention. Best Practices for Comprehensive Tobacco Control 

Programs. Atlanta, GA: US Dept of Health and Human Services, Centers for Disease Control and 

Prevention; 2014. http://www.cdc.gov/tobacco/stateandcommunity/best_practices/index.htm. Accessed 

May 21, 2015.

6. Lavinghouze R, Snyder K, & Rieker P. (2014). The component model of infrastructure: A practical approach 

to understanding public health program infrastructure. American Journal of Public Health, 104(8), e14-e24.

7. Lavinghouze R, Snyder K, Rieker P, & Ottoson J. (2013, Nov/Dec). Consideration of an applied model of 

public health program infrastructure. Journal of Public Health Management and Practice, 19(6).

8. Patton MQ. (2008). Utilization-focused evaluation (4th edition). Thousand Oaks, CA: Sage Publications. 

9. Centers for Disease Control and Prevention. (1999). Framework for program evaluation in public health. 

MMWR, 48(NoRR-11), 1-40. Retrieved from http://www.cdc.gov/mmwr/preview/mmwrhtml/rr4811a1.htm

10. Yarbrough DB, Shulha LM, Hopson RK, & Caruthers FA. (2011). The program evaluation standards: A 

guide for evaluators and evaluation users (3rd edition). Thousand Oaks, CA: Sage Publications.

11. Knowlton LW, Philips CC. The Logic Model Guidebook: Better Strategies for Great Results. Thousand 

Oaks (CA): Sage Publications, 2009. 

12. Centers for Disease Control and Prevention. (2008). Introduction to process evaluation in tobacco use 

prevention and control. Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease 

Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on 

Smoking and Health. Retrieved from http://www.cdc.gov/tobacco/publications/index.htm.

13. NAQC. (Tinkelman D.) . (2010a). Assessing the cost-effectiveness of quitline programs. Phoenix, 

AZ. Retrieved from http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/issue_papers/

assessingcosteffectivenessof.pdf.

14. MacDonald G, Starr G, Schooley M, Yee SL, Klimowski K, & Turner K. (2001). Introduction to program 

evaluation for comprehensive tobacco control programs. Atlanta, GA: Department of Health and Human 

Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and 

Health Promotion, Office on Smoking and Health.
58 | Conducting Quitline Evaluations

15. NAQC. (Cummins S.). (2009b). Measuring reach of quitline programs. Quality improvement initiative.

Phoenix, AZ. Retrieved from http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/docs/naqc_

issuepaper_measuringrea.pdf.

16. North American Quitline Consortium (NACQ). (2015). Calculating quit rates, 2015 update. (Betzner A, Lien 

B, Rainey J, et al.) Phoenix, AZ. Retrieved from http://c.ymcdn.com/sites/www.naquitline.org/resource/

resmgr/Issue_Papers/WhitePaper2015QRUpdate.pdf.

17. Centers for Disease Control and Prevention. (2005). Key outcome indicators for evaluating comprehensive

tobacco control programs. Atlanta, GA: U.S. Department of Health and Human Services, Centers for

Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion,

Office on Smoking and Health. Retrieved from http://www.cdc.gov/tobacco/tobacco_control_programs/

surveillance_evaluation/key_outcome/.

18. Centers for Disease Control and Prevention. (2004, September). Telephone quitlines: A resource for

development, implementation, and evaluation. Atlanta, GA: U.S. Department of Health and Human

Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and 

Health Promotion, Office on Smoking and Health, Final Edition. Retrieved from http://www.cdc.gov/

tobacco/quit_smoking/cessation/quitlines/pdfs/quitlines.pdf.

19. NAQC. (Reynolds P.). (2010b). Call center metrics: Best practices in performance measurement and

management to maximize quitline efficiency and quality. Quality Improvement Initiative. Phoenix,

AZ. Retrieved from http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/issue_papers/

callcentermetricspaperbestpr.pdf.

20. NAQC. (2009c). Review of published data on quitline quit rates. Unpublished report. Available upon

request. (naqc@naquitline.org).

21. NAQC. (2012). Results from the 2011 NAQC annual survey of quitlines. Prepared by: Saul J, Davis RA and

the NAQC Annual Survey Workgroup. Retrieved from http://www.naquitline.org/?page=2011Survey.

22. Bush T, Zbikowski SM, Mahoney L, Deprey M, Mowery P, Cerutti B. State quitlines and cessation patterns

among adults with selected chronic disease in 15 states, 2005-2008. Prev Chronic Dis. 2012;9:E163.

23. Heath C, & Heath D. (2007). Made to stick: Why some ideas survive and others die. New York, NY:

Random House.

24. Torres R, Preskill H, & Piontek ME. (2004). Evaluation strategies for communicating and reporting (2nd

edition). Thousand Oaks, CA: Sage Publications.

25. Centers for Disease Control and Prevention. (2007). Impact and value: Telling your program’s story.

Atlanta, GA: U.S. Department of Health and Human Services, Centers for Disease Control and Prevention,

National Center for Chronic Disease Prevention and Health Promotion, Division of Oral Health. Retrieved

from http://www.cdc.gov/OralHealth/publications/library/pdf/success_story_workbook.pdf

26. Lavinghouze R, Price AW, & Smith K-A. (2007). The program success story: A valuable tool for program

evaluation. Health Promotion Practice. 8(4), 323-331. 
59

Part II. Exercises, Worksheets, and Tools

The exercises, worksheets, and tools found in Part II of this workbook are to help reinforce 

the concepts discussed in Part I. These tools are only examples, and every evaluation will 

vary on the basis of program and stakeholder priorities and context.
60 | Conducting Quitline Evaluations 

1 2 3 4 5 6

STEP 1. TOOLS TO ENGAGE STAKEHOLDERS

1.1 Stakeholder Mapping Exercise

It is suggested that the program enlist the aid of an evaluation stakeholder workgroup

(ESW) of 8-10 members that represents the stakeholders who have the greatest stake 

or vested interest in the quitline evaluation (Centers for Disease Control, 2008). These 

stakeholders or primary intended users will serve in a consultative role on all phases of the 

evaluation. To begin the process of selecting those members who will best represent your 

primary intended users, it is suggested that you compile a list of all possible users with 

corresponding comments about their investment in the quitline evaluation and potential 

uses for evaluation results. 

1.1 Stakeholder Mapping 

Priority Person or Group Comments

Now, go back over your list of potential users of the quitline evaluation results and consider 

their level of priority on the list. For example, providing the information that funders or 

decision makers need may take a higher priority than other users, even though all users are 

still very important. You might rate stakeholders in terms of high, medium, or low, or you 

might rank order them from 1 to n. 
61

1 2 3 4 5 6

1.2 Evaluation Purpose Exercise

Identifying the end users and the evaluation stakeholder workgroup is as important as 

identifying the purpose of the quitline evaluation. These two aspects of the evaluation serve 

as a foundation for evaluation planning, focus, design, interpretation, and use of results. 

The purpose of an evaluation influences the identification of stakeholders for the evaluation, 

selection of specific evaluation questions, and the timing of quitline evaluation activities. 

It is critical that the quitline be transparent about intended purposes of the evaluation. If 

evaluation results will be used to determine whether a quitline component should be added, 

continued, or eliminated, then stakeholders should know this up front.

To determine the evaluation purpose, the evaluation team should work with those who are 

requesting the evaluation to identify the possible multiple purposes for the evaluation from 

multiple sources. The first task is to consider what groups are interested in an evaluation of 

the quitline. This might include the quitline service provider staff, health department staff, 

funders, state-level decision makers, and other stakeholders. The second task would be to 

align the specific group with what they are requesting to be evaluated. The third task would 

be to ascertain what the potential uses of the evaluation results will be by each group 

interested in the evaluation. And fourth, the team should develop a purpose statement 

relevant to each group and evaluation requested. 

1.2 Evaluation Purpose 

Group interested 

in an evaluation

What is to 

be evaluated

How will the results 

be used

Evaluation purpose 

statement
62 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Next, the team should consider each purpose statement for duplication and overlap. What 

statements could be combined? The final step in the process is to merge the statements 

into one overall purpose statement.

Evaluation Purpose Statement:

1.3 Stakeholder Inclusion and Communication Plan Exercise

It is important to explore agendas at the beginning of the quitline evaluation process and 

come to a shared understanding of roles and responsibilities, as well as the purposes 

of the evaluation. Some stakeholders will be represented on the evaluation stakeholder 

workgroup (ESW), and some will not. It is important to include a clear communication plan 

in the evaluation plan to meaningfully engage all appropriate quitline stakeholders and 

increase participation and buy-in for the evaluation, as well as use of final results.

For each stakeholder relevant to the evaluation, list their appropriate role and how and 

when they might be engaged in the evaluation. Consider their expertise, level of interest, 

and availability when developing the communication plan. If there are specific deadlines 

for information, such as a funding opportunity or quitline contract rebidding process, note 

those as well. Additional columns could be added for comments.
63

1 2 3 4 5 6

1.3 Stakeholder Inclusion and Communication Plan 

Evaluation 

stakeholder

Role related to 

the evaluation

Mode of 

communication

Timing of 

communication

A note on roles: Stakeholders need not be a member of the evaluation stakeholder 

workgroup to have a role related to the evaluation. Given a stakeholder’s specific expertise, 

interest, availability, or intended use of the evaluation results, they may be involved in 

part or all of the evaluation without being a specific member of the evaluation stakeholder 

workgroup. Roles might include, but are not limited to the following:

 Development of the quitline evaluation plan.

 Feedback on focusing the evaluation or selecting evaluation questions.

 Needing information about specific quitline evaluation activities or progress of the 

evaluation.

 Facilitating implementation of specific aspects of the quitline evaluation.

 Included in interpretation meetings.

 Disseminating and promoting use of quitline evaluation results.
64 | Conducting Quitline Evaluations 

1 2 3 4 5 6

1.4 Stakeholder Information Needs Exercise

Although focusing the evaluation occurs in Step 3, the groundwork begins with the 

identification of quitline stakeholders relevant to the evaluation or the primary intended 

users. Membership in the evaluation stakeholder workgroup is designed to reflect priority 

information needs, as well as those that will use the evaluation information. However, it is 

not always possible to include some groups who need information, and it is certainly not 

possible to include representation from every group that would benefit from evaluation 

results. This should not prevent evaluation staff and the evaluation stakeholder workgroup 

from considering all points of view and needs for information when considering how best 

to focus the quitline evaluation. Therefore, determining stakeholder information needs is 

both useful for considering membership in the evaluation stakeholder group (Step 1) and 

focusing the evaluation (Step 3).

From the list of primary intended users (i.e., those who have a stake in the quitline 

evaluation results), identify what information each stakeholder will use. 

1.4 Stakeholder Information Needs 

Primary Intended User (Stakeholder) Evaluation Information Needed

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.
65

1 2 3 4 5 6

STEP 2. TOOLS TO DESCRIBE THE PROGRAM

2.1 Developing a Quitline Logic Model Exercise

As a logic model for your quitline program is developed, carefully consider the purpose of 

the quitline evaluation (see 1.2 Evaluation Purpose Exercise). It can be helpful to start with 

the intended outcomes of the quitline, such as increased calls or referrals, and increased 

quit attempts. Or it may be easier to start with the inputs (resources) available to the 

quitline, such as funding amounts or referral networks, as well as quitline activities. In either 

case, the goal will be to identify as many relevant elements of the context in which the 

quitline exists that will contribute to or detract from its intended outcomes. 

Inputs: Infrastructure and resources necessary for quitline implementation.

Activities: The actual activities conducted by the quitline to achieve its goals.

Outputs: Direct products obtained as a result of program activities (e.g., counseling 

and medication provision).

Outcomes: (Short-term; intermediate; long-term; distal) The changes, impacts, or 

results of quitline implementation (activities and outputs). 

Environmental Context: Larger cessation program and policy environment in which 

the quitline is operating.
66 | Conducting Quitline Evaluations 

1 2 3 4 5 6

First, identify the elements in the logic model.

Inputs Activities Outputs

Short-term 

Outcomes

Intermediate 

Outcomes

Long-term 

Outcomes

1. 1. 1. 1. 1. 1.

2. 2. 2. 2. 2. 2.

3. 3. 3. 3. 3. 3.

4. 4. 4. 4. 4. 4.

5. 5. 5. 5. 5. 5.

6. 6. 6. 6. 6. 6.

Next, go through the items listed above, and start identifying how the inputs, activities, and 

outputs might link to outcomes (both intended and unintended). Your ESW can help identify 

some of the links. On the basis of the identified links, place the logic model elements into 

a logic model diagram. This will form the starting point for a shared understanding of the 

quitline program and its intended outcomes. An overview of the environmental context can 

also be highlighted on your logic model (see next).
67

1 2 3 4 5 6

2.2 Describe the Quitline’s Environmental Context Exercise

Developing a clear description of the quitline’s environmental context can be a critical 

component when determining the factors that can contribute to, and detract from the 

quitline’s goals. In addition, the environmental context can help identify what evaluation 

questions are both feasible to answer and most important to answer. 

Start with the quitline’s inputs listed above. For each input identified, transfer them to 

the tables below. Consider environmental factors that might influence those inputs. The 

same environmental factor may influence more than one input. Repeat the process for 

quitline activities, outputs, and outcomes (short-term, intermediate, and long-term). When 

considering the environmental context, think back to the key stakeholder identification 

exercise in Step 1. Which individuals and groups are most likely to take an interest in 

the quitline? Which ones should be more engaged? Which are strong supporters, weak 

supporters, or have negative views of your quitline? In addition, think about the following 

questions: 

 What factors are likely to influence your quitline’s available funding capacity?

 What state or national activities, events, or policies might influence your quitline’s 

activities or outcomes?

 How might the tobacco industry’s promotional efforts affect your quitline’s work?

2.2 Environmental Context Exercise

Inputs 1. 2. 3. 4. 5. 6.

Influential 

Environmental 

Factors

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.
68 | Conducting Quitline Evaluations

1 2 3 4 5 6

Activities 1. 2. 3. 4. 5. 6.

Influential 

Environmental 

Factors

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

Outputs 1. 2. 3. 4. 5. 6.

Influential 

Environmental 

Factors

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.
69

1 2 3 4 5 6

Short-term 

Outcomes

1. 2. 3. 4. 5. 6.

Influential 

Environmental 

Factors

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

Intermediate 

Outcomes

1. 2. 3. 4. 5. 6.

Influential 

Environmental 

Factors

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.
70 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Long-term 

Outcomes

1. 2. 3. 4. 5. 6.

Influential 

Environmental 

Factors

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.

1.

2.

3.
71

1 2 3 4 5 6

STEP 3. TOOLS TO FOCUS THE DESIGN

Illustrating Decisions used to Focus the Evaluation 

The following scenarios illustrate how a quitline’s stakeholders, the evaluation purpose, and 

the quitline components and context interact to help focus a quitline’s evaluation. The types 

of evaluation questions that can and should be asked will vary on the basis of the length of 

time a quitline has been in operation, the degree of engagement by stakeholders, and the 

amount of resources (human and financial) to which a quitline has access, as well as other 

factors. It may be helpful to share these examples with stakeholders, if necessary, to start 

the conversation, or to determine if these scenarios apply in your context. Try to discuss 

what factors apply in your context.

Scenario 1: A New Quitline Service

When technological services (e.g., texting, web services) are added to a quitline, it may not 

be appropriate to conduct outcome evaluation on quit rates or cost-effectiveness during 

the first few years. Instead, it may be appropriate to focus on process evaluation questions, 

such as the number of tobacco users served by each method, how participants who only 

use technology services differ from those who also use telephone counseling, and the 

relationship between promotional efforts and reach. Following some basic monitoring and 

process evaluation and related program improvement, a program should be collecting 

follow-up data to enable outcome evaluation during subsequent years. It may also want 

to ask primarily quality-related questions related to whether the quitline program is being 

delivered as intended, and whether participants are satisfied with their experience.

Scenario 2: Determined Stakeholders

A quitline receives funding by using a state legislative appropriations process. One 

state legislator continually requests information about how the state quitline compares 

to quitlines in neighboring states. By inviting her to serve on the evaluation stakeholder 

workgroup, her larger questions about quitline performance can be addressed through 

an educational process related to quitline context and the feasibility of producing certain 

comparative metrics. 
72 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Scenario 3: Limited Funding

A cut in a quitline’s budget can be cause for reconsideration of the relative importance 

of various evaluation questions. If a quitline has not changed the constellation of service 

offerings since the last outcome evaluation, it may not be necessary to collect quit 

outcomes during the period of budget shortfall, and instead, focus on questions related to 

demonstrating high reach across multiple demographic groups.

Once you have identified the various environmental factors that can both enhance and 

detract from achievement of your quitline’s goals, it can become easier to determine which 

evaluation questions are most important to answer.

3.1 Focus the Evaluation Exercise

The amount of information you can gather concerning your quitline is potentially limitless. 

Evaluations, however, are always limited by the number of questions that can be realistically 

asked, the methods that can actually be employed, the feasibility of data collection, and 

the available resources. These are the issues at the heart of Step 3 in the CDC framework: 

focusing the evaluation. The scope and depth of any program evaluation is dependent on 

program and stakeholder priorities; available resources including financial resources; staff 

and contractor availability; and the amount of time committed to the evaluation. The quitline 

staff should work together with the ESW to determine the priority of the questions, the 

feasibility of answering the questions, and how the results will be used before designing the 

evaluation plan. 

In this exercise, you will need to consider all the information from previous exercises in Step 

1 through Step 2, the logic model, and your stakeholders’ vested interest in the evaluation. 

From the Stakeholder Mapping exercise (1.1), list the stakeholders categorized as high 

priority for information needs: 
73

1 2 3 4 5 6

Stakeholders in High Priority Category of Importance for Information Needs 

(from exercise 1.1)

From the Evaluation Purpose Identification exercise, indicate your overall evaluation 

purpose statement:

Evaluation Purpose Statement (from exercise 1.2):

Consider each stakeholder’s evaluation needs and the information most appropriate for 

answering that question (from exercise 1.4):

Primary Intended User Evaluation Information Needed

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.
74 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Given the overall purpose statement, what questions from the high-priority stakeholder 

group are viable for the current evaluation effort?

Evaluation Purpose Statement:

High Priority Stakeholders Evaluation Question
75

1 2 3 4 5 6

Next, the team should consider issues of feasibility related to those evaluation questions 

that are viable options given the evaluation purpose.

Evaluation 

Question

Methods that might 

be used to answer 

the question

Assumptions or 

conditions for this 

method to be viable

Resources needed 

to implement this 

method

Limitations of 

this method
76 | Conducting Quitline Evaluations 

1 2 3 4 5 6

No chart, grid, or exercise can fully answer the question of how best to focus the 

evaluation. However, the above information should facilitate informed discussions and 

can help avoid evaluation activities that are misaligned with the quitline purpose or 

activities, are underfunded, or not of the highest priority for information needs. Additional 

considerations that might help prioritize your evaluation questions include

 The questions most important to you and your key stakeholders (the “must answer” 

questions). 

 Questions that provide results that can be used (e.g., for improvement).

 Questions you can answer fully with available or easy to gather data.

 Questions within your resources to answer.

The evaluation questions for the current evaluation are:
77

1 2 3 4 5 6

STEP 4. TOOLS TO GATHER CREDIBLE EVIDENCE

Matching Measures to Evaluation Questions

In Step 3, we discussed the difference between types of evaluation questions, such as 

process and outcome questions. For each area of interest for quitlines, we provided sample 

evaluation questions in Tables 1 and 2. For Step 4, you will need to link each evaluation 

question to a method and related measures that can help answer it. In Tables 6 and 7 

below, a sample of quitline evaluation questions is linked to a measure (or indicator if a 

measure needs more specification by the program) that can help answer it. Please note that 

these are examples only, and if your quitline’s evaluation questions are different than the 

ones listed below, the measures needed to answer them will also be different. 

You may wish to consider these examples with the ESW if they relate to areas of interest 

in the evaluation plan. It’s important to keep use in mind as you go through this exercise 

that data should not be collected for their own sake, but because they relay some useful 

information about the program that will be used for a specific purpose that has been laid 

out by the ESW.
78 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Table 6. Sample Process Evaluation Questions and Sample Measures

Process Evaluation 

Areas of Interest

Sample Process 

Evaluation Questions 

Sample Process 

Measures or Indicators

Demand for quitline 

services

What is the call volume for the 

quitline on a weekly, monthly, or 

annual basis? 

How does it change over time?

The total number of calls received by the quitline. 

The total number of (unduplicated) tobacco users 

calling for help for themselves.

The total number of (unduplicated) proxy callers. 

Quitline Promotion Is promotion of the quitline being 

done according to plan and meeting 

set targets? 

How does call volume relate to 

quitline promotional efforts?

The total number of calls related to paid or earned 

promotional efforts or outreach activities.

The number of tobacco users reporting hearing about 

the quitline from various sources.

The content, placement, and coverage of paid 

advertising.

The content, placement, and timing of earned media.

The number and type of outreach efforts with provider 

groups or other potential referral sources.

Quitline Use How many tobacco users receive 

services (counseling or medications) 

from the quitline annually? 

What are the characteristics of the 

callers?

How does the population of quitline 

participants compare with the 

population of tobacco users in the 

state?

Are callers representative of the 

population we are trying to reach?

What is the quitline’s reach?

Education level, gender, age, and race/ethnicity.

 Chronic disease and mental illness status (optional 

MDS questions).

Promotional reach, “registration reach,” and treatment 

reach for the quitline.
79

1 2 3 4 5 6

Process Evaluation 

Areas of Interest

Sample Process 

Evaluation Questions 

Sample Process 

Measures or Indicators

Quitline referral 

networks

How many referrals are received by 

the quitline? 

How many referral sources (e.g., 

individual health care providers, 

clinics, health systems) are there for 

the quitline? 

What recruitment strategies/outreach 

activities are being used to add new 

referral sources? 

How do referral sources correlate 

with outreach activities? 

What proportion of people who were 

referred is successfully contacted by 

quitline?

What proportion of people who were 

referred is successfully enrolled by 

the quitline?

The total number of referrals received by the quitline. 

The source of referrals.

The number and type of outreach activities conducted.

The number of referrals successfully reached by the 

quitline.

The number of referrals registering for quitline 

services (completing an intake survey).

The number of referrals receiving services (counseling 

and/or medications).

Quality Assurance How well are provided services 

meeting quality standards? 

Is the counseling being provided by 

using evidence-based methods? 

Are tobacco users receiving 

proactive calls or responding to 

voicemails within the time frame 

specified by the quitline contract? 

Are referrals being processed in a 

timely fashion? 

Are reports accurate and complete?

The average time from initial contact with the quitline 

to start of counseling for those requesting counseling.

The average number of counseling sessions 

completed per registration/quit attempt.

The average number of minutes of counseling per 

registration/quit attempt.
80 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Process Evaluation 

Areas of Interest

Sample Process 

Evaluation Questions 

Sample Process 

Measures or Indicators

Participant Satisfaction What are participants’ satisfaction 

levels with the quitline? Materials 

provided? Messaging or other 

communication types?

What quitline factors increase or 

decrease participant satisfaction? 

Quitline satisfaction from quitline follow-up surveys 

(see MDS follow-up survey questions for standard 

optional question wording).

Investment in Services What is the annual investment 

in quitline services, medications, 

promotions, and outreach?

Quitline budget for services and medications divided 

by the total number of adults in the state.

Quitline budget for services and medications divided 

by the total number of adult smokers in the state.

Quitline budget for promotions and outreach divided 

by the total number of adults in the state.

Quitline budget for promotions and outreach divided 

by the number of quitline callers who received 

counseling or medication in the state .

Quitline Staffing Is the quitline staffed sufficiently to 

respond to all incoming calls and 

referrals? 

Are the hours of operation adequate 

to meet the demand for services?

Did demand for services exceed 

capability?

Are quitline coaches/counselors 

trained appropriately (e.g., language, 

cultural competency)? 

What is the level of supervision for 

coaches/counselors? What form does 

supervision take?

The type of training completed by quitline counseling/

coaching staff, both content and duration of initial 

training as well as ongoing continuing education.

The number of coaches/counselors staffing the 

quitline.

Supervision type and frequency for coaches/

counselors (e.g., coaching calls recorded and 

reviewed).

Quitline Efficiency How much did the quitline spend per 

enrollee?

How are the quitline’s spending per 

smoker amounts related to reach?

Did we have the right strategies (e.g., 

promotion, treatment mix) to reach 

or target our desired population 

efficiently?

Spending per smoker on promotions and outreach 

compared with promotional reach.

Spending per smoker on services and medications 

compared with treatment reach.
81

1 2 3 4 5 6

Table 7. Sample Outcome Evaluation Questions and Sample Measures

Outcome Areas 

of Interest 

Sample Outcome 

Evaluation Questions 

Sample Outcome 

Measures or Indicators

Quitline Awareness What proportion of the state’s 

adult population (tobacco user and 

nontobacco user) is aware of the 

quitline?

Quitline Awareness as measured by the Behavioral 

Risk Factor Surveillance System (BRFSS) module 

question.

Changes in Motivation 

to Quit/Confidence in 

Quitting

How are participants’ motivation to quit 

or confidence in quitting changing as a 

result of quitline counseling (especially 

among those not ready to make a quit 

attempt upon registration)?

Motivation to quit as measured at intake. 

Motivation to quit as measured at follow-up (for 

those not quit at follow-up).

Motivation to stay quit as measured at follow-up (for 

those who are quit at follow-up).

Confidence in quitting as measured at intake.

Confidence in quitting as measured at follow-up (for 

those not quit at follow-up).

Confidence in staying quit as measured at follow-up 

(for those who are quit at follow-up).

Quit Attempts How many/what proportion of tobacco 

users are making at least one 24-

hour quit attempt since registering for 

quitline services?

“Since you first called the quitline on (Date of first 

contact), 7 months ago, did you stop using tobacco 

for 24 hours or longer because you were trying to 

quit?” (MDS follow-up item)

Longer-term quit 

success

How many quitline participants report 

no use of any tobacco product for the 

past 7 (or 30) days at 7-month followup (point prevalence abstinence)?3

“Have you used any tobacco, even a puff or a pinch, 

in the last 7/30 days?” as measured at seven-month 

follow-up.

“Have you used any tobacco, even a puff or a pinch, 

in the last 6 months?” as measured at 7-month 

follow-up.

Reduction in 

prevalence

How has tobacco use prevalence in the 

state changed over time?

BRFSS smoking prevalence rate (see OSHData for 

results) trends.

BRFSS tobacco use prevalence rate trends for 

cigarettes and smokeless tobacco. 

Adult Tobacco Survey (ATS) prevalence rate trends.

TUS-CPS national and state prevalence rate trends.
82 | Conducting Quitline Evaluations

1 2 3 4 5 6

Outcome Areas 

of Interest 

Sample Outcome 

Evaluation Questions 

Sample Outcome 

Measures or Indicators

Economic Evaluations

Cost effectiveness4

Cost-benefit analysis

Cost-utility analysis

Return on investment

What is your quitline’s cost per quit, 

including the cost of promoting the 

quitline? 

How does cost per quit compare with 

other programs your department, 

agency, or organization provides? How 

does it compare with other quitlines?

How much does each dollar spent on 

quitline services save the state (or other 

payers such as health plans) in terms of 

prevented medical care costs? 

How many Life Years Saved can be 

attributed to the quitline? What is the 

cost for each Quality Adjusted Life Year 

saved?

Quitline expenditure on services and promotions.

“Have you used any tobacco, even a puff or a pinch, 

in the last 7/30 days?” as measured at 7-month 

follow-up.

Smoking attributable health care costs (SAMMEC).

LYS and QALY.

Attribution of 

outcomes to the 

quitline

Is there a clear link between outcomes 

related to your quitline, as opposed 

to other events occurring at the same 

time?

What programmatic or policy changes 

have occurred in your state or various 

local jurisdictions during the evaluation 

period in question? 

Has the tobacco tax rate increased? 

Have smokefree policies gone into 

effect? 

Have media campaigns promoting the 

national portal number 1-800-QUITNOW been in the field?

Are there other media campaigns in the 

field?

What about Medicaid expansion 

coverage?

Date of policy change announcement and initiation.

Date of tobacco tax increase announcement and 

implementation.

Media strategy/buy for quitline promotions.

Partner organization activity.

For guidance on calculating quit rates for quitlines, see the NAQC Issue Paper “Calculating Quit Rates, 2015 Update” at http://c.ymcdn.

com/sites/www.naquitline.org/resource/resmgr/Issue_Papers/WhitePaper2015QRUpdate.pdf.

4 For more on calculating cost effectiveness, see the NAQC Issue Paper “Assessing the Cost Effectiveness of Quitline Programs” at http://c.

ymcdn.com/sites/www.naquitline.org/resource/resmgr/issue_papers/assessingcosteffectivenessof.pdf.

3
83

1 2 3 4 5 6

4.1 Evaluation Methods Checklist Exercise

Here is a checklist of issues based on the evaluation standards that can help the ESW 

recommend the most appropriate data collection methods: 

UTILITY

Questions Notes

Purpose and use of methods: Do you seek a 

point-in-time determination of a behavior (quit 

status), or to examine the range and variety of 

experiences (satisfaction survey or focus groups), 

or to tell an in-depth story (case study of the 

integration of the quitline into a health system)? 

Users of evaluation data: Will some methods make 

the data more credible with skeptics or key users? 

FEASIBILITY

Questions Notes

Resources available: Which methods can you 

afford? 

Time: How long until the results are needed? 

Frequency: How often do you need the data?

Your background and capacity: Are you trained in 

the methodology you want to use, or will you need 

help from an outside consultant? Do you have 

the internal capacity to conduct the evaluation 

yourself, or is it beyond the scope of what your 

staff can handle?
84 | Conducting Quitline Evaluations 

1 2 3 4 5 6

PROPRIETY

Questions Notes

Characteristics of the respondents: Will issues 

such as cultural background or language make 

some evaluation questions more appropriate 

than others? For example, if your quitline serves 

a large number of Native Americans, it will be 

important to tailor your methods to acknowledge 

the importance of sacred or ceremonial uses of 

tobacco.

Degree of intrusion to program/participants: Will 

the data collection method disrupt the program 

or be seen as intrusive by participants? This is 

particularly relevant with respect to quitlines, 

where asking a long series of intake questions 

before beginning cessation counseling can be 

perceived as being irrelevant or irritating.

Other ethical issues: To what extent can you 

justify expending resources for data collection 

and evaluation efforts at the expense of providing 

services to a larger number of tobacco users? 

ACCURACY

Questions Notes

Respondent bias: Are the data collection 

methods likely to influence the answers given 

by respondents? For example, follow-up surveys 

should not be conducted by the same counselors 

or coaches that provided the quitline intervention.

Respondent memory: Are the questions you are 

asking of respondents too complex that there may 

be errors introduced because of their inability to 

remember accurately? For example, asking about 

how many patches were used on each day for 

the first 3 weeks of treatment may not produce 

reliable results. Techniques to increase accuracy 

of reporting could be researched.

Self-report: Although it is a generally accepted 

practice to ask tobacco users to self-report on quit 

status for follow-up surveys, it should be noted 

that respondents tend to be more likely to have 

quit than non-respondents. 
85

1 2 3 4 5 6

4.2 Evaluation Methods Grid Exercise

One tool that is particularly useful in your quitline evaluation is an evaluation methods 

grid. This tool is helpful to align evaluation questions with indicators or performance 

measures and data sources and roles and responsibilities, and it can facilitate advocating 

for resources for the evaluation. In addition, this tool facilitates a shared understanding of 

the overall evaluation plan with stakeholders. This tool can take many forms and should be 

adapted to fit your specific evaluation and context; examples of the forms it takes follow.

Evaluation Methods Grid Example A

Evaluation 

Question

Indicator/ 

Performance 

Measure Method Data Source Frequency Responsibility

What is the 

impact of 

offering free 

NRT through the 

quitline?

Number of 

calls, number 

of tobacco 

users receiving 

services, sevenmonth quit rate, 

cost per quit

Cross-sectional 

observational 

pre-post design

Quitline intake 

data, quitline 

administrative and 

utilization data, 

seven-month followup survey data, 

quitline budget data

Before and and 

after introduction 

of NRT

Quitline service 

provider (intake 

administrative 

data), quitline 

evaluator (followup survey and 

analysis)

Evaluation Methods Grid Example B

Evaluation Question

Indicators/ 

Performance Measure

Potential Data Source 

(Existing/New) Comments

How effective are quitline 

promotional activities at 

increasing demand for 

quitline services?

Description of promotional 

activities (timing, duration, 

content); number of tobacco 

users reaching the quitline 

and their demographics; 

referral source (“how heard 

about the quitline”)

TRP and GRP data sources

Registration/intake data

Call volume data
86 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Choose the grid that is most appropriate for your program, and complete it given your 

chosen evaluation questions from Step 3.

Evaluation 

Question

Indicator/ 

Performance 

Measure Method Data Source Frequency Responsibility
87

1 2 3 4 5 6

Evaluation Question

Indicators/ 

Performance Measure

Potential Data Source 

(Existing/New) Comments

Evaluation 

Question Timeline Methods Data Sources

Instruments 

Needed

Staff/Persons 

Responsible

Evaluation 

Question Methods

Instruments 

Needed Timeline

Respondents/ 

Population/ Sample Responsibility

Evaluation 

Question Indicators

Data Collection 

Sources

Data Collection 

Methods Timeline

Data Analysis 

Plan
88 | Conducting Quitline Evaluations 

1 2 3 4 5 6

4.3 Evaluation Budget Exercise

For this exercise, you will need the Evaluation Methods Grid you completed earlier in Step 

4. For this exercise, we have used one grid as an example, but you should use the one you 

have chosen as most appropriate for your program.

The team should now consider roles and responsibilities, what services might be in-kind, 

and what activities will cost additional money. Will you need to pay for additional questions 

on existing surveys, or can you use items that already exist? Are there existing data 

sources, or will you need to create new ones? Do not forget items such as copying costs 

for surveys or web services or technology needed in the field, such as recorders or mobile 

data collection devices.

Evaluation 

Question

Indicator/ 

Performance 

Measure Method

Data 

Source Frequency Responsibility

Cost 

Considerations

During this exercise, you may find it necessary to revisit Step 3 or earlier portions of Step 

4. Often the budget available doesn’t match the evaluation desired. Either the evaluation 

scope will need to be reduced or additional resources obtained. It is better to thoroughly 

consider this now before implementation begins than have to change course midimplementation cycle.
89

1 2 3 4 5 6

STEP 5. TOOLS FOR JUSTIFYING CONCLUSIONS

5.1 Stakeholder Interpretation Meeting Exercise

Justifying conclusions includes analyzing the information you collect, interpreting what the 

data mean, and drawing conclusions based on the data. This step is needed to turn the 

data collected into meaningful, useful, and accessible information. This is often the step in 

which programs incorrectly assume they no longer need the stakeholder workgroup and 

that this step is better left to the “experts.” However, including your stakeholder group in 

this step is directly tied to the previous discussion on credibility and acceptance of data 

and conclusions. 

Moreover, it is critical that plans include time for interpretation and review from 

stakeholders, including critics, to increase transparency and validity of the process and 

conclusions. The emphasis here is on justifying conclusions, not just analyzing data. 

This step deserves due diligence in the planning process. The propriety standard plays 

a role in guiding the evaluator’s decisions in how to analyze and interpret data to ensure 

that all stakeholder values are respected in the process of drawing conclusions (Program 

Evaluation Standards, 1994). This may include one or more stakeholder interpretation 

meetings to review interim data and further refine conclusions. A note of caution, as a 

stakeholder driven process, there is often pressure to reach beyond the evidence when 

drawing conclusions. It is the responsibility of the evaluator and the evaluation workgroup 

to ensure that conclusions are drawn directly from the evidence. 

A variety of activities can be included in your evaluation plan to solicit stakeholder input 

and facilitate interpretation of evaluation data. An example is provided below:

Interpretation and Review Activities Timeline

Initial quitline evaluation reports As needed to produce final reports to meet data needs of 

stakeholders

Check-in with quitline service provider As appropriate during analysis phase

Stakeholder interpretation meeting Immediately following preparation of preliminary results

Stakeholder review of draft final report Within 3 months following the stakeholder interpretation 

meeting

Clearance and review process of final report Within 2 months following stakeholder review of draft final 

report
90 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Complete an outline of proposed activities appropriate to the evaluation project to include 

opportunities for stakeholder interpretation and feedback: 

Interpretation and Review Activities Timeline

It is important to consider the time it takes to solicit and incorporate stakeholder feedback 

in the evaluation project timeline. Check that the budget and timeline created earlier 

ensures adequate time and funding for the stakeholder inclusion process.

To ensure the stakeholder interpretation meeting is a success, plan for steps to help things 

run smoothly. Time for these activities needs to be included in the evaluation timeline.

 Send the initial invitation at least 2 months in advance so that stakeholders can plan for 

the meeting. Remind stakeholders of the overall evaluation purpose and questions.

 Send the preliminary report or PowerPoint presentation with enough lead time before 

the meeting to allow stakeholders time to review. It is important to remind stakeholders 

that results are draft and should not be shared outside of the review group.

 Send reminders about the meeting 1 or 2 weeks before the date. Identify any preexisting documentation that may be useful for understanding context.

 Plan for appropriate technology (and backup) needed, such as recorders, laptop, 

screen, flipcharts, etc.

 If feasible, use a professional meeting facilitator.

A checklist to facilitate the development of a formal stakeholder interpretation meeting can 

be found at: https://www.wmich.edu/evaluation/checklists.
91

1 2 3 4 5 6

STEP 6. TOOLS TO ENSURE USE AND SHARE LESSONS LEARNED

6.1 Reporting Checklist

Below is a checklist of items that may be worth discussing during the evaluation planning 

stage to ensure adequate time and resources are devoted to the implementation and 

reporting process.

Tools and Templates: Checklist for Ensuring Effective Evaluation Reports**

 Provide interim and final reports to intended users in time for use.

 Tailor the report content, format, and style for the audiences by involving audience 

members.

 Include an executive summary.

 Summarize the description of the stakeholders and how they were engaged.

 Describe essential features of the program (e.g., in appendices).

 Explain the focus of the evaluation and its limitations.

 Include an adequate summary of the evaluation plan and procedures.

 Provide all necessary technical information (e.g., in appendices).

 Specify the standards and criteria for evaluative judgments.

 Explain the evaluative judgments and how they are supported by the evidence.

 List both strengths and weaknesses of the evaluation.

 Discuss recommendations for action with their advantages, disadvantages, and 

resource implications.

 Ensure privacy protections for program clients and other stakeholders.

 Anticipate how people or organizations might be affected by the findings.

 Present minority opinions or rejoinders, where necessary.

 Verify that the report is accurate and unbiased.

 Organize the report logically, and include appropriate details.

 Remove technical jargon.

 Use examples, illustrations, graphics, and stories.

** Adapted from Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L. (1997). Program evaluation: Alternative 

approaches and practical guidelines (2nd edition). New York, NY: Addison, Wesley Logman, Inc. and 

presented in Comprehensive Cancer Control Branch Program Evaluation Toolkit, Centers for Disease Control 

and Prevention, June 2010.

Also visit The Evaluation Center at Western Michigan University online for a free evaluation 

report checklist:

https://www.wmich.edu/evaluation/checklists.
92 | Conducting Quitline Evaluations 

6.2 Communicating Results Exercise

Evaluation results may not reach the intended audience just because they are published; 

an intentional communication and dissemination plan should be included in your evaluation 

plan. As previously stated, the planning stage is the time for the program to begin to 

think about the best way to share the lessons you will learn from the evaluation. The 

communication-dissemination phase of the evaluation is a two-way process designed 

to support use of the evaluation results for program improvement and decision making. 

To achieve this outcome, a program must translate evaluation results into practical 

applications and must systematically distribute the information or knowledge through a 

variety of audience-specific strategies. 

Communicating evaluation results involves sharing information in ways that make it 

understandable and useful to stakeholders. Successful communication is key to evaluation 

results being used. You can do this by using a variety of communication formats and 

channels. A communication format is the actual layout of the communication you will use, 

such as reports, brochures, one-page descriptions, newsletters, executive summaries, 

slides, and fact sheets. A communication channel is the route of communication you 

will use, such as oral presentations, videos, e-mails, webcasts, news releases, and web 

or phone conferences. Both the formats and channels should consider the needs of 

different audiences, the type of information you wish to provide, and the purpose of the 

communication. 

When developing your communication or dissemination strategy, carefully consider the 

following:

 With which target audiences or groups of stakeholders will you share findings?

 What formats and channels will you use to share findings?

 When and how often do you plan to share findings?

 Who is responsible for carrying out dissemination strategies?

You can plan and track the communication process by using tools similar to the following 

examples.
93

1 2 3 4 5 6

6.2 Communicating Results 

What do you want to 

communicate?

With whom do you 

want to communicate?

How do you want to communicate?

Format(s) Channel(s)

** This tool was adapted from DASH’s Communication Matrix in “Using Evaluation to Improve Programs: 

Strategic Planning” in the Strategic planning kit for school health programs. Available at: http://www.cdc.

gov/healthyyouth/evaluation/strategicplan.htm Last accessed 5-19-2014.

This tool can help you track communications with your various audiences, including the 

communication format(s) (the layout of the communication, such as newsletters) and 

the communication channel(s) (the route of communication, such as oral presentations), 

audience feedback on the communication message, and next steps you need to take in 

response. 
94 | Conducting Quitline Evaluations 

1 2 3 4 5 6

Here are two examples of a communication tracking charts. Select the best items for your 

context.

Communication Date Audience(s)

Communication 

Format(s)

Communication 

Channel(s)

Audience 

Feedback and 

Next Steps

Priority Audience 

Objectives for the 

Communication Tools Time Table
95

1 2 3 4 5 6

6.3 Success Story Examples and Tools

Although not your main method of presenting data, success stories can contribute to the 

communication of your evaluation results. Kansas worked with OSH to put together a 

success story for their quitline campaign (http://www.cdc.gov/tobacco/tobacco_control_

programs/surveillance_evaluation/success-stories/pdfs/kansas.pdf). Their story highlighted 

focus group work to identify effective messaging, and the number of additional residents 

that were reached with services because of the campaign. 

Success stories can put a “face” on the data, and can document how the program 

changed participants’ lives (CDC, 2007). The key to a good success story is identifying your 

audience and concentrating on their information needs. Going back to Exercise 1.4 may 

help with identifying stakeholders’ information needs that are unmet by the existing items 

in your communication plans. For example, legislators and other decision makers often 

want to see concrete examples of how government funds have helped their constituents. 

Credible evidence in this case may mean interviewing one or two individuals affected by the 

quitline and using quotes from them. Stakeholders may be useful in this process to recruit 

and identify participants. Discussing potential topics with the ESW may generate some 

ideas about dissemination you had not identified previously.

More information and success story data collection tools can be found in the CDC (2007) 

guide. In addition, CDC has an online success story application (http://www.cdc.gov/

NCCDPHP/dch/success-stories/), which can help states or communities low on time and 

other resources. Features of the online tool include:

 Easy-to-use tool to develop your story. 

 Guidance on how to write a success story. 

 A downloadable worksheet to begin the prewriting process. 

 Three professionally developed templates. 

 A free photo library. 

 A Success Stories Library that you can search and share. 
96 | Conducting Quitline Evaluations 

Resource Section

*Resources are listed for the convenience of the user and do not constitute endorsement by the U.S. Government.

QUITLINE RESOURCES

 “Call Center Metrics: Best Practices in Performance Measurement and Management 

to Maximize Quitline Efficiency and Quality” (2010).

• http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/issue_papers/

callcentermetricspaperbestpr.pdf. 

• Many metrics for process or implementation evaluation can be found in this 

NAQC Issue Paper. This paper defines the most critical operational, service 

performance and efficiency-related call center metrics for establishing 

and maintaining quitline quality. The paper, authored by Penny Reynolds 

of The Call Center School (http://www.thecallcenterschool.com/), makes 

recommendations on important metrics to measure (including why they are 

important for funders, service providers and callers); recommendations on how 

to measure and report these metrics; and describes how these reports may be 

used by both the service providers and funders to improve quality.

 Centers for Disease Control and Prevention. (2004, Sep). Telephone quitlines: A 

resource for development, implementation, and evaluation (final edition). Atlanta, GA: 

U.S. Department of Health and Human Services, Centers for Disease Control and 

Prevention, National Center for Chronic Disease Prevention and Health Promotion, 

Office on Smoking and Health.

• http://www.cdc.gov/tobacco/quit_smoking/cessation/quitlines/pdfs/quitlines.pdf

 Guide to Community Preventive Services. (2013). Reducing tobacco use and 

secondhand smoke exposure: quitline interventions. Retrieved from 

www.thecommunityguide.org/tobacco/quitlines.html.

§ Minimal Data Set for Quitlines (MDS) (see http://www.naquitline.org/mds for more

information, including intake and follow-up survey instruments). 

• Demographic and tobacco use characteristics of quitline participants can be 

collected through implementation of the MDS.
97

• The MDS offers a standard approach to evaluating tobacco cessation quitlines.

Having a standard set of variables is valuable for:

» Establishing commonly defined performance indicators to assist in

assessing quitline performance, improving the quality of quitlines,

identifying knowledge gaps and designing new strategies to fill the

gaps.

» Providing a common language allowing for consistent communication

with others within and external to the quitline community.

» Identifying quitline performance benchmarks that can be used to

determine effective, cost-efficient tobacco cessation interventions.

» Testing and assessing new treatment techniques across large diverse

populations not possible by a single quitline.

» Collecting consistent data and allowing aggregation of data across

quitlines for improved analyses of a variety of variables relevant to the

success of quitlines in North America.

 NAQC. (Cummins, S.). (2009). Measuring reach of quitline programs. Quality

improvement initiative. Phoenix, AZ.

http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/docs/naqc_

issuepaper_measuringrea.pdf.

Reach calculations and definitions can be found in this NAQC Issue Paper

This paper recommends a standard definition of “reach” and provides

recommendations for standard measuring and reporting of the reach of quitline

programs. 

•

•

•



•

•

http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/Issue_Papers/

WhitePaper2015QRUpdate.pdf.

This paper provides a standard formula for measuring quit rates for quitline

programs and recommends standard reporting methods as well.

North American Quitline Consortium. (2015). Calculating Quit Rates, 2015 Update. 

(Betzner A, Lien B, Rainey J, et al.). Phoenix, AZ.
98 | Conducting Quitline Evaluations 

 NAQC. (Tinkelman D) . (2010). Assessing the cost-effectiveness of quitline programs. 

Phoenix, AZ. 

• http://c.ymcdn.com/sites/www.naquitline.org/resource/resmgr/issue_papers/

assessingcosteffectivenessof.pdf.

 NAQC. (Wendling A, & Daigh R.). (2013). Quitline referral systems. Phoenix, AZ. 

Retrieved from:

• http://c.ymcdn.com/sites/naquitline.site-ym.com/resource/resmgr/Issue_

Papers/QuitlineReferralSystemsQuali.pdf. 

NON-OSH EVALUATION RESOURCES

 American Evaluation Association

• www.eval.org

• The American Evaluation Association is an international professional 

association of evaluators devoted to the application and exploration of 

program evaluation, personnel evaluation, technology, and many other forms 

of evaluation. Evaluation involves assessing the strengths and weaknesses of 

programs, policies, personnel, products, and organizations to improve their 

effectiveness. AEA has approximately 5500 members representing all 50 states 

in the US as well as over 60 foreign countries. (Accessed 7/19/2011)

 CDC Division of Adolescent and School Health’s Program Evaluation Resources and 

Tools

• http://www.cdc.gov/healthyyouth/evaluation/

 CDC Division of STD Prevention’s Practical Use of Program Evaluation among 

Sexually Transmitted Disease (STD) Programs

• www.cdc.gov/std/program/pupestd/Introduction-SPREADS.pdf
99

 CDC Framework for Program Evaluation

• http://www.cdc.gov/eval/framework/index.htm

• Effective program evaluation is a systematic way to improve and account 

for public health actions that involves procedures that are useful, feasible, 

ethical, and accurate. The framework guides public health professionals in 

their use of program evaluation. It is a practical, nonprescriptive tool, designed 

to summarize and organize essential elements of program evaluation. The 

framework comprises steps in program evaluation practice and standards

for effective program evaluation. Adhering to the steps and standards of this 

framework will allow an understanding of each program’s context and will 

improve how program evaluations are conceived and conducted.

 Centers for Disease Control and Prevention. (2010). Comprehensive cancer control 

branch program evaluation toolkit. Atlanta (GA): U.S. Department of Health and Human 

Services, Centers for Disease Control and Prevention, National Center for Chronic 

Disease Prevention and Health Promotion, Division of Cancer Prevention and Control. 

• http://www.cdc.gov/cancer/ncccp/pdf/CCC_Program_Evaluation_Toolkit.pdf

 CDC Introduction to Program Evaluation for Public Health Programs: A Self Study 

Guide

• http://www.cdc.gov/eval/guide/

 Haddix A, Teutsch SM, & Corso PS. (Eds). (2003). Prevention effectiveness: A guide to 

decision analysis and economic evaluation (2nd edition). Oxford: Oxford University Press.

 Disseminating Program Achievements and Evaluation Findings to Garner Support

• www.cdc.gov/healthyyouth/evaluation/pdf/brief9.pdf

 Knowlton LW, & Philips CC. (2009). The logic model guidebook: Better strategies for 

great results. Thousand Oaks, CA: Sage Publications.

 National Heart Disease and Stroke Prevention Program’s Evaluation Guides: Writing 

SMART Objectives; Developing and Using Logic Models

• http://www.cdc.gov/dhdsp/programs/spha/evaluation_guides/index.htm

 Penn State Extension Program Evaluation Resources

• http://extension.psu.edu/evaluation/
100 | Conducting Quitline Evaluations 

 Western Michigan University. The Evaluation Center. Evaluation Checklists.

• https://www.wmich.edu/evaluation/checklists

• This site provides refereed checklists for designing, budgeting, contracting, 

staffing, managing, and assessing evaluations of programs, personnel, 

students, and other evaluands; collecting, analyzing, and reporting evaluation 

information; and determining merit, worth, and significance. Each checklist is a 

distillation of valuable lessons learned from practice.

 University of Wisconsin Extension: Program Development and Evaluation 

publications

• http://www.uwex.edu/ces/pdande/evaluation/evaldocs.html

• This site provides a range of publications for planning and implementing an 

evaluation and offers on-line evaluation curriculums and courses.

 W.K. Kellogg Foundation – logic model and evaluation guides

• http://www.wkkf.org/knowledge-center/resources/2006/02/WK-KelloggFoundation-Logic-Model-Development-Guide.aspx

• A guide to logic modeling to facilitate program planning and implementation 

activities. (Accessed 7/19/2011)

 Worthen BR, Sanders JR, & Fitzpatrick JL. (1997). Program evaluation: Alternative 

approaches and practical guidelines (2nd edition). New York, NY: Addison, Wesley 

Logman, Inc.
101

TELLING YOUR STORY: REPORTING EVALUATION RESULTS

 Atkinson C. (2007). Beyond bullet points: Using Microsoft Office PowerPoint 2007 to 

create presentations that inform, motivate, and inspire. Microsoft Press.

 Becker HS. (2007). Writing for social scientists: How to start and finish your thesis, 

book, or article (2nd edition). Chicago, IL: University of Chicago Press.

 CDC. Impact and Value: Telling Your Program’s Story

• http://www.cdc.gov/oralHealth/publications/library/pdf/success_story_

workbook.pdf

 CDC. Division of Community Health. Success Stories Application. 

• http://www.cdc.gov/NCCDPHP/dch/success-stories/

• The Division of Community Health’s Success Stories Application is a free online 

tool to help you turn your success into strong, professional, polished documents. 

» Easy-to-use tool to develop your story. 

» Guidance on how to write a success story. 

» A downloadable worksheet to begin the pre-writing process. 

» Three professionally developed templates. 

» A free photo library. 

» A Success Stories Library that you can search and share. 

 Evergreen S. (2011). Evaluation report layout checklist. Atlanta, GA: Tobacco 

Technical Assistance Consortium. Retrieved from http://www.ttac.org/resources/ 

pdfs/022912_Reporting_Well_ERLC-handout.pdf

 Few S. (2004). Show me the numbers: Designing tables and graphs to enlighten. 

Oakland, CA: Analytics Press.

 Heath C, & Heath, D. (2007). Made to stick: Why some ideas survive and others die. 

New York, NY: Random House.

 Lavinghouze R, Price AW, & Smith K-A. (2007). The program success story: A 

valuable tool for program evaluation. Health Promotion Practice. 8(4), 323-331.

 Torres R, Preskill H, & Piontek ME. (2004). Evaluation strategies for communicating 

and reporting (2nd edition). Thousand Oaks, CA: Sage Publications.
102 | Conducting Quitline Evaluations 

QUALITATIVE METHODS

 Guest G, MacQueen KM, & Namey EE. (2011). Applied thematic analysis. Sage 

Publications.

 Guest G, Bunce A, & Johnson L. (2006). How many interviews are enough? An 

experiment with data saturation and variability. Field Methods, 18(1), 59-82.

 Miles MB, & Huberman MA. (1994). Qualitative data analysis (2nd edition). Thousand 

Oaks, CA: Sage Publications.

 Patton MQ. (2001). Qualitative research and evaluation methods (3rd edition). 

Thousand Oaks, CA: Sage Publications.

 Yin RK. (2008). Case study research: Design and methods (Applied social research 

methods) (4th edition). Thousand Oaks, CA: Sage Publications.

 Yin Robert K. (2010). Qualitative Research from Start to Finish. New York, NY: The 

Guilford Press.

QUANTITATIVE METHODS

 Few S. (2004). Show me the numbers: Designing tables and graphs to enlighten (Vol. 

1, No. 1). Oakland, CA: Analytics Press.

 Few S. (2006). Information dashboard design: The effective visual communication of 

data (pp. 120-206). Sepastopol, CA: O’Reilly Media.

 Few S. (2009). Now you see it: Simple visualization techniques for quantitative 

analysis. Burlington, CA: Analytics Press.

 Kleinbaum DG & Klein M. (2010). Logistic regression: A self-learning text (Statistics 

for biology and health) (3rd edition). New York, NY: Springer.

 Rothman KJ, Greenland S & Lash TL. (2008). Modern Epidemiology (3rd edition). 

Philadelphia, PA: Lippincott Williams & Wilkins.

 Tufte ER. (1990). Envisioning information. Cheshire, CT: Graphics Press. 

 Tufte ER. (2001). The visual display of quantitative information. Cheshire, CT: 

Graphics Press. 
103

EVALUATION USE

 Butterfoss FD. (2007). Coalitions and partnerships in community health. San 

Francisco, CA: Jossey-Bass.

 Mattessich PW. (2003). The manager’s guide to program evaluation: Planning, 

contracting, and manging for useful results. St. Paul, Minnesota: Amherst H. Wilder 

Foundation.

 Patton MQ. (2010). Developmental evaluation: Applying complexity concepts to 

enhance innovation and use. New York, NY: The Guilford Press. 

 Patton MQ. (2008). Utilization-Focused Evaluation (4th edition). Thousand Oaks, CA: 

Sage Publications. 

OSH EVALUATION RESOURCES

 Best Practices for Comprehensive Tobacco Control Programs—2014 

• http://www.cdc.gov/tobacco/stateandcommunity/best_practices/index.htm

• CDC’s Best Practices for Comprehensive Tobacco Control Programs—2014 is 

an evidence-based guide to help states plan and establish effective tobacco 

control programs to prevent and reduce tobacco use.

 Introduction to Process Evaluation in Tobacco Use Prevention and Control 

• www.cdc.gov/tobacco/tobacco_control_programs/surveillance_evaluation/

process_evaluation/index.htm

• Published in 2008, this guide will help state and federal program managers and 

evaluation staff design and implement valid, reliable process evaluations for 

tobacco use prevention and control programs.

 Introduction to Program Evaluation for Comprehensive Tobacco Control Programs 

• http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_

evaluation/evaluation_manual/index.htm

• Published in 2001, this “how to” guide for planning and implementing 

evaluation activities will help state tobacco control program managers 

and staff in the planning, design, implementation, and use of practical and 

comprehensive evaluations of tobacco control efforts.
104 | Conducting Quitline Evaluations 

 Key Outcome Indicators for Evaluating Comprehensive Tobacco Control 

Programs

• http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_

evaluation/key_outcome/index.htm

• Published in 2005, this guide provides information on 120 key outcome 

indicators for evaluation of statewide comprehensive tobacco prevention and 

control programs.

 Question Inventory on Tobacco (QIT)

• https://chronicdata.cdc.gov/Survey-Questions-Tobacco-Use-/QuestionInventory-on-Tobacco-QIT-/vdgb-f9s3

• This web-based tool developed by CDC’s Office on Smoking and Health, 

categorizes more than 6,000 tobacco-related questions. This site can be 

used to collect information on survey questions used in the past, locate 

available data for secondary analyses, and gather ideas for future instrument 

development.

 Quitlines: A Resource for Development, Implementation, and Evaluation

• http://www.cdc.gov/tobacco/quit_smoking/cessation/quitlines/index.htm

• This 2005 document is intended to help state health departments, health care 

organizations, and employers to contract for and monitor telephone-based 

tobacco cessation services. It is also intended to help states, health care 

organizations, and quitline operators enhance existing quitline services, and 

to inform those who are interested in learning more about population-based 

approaches to tobacco cessation.

 Smoking-Attributable Mortality, Morbidity, and Economic Costs (SAMMEC).

• http://www.cdc.gov/oshdata/

• This online application allows you to estimate the health and health-related 

economic consequences of smoking to adults and infants.
105

 Office on Smoking and Health’s Interactive Data Dissemination Tool: OSHData 

• Available at: http://www.cdc.gov/oshdata/

• OSHData presents comprehensive tobacco prevention and control data in an 

online, easy to use, interactive data application

• Users can:

» Access data online.

» Reuse, redistribute, and download data sets for further analysis.

» Explore and download methodology and data source information.

» Create visualizations to share in presentations and reports.

» Subscribe to data updates.

 State Tobacco Activities Tracking and Evaluation (STATE) System

• Available at http://www.cdc.gov/statesystem/.

• The STATE System is an interactive application that houses and displays 

current and historical state-level data on tobacco use prevention and control in 

preformatted reports for easy access.

• Users can:

» View data online by topic or source (in addition to NQDW data).

» Choose multiple topics to compare data.

» Choose specific states to compare data.

» Access data by state in a Highlights Report.

» Compare data across states.

 Surveillance and Evaluation Data Resources for Comprehensive Tobacco 

Control Programs

• http://www.cdc.gov/Tobacco/tobacco_control_programs/surveillance_

evaluation/surveillance_manual/index.htm

• Published in June 2014, this compilation of data sources for tobacco 

control programs is useful for tobacco control programs that are conducting 

surveillance or evaluation.
106 | Conducting Quitline Evaluations 

 Surveillance and Evaluation Net Conferences

• Archived presentations available at http://www.ttac.org/resources/cdc_

netconferences.html

• The Surveillance and Evaluation Net-conference series provides information on 

evaluation best and promising practices and describes the role of evaluation 

in tobacco control work. The Net-conference series was originally designed 

for state surveillance and evaluation staff, but the material covers a variety of 

interesting and emerging topics in surveillance and evaluation that are valuable 

to other public health professionals. Each conference consists of a lecture 

followed by a question and answer session.

 Surveillance and Evaluation web page on CDC-OSH’s Smoking and Tobacco 

Use website 

• http://www.cdc.gov/tobacco/tobacco_control_programs/surveillance_

evaluation/index.htm

 Tobacco Control State Highlights 2012

• http://www.cdc.gov/tobacco/data_statistics/state_data/state_highlights/2012/

• Tobacco Control State Highlights 2012 guides states in developing and 

implementing high-impact strategies and assessing their performance.

• Highlight how some states are making great strides in reducing smoking rates 

by using evidence-based strategies while also showing that more work needs 

to be done in other states.

• Enable readers to see how their own states perform.

• Help policymakers with decision making.
107
108 | Conducting Quitline Evaluations 
109
National Center for Chronic Disease Prevention and Health Promotion 

Office on Smoking and Health

CS257353