56

IV. Surveillance and Evaluation

Justification 

Publicly financed programs need to have accountability and demonstrate effectiveness, as well 

as have access to timely data that can be used 

for program improvement and decision making. 

Therefore, a critical infrastructure component of 

any comprehensive tobacco control program is a 

surveillance and evaluation system that can monitor and document key short-term, intermediate, 

and long-term outcomes within populations.1,2 Data 

obtained from surveillance and evaluation systems 

can be used to inform program and policy direction, demonstrate program effectiveness, ensure 

accountability to those with fiscal oversight, and 

engage stakeholders.2-6

Surveillance and evaluation planning may 

be integrated into the overall strategic plan of 

a comprehensive tobacco control program and 

be compatible and comparable with systems in 

other states and nationally.2 A strategic plan, with 

well-defined goals, objectives, and outcomes, 

requires appropriate data collection methods that 

can monitor the program, as well as evaluate 

key outcome indicators in a valid manner.7

Additionally, the collection of baseline data 

related to each objective and outcome indicator 

is critical to ensuring that program-related effects 

can be clearly measured.3,5 For this reason, 

surveillance and evaluation systems must have 

priority in the strategic planning process.

Surveillance 

Surveillance is the process of continuously monitoring attitudes, behaviors, and health outcomes 

over time.8 Although data gathered by surveillance 

systems can be useful for evaluation, they serve 

other purposes besides evaluation. For example, 

data collection for the purposes of evaluation is 

more flexible than for surveillance and may allow 

program areas to be assessed in greater depth. 

Statewide tobacco control surveillance programs 

should consider monitoring the achievement of the 

four overarching goals of comprehensive tobacco 

control programs:

 Preventing initiation among youth and 

young adults 

 Promoting quitting among adults and youth

 Eliminating exposure to secondhand smoke

 Identifying and eliminating tobacco-related 

disparities among population groups

Implementing state surveillance systems, 

such as the Behavioral Risk Factor Surveillance 

System (BRFSS), Youth Risk Behavior Surveillance 

System (YRBSS), Pregnancy Risk Assessment 

Monitoring System (PRAMS), and the Adult or 

Youth Tobacco Surveys (ATS, YTS), affords each 

state the opportunity to collect data on tobacco 

use behaviors and other important risk factors and 

health outcomes.9–11 Data from these systems also 

allow a state to compare its individual program 

impact and long-term tobacco indicators with other 

states as well as with national benchmarks from 

national surveillance systems. In addition to the 

standard core questions included in these surveys, 

there is flexibility to add state-specific questions 

and modules. States also have the flexibility to 

increase sample size in order to capture local 

and specific population data or to provide more 

data on intermediate performance outcomes. 

Evaluation 

Evaluation has been defined as the systematic 

collection of information about the activities, 

characteristics, and results of programs to make 

judgments about the program, improve or further 

develop program effectiveness, inform decisions 

about future programming, and/or increase 

understanding.12 Evaluation data can be used for 

assessing the effectiveness of individual program 

activities, program improvement, decision making, 

and to engage stakeholders. However, in order to 

do all these things, a written evaluation plan must 

first be integrated with the overall strategic plan. 

An effective evaluation plan:13

 Is collaboratively developed with a 

stakeholder workgroup

 Is responsive to program changes and priorities

 Covers multiple years if projects are ongoing
57

Section A: Surveillance and Evaluation

 Addresses the entire program rather 

than focusing on a single funding 

source, objective, or activity 

States can also consider publishing their 

evaluation results in order to contribute to 

the scientific literature on best practices 

for tobacco control programs.7

A typical approach to evaluation in public 

health is to design data-collection systems that 

monitor progress toward meeting a program’s 

process and outcome objectives.8 Process 

evaluations are used to document how well a 

program has been implemented and are conducted 

periodically during a program.8 This type of an 

evaluation is used to examine the program’s 

operations, including which activities are taking 

place, who is conducting the activities, and who 

is reached through the activities. In contrast, 

outcome evaluations are used to assess the 

effectiveness of a program on the stated shortterm, intermediate, and long-term objectives.8

This type of evaluation assesses what has 

occurred because of the program and whether 

the program has achieved its objectives. 

The program’s stages of development must be 

considered in the evaluation plan, particularly when 

determining the appropriate evaluation questions. 

Outcome evaluations are best conducted only 

when the program is mature enough to produce 

the intended outcome. However, consideration 

for future evaluations can be included in the 

evaluation plan so that programs can prepare 

datasets and baseline information for evaluations 

that consider more distal impacts and outcomes.14

An evaluation plan can include both process 

and outcome evaluation questions at the same 

time.14 Program evaluation also requires that a wide 

range of short-term, intermediate, and long-term 

indicators of program effectiveness be measured, 

including changes in policies, social norms, and 

exposure of individuals and communities to 

statewide and local program efforts. For example, 

evaluation efforts might include countermarketing 

surveillance to track new products and examine 

the impact of pro-tobacco influences, including 

tobacco product marketing, pricing, and promotion. 

Additional indicators for program evaluation 

can include, but need not be limited to, vital 

statistics, quitline utilization, policy compliance 

and enforcement, air quality, or media related 

measures. Practice-based criteria to be considered 

in the selection of indicators for monitoring and 

evaluation have previously been listed elsewhere.15

Qualities of Effective Program Evaluations2,13,16

 Ongoing and include a written evaluation plan that is 

integrated with the program’s overall strategic plan.

 Flexible, adaptive, transparent, and designed to inform 

and engage stakeholders at each step, including 

implementation, interpretation, dissemination, and 

utilization of results.

 Focus on priority evaluation questions and not special 

research interests or what is easiest to implement.

 Confirm that the methods align with the evaluation 

questions and objectives.

 Identify credible evidence and verify its accuracy and 

appropriateness with stakeholders.

 Make effective use of surveillance data by linking 

statewide and local program efforts to monitor progress 

toward program objectives.

 Plan for dissemination and sharing of lessons learned 

throughout the evaluation process. 

 Include technical assistance to disseminate information 

on how to implement effective evaluations to funded 

sites, partners, stakeholders, and local programs. 
58

Section A: Surveillance and Evaluation

Selected Surveillance and 

Evaluation Resources

Surveillance and evaluation can be conducted 

simultaneously.8 To assess tobacco-use prevention 

and control efforts adequately, states will usually 

need to supplement surveillance data with data 

collected to answer specific evaluation questions. 

States can collect data on, for example, knowledge, 

attitudes, behaviors, and environmental indicators. 

They can also collect information on infrastructure, 

program planning, and implementation to 

document and measure the effectiveness of a 

program, including its policy and media efforts. 

Some existing tools for both surveillance and 

evaluation at the state and national levels include: 

Adult Tobacco Survey (ATS): ATS is a state level 

landline and cellular telephone survey of adults 

aged 18 years or older.17 Core questions assess 

adults’ knowledge, attitudes, and behaviors related 

to tobacco use, secondhand smoke exposure, use 

of cessation assistance, and their awareness of and 

support for evidence-based policy interventions. 

In addition to these core questions, ATS 

allows for the inclusion of questions addressing 

state-specific program activities. CDC’s Key 

Outcome Indicators for Evaluating Comprehensive 

Tobacco Control Programs was used to inform the 

development of the ATS survey.1 CDC’s Office on 

Smoking and Health can provide technical assistance 

to states regarding the administration of ATS.

Behavioral Risk Factor Surveillance System (BRFSS):

BRFSS is a state-based telephone survey of noninstitutionalized U.S. adults aged 18 years or older 

that CDC initiated in 1984.9 Data are currently collected annually in all 50 states, the District of 

Columbia, and five U.S. territories. With assistance 

from CDC, state health departments contract with 

telephone call centers to conduct BRFSS surveys 

continuously throughout the year using a standardized core questionnaire and optional modules 

plus additional state-added questions. Beginning in 

2011, several enhancements were made to BRFSS 

to ensure optimal survey coverage and validity, 

including the addition of cellular telephone households and improvements to the sampling methods 

and statistical weighting.18

National Adult Tobacco Survey (NATS): NATS is a 

landline and cellular phone survey of U.S. adults 

aged 18 years or older.19 NATS was first conducted 

during 2009–2010, and the sample was designed 

to provide data representative at both national 

and state levels.19 Additional waves of NATS were 

fielded in 2012-2013 and 2013-2014 in collaboration 

with FDA; however state-level estimates will only 

be obtainable during 2009-2010.

National Youth Tobacco Survey (NYTS): NYTS is 

a nationally representative school-based survey 

of youth in middle school (grades 6–8) and high 

school (9–12).20 NYTS cannot be used to obtain 

state-level estimates, but estimates from the survey can serve as a national benchmark for those 

obtained from state YTS surveys. NYTS is a multitopic survey that includes measures that assess 

tobacco use, cessation, knowledge and attitudes, 

access, media and advertising, and secondhand 

smoke exposure.1 Survey years include 2000, 2002, 

2004, 2006, 2009, 2011, and 2012. As of 2012, 

NYTS will be fielded annually until 2017 in collaboration with FDA.

Pregnancy Risk Assessment Monitoring System 

(PRAMS): PRAMS is a surveillance system that 

CDC and state health departments have conducted 

in multiple phases since 1987; PRAMS data were 

most recently collected in 2011.11 PRAMS collects 

state-specific, population-based data on maternal attitudes and experiences before, during, and 

shortly after pregnancy. The PRAMS questionnaire 

comprises two parts, including core questions 

that are asked by all states and a pretested list of 

standard questions that CDC or individual states 

develop. The core PRAMS questionnaire includes 

questions on maternal tobacco consumption. 

Quitline Minimum Data Set (MDS): The quitline 

MDS identifies a recommended set of indicators to 

assist in assessing telephone quitline performance, 

improving the quality of telephone quitlines, 

identifying knowledge gaps, and designing new 

strategies to fill the identified gaps.21

State Tobacco Activities Tracking and Evaluation 

(STATE) System: The STATE System is an online 

data warehouse that includes epidemiologic data 

on many long-term key outcome indicators, as 

well as economic data and tobacco-related state 

legislation.22

Tobacco Use Supplement to the Current Population Survey (TUS-CPS): TUS-CPS is an in-person 

and telephone survey of U.S. adults aged 18 

years and older that was administered during 

1992–1993, 1995–1996, 1998–1999, 2002–2003, 
59

Section A: Surveillance and Evaluation

2006–2007, and 2010–2011; the next wave is 

slated for 2014–2015.23 These tobacco-use modules provide national and state-specific estimates 

on factors such as tobacco use, quit attempts, 

secondhand smoke exposure, smokefree policies, and clinician cessation counseling.

Youth Risk Behavior Surveillance System (YRBSS):

YRBSS is a national school-based survey of middle 

and high school students conducted biennially by 

CDC.10 YRBSS also includes state, territorial, tribal,

and local surveys conducted by state, territorial,

and local education and health agencies and tribal

governments. YRBSS monitors six types of healthrisk behaviors that contribute to leading causes

of death and disability among youth and adults,

including tobacco use.

Youth Tobacco Survey (YTS): YTS is a schoolbased, state-level survey of students in grades 

6–12.24 Core questions assess students’ knowledge, 

attitudes, and behaviors related to tobacco use and 

exposure to secondhand smoke, as well as their 

exposure to prevention curricula, community programs, and media messages aimed at preventing 

and reducing youth tobacco use. In addition to 

the core set of questions, YTS allows for the inclusion of questions addressing state-specific program 

activities. CDC’s Key Outcome Indicators for Evaluating Comprehensive Tobacco Control Programs 

was used to inform the development of the YTS 

survey.1 CDC’s Office on Smoking and Health can 

provide technical assistance to states regarding the 

administration of YTS.

In addition to the previously described 

surveillance and evaluation tools, several resources 

are available to provide guidance and support 

to states on the selection and implementation of 

appropriate surveillance and evaluation data systems. 

Surveillance and Evaluation Resources 

 Surveillance and Data Resources for Comprehensive 

Tobacco Control Programs provides a summary 

of tobacco-related measures, sampling frames, and 

methodology for multiple national and state surveys 

as well as tools for use in conducting surveillance and 

evaluation efforts.25

 Introduction to Program Evaluation for 

Comprehensive Tobacco Control Programs is 

a “how-to” guide for planning and implementing 

evaluation activities.8

 Key Outcomes Indicators for Evaluating 

Comprehensive Tobacco Control Programs provides 

information on selecting evidence-based indicators and 

linking them to program outcomes.1

 Introduction to Process Evaluation in Tobacco Use 

Prevention and Control provides guidance to states 

on how to evaluate inputs, activities, and outputs of a 

tobacco control logic model.26

 Developing an Effective Evaluation Plan can help 

public health program managers, administrators, and 

evaluators develop an effective evaluation plan in the 

context of the planning process. It is intended to be 

used along with other evaluation resources and is not 

a complete resource on how to implement program 

evaluation.13 For example, disseminating surveillance 

and evaluation findings in brief updates or newsletters 

to key stakeholders may also be beneficial. 

 Developing an Effective Evaluation Reportcan help 

public health program managers, administrators, and 

evaluators develop an effective evaluation report. It is 

intended to be used along with other evaluation resources 

and is not a complete resource on how to write reports or 

communicate and use your evaluation results.14

 Impact and Value: Telling Your Program’s 

Story offers public health program managers 

practical steps for creating success stories 

that highlight their achievements.27
60

Section A: Surveillance and Evaluation

State-Level Examples

Surveillance and evaluation data can be used by 

states in multiple ways to help inform and sustain 

comprehensive state tobacco control programs. For 

example, states can collect their own state-level 

surveillance and evaluation data using previously 

developed instruments and resources, supplement existing surveillance systems with indicators 

related to specific state tobacco control program 

objectives, or utilize secondary data sources to 

assess key indicators and make comparisons with 

other states or national benchmarks. Examples of 

some recent state-level surveillance and evaluation 

activities are described below. 

The New York Tobacco Control Program 

has fielded variations of the ATS and YTS 

questionnaires regularly for more than a decade. 

The program utilizes these data to provide 

a comprehensive summary of multiple key 

outcomes indicators in its annual Independent 

Evaluation Report.28 These reports help to 

clearly and objectively illustrate the impact that 

the state tobacco control program has had on 

key outcome indicators, as well as to highlight 

gaps that need to be addressed in the future. 

Multiple states supplement the core BRFSS 

questionnaire with optional modules to inform 

state-level tobacco control program efforts.9

For example, in 2011, two optional modules 

pertaining to smoking cessation and secondhand 

smoke were proposed and ratified. The 

smoking cessation module was administered 

by Arizona, Guam, Kentucky, Louisiana, 

Maryland, and Nebraska; the secondhand 

smoke module was administered by Guam, 

Indiana, Kentucky, Louisiana, and Mississippi. 

The Public Health Division of the Wyoming 

Department of Health recently utilized statelevel data from the 2009–2010 NATS to measure 

progress toward attaining the objectives of 

Wyoming’s Tobacco Prevention and Control 

Program. A summary of the findings were 

described in a comprehensive report organized 

according to CDC’s four overarching goals for 

comprehensive tobacco control programs.7,29

Achieving Equity to Eliminate 

Tobacco-Related Disparities 

Dissemination of surveillance and evaluation 

data that show disparities can be very effective 

in mobilizing community involvement. In order 

to develop effective interventions and monitor 

progress, most states need more information on 

populations disproportionately affected by tobacco 

use.7 Many of the surveillance and evaluation 

resources described in this report include 

questions related to population characteristics 

for which tobacco-related disparities have been 

shown to exist, including but not limited to: 

race/ethnicity, educational attainment, income, 

occupation, geographic location, sex, age, sexual 

orientation and gender identity, veteran and 

military status, disability status, mental health 

status, and substance abuse conditions. 

However, it is important to note that existing 

surveillance and evaluation methods may 

not provide adequate sample size or enough 

information to fully characterize health disparities 

related to tobacco use. Therefore, additional data 

collection systems or approaches may be needed. 

For example, the use of oversampling, combining 

multiple years of data, and qualitative methods 

are often necessary to adequately assess these 

outcomes among some population groups. 
61

Section A: Surveillance and Evaluation

Budget

All federally funded tobacco prevention and control programs are expected to engage in strategic 

surveillance and program evaluation activities. To 

accomplish this, best practices dictate that 10% 

of total annual tobacco control program funds be 

allocated for surveillance and evaluation. 

It is important that tobacco control programs 

develop and maintain the appropriate infrastructure 

to enhance their surveillance and evaluation 

resources as needed. For example, conducting a 

detailed evaluation of a specific intervention, such 

as a cohort study to assess the effectiveness of 

a media campaign, can be resource intensive.7,30

Similarly, additional resources beyond the 

standard 10% of tobacco control program funds 

may also be required for developing increased 

technical capacity of local programs to perform 

process and outcome evaluation.7,30 For example, 

in California, every grantee was required to 

spend 10% of its budget on evaluating its own 

activities. To aid this activity, the state program 

published a directory of evaluation consultants 

and funded a local program evaluation center that 

provides technical assistance to contractors.31

In addition, programs may need to be flexible 

in shifting funding to address new and emerging 

products or trends of public health concern. 

For example, recent increases in electronic 

cigarette marketing and use warrant targeted 

surveillance, monitoring, and evaluation that 

may not have been recognized if a program’s 

plan was developed several years ago.

Realizing the national goal of eliminating 

tobacco-related disparities will require improved 

collection and use of standardized data to correctly 

identify disparities in both health outcomes and 

interventional efficacy.7 Accordingly, additional 

resources may also be required to fund data 

collection mechanisms and standardized systems 

to better characterize health disparities related 

to tobacco use among special populations 

and to effectively measure progress.
62

Section A: Surveillance and Evaluation

References

1. Starr G, Rogers T, Schooley M, Porter S, Wiesen E, 

Jamison N. Key Outcome Indicators for Evaluating 

Comprehensive Tobacco Control Programs. Atlanta: 

Centers for Disease Control and Prevention, National 

Center for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, 2005.

2. Lavinghouze R, Snyder K. Developing your 

evaluation plans: a critical component of public 

health program infrastructure. American Journal 

of Health Education 2013;44(4):237–43.

3. Giovino GA, Biener L, Hartman AM, Marcus 

SE, Schooley MW, Pechacek TF, Vallone D. 

Monitoring the tobacco use epidemic. I. Overview: 

optimizing measurement to facilitate change. 

Preventive Medicine 2009;48(1 Suppl):4S–10S.

4. Institute of Medicine. Ending the Tobacco 

Problem: A Blueprint for the Nation. Washington: 

National Academies Press, 2007.

5. World Health Organization (WHO). WHO Report on 

the Global Tobacco Epidemic, 2008: The MPOWER 

Package. Geneva, Switzerland: WHO, 2008. 

6. Lavinghouze R, Snyder K, Rieker P, Ottoson J. 

Consideration of an applied model of public 

health program infrastructure. Journal of Public 

Health Management and Practice 2013;19(6):E28–

E37. DOI: 10.1097/PHH.0b013e31828554c8. 

7. Centers for Disease Control and Prevention. 

Best Practices for Comprehensive Tobacco Control 

Programs—October 2007. Atlanta: U.S. Department 

of Health and Human Services, Centers for 

Disease Control and Prevention, National Center 

for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, 2007.

8. MacDonald G, Starr G, Schooley M, Yee SL, Klimowski 

K, Turner K. Introduction to Program Evaluation for 

Comprehensive Tobacco Control Programs. Atlanta: 

Centers for Disease Control and Prevention, National 

Center for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, 2001.

9. Centers for Disease Control and Prevention. Behavioral 

Risk Factor Surveillance System; <http://www.

cdc.gov/brfss/>; accessed: December 2, 2013.

10. Centers for Disease Control and Prevention. 

Youth Risk Behavior Surveillance System; 

<http://www.cdc.gov/HealthyYouth/yrbs/

index.htm>; accessed: December 2, 2013.

11. Centers for Disease Control and Prevention. Pregnancy 

Risk Assessment Monitoring System; <http://www.

cdc.gov/PRAMS/>; accessed: December 2, 2013.

12. Patton, MQ. Utilization-Focused Evaluation. 4th 

ed. Los Angeles: Sage Publications, 2008.

13. Centers for Disease Control and Prevention. 

Developing an Effective Evaluation Plan. Atlanta: 

U.S. Department of Health and Human Services, 

Centers for Disease Control and Prevention, National 

Center for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, Division 

of Nutrition, Physical Activity, and Obesity, 2011.

14. Centers for Disease Control and Prevention. 

Developing an Effective Evaluation Report. Atlanta: 

Centers for Disease Control and Prevention, National 

Center for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, Division 

of Nutrition, Physical Activity and Obesity, 2013.

15. MacDonald, G. Criteria for Selection of HighPerforming Indicators: A Checklist to Inform 

Monitoring and Evaluation; <http://www.wmich.edu/

evalctr/checklists/>; accessed: December 2, 2013. 

16. Lavinghouze SR, Price AW, Smith KA. The program 

success story: a valuable tool for program evaluation. 

Health Promotion Practice 2007;8(4):323–31.

17. McClave AK, Whitney N, Thorne SL, Mariolis P, 

Dube SR, Engstrom M. Adult Tobacco Survey—19 

states, 2003–2007. Morbidity and Mortality Weekly 

Report Surveillance Summary 2010;59(SS3):1–75.

18. Centers for Disease Control and Prevention. 

Methodologic changes in the Behavioral Risk 

Factor Surveillance System in 2011 and potential 

effects on prevalence estimates. Morbidity and 

Mortality Weekly Report 2012;61(22):410–3.

19. King BA, Dube SR, Tynan MA. Current tobacco 

use among adults in the United States: findings 

from the National Adult Tobacco Survey. American 

Journal of Public Health 2012;102(11):e93–e100. 

20. Centers for Disease Control and Prevention. 

National Youth Tobacco Survey; <http://www.

cdc.gov/tobacco/data_statistics/surveys/

nyts/>; accessed: December 2, 2013.

21. North American Quitline Consortium. 

Quitline Minimal Data Set; <http://www.naquitline.

org/?page=mds>; accessed: December 2, 2013.

22. Centers for Disease Control and Prevention. 

State Tobacco Activities Tracking and Evaluation 
63

Section A: Surveillance and Evaluation

(STATE) System; <http://www.cdc.gov/tobacco/

statesystem>; accessed: December 2, 2013.

23. National Cancer Institute. Tobacco 

Use Supplement to the Current Population 

Survey; <http://riskfactor.cancer.gov/studies/

tus-cps/>; accessed: December 2, 2013.

24. Centers for Disease Control and Prevention. 

Youth Tobacco Survey; <http://www.cdc.

gov/tobacco/data_statistics/surveys/YTS/

index.htm>; accessed: December 2, 2013.

25. Yee SL, Schooley M. Surveillance and 

Evaluation Data Resources for Comprehensive 

Tobacco Control Programs. Atlanta: Centers for 

Disease Control and Prevention, National Center 

for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, 2001.

26. Centers for Disease Control and Prevention. 

Introduction to Process Evaluation in Tobacco Use 

Prevention and Control. Atlanta: U.S. Department 

of Health and Human Services, Centers for 

Disease Control and Prevention, National Center 

for Chronic Disease Prevention and Health 

Promotion, Office on Smoking and Health, 2008.

27. Centers for Disease Control and Prevention. 

Impact and Value: Telling Your Program’s Story. 

Atlanta: Centers for Disease Control and Prevention, 

National Center for Chronic Disease Prevention and 

Health Promotion, Division of Oral Health, 2007.

28. New York State Department of Health. 

Reports, Brochures, and Fact Sheets; <http://

www.health.ny.gov/prevention/tobacco_

control/reports_brochures_fact-sheets.

htm>; accessed: December 2, 2013.

29. Wyoming Survey and Analytic Center. 2010 

Wyoming National Adult Tobacco Survey Report; 

< http://wysac1.uwyo.edu/wysac/ReportView.

aspx?DocId=529&A=1>; accessed: December 2, 2013. 

30. World Health Organization European Working Group 

on Health Promotion Evaluation. Health Promotion 

Evaluation: Recommendations to Policy-makers: 

Report of the WHO European Working Group on Health 

Promotion Evaluation. Copenhagen, Denmark: World 

Health Organization, Regional Office for Europe, 1998.

31. Roeseler A, Hagaman T, Kurtz C. The 

use of training and technical assistance to 

drive and improve performance of California’s 

Tobacco Control Program. Health Promotion 

and Practice 2011;12(6 Suppl 2):130S–143S.