Appendices and Glossary�

A P P E N D I X A�

National Tobacco Control Program �

An Overview

The Centers for Disease Control and Prevention (CDC) is the lead federal agency 

for comprehensive tobacco prevention and control. CDC develops, conducts, and 

supports strategic activities to protect the public’s health from the harmful effects 

of tobacco use.

To carry out its mission, CDC: ▲ ▲ ▲ ▲ ▲

Expands the science base for effective tobacco control.�

Builds sustainable capacity and infrastructure for comprehensive tobacco �

control programs and policies.�

Communicates information about tobacco issues to policy makers, health �

professionals, and the public.�

Provides technical assistance on developing, implementing, and evaluating �

tobacco control policies, strategies, and initiatives.�

Builds strategic partnerships with national and international organizations. �

Through its Office on Smoking and Health, CDC manages the National Tobacco 

Control Program (NTCP), which funds comprehensive tobacco control programs in 

state health departments and territories. NTCP-funded programs work to implement 

the strategies described in the following publications: ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

Best Practices for Comprehensive Tobacco Control Programs1

Reducing Tobacco Use: A Report of the Surgeon General2

The Guide to Community Preventive Services: Tobacco Use Prevention 

and Control3 

Treating Tobacco Use and Dependence: Clinical Practice Guideline4

The Health Consequences of Smoking: A Report of the Surgeon General5

Preventing Tobacco Use Among Young People: A Report of the Surgeon General6

Women and Smoking: A Report of the Surgeon General7

Tobacco Use Among U.S. Racial/Ethnic Minority Groups—African Americans, 

American Indians and Alaska Natives, Asian Americans and Pacific Islanders, 

and Hispanics: A Report of the Surgeon General8

CDC created NTCP to encourage coordinated, nationwide activities to reduce 

tobacco-related disease and death. NTCP provides funds and technical support 

to all 50 states, the District of Columbia, seven U.S. territories, and eight national 

networks of Indian tribes, Alaskan Natives, and other minority ethnic groups. 

A P P E N D I X A 

275
NTCP’s Goals 

The overall goal of NTCP’s comprehensive tobacco control programs is to reduce 

tobacco-related disease, disability, and death. This goal is subdivided into four 

goal areas: ▲ ▲ ▲ ▲

Preventing initiation of tobacco use among young people.

Eliminating nonsmokers’ exposure to secondhand smoke.

Promoting quitting among adults and young people.

Identifying and eliminating tobacco-related disparities.

The Four Strategies of the NTCP � ▲ ▲ ▲ ▲

Population-based community interventions.�

Countermarketing.�

Public policies and regulations to reduce tobacco use.�

Surveillance and evaluation.�

For more information on the NTCP go to http://www.cdc.gov/tobacco.

References

1. �Centers for Disease Control and Prevention. Best practices for comprehensive tobacco 

control programs. Atlanta, GA: Centers for Disease Control and Prevention; 1999. 

2. �U.S. Department of Health and Human Services. Reducing tobacco use: a report of 

the Surgeon General. Atlanta, GA: Centers for Disease Control and Prevention; 2000.

3. �Task Force on Community Preventive Services. The guide to community preventive services: tobacco use prevention and control. American Journal of Preventive 

Medicine. 2001;20(Suppl 2):1–88.

4. �Fiore MC, Bailey WC, Cohen SJ, Dorfman SF, Goldstein MG, Gritz EG, Heyman RB, 

Jaén CR, Kottke TE, Lando HA, Mecklenburg RE, Mullen PD, Nett LM, Robinson 

L, Stitzer ML, Tommasello AC, Villejo L, Wewers ME. Treating tobacco use and 

dependence: clinical practice guideline. Rockville, MD: U.S. Department of Health 

and Human Services. Public Health Service; 2000.

5. �U.S. Department of Health and Human Services. The health consequences of smoking: 

a report of the Surgeon General. Atlanta, GA: Centers for Disease Control and Prevention; 2004.

6. �U.S. Department of Health and Human Services. Preventing tobacco use among young 

people: a report of the Surgeon General. Atlanta, GA: Centers for Disease Control and 

Prevention; 1994.

7. �U.S. Department of Health and Human Services. Women and smoking: a report of the 

Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, 

Public Health Service, Office of the Surgeon General; 2001.

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

276
8. �U.S. Department of Health and Human Services. Tobacco Use Among U.S. Racial/

Ethnic Minority Groups—African Americans, American Indians and Alaska Natives, 

Asian Americans and Pacific Islanders, and Hispanics: a report of the Surgeon General.

Atlanta, GA: U.S. Department of Health and Human Services, Centers for 

Disease Control and Prevention, National Center for Chronic Disease Prevention 

and Health Promotion, Office on Smoking and Health; 1998.

A P P E N D I X A 

277
K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

278
A P P E N D I X B�

Selecting and Rating the Indicators� ▲ ▲ ▲

The Centers for Disease Control and Prevention (CDC) began producing this publication by appraising the logic models for three of the four goal areas of the National 

Tobacco Control Program (NTCP): 

Preventing initiation of tobacco use among young people.

Eliminating nonsmokers’ exposure to secondhand smoke. 

Promoting quitting among adults and young people.

As a result of the appraisal, our previously published logic models were updated, 

and the new versions are published here.1

Selecting the Indicators and Data Sources

After an extensive review of published and fugitive literature, we selected candidate 

indicators for the outcome components of each NTCP goal area’s logic model. Then 

we reviewed the scientific evidence for an association between the candidate indicators and the outcome components in the NTCP logic models. For example, we looked 

for evidence that an increase in levels of support for policies, and enforcement of 

policies, to decrease young people’s access to tobacco (indicator 1.6.4) is associated 

with a reduction in the percentage of teenagers who experiment with tobacco 

(outcome 10 in goal area 1).

Next, we selected example data sources and survey questions for each indicator. 

One important criterion used to select example data sources was their easy availability to state tobacco control programs. Such data sources include the Behavioral Risk 

Factor Surveillance System; Adult Tobacco Survey: CDC-Recommended Questions; 

Youth Tobacco Survey: CDC-Recommended Questions; Current Population Survey: 

Tobacco Use Supplement; CDC Pregnancy Risk Assessment Monitoring System; and 

the CDC Youth Risk Behavior Surveillance System. 

The selected survey questions come primarily from these survey or surveillance 

systems. However, if these sources had no appropriate questions to measure the 

indicator, we developed example questions or chose questions from national or 

state surveys and evaluation protocols (e.g., Legacy Media Tracking Survey) that 

are not widely used by state tobacco control programs, although they are available 

to them.

Rating the Indicators

We assembled a panel of experts (whose names are listed in Appendix C) to rate the 

final set of candidate indicators. The principal reason for having experts rate the indicators was to have them advise CDC on which indicators were key for evaluation of 

comprehensive state tobacco control programs. The experts also assessed the indicators on the basis of several criteria and advised us about which data sources are most 

A P P E N D I X B 

279
useful for tracking these indicators. In developing the rating process, we first did a 

pilot test. As a result of that test, we refined the indicator rating process, instructions 

to raters, and supportive materials (see page 284). 

The panelists were asked to rate each of the 136 candidate indicators separately 

according to the following criteria: 

Strength of the evaluation evidence. The extent to which the literature supports 

use of the indicator for the evaluation of comprehensive, statewide tobacco control 

programs, as characterized by the logic models. 

Reference citations on each indicator rating form were intended to provide 

guidance for reviewer ratings.

Resources needed for data collection and analysis. The amount of funds, 

time, and effort needed to collect reliable and precise data on the indicator and 

to analyze primary or secondary data. 

In making their judgments, reviewers were instructed to consider the availability 

of existing data (e.g., archival records or other secondary data) and the difficulties related to sampling and data collection methods. We reminded reviewers that 

many state health departments do not have extensive data collection systems for 

use in comprehensive evaluations of their tobacco control programs. However, 

all states have access to data on adults from the Behavioral Risk Factor Surveillance 

System, as well as periodic data on attitudes and policies through the Tobacco 

Use Supplements of the Current Population Survey. In addition, CDC synthesizes 

behavioral and policy data on the State Tobacco Activities Tracking and Evaluation (STATE) system. The resources needed for data collection and analysis 

are less when data are already available than when new data must be collected 

and analyzed.

Utility. The extent to which the indicator would help to answer key evaluation 

questions for a state comprehensive tobacco control program. 

Although many indicators are also appropriate and useful for evaluating local 

tobacco control programs, reviewers were asked to consider the utility of each 

indicator for evaluating state tobacco control programs.

Face validity. The extent to which judgments about and measurements of the 

indicator would appear valid and relevant to policy makers and other decision 

makers who use the results of an evaluation to justify their continued support. 

Uniqueness. Whether the indicator contributes distinctive information for the 

evaluation of tobacco control efforts. 

Reviewers who believed that an indicator was not unique were instructed to 

identify the redundant indicator.

Conformity with accepted practice. The degree to which use of the indicator as 

a measure of a tobacco control program’s progress is consistent with accepted, 

real-world tobacco control practice. 

Overall quality. A global rating that reflects the reviewer’s opinion of the overall 

quality of the indicator.

▲

▲

▲ ▲ ▲ ▲ ▲

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

280
▲

Summary rating. The reviewer’s opinion of how essential a particular indicator is 

for the evaluation of comprehensive, statewide tobacco control programs.

After the rating process, 31 indicators were merged, 4 eliminated, and 7 added, leaving a total of 120 indicators for which we provide information in this publication.

In addition, we asked the expert raters to: ▲ ▲ ▲

Comment on the data sources and survey questions that CDC had selected for 

each proposed indicator.

Suggest alternative data sources and questions.

Suggest additional indicators that would be useful for evaluation of comprehensive state tobacco control programs.

Each expert used a separate rating form for each indicator (see end of this appendix 

for a reprint of the rating form and rater instructions). 

The form has three sections: ▲ ▲ ▲

A summary of information on CDC’s proposed indicator and logic model component to which it relates, suggested data sources and survey questions, and (when 

available) a reference to the scientific evidence supporting the use of the indicator.

A rating scale for each criterion.

Space for reviewer comments. 

We also encouraged the experts to write notes on the rating forms and to provide 

additional information, references, or other documentation.

Analysis and Synthesis of Data from the Expert Reviews

After CDC received the completed rating forms from the experts, all data (including

written comments) were entered into an electronic file. We adjusted for multiple 

responses, skipped items, and coding errors. If, for example, a rater circled more than 

one response for a criterion, we averaged the responses unless the rater had noted a 

preference for one response over another. Skipped items and “don’t know” responses 

were combined into a “no answer” category. All data were analyzed using the Statistical Analysis System (SAS v.8.02).2 

For each type of rating, numerical data were analyzed in various ways. Frequency 

distributions of numerical data were analyzed to help us understand the raters’ 

perceptions about the indicators. Narrative comments included on the raters’ rating 

sheets were also reviewed to help us understand why raters gave an indicator a 

particularly high or low rating. To limit the effect of outliers, we used the median 

scores for each indicator.

After reviewing the experts’ ratings, we decided to combine indicators that were 

originally divided by population group (e.g., young people, adults). The experts’ 

A P P E N D I X B 

281
numeric ratings for the 31 merged indicators are not provided in this publication 

but are noted with NR. In addition, after reviewing the rating data and comments 

carefully, we eliminated four indicators that were rated “not essential” by most panel 

members.

CDC also reviewed the expert panelists’ “resources needed” scores (their estimate 

of the intensity of resources required to collect and analyze data on each indicator). 

CDC substituted scores for six indicators that were rated by the experts. For example, 

the experts rated the “resources needed” criterion for indicator 1.9.12 (amount 

of tobacco industry campaign contributions to local and state politicians) as 2.5 out 

of 4. We know, however, that data about this indicator are readily available from 

archival sources, so we lowered the score to 1 out of 4. 

The indicator rating tables include seven indicators that were not rated by the 

experts. Most of those were suggested by the experts themselves, and CDC used

its best judgment to select which expert-proposed indicators to include. These 

indicators are not rated (and noted by an NR), but some information about them 

is provided in the indicator profiles. 

Two criteria used by expert panelists were not included in the final rating tables: 

“uniqueness” and the “summary rating.” “Uniqueness” was only used to determine 

redundant indicators, and we found that the “summary rating” was highly correlated 

with “quality.” 

After extensive analysis and consideration, we also decided not to use the expert 

panelists’ assessment for the “strength of evaluation evidence” criterion because, 

among other reasons, several panelists were concerned that their knowledge of the 

scientific literature on certain areas of tobacco control was limited. Instead, ratings 

for this criterion are based on the findings from an independent literature review 

conducted by the Battelle Centers for Public Health Research and Evaluation under 

contract to CDC. Battelle staff reviewed 847 articles to assess the evidence supporting the use of each indicator to measure a downstream outcome of a tobacco control 

program. 

We evaluated and scored each relevant article or report on the following factors: 

Type of Article

One designation per article as follows:

Research article. Article with new data, generally from a single study. 

Review article. Article with summaries of multiple published studies and 

no original data.

Background article. Article with information relevant to the indicator but 

no evidence of a relationship between the indicator and outcomes.

Score: 

Research article = 0.5

Review article = 1.0

Background article = 0.0

▲ ▲ ▲

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

282
Linkage

The extent of evidence provided in the article for a link between the indicator and 

the expected downstream outcomes in the NTCP goal area logic models. 

Score:

Article shows any evidence of link between the indicator and 

an expected outcome = 1.0

Article shows only evidence against a link between the indicator 

and expected outcome = –1.0

Relevance

The degree to which the article specifically focuses on the indicator.

Score:

Article focuses directly on the indicator = 1.0

Article does not focus directly on the indicator = 0.0

Study Strength

How well the study was designed and how well it showed a link 

between the indicator and outcomes in the NTCP goal area logic models. 

Score:

Article shows strong links between the indicator and an expected outcome = 1.0

Article shows a weak link = 0.5* 

These data were used to calculate the Strength of Evaluation Evidence (SEE) criterion, 

as follows: 

SEE = ∑(T*L*R*S)

where, for each article,

T = article type

L = linkage

R = relevance

S = study strength

The product of T*L*R*S for each article was summed across all articles for each 

indicator. The result was translated into the relative score in the indicator rating 

tables, symbolized as follows:

No data ( ): Indicators for which no studies tested an association between the 

indicator and a downstream outcome in one of NTCP’s goal area logic models.

No support ( ): Indicators for which most studies that tested an association 

between the indicator and outcomes in the logic models found that the association 

was not significant (SEE score = – 0.5–0.0). 

▲ ▲ *An article that showed a weak link was given a value of 0.5 rather than 0 (zero) because a weak link is 

stronger than no link. 

A P P E N D I X B 

283
▲

▲

▲

Minimal support ( ): Indicators for which roughly an equal amount of research 

showed a significant association as showed no association between the indicator 

and downstream logic model outcomes. This category also includes indicators for 

which studies with weak designs supported an association between the indicator 

and an outcome (SEE score: 0.01–0.5).

Moderate support ( ): Indicators for which more research showed a significant 

association between the indicator and a logic model outcome than research 

showing a non-significant association. This category also includes indicators 

for which studies supported an association between the indicator and a downstream outcome in the logic models, but the study designs were not strong 

(SEE score = 0.51–2.5).

Strong support ( ): Indicators for which research showed a strong relationship 

between the indicator and a logic model outcome. Included in this category are all 

long-term indicators because the research supporting these indicators as predictive 

of beneficial health effects is well established (SEE score > 2.5).

We also footnoted indicators that had low reviewer response, low agreement among 

reviewers, or a modified “resources needed” criterion with the following symbols: 

▲

▲

▲

An asterisk (*) indicates low reviewer response: if less than 75% of experts rated 

the indicator or if more than 75% of experts gave a certain criterion an invalid 

rating (e.g., “don’t know”), we considered the indicator to have low reviewer 

response. A low response suggests a high degree of uncertainty among raters. 

An example of such an indicator is 2.3.2: Level of receptivity to media messages 

about secondhand smoke. 

A dagger (†) indicates a low level of agreement among reviewers: if less than 75% 

of the valid ratings were within one point of each other, we considered the rating 

to have a low level of agreement. An example of an indicator with a low level of 

agreement is 1.6.3: Proportion of students who would ever wear or use something 

with a tobacco company name or picture. This low level of agreement represents a 

relatively high degree of variability in the raters’ responses for the criterion. 

A diamond (◊) indicates that the “resources needed” rating for this indicator was 

modified by CDC after the experts provided their ratings for this criterion. An 

example of such an indicator is 1.9.1: Extent and type of retail tobacco advertising 

and promotions. 

Review of this Publication

This publication was peer reviewed internally at CDC and externally by program 

managers of state tobacco control programs and by other experts in the field of 

tobacco control. 

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

284
▲ ▲ ▲ ▲

CDC/OSH Key Indicators Report: Instructions for Expert Panel Reviewers

Purpose

CDC’s Office on Smoking and Health (OSH) is developing a report intended to 

assist state and territorial tobacco control program evaluation efforts under the 

National Tobacco Control Program (NTCP). State Program Managers, State 

Evaluators, OSH staff, and national partners will be the primary audiences 

for the report. The report will aim to accomplish the following functions:

Serve as a companion to OSH’s Best Practices for Comprehensive Tobacco 

Control Programs and Introduction to Program Evaluation for Comprehensive 

Tobacco Control Programs.

Describe key outcome indicators for evaluation of statewide, comprehensive 

tobacco control programs, and suggest appropriate data sources and measures 

for these indicators.

Encourage states to use consistent evaluation measures and comparable data 

sources.

Help OSH determine evaluation criteria for the NTCP, assess Best Practices 

recommendations, and provide consistent surveillance and evaluation technical 

assistance to states.

Methods

Report development began with a critical appraisal of OSH logic models for three 

of the four NTCP goal areas: (1) preventing initiation of tobacco use among youth; 

(2) eliminating nonsmokers’ exposure to secondhand smoke; and (3) promoting 

quitting among youth and adults. The logic models (figures 1, 2, and 3) graphically 

display the links among input, activity, output, and short, intermediate, and longterm outcome components. 

The fourth NTCP goal area—identify and eliminate disparities among population 

groups—will be incorporated through guidance on population-specific data collection methods and measures.

Almost every identified outcome indicator may be tracked for various population 

groups, including groups with high tobacco use prevalence rates or excess tobaccorelated disease morbidity and mortality. In addition, OSH is currently developing 

a logic model specific to this disparities goal. The primary focus is currently on 

identifying appropriate program activities and process measures. 

The indicators are organized by CDC/OSH goal area and logic model component. 

Extensive review of published and fugitive literature identified candidate indicators 

for the outcome components of each logic model. Selection decisions were guided by 

a need to highlight key indicators for evaluation of statewide, comprehensive tobacco 

control programs. Linkages connecting antecedent and consequent indicators were 

reviewed for evidence of association; for example, what is the evidence that implementation of tobacco-free policies in schools is associated with “downstream” outcomes? Each goal indicator list (tables 1, 2, and 3) shows the proposed indicators and 

references to supportive evaluation research. However, the references provided are 

not intended to be a comprehensive bibliography.

A P P E N D I X B 

285
Next, optimal data sources and measures were selected for each indicator. The 

primary criterion used to select measures was whether the data sources are 

readily available to state tobacco control programs. These include the Behavioral 

Risk Factor Surveillance System (BRFSS), CDC Adult and Youth Tobacco Surveys, 

and other similar surveys and surveillance data sources. Where necessary, 

measures were drawn from other national and state-specific surveys and 

evaluation protocols that are not widely used at present but are accessible 

to state tobacco control programs.

Finally, a pilot study was conducted to test the rating process. Refinements in 

the instructions, rating forms, and supportive materials were made in response 

to feedback from pilot study participants. 

Rating Process

The principal purpose of this expert review process is to advise CDC/OSH on which 

of the proposed indicators are considered key for the evaluation of comprehensive 

state tobacco control programs, and what data sources and measures would be most 

useful for tracking these indicators. Reviewers are asked to do the following: ▲ ▲ ▲ ▲

Rate each indicator on a set of criteria.�

Comment on the data sources and measures that have been identified for each �

proposed indicator.�

Suggest alternative data sources and measures.�

Offer additional indicators that may be useful for state tobacco control �

program evaluation. �

Rating Form

Each indicator is presented on a separate rating form in the same order as the 

indicators are listed in tables 1, 2, and 3. The rating forms have three sections: 

▲ ▲ ▲

Summary information on the proposed indicator, including the goal area, logic �

model component, suggested data sources and measures, other relevant informa-�

tion, and a reference regarding the evidence supporting use of the indicator, �

where available.�

Eight rating criteria scales for reviewer response.�

Space for open-ended reviewer comments on the proposed indicator and data �

sources/measures.�

In the summary information section on the rating forms, the data sources/measures 

suggested are intended only to help operationalize the indicators and do not represent a comprehensive list of all possible measures for the indicators. In several 

instances where existing data sources or measures have not been identified, they 

have been labeled generically (e.g., “State Adult Tobacco Survey”) and the measure 

noted as “No question identified.” This suggests that a measure could be added to a 

state-specific survey. For measures involving data collection at levels other than for 

an individual respondent, only the data source is identified (e.g., “Environmental 

scan of tobacco advertising and promotional practices in retail outlets” or “Local 

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

286
level policy tracking system”). Finally, to conserve space, response options for the 

suggested measures have been abbreviated.

Rating Criteria

The following criteria are to be used to rate each indicator:

1. Strength of the evaluation evidence—extent to which you believe that the 

literature supports use of the indicator for the evaluation of comprehensive, 

statewide tobacco control programs, as characterized by the logic models. The 

reference citations included in tables 1, 2, and 3 and on each indicator rating 

form are intended to provide guidance in your ratings on this criterion, but 

your knowledge about other citations should also be used.

2. Data collection and analysis resource needs—your rating of the intensity of 

resource use (cost, time, and effort) required to collect reliable and precise measures, and to analyze appropriately primary or secondary data on the indicator.

In making your judgments, please consider availability of existing data (e.g., 

archival records or other secondary data) and methodology and sampling frame 

issues. Please recognize that, with few exceptions (e.g., California, Massachusetts, 

Florida, Oregon, Texas, and a few others), most state health departments currently 

do not implement comprehensive, statewide evaluations of their tobacco control 

programs.

All states have access to basic prevalence data for adults from the BRFSS, 

periodic data on attitudes and policies through the Current Population Survey 

(CPS) tobacco use supplements, and School Health Education Profile (SHEP). 

CDC synthesizes the available state-level data on many behavioral and policy 

areas in the State Tobacco Activities Tracking and Evaluation System (STATE). 

Beyond these “common denominator” data sources, some states collect additional 

data through youth or adult surveys, policy tracking systems, media tracking 

systems, or other specific data collection methods. The intensity of resource use 

for data collection and analysis will obviously be less for those “common denominator” data sources than for other sources.

3. Utility—extent to which you believe that the indicator would help to answer key 

statewide comprehensive tobacco control program evaluation questions. Although 

these indicators may also be appropriate and useful for community-level evaluation, the utility criterion refers primarily to state efforts.

4. Face validity—your estimation of how valid the indicator would appear to be in 

the eyes of policy makers and decision makers who may be users of tobacco control program evaluation results.

5. Uniqueness—your opinion of whether the indicator contributes distinct information for the evaluation of tobacco control efforts. If you believe that the indicator is 

not unique, please note the redundant indicator in the space provided. [Note: Pilot 

study reviewers suggested that the best way to rate indicators on their uniqueness 

was to review all indicators in a given area once through, and then adjust ratings 

on this criterion as necessary.] 

6. Conformity with accepted practice—your opinion of the degree to which use 

of the indicator is consistent with currently accepted, “real-world” tobacco 

control practice.

A P P E N D I X B 

287
7. Overall quality—a summary rating that reflects your opinion of the overall 

quality of the indicator.

8. Priority rating—your opinion of how essential this indicator is for the evaluation 

of comprehensive, statewide tobacco control programs. [Note: Pilot study participants suggested that this criterion be reviewed again and adjusted once all indicators in an area have been rated.]

Reviewer Comments

In addition to providing comments and suggestions regarding the proposed 

indicator, data sources, and measures in the spaces provided, reviewers are 

encouraged to write notes anywhere on the rating forms or provide additional 

information, references, or other documentation, as necessary.

Product

Expert ratings of the indicators will be taken into account when determining the final 

list of key indicators. The report will also present information on each indicator, as in 

Box 1.

Box 1: Indicator Summary (Sample)

Proposed Indicator: Proportion of youth who report never having tried a cigarette

Goal Area: Preventing Initiation of Tobacco Use Among Youth

Logic Model Component: Long-term—Reduced initiation among youth

Definition: Proportion of respondents under 18 years of age who report that they 

have never tried even one puff of a cigarette.

Purpose:� By employing periodic cross-sectional surveys of youth sampled from 

school or communitywide frames, this indicator may be used to track the 

rate of initiation of cigarette smoking among youth in a given population. 

With sufficient sampling, initiation may be measured with good precision 

in various subpopulation groups to look at gender, age, geographic, and 

ethnic/racial group disparities.

Rationale:� Reduced initiation of tobacco use by youth will lower the youth smoking 

prevalence rate in the population. And, if youth reach adulthood without 

any tobacco use, chances are they will not initiate use as an adult.

Demographic Group:� Youth, under the age of 18 years.

Data Sources/Measures:� CDC Youth Tobacco Survey

Have you ever tried cigarette smoking, even one or two puffs?

Yes

No

Additional Data Needs:� Age, gender, race, ethnicity, city/county of residence. 

Limitations:� None

Other Information:� This indicator may also encompass measurement of other forms of 

tobacco use, such as smokeless tobacco.

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

288
References

1. �MacDonald G, Starr G, Schooley M, Yee SL, Klimowski K, Turner K. Introduction 

to program evaluation for comprehensive tobacco control programs. Atlanta, GA: 

Centers for Disease Control and Prevention; 2001.

2. �SAS Institute, Inc. SAS Language reference: dictionary. Version 8. Cary, NC: SAS 

Institute Inc.; 1999.

A P P E N D I X B 

289
CDC/OSH Tobacco Control Indicator Rating Form�

Proposed Indicator: Proportion of schools/districts with policies that regulate 

display of tobacco industry promotional items (01.06.XX)

Goal Area: Preventing Initiation of Tobacco Use Among Youth (01)

Logic Model Component: Short-term—Changes in school curricula and policies (06)

Data Sources/Measures: CDC SHPPS, State School Policy and Environment (2000)

Has your [school/district] adopted a policy that prohibits 

students from wearing tobacco name-brand apparel or 

carrying merchandise with tobacco company names, 

logos, or cartoon characters in it?

Other Information:� Question modified for use with school and/or district 

samples

Reference: � __________________________________________________

Indicator Ratings

a. Please circle the response number that reflects the extent to which evaluation evidence supports use of the indicator for the associated construct:

No Minimal Moderate Strong Don’t�

Support Support Support Support Know�

1 2 3 4 � 0

b. Please circle the response number that reflects your estimate of the intensity of 

resource utilization required to collect and analyze indicator data adequately:

Low Moderate High Very High Don’t�

Intensity Intensity Intensity Intensity Know�

1 2 3 4 � 0

c. �Please circle the response number that reflects your rating of the utility of the indicator to answer important questions on program effectiveness and impact:

No Low Moderate High Don’t�

Utility Utility Utility Utility Know�

1 2 3 4 � 0

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

290
d. Please circle the response number that reflects your estimation of how face valid 

the indicator would appear to be in the eyes of policy- and decision-makers:

Not at A Little Somewhat Highly Don’t�

All Valid Valid Valid Valid Know�

1 2 3 4 � 0

e. Please circle the response number that reflects your opinion of whether the indicator contributes unique information for tobacco control evaluation efforts:

Unique Not If “Not Unique” write Don’t

Unique the number(s) of the Know

redundant indicator(s):

1 2 � 0

f. �Please circle the response number that reflects your opinion of the degree to which 

use of the indicator is consistent with currently accepted, “real-world” tobacco 

control practice:

Not at all A Little Somewhat Highly Don’t�

Consistent Consistent Consistent Consistent Know�

1 2 3 4 � 0

g. Please circle the response number that reflects your view of the overall quality 

of the indicator:

Low � High

1 2 3 4 � 5

h. Please circle the response number that reflects your summary rating of how 

essential this indicator is for the evaluation of comprehensive state tobacco control 

programs:

Not Essential Optional Essential

1 � 2 3

Reviewer Comments

a. Please provide any additional comments on your ratings of this indicator: 

b. If you feel there is a better indicator of this logic model construct, 

please specify here:

A P P E N D I X B 

291
c. �Please provide comments on the proposed data sources/measures for 

this indicator:

d. If you feel there are better data sources/measures, please specify here:

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

292
A P P E N D I X C�

Expert Panel Members�

We thank the following panel of experts members (in alphabetical order) who rated 

the indicators. Without their generosity in sharing their expertise and donating their 

time, this publication would not have been possible. 

Ursula Bauer, Ph.D., M.P.H. 

Director

Tobacco Control Program

New York State Department of Health

Carolyn Celebucki, Ph.D.

Associate Professor

Department of Psychology

University of Rhode Island, Providence Campus

Thomas Chapel, M.A., M.B.A.

Senior Health Scientist

Office of Program Planning & Evaluation

Centers for Disease Control and Prevention

K. Michael Cummings, M.P.H., Ph.D.

Chairman

Department of Behavioral Epidemiology

Division of Cancer Prevention and Population Sciences

Roswell Park Cancer Institute

Cristine Delnevo, Ph.D., M.P.H.�

Associate Professor �

School of Public Health�

University of Medicine and Dentistry of New Jersey �

Matthew Farrelly, Ph.D.�

Senior Program Director�

Tobacco Use Research Program�

Research Triangle Institute�

Ellen Feighery, R.N., M.S.�

Program Director�

Public Health Institute�

Gary Giovino, Ph.D., M.S.�

Senior Research Scientist�

Tobacco Control Research Program�

Department of Health Behavior�

Division of Cancer Prevention and Population Sciences�

Roswell Park Cancer Institute�

A P P E N D I X C 

293
Stanton Glantz, Ph.D.�

Professor of Medicine (Cardiology)�

University of California at San Francisco�

Nell Gottlieb, Ph.D.�

Professor�

Department of Kinesiology and Health Education�

University of Texas at Austin�

Douglas Luke, Ph.D.�

Associate Professor of Community Health�

School of Public Health�

Saint Louis University�

Danny McGoldrick, Ph.D.�

Director of Research�

Campaign for Tobacco-Free Kids�

Jesse Nodora, Dr.P.H.�

Evaluation Administrator�

Tobacco Education and Prevention�

Arizona Department of Health Services�

John Pierce, Ph.D.�

Sam M. Walton Professor for Cancer Research �

Department of Family and Preventive Medicine �

Associate Director, Cancer Prevention and Control Program�

University of California at San Diego�

April Roeseler, M.S.P.H.�

Chief�

Local Programs and Evaluation�

Tobacco Control Section�

California Department of Health Services�

Mike Stark, Ph.D.�

Evaluation Manager�

Office of Disease Prevention and Epidemiology�

Oregon Department of Human Services and Multnomah County Health Department�

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

294
A P P E N D I X D�

Data Source Indicator Table�

The following table cross-references example data sources and indicators in this 

publication. The example data sources do not represent all data sources available. 

When possible, Web addresses are provided. For additional information on tobaccorelated data sources and data collection methods, refer to The Introduction to Program 

Evaluation for Comprehensive Tobacco Control Programs or Surveillance and Evaluation 

Data Resources for Comprehensive Tobacco Control Programs 1,2.

Data source Indicator numbers For more information�

Addressing Tobacco in Managed 3.7.5; 3.9.1; 3.9.8; http://www.aahp.org/atmc/mainindex.cfm Care (ATMC), Survey of Health 3.10.1

Plans, 1997–1998

Adult Tobacco Survey (ATS): 2.3.5; 2.3.6; 2.3.7; State health departments�

CDC Recommended Questions: 2.4.2; 2.4.3; 2.4.4; Office on Smoking and Health,�

Core, 2003 2.6.1; 2.6.4; 2.7.3; Centers for Disease Control and Prevention, �

2.8.2; 2.8.3; 3.8.3; (770) 488–5703�

3.9.2; 3.9.3; 3.9.5; �

3.11.1; 3.11.3; �

3.13.1; 3.13.2NR; �

3.14.1

▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲ ▲

▲

▲

▲

▲

Adult Tobacco Survey (ATS): 3.7.4; 3.8.6; 3.9.2; �

CDC Recommended Questions: 3.9.3�

Supplemental Section C: �

Cessation, 2003�

State health departments�

Office on Smoking and Health, �

Centers for Disease Control and Prevention, �

(770) 488–5703 �

Adult Tobacco Survey (ATS): 2.3.4; 2.3.7

CDC Recommended Questions: 

Supplemental Section D: Environmental Tobacco Smoke, 2003

State health departments�

Office on Smoking and Health, �

Centers for Disease Control and Prevention, �

(770) 488–5703�

Adult Tobacco Survey (ATS): 1.6.4; 1.6.5; 1.6.7NR;

CDC Recommended Questions: 2.3.10NR; 3.8.5

Supplemental Section F: Policy 

Issues, 2003

State health departments�

Office on Smoking and Health, �

Centers for Disease Control and Prevention, �

(770) 488–5703�

Adult Tobacco Survey (ATS): 1.10.4

CDC Recommended Questions: 

Supplemental Section G: Parental 

Involvement, 2003

State health departments�

Office on Smoking and Health, �

Centers for Disease Control and Prevention, �

(770) 488–5703�

American Lung Association’s State 1.8.1 http://slati.lungusa.org Legislated Actions on Tobacco Issues See “Policy tracking” (SLATI)

A P P E N D I X D 

295
2000

Data source Indicator numbers For more information�

American Smoking and Health 3.8.7; 3.9.5

Survey (ASHES), 2003

Americans for Nonsmokers’ Rights 1.8.1; 1.8.2; 1.8.3; �

(ANR) 1.8.4; 2.4.1

Arizona Workplace Survey 2.4.2

Behavioral Risk Factor Surveillance 3.11.1; 3.13.2NR

System (BRFSS), 2002

Behavioral Risk Factor Surveillance 2.8.3; 3.14.1

System (BRFSS), 2003

Behavioral Risk Factor Surveillance 1.6.7NR; 2.3.7;

System (BRFSS): Tobacco Use 2.3.10NR

Prevention Module, 2000

Behavioral Risk Factor Surveillance 3.13.1

System (BRFSS): Tobacco Use 

Prevention Module, 2002

Birth certificate data 3.14.2

California Adult Tobacco Survey 2.3.4; 2.7.1; 2.7.2 �

(CATS), 1999

California Independent Evaluation: 2.3.9

Adult Survey, 1997

California Independent Evaluation: 2.7.5

Adult Survey, 2000

California Independent Evaluation: 2.5.1; 2.5.2; 2.5.3�

Policy Enforcement Survey: Exposure 

to Environmental Tobacco Smoke,

California Independent Evaluation: 1.8.5; 1.8.6

Policy Enforcement Survey: Youth 

Access to Tobacco, 2000

California Independent Evaluation: 1.6.8NR; 1.7.9;

Youth Survey, 2000 1.7.10; 2.6.5

California Tobacco Industry 1.9.5; 1.9.10

Monitoring Evaluation: 

Project SMART Money

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

http://tobacco.rti.org/data/New/surveys.cfm �

http://www.no-smoke.org

See “Policy tracking”

http://www.tepp.org/evaluation

See “Worksite survey”

http://www.cdc.gov/brfss

http://www.cdc.gov/brfss

http://www.cdc.gov/brfss

http://www.cdc.gov/brfss

State vital statistics and records

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/�

html/Evaluation_Resources.htm

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/�

html/Evaluation_Resources.htm

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/�

html/Evaluation_Resources.htm

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/�

html/Evaluation_Resources.htm

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/�

html/Evaluation_Resources.htm

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/�

html/Evaluation_Resources.htm

http://www.ttac.org/enews/

mailer09-30-03full.html

See “Event sponsorship tracking system” 

and “Tobacco industry monitoring system”

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

296
Data source

California Tobacco Use Prevention 

Education Evaluation: District

Indicator numbers

1.7.4

For more information

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/ ▲

Coordinator Survey, 2003

California Tobacco Use Prevention

Education Evaluation: Teacher Survey,

2003

1.7.2; 1.7.4; 1.7.5

html/Evaluation_Resources.htm

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/

html/Evaluation_Resources.htm

▲

California Youth Tobacco Survey 

(CA YTS), 1999

California’s BREATH (Smoke-Free 

Bars, Workplaces, and Communities

1.11.6NR

2.6.3

▲

▲

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/

html/Evaluation_Resources.htm

http://www.breath-ala.org

Program)

Campaign for Tobacco-Free Kids 

(CTFK)

CDC Pregnancy Risk Assessment 

Monitoring System (PRAMS), Phase 4,

1.12.1; 3.12.1

3.9.7; 3.14.2; 3.14.3

▲

▲

http://www.tobaccofreekids.org

http://www.cdc.gov/reproductivehealth

2000–2003

CDC School Health Profiles: Lead 1.7.2; 1.7.3; 1.7.4; ▲

Division of Adolescent and School Health,

Health Education Teacher Question- 1.7.5 Centers for Disease Control and Prevention,

naire (Profiles), 2002 (888) 231–6405

▲

http://www.cdc.gov/HealthyYouth/index.htm

State health departments

CDC School Health Profiles: School 1.7.1; 1.7.6; 1.7.11; ▲

Division of Adolescent and School Health,

Principal Questionnaire (Profiles), 1.9.7; 2.4.5 Centers for Disease Control and Prevention,

2002 (888) 231–6405

▲

http://www.cdc.gov/HealthyYouth/index.htm

State health departments

CDC State Tobacco Activities Tracking 

and Evaluation (STATE) system

1.8.7; 1.12.1; 2.4.6; 

2.8.1; 2.8.2; 3.12.1; 

3.14.4

▲

http://www.cdc.gov/tobacco/STATESystem

▲

▲

▲

▲

CDC Youth Risk Behavior 1.7.10; 1.11.2; http://www.cdc.gov/nccdphp/dash/yrbs/ Surveillance System (YRBSS), 2003 1.11.4; 1.11.5; index.htm

1.13.1; 1.13.2; 

1.14.1; 1.14.2; 

2.6.5; 2.8.2; 2.8.3; 

3.11.2; 3.14.1

Center for Responsive Politics (CRP) 1.9.11; 1.9.12 http://www.opensecrets.org�

Current Population Survey: Tobacco 2.4.2; 2.4.3; 3.11.1 http://www.riskfactor.cancer.gov/studies/ Use Supplement (CPS TUS), 2003 tus-cps

http://www.census.gov/apsd/techdoc/cps/

cps-main.html

A P P E N D I X D 

297
Data source Indicator numbers For more information�

Decision Maker or Opinion 3.8.8

Leader Survey

Direct observation of employees’ 2.6.3

and patrons’ behavior

Enforcement Agency Survey 1.8.5; 1.8.6; 2.5.1; �

2.5.2; 2.5.3

Environmental scan of tobacco 1.9.1; 1.9.3; 1.9.7�

advertising and promotional 

practices in retail outlets

Event sponsorship tracking system 1.9.5

Federal Election Commission (FEC) 1.9.12

Legacy Media Tracking Survey 1.6.1; 1.6.2; 2.3.1;

(LMTS), 2003 2.3.2; 3.8.1; 3.8.2

Media Tracking Service 1.9.8; 1.9.9

National Social Climate Survey 2.3.3

of Tobacco Control, 2001

Operation Storefront: Youth 1.9.1; 1.9.3; 1.9.7�

Against Tobacco Advertising 

and Promotion Initiative

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

▲

State Decision Maker Tobacco Survey �

(California Independent Evaluation, Opinion 

Leader Survey), 1997 

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/

html/Evaluation_Resources.htm

http://www.breath-ala.org�

See “California’s BREATH (Smoke-Free Bars, 

Workplaces, and Communities Program)”

California Independent Evaluation: �

Policy Enforcement Survey, Youth Access 

to Tobacco, 2000

Operation Storefront: Youth Against Tobacco �

Advertising and Promotion Initiative 

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/

html/Evaluation_Resources.htm

Project SMART Money

http://www.ttac.org/enews/

mailer09-30-03full.html#LinkF

Rosenberg NJ, Siegel M. Use of corporate 

sponsorship as a tobacco marketing tool: a 

review of tobacco industry sponsorship in the 

USA, 1995–99. Tob Control. 2001; 10(3):239–46

http://www.fec.gov

See “Public records of political 

contributions”

http://tobacco.rti.org/data/lmts.cfm�

See “TNS Media Intelligence Competitive �

Media Reporting (CMR)”�

Stillman FA, Cronin KA, Evans WD, �

Ulasevich A. Can media advocacy influence �

newspaper coverage of tobacco: measuring �

the effectiveness of the American Stop �

Smoking Intervention Study’s (ASSIST) �

media advocacy strategies. Tob Control. �

2001;10(2):137–44.�

http://www.ssrc.msstate.edu/socialclimate

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/

html/Evaluation_Resources.htm

See “Environmental scan”

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

298
Data source

Partnership for Prevention, Tobacco

Indicator numbers

3.7.6

For more information

http://www.mercerhr.com ▲

Survey: National Survey of Employersponsored Health Plans, 2002

Policy tracking system

Public records of political 

contributions

Quitline call monitoring

1.8.1; 1.8.2; 1.8.3; 

1.8.4; 1.9.2; 1.9.4; 

1.9.6; 2.4.1

1.9.11; 1.9.12

3.7.1; 3.7.2NR; 3.7.3

Americans for Nonsmokers’ Rights 

http://www.no-smoke.org

State Legislated Actions on Tobacco Issues 

(SLATI) online database

http://slati.lungusa.org

▲

▲

▲

▲

▲

▲

Collected by the Office of State Secretary 

or equivalent at local level in each state

See “Federal Election Commission (FEC)”

Givel MS, Glantz SA. Tobacco lobby political 

influence on US state legislatures in the 1990s. 

Tob Control. 2001; 10 (2):124–34.

Miller CL, Wakefield M, Roberts L. Uptake 

and effectiveness of the Australian telephone

State departments of revenue

Substance Abuse and Mental Health 

Services Administration (SAMHSA) 

Compliance Checks

TNS Media Intelligence Competitive 

Media Reporting (CMR)

Tobacco industry fiscal reports

1.12.1; 2.8.1; 

3.12.1; 3.14.4

1.11.1

1.9.8

1.9.11

▲

▲

▲

▲

▲

quitline service in the context of a mass media 

campaign. Tob Control. 2003; 12 (Suppl 2):

ii53–8.

State tax sales data, tobacco product excise 

taxes

http://prevention.samhsa.gov/tobacco/

guidance.asp

http://www.tnsmi-cmr.com/products/

index.html

See “Media Tracking Service”

http://www.altria.com/investors/

Tobacco industry monitoring system

University of California at San Diego, 

California Tobacco Survey (CTS): 

Adult Attitudes and Practices, 1996

Worksite Survey

1.9.10

1.6.7NR; 2.3.8; 

2.3.10NR

2.4.2

▲

▲

▲

▲

▲

02_01_annualreport.asp

http://www.reynoldsamerican.com/

Investors/sharedocs_cover.asp

See “California Tobacco Industry Monitoring 

Evaluation: Project SMART Money”

http://ssdc.ucsd.edu/tobacco

http://www.dhs.ca.gov/ps/cdic/ccb/TCS/

html/Evaluation_Resources.htm

See “Arizona Workplace Survey”

A P P E N D I X D 

299
Data source Indicator numbers For more information

Youth Tobacco Survey (YTS): CDC 

Recommended Questions: Core, 2004

1.6.3; 1.7.8; 1.7.9; 

1.7.10; 1.10.1; 

1.10.2; 1.10.3; 

1.10.5; 1.11.2; 

1.11.3; 1.11.4; 

1.11.5; 1.13.1; 

1.13.2; 1.14.1; 

1.14.2; 2.3.5; 2.6.5; 

2.7.3; 2.7.5; 2.8.2; 

2.8.3; 3.8.3; 3.11.2; 

3.13.1; 3.13.2NR; 

3.14.1

▲

▲

State health departments

Office on Smoking and Health, 

Centers for Disease Control and Prevention, 

(770) 488–5703

Youth Tobacco Survey (YTS): 

Supplemental Questions, 

2004

3.11.3 ▲

Office on Smoking and Health, 

Centers for Disease Control and Prevention, 

(770) 488–5703

References

1. �MacDonald G, Starr G, Schooley M, Yee SL, Klimowski K, Turner K. Introduction to program 

evaluation for comprehensive tobacco control programs. Atlanta, GA: Centers for Disease Control and 

Prevention; 2001.

2.�Yee SL, Schooley M. Surveillance and evaluation data resources for comprehensive tobacco control programs.

Atlanta, GA: Centers for Disease Control and Prevention; 2001.

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

300
Glossary�

Activities

The events or actions that are part of a tobacco control program.

Attitudes

Biases, inclinations, or tendencies that influence a person’s response to situations, �

activities, other people, or program goals.�

Awareness

The extent to which people in the target population know about an event, activity, 

or campaign.

Capacity

The resources (e.g., staff, data-collection systems, funds) needed to conduct a �

tobacco control program or to evaluate such a program.�

CDC

Centers for Disease Control and Prevention.

Cognitive-behavioral interventions

Activities based on the premise that people can learn new behaviors to use in 

response to stimuli and that the thought processes that serve as intermediate steps 

between stimuli and behaviors can be altered, thereby influencing behavior. Basic 

applications of this theory for tobacco-use cessation are:

Establishing self-awareness of tobacco use.

Providing the motivation to quit.

Preparing to quit.

Providing strategies to maintain abstinence.

Consumption

The number of tax-paid cigarettes (pack of 20) purchased by consumers in a �

particular calendar year.�

Data

Documented information or evidence.

Data sources

Surveys or surveillance systems used to gather data.

Evaluation

The process of determining whether programs—or certain aspects of programs—�

are appropriate, adequate, effective, or efficient and, if not, how to make them so.�

Ever-smoker

A person who gives a positive answer to the question “Have you tried cigarette �

smoking, even one or two puffs?”�

Example data source

Surveys or surveillance systems used to measure an indicator and the population �

on which the data are needed.�

Face validity

The degree to which data on an indicator appear reliable to stakeholders and �

policy makers.�

G L O S S A R Y 

▲ ▲ ▲ ▲

301
FDA

U.S. Food and Drug Administration.

Goal area

One of the four components of the overall goal of CDC’s National Tobacco �

Control Program.�

HHS

U.S. Department of Health and Human Services.

Implementation

Carrying out or putting into effect a plan or program.

Indicator

An observable and measurable characteristic or change that shows the progress a 

program is making toward achieving a specified outcome.

Indicator profile

The term used in this manual for a table with detailed information on one indicator listed in this publication (see page 29 for an example).

Indicator rating table

The term used in this publication for the list of the indicators associated with one 

outcome in one NTCP logic model. The experts’ rating for each indicator is also 

included (see page 28 for an example).

Inputs

Resources used to plan and set up a tobacco control program.

Intervention

The method, device, or process used to prevent an undesirable outcome or create 

a desirable outcome.

Logic model

A graphic depiction of the presumed causal pathways that connect program 

inputs, activities, outputs, and outcomes.

Media messages

Anti-tobacco information provided to the public through various media�

 (e.g., television, radio, billboards).�

Minors

Persons younger than 18 years of age.

Morbidity

Disease or disease rate.

NCI

National Cancer Institute.

Never-smoker

A person who gives a negative answer to the question “Have you tried cigarette 

smoking, even one or two puffs?”

NIH

National Institutes of Health.

NTCP

National Tobacco Control Program.

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

302
Observation

A method of collecting data that does not involve any communication with the �

subjects being studied. The investigators merely watch for particular behaviors �

and record what they see.�

Opinion leader survey

Collection of information (data) from leaders in the community.

Outcome

The results of an activity such as a countermarketing campaign or an effort to �

reduce nonsmokers’ exposure to smoke. Outcomes can be short-term, intermedi-�

ate, or long-term.�

Outcome components

The term used in this publication for the short-term, intermediate, and long-term �

results described in the NTCP logic models for the first three goal areas. These are �

the results expected if tobacco control programs provide the needed inputs and �

engage in the recommended activities also described in the logic models.�

Outcome evaluation

The systematic collection of information to assess the effect of a program or an �

activity within such a program to reduce the adverse health effects of tobacco �

use. Good evaluation allows evaluators to draw conclusions about the merit �

of a program and make recommendations about the program’s direction.�

Outcome overview

The term used in this publication for the summary of the scientific evidence in 

support of the assumption that achieving an outcome on an NTCP logic model 

affects all concurrent and later activities and outcomes (see page 25 for an example).

Outputs

The direct products of a program (e.g., the materials needed for a media campaign).

Payers

Health insurance organizations that reimburse providers for services when coverage is purchased by companies, government agencies, or other consortia. Also selfinsured companies, government agencies, or other consortia that purchase health 

care benefits for a group of individuals and use an insurer as a fiscal intermediary 

to process claims and reimburse for services.

Population group

Individuals from which data about a given indicator can most commonly be �

collected.�

Preemption

Federal or state legislation that prevents states or local jurisdictions from enacting 

tobacco control laws more stringent than or otherwise different from the federal or 

state law.

Prevalence

The amount of a factor of interest (e.g., tobacco use, awareness of a media �

campaign) present in a specified population at a specified time.�

Process evaluation

Systematic collection of information to determine how well a program is set up �

and operating.�

G L O S S A R Y 

303
Program evaluation

Systematic collection of information about activities, characteristics, and outcomes 

of programs, used to make judgments about a program, improve its effectiveness, 

or inform decisions about future program activities.

Purchaser

Purchasers include companies, government agencies, or other consortia that �

purchase health care benefits for a group of individuals.�

Rate

A measurement of how frequently an event occurs in a certain population at �

one point in time or during a particular period of time.�

Reach

The number of people or households that receive a program’s message or �

intervention.�

Recent successful quit attempts

Proportion of former smokers who have quit in the previous 12 months.

Resources

Assets available or expected to be available for program operations. Resources 

include people, equipment, facilities, and other items used to plan, implement, 

and evaluate public health programs whether or not they are paid for directly 

with public funds.

Self service tobacco sales

Sales that allow customers to handle tobacco products before purchasing them.

Social source

A person or location from which tobacco products are obtained other than a �

tobacco product retailer.�

Some-day smoker

A current smoker who gives a “smoked on some days” response.

Surveillance

The ongoing, systematic collection, analysis, and interpretation of data about a 

hazard, risk factor, exposure, or health event.

Survey

A quantitative method of collecting information on a target population at one point 

in time. Surveys can be conducted by interview (in person or by telephone) or by 

questionnaire.

Susceptibility

The intention to smoke or the absence of a strong intention not to smoke.

Sustained abstinence

Complete cessation of tobacco use for 6 months or longer.

Theory of change

Intellectual framework for understanding the process of behavior change.

Utility

The extent to which evaluation produces reports that are disseminated to relevant 

audiences, that inform program decisions, and that have a beneficial effect.

K E Y O U T C O M E I N D I C A T O R S for Evaluating Comprehensive Tobacco Control Programs

304
Outcome 7

and Programs in Schools

Number Indicator Overall quality

1.7.4 | | | | | | $$

Indicator number

1.7.4

Outcome Indicator

component within

better

Indicator Rating

Resources

needed

Strength of

evaluation evidence

Utility

Face validity

Accepted

practice

low high

Overall quality:

Resources needed:

Utility:

such as policy makers.

Accepted practice:

consistent with accepted practice.

Increase in Anti-tobacco Policies

Proportion of schools or school districts that provide 

program-specific training for teachers

Goal area

the goal area

The general worth of the indicator as it relates to evaluating tobacco control programs.

Dollar signs show the amount of resources (funds, time, and effort) needed to collect and 

analyze data on the indicator using the most commonly available data source: the more dollar signs (maximum 

four), the more resources needed. The dollar signs do not represent specific amounts because the actual cost of 

measuring and analyzing an indicator varies according to the existing capacity of a state health department or 

organization to evaluate its programs.

Strength of evaluation evidence: The degree to which scientific evidence supports that implementing interventions 

to affect change in a given indicator (e.g., proportion of schools or school districts that provide program-specific 

training for teachers) will lead to a measurable downstream outcome (e.g., reduced susceptibility to experimentation with tobacco products). 

 The extent to which the indicator is useful for answering evaluation questions for comprehensive state 

tobacco control programs. 

Face validity: The degree to which data on the indicator would appear valid to tobacco program stakeholders, 

 The degree to which using the indicator to measure a tobacco control program’s progress is

How to Use the Rating Tables