Program Sustainability Action Planning Training Project
Phase: NA
Conditions: Program Sustainability, Tobacco Use
Sponsor: Washington University School of Medicine

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
1 
 
 
 
 
Date: January 22, 2018 
 
Protocol Number: 201801196 
 
Title:  Program Sustainability Action Planning Training Project 
 
Principal Investigator: Sarah Moreland-Russell, PhD, MPH 
George Warren Brown School of Social Work 
Division of Public Health Sciences and Siteman Cancer Center,  
Washington University School of Medicine 
Washington University in St. Louis 
One Brookings Drive, Campus Box 1196 
St. Louis, MO 63130 
(tel.) 314-935-8382 
(email) smoreland-russell@wustl.edu 
 
Co-Investigators: Ross Brownson, PhD, Douglas Luke, PhD 
 
Statistician: Todd Combs, PhD 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
2 
 
1.0 Introduction – Background and Rationale 
Background 
The emergence of dissemination and implementation (D&I) science has driven a rapid increase in 
studies of how new scientific discoveries are translated and developed into evidence-based 
programs and policies. However, D&I science has paid much less attention to what happens to 
programs once they have been implemented. Public health programs can only deliver benefits if 
they reach maturity and sustain activities over time. In order to achieve the full benefit of 
significant investment in public health research and program development, we need to understand 
the factors related to sustainability and develop tools and trainings that support strategic long-term 
program sustainability. The goal of this proposed study is to increase the capacity for sustainability 
among evidence-based tobacco control (TC) programs.  
 
Rationale  
Sustainability as an important next step in dissemination and implementation (D&I) 
research.  
Demonstrating that an intervention is effectively implemented, (the initial process of embedding 
interventions within settings12), is only the first step in impacting the health and wellness of a target 
population. In order for the population to reap the benefits of an implemented evidence-based 
intervention, it must be sustained over time. In recognition of this, several recent D&I papers have 
called explicitly for sustainability research as an essential next step in the field.10,11,13–21 Program 
sustainability—the extent that interventions can continue to be delivered over time and 
institutionalized within settings12—is a complex process, often fraught with challenges.16,21–23 
Research consistently indicates that even effectively implemented interventions risk failure when 
funding ends10,20,24 or when planning and training stops.25 In fact, it is estimated that up to 40% of 
programs fail within two years of losing funding.26 Implementing a program and then failing to 
sustain it can have disastrous consequences for the community though loss of trust and waste of 
valuable community resources.27 Though there is growing interest in the study of program 
sustainability, there is little agreement on how to best sustain initiatives over time.23,28 This is due, 
in large part, to the historical lack of sustainability assessment tools that are both reliable and 
widely disseminated16,29 and an essential absence of an evidence-based action planning process for 
sustainability.  
 
Importance of sustaining evidence-based TC for public health.  
As of 2013, 42.1 million– or 1 in 5— adults in the US smoke, which leads to an estimated 480,000 
preventable deaths per year due to tobacco use. Approximately 61,000 of those deaths occur from 
secondhand smoke exposure. According to the Center for Disease Control and Promotion (CDC), 
if smoking continues at its current rate, more than 5 million of today’s youth will die prematurely, 
and the economic cost of smoking will rise to more than $300 billion per year.30  
Tobacco use has long been identified as a major preventable cause of death and disease in the 
US. There is also little debate that TC expenditure directly impacts tobacco use rates in the US. 
Between 1985 and 2003, US adult smoking prevalence declined from 29.5% to 18.6%. This sharp 
decline directly correlated with state TC program funding rates.1 Farrelly et al. estimates, “If, 
starting in 1995, all states had funded their TC programs at the minimum or optimal levels 
recommended by the CDC, there would have been 2.2 million to 7.1 million fewer smokers by 
2003.”1 Given the established evidence for the state TC model and the work left to do in this field, 
it is vital that these programs continue in all 50 states, both to improve quality of life and reduce 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
3 
 
the massive healthcare costs incurred by smoking-related illness.1 Failing to sustain evidence-
based TC initiatives will likely lead to stagnation in smoking cessation rates and an increase in 
new smokers.1 
 
The need for evidence-based sustainability training and technical assistance. Empirical 
evidence has established that program sustainability can be improved through training and 
technical assistance;11,31–33 however, to date, no evidence-based program sustainability training 
curricula exists. Thus, systematic methods are needed to empirically develop, test, and disseminate 
sustainability training to improve institutionalization of evidence-based programs. While there is 
a growing body of research on the factors affecting sustainability,11,15,20–23,34 virtually no empirical 
work has been done to translate the components of program sustainability capacity into practical 
guides and tools for practitioner utilization.11,28 Our work with over 50 public health programs 
indicates that managers, evaluators, and practitioners need assistance in using sustainability 
assessment results to inform sustainability planning priorities, along with a clear process for 
completing a written sustainability plan that is easy to implement.  
 
Importance of utilizing an action-oriented approach. Literature indicates that the best practice 
for impacting long-term behavior and institutional change is a hands-on, action-oriented, in-
person35–37 training program. A study that involved action planning training with coalition groups 
working on substance abuse initiatives in Tennessee found positive changes in sustainability 
readiness (increased infrastructure capacity) post training,11 and a meta-review of health behavior 
change outcomes found that intervention amount, duration, and penetration all impact level of 
behavior change.38,39  
Research also highlights the importance of creating an action plan to move sustainability 
progress forward. Creating a sustainability plan has predicted both program survival and post-
launch funding,40 and obtaining future funding facilitated the institutionalization process of a new 
intervention.41 Additionally, plan specificity and attitude toward intervention predicted plan 
adherence,42 and perceived self-efficacy, policy support, and level of institutionalization predicted 
sustainment of plan initiatives.25 
 
Innovation 
This study makes important contributions to D&I science and is innovative because it:       
1. Develops the first ever evidence-based Program Sustainability Action Planning Model and 
Training Curriculum. By establishing an evidence-based method for action planning and technical 
assistance surrounding program sustainability, we are supporting state TC programs and other 
evidence-based public health initiatives to sustain their positive impact in a tumultuous funding 
climate. In addition, this work will advance the field of study of action planning and technical 
assistance in general which can contribute to aspects of the D&I field beyond sustainability. 
2. Links closely with practice at an opportune time. As part of the DP15-1509, National State-
Based TC Program Funding Announcement set forth by the CDC, all 2015 awarded applicants 
(includes all states and territories) are “required to develop a sustainability plan to ensure 
sustainability and maintaining a state based TC program” and “provide measures of execution of 
all activities as outlined in the sustainability plan.”2 Our development of the Program Sustainability 
Action Planning Training and broad dissemination will assist states in fulfilling this requirement 
and establishing their capacity to continue sustainability planning over time. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
4 
 
3. Further advances D&I science related to establishing validated measures for 
sustainability. The study will also provide further clarity on the challenges, benchmarks, and 
programmatic factors that contribute to sustainability (versus implementation) of an intervention. 
We will also use the data collected to explore the predictive validity of the Program Sustainability 
Assessment Tool, which will enable programs to better plan for and improve the sustainability of 
effective programs. 
4. Provides broad dissemination of training materials and results. The curriculum and lessons 
from the training program will be systematically disseminated (at no cost) to several key audiences 
(e.g., trainers in federal and state agencies, staff in public health agencies). Since our training 
program focuses on capacity for sustainability, our methods and approaches will be applicable to 
numerous other public health issues. 
 
 
1.0  Objectives 
 
The overall goal of this proposed study is to increase the capacity for sustainability among 
evidence-based tobacco control (TC) programs. Although all 50 states have implemented 
evidence-based TC programs and policies, states vary in their abilities to support and sustain these 
programs over time. Most states still do not meet the CDC recommended level for funding their 
TC program allowing for tobacco use to remain the leading cause of preventable disease and death 
in the US.1 Also, as part of the 2015 National State-Based Tobacco Control Program Funding 
Announcement released by the CDC,2 all awarded state applicants are “required to develop a 
sustainability plan” and “provide measures of execution of all activities as outlined in the 
sustainability plan”. There are currently no available resources specifically designed to help states 
meet this requirement making this project imperative for states to effectively meet this important 
funding requisite. 
 Phase 1 (Aim 1) focuses on refining and finalizing our evidence-based Program 
Sustainability Action Planning Model and Training Curriculum and deliver to 12 states. 
The Program Sustainability Action Planning Training will incorporate experiential 
learning methods3–6 and includes: action planning workshops, development of action plans 
with measurable objectives to foster institutional changes, and technical assistance.   
 Phase 2 (Aim 2) will use a randomized effectiveness trial to assess the Program 
Sustainability Action Planning Training in 24 states (12 intervention, 12 comparison). 
Evaluation of our training program is centered on the theory of change.7,8  
 Phase 3 (Aim 3) focuses on responding to study results by making any needed revisions 
and conduct widespread dissemination of our Program Sustainability Action Planning 
Model and Training Curriculum starting with the 12 comparison states. 
 
 
3.0 Selection of Participants – Eligibility Criteria 
 
Recruitment. A total of 24 state TC programs will be recruited to participate in this study. We 
believe this number is feasible for three main reasons: (1) this training will be in high demand 
because of the CDC’s requirement for all state TC programs to complete a sustainability plan; (2) 
our strong relationships and past success in evaluating state level TC programs; (3) our project 
consultant’s 20+ years working directly with state level tobacco programs (Eischen).  

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
5 
 
State selection. We will recruit 24 states, and then randomly assign each state into intervention and 
comparison conditions. To ensure that both the intervention and comparison groups are equally 
diverse in terms of organizational factors relevant to tobacco control, we will stratify states using 
the following three criteria: (1) tobacco control policy progress, (2) smoking rates, and (3) funding 
for TC program. Tobacco control policy progress is operationalized as the American Lung 
Association’s (ALA) 2015 Smoke-free Score for each state.88–90 The ALA “grades” each state 
annually according to the comprehensiveness of its smoke-free policies (higher scores indicate 
more comprehensive smoke-free policies).  2013 adult smoking rates are used for smoking rates90 
and funding is measured as the actual amount spent on tobacco control as a percentage of the CDC-
recommended amount of spending for FY2015.88,91 Together, these three indicators characterize 
states’ needs (smoking rates), inputs (funding), and environments (policy). States will also be 
chosen based on willingness to participate and receipt of prior sustainability training. Figure 3 is 
divided into quadrants by the median smoking rate and median Smoke-free Score. The size of the 
circles within each quadrant represents the varying levels of funding. The quadrants delineate the 
primary selection guidelines. Six states from each quadrant representing varying levels of funding 
will be randomly assigned to either the intervention or comparison group. This process will guard 
against selecting a disproportionate number of well- or poorly-funded programs for each group. 
Furthermore, it will ensure both the intervention and comparison groups are evenly varied in terms 
of needs and policy environments. All participants in this study are aged 18 years or older. States 
agreeing to participate in the study will learn of their group assignment after completing the PSAT. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Figure 3. Selection criteria for intervention 
and comparison groups 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
6 
 
4.0 Participant Registration 
 
We will work with the Office on Smoking and Health (OSH) at the CDC to identify a list of state 
TC program managers to contact during study recruitment/registration. We will contact all 
identified program managers and invite them to participate in the study, as well as provide project 
fact sheets, outlining benefits and responsibilities of being a part of the study. Once we divide 
states into intervention or comparison group, we will work with each state program manager to 
identify a sustainability workgroup consisting of 5-12 members actively involved in TC initiatives 
in the state. In our experience, we have found that the desired organizational outcome 
(institutionalization) is best achieved if an integrated workgroup comprised of internal state health 
department staff (i.e., the program coordinator and TC program director); external stakeholders 
(i.e., TC coalition members or national TC advocacy groups); and policy makers (especially if 
there is a TC champion in the state) complete the PSAT and actively participate throughout the 6-
step action planning process. Therefore, the composition of each state program’s sustainability 
workgroup will not be randomized.  
Participants receive a consent letter (Appendix D, attached) that details participation in the 
study. Participants are able to review this letter privately and for several days. To register, 
participants email the research project manager identified in the letter. Participants are able to ask 
questions at all times using phone and email contact information for the research team located at 
Washington University in St. Louis. 
 
 
5.1 Treatment Plan 
 
Phase 1 (Aim 1) 
 
Delivery of Program Sustainability Action Planning Training (intervention protocol).  
The Program Sustainability Action Planning Training will be designed to assist states in 
completing each of the 6 steps in the action planning process and consists of three components (1) 
pre-workshop preparation, (2) an in-person two day workshop, and (3) ongoing technical 
assistance and support. Each component will be defined by experiential learning and participation 
by multiple stakeholders. A sustainability workgroup will be defined for each state consisting of 
5-12 members who are actively involved in TC initiatives in the state. To ensure training fidelity, 
delivery of all training components will be overseen by Dr. Sarah Moreland-Russell, Principal 
Investigator and Ms. Rebecca Vitale, Project Manager. 
Preparing for the workshop. Approximately two months before scheduled workshop, we will 
work with the state tobacco control program manager to identify the list of TC stakeholders who 
will be involved in workgroup based on our standardized workgroup selection criteria.31 Once a 
list is determined, we will work with the program manager to plan a time to conduct an 
“Introduction to Sustainability” webinar. This webinar educates listeners on the Program 
Sustainability Framework (8 domains), defines the parameters one should use when completing 
PSAT items (i.e., define the “program”), explains how to complete the PSAT online, and provides 
a time for state-specific Q & A. Prior to the webinar, we will ask the program manager and all 
stakeholders to complete an optional Qualtrics survey, assessing general state tobacco control 
program sustainability perceptions and individual demographics (age, gender, race, per NIH 
reporting requirements). Personally identifiable information will not be collected. The pre-

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
7 
 
workshop webinar is important for two reasons: (1) it familiarizes all workgroup members with 
the components of sustainability as outlined in our framework, and (2) introduces the PSAT and 
steps for completing the PSAT online. The webinar instructs users to answer N/A to PSAT items 
they are unable to accurately assess. The pre-workshop webinar will last 45 minutes and provide 
a forum for participants to ask questions about the project. The PSAT will be administered online 
to all workgroup members and results will be aggregated into a report that will be disseminated to 
the participants prior to the start of the in person workshop. There will also be follow-up 
correspondence, discussing state-level results of the PSAT, as well as recommendations for action 
planning next steps and a summary of resources. 
Workshop Day 1. The workshops will take place in each state at the state health department 
(unless a more convenient site is determined). Trainers will follow an instructor manual and will 
provide an interactive workbook for completing the main goals for each day. This workbook will 
prompt users throughout the training to document reactions, ideas, and plans that arise in the 
workshop. The main goals for Day 1 will include defining the program and reviewing assessment 
results. Activities for Day 1 will include clarifying the program’s mission and vision and 
specifying the population it currently serves; reviewing the current activities and services that the 
program is implementing; and clarifying any program and organizational capacities that will be 
necessary to sustain the activities. PSAT scores will identify sustainability areas that need 
attention. These goals align with Steps 1-2 of the Program Sustainability Action Planning Model. 
Workshop Day 2. The main goal for Day 2 is to develop an action plan. Activities for Day 2 
include outlining action plan components including: (1) specific, measureable, attainable, realistic, 
and time-sensitive (SMART) objectives; (2) specific action steps (activities) for achieving 
objectives; (3) a timeline for completion of each objective; (4) persons responsible for completing 
each objective; (5) a clear definition of outcomes and measures for successful completion of each 
objective and overall plan. The workbook will include activity handouts encouraging discussion 
around different areas of program sustainability. Handouts will also include templates for action 
planning and materials for tracking technical assistance interactions. These goals align with Step 
3 of the Program Sustainability Action Planning Model. 
 
Technical Assistance and Support. Following the in person two-day workshop, the state TC 
program will be responsible for executing the action plan and tracking its progress in achieving 
plan objectives (Steps 4-6 in action planning model).  Technical assistance is an essential step in 
the proposed action planning process and will be focused on building each state program’s capacity 
for quality implementation of the action plan.92,93 Technical assistance built on best practice 
indicators17,92–100 will include state-specific quarterly plan review and feedback, assistance in 
troubleshooting problems with step completion, sharing insights on plan successes from other 
similar states, and facilitating connections between  similarly-focused states. The research team 
will remain “on call” for states who need assistance beyond the planned quarterly calls. States will 
also have access to our website https://sustaintool.org, which contains a wealth of information on 
general action plan execution and domain-specific resources to support plan completion and 
execution. The website is updated quarterly with new resources and materials. States will be 
encouraged to retake the PSAT at 12 months and repeat the action planning cycle to revise their 
action plan based on PSAT results and environmental changes, using the indicators and process 
provided in the original training cycle. 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
8 
 
 
Comparison state protocol 
Comparison state programs will participate in the PSAT assessment, record abstraction, and a 
key informant interview and survey with state program managers at three time points. Before 
taking the PSAT, we will work with the state tobacco control program manager to identify the list 
of TC stakeholders who will be involved in workgroup based on our standardized workgroup 
selection criteria.31 Once a list is determined, we will work with the program manager to plan a 
time to conduct an “Introduction to Sustainability” webinar. This webinar is identical to the 
webinar issued to the intervention group, because state programs will be informed of their group 
selection after completing the PSAT. The webinar educates listeners on the Program Sustainability 
Framework (8 domains), defines the parameters one should use when completing PSAT items (i.e., 
define the “program”), explains how to complete the PSAT online, and provides a time for state-
specific Q & A. Prior to the webinar, we will ask the program manager and all stakeholders to 
complete an optional Qualtrics survey, assessing general state tobacco control program 
sustainability perceptions and individual demographics (age, gender, race, per NIH reporting 
requirements). Personally identifiable information will not be collected. The pre-workshop 
webinar is important for two reasons: (1) it familiarizes all workgroup members with the 
components of sustainability as outlined in our framework, and (2) introduces the PSAT and steps 
for completing the PSAT online. The webinar instructs users to answer N/A to PSAT items they 
are unable to accurately assess. The pre-workshop webinar generally lasts 45 minutes. Comparison 
states will receive a detailed sustainability assessment report outlining the state’s PSAT scores. 
There will also be follow-up correspondence, discussing state-level aggregated results of the 
PSAT, as well as recommendations for action planning next steps. At this point, we will provide 
a brief packet with resources about program sustainability. However, comparison states will not 
receive any other component of the training until year four of this project, when they we 
disseminate the finalized curriculum materials. Some comparison states might independently 
implement sustainability plans over the course of the grant. We will track implementation and 
account for it in analyses. 
 
 
Phase 2 (Aim 2) 
 
(Aim 2): Demonstrate change in sustainability outcomes in states who received the Program 
Sustainability Action Planning training compared to states who did not. 
The overall goal for the Program Sustainability Action Planning Training and subsequent 
dissemination activities is to increase capacity for sustainability of evidence-based state TC 
programs through the institutionalization of the program. To determine the extent of reaching this 
goal, we will design a randomized effectiveness trial to document differences in organizational 
and programmatic measures and Program Sustainability Assessment Tool scores between 
intervention and comparison states. We will rely on three sources of data for this study: 1) state-
level program records for measures of predictors of sustainability and institutionalization,9,11 2) 
PSAT scores and 3) a key informant interview and survey with state program managers. While we 
anticipate that our study will impact behavioral risk factors for tobacco-related diseases (e.g., rates 
of smoking and initiation), the dependent variable for our study involves measures of program 
institutionalization and sustainability.  
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
9 
 
C.6.a. Data collection methods 
Pre-Introduction to Sustainability Webinar Survey 
All participants will receive an optional Qualtrics survey via email, prior to participating in the 
“Introduction to Sustainability” webinar. Participants will be asked program sustainability 
perception questions, as well as demographic questions (age, gender, race, per NIH reporting 
requirements). Personally identifiable information, such as the participant’s name, will not be 
collected.   
Program Sustainability Assessment Tool (PSAT)  
The PSAT has established reliability and face validity for effectively measuring program capacity 
for sustainability.43 The PSAT contains 40 questions (5 items per 8 domains) and is measured on 
a 1-7 scale where 1 indicates “To little or no extent” and 7 indicates “To a very great extent”. Users 
also have the option to choose N/A (Not able to answer), and are encouraged to do so if they are 
not familiar with that aspect of the program. The PSAT is directly linked to the eight domains 
outlined in the sustainability framework. Since this framework will be used to guide the action 
planning for each state, the scores from the PSAT will be important in assessing programmatic 
challenges and strengths related to sustainability. The sample for our survey is the workgroup 
defined by the program manager in each of the 24 participating states. We anticipate that 5-12 
people in each state will complete the survey. PSAT scores are aggregated across participants for 
each state so that each state will have an overall PSAT score. Power calculations indicate that 
surveying workgroups in 24 states (12 intervention, 12 comparison) at three time points- 
intervention states at baseline and 12 and 24 months post training and comparison states at baseline 
and 12 and 24 months after baseline collection- will sufficiently capture group differences (Figure 
4).  
The survey will be conducted online using established methods for Web-based surveys.102 To 
enhance the survey response, we will employ the well-established method developed by 
Dillman.103 First, all members of the sample will receive a personalized, advanced notice email. 
This helps to identify the purpose of the survey and establish its legitimacy. Second, approximately 
one week after the advance email is sent, all members of the workgroup receive the survey link. 
In all cases, the replies to the surveys will be anonymous. Third, approximately one week after 
completion of the PSAT, a follow-up email will be sent to all members of the sample. It will thank 
those who have already responded and request a response from those who have not yet responded. 
Finally, once data collection is closed, the workgroup members will receive a detailed 
sustainability report that includes their responses and an aggregated score for the program. For 
intervention states, this report will serve in the assessment and planning phases. All 
correspondence will come from WU and the TCN (an important professional association for our 
sample). The CDC has also agreed to allow us to participate on the OSH Program Service Branch 
calls to make CDC program officers aware of their state’s participation and for promotion of our 
study. We have had previous success with online administration of surveys including the PSAT. 
Response rate for PSAT administration across the six TC states was 86% (123 out of 143). In 
another study, in which we administered an email survey among all 50 state TC managers in three 
waves, with an overall response rate of 97%.51,78 In addition, we have years of experience and 
building partnerships with state TC programs and CDC-OSH. 
 
Program record abstraction 
A review and abstraction of state level TC program public records will be performed for all 24 
states at three time points- baseline and at the annual CDC progress reporting time (June) in years 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
10 
 
2 and 3. This will help in evaluating the impact of the training intervention on the capacity for 
sustainability and institutionalization of the TC program within the organization.  
Step 1: Identifying relevant records and measures. Relevant state-level TC program records 
will be identified within the 24 selected states from various sources. First, all state tobacco 
programs are required by CDC (DP15-1509 funding requirements2) to submit an annual progress 
report. We will begin by using our consultants and empirical literature to identify relevant state-
level programmatic measures documented in these annual progress reports. We have also reviewed 
the DP15-1509 funding requirements for relevance and have outlined the connection between state 
reporting requirements and established predictors of sustainability and outcomes in Appendix C. 
This will increase our likelihood of retrieving all relevant information and provide empirical 
evidence with which to substantiate the responses in the PSAT and identify and address any 
potential bias therein.  In addition, for those metrics not included in the report, we will review state 
health agency websites and publically available state-level government documentation. Dr. 
Brownson has extensive experience in collecting this type of information and has incorporated 
similar strategies in three other studies. 
Step 2: Data abstraction. TC program records will be reviewed and data will be abstracted by 
two trained evaluators using standardized methods and data collection tools created by the research 
team. The development and use of a standardized instrument for the review and abstraction will 
improve inter-rater reliability and will provide quantitative data for monitoring trends and 
intervention effects.  
Step 3: Establishing reliability and validity. Data collected via record review and abstraction 
are subject to bias that may result from rater subjectivity.104 In order to determine the agreement 
between the two evaluators, a baseline reliability evaluation will be conducted early in the process 
to calculate inter-rater reliability. In order to determine validity of self-reported data from the 
PSAT, agreement between self-reported and record-based information will be assessed. Both inter-
rater reliability and validity of self-reported data will be checked at baseline and two other time 
points. 
 
Key Informant Interview and Survey  
In order to collect additional organizational metrics, a key informant interview and survey will be 
conducted with state program managers. These will be performed for all 24 states at three time 
points- baseline and in June during years 2 and 3. This will help in evaluating the impact of the 
training intervention on the capacity for sustainability and institutionalization of the TC program 
within the organization.  
Step 1: Identifying relevant measures. Relevant state-level sustainability metrics will be 
identified within the 24 selected states from various sources, including CDC Best Practices, ALA 
State of Tobacco Control, empirical literature, and consultation of our advisory group.  
Step 2: Key Informant Interview and Survey. TC program managers will be contacted to 
complete a phone interview and survey. The interview and survey will only collect organizational 
metrics not collected in reporting records, and no personal information will be collected. We will 
record phone interviews  to ensure comprehensive analysis by trained abstracters. Prior to the 
interview, program managers will be provided a copy of the questions to be asked from a standard 
script created by the research team. Each question is formatted in multiple choice, e,g,, “How long 
has the program been in existence?” [1: Less than 1 year; 2: 1-3 years; 3: 3-5 years; 4:5-10 years; 
5: 10 or more years; 6: Don’t know]. Using this method, program managers will provide 
unambiguous answers to data points not collected by record abstraction.  

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
11 
 
Step 3: Establishing reliability and validity. Data collected via interview/survey and 
abstraction are subject to bias that may result from rater subjectivity.104 In order to determine the 
agreement between the two evaluators, a baseline reliability evaluation will be conducted early in 
the process to calculate inter-rater reliability. In order to determine validity of self-reported data 
from the PSAT, agreement between self-reported, record-based information, and interview/survey 
information will be assessed. Both inter-rater reliability and validity of self-reported data will be 
checked at baseline and two other time points. 
 
 
Phase 3 (Aim 3): 
C.7. (Aim 3): Actively disseminate The Program Sustainability Action Planning Model and 
Training Curriculum materials and study results to support future adoption. 
No treatment with intervention or comparison groups will take place during this phase. 
 
5.2 Administration 
 
Not applicable. 
 
5.3 Schema 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
12 
 
 
 
5.4 Dose Modification 
 
Not applicable. 
 
 
 
 
 
5.5 Duration of Therapy 
 
Not applicable 
 
 
 
 
 
5.6 Supportive Care 
 
Not applicable 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
13 
 
 
 
 
 
 
6.0 Response Assessment 
 
Not applicable 
 
 
 
 
 
7.0 Study Parameters 
 
Not applicable 
 
 
 
 
 
8.0 Drug Formulation and Procurement 
 
Not applicable 
 
 
9.0 Statistical Considerations 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
14 
 
C.6.d Statistical power  
To estimate the sample size needed for the study, an a priori 
power analysis was performed using data from a natural 
experiment. The data consists of Pre and Post PSAT averages 
from previous work with five tobacco control programs that 
received action planning and training, and five programs that 
did not receive action planning or training for which pre- and 
post-PSAT scores are available. To determine effect size, t-
tests of differences-in-means were conducted for the two 
groups using the changes in PSAT scores between the Pre and 
Post PSAT data. The standardized differences and t-statistics 
yielded an effect size of 1.25, which is a large effect size 
according to Cohen’s criteria.107 Using R statistical 
environment with packages from Del Re108 and Champely,109 
the effect size with power = 0.8 and α = 0.05 project the 
necessary sample size to be approximately 9 (programs) per 
group. Since these calculations are based on a natural 
experiment, which we note occurred over a period of time that 
also saw changes to the recommended action planning,2 we 
are careful not to overstate the anticipated power and 
therefore conservatively propose a sample size one-third 
larger than calculated, 12 per group as shown in Figure 4. This sample is also sufficient to measure 
the main goal of the study, to discern whether the training, and accompanying PSAT tool, 
significantly increase institutionalization of state TC programs.108–110  
 
C.6.e. Data analyses. Descriptive statistics will be calculated (e.g., frequencies and measures of 
central tendency and dispersion), for both the intervention and comparison group at each of the 
three data collection time points to assess baseline averages and changes. Multiple comparisons 
across groups and times will be drawn for independent and dependent variables. All continuous 
measures will be examined for non-normality and all relationships will be examined for linearity 
where applicable. Strategies for addressing issues encountered will be applied as necessary such 
as different functional forms, category scaling, and robust error calculations. In addition, the dose 
reception and delivery, and participant measures from Aim 1 (C.5.d). will be used to calculate rates 
of adoption and implementation of training recommendations (intervention states) and other 
potential actions taken (comparison states, C.5.e).  
To incorporate the influence of idiosyncratic characteristics of the state programs, longitudinal 
regression analysis will be used to model the outcome for each of the dependent variables. This 
analysis is modeled in Equation 1.  
 
                    𝛽0 + 𝛽1𝑃𝑆𝐴𝑇𝑖 𝑡=0 + 𝛽2𝐺𝑖+ 𝛽3𝐷𝑖𝑡+ 𝛽4(𝐺𝑖𝑡∗𝐷𝑖𝑡) + 𝐵5−𝑘𝑋𝑖𝑡= 𝑌𝑖𝑡   
  
Equation 1 
 
where i = state, t = time 1 or 2, 𝑃𝑆𝐴𝑇𝑖 𝑡=0 = baseline PSAT score, 𝐺𝑖 = group (intervention or 
comparison), 𝐷𝑖𝑡 = dose variable,  (𝐺𝑖𝑡∗𝐷𝑖𝑡) = an interaction between the group and dose terms, 
𝑋𝑖𝑡 = a vector of programmatic, organizational, community and funder variables, and 𝑌𝑖𝑡 = the 
Figure 4. Sample size and power calculation 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
15 
 
outcome variable. This analysis will ultimately be used to test Hypothesis 1, that the impact of the 
training is nonzero and positive.  
 
                                               𝐻𝑜: 𝛽̂𝑖𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛= 0                         𝐻𝑎: 𝛽̂𝑖𝑛𝑡𝑒𝑟𝑣𝑒𝑛𝑡𝑖𝑜𝑛> 0                     
Hypothesis 1 
 
The multivariate approach allows us to account for the numerous influences found in 
programmatic, organizational, community, and funder attributes. Data collection at different time 
points also allows for measuring changes in these variables, and the influence these changes have 
on institutionalization outcomes. In addition, the level of adoption and implementation of action 
plans both from the training (intervention states) and from other initiatives (comparison states) will 
vary across programs and time. Varying levels of action plan implementation across states must 
be acknowledged. Multivariate longitudinal analyses also allow us to control for differing degrees 
of dosage reception and delivery. We have successfully used similar dose variables. 111,112 
 
10.0 Recordkeeping 
 
The Principal Investigator, Sarah Moreland-Russell, will be responsible for monitoring the data 
safety and quality of the overall study. This is a minimal risk study that does not warrant a 
formal Data Safety Monitoring Board.  
 
Program Sustainability Assessment Tool (PSAT). The PSAT is a 40-item measure of a program’s 
sustainability capacity which requires 10-15 minutes to complete. Individuals selected by state 
TC program managers will be invited to complete the PSAT online via our secure website 
https://sustaintool.org/. Data collected through the website is automatically de-identified by the 
website and aggregated at the state level. E-mail addresses are collected on the site, but solely for 
website management, technical assistance, and user access. See Appendix A for full copy. 
 
Workshop Data Collection Form. Several process measures will be collected to assess the 
effectiveness of the training and support delivered. All measures will be collected via survey at 
the completion of each training component or tracked by review of programmatic records (i.e., 
action plan, meeting minutes). Quantitative data collected for this aim will be analyzed using 
descriptive statistics. Qualitative analysis of programmatic records will be conducted following 
procedures outlined. See Appendix B for full copy.  
Dosage delivered is the number of units of an intervention delivered or provided.101 For 
this study, this measure will be defined by the number of hours of training and technical 
assistance provided to each state.  
Dosage received is the extent to which participants engage with or use the materials 
provided.101 This will be measured by whether or not the action plan is completed and 
implemented, whether the workgroup utilizes the resources and materials provided, and 
the extent to which the action plan is implemented.  
Participant reactions will be measured by the extent to which the participants felt that the 
objectives of the training and technical assistance were achieved, satisfaction with the 
training and technical assistance and perceived usefulness of the training provided. 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
16 
 
Data abstraction. State level TC program records will be reviewed and data will be abstracted by 
two trained evaluators using standardized methods and data collection tools created by the 
research team. Examples of variables of interest include: implementation of action plan, amount 
and source of funding allocated for the implementation of tobacco control programs, size of the 
program and organization, and number of staff dedicated to support programmatic activity. 
Funding-related information will be collected from annual progress reports of state health 
departments, annual reports, and funding opportunities of funding agencies relevant to tobacco 
control. Abstraction will include only records that are publically available and only 
programmatic and organizational information will be recorded. No individual identifying 
information will be recorded. 
 
Furthermore, the Prevention Research Center has developed a safety monitoring plan to address: 
1) training staff on data sensitivity and protocols for safeguarding confidentiality; 2) storing and 
processing data in a secured, centralized location; 3) securing any hardcopy data in locked files 
when not in use; 4) removing names, addresses, and other direct identifiers from hardcopy and 
computer-readable data when they are no longer necessary for respondent tracking and then 
using encrypted codes for subsequent identification of participants; 5) destroying all identifiable 
linkages to data after data accuracy has been verified and final analyses have been completed; 6) 
using restricted logon identification and password protection computer protocols for all 
computerized entry, retrieval, and analysis; 7) ensuring that the research team members and 
others with direct contact with participants and/or access to identifiable data complete human 
research certification (e.g., CITI, and other human research training programs); 8) including 
human research concerns as a standard agenda item for regular project meetings; 9) 
incorporating responsibility for the human research protection process into the Project Manager’s 
job description; and 10) regular review of research data to ensure validity.  All data will be stored 
on a section of the Washington University in St. Louis password protected server protected by 
the university’s firewalls. The only people who will have access to the raw survey data are the 
study staff and faculty based at the Washington University in St. Louis. Each has taken the CITI 
training and will be approved by the Washington University Institutional Review Board for this 
specific study.  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
17 
 
 
 
11.0 References 
 
1. 
Farrelly MC, Pechacek TF, Thomas KY, Nelson D. The impact of tobacco control 
programs on adult smoking. Am J Public Health. 2008;98(2):304-309. 
doi:10.2105/AJPH.2006.106377. 
2.  
Centers for Disease Control and Prevention. National State-Based Tobacco Control 
Programs. CDC-RFA-DP15-1509. http://www.grants.gov/web/grants/search-
grants.html?keywords=DP15-1509. Accessed January 1, 2015. 
3.  
Kolb D. Assessing individuality in learning: the learning skills profile. Educ Psychol. 
1991;11(3,4):279-295. 
4.  
Akella D. Learning together: Kolb’s experiential theory and its application. J Manag 
Organ. 2010;16(1):100-112. doi:10.5172/jmo.16.1.100. 
5.  
Abdulwahed M, Nagy ZK. Applying Kolb’s experiential learning cycle for laboratory 
education. J Eng Educ. 2009;98:283-294. https://dspace.lboro.ac.uk/dspace-
jspui/handle/2134/5412. 
6.  
Kolb AY, Kolb DA. The learning way: meta-cognitive aspects of experiential learning. 
Simul Gaming. 2009;40(3):297-327. 
7.  
Weiss C. Nothing as practical as good theory: exploring theory-based evaluation for 
comprehensive community initiatives for children and families. In: Connell J, Kubisch A, 
Schorr L, Weiss C, eds. New Approaches to Evaluating Community Initiatives: Vol 1, 
Concepts, Methods, and Contexts. Washington, DC: The Aspen Institute; 1995. 
8.  
Huey-Tysh C. Theory Driven Evaluation. Newbury Park, CA: Sage Publications Ltd.; 
1990. 
9.  
Savaya R, Spiro S. Predictors of sustainability of social programs. Am J Eval. 
2012;33(1):26-43. doi:10.1177/1098214011408066. 
10.  
Scheirer MA, Dearing JW. An agenda for research on the sustainability of public health 
programs. Am J Public Health. 2011;101:2059-2067. doi:10.2105/AJPH.2011.300193. 
11.  
Johnson K, Collins D, Wandersman A. Sustaining innovations in community prevention 
systems: a data-informed sustainability strategy. J Community Psychol. 2013;41(3):322-
340. 
12.  
Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: 
addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8(117). 
doi:10.1186/1748-5908-8-117. 
13.  
Cooper BR, Bumbarger BK, Moore JE. Sustaining evidence-based prevention programs: 
correlates in a large-scale dissemination initiative. Prev Sci. 2015;16:145-157. 
doi:10.1007/s11121-013-0427-1. 
14.  
Chelimsky E. Public-interest values and program sustainability: some implications for 
evaluation practice. Am J Eval. 2014;35(4):527-542. doi:10.1177/1098214014549068. 
15.  
Scheirer MA. Linking sustainability research to intervention types. Am J Public Health. 
2013;103:e1-e8. doi:10.2105/AJPH.2012.300976. 
16.  
Schell S, Luke D, Schooley M, Elliott M, Mueller N, Bunger A. Public health program 
capacity for sustainability: a new framework. Implement Sci. 2013;8(15). 
doi:10.1186/1748-5908-8-15. 
17.  
Escoffery C, Glanz K, Hall D, Elliott T. A multi-method process evaluation for a skin 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
18 
 
cancer prevention diffusion trial. Eval Heal Prof. 2009;32(2):184-203. 
doi:10.1177/0163278709333154.A. 
18.  
Elliott DS, Mihalic S. Issues in disseminating and replicating effective prevention 
programs. Prev Sci. 2004;5(1):47-53. 
19.  
Bumbarger BK, Perkins DF. After randomised trials: issues related to dissemination of 
evidence-based interventions. J Child Serv. 2008;3(2):53-61. 
20.  
Wiltsey Stirman S, Kimberly J, Cook N, Calloway A, Castro F, Charns M. The 
sustainability of new programs and innovations: a review of the empirical literature and 
recommendations for future research. Implement Sci. 2012;7(17). doi:10.1186/1748-5908-
7-17. 
21.  
Gruen R, Elliott J, Nolan M, et al. Sustainability science: an integrated approach for 
health-programme planning. Lancet. 2008;372:1579-1589. doi:10.1016/S0140-
6736(08)61659-1. 
22.  
Scheirer MA. Is sustainability possible? a review and commentary on empirical studies of 
program sustainability. Am J Eval. 2005;26:320-347. doi:10.1177/1098214005278752. 
23.  
Hanson H, Salmoni A, Volpe R. Defining program sustainability: differing views of 
stakeholders. Can J Public Heal. 2009;100(3):304-309. 
24.  
Wright C, Catty J, Watt H, Burns T. A systematic review of home treatment services: 
classification and sustainability. Soc Psychiatry Psychiatr Epidemiol. 2004;39(10):789-
796. 
25.  
Mâsse LC, McKay H, Valente M, Brant R, Naylor P. Physical activity implementation in 
schools: a 4-year follow-up. Am J Prev Med. 2012;43(4):369-377. 
doi:10.1016/j.amepre.2012.06.010. 
26.  
Savaya R, Spiro S, Elran-Barak R. Sustainability of social programs: a comparative case 
study analysis. Am J Eval. 2008;29(4):478-493. doi:10.1177/1098214008325126. 
27.  
Akerlund KM. Prevention program sustainability: the state’s perspective. J Community 
Psychol. 2000;28(3):353-362. 
28.  
Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention 
innovations: a sustainability planning model. Eval Program Plann. 2004;27(2):135-149. 
doi:10.1016/j.evalprogplan.2004.01.002. 
29.  
Steinberg M, Hutchinson K. Literature Review of Program Sustainability Assessment 
Tools. Burnaby, BC; 2010. 
30.  
Office of the Surgeon General. The Health Consequences of Smoking — 50 Years of 
Progress. Rockville, MD; 2014. 
31.  
Calhoun A, Mainor A, Moreland-Russell S, Maier RC, Brossart L, Luke DA. Using the 
Program Sustainability Assessment Tool to assess and plan for sustainability. Prev 
Chronic Dis. 2014;11:131-185. doi:http://dx.doi.org/10.5888/pcd11.130185. 
32.  
Feinberg ME, Ridenour TA, Greenberg MT. The longitudinal effect of technical 
assistance dosage on the functioning of Communities That Care prevention boards in 
Pennsylvania. J Prim Prev. 2008;29(2):145-165. 
33.  
Perkins DF, Feinberg ME, Greenberg MT, Johnson LE, Chilenski, Sarah M. Mincemoyer 
CC, Spoth RL. Team factors that predict to sustainability indicators for community-based 
prevention teams. Eval Program Plann. 2011;34(3). 
doi:10.1016/j.biotechadv.2011.08.021.Secreted. 
34.  
Scheirer MA, Hartling G, Hagerman D. Defining sustainability outcomes of health 
programs: illustrations from an on-line survey. Eval Program Plann. 2008;31:335-346. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
19 
 
doi:10.1016/j.evalprogplan.2008.08.004. 
35.  
de Vries H, Eggers SM, Bolman C. The role of action planning and plan enactment for 
smoking cessation. BMC Public Health. 2013;13(393). doi:10.1186/1471-2458-13-393. 
36.  
van Osch L, Lechner L, Reubsaet A, De Vries H. From theory to practice: an explorative 
study into the instrumentality and specificity of implementation intentions. Psychol 
Health. 2010;25(3):351-364. doi:10.1080/08870440802642155. 
37.  
De Vet E, Oenema A, Brug J. More or better: do the number and specificity of 
implementation intentions matter in increasing physical activity? Psychol Sport Exerc. 
2011;12(4):471-477. doi:10.1016/j.psychsport.2011.02.008. 
38.  
Roussos ST, Fawcett SB. A review of collaborative partnerships as a strategy for 
improving community health. Annu Rev Public Health. 2000;21:369-402. 
39.  
Elias MJ, Zins JE, Graczyk PA, Weissberg RP. Implementation, sustainability, and 
scaling up of social-emotional and academic innovations in public schools. School Psych 
Rev. 2003;32(3):303-319. 
40.  
Feinberg ME, Bontempo DE, Greenberg MT. Predictors and level of sustainability of 
community prevention coalitions. Am J Prev Med. 2008;34(6):495-501. 
doi:10.1016/j.amepre.2008.01.030. 
41.  
Goodson P, Murphy Smith M, Evans A, Meyer B, Gottlieb NH. Maintaining prevention in 
practice: survival of PPIP in primary care settings. Am J Prev Med. 2001;20(3):184-189. 
doi:10.1016/S0749-3797(00)00310-X. 
42.  
Verbiest ME, Presseau J, Chavannes NH, et al. Use of action planning to increase 
provision of smoking cessation care by general practitioners: role of plan specificity and 
enactment. Implement Sci. 2014;9(180). doi:10.1186/s13012-014-0180-2. 
43.  
Luke DA, Calhoun A, Robichaux CB, Elliott MB, Moreland-Russell S. The Program 
Sustainability Assessment Tool: a new instrument for public health programs. Prev 
Chronic Dis. 2014;11:130-184. doi:10.5888/pcd11.130184. 
44.  
Schoen M, Moreland-Russell S, Prewitt K, Carothers B. Social network analysis of public 
health funding programs to measure partnership. Soc Sci Med. 2014;123:90-95. 
45.  
Tabak R, Duggan K, Smith C, Aisaka K, Moreland-Russell S, Brownson R. Assessing 
capacity for sustainability of effective programs and policies in local health departments. J 
Public Heal Manag Pract. 2015;e-pub ahea. 
46.  
Barbero C, Moreland-Russell S, Bach L, Cyr J. An evaluation of public school district 
tobacco policies in St. Louis County, Missouri. J Sch Health. 2015;83(8):525-532. 
doi:doi: 10.1111/josh.12061. 
47.  
Moreland-Russell S, Benson P, Cyr J, Colditz G, Barnoya J. Study of secondhand smoke 
exposure in St. Louis City and County: need for comprehensive smokefree law adoption 
across Missouri. J Missouri Med. 2012;109(9). 
48.  
Tabak R, Moreland-Russell S. Food service director perspectives on implementation of 
the new National School Lunch Program guidelines: a qualitative study. Heal Behav 
Policy Rev. 2015;in press. 
49.  
Center for Public Health Systems Science. St. Louis County Communities Putting 
Prevention to Work (CPPW) Final Report. St. Louis, MO; 2012. 
50.  
Center for Public Health Systems Science. Point-of-Sale Strategies: A Tobacco Control 
Guide. St. Louis; 2014. 
51.  
Center for Public Health Systems Science. Point-of-Sale Report to the Nation: Policy 
Activity 2012 - 2014. St. Louis, MO; 2015. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
20 
 
52.  
Center for Public Health Systems Science. Point-of-Sale Report to the Nation: The 
Tobacco Retail and Policy Landscape. St. Louis, MO; 2014. 
53.  
Luke DA. Multilevel growth curve analysis for quantitative outcomes. In: Handbook of 
Longitudinal Research: Design, Measurement, and Analysis across the Social Sciences. 
Academic Press; 2007. 
54.  
Luke DA. Multilevel Modeling: Quantitative Applications in the Social Sciences. London: 
A Sage University Paper Series; 2004. 
55.  
Luke DA, Stamatakis KA, Brownson RC. State youth-access tobacco control policies and 
youth smoking behavior in the United States. Am J Prev Med. 2000;19(3):180-187. 
56.  
Luke DA, Ribisl KM, Smith C, Sorg AA. Family smoking prevention and tobacco control 
act: Banning outdoor tobacco advertising near schools and playgrounds. Am J Prev Med. 
2011;40(3):295-302. 
57.  
Luke D a., Stamatakis K a. Systems Science Methods in Public Health: Dynamics, 
Networks, and Agents. Annu Rev Public Health. 2012;33(1):357-376. 
doi:10.1146/annurev-publhealth-031210-101222. 
58.  
Luke D a., Harris JK, Shelton S, Carothers BJ, Mueller NB, Allen P. Systems analysis of 
collaboration in 5 national tobacco control networks. Am J Public Health. 
2010;100(7):1290-1297. doi:10.2105/AJPH.2009.184358. 
59.  
Brownson R, Baker E, Leet T, Gillespie K, True W. Evidence-Based Public Health. 2nd 
ed. New York: Oxford University Press; 2011. 
60.  
Brownson R, Diem G, Grabauskas V, et al. Training practitioners in evidence-based 
chronic disease prevention for global health. Promot Educ. 2007;14(3):159-163. 
61.  
Gibbert W, Keating S, Jacobs J, et al. Training the workforce in evidence-based public 
health: An evaluation of impact among US and international practitioners. Prev Chronic 
Dis. 2013;10(E148). 
62.  
Brownson RC, Petitti DB, eds. Applied Epidemiology: Theory to Practice. 2nd ed. New 
York, NY: Oxford University Press; 2006. 
63.  
Remington P, Brownson R, Savitz D, eds. Chronic Disease Epidemiology and Control. 
3rd ed. Washington, DC: American Public Health Association; 2010. 
64.  
Rabin, B. A., Brownson, R. C., Haire-Joshu, D., Kreuter, M. W., & Weaver NL. A 
glossary for dissemination and implementation research in health. J Public Heal Manag 
Pract. 2008;14(2):117-123. 
65.  
Brownson, R., Colditz, G., Proctor E. Dissemination and Implementation Research in 
Health: Translating Science to Practice. New York, NY: Oxford University Press; 2012. 
66.  
Brownson RC, Ballew P, Brown KL, et al. The effect of disseminating evidence-based 
interventions that promote physical activity to health departments. Am J Public Health. 
2007;97(10):1900-1907. 
67.  
Brownson R, Kreuter M, Arrington B, True W. Translating scientific discoveries into 
public health action: how can schools of public health move us forward? Public Health 
Rep. 2006;121(1):97-103. 
68.  
Rabin BA, Brownson RC, Kerner JF, Glasgow RE. Methodologic challenges in 
disseminating evidence-based interventions to promote physical activity. Am J Prev Med. 
2006;31:24-34. 
69.  
Center for Public Health Systems Science. STARS-Policy Crosswalk. St. Louis; 2015. 
70.  
Center for Public Health Systems Science. Reducing Cheap Tobacco & Youth Access: 
New York City. St. Louis; 2015. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
21 
 
71.  
Center for Public Health Systems Science. Assessing Retail Environments with S.T.A.R.S. 
St. Louis; 2015. 
72.  
Moreland-Russell S, Combs T, Jones J, Sorg A. State Level Point-Of-Sale Policy Priority 
As A Result of the FSPTCA. AIMS Public Heal. 2015;2(4):681-690. 
73.  
Combs T, Moreland-Russell S, Roche J. Evaluation of Measurement Tools For Tobacco 
Product Displays: Is There an App for That? AIMS Public Heal. 2015;2(4):810-820. 
74.  
Center for Public Health Systems Science. Policy Strategies: A Tobacco Control Guide. 
St. Louis; 2014. 
75.  
Center for Public Health Systems Science. Pricing Policy: A Tobacco Control Guide. St. 
Louis; 2014. 
76.  
Centers for Disease Control and Prevention. Best Practices User Guide: Coalitions- State 
and Community Interventions. Atlanta; 2009. 
77.  
Centers for Disease Control and Prevention. Best Practices User Guide: Youth 
Engagement- State and Community Interventions. Atlanta; 2009. 
78.  
Centers for Disease Control and Prevention. Best Practices User Guide: Health Equity in 
Tobacco Prevention and Control. Atlanta; 2015. 
79.  
Mueller NB, Luke DA, Herbers SA, Montgomery TP. The best practices: use of the 
guidelines by ten state Tobacco Control programs. Am J Prev Med. 2006;31(4). 
80.  
Luke DA, Wald LM, Carothers BJ, Bach LE, Harris JK. Network influences on 
dissemination of evidence-based guidelines in state tobacco control programs. Heal Educ 
Behav. 2013;40:33S - 42S. doi:10.1177/1090198113492760. 
81.  
Sheeran P, Milne S, Webb T, Gollwitzer P. Implementation intentions and health 
behavior. In: Connor M, Norman P, eds. Predicting Health Behaviour Research and 
Practice With Social Cognition Models. 2nd ed. Berkshire, England: Open University 
Press, McGraw-Hill Education; 2005:276-323. 
82.  
Friedman L, Schreiber L. Healthy Families America state systems development: an 
emerging practice to ensure program growth and sustainability. J Prev Interv Community. 
2007;34(1-2):67-87. 
83.  
Edwards NC, Roelofs SM. Sustainability: the elusive dimension of international health 
projects. Can J Public Heal. 2006;97(1):45-49. 
84.  
Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model 
(PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 
2008;34(4):228-243. 
85.  
Kolb DA. Experiential Learning: Experience as the Source of Learning and Development. 
1st ed. Englewood Cliffs, NJ: Prentice Hall, Inc.; 1984. doi:10.1002/job.4030080408. 
86.  
Friedman AR, Wicklund K. Allies against asthma: a midstream comment on 
sustainability. Health Promot Pract. 2006;7:140S - 148S. 
doi:10.1177/1524839906287058. 
87.  
Batan M, Butterfoss F, Jaffe A, LaPier T. A Sustainability Planning Guide for Healthy 
Communities. Atlanta, GA; 2010. 
88.  
American Lung Association. State of Tobacco Control Report 2015. Washington, DC 
http://www.stateoftobaccocontrol.org/. 
89.  
American Lung Association. Unpublished data provided upon request. 2015. 
90.  
Centers for Disease Control and Prevention. State Tobacco Activities Tracking & 
Evaluation (STATE) System. Atlanta; 2015. 
http://www.cdc.gov/tobacco/data_statistics/state_data/state_system/index.htm.  Accessed 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
22 
 
May 12, 2015. 
91.  
Centers for Disease Control and Prevention. Best Practices for Comprehensive Tobacco 
Control Programs—2014. Atlanta, GA; 2014. 
92.  
Chinman M, Hannah G, Wandersman A, et al. Developing a community science research 
agenda for building community capacity for effective preventive interventions. Am J 
Community Psychol. 2005;35(3-4):143-157. doi:10.1007/s10464-005-3390-6. 
93.  
Wandersman A, Chien VH, Katz J. Toward an evidence-based system for innovation 
support for implementing innovations with quality: tools, training, technical assistance, 
and quality assurance/quality improvement. Am J Community Psychol. 2012;50(3-4):445-
459. doi:10.1007/s10464-012-9509-7. 
94.  
Chinman M, Hunter SB, Ebener P, et al. The getting to outcomes demonstration and 
evaluation: an illustration of the prevention support system. Am J Community Psychol. 
2008;41(3-4):206-224. 
95.  
Kegeles SM, Rebchook GM, Tebbetts S. Challenges and facilitators to building program 
evaluation capacity among community-based organizations. AIDS Educ Prev. 
2005;17(4):284-299. 
96.  
Hamdallah M, Vargo S, Herrera J. The VOICES/VOCES success story: effective 
strategies for training, technical assistance and community-based organization 
implementation. AIDS Educ Prev. 2006;18(Supplement A):171-183. 
doi:10.1521/aeap.2006.18.supp.171. 
97.  
Spoth R, Clair S, Greenberg M, Redmond C, Shin C. Toward dissemination of evidence-
based family interventions: maintenance of community-based partnership recruitment 
results and associated factors. J Fam Psychol. 2007;21(2):137-146. 
98.  
Collins C, Harshbarger C, Sawyer R, Hamdallah M. The diffusion of effective behavioral 
interventions project: Development, implementation, and lessons learned. AIDS Educ 
Prev. 2006;18(SUPPL. A):5-20. 
99.  
Klesges LM, Estabrooks PA, Dzewaltowski DA, Bull SS, Glasgow RE. Beginning with 
the application in mind: designing and planning health behavior change interventions to 
enhance dissemination. Ann Behav Med. 2005;29 Suppl:66-75. 
doi:10.1207/s15324796abm2902s_10. 
100.  Clements DH, Sarama J, Wolfe CB, Spitler ME. Longitudinal evaluation of a scale-up 
model for teaching mathematics with trajectories and technologies. J Res Educ Eff. 
2013;5(4):812-850. doi:10.1080/19345747.2011.627980. 
101.  Linnan L, Steckler A. Process Evaluation in Public Health Interventions and Research. 
(Linnan L, Steckler A, eds.). San Francisco, CA: Jossey Bass Publishers.; 2002. 
102.  Best S, Krueger B. Internet data collection. In: Lewis-Beck M, ed. Quantitative 
Applications in the Social Sciences. Thousand Oaks, CA: Sage Publications Ltd.; 2004. 
103.  Dillman D. Mail and Telephone Surveys: The Total Design Method. New York, NY: 
Wiley; 1978. 
104.  Cassidy LD, Marsh GM, Holleran MK, Ruhl LS. Methodology to improve data quality 
from chart review in the managed care setting. Am J Manag Care. 2002;8(9):787-793. 
105.  Miller TR, Levy DT. Cost-outcome analysis in injury prevention and control: eighty-four 
recent estimates for the United States. Med Care. 2000;38(6):562-582. 
106.  Statistical Analysis System Institute Incorporated. SAS User’s Guide: Basics, Version 5 
Edition. 1985. 
107.  Cohen J. Statistical Power Analysis for the Behavioral Sciences. 2nd ed. New Jersey: 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
23 
 
Erlbalm; 1988. doi:10.1234/12345678. 
108.  Del Re A. Compute.es: Compute Effect Sizes. 2014. http://cran.r-
project.org/web/packages/compute.es. 
109.  Champely S. Pwr: Basic Functions for Power Analysis. 2015. http://cran.r-
project.org/package=pwr. 
110.  Team RC. R: A Language and Environment for Statistical Computing. 2014. 
http://www.r-project.org. 
111.  Brownson RC, Hagood L, Lovegreen SL, et al. A multilevel ecological approach to 
promoting walking in rural communities. Prev Med (Baltim). 2005;41(5-6):837-842. 
112.  Brownson RC, Smith CA, Pratt M, et al. Preventing cardiovascular disease through 
community-based risk reduction: the Bootheel Heart Health Project. Am J Public Health. 
1996;86(2):206-213. 
 
 
12.0 Personnel 
 
There are no laboratory or ancillary laboratory instructions to report for this study. However, we 
have included below a list of all personnel working on the study. 
 
 
Principal Investigator: Sarah Moreland-Russell, PhD, MPH* 
Co-Investigators: Ross Brownson, PhD*; Douglas Luke, PhD† 
Statistician: Todd Combs, PhD† 
Research Manager: Rebecca Vitale, MPH* 
Research Staff: Elizabeth Zofkie, MPH*; Timothy Blaine-Kuklo*, Kim Prewitt†, Rachel Hackett†, 
Laura Brossart† 
 
* 
Prevention Research Center in St. Louis 
George Warren Brown School for Social Work 
Washington University in St. Louis 
St. Louis, MO 63130 
 
† 
Center for Public Health Systems Science 
George Warren Brown School for Social Work 
Washington University in St. Louis 
St. Louis, MO 63130 
 
 
12.0 Informed Consent 
 
**Please see separate informed consent letter document (Appendix C)** 
 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
24 
 
13. Additional Elements 
There are no known risks from being in this study, and participants will not benefit personally.  
However, we hope that others may benefit in the future from what we learn as a result of this study. 
Participation comes at no additional cost to the participant. Participation is entirely voluntary and 
participants can withdraw from the study at any time simply by indicating that they do not wish 
to continue. Should participants withdraw from the study, they are not penalized for benefits for 
which they otherwise qualify. This study anticipates a maximum of 288 participants, with a 
maximum of 12 participants at each state tobacco control program site. 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
25 
 
 
Appendices 
 
 
Appendix A. Program Sustainability Assessment Tool (PSAT) Survey 
 
 
Program Sustainability Assessment Tool 
  
 
What is program sustainability capacity?  
We define program sustainability capacity as the ability to maintain programming and its 
benefits over time.  
 
Why is program sustainability capacity important?  
Programs at all levels and settings struggle with their sustainability capacity. Unfortunately, 
when programs are forced to shut down, hard won improvements in public health, clinical care, 
or social service outcomes can dissolve. To maintain these benefits to society, stakeholders must 
understand all of the factors that contribute to program sustainability. With knowledge of these 
critical factors, stakeholders can build program capacity for sustainability and position their 
efforts for long term success.  
 
What is the purpose of this tool?  
This tool will enable you to assess your program’s current capacity for sustainability across a 
range of specific organizational and contextual factors. Your responses will identify 
sustainability strengths and challenges. You can then use results to guide sustainability action 
planning for your program.  
 
Helpful definitions  
This tool has been designed for use with a wide variety of programs, both large and small, across 
different settings. Given this flexibility, it is important for you to think through how you are 
defining your program, organization, and community before starting the assessment.  
Below are a few definitions of terms that are frequently used throughout the tool.  
Program refers to the set of formal organized activities that you want to sustain over 
time. Such activities could occur at the local, state, national, or international level and in 
a variety of settings.  
Organization encompasses all the parent organizations or agencies in which the program 
is housed. Depending on your program, the organization may refer to a national, state, or 
local department, a nonprofit organization, a hospital, etc.  
Community refers to the stakeholders who may benefit from or who may guide the 
program. This could include local residents, organizational leaders, decision-makers, etc. 
Community does not refer to a specific town or neighborhood.  

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
26 
 
 
The name of the program or set of activities I am assessing is:  
_________________________________________________________  
 
 
In the following questions, you will rate your program across a range of specific factors that 
affect sustainability. Please respond to as many items as possible. If you truly feel you are not 
able to answer an item, you may select “NA.” For each statement, circle the number that best 
indicates the extent to which your program has or does the following things.  
 
 
Environmental Support: Having a supportive internal and external climate for your program  
1. Champions exist who strongly support the program. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. The program has strong champions with the ability to garner resources.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. The program has leadership support from within the larger organization. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
NA 4. The program has leadership support from outside of the organization. 1 2 3 4 5 6 7 NA 
5. The program has strong public support. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Funding Stability: Establishing a consistent financial base for your program  
1. The program exists in a supportive state economic climate.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. The program implements policies to help ensure sustained funding.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. The program is funded through a variety of sources.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. The program has a combination of stable and flexible funding.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The program has sustained funding.  

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
27 
 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Partnerships: Cultivating connections between your program and its stakeholders 
1. Diverse community organizations are invested in the success of the program.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. The program communicates with community leaders.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. Community leaders are involved with the program. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. Community members are passionately committed to the program.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The community is engaged in the development of the program goals. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Organizational Capacity: Having the internal support and resources needed to effectively 
manage your program and its activities 
1. The program is well integrated into the operations of the organization.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. Organizational systems are in place to support the various program needs. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. Leadership effectively articulates the vision of the program to external partners. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. Leadership efficiently manages staff and other resources.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The program has adequate staff to complete the program’s goals. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Program Evaluation: Assessing your program to inform planning and document results 
1. The program has the capacity for quality program evaluation. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. The program reports short term and intermediate outcomes. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
28 
 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. Evaluation results inform program planning and implementation. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. Program evaluation results are used to demonstrate successes to funders and other key 
stakeholders.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The program provides strong evidence to the public that the program works. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Program Adaptation: Taking actions that adapt your program to ensure its ongoing 
effectiveness 
1. The program periodically reviews the evidence base. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. The program adapts strategies as needed. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. The program adapts to new science. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. The program proactively adapts to changes in the environment.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The program makes decision about which components are ineffective and should not continue. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Communications: Strategic communication with stakeholders and the public about your 
program 
1. The program has communication strategies to secure and maintain public support. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. Program staff communicate the need for the program to the public. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. The program is marketed in a way that generates interest. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. The program increases community awareness of the issue. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
29 
 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The program demonstrates its value to the public. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
Strategic Planning: Using processes that guide your programs direction, goals, and strategies 
1. The program plans for future resource needs. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
2. The program has a long-term financial plan. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
3. The program has a sustainability plan. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
4. The program’s goals are understood by all stakeholders.  
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
5. The program clearly outlines roles and responsibilities for all stakeholders. 
Likert [1: To little or no extent; 2; 3; 4; 5; 6; 7: To a very great extent; or Not able to answer] 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
30 
 
 
Appendix B. Data Collection Form for Workshop 
 
I.  
DOSAGE DELIVERED:  
number of units of an intervention delivered or provided  
1. # Stakeholders to complete PSAT  
2. # Stakeholders to attend training  
3. # Stakeholders to attend TA calls  
4. # hours of training  
5. # hours of TA  
 
 
II.  
DOSAGE RECEIVED:  
extent to which participants engage with or use materials or recommended resources.  
 
Service-Related Questions related to Comprehension- required for properly completing tools  
1. After viewing the Introduction to Sustainability Webinar, I now understand the concept of 
program sustainability completely (5 point Likert scale strongly agree- strongly disagree).  
 
2. After attending/viewing the Action Planning Workshop, I now understand the process of 
Program Sustainability Action Planning completely (5 point Likert scale strongly agree- strongly 
disagree).  
 
3. The handouts provided helped me to understand the Action Planning Workshop material (5 
point Likert scale strongly agree- strongly disagree).  
 
Action Plan Implementation-Related Questions (Implementation- “measures the extent to 
which users engaged with or used”)  
1. My program created a program sustainability action plan based on the results of the Program 
Sustainability Assessment Tool and Action Planning Training (Y/N).  
 
2. My program implemented the activities outlined in our program sustainability Action Plan 
(Y/N).  
 
3. Please indicate the percentage of your sustainability plan your program has implemented (i.e., 
If you met 3 out 4 objectives, you would select 75%).  
 
_____ 10% or less _____ 25% _____ 50% _____ 75% _____ 90% or more  
4. Please estimate the amount of time per month your sustainability plan was discussed among 
staff.  
 
_____ Less than 1 hour per month _____ 1-3 hours _____ 3-5 hours _____ 5-7 hours _____ 7-9 
hours _____ 10+  

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
31 
 
5. Please estimate the amount of time per month your program staff focused on implementing 
your sustainability plan steps.  
 
_____ Less than 1 hour per month _____ 1-3 hours _____ 3-5 hours _____ 5-7 hours _____ 7-9 
hours _____ 10+  
 
 
III. 
PARTICIPANT REACTION  
 
Service Delivery: PSAT and Report-Related Questions (Relevance, Satisfaction)  
 
1. I found the Program Sustainability Assessment Tool to be informative (5 point Likert scale 
strongly agree- strongly disagree).  
 
2. The Program Sustainability Assessment Tool and Report helped me to think about program 
sustainability in ways I hadn’t considered before (5 point Likert scale strongly agree- strongly 
disagree).  
 
3. My program’s future planning was impacted positively by the process of completing the 
Program Sustainability Assessment Tool (5 point Likert scale strongly agree- strongly disagree).  
 
4. Planning for my program’s sustainability was made easier by completing the Program 
Sustainability Assessment Tool (5 point Likert scale strongly agree- strongly disagree).  
 
Service Delivery: Action Planning Training-Related Questions (Relevance, Satisfaction)  
1. Objectives of the Action Planning Training were useful for my work (5 point Likert scale 
strongly agree- strongly disagree).  
 
2. Objectives of the Action Planning Training were clearly articulated (5 point Likert scale 
strongly agree- strongly disagree).  
 
3. Objectives of the training were achieved during the Action Planning Training (5 point Likert 
scale strongly agree- strongly disagree).  
 
4. Overall, how would you rate the quality of the facilitator’s presentation for the Action 
Planning Training? (5 point Likert scale excellent- poor).  
 
5. The length of the Action Planning Training was: (3 point Likert scale too long- too short).  
 
6. The Action Planning Training made planning for my program’s sustainability easier. (5 point 
Likert scale strongly agree- strongly disagree).  
 
7. The Action Planning Training was beneficial for my program. (5 point Likert scale strongly 
agree- strongly disagree).  
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
32 
 
8. The Action Planning Training was directly relevant to my program. (5 point Likert scale 
strongly agree- strongly disagree).  
 
9. Overall, how satisfied were you with the Action Planning Training? (5 point Likert scale very 
satisfied to - very dissatisfied).  
 
Service Delivery: Technical Assistance-Related Questions (Relevance, Satisfaction)  
1. The technical assistance provided following the Action Planning Training was beneficial for 
my program. (5 point Likert scale strongly agree- strongly disagree).  
 
2. The technical assistance provided following the Action Planning Training was directly 
relevant to my program. (5 point Likert scale strongly agree- strongly disagree).  
 
_  
3. The technical assistance provided following the Action Planning Training made planning for 
my program’s sustainability easier. (5 point Likert scale strongly agree- strongly disagree).  
 
4. Overall, how satisfied were you with the technical assistance provided following the Action 
Planning Training? (5 point Likert scale very satisfied to - very dissatisfied).  
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
33 
 
 
Appendix C. Sample Consent Letter 
 
 
INFORMED CONSENT DOCUMENT 
 
Project Title: 
Program Sustainability Action Planning Training Project 
 
Principal Investigator: 
Sarah Moreland-Russell 
 
Research Team Contact: 
Rebecca Vitale (314) 935-8025 
 
This consent form describes the research study and helps you decide if you want to participate.  
It provides important information about what you will be asked to do during the study, about the 
risks and benefits of the study, and about your rights and responsibilities as a research 
participant.   
 You should read and understand the information in this document including the 
procedures, risks and potential benefits.  
 If you have questions about anything in this form, you should ask the research team 
for more information before you agree to participate. 
 You may also wish to talk to your family or friends about your participation in this 
study. 
 Do not agree to participate in this study unless the research team has answered your 
questions and you decide that you want to be part of this study.  
 
WHAT IS THE PURPOSE OF THIS STUDY? 
 
This is a research study.  We invite you to participate in this research study because you were 
identified as an important or relevant tobacco control program stakeholder in your state. 
 
The purpose of this research study is to test education materials to see if they can help improve 
state-level tobacco control initiatives across the United States. 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
34 
 
WHAT WILL HAPPEN DURING THIS STUDY? 
 
If you agree to participate, we would like you to indicate your intent to enroll to the research 
team member identified at the bottom of this document.  
 
If you do not wish to participate, please indicate this to the research team member identified at 
the bottom of this document. We will not contact you again.   
 
For state employees who are not program managers: Although your program manager will 
know who participates in the study, you do not have to participate and your position and 
evaluations by your employer will not be affected by your choice.  Your program manager might 
changes to the way your state tobacco control program functions after the study.  If this occurs, it 
will be done separate from this research project and you will continue to do your job as normal. 
 
Your state program will either be in the “intervention group” or the “control group”.  The 
intervention group will do some additional education activities. 
 
 
All states: 
We will ask you to attend a webinar during work hours introducing you to the study. Following 
this, we will ask you to complete an electronic survey tool through http://www.sustaintool.org, in 
which you will rate various aspects of your state-level tobacco control program. The webinar 
will take approximately one hour to complete, while the survey will take approximately twenty 
minutes to complete. On the survey, you are free to choose “Not Applicable” for any question 
you choose.  This survey will be repeated again 12 months after the study starts and a third time 
24 months after the study starts. Program managers will complete a survey through Qualtrics 
providing information about their program. Program managers will also complete an audio-
recordedphone interview with the research team for information about their program. 
 
 
Intervention States: 
The WUSTL research team will come to your office to deliver in-person workshops.  The 
workshop includes a discussion of the survey results as well as a personalized training to 
establish action plans in fulfillment of CDC sustainability planning guidelines. You will also 
complete a short questionnaire rating your experience in the workshops.  

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
35 
 
 
Workshop Day 1:  
The main goals for Day 1 will include defining the program and reviewing survey results. We 
will clarify the program’s mission and vision and specify the population it currently serves.  We 
will review the current activities and services that the program is implementing and clarify any 
program and organizational capacities that will be necessary to sustain the activities.  
 
Workshop Day 2:  
The main goal for Day 2 is to develop an action plan. We will outline action plan components 
including: (1) specific, measureable, attainable, realistic, and time-sensitive objectives;  
(2) specific action steps (activities) for achieving objectives;  
(3) a timeline for completion of each objective;  
(4) persons responsible for completing each objective;  
(5) a clear definition of outcomes and measures for successful completion of each objective and 
overall plan.  
 
Handouts will be provided including an action plan template and spreadsheets that allow for easy 
tracking and analysis of action plan outcomes. 
 
Following the workshops, you will receive a list of resources relevant to your program’s needs.  
Your program will be able to work with the WUSTL research team for technical assistance 
towards your action plans.  
 
Will you save my research data to use in future research studies?  
 
As part of this study, we are obtaining survey data from you. We would like to use this data for 
studies going on right now as well as studies that are conducted in the future.  These studies may 
provide additional information that will be helpful in understanding how to best improve tobacco 
control programs.  It is unlikely that what we learn from these studies will have a direct benefit 
to you.  There are no plans to provide financial compensation to you for use of your data.  By 
allowing us to use your data you give up any property rights you may have in the data. 
 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
36 
 
We will share your data with other researchers. They may be doing research in areas similar to 
this research or in other unrelated areas.  These researchers may be at Washington University, at 
other research centers and institutions, or commercial sponsors of research.  We may also share 
your data with large data repositories (a repository is a database of information) for broad 
sharing with the research community.  If your individual data is placed in one of these 
repositories only qualified researchers, who have received prior approval from individuals that 
monitor the use of the data, will be able to look at your information.   
 
If you change your mind and do not want us to store and use your data for future research, you 
should contact the research team member identified at the top of this document.  The data will no 
longer be used for research purposes. However, if some research with your data has already been 
completed, the information from that research may still be used. Also, if the data has been shared 
with other researchers it might not be possible to withdraw the data to the extent it has been 
shared. 
 
The use of your data for the purposes described above is optional.  Please let the WU 
research team know whether you agree to this future use. 
 
Audio/Video Recording or Photographs 
 
One aspect of this study involves making audio recordings. Only interviews with program 
managers will be recorded. Interactions with participants that are not program managers will not 
be recorded by the research team. The research team will record audio from interviews with 
program managers to ensure accurate transcription of answers collected over the phone. Because 
of this, audio recording is not optional for program managers. Only members of the research 
team at Washington University will have access to these recordings, and they will not be released 
outside of the research team in any way. The recordings will be destroyed at the conclusion of 
the study. 
 
 
HOW MANY PEOPLE WILL PARTICIPATE? 
 
Approximately 288 people will take part in this study conducted by investigators at Washington 
University.   

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
37 
 
 
HOW LONG WILL I BE IN THIS STUDY? 
 
If you agree to take part in this study, your involvement will last for a total of 24 months. 
 
WHAT ARE THE RISKS OF THIS STUDY? 
 
You may experience one or more of the risks indicated below from being in this study. In 
addition to these, there may be other unknown risks, or risks that we did not anticipate, 
associated with being in this study. 
 
One risk of participating in this study is that confidential information about you may be 
accidentally disclosed.  We will use our best efforts to keep the information about you secure.  
Please see the section in this consent form titled “How will you keep my information 
confidential?” for more information. 
 
WHAT ARE THE BENEFITS OF THIS STUDY? 
 
You will not benefit from being in this study. 
However, we hope that, in the future, other people might benefit from this study because of what 
we will learn. 
 
WILL IT COST ME ANYTHING TO BE IN THIS STUDY? 
 
You will not have any costs for being in this research study.   
WILL I BE PAID FOR PARTICIPATING? 
 
You will not be paid for being in this research study. 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
38 
 
 
WHO IS FUNDING THIS STUDY? 
 
The National Institutes of Health National Cancer Institute (NCI) is funding this research study.  
This means that the Washington University is receiving payments from NCI to support the 
activities that are required to conduct the study.  No one on the research team will receive a 
direct payment or increase in salary from NCI for conducting this study.    
 
HOW WILL YOU KEEP MY INFORMATION CONFIDENTIAL? 
 
Other people such as those indicated below may become aware of your participation in this study 
and may inspect and copy records pertaining to this research. Some of these records could 
contain information that personally identifies you.  We will keep your participation in this 
research study confidential to the extent permitted by law.   
 Government representatives (including the Office for Human Research Protections) to 
complete federal or state responsibilities 
 The National Institutes of Health and National Cancer Institute 
 University representatives to complete University responsibilities 
 Washington University’s Institutional Review Board (a committee that oversees the 
conduct of research involving human participants) and Human Research Protection 
Office.  The Institutional Review Board has reviewed and approved this study.  
 
To help protect your confidentiality, we will automatically de-identify your survey responses. 
This means that any information you input into our survey is not connected to your name, or you 
personally, and is instead aggregated at the state level. Email addresses are collected on the site, 
but this is solely for website management, technical assistance and user access. Your email 
address is not connected with your responses. All collected data are secured using passwords and 
encryption. Any report or article that we write will not include information that can directly 
identify you.  The journals that publish these reports or articles require that we share your 
information that was collected for this study with others. Sharing this information will allow 
others to make sure the results of this study are correct and help develop new ideas for research. 
Your information will be shared in a way that cannot directly identify you.  
 
IS BEING IN THIS STUDY VOLUNTARY? 
 
Taking part in this research study is completely voluntary.  You may choose not to take part at 
all.  If you decide to be in this study, you may stop participating at any time.  Any data that was 

PI: Moreland-Russell, Sarah 
Sustainability Action Planning 
39 
 
collected as part of your participation in the study will remain as part of the study records and 
cannot be removed.   
 
If you decide not to be in this study, or if you stop participating at any time, you won’t be 
penalized or lose any benefits for which you otherwise qualify.   
 
What if I decide to withdraw from the study? 
 
You may withdraw by telling the study team you are no longer interested in participating in the 
study.  
 
WHAT IF I HAVE QUESTIONS? 
 
We encourage you to ask questions.  If you have any questions about the research study itself, 
please contact: Rebecca Vitale, (314) 935-8025.  If you feel that you have been harmed in any 
way by your participation in this study, please contact Rebecca Vitale, (314) 935-8025. 
 
If you have questions, concerns, or complaints about your rights as a research participant please 
contact the Human Research Protection Office at 660 South Euclid Avenue, Campus Box 8089, 
St. Louis, MO  63110, 1-(800)-438-0445, or email hrpo@wustl.edu.  General information about 
being a research participant can be found on the Human Research Protection Office web site, 
http://hrpo.wustl.edu.  To offer input about your experiences as a research participant or to speak 
to someone other than the research staff, call the Human Research Protection Office at the 
number above. 
 
 
This consent form is not a contract. It is a written explanation of what will happen during the 
study if you decide to participate. You are not waiving any legal rights by agreeing to participate 
in this study.