Sustainability research: A

review of concepts, concepts, methods, methods,

and findings

Shannon Shannon Wiltsey Wiltsey Stirman Stirman, PhD

VA National Center for PTSD and Boston University; VA Ce

Organization, Leadership, & Management Research
Acknowledgements

Collaborators:Martin Charns, D.B.A.John Kimberly, Ph.DAmber Calloway, B.A.Sarah Beehler, PhD

Frank Castro, Ph.D.Natasha Cook, B.A.

Funding and support :NIMH (R00 4MH 010800; T32MH019836)

NIMH and VA QUERI‐funded Implementation Research Institute
Poll Question #1

• What is your primary role?

– Student, trainee, or fellow– Clinician– Researcher– Manager or policy‐maker– other
Poll Question Question #2

What is your experience withimplementati / on sustai bili na ty projects or rese

‐Current/prior Current/prior implementation implementation (not sustain

sustain

research‐Current/prior Current/prior research research on sustainability sustainability

‐Planning to conduct research on sustainab

‐Involved Involved in an implementation implementation project project (no

research)‐None of the above
Sustainability as a key implementation

construct• Successful Successful implementation implementation doesn’t guarantee guarantee that 

or intervention will be sustained

• P li o cymakers who invest in implementation expect 

effective practices will be sustained

• There h b as een reli l at ve y li l tt e emph i as s on sustai 

n

the implementation literature

• To date, we know very little about how to promote

sustainability

• The study of sustainability presents numerous cha

(conceptual and methodological)
Overview

• Background and considerations• What do existing conceptualizations tell us about c

elements for sustainability?

• Wh t a meth do s have been used to st du y sust i a nab

• What influences influences on sustainability sustainability have been ident

ident

research and conceptual literatures?

• How do the research findings overlap with existing

conceptualizations?
Defining sustainability

• A very basic definition: the continuation of p g ro ram

practices, or interventions after initial implementaor funding have ended

• A program or intervention can be considered to be 

after initial implementation support has been with

‐ Core elements are recognizable, or delivered 

sufficient level of f di elity or intensity to yield d

benefits

‐ Adequate capacity for conti ti nua on of these

elements is maintained
Sustainability outcomes

• Continued fidelity to core elements• Sustaining Sustaining program activities• Maintenance Maintenance of desired desired health benefits benefits• The extent, nature, and impact of modifications modifications to 

peripheral elements

• Maintenance of capacity to function at the require

maintain desired health benefits
Stakeholder Stakeholder goals for sustainabilit

sustainabilit

• What are the “bottom line” goals of stakeholders a

levels?

• Is maintenance at the same level acceptable, or m

outcomes outcomes be improved improved upon?

• Is implementation fidelity valued?

• Would continuation of some components but not 

acceptable?

• Is adaptation required?

• Under what circumstances is discontinuation or

implementation of a new practice desirable?
Conceptualizations of sustaina

A consolid ti a on an

d synthes

is
Rationale

• Existing Existing conceptualizations conceptualizations are designed designed fo

particular innovations or fields

• Variations in terms and categorization of co• All contain contain some related related, overlapping overlapping conce

conce

none include all identified concepts

• Consolidation and synthesis can promote a 

understanding understanding and guide research research and

implementation efforts
Method

• 27 conceptualizations found through a liter

review

• Each unique element was identified from e

conceptualization

• Elements were sorted into groupings of sim

concepts

• Concepts were grouped into broad categor
Preliminary findings

• Five Broad C t a egories:

– Outer context (e.g., policies, system support)

– Internal conditions (e.g., leadership support, cli

– Resources (e.g., Funding, workforce)

– Processes (e.g., training, feedback, adaptation)

– Intervention characteristics (e.g., fit, effectiven
Remaining questions

• Do the identified influences impact sustaina

d ff i erently th h an t ey impact implementatio

• How do these influences interact with one 

over time, and how does this impact sustai

• Which are most critical? critical?• Can some compensate for deficits in others


Processes

Intervention

Resources

Context
Challenges in Research on Sustainability

• Definitions: Multiple ways of conceptualizing susta• Variability of innovations: A study on electronic me

records may look different from the study of a men

treatment

• Funding for research: Without planning, funding ca

research on implementation is completed

• Measurement: How do we assess sustainability? N

of key constructs
Challenges Challenges in Research Research on Sustainability Sustainability

• Timing: Retrospective studies often lack appropria

prospective measures. When does implementation 

sustainment begin?

• Dynamic Systems : Potential influences may chang

and can’t always be accurately characterized at a stimepoint

• Interrelated Constructs: Potential influences are in

and may mutually influence one another

• Interpreting Results: What are the implications of 

sustainability? To what do we attribute success/fai
Initial Initial Review of Sustainability Sustainability Research Research

• To understand the “state of the science”

• To describe:

– The methods used to examine sustainability thu– The way authors authors have defined defined sustainability sustainability– The types of outcomes reported– Findings Findings related related to influences influences on sustainability sustainability

• To develop develop recommendations recommendations for future res

res
Method‐Search Procedure

• Searched databases, employed a snowballin

strategy strategy to search the literature literature in healthcar

healthcar

health, prevention and health promotion, eand business

• Inclusion criteria:

– Identified Identified a post‐implementation implementation outcome outcome or ex

ex

factors associated with sustainability

– Peer reviewed research (no “lessons learned” re– No longer receiving funding/support from origin– Sufficient information to determine timeframe, 

st t a us
Search Procedure Procedure and Results Results

• One coder identified potentially relevant pape

searched full text

• 70% were independently screened by two cod

agreement on inclusion

• 460 potentially relevant studies

– 301 excluded

• Focused on implementation

• Insufficient information

• Narrative/Lessons learned• Timeframe unclear

• 125 studies reviewed
Coding procedure procedure

• Initial coding scheme based on conceptualizatio

implementation and sustainability

• Additional codes generated deductively

• Related constructs identified and collapsed into 

categories

• 65% of papers coded by two raters

• Cohen’s kappa .85‐1 for broad categories, .61‐1 

specific categories

• Disagreements resolved by consensus, consulta

other authors where necessary
Results-Definitions and terms

• Focused on sustainability

– Yes‐102– No‐23

• Defined sustainability

– Yes‐36– No‐80– Cited multiple definitions; didn't specify an operational definition‐9

• Term used:

– 

Sustainabilit

y‐77

– Long‐term/follow‐up implementation‐12– Institutionalization‐6– Durability‐3– Discontinuation‐1– De‐adoption‐1– Maintenance‐1

– Sustained/continued implementation‐1

– Routinization‐0

• Definition cited

– Created definitions‐8

– Scheirer ‐6

– Shediac‐Rizkallah and Bone‐4

– Glasgow et al. ‐

2

– Pluye et al.‐2

– G d oo

dman an

d

Steckler ‐2

– Other ‐12
Results Results-Timeframe 

Cdd o e for l t as post‐implement ti a on time peri 

reported:

– 64% at 2 years or more

– 16% at 12 months

– 12% at 12‐23 months

– 6% at less than 12 months
Results Results-Area of Area of Study 

33% medical interventions/healthcare 34% public health or health promotion pro27% mental health treatments 9% school based interventions 

7% educational interventions 

72% examined either p g ro rams or 

multi-component interventions 
Results Results-Methods and Design and Design 

54 % Quantitative

22 % Qualitative

23 % Mixed

6 % Experiments (e.g., training conditions)

94 % Naturalistic (generally post‐hoc)

7% Followed up on implementation after clinical trial43 % Self‐report

40 % Interviews

43 % Observation

19 % Fidelity monitoring
Results Results-Unit of Unit of Analysis Analysis 

54 % Multiple implementation sites16 % Within systems/communities systems/communities12 % Provider level

9 % Single site6 % Providers within site4 % Team‐level
Results Results-Outcomes Reported Reported 

‐Continuation/Discontinuation‐Presence/absence of indicators

(e.g., key positions staffed; space allocate

‐Fidelity or integrity

‐Full implementation vs. components

‐Sustained impact/effectiveness

Results Results-Associated Factors Factors 

• 68 studies

• Most were not guided by a model

• Four broad categories of i fl n uences

– Characteristics of the Innovation– F t ac ors rel t a ed to the Organization/Context– Capacity (Internal and External)– Processes Processes that facilitate facilitate sustainability sustainability

• Qualitative studies identified processes most c• Quantitative studies identified capacity most c
Findings from review of research
*

Findings from review of conceptualizations

Construct n Construct 

Outer context/external conditions Processes Policy/environmental context (system supports policy) Policy/environmental context (system supports policy) * 13 Collaboration between key stakeholders 

(support/commitment/participation) * 

Inner context/internal conditions Training/HR development* Leadership support* Alignment/Integration (program Alignment/Integration (program-need/priority alignment) need/priority alignment) *

17 12 

Evaluation (outcome)* 

Organization/community structure/procedures/policies* 11 Planning for sustainability 

Climate/culture/social context* 10 Monitoring/fidelity (performance)* 

Role clarity 9 Communication Communication* 

Readiness 6 Feedback (employees/communities; resear

reinforcement) 

Motivation 6 

Lack of opp osition 2 Adaptation (of intervention)* Resources Publicize results burnout); workforce (qualified, competent, self-efficacy); champions* Resources (human); workforce (low turnover, stable, low 

17 Intervention attributes 

Developing local expertise 

Intervention effective Intervention effective * 

Funding* champions 

16 Fit/compatibility* 

Resources (general, material)* 14 Adds value/cost effective 

IT t / i t IT systems/ongoing support 8 Flexible 

Allocated time 3 Complexity/added burden 

*Findings from this review comparable to those from Stirman et al. (2012). 
Recommendations

• Identif

y

a workin

g definition of sustainabilit

y

• Define key sustainability outcomes

• When studying determinants of sustainability, iden

multilevel model

• Assess at multiple timepoints

• Observation Observation or fidelity fidelity ratings ratings• Study both fidelity and adaptation/modification

• Consider

a mixed

‐methods strategy
Limitations Limitations 

• We cast a broad net• May have missed studies d eu to variation ariation in 

reporting

• The sample was not limited to EBPs in healt

• Variety of methods used precluded the use 

analytic strategies

• Conclusions about the level of sustainability 

be expected were not possible due to varia
Key findings

• Fully sustaining interventions was not common• Few studies examined influences at multiple levels 

influence

• Relatively little research has been conducted on ad

and how it impacts outcomes of interest

• Process‐oriented findings were common in qualita• Both f h o t e reviews identifi de si il m ar i fl n uences on

sustainability
Conclusions

• Sustainability has been defined, assessed, a

in a number of ways

• Some of this variety is due to the variety of 

• We need better measurement, use of mode

standardization of terms to guide research

• There is still an important role for qualitativ

• Efforts Efforts to develop develop a consolidated consolidated conceptua

conceptua

facilitate standardized measurement, interp

findings findings and shared understanding understanding of conce

conce
o a e a easu s ste ce o e tat o Se es e Sc 3

References References and Suggested Suggested Readin

Readin

• Bowman C, et al.: Measuring persistence of imp el mentation: QUERI Series. Imp el ment Sci 2008, 3( 

1 

• Shediac‐Rizkallah MC, Bone LR: Planning for the sustainability of community‐based health program

frameworks and future directions for research, practice and policy. Health Educ Res 1998, 13(1):87‐

• Scheirer Scheirer MA, Dearing Dearing JW: An agenda for research research on the sustainability sustainability of public health programs programs. A

101(11):2059‐2067

• Racine DP: Reliable effectiveness: a theory on sustaining and replicating worthwhile innovations. A

2006, 33(3):356‐387

• Gruen RL, et al.: Sustainability science: an integrated approach for health‐programme planning. La

372(9649):1579

• Scheirer MA, Hartling G, Hagerman D: Defining sustainability outcomes of health p g ro rams: Illustra

survey. Eval Program Plann 2008, 31(4):335‐346

• Scheirer MA: Is sustainability possible? A review and commentary on empirical studies of program 

Eval 2005, 23:320‐347

• Stirman, S.W., Kimberly, J., Cook, N., Calloway, A., Castro, F. & Charns, M., (2012). The sustainability o

innovations: A review of the empirical literature and recommendations for future research. Impleme
• Questions?
Thank you!

Contact: sws@bu.edu or

shannon.wiltsey‐stirman@va.gov