QUERI Implementation Research - 18 -Department of Veterans Affairs

Sustainability of Evidence-Based Practices: Concepts, Methods and Findings

**Transcript of Cyberseminar**

**Department of Veterans Affairs**

**QUERI Implementation Research**

**Sustainability of Evidence-Based Practices: Concepts, Methods and Findings**

**Shannon Wiltsey-Stirman, PhD**

**June 14, 2012**

Moderator:  We are at the top of the hour now, so I would like to introduce our speaker for today.  Dr. Shannon Wiltsey Stirman is a clinical psychologist in the Women's Health Sciences Division of the National Center for PTSD and an assistant professor in the Division of Psychiatry at Boston University.  Her research focuses on the implementation and long-term sustainability of evidence based practices in public sector mental health settings.  Training and consultation, treatment and modification of treatment severity and modification and the implementation for treatment for trauma, depression, and suicide prevention.  Dr. Stirman recently completed a fellowship at the NIMH and VA funded implementation research institute.  

And with that I would like to turn it over to you, Shannon.  Are you prepared to share your screen?

Shannon Wiltsey-Stirman:  I am.

Moderator:  Thank you.  

Shannon Wiltsey-Stirman:  Okay.  Well, let me see if I can—are you all seeing my—oh, there we go.  Let me minimize that.  Okay.  

Moderator:  Thank you.  

Shannon Wiltsey-Stirman:  Can everyone see the screen?  Great.  Well thank you very much for inviting me to speak today.  Today I'm going to be providing an overview of key reviews that we've done or one is actually in progress so I'll share some early findings on sustainability research and models of sustainability.

I'd like to acknowledge several collaborators who have been working with me on this project including Martie Charns and Sarah Beehler from COLMR and John Kimberly from the Wharton School, Amber Calloway, Frank Castro and Natasha Cook.  

And Molly told me a little about those of you who are participating.  But I'd like—and a little bit about me—but I'd like to find out a little bit about who is participating so that I can—to make sure that I kind of gear this towards the people who are in the roles that are listening.

So there is a poll that I think Molly was going to put up.  So if you could just click on the buttons for your primary role.  

Moderator:  Thank you, Dr. — or thank you Shannon.  The poll is up and we have received 55 percent of respondents loading.  The responses are coming in very quickly.  For those of you that are only joining us via telephone, the answer choices—the question is "what is your primary role?"  And the answer choices are: student, trainee, or fellow; clinician; researcher; manager or policy maker; or other.  

We have had about 80 percent response rate thus far and responses are still coming in so I am going to leave this open for just a few more moments.  Thank you to all of you who are joining us  and please do click on the circle next to one of the five answer choices that best describes your primary role.

And it looks like the responses have slowed down, so at this time I am going to close the poll and share the results.  Dr.—uh, Stokeman, I'm sorry, Shannon?  Can you see that?

Shannon Wiltsey-Stirman:  I actually can't right now.

Moderator:  One second.  

Shannon Wiltsey-Stirman:  Okay, so it looks like 11%  are students, trainees, or fellows; 13% are clinicians; 50% are researchers; 17% are managers or policy makers; and then 13%  responded "other."  

Shannon Wiltsey-Stirman:  So that's very helpful.  Thank you.  We have one more poll and then we will get on with the presentation.  But I wanted to, let's see, the next poll is—I'd like to find out about the audiences experiences with implementation and sustainability projects or research.  So we have five choices:  Current or prior implementation research but not sustainability; current or prior research on sustainability; planning to conduct research on sustainability; involved in an implementation project or program; not conducting research or none of the above.

Moderator:  Thank you, Shannon.  We have had about 54% response rate and the respondents are still actively responding.  So we are going to leave it open for just a few more moments and then we will share the results and talk through them.  Thank you, everyone, for joining us and for taking the time to respond to this poll.  It really does help guide the context or the level of the presentation. 

 And we have had about 80% response rate so I am going to leave it open for just a few more seconds for those of you just joining us.  And this is a friendly reminder that during the presentation if you would like to submit questions, you must do so in writing using the question function on the Go To Webinar dashboard located on the right-hand side of your screen.

And the responses have stopped coming in so I am going to go ahead and close the poll and share the results.  And Dr. Stirman, you should be able to see those now.

Shannon Wiltsey-Stirman:  Okay.  So it looks like about 30% are currently conducting or have conducted implementation research; 7% are conducting research on sustainability, and 15% are planning to.  That's great to hear.  About almost 30%, 29%, are involved in an implementation project but not conducting research and the remainder, 19% selected none of the above.

Moderator:  Thank you.

Shannon Wiltsey-Stirman:  Okay, so, now to get started.  I guess some reasons to think about studying sustainability and some things that got me interested in the topic.  Okay so there's some increasing evidence that successfully implementing a program or practice doesn't actually guarantee that it will be sustained over the long term.  But that certainly is not what policy-makers who invest in implementation would expect.  I think that there's an expectation that once we start a practice or a program that it will continue until there is a reason to sort of actively discontinue it.

But I think more and more we're finding out that it's actually quite challenging to actually continue interventions or programs over the long term.  Despite that, there's relatively little emphasis on sustainability in the literature.  So we know very little empirically about how to promote sustainability.  

And one of the reasons that we might not know very much and why there may not have been a lot of research in the literature is that it's actually fairly challenging to study sustainability for a number of reasons.  There are some conceptual and methodological challenges which we shall discuss.

So in today's talk I'll just give a little background and we'll discuss some of these different challenges and important considerations in sustainability research.  I'll share the results—some early results of a review that Sara Beehler and Martie Charns and some others—Amber Calloway and I are doing on conceptualizations of sustainability.  

We'll look at methods that have been used so far to study sustainability and what these studies have found so far about influences upon sustainability.  And then we'll discuss how the research findings actually overlap with the conceptualizations and where we need to see more research.

So a very basic definition of sustainability is the continuation of programs, practices, or interventions after initial implementation efforts or funding have ended.  But to flesh that out a little better, it probably—it's probably better to think a little more about what we mean by that.

So one definition that we propose is that a program or intervention can be considered to be sustained after an initial implementation support has been withdrawn if the core elements are recognizable or delivered at a sufficient level of fidelity or intensity to yield the desired benefit.  And if adequate capacity for continuation of these elements is maintained.

Now just a word about why we would say that core elements are recognizable or delivered at a level of fidelity.  For some programs there are programs and probably some interventions where the work hasn't really been done to identify what the key components are that are most closely associated with the desired health outcome.

So in those cases—there are many including Caroll's Framework for Fidelity that suggest that in most cases one should try for full fidelity.  But then there are other things more like public health programs where it might make more sense to make sure that the elements are recognizable.  For example if there are some policy changes, if there are some PR efforts, et cetera.  Those are harder things to measure in terms of the fidelity.  So in those cases it might make more sense to just make sure that the core elements are recognizable.  But we'll return and talk a little bit more about fidelity later.

Some of the sustainability outcomes that have been discussed in the literature and Marian Scheirer actually did a survey with stakeholders to see what—how they thought about sustainability outcomes.  So some of these come from her review and then others come from some of the models and some of the other papers that have been out in the literature.

So some of these outcomes are continued fidelity to core elements, sustaining the program activities, maintaining the desired health benefits.  So really sustainability of the health outcomes.  The modifications and adaptations that are made and the impact of those modifications.  And then the maintenance of the capacity to function at the required level to maintain those benefits.  So those are some of the different ways that the research and the literature thus far has looked at sustainability and some of the ways that it's been suggested.

But when you're either conducting research or an evaluation and you're trying to select which of the outcomes are most important, I think it becomes very important to look at the different stakeholder goals for sustainability.  So trying to identify what the bottom line goals for different stakeholders are at different levels.  

You know, at one level, at a policy level, the most important thing might be to make sure that a particular treatment or an intervention is available. But for others it might be that—to actually make sure it's being delivered to all the eligible patients.  But then for others it might be to be able to get the intervention delivered and the amount of time allocated to see each patient.  So  I think it becomes important to look at the different goals of stakeholders.  And particularly when you're involved in an implementation program to understand how those might conflict or complement one another.

But then when choosing the outcomes to look at for evaluation or for research some questions to ask are whether maintaining the intervention or program at the same level is acceptable.  Or whether there is actually a goal to continue to improve upon this outcome.  Additionally finding out whether implementation fidelity is valued.  It might be valued at one level but not at others.  And I think that becomes important to understand.

There is evidence that fidelity or a certain level of fidelity is necessary that I think is very important to make sure that that information is communicated to stakeholders and to find ways to kind of gather some consensus around that goal.

But additionally, you know some questions are would continuation of some components of the intervention or program be acceptable but not others?  Whether the intervention or program actually needs to be adapted to fit the needs of a particular context.  And then when might it be acceptable to discontinue the program or implementation?

So, you know, the flip part of sustainability is that it can become entrenchment.  If we have a program that's really the best practice today but within, you know, in two decades it hasn't changed.  That might actually not be desirable.

So first I'll give a little bit of an overview of some of the thinking that's been done on sustainability.  And this is a review that we have in progress; we're starting to consolidate the results.  The way that we organize some of the results might change, but I wanted to share with  you kind of where we are with this right now.  

So we found probably close to about thirty conceptualizations or models of sustainability in the literature.  These are not implementation models, they are models that specifically address sustainability.  And many of them are designed for a particular type of program or innovation or for a specific field.  

And there's definitely overlap in terms of the constructs and the potential influences that they identify.  But none seem completely comprehensive.  So we wanted to really take a look at the different models and conceptualizations that are out there and try to do a consolidation and synthesize those to—so that we could really make sure that we were looking at all the elements that are hypothesized to be important in terms of promoting sustainability.

So thus far we've looked at 27 conceptualizations that we found through a literature review where we used a snowballing method.  We also did searches from the databases that we've actually looked in in our implementation science review.  Although that was for the—the review of the research we used the same process.  We'll talk about that a little bit later.

We identified each unique element and then we sorted them into groups with similar concepts, and then we grouped those concepts into broader categories.  So at the broadest level, the broad categories that we had identified were very similar to those of implementation  models that are out there right now.  

Outer context; and these are things like political support, policies, kind of broader system or community level factors.  Internal conditions—this is kind of the inner context; leadership support, climate, culture, etcetera.  Resources such as funding and workforce availability.  Processes such as training, feedback,  adaptation.  And then the characteristics of the intervention.  And the models in particular highlight things like fit and effectiveness that are probably particularly important, you know, over the course of the life cycle of the implementation.  That, you know, they need to continue to sustain the fit and be effective in those in the context where they're being implemented.

But there are a number of questions that really remain, even after we visit review.  So, one question that our group really still has is do the identified influences impact sustainability differently than they impact implementation?  And, you know, that's something that we're going to need to do some research on. 

But some of the models don't go into a lot of details about differences or, you know, whether particular influences become more salient, you know, the further out you get from the initial implementation.

We also don't know—a lot of the models don't really speak about—or don't have a lot to say about how the influences might interact with one another over time and have it impact sustainability.  Some of them do actually, you know, describe that there is some interaction, but they're not highly specific about how.

We also still don't know whether there are particular influences that are the most critical.  And whether some can—having very strong representation of some constructs or factors can compensate for deficits within others.  

So in other words what we really still don't understand is do the models look more like this where if you have a number of different elements in place, that if one decreases.  If they, for example, if there are fewer resources, that they can continue to support the program or intervention.

It could also look more like this where, you know, over time if the political support erodes, the resources will disappear, and then over time the processes that were in place to support the intervention will also no longer be in place.  And that that will make it very difficult for the program or intervention to remain.

But there are some others.  Gruen's model, which is not very specific—well it's not specific at all about implementation drivers itself, but some models do suggest that there are actually—the systems that these programs and interventions are in are very dynamic in nature.  And so as one changes and evolves, it's going to influence others.  

So, you know, the processes of things like adaptation will change, there will be changes in resources, and that in and of itself can change the intervention—what it looks like.  So, for example, if over time you have more people coming into a system that need an intervention but the resources remain flat or actually decrease.  So more people need the intervention than something like an eight-session individual's psycho-social treatment be turned into a four-session group treatment.

And what the research still hasn't told us is whether and to what extent those types of changes will continue to support and promote the outcomes that the original interventions did.  So there's a lot of room for research in these areas.

As I mentioned earlier, one of the reasons why we might still not found the answers to some of those are the challenges that are inherent in conducting research on sustainability.  

Sustainability has been defined in a number of  ways and as such you'll see what's meant by "sustainability" hasn't always been clear in the research that's been conducted.  Additionally, there is such variability in the types of innovations that have been studied and that need to be studied that the results from one study might not generalize to another.  And the way that a study is designed might look very different.

So a study on electronic medical records might look very different from a study of mental health treatment.  And different resources and different contextual factors might be more or less important depending on what you're studying.  Studying for research I think is typically—with the timelines depending—typically really only allowed for research on implementation.  So without planning a follow-up grant for example, the funding will end after a research on implementation is completed without really allowing time to assess sustainability.

Additionally I think that the field of implementation science is certainly making strides in terms of developing measures of a key construct, but we don't really have measures—we don't have a measure of sustainability.

There have been some that have been developed that I think more work around the psychometrics than how applicable they are to the program that could be done. But we still really need to develop measurements of some of the key constructs at least and determine how to measure and assess things like the sustainability outcomes that we described earlier.

  Additionally,  timing is challenging for a couple of reasons.  First of all, you know, it seems like some of the studies and the literature have kind of gone back and looked after an implementation has been completed or after the support has been withdrawn.  That because they were designed retrospectively they didn't really get the appropriate measures.  So planning and timing, you know, when to actually measure the constructs that are of interest is important. 

   But then also conceptually trying to identify when implementation ends and sustainability begins is another challenge.  There are some models that really talk about how things that happen very early on in an implementation process can have a big impact on sustainability later.  I think that certainly can be the case.  But also there have been models of implementation, sort of the phased models, that suggest that sustainability might begin in about two years.  But even those models kind of acknowledge that that's something of a line in the sand.

Additionally as the kind of potential models that I showed you earlier indicate, the—well they're interrelated constructs.  So potential influence is—may influence one another.  It may also influence the intervention.  But also these things are happening within a dynamic context.  And potential influences that we might need to measure will most likely change over time.  So trying to measure at one time point might not really give us the whole picture of what's going on and how the differences and changes in terms of influences on sustainability actually impact the intervention over the long term.

And then finally as you'll see a little bit later, some of the research has indicated that often full sustainability isn't achieved.  And terms like partial sustainability or sustainment of components has been achieved, but we don't really know what the implications are of that.  So we don't really know how that might influence health outcomes.

And additionally we don't yet know how to attribute—or what we should attribute the success of failure to.  Because although some of the studies have listed influences, again it's really too early to generalize.  So we were curious to see what the state of the science was in terms of the research that has been conducted on sustainability.  So we did a literature review which was published in Implementation Science I think in January.

And the goal was really to describe the methods that we—that have been used so far to examine sustainability, how it's been defined, the types of outcomes reported and then findings related to the influence upon sustainability.  And the goal was really to figure out where we need to go next in terms of future research.  How can we design future research, what types of things do we need to be studying, etcetera.

So we conducted a search procedure which was somewhat challenging because there are so many different terms that have been used.  And the literature is spread over a number of different fields.  So we searched databases, we used the snowballing strategy, we also gave the list of papers that we had found to an implementation expert and asked them how much they knew about our papers that we had missed.

The inclusion criteria included the identification of a post-implementation outcome or the examination of factors associated with sustainability.  We only looked at peer reviewed research for this .  And we also didn't include papers that were kind of lessons learned papers unless there was some sort of  analytic strategy applied.  Like a—if it was a systematic case study we would use that but the papers that just described efforts to implement and sustain a program and what happened we did not include.  The studies that the projects had to be—had to have had initial implementation support removed.  

And we had to have sufficient information to determine things like time frame and funding status.  So we had one coder identify potentially relevant papers, then they were screened by two coders.  We had high agreement on what should be included.    

We found about 460 potentially relelvant studies.  that was after screening out all of the search terms or all of the results that were talking about environmental sustainability and other uses of the word sustainability. 

Some of them were models of sustainability and things like that that we included in the other review, but we excluded a number of them for having a focus that was really on initial implementation without information about sustainability.  We also did not include studies that talked about prospects for sustainability.  They actually had to make an effort to measure actual sustainability.

If there wasn't sufficient information about funding or timing some [inaudible]. And then we ended up with 125 studies.

So we came up with an initial coding scheme based on implementation concepts—basically implementation models that were out there.  We tried to identify the different constructs that have been identified as important for implementation.  And then we generated additional code deductively when necessary.

We collapsed related constructs into more general categories.  And we had two writers code more than half.  We had high agreement for the broad categories and still we had agreement for the more specific categories.  There were just a few where agreement was a little lower and then those pieces we resolved disagreement by consensus and consulting other authors where we needed to.

So just a kind of a quick overview of what was out there.  Most of the papers did focus on sustainability, but there were some that were follow-ups from clinical trials meaning that they had implemented intervention in a particular setting and then they went back later to see if people were still doing it for example.  Or there were some papers that focused more on implementation but they provided long-term—outcomes over the longer term after the initial implementation support had been withdrawn.

We were a little bit surprised because only 36 of these studies actually provided an operational definition of sustainability.  The rest kind of used the term but there was something of an assumption that there was a shared understanding about what that meant.  And then they would usually identify one sustainability outcome on that list I provided and we'll provide some data on that.

Most of the papers did use the term sustainability but then a number of other terms that have been used in the literature were also found in these studies.   And then there really wasn't a lot of consensus around definitions either.  Probably the most commonly cited when you combined them were Scheirer's and the Shediac-Rizkallah and Bone—I don't actually know how to pronounce that last name—but those two are very similar.  And in fact Scheirer's cites the earlier literature from Shedica-Rizkallah and Bone.  So those are probably the most common.  Some created definitions and then there were a few others that mentioned the reviews.

So in terms of time frame most stuck with the two-year convention.  But about I guess 28% looked somewhere between 12 and 20 months.  We had a few studies, about 6%, that looked at less than 12 months.  And usually those were the sort of outcomes that were maybe a little underwhelming.  So, for example, there was one paper that showed that at that point somewhere between 30 and 40% of the clinicians who were trained for implementing an intervention were salaried.  So that would probably give us some idea of what the longer-term prospects were so we included those papers for that reason.  But most of the papers did stick with the two-year convention. And some of them actually followed things out for decades later which was very interesting.

The areas we studied—the field were relatively even between healthcare and medical intervention, public health or health promotion, and mental health treatments.  We did look at school-based interventions as well and most of those were either related to mental or physical health.  But we looked at educational interventions in addition because the education field struggles a lot with the same topic.  How do we, you know, how do they get curricula implemented and how do they make sure that teachers continue to deliver them at a high level, for example.  So we wanted to include those types of things because if there were something to learn from that literature, we wanted to make sure we could capture it.

We also looked at the business literature and management literature but didn't really find as much that was actually kind of empirical research.  But most of the innovations that were examined were either programs or multi-component interventions.  So we were looking mainly—we ended up looking at mainly more complex interventions and programs with multiple parts.

In terms of how the studies were designed, just over half were quantitative in nature, but then about 22% were qualitative and then 23% were mixed methods.  Most of the studies, 94% were naturalistic studies, generally sort of post-hoc studies after an implementation effort.  But there were a few experiments mainly around things like which training condition resulted in highest fidelity over the long term, things like that.  And 7% followed up on implementation after clinical trials.  

And then in terms of the way that either the sustainability outcomes or the constructs that were potential influences on sustainability were examined, the majority were self-report measures or interviews. So we had 43% that were self-report, 40% were interview, 43% included some form of observations, but we were somewhat broad in terms of how we thought about observations and that could include a chart review as well.  And only 19% had some type of fidelity monitoring.  

So I think that that's an area where future implementation research could really develop in terms of having some kind of external observation or some way of actually evaluating whether the original program or intervention was actually being implemented the way it was designed.  Or being able to observe modifications that might be happening either planfully or naturally.  

Because the self-report data I think can provide some very rich contextual information but it's also—there might be things happening that we don't capture because sometimes I think clinicians might endorse that they're doing a program or practice and they might in fact be implementing it fairly well, but the fidelity might be slipping in other cases, or they might be making some adaptations that they're not aware of that could be really interesting to capture.

In terms of units of analysis, just over half of the research focused on multiple implementation sites.  And about 16% looked at programs or interventions within systems or communities.  12% looked at the provider level. And then the rest were either single sites or looking at providers nested within sites or at the team level.

The types of sustainability outcomes that were recorded most commonly was kind of probably continuation or discontinuation.  But often what was recorded instead of questions like is the intervention actually being delivered or does it continue to be delivered—some of the studies looked at the presence or absence of indicators.  So rather than observation or accepting you know are you still doing this—the program or the intervention.  They looked at whether key positions were staffed or whether space was allocated; basically whether the capacity was there.

Fidelity or integrity somewhat less often.  Some of the studies looked at full implementation versus partial or components.  And then some looked at sustained impact or effectiveness.  So whether or not the desired health benefits—or benefits of the interventions or programs had been continued.  And actually we presented preliminary data on this I guess a little over a year ago at the NIH implementation conference.  And initially we had not found very many papers at all that looked at sustained impact.  But just in the last year—we continued the lit search after that—and ended up finding many more.  

So in the last couple of years it seems that this is an area that has received more attention.  So I'm not going to dwell on this figure for too long, but a couple of things that I wanted to point out here that are somewhat interesting.  So we divided finding by field; medical, mental health, or public health or health promotion.  And more rigorous methods were those that included some sort of observation or fidelity monitoring.  Less rigorous were more kind of self-report studies.

So really across those types of methodologies or the different methodologies that were used.  One thing that's interesting is that when these studies reported on full sustainability, there really are not a lot of studies that achieve high rates of full sustainability.  What was a lot more common were things like low sustainability, partial sustainability.  So it's possible, I mean it probably wouldn't surprise a lot of us when we're looking at complex interventions—but it's challenging to sort of fully sustain a complex intervention as it was originally designed.  

So I think that more research can be done to really kind of unpack this and look at what's going on at the different levels; kind of the provider level and the site level.  What are we seeing in terms of how the interventions stay the same and where we see things like partial sustainability or low sustainability what are the impacts on the actual outcomes—the health outcomes that we were most interested in.

So turning our attention now to factors associated with sustainability or potential influences on sustainability, there were 68 studies in our literature that examined this.  And what we found—one thing that was interesting—was that most were not guided by a model of implementation or sustainability.  

So in terms of the way that the studies were either designed or the way that analyses were done or the way the measures were chosen, it didn't seem like there were a lot of studies that were, you know, choosing a theoretical framework and then trying to measure potential influences that had been highlighted in this construct.  I think there were right around  maybe 20 that did and some of those developed models based on very extensive literature reviews or at least had a list of potential influences that they had identified in the literature.  

Others actually identified some sort of a framework.  But the majority didn't which means that many of them didn't actually measure potential influences in the four categories that we found that I have below.  

So one caveat in terms of looking at the results we found is that I think we can't have a lot of confidence that these are really highly generalizable findings or the findings of perhaps like in a stick because some of those—the studies—really didn't look at multi-level influences.

So the four broad categories that we did find some evidence in the literature that these things might influence sustainability were again not that different from implementation models.  Characteristics of the innovation, factors related to the organization or context, capacity, and processes that facilitate sustainability.  So these again are not too different from some of the implementation models that are out there.  It differed a little bit from the way that we organized our findings from the review of sustainability models in that, as you'll see in a minute, the factors related to the outer context really were not as well studied or well represented in the research.  So we just basically collapsed those into the context or contextual factors, rather than outer and inner context.

And again we organized capacity rather than resources.  Capacity for this model is a little bit broader as you'll see in just a second.  But we found qualitative studies, identified processes most commonly, and quantitative studies identified things that are related to capacity most commonly.

So in this table of findings, I'm really just going to call your attention to the first two columns where we look at quantitative and qualitative studies.  And identified innovation characteristics was one.  And the things that seemed most important again, maybe not surprisingly because these are things that probably emerge over time, but still whether the innovation could be modified or whether modifications had been made; so whether the intervention was modified.  And the effectiveness or benefit of those—of the intervention were probably the most common findings related to intervention and characteristics.

In terms of context, setting characteristics were most common in the quantitative studies.  And this was probably because some of these things were more easily measured.  The things like size, number of providers, or number of units and things like that could be kind of more easily measured and plugged into a model.

It was sort of interesting to me that in terms of the qualitative findings, leadership was something that came up a lot and much more so than climate and culture.  So it might be that what we would think of as climate and culture was something that the key informants were actually describing in terms of how leadership approached things or what leadership was reinforcing, etcetera.  

In terms of capacity, workforce really emerged as particularly important not surprisingly.  And this included attributes like their attitudes or how motivated they were, how well trained they were, but then also just basic things like were the key positions staffed.  So those emerged in both qualitative and quantitative findings.  

And then a little more commonly in the qualitative studies were these things that were probably maybe a little bit harder to measure or at least the researchers were not using paper and pencil measures to look at things like community or stakeholder support and involvement.

Another thing that well might be a little bit surprising on the one hand is that funding didn't emerge more often.  Because I think that one thing that many of us think of as being very important for at least certain types of interventions and programs is that the funding to provide the resources remain available or that there is a model to continue funding.  That might not have emerged as much because it's something that maybe wasn't easily measured or was not something that the researchers were measuring.  So then also some of it might have fallen into other categories.  So funding and resources—it might be that in terms of what the key informants found important, rather than focusing on funding they were talking about whether resources were made available, etcetera.

So then as I mentioned earlier in terms of the qualitative findings, you know, for almost all of the studies some sort of processes were identified as particularly important.  Now again these were things that might not have been as easily measured in terms of the quantitative research.  But some of the things that emerged as particularly important were the integration of polices and rules.  So integrating the intervention or the program into the policies themselves of the organization. 

Collaborative relationships and partnerships were mentioned frequently and some sort of ongoing support.  So not only things like training and education some training the workforce and providing evaluation and feedback.  Those emerged as somewhat important though not as frequently as kind of that ongoing support.  But as you'll see only one qualitative study mentions planning for sustainability as important.

Now again, these were not constructs that were measured in every single study.  In fact, most of them were not measured in most of the studies, but this is what's been found so far in the research on sustainability up until—I think we stopped pulling new papers into the review in last August maybe.  So it's a fairly recently, um...

So let's go back to our review of conceptualizations and to kind of compare how these findings stack up with what emerged in the models of sustainability and the conceptualizations.  You'll see where the asterisks are is where we—where our findings aligned with the findings from the conceptualization review. 

So many of the 27 models that we reviewed mentioned the policy and environmental context.  And in fact at least one model says that this might be more important in some ways than the inner context.  So whether there continues to be political support and the funding streams, etcetera.

Leadership support under internal conditions emerged as very—was something that was mentioned in a lot of the sustainability models; 17 of the 27.  Somewhat more so than the climate, culture, and social context constructs although that emerged both in our findings—the research review findings—and the conceptualization findings.  And some of that might have been captured under leadership in some of these models.

So other factors that emerged in both findings as important in both reviews were the alignment and integration—so the way that the program needs and the priorities were aligned with the intervention or program itself.  And then sort of some structural/policy types of things at the organization or community level.

But some of the other things that were mentioned in the model under inner context have not been studied or have not emerged in terms of the research—the empirical findings.  So role clarity within the organization, readiness for change and motivation are things that are mentioned in the conceptual reviews or the conceptualizations but not in the research literature.

And then lack of observation.  I'm sorry—lack of opposition.  At the research level, research is clearly identified as important in the models as well as in our empirical findings.  

What's kind of human resources, you know, workforce characteristics being particularly important and funding maybe not surprisingly.  But then also kind of more general resources like space, material resources, etcetera.

One thing that we did not find in the literature or the research review that we did find in the conceptual review was IT systems, but we did find again ongoing support was important.

And then in terms of processes, stakeholder collaboration emerged as pretty important in the conceptualizations.  Many of them identified them as being important and it was also a finding in our research review.  And then some other processes like training, evaluation, planning, fidelity monitoring, communication and feedback, and adaptation emerges as important.  And I think that this is something—it also emerged in our research finding in terms of—it actually shows up as a process.  And then also in a way, you know, whether or not the intervention can be modified.

I think that there's a recognition that this is important at least in some models, but this is an area that really very understudied.  Right now we're working on a study to characterize the different types of modifications that are made to interventions and a coding scheme around that.  But there really hasn't been a lot done in terms of identifying ways to really characterize and measure adaptations that we can then look and see what the impact is on outcomes.

Intervention attributes, again not surprisingly, it needs to be effective, compatible—those were a couple of the main ones.  But then also it adds value not surprisingly it was something that was mentioned in the conceptual literature.  Flexibility, you know, whether or not it can be modified or adapted in some ways.

So these are things that have emerged in conceptualizations but again you know very few of these conceptualizations if any really talk about the relative weight or importance of these different constructs and potential influences.  And although like I said earlier there's mention of how they overlap and how they might influence one another, there's still a lot of unpacking to do on actually what that process looks like and what the impact ultimately is on sustainability.

So just to wrap up, in terms of recommendations for future research, when embarking on research on sustainability, I would recommend identifying a working definition of sustainability, and really defining what key sustainability outcomes you're planning on examining.  But then also identifying a multilevel model and selecting measures or developing interview questions based on that model so that you capture the different potential influences at different levels.

And then also particularly if you're doing qualitative research or if you have a sample size that's large enough and you can really start trying to look at relationships between those influences and how they might mutually influence one another or how they might interact and change other influences or actually change the intervention in some way.  I think that's going to be really important to look at.

Assessment at multiple timepoints; and that's so that you can capture really what's going on with the different resources and with the different influences at different times.  So rather than just assessing sustainability at one time point, looking at sustainability outcomes and potential influences at several timepoints over a period of years if possible.  As I mentioned earlier, some type of external evaluation or observation; fidelity ratings when possible.  

And then looking both at sustainability and adaptation so that we can start to understand what the impact of adaptation is, impact on sustainability and then also some of the sustainability outcomes like whether or not it continues.  But then also on the house outcomes that are of greatest interest.

And then because we really do need to understand relationships between these different influences, and we need to understand more about why interventions and programs are sustained or are not sustained, consider a mixed-methods strategy so you really can capture a lot of rich detail about the context and about the processes that were in place.

So a couple of limitations to mention, we cast  broad net.  We looked across a number of different interventions and programs and fields.  And we wanted to do that to make sure that we could learn what has been learned in all of these different fields that might apply more to healthcare and mental health.  But because of variations in terms we might have missed some studies.  We also did not limit the sample to evidence based practices in healthcare.  Most of the healthcare interventions that we looked at, if not all, did have an evidence base, but we were really interested in the phenomena of sustainability so that's what we prioritized.

And again, because of the variation in the literature so far and the way that the studies were done at this point it's really not possible to draw conclusions about the level of sustainability that can be expected.  Although it does seem fairly challenging to maintain strict fidelity.  That doesn't mean there shouldn't be a goal of strict fidelity if necessary, but in terms of further generalization I think all I can say is that we need to do more research before we can really conclude that a particular level of sustainability is what can be expected for different types of intervention or in different types of systems.  

So key findings, again, fully sustained intervention wasn't common.  Very few studies really looked at multiple levels of influence or influences at every level.  Not much has been done on adaptation.  Process-oriented findings were common in qualitative studies and I think that we really need to pay attention to that.  The way that the different processes are put into place or the things that naturally happen within organizations can influence sustainability.  

And then were there were some differences, both of the reviews identified similar influences on sustainability, but a lot more needs to be done to understand the relationship between those influences.

So with that I think I will just about—I know we've just got a couple of minutes for questions—but I would say looking at measurement, use of models, and standardization or defining terms is going to be really important in the future.  And I'll just put up a list of—this is on the handout that you can download—but some references and suggested reading if you're interested in learning more about sustainability.  So with that I'll turn it back over to Molly for questions.  And I'll just keep a slide up with my contact information in case anybody would like to back channel me with questions or anything else I can be helpful with.

Moderator:  Thank you so much Dr. Stirman.  That was a very excellent presentation.  And we do have several questions that have come in.  I understand that some of our audience members may have to leave early or leave at the top of the hour, so I am going to strongly encourage you to please when you exit the session, do wait a few moment for our survey—our feedback survey—to load.  And please provide us our feedback because it does influence how the progression of  these sessions go.  And with that I'll go to the first question in the order that they were received.

About definition of sustainability—why is capacity a part of the definition rather than a factor affecting sustainability as defined by continued presence of core elements?

Shannon Wiltsey-Stirman:  I think that's a really good question and I certainly wondered about that as well.  I think that some of that might be because of the different types of programs that have been examined and are being considered in this definition.  So for example if you have a collaborative that focuses on a particular disease prevention or something.  Then what might be more important than actually particular activities where if it's valued that the program or the collaborative will be making changes with the best available evidence or based on the needs of the system, that what's important is the capacity to do that. 

But I think that in terms of healthcare intervention, capacity while important, you're right it's probably more of a predictor.  But I think for other programs the capacity might be more having the framework in place to make the adjustments and continue the program activities is valued more than particular activities in and of themselves.  So I think that's partially why capacity has been considered to be a key part of the definition.  

But that's why I think it's important when conducting research that you consider the type of intervention and really select the definition that's going to be most meaningful and relevant to the intervention at hand.  So for some capacity probably is not important.  You could have a mental health intervention and have the staff hired and have the space to deliver it, but if it's not actually being delivered, people aren't actually doing that, then capacity really doesn't matter.

Moderator:  Thank you for that response.  I just want to let you know, Dr. Stirman, that many people have been writing in thanking you for such an excellent presentation.  And to those people I would encourage you once again to complete the feedback survey once you exit the session.

And our next question—actually I should check in.  Dr. Stirman are you available to continue answering questions for a moment?

Shannon Wiltsey-Stirman:  Yes, I can.

Moderator:  Thank you.  The reason why we do this is to capture them on the recording so that they will be available in the archive which gets posted in 48 hours in the cyber seminar archive.

The next question—could you talk more about the quantitative studies assuming it's not in the following five:  what measures, what methods, none of the methods listed on the general slide sound very quantitative.

Shannon Wiltsey-Stirman:  Yeah, you know, for a lot of the quantitative studies what seemed to be happening was either the quantitative studies were just some sort of measurement of things like percent of providers or degree to which an intervention continued to be delivered or percent of patients who continued to receive an intervention.  So some of them were along those lines, but in terms of the influences, what some of the studies did was basically take—you know, it looked like some of them basically took variables that they were able to measure or identify and kind of plugged them into a model.  And so that might be things like the size of the different organizations or let me try to think of some other factors that might have been plugged in—maybe survey responses around barriers for example.   And then they would associate those, you know, there would be a correlation or they would plug them into a model where they would try to predict sustainability based on those.

So the quantitative methods were—I think there's room to develop a lot more sophisticated studies around implementation, which of course gets challenging because you  need a number of sites often to really be able to do it.  But what we did see were things like that; more correlational.  Or, you know, taking three or four potential predictors and putting them into a model.

Moderator:  Thank you very much for that response.  The next question we have received and please note that these come on early on in the presentation so if you have addressed them, feel free to let us know.  Seeing the definitions used in your lit review was very interesting because as I was listening I wondered how much you had looked at the "scale-up" literature.  And I see that none of the articles you looked at used scale up as a defined term.  I find this interesting since at least in the international health field in which I work, we define scale up as the long-term or sustained outcome/impact of the intervention.  You may want to consider looking at the scale up of literature to find additional data, framework, constructs, etcetera.  And in particular,ExpandNet's website includes a relevant bibliography.

Shannon Wiltsey-Stirman:  Yeah, I appreciate that.  Yeah, I think, you know, scale up is definitely a closely related construct and concept.  And yeah, I think that for some stakeholders and for some types of programs the goal isn't just to continue what's been done but to continue to spread the innovation.  For this review we basically kind of limited it to sustainability just to keep it manageable.  But I agree, particularly for some types of programs and interventions, the scale up and the spread is really considered to be a key part and it is an important literature to consider as well.  And I will definitely be taking a look at that—at the website as well.  You know, again, with the conceptualization I think it would be worth looking at the literature.  Again, we're trying to sort of keep it somewhat manageable but I do recognize that there is definitely overlap and similarity and there are certainly some things that could be learned from that.

Moderator:  Thank you for your response and for that submitter's information.  The next question which you may have covered, what is fidelity monitoring again?

Shannon Wiltsey-Stirman:  So fidelity monitoring is probably more relevant to certain types of interventions than others, but it is essentially a measure of this different kind of core components—and sometimes peripheral components too—of an intervention.  And actually I should probably say that for some interventions there really hasn't been an effort to distinguish core from peripheral components.  Either the theoretically or empirically derived components of the intervention that seem most related to outcome.  

So basically it's a list of the constructs and then either a measure of whether or not those pieces of the intervention happened or took place, or a measure of the skill with which they took place.  So that will be relevant for mental health interventions, some of the checklists that have been studied would probably be considered to be a fidelity measure.  You know, where they're looking at adherents and whether or not certain things were done.

For other things like public health and community level interventions I think that thinking about fidelity it's a little bit different.  It's probably more of whether or not larger program activities were made recognizable and continue to be implemented. 

Moderator:  Thank you very much for that response.  I'm going to skip ahead one question because I think this is very applicable to all of the people still on our call.  This review was very interesting.  When do you expect it to be published?

Dr. Stirman;  So on this—I don't know if my slides are still up but the last—the very last reference on this list of references is the reference for their review of the literature.  So that's already out.  That was published in Implementation Science I think in January. 

And then the Review of Conceptualizations is something that we're writing up right now and we're still kind of organizing some of the results.  So hopefully we'll get that submitted fairly soon.  But I wish I could give you a better timeline than that—a better conceptual overview.

Moderator:  Thank you for that response.  Next question, when looking at—okay, sorry.  When looking at full versus low or partial sustainability, I wonder if it is a really bad thing for partial sustainability to occur.  Wouldn't that result potentially indicate that the intervention was being modified to better fit the context rather than what the intervention developers had developed?  Alternatively, programs just aren't sustaining evidence based practices however I think that it really is crucial to debate whether partial sustainability is problematic.  This is just a comment.

Shannon Wiltsey-Stirman:  Yeah, and I agree.  I think we don't know for a lot of interventions.  I think there are some that we do have a sense that, yeah—for example for hand hygiene if you skip or kind of don't do some of the components well you're probably not meeting the goals of the program.  So if you're, you know,  just running your hands under the faucet without using soap, for example, or if you're immediately putting your hand back on the faucet and potentially re-exposing yourself to bacteria, etcetera, that's probably not—that type of modification which is not exactly what people have in mind when they think about modifications, but they are the types of things that can happen naturally.

In those cases, I think an observer might say, "Well it's being partially sustained."  Some of the things are—some of the elements are in place and people seem to be doing some parts of the intervention.  You know for psychotherapy there are some elements that might not be delivered as originally intended and I think we really need to do more research to find out what the implications are.

I don't think that we can always assume that adaptation is bad.  And I don't think we can always assume that it's good.  And I think we might find it's very different for different types of interventions.  I think that's some research that really really needs to be done.

I agree that sometimes modification can be really important in terms of actually promoting sustainability and making sure that things fit the context and can continue as long as that doesn't happen at the expense of the effectiveness.  Because I think when things get modified to the point where they compromise effectiveness, then that's recognized within the system.  And then it becomes kind of the story of the intervention not working in that context and then it's discontinued.  So I think those are areas that are really ripe for more research and wider debate.  But I really think we need more research to inform that debate.

Moderator:  Thank you for that response.  We do only have five questions pending.  Are you able to stay on and address as many as possible?

Shannon Wiltsey-Stirman:  Yes, I can.

Moderator:  Great, because the submitters are still in the meeting, so I'm sure they are very eager to hear the responses.

Next question as we are doing an implementation/study/project, at what point do you think should sustainability be addressed and planned for?

Shannon Wiltsey-Stirman:  I think it needs to be developed or really addressed as early as possible.  So I think even in the implementation planning stages.  You know, if you're planning to do something to make the implementation work that's not going to be sustainable, then that will really have an impact on sustainability.  So I'll give you an example.  If you're planning on having consultants available to help support the implementation of a particular intervention, but over the long term there is no plan to continue that, then it might make it more difficult unless you can really develop local expertise to be able to really sustain the intervention at the level that you would like.

So early on in the planning stages I think there needs to be some consideration of like, "Well what do we do when these consultants go away?"  "How are we going to continue to support the intervention."  So I think that right during the planning, it becomes important to think about which aspect of this implementation support are we going to be able to continue and how do we plan for it when we no longer can?

In terms of measuring sustainability I think that probably happens later in the game; sometime after implementation support has been withdrawn.  But I do think being able to look even early on at what types of things might be happening early in the process that could influence sustainability could be important.  I think when you're implementing something you can kind of, with enough political support and resources you can make things happen.  

You can get clinicians to training if you can clear time, etcetera.  But over the longer term when that implementation program is no longer in full swing, what do you do when you need to replace people?  You know, are they  going to have time allocated to go and get that training, etcetera.  So those are the types of things that I think the earlier you can start planning for the better.

Moderator:  Thank you for that response.  I am going to preface this question by noting that it is in reference to slide 32,  Table 2—so if you'd like to pull that up while I ask?  It seems like the development or emergence of constructs may be driven more by methodology and choice of measurement more than anything else.  Your Table 2 really seems to illustrate this.  So I wonder if we put some creative thinking into new methodologies or measurements/indicators would be unearth new constructs?  I think it's would unearth new constructs.  Then again, there seems to be a fair amount of repetition around the four or five basic constructs.  So maybe that's just it and it's a matter of how you categorize something.

Shannon Wiltsey-Stirman:  Yeah, I think that's true.  I mean the qualitative studies particularly because, you know, many of them were not, you know, questions weren't asked based on these types of constructs or influences.  They really tended to be very general questions around things like, you know, what barriers were experienced, etcetera.  And about the process of implementation and sustainability.  But there could be some things that would emerge through that that could point to the development of new measures.  But essentially this is what we found with the qualitative studies that used that process.

So it is a question of whether when we're coding these things, if we do it with a particular lens that we're looking for these things that have already come up before.  But again because these studies—the qualitative studies in particular—but really a lot of them were not driven or influenced by implementation models.  It seems like these are the things that emerged.

These are the types of things that the key informants were finding to be important.  But again I think that looking at how we can better measure these things and then seeing if they hold up when these measures are put into place would be important in developing these measures along the 

web line as well.

It would be great if we could figure out, you know, the three key things for sustainability seem to be, you know, one, two, and three; whatever they might be.  But with the methodologies that have been used so far I don't think that we're quite there.

Moderator:  Thank you for that response.   The next question—and we're down to two—in terms of determining which constructs play a bigger or lesser role in influencing sustainability, approximately what percentage of studies compared low sustainability sites with high sustainability sites to see which constructs appeared to help explain this variability?  

Shannon Wiltsey-Stirman:  I don't know the exact proportion.  I know that some did; I don't think it was a huge number.  I was looking back over some of the studies recently.  It's certainly something that some of the qualitative studies did, but in terms of setting it up as a comparison like that, I would say probably fewer than ten just off the top of my head.  I would have to go back and take a look to really be sure about that.  But it wasn't—it was not a really common design and I do think that that's something we could certainly look and do more of in this type of research.

Moderator:  Thank you for that response and information.  It is helpful.  The final question, is there any discussion in literature about whether or how sustainability is affected by additional advancements, for example, new medications for the treatment of heart disease, and at what point sustainability becomes an impediment for further advances?

Shannon Wiltsey-Stirman:  I think that that concern has been voiced, you know, in the literature.  Some people would say that the goal is, for example, for the organization to become a learning organization.  So rather than focusing on specific intervention, what you need to be focusing on is developing the capacity for that organization to be able to recognize and identify when new practices actually need to replace or supplement an existing practice.  But I don't know that a lot's been done.  I think that it's something that's probably been discussed more than it's been studied.

There are some terms like divestment which doesn't always mean, you know, kind of removing one intervention in favor of another.  But the term exnovation is being increasingly discussed and that.  So removing an old practice whether or not it was once the strongest most evidence based practice or not in favor of the new ones.  But that's a term that's being discussed more.  I wouldn't be surprised if we don't start seeing more research on it but so far I don't think that there's really been a lot.

Moderator:  Thank you for that response.  And at this point I would like you to invite—I would like to invite you to make any concluding comments to the audience.

Shannon Wiltsey-Stirman:  Well, I appreciate that you all stuck through the presentation of all of the data.  I'm certainly happy to be in touch with anyone back channel if I can provide additional information.  The paper that was referenced is online at Implementation Science and that's an open access journal.  So it's available to be downloaded.  I think it's ImplementationScience.com.  You know, so certainly feel free to be in touch if I can provide additional information.

Moderator:  Thank you very much.  And at this point in time, I would like to remind our attendees that as you exit the session you will be prompted to complete a survey.  Please do so.  We do take your comments into consideration when planning future sessions.  

And while I have everybody's ear, I am going to plug the next implementation research science session which is taking place on July 10 at 12:00 p.m. Eastern.  It will be presented by   Dr. Shirley Moore.  And she is part of the Edward, Jerry, and Lewis Mellon professor of nursing and an associate dean for research, research case at Western Reserve University.  So please do join us.  Further information will be available but you can always look at upcoming sessions at the cyber seminar catalog.

Thank you so much, Dr. Stirman.  That was a great presentation, and I really appreciate you taking the time to address all the questions.

Shannon Wiltsey-Stirman:  No problem.  I was glad to do it.

Moderator:  Great.  And with that this does formally conclude today's HSI&D cyber seminar.  Thank you all for joining us.

**[End of Recording]**