QUERI Implementation Research11Department of Veteran Affairs

**Department of Veteran Affairs**

**Researching the Context of Programs to Improve Health and Health Care**

**QUERI 3-8-12**

**John Øvretveit**

Molly:  Without further ado, I would like to turn it over to Dr. Brian Mittman at this time.  He is the director of the Center for Implementation, Practice, and Research.  He will be introducing our speaker today.

Dr. Brian Mittman:  Thank you.  I’d like to add my welcome as well to all of you to our March QUERI Implementation Research cyber seminar.  This is one in a series that serves two purposes.  It represents a follow-up to last fall’s Enhancing Implementation Science training program as well as a seminar that we hope is of general interest as well.  The speaker today, John Øvretveit, is an international expert in patient safety improvement, quality improvement, implementation science.  He’s spoken in this series before; we’re pleased to have him with us.  in Sepulveda for the winter months.  He will actually be delivering another seminar next week, so we’re trying to benefit from his time and expertise as much as possible before he returns to Sweden. 

The topic today is the issue of context and contextual influences on implementation.  I think there’s growing recognition and appreciation of the fact that contextual influences in the implementation field, in implementation science, are as important, and in many cases more important, than the actual interventions that we study, the implementation programs and strategies.  The extreme version of the argument states that the main effects of the implementation strategies are so weak that in many cases it doesn’t matter much which strategy one chooses, that the key is to manage context and understand contextual influences and factors.  So with that version of the argument, I’d like to turn the mike over to Professor Øvretveit.

John Øvretveit:  Thank you very much, Brian.  First of all, I want to say a big thank you to Brian for making it possible for me to visit and for hosting the VA Implementation CIPRS Center, which we identified at the Karolinska Institutet as the place to work and learn at.  And I’ve certainly benefitted tremendously from the center.  Today I’ll be talking about issues around context, and I think Brian put it very well.  There were three concepts he mentioned there: intervention, implementation, and context.  Those ideas will come up again.  Sometimes we don’t separate the intervention from implementation.  In the course of this I may need to say a few words about that.

On slide two you’ll see ways in which I’m aiming for you to be different.  In one hour you’ll be better able to explain three designs for evaluating interventions and implementations, frameworks to guide data gathering about context but also about intervention and implementation, and when and why to collect data about context and how to present the data to customers for their use.  And by customers I mean mostly practitioners at different levels of the Veterans Health Administration.  But I recognize also as researchers our customers are reviewers of proposals and publications as well.  That list of what we’re going to be covering, you may well say, “Actually, it’s all a bit back to front, why not when and why to collect data about context?”  In some ways it is, and I’m going to go straight into three different designs and then spend some time on the different frameworks for data gathering because I see this session primarily as a how-to-do-it for researchers and for people interested in this subject.  I know time goes by quickly, so I wanted to give you tools and ideas that you can use before going on to the more philosophical or rationale issues around why we’re looking at context and why there is more interest in looking at context, especially in the implementation sciences.

What I’ll be talking about on slide three is basically three designs for evaluating complex programs.  The first is to use an experimental evaluation together with a process evaluation and to run them in parallel.  And I’ll give you an example of that.  The second set or category of approach is what I’ll be calling a theory-informed, context-sensitive observational evaluation, possibly a case study of an implementation process.  The third is action evaluation, where the researchers give feedback to the people doing the implementation as it goes.  That helps develop the intervention iteratively.

As a result of the experimental outcome, the first one—remember, we’re going to be looking at two things here; we’re looking at two studies: a controlled trial outcome evaluation, and then I’ll talk to you about the process evaluation.  Let me tell you about the outcome evaluation first.  The intervention—it was guideline implementation intervention where you had primary healthcare facilitators, paired facilitators, giving educational support to primary care physicians.  The idea was to increase compliance with a sore-throats guideline and women-with-urinary-tract-infection guideline.  

Now, I’m not going to go into the details of the intervention.  It was done in Norway; colleagues in Oslo did it for 142 primary healthcare centers.  As with a number of interventions of this time, there was little change overall.  There are a number of different interventions used.  It wasn’t simply giving people lectures.  They did use a fairly active approach.  When I say little change overall, I mean in relation to the three measured outcomes which was the use of antibiotics in sore-throat guidelines, use of laboratory tests, and use of telephone consultations.  That in itself is unremarkable, and it’s fairly common not to see a lot of change.  What was extremely interesting and very important, I think, is the great variation between the practices in the amount of change that happened.

The next slide I’m going to show you shows you the percent of patients receiving antibiotics before and after the intervention for the controls, which are the red practices in the diagram, and for the intervention groups, which are the white practices.  Now, this slide is a bit too clever.  I think it’s brilliant.  But every time I look at it I have to do a “Goodness, now what’s going on here?  How does this work?”  Let we walk you through this.  It’s a great way of presenting, very well, some data.  Basically, the control practices are the red ones, and the white practices are the intervention practices.  The best way to understand this is to look at where you see the text there, “These reduced use after practices.” The practices below the line are those which did reduce their use.  You see there are as many red as white ones.

If you look at that white one next to the “t” on the text, it helps understand it a bit better.  If you’re going down there, it’s about forty-three percent of the patients receiving antibiotics before the intervention.  Then if you go left to the percent of patients receiving antibiotics after the intervention, you see... well just a minute what’s going on here?...  forty-three percent were receiving it before the intervention.  And then after, twenty percent were.  So I thought the intervent, ah, so that one would be worthless.  The general point here is the big variation and the interesting before-after changes in practices.  

There is a question of “Was it bad data?  Have we got a problem with the data here?”  The process evaluation looked at that because—don’t ask me how.  I still can’t get out of them why they decided to do a parallel-process evaluation.  But this is one of the first that was done of a parallel type like this.  This is going back ten years ago.  At the same time as the trial, they collected data that might explain—I think they foresaw that there might be differences, but not so amazing differences.  They did get data from 120 practices.  The data they got was from observations, from telephone interviews, a postal survey, and they also looked at data on the electronic medical records.  The question there really is what did they look for?  How did they look at those data?  It was mostly an inductive approach.  There wasn’t any pre-theory saying, “We predict that these practices with these features, there’ll be a major change.  And these, there won’t be.  These are the critical factors.”  My point there about the theory is relevant later in the second and third categories that I’ll be talking about later.  So this is basically inductive.  They trawled through these observations and interviews and put together a bit of their own, I think, chaps in the team.  At least that’s what I ascertained by talking to my colleagues out there in Oslo.  

What they found was that these things were the critical factors, that this is what the data shows.  Basically it’s the physicians agreed with the guidelines; they’re very positive about them.  And there was a degree of participation in the project, pretty much subjectively judged, taking time to discuss the guidelines, so working through some implementations.  That’s the beginnings of the ownership of it.  Then those practices that actually used all of the different components of the intervention, like they used the data that was being fed back to them.  They all had laminated one-pagers posted, etc.  So that’s the implementation fidelity to some degree.  And they had procedures in the practice so they could do telephone consultations with patients quickly and easily.

Then there was this other interesting, compelling, mysterious aspect which was communication within each practice, which was more of a team issue.  What we’re seeing here is an interesting—in some ways the richness of these studies could’ve been completely missed because we just saw another fairly common finding that, well, overall there wasn’t a big change in the 140 or so.  It’s not of significance to say this is an effective intervention.  But by going through into a process evaluation and collecting data in a pretty—well, I won’t say random way, but not in a guided way.  They were able to come out with some sort of an explanation of why—first of all, they were able to spot the big variations between the practices, and then to explain why those big variations and get some sort of explanation for what was going on.  They could then, on the “What to Do Differently” slide, they could then say, “Well, actually, coming out of this we’ve actually got data and evidence to say these are evidenced-based improvements to this intervention.  Use more active interventions, etc.”  

The next iteration of this guideline implementation approach could then add other interventions to change the context because those last two, the “Focus more” and “On-line access” one on what to do differently, those are context factors.  Although they didn’t know it at the time when they were showing it, what they were showing is you’ve got to change the context to get more success on this.  In principle, what they could then do is a phase two of this where the intervention included those aspects of context that were important to success and to test that theory.  The words we’re using, the context is enfolded into the intervention.  The context is enfolded into the intervention.  You can see that—I don’t know if some of you are very familiar with Peter Provenost’s central line-associated bloodstream infection studies in Michigan in the Keystone Project, but some of us have a nerdish interest in all the various stages of that program and how and why it was successful.  Now, a similar phenomenon in the ICUs up in Michigan, that big variation, and Mary Dixon once did a great social science study of that, a bit like this parallel approach, where similar sorts of issues came out.  I think Peter himself said in a later paper, “We now have changed intervention to include aspects of context.”  

Onward to approach two, here we’re looking more at a single case study or a program of spread, possibly.  This example comes from Zapka’s study of an intervention to promote smoking cessation in community health centers.  This was able to highlight some of the issues about which what needed to be done to increase the chances of community health centers being successful with their smoking cessation programs.  Why I’m showing you this one is because a key part of this study was having a program theory, which is sometimes called a logic model.  There is a whole family of terms to describe slightly different things.  But basically the program theories are assumptions about which actions lead to which intermediate results and then later results.  So intermediate, in this case, a smoking cessation might be, for example, people attend the classes on smoking cessation or they get information, the later results might be quit rates, that sort of thing.

But also, some program’s theories are quite good at including a theory about which context factors are critical to enable the actions to implement the intervention.  In the next slide I’m going to show you a program theory model.  Remember, we’re talking about intervention to promote smoking cessation.  This is the overall broad-brush approach; you see the concepts of the QT concept in the organizational context, and then go right and you see the planning and implementation process that they went through to implement this intervention, and a box there that attaches how much the providers took part, then interactions, and then patient impact and final outcomes.  What this does is blows up in figure the organizational context aspect.  Basically this study collected data about the structure of the different community clinics, different work characteristics, the social environment, the part of town they’re in, patient profiles, undocumented and so on, and aspects of the external environment with each clinic.  Some of them were funded differently, were working under different regulations.  

The point here is that they had a pre-study theory that guided the data that they then collected, not just about the implementation process and the different outcomes they’re going to focus on, but also about the context of the different clinics that they were looking at.  You probably can’t see the detail there.  But if you have it on full-screen mode and if you have a big screen, you can actually see different aspects of the organization context and what they think.  You can see their organizational structure.  You see some of the challenges that they faced that explain why some clinics were more or less successful and what one needs to attend to.  The point here is that what you get out of this study is maybe less certainty about outcomes.  If you did a controlled trial, you would maybe be more certain about the intermediate and final outcomes.  You trade that off for having much richer and more useful, in my mind, details about the context that helps and hinders and the details of the intervention.  When you’ve got a fairly complex intervention like this, that can be more or less successful in different settings.  

What we’re doing is we maybe have less internal validity, but we’ve got more external validity.  The study is more useful and useable by clinic leaders in other clinics to decide what to do and how to do it and what chances they’ll have of success.  My point there is that it answers the practical questions of some of our customers much more effectively.  Now, I’ll get on to a third approach, which I’m generally going to term action evaluation.  This is where you have interventions that actually demand or require the iteration, that they are developed, that they are tested, developed, and continually worked on.  I’ll talk about one version of this category, which I’ll call integrated research-implementation evaluation, where the distance between researchers and the implementers is very close collaborative working.  The aim here is not to control the intervention, but it’s to describe and improve it—this is the researcher’s aim—to describe and improve the intervention and its result.  And you work closely with practitioners, actually, not just to give feedback but also to help the implementation.  

What I’m going to focus on is—I’ll probably focus on the TIDES.  Interestingly, Lisa Rubenstein here at Sepulveda— and Becky Yano and  [inaudible] too in European terms—are looking at three demo sites for Patient Centered Medical Home in the Southern California VISN 22.  Some of the learning and lessons from the TIDES depression study is being used in the PCMH in a very interesting way.  So going to the TIDES, this was a collaborative action research project.  The idea was to use evidence-based quality improvement methods.  The intervention was using proven depression care models.  To some extent, there was some similarity to the diagonal-chronic-care-model kind of approach.  I know people involved in this would onject to that.  But that just gives people a feel for what the model was.  So it’s the depression care model, and the researchers helped regional leaders to select and adapt different models.  As I said, the researcher’s role was to give evidence about practice models and discourage proven ineffective approaches to this, and to kick start or help people use quoted proven methods to do the implementation.  Then the researchers evaluated the results [inaudible].

Now, that’s a very crude, rough-and-ready study of a considerable number of years and really the development of a new approach to doing research [inaudible].  The point is there’s no randomization or standardization.  To some degree I don’t really think there’s really a fidelity assessment of implementation or a comparison—actually, that’s not strictly correct.  I think there was.  I’m hoping there are some people watching this who can correct this.  The research products that came out were examples as modeled on how they were implemented by local project teams, and exactly the same is coming out of the Patient Centered Medical Home demos that are being done on the moment.  A description of the implementation structure steps and systems is, even in the Veterans Health Administration with policies and execution, you need a structure, steps and systems, to implement.  The mainline system isn’t sufficient.  And then outcome data, so we saw how effective this was.  These were the results quoted there.  What was sort of gamely thrown out was... And the program is undergoing national spread just like that.  Here’s a really great study waiting for someone to do for a PhD or a master’s is, yeah, what happened? What was the fate of this in terms of the national spread, and what were the different ways it went?  

One of the questions is did it spread without research or other types of support?  What was the infrastructure to spread?  The point that I’m making here is this is a third approach where context is in there and a concern and an issue to the researchers.  What I’ve been observing in the Patient Centered Medical Home study that is drawing on and significantly influenced by this is much more attention to collecting data about context and also feeding back to people to say, “Look, one reason why this might not be successful here is that you don’t have these factors in place in the local clinic.”  There are three clinics.  It’d be great if we heard from colleagues working on either of those two studies, a major who could maybe correct or optimize something that they’d say.  So where we go, basically two approaches to study context.  We’ve got a parallel process and outcome examination here.  Then we’ve got theory-informed case evaluation.  And then an action evaluation which uses context theory as part of it as well, where people discover certain elements.  Context is critical.  These then can be immediately introduced in the next iteration of the intervention.  So going to some of the practical questions for colleagues, who are saying, “Okay, I’m doing a study.  Should I collect data about context, and if so, which data should I collect?”  

In a sense, there are two quick things I want to preface here.  The first is if you’re doing a control trial, no.  Don’t bother.  Just meet them in the minimum reporting requirements for the trial where even there people are beginning to say, “You need to say a fair bit more about the setting where the trial is done.”  But you don’t necessarily have to get lots of data about context.  Now, basically the guidance is the simple rule of thumb so far—and maybe others listening in have other thoughts on this—look at previous evaluations of similar types of interventions.  For example, if you’re doing something like a smoking cessation type of study, behavioral change study, obesity prevention program, similar sort of things, then look at a study that has looked at context before and say, “What context factors at different levels of the system do they suggest were critical for success?”  I think you’ll find that most of them will say, “There’s leadership and culture.”  That’s pretty common to all of the explanations and understanding.

But you need to go further than that.  You have to say, “Okay, which level of leadership are we looking at or might be most important?  How are we going to measure leadership that is important for implementation?”  This is also especially true for that [inaudible] concept culture, which aspects of cultural reasoning to look into and how to measure.  What’s our theory about specifically which aspects of culture affect implementation?  Now, having looked at the literature and seeing what others evaluating similar interventions have come up with in relation to context to explain what’s going, if that fails then use a generic framework and maybe adapt it to the intervention.  

If we’re looking at implementing evidence-based factors in nursing, an obvious contender is the PARiHS framework.  This is Rycroft-Malone, et al’s summary of that.  Again, it’s a little bit sketchy.  There is later work that has specified this in more detail, but you can see there the separating out of the different aspects of context that will give you some guidance as to what to collect data about.  Notice also that they have this idea of receptive context, and that is a historical thinking about context which is ideas about how welcoming of the change people are likely to be, rather than, as it were, the immediate current.  A lot of that is based on experience, history, and other factors, rather than the immediate financing regulation and other things.  I do actually feel this is missing finance and regulation, which are really important aspects of context usually.

Turning to another generic model you might use or think about, the Kaplan MUSIQ model, based on a review of research.  This is interesting in that they quite well differentiate four levels of context, although I’m a bit nervous about QI team being context.  I don’t know why, but I’m just uncomfortable about that.  Again, also microsystem; we all define that in different ways, usually those external-internal environment contexts.  But looking at the actual model, they reviewed the research; what they found was, yes, [inaudible] leadership and culture, data infrastructure, years involved in QI.  That’s on the receptive context side.  Other potentially important factors—and this is interesting—I wonder if it’s depending on the type of intervention these are more or less important: physician involvement, microsystem motivation, and QI team leadership.  These kept coming up but not as strongly as the others.  I’ll show you another one which is coming from a different angle.  French, et al, did a synthesis of thirty instruments measuring context.  This paper is really helpful for saying, “Here are all the references, and here’s a summary of it.”  This is a great way into the literature and the models and guidance on this.  I think it’s very interesting also the way they’ve differentiated these two types of aspects of context; the receptive capacity, which is the bring-it-on side of things, and then the absorbative capacity is separate from what we think we can do and what we’re up for doing and ready to do.  There’s the actual hard “Can we handle this change?” in terms of hard factors.  

Now, another one I’ll briefly mention is the Shekelle AHRQ study, the safety study that I was involved in with Peter Provenost and UCSF and Rand here.  These were some of the measures of aspects of context the team identified.  We were looking at what measures are there of context that you could use?  Those were some of the better ones that we selected.  For those of you who maybe at this point sort of, “What is all this about?” this gives you a useful anchor in thinking about this.  If you think of the context of the intervention and perceive these are the proven factors for evidence-based change, then you have to implement it.  Your gardeners are the implementers; they plant and nurture the change.  But then you’ve got the climate and soil, and it may be that the climate and soil beats the skills of the gardener.  Or it may be that even the most unskilled gardener can abuse the plant, but the climate and soil does the rest.  It makes sure that it gets watered properly even when the gardener forgets.  This principle here is that the climate and soil is the internal and external context, and the gardener is the implementer.  Basically that’s John’s theory of what counts in implementing change.  And I would simply say that the climate and soil is the most important one.  In some ways it goes back to what Brian was saying: “Maybe not as important as the seed is how well it’s implemented.”  Then I would also add: “Maybe even more important than how well it’s implemented is how receptive or whether the climate and soil is nurturative or fertile for that particular seed in that setting.”

We come finally to why we should bother about this.  I’m gonna run out of time and probably another five minutes is quite sufficient so I’ll run through this quickly and you can come back to it. So Why do we do? First of all, is there any evidence the context affects outcome?  Well, it’s a bit like the one about why do they rob banks.  That’s where the money is.  Why do they do RCTs?  Because context affects outcome.  And in this case, patients are affected differently by medications.  In this case the body is the context.  But then there’s another, wider context, which is to do with whether the patient adheres and what their life situation is that helps or hinders them adhering with the meds.  Training affects providers differently; the context of their work may help or hinder them actually following through on the training.  Then QI interventions to organization changes some and not others, and part of that has to do with context.  The problem is that there isn’t much evidence of how context affects and which are most influential.  I’ve been running through those models.  But in terms of looking at different interventions, there isn’t a lot of empirical research out there, which is why this is such an interesting field.  You guys can be at the front line on this, pioneering this type of research.

As regards the evidence, we’re coming along to a vote here.  What I’m going to do is go through and ask you to vote yes or no.  Would you say leadership from top management was one of the most important things to quality improvement success?  The next one for your vote is organizational culture.  Would you say that is a really critical factor in whether quality improvement intervention succeeds?  What about data infrastructure and information systems for the improvement teams and management?  I’m talking here about data on the progress and the results of the project, patient and intermediate outcomes.  Would you say that’s absolutely fundamental to—one of the things I’m doing here is giving examples of what I mean by context, data infrastructure and information systems to the team doing the improvement.

The next one is, is this very important to success, how many years are people involved in quality improvement?  Let’s say the organization has been involved in doing quality improvement in different ways in different projects.  Would you say that’s absolutely critical?  The last one is whether the CMO of the new organization supports the Dallas Cowboys.  How critical do you think that would be to the success of quality improvement?  In this case I’m thinking of that as a context factor.  

To cut to the chase as time’s running a bit short, Kaplan didn’t actually include the last item in their review.  But what they did find was that all of those were absolutely critical to quality improvement success, all of those context factors.  They key question, like I was saying earlier, is how do you operationalize leadership, the top management, to judge simply present or absent, or at least one to five, organizational culture?  We need to get more precise about these.  I’m not going to say much about this.  But this is also, as it were, making the case of why to study context and what the evidence is.  

This is from the AHRQ safety study review that we did.  We looked at five patient safety practices; one of them was interventions but falls in institutions.  There was no strong evidence for or against context practices being important.  I think this is either helping or hindering the implementation of these interventions.  I think that says more about the research than it does about whether—having gone through these in quite some detail—and there’ll be others listening in who also know about the research into falls in institutions in quite some detail—and one of the issues here is how strong does the evidence need to be, which is a whole other debate.  Medication reconciliation, believe it or not, there’s very little data on how effective those are.  But the context also is important, and the obvious one would be blocking functions.  Basically the system doesn’t give you an alternative.  

Now, I’m not going to go through the other evidence from the other reviews that we did on that.  They’re later on.  One option is not to bother with context and use comparison groups and randomize.  But this is expensive, time consuming, and we get into the argument against randomized control trials.  One of the other arguments is “Look, if it works, who cares about the context?”  If you’ve got an intervention—Steve Goodman makes this point, and I think he makes it very well—if you’ve got an intervention that works in, say, sixty percent of the settings, i.e., an experimental design shows that it was reasonably effective in sixty percent of the settings, then why bother with all this business about context?  Just give it a go; whack it in.  There’s enough evidence that it works.  Steve there is arguing, “This is pretty much what we do with clinical medicine clinical trials.  There will be some patients where it’s not so effective.  But if for most of the patients in that lecture group it works, we can just get on and do it.”  

One of the issues—and I think I’ll close off here; I’ll leave the rest for later for you to look through in the references—a lot of what this session’s been about is about how to generalize from research to a variety of settings.  And research is providing information not just about the details of the intervention for some of these programs that they’re looking at, but also providing and not just describing the context in which they were done.  They’re going further than that and saying, “Actually, our best guess so far is if you have these context factors in place then you have more chance of success making explanatory and predictive statements about the context necessary to successfully implement a particular intervention or improvement.  I think those were the main things.

That’s the summary.  I’ve converted to IDrive, so if you go to this Web site you can get other resources related to that.  On that Web site is a guide that pulls together in a very simple two or three pages with all the references, some of the research into context.  You’ve got a guide and a checklist there that can give you a starting point for that.  Then there are other references in the presenter notes on slide thirty-six.  I’m going to finish off there to give others a chance to correct, comment, complain, or to give your views.

Molly:  The first question we have is since context is important, how do scientists mount a study with enough “units” in the samples to produce a believable, generalizable sample size?

Dr. Brian Mittman:  The issue is one that we struggle with with our statistician colleagues and that is as we add additional contextual factors that need to be measured, in other words, additional independent variables that the sample size requirements increase, do we recruit sufficient numbers of hospitals or clinics to generate meaningful estimates with appropriate statistical significance?

John Øvretveit:  The quick answer is I don’t know.  The other answer would be one is enough, that basically I’m coming at this from a completely different paradigm, that basically if we did a case study evaluation—take for example that if we just took one community clinic and did that cessation study and we collected those details about context factors, we would have very weak evidence about its effectiveness.  Even if we were able to go down the chain that the model showed and track the chain from changes to providers to changes to patients to actual health outcome, we could do that with a theory.  That will help make the links in the chain; it would help increase certainty about outcomes.  But we would also be able to explain what context factors helped and hindered that causal chain to operate in this clinic.  

So what then it means is one clinic case with a certain number of patients where we’ve got fairly weak evidence about outcomes but where we’ve got pretty good evidence about what helped and hindered making the change.  If you know that the change is effective in other settings, then that’s a good way to go.  But if you’re starting with a completely new change and you’re wanting to evaluate that, then it may not be such a good approach.  The main thing I’m trying to get across is just a completely different way of thinking about what the purpose of the research is.

Dr. Brian Mittman:  I think for those that do insist on quantitative evidence, we’re pretty much in the realm of large sample observational studies into some like VA where we do have the ability to include, with the use of some administrative data, dozens or hundreds of hospitals or clinics and measure as many of the contextual factors as possible through the existing data and generate the large samples.  But in terms of experimental approaches, once we move beyond a small number of contextual factors, we just can’t include sufficient sample size.  I think personally I too would advocate for more of the qualitative approach and the theory-testing approach as a way of generating better insights into the impacts of contextual factors.

Molly:  The next on is how does your Swedish cultural homogeneity influence your presentation today?

John Øvretveit:  That’s right.  The Swedish society is much more homogeneous than certainly L.A.; that’s for sure.  On the other hand, in my defense, Stockholm is pretty multicultural.  There’s Stockholm and then there’s the rest of Sweden; well, [inaudible] and Gothenburg.  Certainly in terms of the services, the context is pretty similar.  Essentially we have twenty-one geographical health systems, and they’re all governed by the same national laws and regulations.  The other day I was at a community clinic in San Diego, and I spent an hour talking to the CMO there about how they finance their program and how it was managed and organized.  It was absolutely astonishing.  In comparison to a community clinic of that type in Sweden, this is a whole different ballgame; the complexity of the management [amazes you].  

Now, you may say, “In the VA we’ve got a much simpler and more streamlined context.”  But it’s still a lot more complex than Sweden.  I think that’s a fair point, and I think it may even get more complex in the next few years if veterans have a choice of where to go.  It divides up slightly.

Molly:  Can implementation research provide evidence and-or insights into how to change local context to maximize chances of implementation success?  For example, if a site had mediocre leadership, can implementation research help show the way about how to “improve” leadership qualities or perform an effective end-run around that issue?

John Øvretveit:  The quick answer is yes.  Research can show to some degree, and in fact there are various research-based assessment tools that you can use to assess readiness to change, to assess spread issues, like the HRET spread assessment tool that’s based on research.  It’s really useful.  That covers a lot of context factors.  Project leaders or researchers would be able to use one of these tools or develop their own to do an assessment of the context.  And then you would go to present that to leadership and others to say, “Actually, for this change you haven’t got enough of what it’s going to take to implement it successfully.”  And this assessment instrument helps highlight where you would need to do other things before you would even start trying.  

Another way to put it is, for example, in Sweden, in the breakthrough collaboratives that we did over the last ten years or so, we got to a point where we could tell which teams in which organization were likely to be successful pretty much in terms of the leadership and the kind of backup that the team would have and whether they were going to get time off and so on.  There was a debate amongst us about, “Should we exclude from the breakthrough collaborative those teams where our assessment tool shows that their organization is not going to provide them with what they need to actually be successful in this?”  They [ruled] for another team because they’ll be wasting the resources of the—another team could take their place that’s more likely to be successful.  It being Sweden it had to be equally involving everyone, but we could predict who was going to be more successful and less successful from this idea of the context factors.  But then the second point is you said, “Then can we develop leaders and others?”  That’s a whole other ballgame because you may be able to provide a leadership program, but they may not prove to be up to scratch to be able to do it.  But I guess you’ve got to try it.  The point is you can do assessments, decide on intervention, change the context, and then go with implementing interventions.  But, you see, that pre-work is—basically what we’re saying is an intervention with a number of phases where we attend to context first and try and get back to a minimum level.  The tricky issue is deciding what’s good enough to provide the nurturing setting for the implementation.  We’re making progress on those assessment tools to be able to do that, and some of those are referenced.

Dr. Brian Mittman:  I would just add briefly that this area, the organizational readiness change, represents the best illustration of the points that John’s making, recognition of the fact that context is important, that it matters; an organized, systematic way of assessing context; identifying those areas of context that are weak; and then providing guidance in managing and changing the context as a strategy for improving effective implementation rather than focusing only on the implementation strategy itself.  There is a growing body of literature that applies these tools and somewhat less.  But again, a growing body that attempts to do something about deficiencies in readiness that have been identified.

Molly:  Next question is what time frame do you use for collecting data from a specific clinic?

John Øvretveit:  Well, it depends on the intervention, and it also depends on the researcher’s role.  Essentially you’d at least be doing a before-after study probably, although in some cases you might be doing a retrospective.  Now, if it’s a retrospective, which is probably the least good way of doing it, then you could be talking about six months to interview people, look at documents, to ascertain the context factors.  

Here we introduce another aspect of this, which is really tricky, is the context changes over time.  Part of documenting context also is documenting how the context has changed in those different respects that you’re looking at.  Typically, for example, the chief executive leaves, or people leave and come and go who had a role in the project.  It all depends on the intervention and the type of design that you’re using.  And if it’s two or three, you may need to collect data on context at least two or three times to identify any major changes.  The other way is just to interview people and use documents and do it retrospectively.  That’s a lot quicker to do it that way but slightly less accurate. 

Dr. Brian Mittman:  John, I’d like to thank you again for yet another seminar and for sharing your ideas and insights with us.

John Øvretveit:  Thanks, Brian.  And thanks Molly and Debbie.

Molly:  Thanks again, and this does formally conclude today’s HSR&D cyber seminar.  

Cambridge Transcriptionswww.ctran.com

Cambridge Transcriptionswww.ctran.com