Enhancing Implementation Science- 43 -Department of Veterans Affairs

**Department of Veterans Affairs**

**Enhancing Implementation Science**

**EIS- Intro Program Session 1: Introduction to Implementation Science in VA**

**Presenter: Brian Mittman**

Moderator:  And as we have reached the top of the hour, at this time, I would like to introduce a brief introduction of the series and also our presenter. As I mentioned, the 2012 EIS Program is a distance-learning program with lectures delivered by our cyber seminar and small group discussion sections and sessions conducted at individual VA facilities. This is the first presentation of the introductory program, which is being held between May and July of this year, 2012. The advanced program will be held between September and November of 2012. Presenting today on the topic of Introduction to Implementation Science in VA is Dr. Brian Mittman. He is the Director of the VA Center for Implementation Practice and Research Support. He is also a Senior Social Scientist at VA, UCLA, and Rand Center for the Study of Healthcare Provider Behavior; both at the US Department of Veterans and both are located at the US Department of Veteran Affairs in the Greater Los Angeles Healthcare System. At this time, I would like to check in. Dr. Mittman, are you available to share your screen at this time?

Dr. Brian Mittman:  I believe we are. Yes.

Moderator:  Okay. You will see a popup and you can go ahead and take control now.

Dr. Brian Mittman:  Thank you. And thank you for the introduction. And for all of you working on getting this organized. I think the production is one reason or one illustration of why we like acronyms within the VA. I would like to welcome you all to the first introductory session and the sixth session in Enhancing Implementation Science Introductory Program. This is, as she said, a distance learning course. Many of you who are listening, I know, are enrolled in the course and will be participating in the post-lecture small group. Others may not. For those who are not, we are happy to share the materials we are using in the small group sessions. 

Let me begin with a brief overview or outline of what I’d like to cover today. That’s on the next slide. We’re having trouble advancing our slides. 

Moderator:  Not a problem. Click anywhere on the actual slide and you can press return or you can use the arrow button.

Dr. Brian Mittman:  I’ve got it. Thanks. What I would like to do is begin with an overview of Implementation Science and share some slides that illustrate what we typically think of when we think implementation science. And talk a bit about why that illustration covers only a small subset about what this field entails. I’ll then spend fifteen minutes describing the key quality and practice foundations for the field. I’ll then move into what is essentially the core of the lecture, and that is presented as the frameworks we’ve developed and used with inquiry to guide the design and conducts and reporting of implementation studies. And then a brief part that provides an overview of the remainder of the program and the remaining lectures.

Moderator:  Brian, I apologize for interrupting. Is it possible to get closer to the microphone? Your volume is a little low.

Dr. Brian Mittman:  Not only can I get closer, but I can speak more loudly if that helps. So, let me begin with an assertion and that is with the argument that increasing investment in activity in implementation and therefore implementation research are critical in achieving key societal goals. The argument here is that we have a number of problems that fields of health research is perusing and a number of gaps in quality and outcomes that concern us as well as mortality and morbidity. There are specific problems related to gender and racial equity. Use of care in prevention and a number of others.

The argument is that at least some of these problems can be at least partly addressed and solved through better implementation of knowledge and innovation in treatments and tools that are currently available. We are not implementing or using those to the extent that we should be, and therefore, we need more and better implementation research in order to better understand the implementation gaps and challenges to help us to, again, contribute to solving of these problems

So, before we proceed, I would like to get a sense as to who our audience is. So, Molly, if you could take over and post the first poll question. And what we’d like you to do is indicate your primary affiliation and role. We know that some of you are VA Clinician Researchers, others are VA Social Scientist Researchers, Non-VA Clinician Researchers, Non-VA Social Science Researchers and we may have some VA employees who are not researchers. If you could let us know which of those categories best describes your situation...

Moderator:  Please give me one second here. I am going to reopen the poll at this time. Brian, I hate to do this. Would you mind if we moved on to poll number two while I readjust poll number one real quick?

Dr. Brian Mittman:  okay. Sure. Poll number two is a single question, so if you go to the next full poll... That is, in fact, in some ways what we are more interested in knowing. And that is some indication of the type of research that you conduct. If you could let us know if you are a basic science or lab researcher, you conduct clinical research in which you study drugs or devices and we’re including health promotion and health behavior researchers in that category if you evaluate health promotion programs. And then if you are health services research or study health use, cost, quality, access and finally those of you who would characterize yourselves as implementation researchers.

Moderator:  Thank you for your patience. We have had about seventy percent of our audience vote and we will be able to return to the first poll momentarily, as soon as we have had all respondents click their answer, I will close the poll and share the results with everyone. We’re up to about eighty percent response rate. And the responses are still coming in. I’m going to close the poll now and share the results. And Brian, you should be able to see the results now. If you could speak through them, that would be great.

Dr. Brian Mittman:  Okay. I need to click onto another screen so I can see the results? Okay, here we go. So, we have, as we would expect, very few basic science and lab researchers. The majority of you are health service researchers. Forty nine percent – twenty six percent of you classify yourselves as clinical researchers. And twenty-four as implementation researchers. Roughly one quarter of you would characterize yourselves at this point as implementation researchers, but our goal is to increase that by a factor of two. So, if we could go back to the first poll, then, and again ask you to indicate your primary affiliation and whether you see yourself or are a VA researcher, a social science researcher, a non-VA clinician or a social science researcher or someone in VA who is not a researcher.

Molly, those results are coming in? 

Moderator:  And we are at about eighty percent response rate at this time. So, I will leave the poll open for about another fifteen seconds or so until we have – everyone has had a chance to select their primary role. Okay. And, it looks like responses have stopped coming in so I am going ahead to close the poll and share the results. Brian, you should be able to see those now.

Dr. Brian Mittman:  Give us a minute for them to appear on our screen and I will go ahead and read them. Actually, try clicking on the screen and see if that helps. Molly, we’re actually not seeing them. Would you like to read the results?

Moderator:  Yes. We have thirty three percent VA Clinical Researchers. We have twenty eight percent social scientist researchers. We have seven percent non-clinician researcher. We have fifteen percent social science researcher. We have seventeen percent non-researcher VA.

Dr. Brian Mittman:  That’s a healthy mix. Let’s go to the last series of these polls. That is the question asking the participants to rate their highest role today in implementation research.

Moderator:  okay. The poll has been launched. And we are seeing results streaming in. We’ve had about eighty percent of our attendees respond thus far, so we will give people a little bit longer to select their answer.

Dr. Brian Mittman:  Molly, I think in the interest of time if you could go ahead and read the results of this poll, we’ll be ready to proceed with the content.

Moderator:  Not a problem at all. I will go ahead and close the results now and share them. So, we have sixteen percent principle investigator. We have seventeen percent co-investigator slash consultant. We have twenty percent reporting that they have watched colleagues conduct implementation research and we have thirty percent saying that they have no role to date in implementation research.

Dr. Brian Mittman:  Again, the goal of this program is to shift those numbers upwards and decrease the number who have not conducted or have not watched as well as those who have watched into active PIs and Co-investigators in this project. Let’s proceed, then and move into a set of slides that indicate what I will briefly illustrate what we typically think of when we hear the words implementation research.

Moderator:  Brian? 

Dr. Brian Mittman:  Yes.

Moderator:  We cannot see your slides at this time.

Dr. Brian Mittman:  Do we need to begin sharing?

Moderator:  You should have, up on your screen... There you are.

Dr. Brian Mittman:  Okay. Thank you. Our apologies. This is the typical sequence that we see when new evidence or new innovations are introduced. We will see the release of the new innovation and the publication of the new study results and new evidence. Upon release or publication, we will often see initial efforts to promote the use of that evidence or that new innovation and some years later we will also see a study that measures rates of adoption and quantifies implementation gaps or quality gaps. Essentially, it is assessing whether the new evidence or innovation have been adopted and are yielding the benefits that were promised.

And when the results of that measurement show significant implementation gaps or quality gaps, we will often see studies conducted and published that develop and evaluate implementation programs that are designed to increase implementation. So let me talk you through an example of this in a case of a research finding in the heart area. There were some publications in the year 2000 for a very large and highly anticipated study that essentially put to rest the question that had been outstanding for a number of years in the cardiology and primary care community – and that is whether beta blockers are effective or perhaps even harmful for patients with heart failure. There has been some evidence suggesting that they are helpful and indicated the evidence was not full tested in this very large study, but it was put to rest. This is the example of publication and new evidence that has a significant policy and practice indication.

At the same time the evidence was published, there was an accompanying editorial but the key message of the editorial was captured by the subtitle ‘The Evidence is in. Now the Work Begins.’ And what the authors meant by the work is work to assure that this evidence was used and that the finding was implemented and all patients with heart failure were on an appropriate dose of beta-blockers and that they all benefited fro this new search. And then a few years later, we saw a number of steps taken that essentially offered the clinician community guidance for adoption of the heart failure study findings. The American College of Cardiology and the American Heart Association have a very active and well-regarded clinical practice guy on program and their heart failure study results were incorporated in their clinical practice guideline. The ACC and the AHA also developed performance measures for use by healthcare systems such as VA where the measures were used to monitor clinical practices in order to identify quality in implementation gaps and identify areas where more active and intensive efforts to implement findings are appropriate. So, again, providing findings and incentives for adoption.

We also saw a number of medical societies and healthcare systems taking steps to try to facilitate adoption implementation of this guideline in their heart associations. With the guidelines program, the VA and the DoD find clinical practice guidelines and a committee will go through development of its guidelines as well as implementation efforts and work to try to encourage implementation of the finding and increase appropriate utilization of beta blockers. They try to help other health partners and other private systems engaged in similar efforts.

So, continuing the story of the typical sequence, a number of years later, we saw the publication of a couple of articles and there may have been more as well that looked back in time and tried to determine whether there were significant increases in the use of beta-blockers for heart failure patients. They found the publication of these findings and their incorporation into the practice guidelines and performance majors and so on and as is too often the case, the results of both of these studies showed very little change, if any, in rates of utilization of beta-blockers. So the publication of a major study and accompanying editorial and a number of efforts by large medical societies and patient advocacy groups and health systems that attempt to increase physician prescribing patterns and prescriptions of beta-blockers all seemed to have relatively little impact.

So, that then triggered, again, what we typically think of when we think implementation science or implementation research. And that is the design and evaluation of specific quality improvement programs or practice change programs that attempt to increase appropriate use of beta-blockers within healthcare systems or communities. This first study that I’m showing you actually did not succeed in achieving significant rates of increasing beta-blocker use. The second study that I’m showing you, however, conducted by Paul Heiberweiss with colleagues at the VA did actually lead to significant increases. It was modest, but they were significant increases in the rates of corporate utilization of beta-blockers. So that study shows us, of course, that it is possible to increase use. Although the question is at what level of effort and why is it that many of these programs do not succeed?

So let me go back to my previous slide, and again show you this sequence of events. I offer you at least one reading of the literature and the experience in this field. I think it’s one that I think is shared by most observers and researchers. And that is by and large, the implementation or quality improvement program that we as researchers design and conduct tend to have relatively low rates of success. We do sometimes see improvements in terms of quality or implementation, but the other programs – there’s got to be an inverse relationship in the published literature between the rate of improvements shown and the quality or rigors of the study where the studies that are well-designed, use appropriate control groups and attempt to separate implementation issues from secular trends show much lower rates of improvement per change in clinical practice. When we see, especially the quality and safety journals, individual quality improvement reports or stories that often show successful improvement is the result of an implementation effort oftentimes explained by secular trends or other kinds of activities. But again, the bottom line for the most part is the effectiveness of the implementation strategies and quality improvement strategies for programs is modest. So, as a field, if that’s what our primary goal is, we have some work to do.

The next slide shows instead of some of the barriers to progress and success in implementation in research, there’s a road to the large and growing literature that identifies and discusses these in some cases, as you will see in reading through this list, there actually are arguments on both sides of an issue. Some observers will argue that the field of implementation research is not making more and better and faster progress because the research that we do lacks rigor, the internal validity of our studies is too low, and we need more RCTs. There are other observers and commentators on the other hand, who point to the lack of external validity of our research. 

They would argue that we have to many RCTs or perhaps that we have too many flawed RCTs in the field and too much reliance on black box evaluation approaches. Others would point to the lack of theory of lack of use of appropriate theory in the use of theory and implementation and that’s a topic we will cover in this program over the coming weeks. 

Some would argue that we have too many theories rather than not enough theories and some say that what we need is better guidance in how to use those theories. And then some are pointing to the possibility that the implementation phenomena that we are interested in addressing and studying are not simple problems or complicated problems, but instead they are complex problems or wicked problems. I don’t have enough time to go into the distinctions, but the bottom line is they are problems for which simple solutions such as a single implementation strategy or even one with two or three different components simply aren’t enough. The problems that we are addressing are complicated and multifaceted and they change over time. And no matter how hard we try, we’re not likely to achieve any sort of situation in which we can reliably change clinical practices and improve quality across the board.

So with that not so positive outlook, let me look to the next slide that turns into the brief summary of the key policy and practice foundation for the field of implementation science. It basically is two. There are two that are largely the same in terms of the underlying research issues and the nature of these issues. And yet, as researchers, when we talk to colleagues at NIH and in med school and literally pursue funding from NIH, we tend to talk about the implementation gap. When we talk to colleagues at schools of public health, Academyhealth,  and pursue funding from the Robert Johnson Foundation and other more policy-oriented entities, we tend to talk about the quality chasm. I think it is important to recognize the key essence of these two arguments.

Let me provide a very brief summary about what the implementation gap was all about. The top half of this slide shows what we would like to see in a well-functioning physician health research system. We’ve also seen basic and lab science results that are critically taken up by clinical researchers in that clinical research findings will quickly result in improved health. What we tend to see instead are too many flurries of ideas around the yellow and red arrows that are indicated at the bottom half of this file. Yellow arrows are actually ideas that emerge or findings that emerge from research, but actually don’t have any immediate value and therefore are published and put on a shelf where that is appropriate. What we are trying to do is reduce the flow of ideas and findings around the red arrows. Those red arrows are meant to indicate findings that emerge from research that actually do have value and potential application. The findings are published and put on the shelf because their value is not recognized. So the implementation gap is all about an effort to reduce those squares along the red arrows.

This next slide depicts the implementation gap in a slightly different way. I don’t know if any of you can hear the background noise from – somebody was trying to ignore, but if it’s distracting... There’s a code going on outside. In any event, this slide is a different depiction of the transitional roadblock that has been discussed by NIH, the Institute of Clinical and Medical Research Roundtable and others. And they’re basically pointing out the two types of translational roadblocks and the two types of translational research. This next slide – I actually list some of the key reports that I just alluded to – the implementational gap, the crisis and clinical research as it’s been called has first been identified and discussed in the states in a summit that was convened by the Association of American Medical Colleges in 1999. The clinical research roundtable met for a good five years to address these issues and develop different strategies and policy proposals to accelerate the flow of ideas along the research pipeline.

Efforts to increase the proportion of basic science findings with potential value was actually taken up and used by clinical researchers. And then efforts to increase the proportion of research findings that are published were taken up and used by practicing clinicians in the health system. So that’s a very brief summary of the implementation gap and translational roadblock issue.

The second policy and practice foundation that I’ve listed on my previous slide is the quality chasm. I don’t need to spend time on this issue because it’s one that we’re all familiar with. Basically, I’ve shown that the two key quality reports have emerged from the series. In the bottom half, a highly cited article documents significant gaps in quality and care within the US. Their basic finding was that the typical average American patient receives almost half of the indicated care. The quality gap is approaching half of all indicated care. 

This next slide shows results of a couple of studies that compare VA quality to non-VA quality. In the case of the first article on this slide, they actually applied the same quality measurement scheme that has been used in the Rand studies in the previous slide. They applied the same scheme to a set of data in VA. The results for the VA sample was approximately sixty-six percent. This data said that the typical average VA patient receive roughly two thirds of the indicated care. Two thirds or roughly half for non-VA patients. 

The second article in this slide used a different data set and a different set of data and measures but they showed significant differences in quality and rates of implementation or appropriate utilization of practices in VA versus non-VA. And I show these slides especially when I speak to non-VA audiences, not necessarily to boast a bit about VA and our accomplishments, but primarily to make the point that improvement in quality and increases in appropriate utilization or implementation is possible. In the VA, we all know very well what went into our improvements and the significant investment of time and effort and funds. The question is whether we can achieve similar improvement outside of the VA perhaps more quickly in a less cost. The point is improvement is possible.

Let me then turn to, as I said, the core of my presentation, and walk you through a set of frameworks that have been developed and used within the VA to guide the design and conduct, and recording is on, I would say a program of implementation and research. I’ll begin with a formal definition of implementation research. This is a definition that Martin Eccles and I adapted from previous sources and published in the opening editorial in the journal Implementation Science. It describes some implementation research as the study of methods to promote the uptake of research findings and other evidence based practices into routine practices and to improve the quality and effectiveness of health services. It also states that implementational research includes the study of influences on healthcare professionals and organizational behavior.

Now, the first half of this definition basically characterizes the type of research that I showed in my introductory slides. Basically, studies that evaluate specific strategies to facilitate better implementation or adoption of innovations and effective practices – I should point out that this is actually not a definition of implementational research, but instead a definition of implementation research in health. There is a fair amount of implementation research in other domains such as criminal justice, civil justice, education and so on.

But to get back to this definition, the second half of this definition describes more of an observational rather than an interventional type of research. It is research that attempts to understand influences on healthcare behavior practices and processes of implementation. The next slide restates this definition in a slightly different way by specifying three major goals for implementational research. The first goal maps to the first half of my definition. Developing reliable strategies or changing health practices or improving health related processes and outcomes in efforts to facilitate their widespread adoption. So, this goal is essentially what the interventional and experimental research in the field is trying to achieve.

But the second goal maps the second half of the definition. Again, it describes insights and knowledge regarding the limitation of processes and barriers facilitators, and strategies. Much of this goal is achieved through observational studies that attempt to study the ways in which practices change over time and the barriers and facilitators. Finally, as with the case with any field in science, implementation research roles include efforts to develop and test and refine theories, hypotheses, methods and measures. 

So this slide, again, is a slightly different form of the definition or description of the field of implementation science. I think if we were to go to the journal Implementation Science or other publications, much of the research if not most of the research that you would see can be placed into one of these three categories of activity. 

So the next slide – our group did a set of frameworks, again, that I’ve mentioned that were developed and used inquiry within VA to guide the planning and design of a portfolio or program of implementation research and the conduct and reporting of that research. I’ll talk you through the first three of these. I’ll present on the first three frameworks today. The fourth, I’ll not cover. We’ll hope to have time in one of the other sessions in the other series to cover that. 

The first framework that I’ll talk through is a set of expanded pipeline diagrams, many of which are useful in identifying gaps in implementation research. Getting back to my earlier comment about the shortcomings of implementation research, it leaves one possible explanation for the lack of more consistent findings and more success – we’re failing to conduct some of the research activities that are need as part of a more complete portfolio of research activities in the field and I’ll talk about that. The next framework that I’ll present is the QUERI six-step process and I’ll emphasize some of the pre-implementation steps in that process that again, we tend to overlook too often when we design and conduct programs of implementation research. And then I’ll spend a few minutes presenting the query four-phase framework for designing conduct of implementation trial.

The next slide, we have the first of a set of pipeline diagrams that I will show. This is, again, a restatement of the earlier diagram. You’ll notice that I relabeled the two types of transitional blocks – one of which is translational research and the other is implementation research. This is part of a broader effort that I won’t spend too much time on and that is an effort to try to clarify terminology and reduce confusion at the results from the multiple abuses of the word translation. So we have to talk about efforts to stamp out use of the T word in the implementation field.

We talk about translational research, especially in medical school settings and others. Too many of our colleagues aren’t sure if we meant translation type two or type one. For many in the field, the solution is to allow the basic scientists to use the word translation and translational and we should be focusing on and relying on implementation instead. But again, that’s an issue I won’t spend any more time on.

Let’s move to the next slide and I’ll talk you through the first of the gaps that I alluded to earlier. One of the gaps that we see and one of the hypothesized reasons for the lack of more complete adoption and implementation of findings is the idea that many researchers publish results of efficacy studies and are surprised when a practicing clinician won’t view those results as credible or relevant and don’t rush to implement or adopt those findings. Practicing clinicians often recognize that the result of the studies conducted in an academic setting using healthy weight males with a single condition rather than multiple chronic conditions are probably not completely relevant or applicable to their patients with multiple issues that come through their doors. We don’t have access to the same kind of care seen in academic settings and for other reasons, they are just different.

So the argument is that we need to see more effectiveness studies and we as implementation researchers and implementation practitioners should be relying on the results of the effectiveness studies rather than on implementation. There’s new work within the field that attempts to identify and characterize implementation studies that combine elements of clinical effectiveness and implementation. I won’t say anymore of that because Alison Hamilton who is sitting right here will provide a full lecture on the topic of hybrid effectiveness implementation studies. My key point in displaying this slide, at least at this point, is to point out the need for more effectiveness studies and for greater reliance on the part of the implementation research community on the results of the effectiveness studies rather than efforts to implement the results of the efficacy studies, which, again, are often not generalizable and applicable to the types of real world clinical settings in which we work.

On the next slide, we’ll talk about yet another gap. I’ll spend a bit more time on this one in a few minutes. This gap is the need for pre-implementation studies and specifically studies that attempt to document and diagnose quality gaps. We will often see, and this was certainly the case in the seventies and eighties but continues to be the case too often – we often see implementation studies that are published with an introduction that begins by highlighting the quality gap and that pointing out the need for specific practicing strategies to close that quality gap. The article would continue then by pointing out that in past research, colleagues have found success in the chief of practice change through use of feedback strategy or computerized reminder strategy. And as a consequwnce of that success. The new research has found that that was an appropriate strategy to employ to address a new quality problem and that was the purpose of their study and the focus of their article. Most of those studies tend to fail. It’s – some are comparable to reading a clinical study that would begin with an introduction that characterized the clinical problems such as HIV/AIDS and pointed out that in the past, clinical researchers have found success with a given drug in treating diabetes and therefore a new set of researchers would evaluate the efficacy of effectiveness of the drug for HIV/AIDS. Those are very different clinical conditions and they have different etiologies. As a consequence, the solutions or the treatment approaches need to differ as well.

The same holds true, more often than not, in the implementation world. The fact that its specific implementation strategy was effective in overcoming a given implementation or quality gap in a given setting in the past does not necessarily mean that it’s appropriate for a new situation. So, that’s at least one of the motivations and reasons for the need for this middle pike that I’ve included in this diagram, and that is the need to document and diagnose the quality gaps, understand their root causes and understand their etiologies, and select or develop implementation strategies that actually address the root causes of the quality problem rather than selecting an implementation strategy that just happened to be effective in a previous study for an entirely different problem in a different setting. Again, I’ll come back to that in a few minutes.

Let me, then, move to the next framework. This is one that is of considerable important within VA and especially within the query program. That’s the six-step QUERI process. We’ll spend some time discussing this in the small group sessions so I won’t spend a lot of time now. I’ll just point out that this could be simplified by highlighting the left hand of this slide in the first three steps which essentially sends the QUERI researcher to identify quality gaps or quality problems and on the right hand side, they aim to close those gaps to address the quality problems. But I’ll spend some time talking through some of the individual steps and sub steps.

The next slide lists some steps that are actually outside of the six step process that capture other forms of work that are part of a portfolio of implementation and pre-implementation research. And these steps are useful in understanding and characterizing and separating certain research activities that could be viewed as part of an overall implementation research portfolio. Again, for some of us within the VA, these are important because these are the activities that QUERI will generally not fund. These are considered the focus of clinical research service within the VA and HSR but not QUERI. Again, it will capture some of the categories of research that are necessary as steps before we get to implementation.

Let me spend a couple of minutes on steps in effectiveness studies, which I mentioned previously. But in this case, talk about the fact that one of the sources of confusion or lack of clarity both with in the field but more to the point among individual researchers is the distinction between a clinical effectiveness study and an implementation study. Many of the abstracts that we have reviewed for those of you who are participating in the EIS program have submitted, as is always the case when we have training in this field – many of the abstracts actually describe clinical studies and clinical effectiveness studies. They do not describe implementation studies.

Oftentimes researchers will use the word implementation as part of their abstract, but generally speaking as a somewhat oversimplified rule, a study that evaluates a clinical program or intervention including health promotion programs that’s meant to change clinical outcomes and where the key outcome measures are clinical measures such as patient clinical status, functional status, quality of life and so on – those are all clinical effectiveness studies. Those are not implementation studies Implementation studies, for the most part, are interventional and observation are meant to focus on practice change processes and strategies and to evaluate those relative to practice guidelines, rates of adoption of an effective practice. Narrowly defined, an implementation study actually should not measure or at least does not need to measure clinical outcomes. 

If we know that a clinical practice is evidence-based and we know that that practice will generate better patient outcomes, in the implementation world what we’re trying to do is increase use of that practice. If we managed to increase use of the practice and if the practice is truly evidence based, we’ll know that we will see improvements in the clinical outcomes and we don’t need to allocate our scarce data crunching resources to measure clinical outcomes. So, again, this is an issue that is the subject of a fair amount of confusion and a lack of clarity and it is one that we will emphasis into small group sessions, especially following this first lecture?

So let me move to the next slide and talk briefly through a couple of examples of the annotative QUERI six-step process. This is the diversion of the six-step process that was published in the QUERI series overview article. That is the assigned reading for today’s lecture for the enhancing implementation science theories because these are pre- QUERI steps and I won’t spend any time on them during the lecture. I encourage you to look at them, especially those of you who are involved in query, because these steps provide guidance in developing a QUERI -strategic plan and then identify and conduct from the pre-implementation activities.

Let’s move, though, to QUERI step three and I will spend a few minutes elaborating on some points that I made earlier, namely the need for a fair amount of pre-implementation work that’s designed to document and more importantly, diagnose quality or implementation gaps before we launch an implementation study. The purpose, of course, is to make sure that the implementation strategy that we are proposing is actually designed to address the causes of the implementation gap or the quality gap. Before we begin an implementation effort in the case of step three B, we’ve spent some time identifying barriers and facilitators to improvement. 

So, this is a category research activity that is critically important and ideally would be included before any interventional implementation study. It is one that we as researchers often don’t like to conduct because it’s not nearly as interesting as developing and evaluating a novel implementation strategy through a rigorous and elegant RCT. But it’s actually some of the hard work that needs to be done in order to give us a better insight into current practices into the quality gaps that we’re trying to address: the causes of those quality gaps and the prospects to success to an implementation strategy to overcoming and closing those quality gaps.

The next slide identifies yet another gap in our typical portfolios or programs of implementation research and that is captured by the green arrows that you see at the top of the slide. That area is meant to illustrate the fact that oftentimes when we see the results of effectiveness studies or we see the release of clinical practice guidelines, we actually see those guidelines or those steady results being moved into practice even without the involvement of researchers. 

In the case of a large healthcare system like VA, VA central office is introducing and implementing new clinical programs all the time, often without the involvement of researchers. Many of those efforts are actually more successful in terms of implementation success and achievement of the high rate of adoption and implementation than the new clinical programs. They are often more successful than many of the research-led efforts that we see. The point here is that we as researchers should be interested in studying those implementation processes and understanding the facilitators and barriers and the way in which the central office program has managed to overcome those barriers and achieve implementation.

And if we were to actually do better in many cases, if our goal is to understand implementation barriers and develop good insight, we would do better by dividing our research time intention to these observational studies than we would in conducting experimental studies that oftentimes are somewhat artificial, that oftentimes pursue quality improvement goals that we as researchers feel is important but aren’t necessarily shared by practicing clinicians or VA leaders. And a number of other aspects of interventional studies that limit their external validity and their value are similar. 

The next slide, actually – the next slide shows at least a couple of the characteristics of observational studies that make them important and we’ll spend a couple of minutes talking through some others. Again, experimental studies maximize internal validity but compromise external validity. Observational studies are often the opposite. In experimental studies, we often see a considerable artificial condition rather than real world conditions. Oftentimes in experimental and interventional research driven studies of implementation, we see elements such as high levels of technical assistance and facilitation on the part of the research team. We often see research funding and grant funds that are used to support new staff to hire and supervise new staff. And there are a number of other experiments that may not be applicable to real world situations and observation studies, of course, we see none of that.

We also have the ability in observational studies to recruit and involve much larger sample sizes. This will maximize the power to detect contextual info, which we will come back to and discuss later, and also it allows us to examine local adaptation processes and effects. Those are some of the key features of the observational studies and why we need to be conducting more of them.

Let’s move, then, to the next slide. I’ll spend a few minutes talking about QUERI step four work and the actual experimental and interventional studies. This is an issue that we will discuss in more detail during the small group sessions, so I won’t spend a lot of time right now. But this part of the annotated and expanded QUERI process identified some of the key sub steps and some of the individual activities that are involved in the design and the evaluation of quality improvement strategies when we do conduct trials of interventional studies.

On to the next slide, I would like to spend a few minutes talking about this new framework and what’s new on this slide is in the lower right hand corner. That is a brief summary of what we were able to do with the QUERI four-phase framework for conducting trials. In this, as with many other parts of the frameworks in these diagrams, it points out aspects of implementation research as if typically conducted where we will also take shortcuts or skip steps and as a result, find ourselves with certain problems or lack of success.

Many of the studies in this tend to be large trials of the implementational strategies where in too many situations during the early weeks of the implementation process, the investigators will discover a significant challenge or barrier and a reason why the implementation strategy is not likely to succeed. It would be nice if we were to make those discoveries during a relatively short and inexpensive pilot test of a novel implementation strategy rather than waiting until we’ve committed to that strategy and have invested hundreds of thousands of dollars in a large study.

Where, for the sake of internal validity, we’re trying to maximize fidelity to an implementation strategy, even when we learn very quickly that that strategy has some major flaws. So at lease one aspect of this four phase framework is meant to encourage and remind us that we should be beginning with pilot projects rather than launching immediately into a large trial.

If you could move into the next slide, this is a slightly different way of depicting and characterizing the four phases. Again, we will spend more time in small groups going through these. I would like to direct your attention briefly to a distinction between the phase two and phase three trials. This is an issue that I alluded to previously in terms of the artificial features of the implementation trials. In many cases, it actually is appropriate for us and we do wish to deploy and evaluate a novel implementation strategy and a novel practicing strategy. Typically, it is appropriate for us to evaluate that strategy under efficacy conditions such as high levels of research team attention and technical support, grant support for the sites and so on. Those studies are described here as phase two trials with an efficacy orientation and we do what we can to try to maximize the steps.

If we do succeed in achieving implementation through that effort, we actually are not done because again, we know that many of those features can’t be replicated in the larger scale application of that implementation strategy. The solution to that problem is what’s labeled on this table as a phase three effectiveness study. That is the second trial of that implementation strategy but under advocacy – I’m sorry, under effectiveness conditions, where we do no provide grant support for staff, we don’t provide grant supervision or technical assistance. Instead, we determine whether the implementation strategy is likely to be effective under routine real world conditions. And ideally, we would follow our phase two trials with an efficacy orientation. We would follow those with a phase three trial or what we sometimes label as regional rollout studies and what we label outside of the VA as scale up and spread studies. We would conduct those studies in order to understand whether our innovative implementation strategy that did seem to be effective in our initial studies actually continues to be effective when it is deployed on a larger scale.

There is a marketing type of phase, but we will discuss that during the small-scale sessions. Let’s move now to the next slide and I will essentially tee it up and point out that these issues will be discussed throughout the remainder of the session but my point at this point in the talk and the EIS program is to point out that there are many features of typical implementation trials as they are often designed and very often funded by VA and NIH and when published, there are many features of those trials that are problematic in terms of specific design issues. 

The standard way of approaching a trial of an implementation strategy is to try to emulate a colleague’s clinical research and to design and conduct a gold standard RTC using the typical kinds of rules of thumb and standards that are applicable to clinical research. The problem is that implementation phenomena and implementation strategies are not the same as clinical phenomena and drugs and devices. We see much greater levels of treatment and greater levels of context dependence and influence, much weaker main effects. We see much higher rates of variability and permutation strategies can vary over time and across different settings. We can actually vary them ourselves, with a typical drug we can often vary the dosage, but that’s about it.

So, the level of heterogeneity, the level of variability and instability – all of those are much great in the implementation world and the results our trials need to be designed in a different way. Again, this is an issue that we will discuss during the remaining sessions of the program. So, let me move, then, to the final poll question. We’d like to get a sense from you of which of these kinds of research you have conducted in the past. Molly, if you could pull up that final poll. 

Moderator:  Not a problem. It is up and the responses are streaming in now.

Dr. Brian Mittman:  Great. So, based on the descriptions provided, how many of you conducted step three pre-implementation research, how many have conducted small-scale or pilot trials or larger efficacy oriented implementation trials, how many have conducted the phase three regional roll out scale spread studies and how many have conducted observational studies? And I think you should go ahead and read the results when they’re ready.

Moderator:  We have had about sixty percent of our respondents vote so far. So we will leave it open for just a few more seconds. The remaining respondents will have a chance to vote. Okay. Looks like the responses have essentially slowed down enough. I am going to close this and share the results now, which are – we have thirty two percent that have done the three pre-implementation research steps. We have forty percent that have done the small-scale pilot implementation trials phase one, we have thirteen percent that have done larger efficacy oriented trials phase two, and we have eleven percent that have done phase three and we have thirty three percent that have done observational studies in implementation processes, step zero. 

Dr. Brian Mittman:  Good, so the numbers – why don’t we skip that last poll question that talks about thoughts on evaluation. That’s one that we can come back to during the session in which we will discuss the forms of evaluation. If you could move back to the outline, what I’d like to do is wrap up with just a brief minute of overview with the remainder of the program on the final slide.

So, as you see and as I’ve indicated, the next lecture in the series, number two, Allison Hamilton will deliver based on hybrid effectiveness implementation strategies. She will also spend some time talking a bit more about query step three activity – documenting and diagnosing quality gaps and the barriers and facilitators to improve. The remaining lectures are the two key components of implementation research. So in order to be an effective implementer and implementation researcher researcher, there’s basically two sets of activities and two sets of skills. The first is designing implementation strategies. We’ll have a couple of lectures that talk about the role of QUERI in designing and deploying permutation strategies. The second set of issues is about evaluation and the final two lectures in the series will present a set of frameworks and approaches with tools and so on for evaluating implementation strategies in an experimental or interventional mode, as well as some approaches for studying permutation processes and barriers and facilitators in more of an observational mode. That’s just a preview of the remaining five lectures in the series.

With that and having left only about three minutes on my clock for questions, Molly if there are any questions that have come in, I would be happy to take them.

Moderator:  Yes. Alright. For the very beginning, would you like me to start with those or go in order?

Dr. Brian Mittman:  Why don’t we take the ones that have come in the most recently? Those would be the easiest for me to answer.

Moderator:  Alright. That sounds great. What is a black box evaluation approach from the barriers implementation research progress?

Dr. Brian Mittman:  Black box is basically an approach where we are interested in the inputs and outputs without any sense of what happened inside. So a typical drug trial will randomize one hundred patients with a hundred for control and you measure changes in clinical condition or the key outcome and draw conclusions about the effectiveness of the drug based on those outcome differences with no interest in understanding anything about the inner workings or mechanisms of action. The implementation field is interested and I would argue in others as well, that we are more interested in the mechanisms of the action and what happens inside the black box and those are the types of evaluation strategies that we will discuss later on in the program.

Moderator:  Thank you for that response. The next question we have – what do you see the barriers to phase three being since this is so crucial to transformation? Are there test sets for this kind of study?

Dr. Brian Mittman:  That’s actually a good question and that’s one that I’m struggling with a group of colleagues. VA had co-funded a couple of years ago a state of the art conference on scale-up and spread research for phase three kinds of studies where we attempt to evaluate implementation programs in a much larger sample, rather than the smaller RCTs that we often conduct. It’s not clear, I think, you could probably point to all of the different stakeholders like research funding agencies that have an issue in funding announcements in a manner that I believe they should to call for this kind of research. Researchers often feel that when they manage to show success with quality improvement in a small-scale study, and they publish it that they are then done. The policy and practice leaders will read the results of their study recognize the value of the implementation strategy and will then rush to deploy that strategy. So, potentially as they fall for the same trap that our clinical research colleagues fall for who publish the results of their clinical studies, they think they will automatically be implemented. There are also issues of course related to a sample size and study budget and lack of data. Within the VA, though, we don’t have those problems at nearly the same level. It’s much easier for us to conduct large-scale scale-up and spread studies. The hope is that our funding programs and officials as well as researchers will become more interested in doing this type of work.

Moderator:  Thank you for that response. As we are approaching the top of the hour, I would like to skip to a general question, which I’m sure many people have on their mind. They are wondering how they can get invited to the follow-up sessions.

Dr. Brian Mittman:  The answer is to contact one of us and depending on the level of interest – the first thing is if you are in VA, please contact your local implementation researchers. It’s possible they are conducting small group sessions at your facility and you weren’t aware of them. It’s possible they have the capacity to include you. If there isn’t anything locally at your VA, get in touch with us and we will see what we can do. We may end up if there’s enough interest and demand, schedule another set of sessions later in the year. We feel that we’re at capacity at this point. The first question is for us to assess the level of interest and see what we can accommodate in given the current situation.

Moderator:  Thank you for that response. I can tell you that the level of interest for this session has been huge. You have a number of remaining questions. A large portion of our audience is still with us. Would you like to continue taking questions or would you like to end now?

Dr. Brian Mittman:  I can stay if there’s interest for those of you that do need to leave; I thank you again for your interest to the extent that the goal of CIPRS and this program is to increase the quantity and quality of implementation sciences. The more interest, the better. Please join us for the remaining sessions, but Molly, I’d be happy to take additional questions.

Moderator:  Excellent. And before I get to the next one, I would like to tack on a comment. If you do leave the session, you will be prompted to complete a feedback survey. Once you exit the session, it will popup on your screen. Please take just a moment to give us feedback because as Brian mentioned, we are trying to gauge the interest and future sessions we should be planning. With that, the next question is – by referring to implementation as quote science end quote means that there is a set of concepts and methods that can be repeated successfully, for example in health. Is implementation an up and coming discipline or is it a mix of disciplines? How is it related to health services research, et cetera?

Dr. Brian Mittman:  That short answer is yes to all of the above. At this time and in the remaining lectures of the program, we will talk about the series, the evaluation methods and the systematic approaches and the reliable and reproducible results as well as some of the complications to obtaining those. It is up and coming and is still developing. I alluded to or mentioned a small number of controversies and there are and lack of consensus and there are many more, some of which we will address in the remaining lectures. It builds on a number of disciplines like sociology and psychology, economics – it uses concepts from program evaluation. It’s not completely new. But it does represent a somewhat new mix of the different disciplines as well as a number of issues that are somewhat unique, even if implementation sciences can help. It is related to health service research, but again is somewhat distinct, hence my initial answer which is yes. I think all of the items that you mentioned are true and they do characterize the field.

Moderator:  Thank you for that response. The next question – whether an intervention gets implemented seems to be a function of the implementing agent’s outcome expectations. In parentheses, how important is it for me to do this?  And efficacy expectations, can I do this? Therefore, what are your thoughts of the notion that implementation is simply a function of proper training slash coaching and holding the implementing agent accountable for implementing the intervention?

Dr. Brian Mittman:  One of the key phrases and concepts or ideas in the field is the idea of necessary but not sufficient conditions. I think that I and others in the field would probably agree that the items you have mentioned are among the list of necessary but not sufficient conditions. We do need training and we need to overcome expectations and external pressure. We do need internal motivation but we need many more things as well in many cases. There’s a very long list of conditions and factors that need to be present in order for us to see a successful implementation. The specific list varies by the problem and by the setting so it is likely that those two items, if they were present, would be sufficient for certain quality or implementation problems. By and large, the list of necessary but not sufficient conditions tends to be longer and that’s what makes the field both interesting as well as challenging.

Moderator:  Thank you for that response. The next question – what does SDT stand for on the framework slide? It is stated SDT template that was not going to be discussed here.

Dr. Brian Mittman:  That’s QUERI speak and it gets back to the opening remarks about acronyms. SDT is the service directed project, a funding mechanism within VA. It is a funding program that supports quality improvement trials and other types of efforts. The SDT template, which there is an example in the QUERI series overview article and another version that has not been published – it’s basically instead of guidelines or recommendations for design or write-up for a grant or funding application for a service directed project. Anyone who is interested who contacts me by email, I’ll send you the QUERI series article that contains one version as well as a second version that I also present.

Moderator:  Thank you. That question was actually submitted by several people so I appreciate you answering that. Also, what do you feel the barriers to phase three being since this is so – oh, I’m sorry? We’ve already done that one. I know it will be covered in later lectures, but could you possibly give a hint at what you would define a successful intervention? For example, what percent adoption or level of participation?

Dr. Brian Mittman:  I suppose it’s the same question that a clinical researcher would ask if you have a level of thirty percent of patients are able to accomplish daily activities of living and you achieve improvement through a medical or behavioral program to get that up to fifty or sixty. That would be a significant improvement. If we agree with appropriate use of beta-blockers, for example, at thirty percent and we’re able to achieve an improvement up to fifty or sixty percent, that would be significant improvement. If we get that number to eighty or ninety percent – I don’t think the answer is all that different from implementation research versus other types of interventional research. We’re trying to improve something.

Moderator:  Thank you for that response. You just let me know if I need to stop giving you more questions.

Dr. Brian Mittman:  Let’s do one more because I know that our small group is scheduled to meet right after this.

Moderator:  Okay. I am going to skip to the one that will probably apply to most people. Do you have a core reference for the query steps?

Dr. Brian Mittman:  Yes. The QUERI series overview article, which we can post as part of the follow-up. It is the most complete description of the QUERI steps and again, for those who are participating in the EIS program, that was the assigned reading for the lecture. If you can post it and make it available, the others would have access to it as well. 

Moderator:  I would be happy to do so and thank you for making note of that. This session has been recorded and we will be posting it in the cyber seminar archives along with any additional handouts or references that Brian provides me with. With that, I would very much like to thank Brian for sharing his expertise and our audience for joining us. Brian, I would like to give you the chance to make any last concluding comments.

Dr. Brian Mittman:  No, other than to thank you again and again and encourage those who are still on the line to join us for the remaining sessions. Molly, thank you again.

Moderator:  Very happy to help and thank you everyone. This formally concludes today’s presentation. Have a nice day.

**[End of Recording]**