**Transcript of Cyberseminar**

**Enhancing Implementation Science**

**Advanced Program Session 1. Pushing the Implementation Science “Envelope” in VA**

**9-20-2012**

**Presenter: Brian Mittman**

**Enhancing Implementation Science**

**9-20-2012**

Molly:So I would like to introduce our speaker for today.  Kicking off this mini-series we have Dr. Brian Mittman.  He is a senior advisor for the Veteran’s Affairs Center for Implementation Practice and Research Support and I’m very happy to have Brian kicking off this series.  Dr. Mittman, are you ready to share your screen?

Dr. Mittman:I believe I am, yes.

Molly:Excellent.

Dr. Mittman:Okay Molly, thank you for the introduction and, as usual, for all the arrangements and welcome to everyone.  I know that we have a mix of EIS program attendees and have some implementation science training program attendees as well as others.  What I would like to do is give my talk first and then talk a bit more about the series and the subsequent sessions in the series at the end of the hour.  So with that let me move to the next slide.  

So just a brief overview of what I propose to say and why.  What I’d like to do is reflect a bit on recent progress, on the current status and the future direction of implementation science both within VA but outside VA as well.  With the specific goal of identifying opportunities for improvement for things that we as researchers can do to, as I indicated in the title, enhance the value and the contributions of the research that we do.  By way of disclosure in explanation this is a presentation that is adapted from a talk that I gave at a conference organized by the Academy for Healthcare Improvement this past May.  The topic that I was assigned was “the future of implementation science”, and so what I’ve done is adapted that talk for this purpose.  I should say that the sessions archive versions of the talks from that conference are available both on the VA website as well as the Academy for Healthcare Improvement website.  

Now, because the topic I was assigned for that talk was the future of implementation science, I spent a bit of time trying to understand and think about what was meant by that topic; why we would be interested in the future of implementation science.  So I was able to locate a few quotes that I think are helpful in thinking through what we as implementation researchers should be doing vis a vis the future of our science.  So that the first point, of course, is that all of us should be interested in the future as indicated by this quote “Because we are going to spend the rest of our life there.” So it’s clearly something that is of concern.  

The next question though is: How do we go about predicting the future and trying to project what the future of the field is so that we can perhaps modify our research and prepare us for the future and insure that what we do is as useful as possible and as relevant as possible?  This is a quote that has been attributed to a number of individuals likely of course that Niels Bohr, someone back at that point, first stated and others have repeated it but that the point the quote states “Prediction is very difficult, especially if it’s about the future.”  So that does pose a bit of a challenge to the task of trying to predict the future of implementation science.  

In terms of specific strategies and techniques for predicting the future, this quote seemed to be most valuable as I looked through a series of possibilities.  This is from Scott Adams who is the creator of Dilbert, the comic strip, and wrote a book a few years back, “The Dilbert Future” and his comment is “there are many methods for predicting the future.  For example, you can read horoscopes, tea leaves, tarot cards, or crystal balls.  Collectively, these methods are known as “nutty methods”. Or you can put well-researched facts into sophisticated computer models, more commonly referred to as “a complete waste of time”.  So again, a bit of a challenge for the task of trying to predict the future in general and the future of implementation science in particular.  

So this quote actually was the one that I stumbled upon which basically provided me with the guidance that I needed to prepare to talk and that is from Alan Kay, director and scientist from Xerox PARC, Palo Alto Research Center, and later joined Apple Computer.  He and colleagues were responsible for the mouse, graphical windows type of interface and so on and so forth.  And his statement is “The best way to predict the future is to invent it.”  And I think that is a relevant and useful idea for us in implementation science.  It probably is not very useful for us to sit here and try to imagine or predict what the future will be; it’s one that we have the ability and the obligation, I think, to invent.  

So before I launch into the core of my talk, I would like to ask this polling question; for you to indicate your VA affiliation and to state your level of prior implementation research training and experience.  Molly?

Molly:Thank you Dr. Mittman.  As you can see, to our attendees, there is a poll up on your screen at this time.  Please do select the answer that most closely indicates your VA affiliation and your level of research training and experience.  We have had half of our respondents reply so far, so we will give the remaining attendees a little bit longer to answer and we do appreciate your participation.  It does help us guide the level of the content within the presentation.  And we have had about an 80% response rate and the responses seem to have stopped streaming in so at this time I am going to close the poll and share the results.  Would you like to talk through those Dr. Mittman?

Dr. Mittman:Yes.  I’m going to give it a minute for them to come up on my screen.  So let me close one window.  So it looks like most of you are VA researchers, VA affiliated with a pretty even mix of those with prior implementation science training and those without.  So, I guess what I would just say to those without, ask you to not take too many of my comments in essentially criticisms of the field as an indication that  it is not a field that offers value or is worthwhile.  This talk was meant to kick-off the advanced program for implementation science based on the understanding that most of the attendees would have had prior training, would have been working in the field and are ready to sort of examine some of the weaknesses and the gaps and flaws in the basic approaches and move to a somewhat higher level of more sophisticated understanding and work in the field.  So, for those without experience, if you would keep that in mind.  

So let me turn back then to the slides and move on to my next slide and begin to talk a bit about predicting the future and inventing our future.  And the first question, of course, is: What has happened in the past and are there some trends that we can use to try to understand where we’re headed and perhaps how we might redirect our trajectory?  

So a somewhat oversimplified short history of the field of quality improvement research or implementation research is captured in this slide.  We began with a large body of research in the 1970s for the most part consisting of work assessing quality, examining small-area variations in practices, and trying to determine whether variations in rates of specific medical interventions were a sign of poor quality care overuse or underuse.  That work and the recognition that there was good levels of evidence of overuse or underuse and general poor quality led to an increase in interest in research that was labeled “changing physician behavior”, and that was for the most part in the 1980s.  This work had begun in the 70s and before and it continued, of course, into the 90s and beyond but I guess my assessment is the bulk of the work that was labeled and viewed as the field of changing physician behavior did take place during the 1980s.  One of the key assumptions here is that physicians were responsible for the vast majority of all clinical decisions and resource allocation decisions and so on and if we have quality problems overuse that the key to addressing those problems was to change the way the physicians practiced.  Around about early 1990s, ideas from industrial quality improvement, TQM (total quality management), CQI (continuous quality improvement) and others were introduced into healthcare.  Don Berwick and others wrote a number of articles that introduced these ideas and advocated for their use in healthcare along with a focus not on individual clinicians in behavior but instead, on the system and system design.  This was followed by really the launch of the field of quality improvement research, AHRQ was responsible for the support funding for much of this work.  

Moving into the first decade of the current century the quality chasm reports’ work from the IOMs clinical research roundtable and publications from NIH identifying translational roadblocks and focusing both on quality gaps and the need to close those as well as deficiencies in the implementation and adoption of research findings from NIH funded research that stimulated and really began to direct attention to the implementation gap in addition to the quality gap and that led to, along with a lot of other trends, movement away from the fields and labels of changing physician behavior and quality improvement of research towards implementation research and implementation science, the Journal of Implementation Science that we launched, the Academy for Healthcare Improvement, and a number of other groups and efforts began in this period as well.  

As we move into the current decade, which is dominated of course by health reform and the Affordable Care Act and its implications, we see yet more transition and evolution of focus now on comparative effectiveness research and patient centered outcomes research, CMS and the CMMI, the Centers for Medicare and Medicaid Innovation Center and the work that it’s funding, the AAMC, the Association of American Medical Colleges, has launched an implementation science initiative and is investing effort in advocating for greater interest in activity in the field of implementation science.  So that’s more or less where we are today.  Along with this trend in this evolution of the field we’ve seen a dramatic increase in the level of awareness and developments focused on methods in the methodologic and scientific foundations of implementation science, in addition to conducting the empirical work and this does include attention and work in the area of theory; attention to the issue of contextual influences; on implementation processes and strategies; a focus on implementation processes and mechanisms and mediators as opposed to impacts.  And all of this is ongoing and I will talk more about each of these during the next few slides.  But again, this is a somewhat oversimplified history of the field and where we are today.  And I show this and review this again as a way of trying to understand where we might be headed; what sort of trajectory we are on; and what kinds of refinements might be useful in trying to improve the field and its contributions to better implementation, better quality, and better outcomes of care.  

So, it’s useful also to sort of step back and assess in a more quantitative manner where we are and these are my own personal scores on a set of dimensions that I would argue are important in understanding where we are.  So, if we think first of all about current levels of quality and safety; value and efficiency in healthcare delivery; current levels of utilization of best practices; rates of adoption of effective practices; and what sorts of improvement we’ve seen since the year 2000, and the goal here, of course, is to try to both understand how much distance we still need to go in terms of improving quality but also how much progress we’ve made in improving quality and as a result, whether the research that we have been doing has been useful.  And again, my own personal assessment here is that we are somewhere along at a 2 on a 5 point scale; that we still see significant gaps in quality and safety and value and utilization.  And the level of progress that we’ve seen since the year 2000, in the past 10 years, is relatively modest.  So as an implementation researcher, in some ways I suppose this is good news; that there is still plenty of work for us to do.  On the other hand, it does raise some questions as to what our contributions have been and what value we have offered and continue to offer to the policy and practice audience that we are hopefully serving.  

So the next dimension, well, I’ll offer my personal assessment, is of the volume of insights, useful findings and practice and policy-relevant guidance that the QI research and implementation research fields have offered over the last 10 years.  And here, I’m definitely going to take a pessimistic view and a more critical view and argue that if we focus only on useful findings and practice policy-relevant guidance, at least in terms of guidance that our target audience seems to appreciate and understand and use, I would argue that we’re only at a 1 on a 5 point scale.  When we talk to our practice and practice and policy colleagues and ask them questions about the implementation science field and specific findings and guidance they don’t offer much by way of positive reinforcement and feedback so their feeling is that the fields have not given them much of the tools and insights and guidance that they need.  And, you know, we can argue about whether I’m being too critical but again, the bottom line is I think we have a long way to go.  

What about the actual volume of QI research activity; the amount of interest in grants; the amount of growth that we’ve seen since 2000?  So, is this a matter of or a problem of quantity or quality? And I would argue that quantity-wise we are doing very well; that we have seen good levels of growth in interest and in some of the trends that I presented and listed on the previous slide, I think, are some of the evidence.  The increased interest in funding from NIH, from AHRQ and other foundations; the annual conference that NIH sponsored; the AAMC interest; the interest on the part of CTSI and so on.  So I think in terms of convincing funding agencies that there is work to be done and that this research is needed, we’re doing reasonably well.  Of course, we could use more but again, I think quantity wise, we’re on a good trajectory.  

Similarly, the volume of QI publications and presentations and the growth that we’ve seen, a good level of interest and activity, lots of attendance at the annual NIH conference for example, at the IHI conference scientific symposium and others internationally, lots of growth in the rate of submissions to the Journal Implementation Science and other special issues of general medical journals, so there is plenty of research that’s being conducted and plenty of research that’s being published and presented.  So what we conclude from a comparison of my scores on the first two bullets relative to the last two, at least one conclusion is that there has been a fair degree of growth in activity but it has not produced comparable growth in insights and guidance and impacts.  So again, taking a somewhat critical view, I would argue that the problem seems to be one of quality, not quantity.  But, there’s something about the research that we do that is not yet leading to the kinds of insights and guidance and value that we need to be producing for the policy and practice leaders who look to us for guidance in understanding how to better improve quality and more rapidly facilitate the adoption, the implementation of effective practices.  

So why is that?  This is a list of five topics or issues that I will cover during the next set of slides that represent possible diagnoses of this problem.  Possible reasons why the quality of the work that we are doing may not be as high as it could be and why we may not be generating as much guidance and useful insights as we would like to.  So let me go through the first of these.  So the first possibility is that perhaps it’s not necessarily us as implementation researchers or implementation practitioners who are, in a sense, to blame but instead something about the evidence that we are trying to implement.  Maybe we’re trying to sell practicing clinicians on a set of findings and evidence that they don’t feel are valuable.  And the key distinction here that’s necessary to think about is the distinction between efficacy and effectiveness research.  Larry Green, Russ Glasgow and others who have argued in a series of publications said the external validity; the relevance of much of the clinical research is not as high as it needs to be.  The set of publications and ideas related to practical clinical trials, pragmatic clinical trials are what are probably more properly labeled, effectiveness trials, has addressed this issue as well.  And the bottom line is that, practicing clinicians recognize that a significant proportion of the published literature provides evidence of the theoretical efficacy of clinical interventions in academic settings in many cases, among healthy white males in many cases, and with other artificial features that limit the relevance of that evidence to them.  In the community settings in which they practice with the patients that they manage who have multiple comorbidities and other challenges.  And that evidence that a certain new innovation in healthcare delivery or treatment is effective under limited circumstances is not seen as relevant and valuable.  And for us, as implementation researchers, to be working hard to try to convince clinicians to use evidence that they don’t see as valuable, no matter how many clinical reminders and audit feedback mechanisms and opinion leader sessions and CME programs we offer, if the evidence itself is rejected no matter how well the clinicians understand it and no matter how much we try to impress upon them the importance of using it, we’re fighting a losing battle.  So that’s a possibility and this is at least one consideration that we need to keep in mind.  Of course, PCORI, Patient Centered Outcomes Research Institute and efforts on the part of NIH, AHRQ and others to fund comparative effectiveness research and effectiveness studies as opposed to or as a compliment to efficacy studies, will be helpful but to us as implementation researchers, of course, one of the key implications is that we should be looking to effectiveness study results when we try to implement evidence based practices, not efficacy results.  And that we should be trying to assess, at a very early step and stage in our process, we should be trying to assess the acceptability of the evidence.  And if we identify early on that clinicians don’t trust the evidence and don’t feel that it is relevant, we need to either do something about that or look to other types of evidence in other practices to try to implement.  So that is one possibility and one set of issues.  

The second set of issues that I would like to talk about briefly are the possibility of external barriers; that we don’t have enough funding; that we aren’t able to recruit study site subjects in large enough numbers, that we face IRB barriers; and that we’re just not able to conduct the research that we need to be conducting.  Just a couple of editorial comments on this, and again these as with many of my comments represent my own opinions that I recognize are not necessarily shared by many of our colleagues, but there has been for a number of years debate over the issue of IRB approval and regulatory constraints on QI research and implementation research as the VA has issued its own new guidance.  My view is that many of our colleagues are focusing on the wrong problem.  That we should not be trying to argue that the research that we do is not research and therefore should not be subject to IRB review, but instead we should be focusing on the review process and trying to insure that our IRBs recognize that for the most part QI research is low-risk.  It can be expedited or exempted but it does require IRB approval.  These are not service delivery projects or operations projects that we then later decide are research.  From the very beginning, much of the work or most of the work ideally that we do, should be viewed as research but it should be viewed and understood that it’s a specific type of research for which the standard IRB review is appropriate if it is done appropriately and if the kinds of regulations that exist to allow for expedited review and to allow for reviews to be conducted in a rapid manner when it’s low-risk research, those are applied.  And of course, we are all waiting for OHRP to issue its new guidance regarding central IRBs and so on but within VA we are not subject to many of the kinds of constraints and difficulties that our non-VA colleagues are subject to as far as multisite reviews.  So that’s one issue.  

Another issue has to do with funding sources and mechanisms.  Do we not have enough funding?  Are there problems in the review committees?  Do we lack sufficient guidance for reviewers so that they can review our studies properly?  Do we need more consensus on design and methods?  And my answer to most of those would be, yes.  On the one hand we do have, maybe not sufficient funding available but, a good level of funding that’s available to us but there are definitely problems in peer review of our grants.  I guess I would direct our attention back to ourselves and as a field, point out the need for us to reach consensus on specific designs and methods and develop guidance for reviewers so that not only our colleagues within our field, when we review our own grants, but also on those occasions when NIH review committees and others recruit reviewers from related fields that those reviewers have some guidance to rely on.  And if we feel that some of the rules differ for us in implementation studies from other forms of clinical or medical research, we need to say that and we need to provide the guidance.  So this is likely to be a barrier to some of our research and to again, more value/better contributions, but I think the responsibility lies on us as a field to address this issue.  

Then finally, access to study sites is one barrier.  The fact that we can’t recruit enough sites and again within VA this is not a problem that we typically face but there is interest in developing better approaches for partnerships with study sites and identifying some of the barriers that they view to participating in our projects.  Especially the university based CTSAs and CTSIs funded by NIH conduct more of this type of research and the hope is that better guidance will be available for effective partnerships and effective ways of working with study sites; not only to convince them to participate in our studies but to convince them to participate in a meaningful way so that they view these as useful projects and they provide the level of support and engagement that we need rather than simply providing access but without the kinds of management support and other support that we need.  I’ll talk about that a bit more over the next few slides.  

The next set of issues I would like to discuss relate to professional and institutional factors; aspects of the disciplines that are involved in this field; the terms and concepts that we use.  And the challenge here is that the field of implementation science has multiple origins and has multiple foundations.  As a consequence we continue to see a very diverse and often times confusing set of labels or set of subfields.  So the fact that some of our research is published in the Journal of Implementation Science that uses that term; some is published with the label of knowledge translation, research utilization, technology transfer, diffusion of innovations and so on.  So it’s sometimes difficult, especially for those who are entering the field, to make sense of this and understand what it is they’re reading and what to make of it.  Those who have been working in the field, of course, recognize all of those labels; know that for the most part those labels describe the same types of research and it’s not a problem for the experts and the experienced researchers to have this situation of multiple terms and labels.  On the other hand, and that’s really the, let a thousand flowers bloom sort of attitude that many of our colleagues feel is the right way to go.  That it’s not really a problem for us to have this confusion and the tower of babble as long as all of us understand what the terms mean.  And that again, that may be fine for those in the field but, you know, if we’re hoping to grow the field, if we’re hoping to increase the value and the impact of the field, if we’re hoping to increase external recognition and appreciation of the field, we’re not doing ourselves any favor by calling ourselves different labels and by describing and publishing our research under different terms.  So on the continuum of let a thousand flowers bloom versus we really need to sit down and hash all of this out and develop a limited set of terms with consistent definitions and agree to use them, I’m definitely on the side of consistency.  The tower of babble phenomenon, of course, is also a barrier to research synthesis and shared learning within the field, not just outside.  There is some slow progress in establishing a more inclusive and more cohesive community, discussions over the establishment of a new professional society, they are going very slowly so to me this remains a problem and again I would point to this as at least one of the contributions to the glass half empty sort of status that I think we have where we haven’t generated as much useful guidance and contribution to improvement in quality as we potentially could.  So this is an area where again, in terms of creating our own future, I hope that we will create a more inclusive and a more consistent future of terms and concepts and labels.

Let me turn now to the issue of the theoretical and conceptual foundations for the field.  The issue here is the possibility that our lack of progress in development of useful insights relates to the possibility that we simply lack adequate theory.  That we either don’t have enough theories or that their scope is wrong or their value is limited.  On the other hand we see just as many arguments, if not more, and especially with the relatively recent publication of a nice review of 60+ different conceptual frameworks and theories, there is also the possibility that there are just too many theories and that we’re all sort of talking past each other because we all apply different theories and in consequence our ability to synthesize our findings and develop strong results and patterns, to see patterns, is limited.  My answer here is that we probably do have more than enough theories, that the search for the universal theory of physician behavior change or system change or clinician practice change is probably not useful.  What we need is better guidance in using theory.  So that, and others have called for a moratorium on new conceptual frameworks and new theories, but instead, more effort to continue to try to synthesize and assess the existing theories and try to provide guidance in when and how to use them.  Not only in designing a study and developing a grant application and obtaining funding but more importantly in conducting the study and perhaps more importantly still, in reporting the results of the study and better use of theory in trying to synthesize findings.  And again, this is an issue that I’ll come back to in a few slides.  So as I would place this on the short list of issues that are responsible for some our limited contributions we do need more attention and better work related to theories and conceptual foundations as a way of trying to enhance our value.  

So the fifth issue that I will spend the most time discussing is the issue of research approaches, designs and methods.  And the question here is whether the published disagreements and debates that we see over research approaches and sort of oversimplifying the issue is do we need more RCTs or fewer RCTs but there are others as well.  The issue is whether their disagreements over the relevance and the value of different research approaches and designs for specific questions or, is the disagreement or lack of consensus focused instead on what are the key research questions and goals.  So if we see two or three different kinds of research approaches that are dramatically different that have been advocated, one possibility is, for any given research question, we just don’t agree which of those is the best and most appropriate approach.  The other possibility is that we are talking about different research goals or questions.  And for a certain set of questions, one approach is clearly better for a different set of questions and another approach is clearly better and we would probably see good agreement with that.  What we don’t agree on then is which of those research questions and goals are most important.  So what is it that we are trying to achieve as a field?  It’s the second version, the second argument that I would like to advocate over the next few slides.  

Before I do that though, let me address a specific issue that relates to the topic or the theme of research approaches and designs and methods.  It ties back to what I said earlier about the need for us to develop better guidance for us but also for reviewers.  This is an example of what is likely to be a long list of features of study designs for implementation science that need to differ from the comparable features of study designs for clinical research and other forms of research that we conduct.  As a consequence, when we ask reviewers who are experts in clinical research to help aid the research to come in and review implementation studies and they apply the usual rules that apply to their field, when they apply those to our fields they get it wrong.  The specific example that I would like to use is the issue of representative sampling; something that I call the representative sampling paradox.  So the question is:  How do we go about sampling clinicians or sites in implementation studies, specifically if we are conducting clinical trials?  What we know from clinical research, evaluating drugs and devices and so on, is the best way to recruit and create a representative sample is to sample randomly from a population.  The reason is that the goal of sampling or the goal of research is to try to estimate the effectiveness of the given intervention two, three, or four years down the line.  So when drug companies and clinical researchers conduct clinical trials of a novel medication, their goal is to provide data that will support an FDA approval decision.  The FDA would like to know if this medication will be effective in three years if it is approved and it’s marketed and made available to the population.  So what clinical researchers do is recruit a representative sample of the population knowing that 200 randomly selected patients now will be highly representative of the two hundred million patients for whom this medication may be offered or approved for and potentially used in the future.  When we study and evaluate implementation strategies we are trying to achieve the same goal.  We are trying to estimate the likely effectiveness of that implementation strategy three, four, five years down the road.  When within VA, for example, VA leadership is likely to endorse that implementation strategy and support its use nationally.  So the question is: How do we identify a sample of VA Hospitals and Clinics that are representative of that future state of affairs so that we can conduct a realistic and valid test of effectiveness of that implementation strategy here and now?  The issue though for us and the challenge is the fact that the effectiveness of that implementation strategy is likely to be very different three or four years from now than it is right now.  The effectiveness of implementation strategy is that it’s highly dependent on contextual factors such as VA National Leadership support.  So if VA decides in three years to endorse a program and to issue a directive mandating use of that implementation strategy, a new care model or whatever the case may be, a new clinical reminder, if it’s rolled out nationally and endorsed and if it’s mandated then the propensities for individual medical centers and clinics to adopt that strategy will be much greater than it is right now.  So the question is again: How do we recruit a representative sample?  The way that we do that is to recruit sites who know us and trust us and are likely to be predisposed to acceptance and support for implementation of that strategy and for implementation outcomes.  Our clinical research colleagues, of course, would label that a biased sampling strategy that is not representative but the issues and the situations, the phenomena for us as implementation researchers are different.  Our context and our situation and our population will not be the same in three years as they are right now and so randomly sampling across the board leads us to a biased sample and we will underestimate the likely effectiveness of the implementation strategy.  So again, it’s just one example of what’s likely to be a long list of ways in which implementation researcher approaches and designs and methods have to be different even when we’re answering the same kinds of questions as our colleagues in clinical research.  So that’s one issue to keep in mind as we again try to invent our future as implementation researchers in this broad theme of research methods and approaches and designs.

What I would like to do though is to return to a broader issue and that is the issue of different kinds of questions.  The question as to whether the debates that we see over, for example, RCTs and are they are good idea or a bad idea?  Are they relevant and applicable or not?  Is it a matter of the relevance and applicability and value of those methods or is it a matter of different research questions that we are trying to answer?  And again, I would argue that we are trying to answer different research questions, or at least similar questions but about different types of phenomena.  The key concept here is the concept of complex social interventions.  When we evaluate implementation strategies as opposed to when our clinical colleagues evaluate drugs and devices, we are evaluating strategies or programs, behavior change or practice change programs or interventions that are highly variable and highly heterogeneous.  So despite our best efforts to maintain fidelity, we just don’t see it.  There is a good argument to be made that we actually don’t want to see fidelity; that we have programs that can be adapted and customized to local conditions.  Unlike our clinical colleagues who evaluate a medication that may come in two or three dosages, but that’s about it, our programs consist of a lot of different moving parts and components and they can be adapted and they are adapted just naturally.  We see that the way in which those programs are implemented will vary across time and place as well; similar heterogeneity.  We also see, as I indicated previously, strong contextual influences; the effectiveness of an implementation strategy or the implementation outcomes.  The rates of improvement that we see in quality or implementation very often times, as much if not more, by virtue of the local leadership, the culture of the sites, their experience and capacity, the extent to which their budgets are sufficient, those factors sometimes dominate the main effect of the implementation strategy or the implementation intervention; so again, different kinds of phenomena that lead to different kinds of questions.  Often times, and I would argue more often than not, the main effects of our implementation programs are relatively weak, relative to the contextual effects.  So the question is: What do we, as researchers, try to do and what kinds of questions should we be trying to answer?  Let me compare two kinds of questions and then talk about their implications for research approaches.  So if we have a complex social intervention that is relatively robust and stable where it does have a strong main effect and where the effect is likely to be consistent across large numbers of sites and across time, then the RCT that is designed to estimate the effect of that intervention is a useful approach.  If we have a large enough sample size an RCT also allows us to examine the strength of the small number of contextual factors or effect modifiers.  Typically though, we see complex social interventions that don’t have strong main effects that do have large contextual effects that are characterized by large degrees of heterogeneity, variability across sites and across time.  Then for those the RCT is not a very useful method but we tend to study the robust complex social interventions and we tend to use the RCTs because that’s what we know how to do and that’s what we like to do; because our clinical colleagues have told us that that method is the gold standard.  The question that is answered by the RCT is: What is the effectiveness or what is the impact of this intervention?  The RCT doesn’t do a very good job of answering why a specific intervention was effective, how it was effective, or how its effectiveness varies over time and place.  So basically what we have is essentially a continuum of robustness, for lack of a better word, of these interventions where at one end they are highly robust, highly effective, strong, consistent main effects, and at that end of the continuum the RCT is a useful method.  At the other end of the continuum where the interventions are highly variable in their impacts, where they are relatively weak in terms of their main effects, where the contextual factors tend to dominate, we see a different, very different situation, and at that end of the continuum the RCT is not very useful.  The final bullets on this slide are an attempt to identify some of the key dimensions or factors that would allow us to determine and predict the likely utility of an RCT.  

So getting back to the key issue here: What is our goal? What is it that we as implementation researchers are trying to achieve? And, what kinds of questions are we trying to answer?  For the most part, the RCTs that we conduct are an effort or a strategy or a tool for answering the first set of questions; does it work or is it effective?  These are the kinds of questions that our clinical research colleagues try to answer through clinical trials of drugs and devices.  Should this drug or device be approved by the FDA?  Should it be included in the formulary? Or, should I use it?  For an average patient, is this drug effective and, therefore, should it be used?  If we are faced with a situation where the kinds of interventions that we study and the kinds of programs and strategies that we use don’t have a simple answer to the question, does it work, because, the answer is, well, it works sometimes under certain circumstances and if it is used in certain ways in conjunction with other kinds of strategies; that’s a very different question.  The kinds of answers that we as researchers should be generating to those questions are not answers to, does it work? But instead, how does it work? Why does it work? When and where does it work?  The guidance that we are providing to our decision makers is not, yes/no, you should approve it or you should use it but instead, here is how you should use it and here is what you can do to make it work.  That’s the kind of question that for the typical clinician, using a specific medication is not a very useful question.  The decision that they face is, should I prescribe this drug or not?  Now there are other questions that they face as well.  What kinds of exercise should I be counseling?  What kinds of changes in home life or stress or other kinds of activities?  So there are contextual factors that apply to clinicians as they think about clinical decisions.  The point or the argument is that the contextual factors that we face as implementation researchers are much more significant.  There is an argument that has actually been published that states that, in the implementation or the quality improvement domain it actually doesn’t matter much which particular implementation strategy or QI technique you as a decision maker or an organizational leader would chose.  What is more important is how you use it and what else you do in your organization.  Now clearly in the clinical setting, that kind of thinking doesn’t make much sense.  When you are faced with a patient with a specific diagnosis, it does matter which medication you choose because we know that certain medications are more useful and more effective for certain diagnoses.  But, in the implementation or the quality improvement domain perhaps that’s not the case.  Perhaps our focus should not be on selecting a specific implementation strategy but instead on managing the organizations properly and doing the right things in conjunction with that implementation strategy.  Also our focus should be on modifying the implementation strategy and adapting it in a way that increases its effectiveness; equivalent to modifying the formulation of a drug to increase its effectiveness.  Again, that can’t be done.  It is possible to titrate the dose and so there are some parallels in the clinical decision making world parallel to the implementation or QI decision making world but again, it’s a matter of degree.  For us, I would argue that most of the action is probably in how we titrate or adapt our implementation strategies and how we modify the organization.  So, if that’s the case, what are the implications for us as implementation researchers and what kinds of questions should we be answering for decision makers and what kinds of evidence and insights and guidance should we be developing and providing?  Here is a list of some of the kinds of questions that I think we should be answering:  How do I choose an appropriate implementation strategy given my context, so to the extent that there might be some connections and some links between the characteristics of a setting and the effectiveness of different strategies?  This is an important question.  But again, based on the idea of the adaptability and complexity of these complex social interventions, the second question I think is an important one.  How do I implement or deploy that strategy to increase its effectiveness?  It’s the equivalent of, do I counsel that the patient take the medication with or without milk, or with or without food, and so on.  So there are questions that relate to how the strategy is deployed or used that influence its effectiveness and as researchers we should be studying these processes and these phenomena in generating guidance and insights.  How do I adapt and customize the strategy to increase its effectiveness both initially and over time?  The clinical decision, the prescribing decision, is often to prescribe a medication and then conduct some monitoring over time but in many cases patients remain on a medication for months and years.  With implementation, that’s probably not the right way to think about it.  We need to be constantly monitoring the phenomenon of the organizational learning curve, where the organization changes and adapts to the organizational change program and where the organization changes that program itself and adapts the program to match the organization.  Those kinds of changes occur continuously and as a consequence we need to both initially adapt the implementation strategy to fit the local context but we also need to constantly monitor and adapt the strategy over time in order to maintain its effectiveness.  Then this additional question, which is quite different but still important and that is: How do I modify or manage the organization or the setting?  So maybe the goal is not to try to select the best magic bullet, the best implementation strategy because any implementation strategy could be effective.  Maybe the goal is not to necessarily adapt that strategy because in some cases the strategy may be fixed.  Perhaps what I need to do instead as decision maker or leader is modify and manage my organization and improve quality by modifying the culture or by providing additional training or staff support in implementation so this implementation strategy that I’m using actually has some hope of success because the staff who are responsible for using that strategy will have the training and support that they need.  Again, all of these are different variants or aspects of the question; how, when, why, and where does it work?  Not, does it work or not, where it is a fixed implementation strategy or intervention.  

Let me wrap up with just a couple of more sets of thoughts.  First of all, what does this mean for us in terms of selecting research approaches and designs and methods?  We know that the RCT and other forms of experiments facilitate our attempts to measure effectiveness.  On the other hand, observational studies are probably much more useful as we try to understand barriers and facilitators; the mechanisms and mediators, the moderators, the ways in which these interventions work.  Not simply whether they work or not.  So if we’re interested in looking inside the black box, the trials are somewhat limited.  Now trials in conjunction with the strong process evaluation can develop insights into mechanisms and in many instances we do prefer a trial over an observational design but we need to accompany that trial with a strong process evaluation.  Techniques such as theory based evaluation, realistic evaluation and others from program evaluation offer guidance in studying the mediators and moderators, the mechanisms and the processes of change, and studying what happens inside the black box.  What we need as a field is better guidance in selecting and applying these kinds of approaches and understanding what theory based evaluation looks like, how to conduct one and how to perhaps accompany or compliment a trial with a theory based evaluation in order to generate the additional insights and guidance that we need.  And again, the idea that I mentioned earlier, that the implementation strategies in situations exist on a continuum; the degree of strength and robustness and strength of main effect and we need to make sure that we match our research approaches to the features of these strategies in the settings and the problems that we are studying.  

In conclusion, the productive and satisfying future that I hope we will create requires significantly more attention and action focused on the features of the evidence and the research and innovations we implement.  Much more advocacy to work with effectiveness study results rather than efficacy study results.  The second issue that I talked about was external barriers.  The action on our part to strengthen our IRBs and insure that they understand the kind of work that we do and review it properly and quickly, work on the funding process to make sure that funding programs are written properly and their reviews are conducted properly.  We need to focus on professional institutional factors and try to work to develop a more cohesive field and more consistency in our disciplines, in our terms and concepts rather, work as I indicated on our theoretical and conceptual foundations and the main issue that I focused on and that is research approaches and designs.  So these are the areas where I believe action and work is needed in order for us to strengthen our field.

So, what does this mean for those of you who are trainees and participating in the EIS 2012 Advanced Program?  This is what we have set up for you for the rest of the fall.  We actually will not have a session in early October.  What I would like you all to do though is engage in a period of independent study and on the next slide is some material that we will send around via email.  I have listed a set of talks that have been given in past programs that addressed key topics that I think are critical as part of an advanced implementation science training program and rather than give you a new version of those talks we would like you to spend time listening to the archive versions.  The four new issues that we will address though are listed here:  Dave Aron will talk about the scale-up and spread; it is one of the under looked areas of implementation research and he will talk more about it.  The area and the set of issues related to our measures and validity and quality of measurement and instruments is one that Cara Lewis and Kate Comtois will talk about November 1st; Tom Valente from USC will speak about network analysis as a way of understanding implementation processes and network interventions is a category of implementation strategies and programs; and then finally, Luci Leykum will speak about complex adaptive systems and concepts, again, similar to Tom’s talk, a set of ideas and strategies for studying implementation processes but also for developing more effective implementation strategies and programs.  There is a slide that I won’t spend any time on that directs you to the different sources of those.  Again, those of you who are participating in the program will receive additional detail by email and then Molly will see if it’s possible to post the links in the list on the cyber seminar catalogue so that others can access that list as well. 

I have left a little bit less time than I had hoped for questions but Molly; let me turn it over to you.

Molly:Excellent.  Thank you Dr. Mittman.  We do have several questions that have come in from the field so we’ll just dive right in with them.  The first came in about halfway through the session and so it might be referencing an earlier slide: What is the recently published article that reviewed the numerous theories? And in parentheses it says 60?

Dr. Brian Mittman:Yeah.  We’ll send that around.  We’ll send a link around so that everybody can have access to it.  It’s a very useful article.

Molly:Great.  The next question; may be the same answer:  Is there a citation for “selective” versus “biased sampling point” that Dr. Mittman made?

Dr. Brian Mittman:Not yet.

Molly:Okay.  Next question is:  It seems to need a layered study, approach contact/studying processes etcetera.  How would an initial study tackle this complexity at the same time as being focused and persuasive to reviewers?

Dr. Brian Mittman:That’s a question that we need to direct in part to our funding agencies and that again, we as a field, need to work out.  You know, to do these studies properly is a much more complicated and more costly endeavor than to study simple interventions.  So the question is: How do we go about designing and conducting the studies?  Can we approach it in a phased approach where we initially get funding only to answer the “did it work” question and we later go back and answer the “how did it work” question?  That’s probably not a good approach because the data need to be collected concurrently but the first task I think is to lay out in detail what these studies should look like and what the required components are; to think a little bit about the scope, scale and cost of data collection; and then to think about the implications for a grant.  And if it turns out that typical implementation study for it to be useful needs to be twice as large and twice as expensive, then I think we have an argument that needs to be made that for us to conduct the small inexpensive studies that again, to be somewhat overcritical perhaps, don’t really generate much by way value or useful insights.  It’s a useful activity for us as researchers in terms of publication and our careers but if we are not generating useful guidance, what’s the point?  So, it’s some hard decisions that need to be made but we need to sit down and first determine what it is that we think is needed and then think about how to achieve what is needed, namely perhaps larger grants and longer timelines.  

Molly:Thank you for that response.  The next question:  Is there an article/publication which speaks to an IRB process which supports implementation science goals which would assist in communicating to our local IRB around how we could best partner to support this work?

Dr. Brian Mittman:Sure, and that’s another resource that we should go ahead and post.  The articles do talk about the key issues as I see them which is the low-risk nature of this kind of work and the specific regulations within the common rule and so on that allow for it to be reviewed appropriately but quickly.

Molly:Thank you for that reply.  The next question:  When you do a process oriented evaluation of an intervention and implementation strategy is it okay to then make changes mid-course in the intervention based on findings or are changes only “allowed” in implementation strategy?

Dr. Brian Mittman:So that’s one of the tensions and one of the areas where it’s really a continuum and the tension between fidelity and adaptation and you know my own response here is that the answer is somewhere in between; that if we know for a fact that implementation strategies can be modified and can be adapted to increase their effectiveness it is not a very useful thing for us to artificially limit modifications for the sake of internal validity because we know that when these are used in practice, they will be modified.  So the answer that’s somewhere in the middle is to develop and apply an adaptation not algorithm.  So what we are trying to achieve is some consistent rule in strategy or algorithm decision rule that allows us to know when and how to modify and adapt.  What we need to be generating for our decision makers is that adaptation algorithm and guidance.  So to the extent that we could develop that algorithm in advance and then use it as a way of guiding the modifications that occur during the trial; we allow for both goals to be met.  We have internal validity and we have fidelity to the adaptation algorithm but we also have the ability to adapt and modify.  So that’s the goal.  Whether that’s feasible or not and exactly how it would work out is unclear but to me it’s very clear that setting up an artificial constraint to say this is the program with a capital P, this is the way it works, if you think you have a better idea and know how to modify it to make it more effective, you're not permitted to do that because it would destroy the internal validity of the study.  That’s just not a useful thing to do and that’s an example of the kind of research that I think is not useful for the field overall even though it sometimes leads to more elegant publications.  

Molly:Thank you for that reply.  We do have another question pending:  I am very much interested in implementation but need to illustrate strength in my area of expertise, which is genomics, consumes so much time that it makes it difficult to focus on implementation frameworks.  Seems there should be an implementation methods expert with whom we can partner.  Is there?

Dr. Brian Mittman:There are.  They’re out there in the QUERI world and I think that kind of strategy, so again with this sort of basic them that, this is multidisciplinary; these are multicomponent kinds of interventions.  The field relies very heavily on behavior and social science but also on clinical science and expertise.  This work should not be conducted by individuals who lack the expertise in all those fields and I’m not sure that there are many who have the expertise in all those fields.  So that points to multidisciplinary teams.  The QUERI is the first place to go to seek local expertise but if you get in touch with me afterwards I will help you find some local colleagues.  There is a shortage of folks in this field which is why this training program is being offered and others as well, by NIH and other groups.

Molly:Thank you for that reply.  We do also have two questions that came in just prior to the session.  Would you like to review those as well?

Dr. Brian Mittman:Yes, please.

Molly:Excellent.  Several of your recommendations will require studies that don’t seem feasible within current budge and timeline constraints.  Is QUERI willing to support implementation studies with long follow-up periods and large budgets to facilitate in-depth mixed method approaches?

Dr. Brian Mittman:That’s actually a variance of the question that was asked earlier and again I think the answer has to be it needs to be willing to support them but we need to have the kind of guidance and justification and the kinds of monitoring and support mechanisms in place so that if we do propose a five year study with lots of moving parts that the funding agencies as well as we as investigators have the confidence that it will be conducted properly.  So that points not only the need for the multidisciplinary team but perhaps some sort of advisory board or steering committee.  So I would hope the answer is yes despite the kinds of funding constraints that we’re operating under.  But again, five $200,000 studies that don’t offer much by way of useful insights are not worth nearly as much as one $1,000,000 study that actually has the needed components and actually has some hope of generating something worthwhile.  

Molly:Thank you.  We do just have two quick questions left.  Is the Academy for Healthcare Improvement endorsing or promoting these recommendations?

Dr. Brian Mittman:Good question.  Not sure.  I know they are working on a journal supplement that will include the papers from the conference but exactly what they're doing by way of advocacy is not clear; something that I will look into.  

Molly:Thank you.  And our final question:  Will PCORI develop method standards and guidance on implementation research?

Dr. Brian Mittman:If I have my way, yes.  So, PCORI, the Patient Centered Outcomes Research Institute that I serve on the methodology committee, we are in the process of revising our first methodology report which contains standards for Patient Centered Outcomes Research does not address explicitly complex social intervention, implementations, strategies and issues just because we had limited time and capacity, we haven’t yet finalized plans for the second report and the contents of it but the hope is that it will focus on these issues.  There will be opportunities for outside involvement in developing those standards including participation on the part of VA researchers.  The answer is, yes we hope so.

Molly:Thank you very much Dr. Mittman.  I have a couple of quick announcements before we wrap up this session.  First of all this session has been recorded and it will be posted in our archive catalogue but it will not be available until the end of next week, around the 28th or so.  Included in a follow-up email to you will be a link to the resources that Brian mentioned.  We will be sure to get those to you and get them posted somewhere on the web.  And, at this time, I would like to let Dr. Mittman make any concluding comments.

Molly:I also do want to thank all of our attendees for joining us today.  It is very helpful to have you join our presentations and have your interaction.  You asked some great questions and we very much appreciate it.  Dr. Mittman, one more time, would you like to address.... I understand.  We have reached the top of the hour and we will have to wrap up the session now.  I do want to again thank our attendees and also thank Dr. Mittman for kicking off the advanced mini-series.  If you are interested in registering for all of the sessions please email:  cyberseminar@VA.gov and you can just simply write in the subject line, please register for all EIS sessions.  And, as I mentioned, this session has been recorded and we will make the archive available to you by the end of next week.  Also, you will be prompted to complete a survey upon exiting today’s session.  When you leave the session a survey will appear on your web browser.  Please do take a moment to provide us some feedback.  It really helps us guide the topics and content for the presentations that you are interested in learning more about.  So again, thank you to everyone for joining us, we hope to see you for our next EIS session which will take place on October 18th at 2:00 p.m. Eastern and you can always visit the HSR&D homepage and look for the cyber seminar catalogue.  Thank you again and have a nice day.  

Page **16** of **16**