**Transcript of Cyberseminar**

**Enhancing Implementation Science**

**Advanced Program Session 2 - Scale-up and Spread of Innovations/Clinical Practices**

**David Aron, MD, MS**

**October 18, 2012**

**Advanced Program Session 2 - Scale-up and Spread of Innovations/Clinical Practices**

**October 18, 2012**

Moderator:We are the top of the hour, so I would like to introduce our presenter for today. We have Dr. David Aron presenting for us. He is the Associate Chief of Staff and Education for the Louis Stokes Cleveland Department of Veterans Affairs Medical Center and the Co-Director of the VA HRS&D Center for Implementation, Practice and Research Support. He is also a Professor of Medicine in Epidemiology and Biostatistics at the School of Medicine and Professor of Organizational Behavior at the Weatherhead School of Management, Case Western Reserve University. I would like to thank Dr. Aaron for presenting for us today. And, if you are ready, I would like to turn it over to you at this time.

Dr. Aron:Okay, and very apropos of the test question, show my screen. So, you should be able to see my slide, my opening slide, I hope. 

Moderator:Yep, we are all set.

Dr. Aron:Okay.

Moderator:Oh, I would just like to make one more announcement for David and for our attendees. To minimize that GoTo Webinar dashboard, on the right hand side, just hit the orange arrow in the upper left hand corner and that will clear it out of your viewing screen, thanks.

Dr. Aron:Okay, well very apropos of the pizza question. If anyone ever is passing through Anchorage, Alaska check out the Moose’s Tooth, which has some toppings that you probably never really thought about, like halibut pizza.  But I digress. Now, I know that you are all on mute so let us start with the ground rules. It is my job to talk and your job to listen, but if you happen to finish your job before I finish mine that is fine.  Do not worry about it. Go to your emails, do whatever you like. It is fine. I have my slides on full screen, so I cannot tell how many people are tuning off. So the topic is Scale-up and Spread. That is Alexander Fleming, by the way, in the middle of the slide. Whoop, I should be able to go down, but I cannot.

Moderator:Just click anywhere on your slide. There you go.

Dr. Aron:Okay, so this presentation will be -- is rated R. Those of you who already know me know that...

[Feedback]

Moderator:Oh, David go ahead and mute the computer speakers and that should get rid of the echo.

Dr. Aron:Okay, they are muted and gone.

Moderator:Okay, I figured it out. We are set. [Giggling]

Dr. Aron:Okay, so all kinds of sarcasm will more than likely seep in so your discretion is advised, but do feel free to challenge everything I say. I am perfectly fine with that. 

Okay, so a definition, Scaling Up means the “deliberate efforts to increase the impact of health service innovations successfully tested in pilot or experimental projects so as to benefit more people.” It is actually an interesting definition that is quite broad, but there is also an underlying assumption there about testing things in pilot or experimental. So it is the efficacy to effectiveness assumption which lately I have begun to question. 

So the “what” of what is scaled up can be almost anything. It could be a “practice”, such as a way of carrying out a work task, checklists being a great example. It could be a combination of practices. It could be a way of organizing a service, like PACT. And it could be other types of intervention, such as a new way of paying providers, or new ways of incentivizing healthy behavior among people who have particular types of insurance, or the use of mobile apps, or just about anything else you want. 

Next, is the “how” of scaling up and there is a very broad spectrum from letting it happen to making it happen and the mechanisms that are assumed differ. Now on the “let it happen”, the defining features are that it tends to be unpredictable and emergent, so you do not really know what is going to happen. On the other hand, the “make it happen,” scientific, orderly, planned, regulated, programmed, and there the assumed mechanism is managerial. In fact, it is managerial to the point where the organization is actually viewed as a machine and you could just plug something in here and there, and there are various metaphors for spread ranging from emergence through diffusion, to dissemination, to re-engineering. 

Next is the “direction” of scaling “up”. Now typical way we think about scaling up is the horizontal. So, we have a PACT team has been implemented in the main facility of the VA and then it is disseminated out to the CBOCs, although I suspect the process was actually in the reverse order. But there are two other directions that one could go. One could go vertically so that you are scaling up a procedure or a mode of organization at different levels of the organization or the depth, the scope of the practice involved, could become much more extensive so that that is increasing in the depth, scaling up by depth.

Now, there are examples of the industrial or mechanical scale-up, the make it happen. And penicillin is one of the best examples; it was discovered by accident in 1928 by Alexander Fleming. He actually did not think of it as something that could be used to fight infection. He noticed that there was inhibition of growth of bacteria in a particular part of a contaminated Petri dish where there was a bit of fungus growing and he thought, “Oh, maybe that fungus is making something that would enable me to differentiate among different types of bacteria.” And it took other people to come up with the idea, Florey and Chain in particular, to use it as a therapeutic agent. And the culture of that fungus started in little bottles; actually it started in urinals, bedpans, and then was scaled up over a pretty short period of time to the point where it was available in very limited quantities during World War II. So it went up from being cultured in little dishes, all the way to these big fermentation tanks. That is mechanical.

Diffusion is more of a let it happen type, okay. And this is the classic Everett Rogers *Diffusion of Innovation*, which is a great book. He was a great person too, I might add. And he studied a number of different things and how particular innovations were adopted looking at corn seeds and so on, and noticed that there was a particular S-curve in the adoption. You will also notice on the right hand side of the screen, the number of years that it took to actually get people to adopt this new particular corn seed. And on the left side, shows the importance of social networks in this particular type of diffusion of innovation. So there was a scientist right over there who came up with this seed and a particular farmer who thought, “Oh that looks interesting. I am going to try it.” Okay, an innovator, and then this guy, this was the second person to adopt it and he kind of liked this guy. And this was a guy who was respected by a lot of people, okay, and then it started to take off. And this has been applied to technologies of all sorts from the telephone, the car, electricity and you can see that the time scales are getting more and more compressed; although it is interesting that the infrastructure to support each of these innovations has gotten smaller and smaller over time. 

Now, one of the things that is particularly notable is this is for relatively simple technology, relatively simple, where we have independent agents. Most of the things in healthcare are a bit different and Everett Rogers came up with this. So, here is the adoption, right from very low to a hundred percent of the market share and there are some people who come in early and early majority, and so on. And then these are these laggards. One of the fundamental problems with this terminology is that it makes the assumption that whatever is being adopted is good. Other people have used different terms, but I would just ask you did you buy the first iPod that came out, the first version? Or did you wait until the second version, or the third version, or the fourth? And any time I ask an audience this question, there is usually one person who got the first one and I ask why and they say because I have to have the newest thing. Okay, very good. I ask other – most people are buying the second, third or fourth versions. How come? Well, it did not meet that individual’s needs until the batteries were fixed; the capacity for music was increased and so on, and so on. So, I think this can be a quite simplistic way of thinking and ditto for the way Paul Plsek looks at it.

Now, the major factors in scaling up are the environment for change, the timing, the aim, the structural context, so this is all the context in which scaling up takes place, the nature of the intervention which is the “what” and the method of the intervention which is the “how”. There are many, many, many frameworks for scale up. 

Okay, the previous one came from Medicare, I believe. This one comes from HEALTHQUAL International. So, better ideas set up on a small scale and then spread throughout a social system with the influence of leadership and so on. This is for U.S.A.I.D. 

This one I find rather interesting. It is a method of evaluating scaling up as a Complex Adaptive System. And in a complex system there – a complex system is a collection of elements that interact in non-linear ways to produce emergent behavior. Now, that is a nice, relatively brief definition. The problem is that each of those words are problematic, with the exception maybe of of. So, what is a collection, what is an element, what is non-linear, what is produce mean, which is about causality, and what is emergent behavior. In any case, this particular model is based on complexity and some of the non-linear type behavior are things like tipping points which we can see in those S-curves. There are initiating conditions, which is the context, the inter-dependence of the various actors and the various parts of the system. There are outcomes, but there are always unintended outcomes. And that is something that tends to be ignored in most scaling up studies that I have seen. 

So, a couple of slides from Becky Yano, I would like to thank for sharing them with me. The other stuff I just stole, but annotated. She starts with building the evidence base and then adding the multi-level context. So consider a primary care teamlet, which is nested in a primary care team, which is nested within a CBOC, which is nested within a facility, which is nested within a VISN, which is nested within the VA, which is nested within the whole federal governmental, and so on, and so on. However, the context for each of these places can be quite different and that is a fundamental problem in scaling up, that if the context is the same everywhere, then it is possible to take a more mechanical type approach. But if contexts differ, than at any social system context differ, you have to use a much more nuanced approach. The question is how nuanced does it have to be? 

Just an example of contextual difference, between something that on the face of it should not be too different, so the Cleveland VA we have a number of CBOCs and there are two CBOCs in particular that are exactly the same size, have the same number of patients, have the same number of staff, have the same ratio within staff of nurses to docs and so on, clerks and so on. The buildings were built on the same architectural plan. So, if they are empty and you go into either one, the building looks the same. But, if you walk in when people are there or actually you walk in with your eyes open and you see what is on the walls, you go to one place you get one kind of feeling, you go to a different place, it is a completely different feeling and that ends up being reflected in the relative ease or difficulty with which change occurs in both of those two. And when you think about trying to scale up nationally, one is faced with the old proverb that if you have seen one VA, you have seen one VA 

Now, all of these issues and more were observed and dealt with in the WAVES/TIDES/COVES/RIPPLE/RETIDES series of projects to scale up collaborative care for depression. So, TIDES was the first one, Translating Initiatives in Depression into Effective Solutions and this TIDES was the beginning of a series of projects led by a very talented group of individuals at the Sepulveda VA, now part of greater Los Angeles VA During the nineties there were a variety of depression care improvement models tested around the world, actually. Thirty-six high quality RCP’s, which kind of raises another question, but I will let you come to that on your own. In 2000 TIDES began “Can VA implement collaborative care as part of routine care?” and the decision makers were the VISN leadership and by 2006 it was part of a national roll out, but the process is still going on. 

So it has been more than a decade and collaborative care, although there is plenty of it comes in a wide variety of flavors and models, and sometimes flavorless and with no model at all. There are a lot of lessons, the trials; the clinical trials may not reflect real world implementation. In fact, in my experience they rarely reflect real world implementation. Interventions only sustain if integrated into an organization’s real world activities. You cannot use – it is difficult to use trials to study quality improvement without distortion and I will give you some examples. And trials do not capture a lot of the determinants of real world program functioning. So, this gets to that issue of does one really have to demonstrate efficacy before looking at effectiveness? Is it even possible in a complex social intervention to test efficacy? I am not sure I still believe that, although I certainly did. 

Now, this is a very, very recent study that just came out in Implementation Science and it was done by Rycroft-Malone and colleagues and it was a pragmatic cluster randomized trial evaluation of three implementation interventions designed to all implement the same thing. A guideline to decrease the amount of time someone was fasting or had no liquids, N.P.O., prior to surgery. And they randomly chose a bunch of hospital trusts and nineteen hospital trusts were randomized to three different techniques. A standard dissemination package, which basically meant, here is the guideline go do it. A standard dissemination plus a web-based tool championed by an opinion leader and standard dissemination plus a quality improvement, facilitated quality improvement. And you can see the end there, and I hope you are getting a sense that a hospital trust, which includes a hospital and a lot of doctors and some primary care practices. But all of this was focused mostly on the hospitals, lots of different wards, lots of different surgeons, lots of different nurses within an individual trust and the end is nineteen. And one site was unable to deliver the intervention due to the sickness of a facilitator. 

Here is that web-based tool, so that you can look up the website if you are particularly interested in what they had. And here is their conceptual model. So they had some evidence they were going to use facilitation for some, they were looking at context, they had their three interventions, they had a very detailed, summative and informative evaluation with numerous interviews, and learning organization surveys, and economic evaluations, and patients, and lots of quantitative things around, fasting times, and looking at interrupted time series as their basic method. I mean very, very extensive. I mean pretty nicely designed study if you accept that you can randomize nineteen trusts.

Well, what did they find? What they found was similar to many of the studies that have been done on implementation of anything, which is nothing works. There were no differences in the mean fluid fasting time, pre and post intervention for any of the – for the standard plus web or standard plus PDSA approach and there was a lot of variation among them at baseline, but no significant differences. And the same applied to mean food fasting time. So, nothing works.

So this was their conclusion, and I want to read it in its entirety. “This was a large, complex study in one of the first national randomized control trials conducted with acute care in implementation research. The evidence base for fasting practice was accepted by those participating in this study and the messages from it simple; however, implementation and practical challenges influenced the interventions’ impact.” And this is my underlining, “A set of conditions for implementation emerges from the findings of this study, which are presented as theoretically transferable propositions that have international relevance.” And now, let us just go. 

So, here is my interpretation, and “although it would have been interesting to have a true control group”, which did not even get the guideline. Nothing works better than anything else independent of context and that is the fundamental issue here. The randomized trial, the randomization is designed to wash out, to eliminate the effective context. It does not, not when you are dealing with a complex social intervention and that is the fundamental problem. 

So here are their set of conditions, “implementation is more likely to be successful in cases where the topic or issue is a strategic or organizational priority.” That is going out on a limb there. I believe Proverbs chapter, I think it is, 17:8, maybe, I am not sure. Without vision the people perish. “Historical lack of clear leadership, structure, and process for local guideline dissemination and implementation, in which staff are unclear about their responsibilities, will negatively impact an organisation’s ability to routinely use guideline recommendations. Robust and believable evidence is not always sufficient to change decision making and practice.” And you can probably read faster than I can talk and since this is an open access journal, you can pull this and get it for yourself. Where there is effect teamwork, “practice change will be easier”. 

I mean you have to ask yourself, did that randomized control trial really contribute anything. Is there any new information here? And unfortunately, I would have to argue that the answer to that is no. And that is a serious problem when you get to the issue of how do you do research on scale up and spread, especially when the time frames and I am talking about more socially complex interventions, a way of changing practice. There are still plenty of places in the country even with making PACT teams mandatory where there may be PACT teams in names, but not in reality. And ditto for mental health, primary care collaboration, however, good each of those are. 

So, let us kind of re-examine what are the underlying assumptions that we as researchers bring with us when we do a study. The first one is that the intervention is independent from other external changes. Now think about the time frame. Scale up and spread has to, by definition, be a dynamic process. It has to occur over time and the likelihood that there will be no external changes in this world is pretty much zero. I mean I can easily plot the increasing inches of paperwork that are required for the IRB and R&D committee that have increased over the past couple of years. The second, that isolating of confounding factors enables the researcher to infer a direct link between intervention and outcome and that is causality and it is a very mechanical, linear causality. A causes B, okay. And internal validity ensures the ability to generalize and differences in contextual factors are eliminated given large samples. 

And N equals nineteen, even if there were fifty wards in each hospital and even if you try to adjust for side effects can be real tough when the intervention these guidelines actually involves cooperation among a variety of services. For example, in the case where I have dealt with it myself, is in the case of diabetes. We have people who have gotten insulin. They are NPO for surgery – NPO after breakfast, or NPO after a period of a specific time of day, but because delivery of food comes too late they are NPO having gotten their insulin and hypoglycemic when they arrive in surgery because there is not a good coordination between the food service and the surgical schedule, and nursing, and so on, and so on. And this is reflective of the kinds of complex social interventions that we do. 

Now on the left hand side, those assumptions are all based on the standard research paradigm that is a pure positivist one, okay. It is not the only paradigm and one has to ask how effective that paradigm is for the nature of the processes that we are looking at. I am not calling all science into question, okay, do not – please do not misunderstand me in that regard. But, I do not think that the standard approach is really been helping us. Realist evaluation which – or realist approach starts with some different assumptions. External changes are part of the intervention and they have to be reported. Real world intervention, in real world interventions, isolation of confounding factors just is not possible. So, the context and mechanisms are part of those things that trigger the causal relationship. So, you cannot have a simple mechanism and the generalization comes not from saying getting rid of all the confounding factors, which is not possible, but rather providing enough information so that the relevant context and the various mechanisms are at least illustrated. 

So let me just give you a very, very quick differentiation between three research paradigms, positivism, critical realism, and the deconstructionist’s favorite, social constructionism. So, think about baseball. It is the playoffs after all and think about the umpire who is calling balls and strikes. The positivist umpire says, “I call them as they are.” The critical realist umpire says, “I call them as I see them.” And the social constructionist umpire says, “They ain’t nothing until I call them.” So, we have to be clear on what our paradigms are. So, here are some take home points, which will leave plenty of time for questions, and comments, and brickbats, and so on.

The first is there is no easy solution. If there were, word would have gotten around by now. The second is in the kinds of things we as health services researchers think about we are dealing with “wicked problems” and “wicked problems” are characterized – and I am using this not in the form, not in the good evil sense, but rather in the Rittel-Weber sense, who described “wicked problems” as problems that involve multiple actors, ethical differences, where there is no completely optimal solution. There are only different solutions. 

Another issue to think about is every instance of scaling up unique? And are our “researcher” mental models not only wrong, as all models are to one degree or another, right? The G.E.P. box claim that all statistician models are wrong, some are useful. But are mental models not only wrong, but dangerously so? That is we get money to do studies that are not going to result in A) improvement or B) knowledge that is actually useful. There are roughly twenty thousand randomized control trials that are picked up by the Cochran collaboration per year now. Twenty thousand randomized control trials, yet if you look at systematic reviews the amount of so-called high quality evidence is very, very small. 

Now, I have added that you will be able to look at on – at your leisure a bunch of things that I just call supplementary material. Things like “What we really do not know?” If you want to investigate scaling up, whatever paradigm you choose. Costs, we have very little on costs. Now the Rycroft-Malone paper did do a nice job on costs. The requirements for sustaining scale up. Comparative effectiveness of approaches, which is going to be very, very difficult and the one thing that almost never gets talked about is what about stopping doing things that do not make any sense, what about scaling down and, of course, unintended consequences. 

These were some recommendations for research that came from papers at a H.R.Q. conference that are in, references, and there are a few more models which I think I have annotated them all. So, you choose any. Everyone comes to roughly the same conclusions. So, this is from the British Civil Service. Make the business case, there has to be credibility, leadership. The kinds of things you could learn by reading the Bible. 

Here are some references and a very, very interesting reference that directly questions the role of randomized control trials in what we do. And with that, I will open it to questions, comments, and whatever you all got.

Moderator:Great!  Thank you very much Dr. Aron. For those of you that joined us after the top of the hour, to submit your questions and comments in writing just go to the GoTo Webinar Dashboard on the right hand side of your screen and write it under the question section and press send and we will get to it in the order that it was received. We have gotten a couple of comments and some questions so far. The first comment was, “I am very impressed by the quoted Proverb without vision the people perish. With the implementation of PACT we have a great vision. Makes me think we need to make sure all the team players understand the vision as the first stepping stone. I am thinking we need to go back to the basics and make sure we all understand the primary care mental health PACT vision.” So, thank you for that.

Dr. Aron:And I would like to say that I share that and I will just tell a little story in that regard. So, I was the first director of the firm system we have here in Cleveland which was started in 1991. It was, I believe, the first integrated inpatient outpatient firm system in the VA and we used as our model the firm system that has been at our local county hospital, Metro Health, since the late seventies. In 1993 we developed a plan to integrate and co-locate in our different facilities, we had very few CBOCs back in 1993, to co-locate primary care, geriatrics, and mental health and I have kept a copy of that. But because of – let us just say politics – that was nixed at a high level in the facility and we have not had that until the development of the CBOCs. And then – so this is another great example I think of scaling up. So Cleveland went from one CBOC  to thirteen and they way they did that was not only build buildings, rent space and so on, but what did they scale up? They scaled up mental health. How come? Mental health, you do not need plumbing. You do not need sinks in the offices. So, mental health actually started first. It was followed by primary care and they co-located and it is just an example of the impact of external context on or types of constraints that are going to have an impact on the scale up. Thank you for that.

Moderator:Thank you for that reply. Hi, Dr. Aron, can I ask you to put your slides back up. Maybe the reference slide just so that we have something to look at while we do the Q and A.

Dr. Aron:Okay.

Moderator:Thank you! We do have several more questions that have come in. The audience is very engaged.

Dr. Aron:I will put that one up, I like that.

Moderator:There we go, perfect. All right, so as I was saying we have several more questions that have come in so we will go ahead and get right to them. The first one, “When should scaling up first be thought about?”

Dr. Aron:I think it depends -- like everything else, it is a very easy answer, it depends. I guess the first question is when you design something, right at the beginning, are you thinking about something that potentially might apply in other places. And if the answer is yes, you should build that in from the beginning. 

Moderator:Thank you for that –

Dr. Aron:And the degree is going to depend upon how complex that particular innovation is. And, by the way, as long as I am all pining, we talk a lot about best practices in the VA or best practices anywhere. Best practices are a load of rubbish. There is no such thing as a best practice independent of the context in which it is conducted. There are certainly places that do things better than others, no dispute there. There are not best practices. There are potentially better practices. When there is something that is truly a best practice, it is like we should treat meningococcal meningitis with antibiotics. I mean, if it is so best, it is independent of the context. But the kinds of things that we in health services research deal with are virtually never independent of the context. 

Moderator:Thank you for that reply. Next question, “What funding opportunities are available in the VHA for scaling up an innovation? I have a one year VHA innovation award for a pilot project to demonstrate feasibility of a novel clinical pathway. Is there a funding mechanism that would allow me to scale up the project beyond the one year time frame?”

Dr. Aron:So, if you think about VA funds. There are two kinds of dollars. There are research dollars and there are operations dollars. For research dollars, whether you go through a service directed project through one of the QUERI centers or something similar, the people who are going to be reviewing it are primarily going to be researchers who are quite frankly totally embedded in this positivist, get rid of context, the only knowledge that counts is what is internally valid and generalizable. So, if that is the kind of thing you want to do, there are certainly opportunities to do it there. The alternative is operations and that means having an innovation that appeals to someone in one of the central offices, whether it is within patient care services, either primary care or specialty care; both of which have been funding a lot of different things. The office of patient centered care and cultural transformation, if you can; there is an R.F.A. that has just come out for projects there. Adam Darkin’s tele-health shop, there are a variety – women’s health is another office that funds things. I think when you think about scaling up and thinking about the context, you have to think about partnered research. That means you have to have an operations partner. So, when QUERI began, at one of the early meetings of QUERI one of the implementation coordinators was giving a talk on lessons learned and I will never forget this, the lesson learned was if you want to do something at the front line you have to talk to the front line staff, okay? I mean think about the implications of that statement, okay. I mean putting aside the fact that it is obvious, what it meant was that the underlying paradigm was researchers as solely as observers. So, think about what is your goal? Is your goal improvement or is your goal generation of what someone else thinks of as generalizable, as new knowledge, new and generalizable knowledge. So, there is money out there, okay? There is a lot of money out there actually, but it is often not in research and you have to find a willing partner, which means you have got to show how – and that willing partner could be a VISN director, but you have got to make it worth their while and appeal to what appeals to them, which means – and they have three things that they are being judged on population health, the patient experience, and unit cost. So, if your project will reduce unit costs, improve population health, and improve the patient experience, you are going to be able to find a willing partner depending upon how much front end investment.

Moderator:Thank you for that.

Dr. Aron:By the way, by the way I should give – I should mention that I actually do not come from a typical health services research background. I mean apart from still practicing as an endocrinologist and having done molecular biology for fifteen years before moving into health services research, I have also been involved in operations and was an associate chief of medical service and a chief of medicine, and the issues that one deals with in operations are not the same as the issues that one deals with in research.

Moderator:Thank you for that reply.

Dr. Aron:Yeah, so I come to my cynicism honestly, but I am an optimistic cynic.

Moderator: [Giggling] Experience has taught you. Can I get you to refresh your screen real quick? 

Dr. Aron:Possibly.

Moderator:Thank you sir.

Dr. Aron:You do not want to know about patient safety.

Moderator:Well, we all have our own that we get to see on the daily. All right, so next question, “If adoption/implementation theoretical models are generally non-generalizable across context where should the research focus shift to in understanding scale up?”

Dr. Aron:So, I would not say that the models are not generalizable. I would rather frame it as all of the models are sufficiently vague as to be able to apply to a lot of things generally. The problem is when you get to the specifics. So, if I were going to do a scale up and spread study. One that would really appeal to me as a reviewer would be to test scale up versus scale up plus descaling of something. That would really appeal to me as a reviewer. Some of the kind of research questions that AHRQ and the VA for that matter were interested in were these. There we go, okay. So, these are some of the areas, the typology of innovations to facilitate comparisons. I think a critical thing that needs to be done in any implementation project is there has to be a far more detailed description of the contexts in which the implementation in which the implementation is taking place and whether one uses the framework of the Squire guidelines or something else, I think that is absolutely critical and I realize that journals are not interested in that. But at least with some journals there is plenty of opportunity for online supplementary material.

Moderator:Thank you for that reply. Next question, “How can the scaling up process be monitored and evaluated?”

Dr. Aron:Creatively! Again, this is one of those questions. Scale up of what? We can easily or relatively easily look at the scale up of the use of a particular medication by monitoring the number of prescriptions for that particular medication or the rate at which the prescriptions for other medications fall, or something like that. Particularly of interest, when a medication is found to have some side effects that were not anticipated. When it gets to complex social interventions, the outcomes tend to be at much, much higher levels. For example, in PACT some of the PACT outcomes are who is on the team, how many patients do they have, how much communication is there among them? The monitoring and evaluation is going to be very, very, very dependent upon the particular issue. So, I am one of the directors of one of the evaluation centers for the office of specialty care transformation and their projects and the way we have been addressing the context issue and we have to evaluate a bunch of different initiatives is we start with a questionnaire that is based on the CFIR guidelines, Consolidated Framework for Implementation Research, papers by Nordam Schroeder et al. I am al, there are other al’s. And we talk to some of the key – we gave this to key people at the sites. They identified which of the thirty-nine constructs they thought were important and then when we sent surveys to all the other folks, those were the ones that we used. And I can tell you it varied. The constructs for econsults were different from the constructs for skin echo, which I have no doubt are going to be different from the consults for mini residency programs, and so on. 

Moderator:Thank you for that reply. The next question, “What do you suggest researchers do if trying to study implementation? What approach should we use if random control trials are not appropriate for implementation science?”

Dr. Aron:I would not – okay, so this is – this gets to the paradigm issue. So, it is not that I am opposed to randomized control trials. I just think that they are – they have limitations that we have to recognize. If one actually can answer a question with a randomized control trial, go for it. Absolutely, I have no problem with that. But, let us not be fooled that it is going to work in some situations where it clearly cannot give us the answers that we want. There are a variety of other kinds of methods, whether it be quasi experimental trials, time series, regression discontinuity evaluations. There are a wide variety of different methods, observational trials. But these, all of these, are based or generally come out of a paradigm of researcher as independent observer and there are a lot of advantages to that researcher as independent observer in that at least ideally it decreases the amount of subjectivity since researchers have a stake in getting the correct answer rather than proving that a particular innovation works. But I would argue that when you deal with complex systems that exhibit non-linear behavior that you might take a page out of Kurt Lewin who said that if you want to understand an organization, try to change it and be involved in the process. So, I am much more interested in, personally, in participatory action research. I think the days of being able to get individual providers to spend a lot of time answering questionnaires are way gone given the kinds of pressures that existed in primary and elsewhere. And we as researchers need to be partners with operations. Not only at the central office level and the VISN level and the facility level, but at the front line level.

Moderator:Thank you for that.

Dr. Aron:And again, just let me point out, if there were an easy solution to this, word would have gotten around by now.

Moderator:All right, we have five pending questions. If you have some time to stay on, that would be great. 

Dr. Aron:Sure, I have at least five pending answers that may not match up with the questions, but what the heck.

Moderator: [Laughter] Hey, we will give it a shot.

Dr. Aron:And, by the way, I think my email was on the first slide, but it is David dot aron at VA dot gov, and feel free, challenge what I say. I am fine with that. I am definitely cynical, but I am also an optimistic cynic. I have passed retirement age, but I am still here fighting. 

Moderator:That is encouraging to hear, thank you. Next question, “If we believe that the crux of implementation is understanding context, should one goal of implementation science be establishing a pragmatic wave for organizations to evaluate their context to decide which mode of implementation will be effective?”

Dr. Aron:Yes. How about that?

Moderator:Great.

Dr. Aron:Why use two words when one will do.

Moderator:Okay, “What about the new model of quote point of care research? Leonard Davolio and colleagues just published a paper describing the implementation of the first P.O.C.-R trial. There are a number of issues with doing this kind of research, but it would appear it could help answer more practical questions. What are your thoughts on this?”

Dr. Aron:That is not one of the five answers I have. 

Moderator:Do you want to take a stab at it?

Dr. Aron:I do not know enough about it to answer intelligently.

Moderator:Not a problem. They can always contact them.

Dr. Aron:But give me that, give me the name of that author. I am going to look it up.

Moderator:It is Leonard Davolio, D-A-V-O-L-I-O. I believe he is head of head of Maverick in the Boston healthcare system.

Dr. Aron:Okay.

Moderator:All right, next one. “Superb and provocative presentation. Comment on the observation that VACO increasingly operates on a managerial model of scaling up often in the faith rather than with the actual evidence.”

Dr. Aron:Oh, I do not think there is any question about that, but that does not mean that there are not individuals within operations or VACO  at large that are willing to experiment. You know what, I talked to my colleagues in the private sector, they are amazed at the kinds of things we can do in the VA, just amazed. The micro management, you think about it, the purpose of a bureaucracy, of a rules based method is to prevent change. That is the purpose of it and then we get angry because the bureaucracy will not change. Well it is designed not to change, but there are always individuals who are chiseling away. Be a chiseler.

Moderator:Great, thank you. 

Dr. Aron:It is a good thing I am past retirement age [Laughter].

Moderator:You can say whatever you please.

Dr. Aron:You got it and I do as it turns out [Laughter].

Moderator:You may or may not know the answer to this one. “How do we get the info for the Office of Patient Centered Care RFP’s and other VA Operations RFP’s?”

Dr. Aron:Sure, so the – that is actually a very interesting question. The way it works is that these RFA’s go to the VISN and the VISN send them to the facilities, and the facilities send them to individuals that they think might go after them. That approach depends upon the interests of the VISN director in sending things to a facility, the interests of the facility directors or whoever are dealing with those particular suspenses in doing anything about them. So, I can tell you even as someone who is pretty well connected in VACO, there are R.F.P.’s that I have never heard about. So, it is good to have contacts in the offices. So, one person who is in both the Office of Primary Care – she may be listening in and will probably shoot me when she finds out I have given her name and email – she is the clinical lead for Office of Special Care Transformation and she is also in primary care and that is Susan Kirsh, K-I-R-S-H, and she is in Outlook. Just Susan dot Kirsh at VA dot gov. Vanessa Ellington is the head of the Office of Specialty Care Transformation. In the Office of Primary Care, it is Gordon Sheckman is the head of that. It is a very, very good way is to contact your VISN directors. 

Moderator:Okay, thank you.

Dr. Aron:And learn who is – there are a lot – this is social network writ large. And this where you have got the – this is the case of the strength of weak ties. If you cannot answer that question it is because your social network does not have the answer readily available, you need a bridge to another one.

Moderator:Thanks!  Last question, “Can you repeat the full name of the Clinical Operations Partner for potential funding the speaker mentioned patient centered dot, dot, dot, dot, dot.”

Dr. Aron:Yes, it is the Office of Patient Centered Care and Cultural Transformation and the head of that office is Dr. Tracy Gaudet, G-A-U-D-E-T. And that RFA went out on the fifteenth of October.

Moderator:Thank you.

Dr. Aron:And if anyone has questions, feel free to email me.

Moderator:Great, well we really appreciate you lending your optimistic cynicism expertise to us today and do you have any final comments you would like to make to the field?

Dr. Aron:Yes, remember -- I have two comments. Number one, life is much too long not to have fun because if you are not having fun, you are not having it for a long time. That is number one. Number two, take it easy, but take it. Thanks very much everybody.

Moderator:Thank you kindly and thanks to all of our attendees for joining us today and this does conclude our presentation. Please do fill out the feedback survey that will pop up on your screen as you exit today’s cyber seminar. And be sure to join us for the next session in this series, which will be taking place on November 1 also at 2 P.M. Eastern and that will be on measures, measurement and instruments. So thank you everyone for joining us.

[End of audio]

Page 13 of 13