Enhancing Implementation Science- 17 -Department of Veterans Affairs

**Department of Veterans Affairs**

**Enhancing Implementation Science**

**Intro Program Session 2: Pre-Implementation and Hybrid Effectiveness-Implementation Study Designs**

**Alison Hamilton**

**May 17, 2012**

Moderator:  This series is on implementation. Implementation science features presenters from VA HSR&D, Quality Enhancement Research Initiative, also known as QUERI. Implementation science focuses on research that examines methods to accelerate the implementation of evidence-based clinical practices in routine healthcare settings. Faculty will share information about QUERI implementation processes, methods and what they’ve learned. Those who may benefit from attending this session include clinicians, researchers, administrators, policy makers who are interested in learning more about implementation science and the tools needed to promote evidence-based medicine. 

Today’s presenter is Dr. Alison Hamilton. She’s an investigator at CIPRS and the VA Women’s Health Research Network at Greater Los Angeles VA. Dr. Hamilton is an Associate Research Anthropologist at UCLA and her work focuses on implementation of best practices in mental health. And I’d like to turn it over to you now, Dr. Hamilton.

Dr. Alison Hamilton:  Thank you Molly. Can you hear me okay?

Moderator:  You are coming through loud and clear.

Dr. Alison Hamilton:  Okay, great. Thank you. Thank you everyone for attending today’s session. Good morning, good afternoon, wherever you may be in the US. We have a lot to cover today and I’m going to do my best to make it through all the material. Before I get started, I just want to acknowledge several individuals that have contributed substantially to this presentation model and the authors of the hybrid paper which many of you read for today’s session. Other colleagues played a critical role in development of this work.

So, today I’m going to cover a few different topics. First I’m going to briefly review characteristics of efficacy trials, effectiveness trials and implementation research in about half the time and roughly will be spent of pre-implementation or what we might think of as QUERI Step Three design and methods. These will be a couple of examples of Step Three studies. The other portion of the presentation will concentrate on hybrid design. I’ll explain what the three hybrid designs are and provide examples of each design and raise some considerations that go along with the hybrid design. We’ll take questions at the end.

So, just briefly, clinical efficacy trials address whether a treatment increase outcomes in a given condition. These are randomized trials and the outcomes are typically clinical. Process measures are not included in these types of trials. The level of analysis is up to the patient and the clinical unit. The emphasis in these types of trials is internal validity, answering the questions if changes are attributable to the intervention and nothing else. They take place in control conditions and we will not be talking about the trial today.

What we will be touching on are clinical effectiveness trials, which typically follow efficacy research trials. The outcomes are generally clinical, but there might also be some process measures included, but they would be secondary. The level of analysis here might be the patient or the clinical unit, but the point here is to look at external validity. How does the intervention work in real clinics in a larger and more diverse sample? 

Moderator:  I apologize for interrupting. I just want to make a quick comment. If you could speak up a little bit, that would be great. Also, to any of our viewers that the dashboard is in the way, just hit the upper left hand arrow in the left hand corner, the orange arrow, and you will be able to see the slides in full. Thank you. 

Dr. Alison Hamilton:  Okay, thanks Molly. Is that better?

Moderator:  Yes.

Dr. Alison Hamilton:  Okay. Finally, implementation research, which you are all already familiar or becoming familiar is about enhancing uptake of, established clinical interventions. So the outcomes here are different from what you might see in effectiveness trials. Implementation research is looking at process measures such as rates of adoption, utilization of service and context. The clinical outcome data may not be of primary interest sense intervention is already established in the context of implementation research. So, levels of analysis in implementation research would therefore differ and might be more focused on providers, clinical units, facility, organization, et cetera.

Next I’m going to talk about pre-implementation design, which is QUERI Step Three. So, I think from prior presentations, you are all familiar with the Clinical Research-Implementation Pipeline. The area that we’re going to be addressing for the time being is pre-implementation, which comes right before implementation research in the pipeline. Step three is about measuring and diagnosing quality and performance gaps. I’ll get to each individual piece within step three in the next slide. 

One of the most important things to remember when it comes to step three works is that it is about systems thinking or understanding social systems. The emphasis is not on isolating factors, but thinking about context and broader issues that might affect implementation. Here, the idea of diagnosis is one that would result in identifying actionable factors. These are things that you want to be able to change. These are factors that would contribute to quality gaps and might be reasons innovation has not been successfully up taken in a given area. This information can be found on the QUERI website if you want to have more references. 

There are four phases or steps involved. These are not sequential. They are four different areas of step three that you might want to pursue in a pre-implementation study. A is to measure existing practice patterns and outcomes across VA and identify variations from evidence-based practices. You might think of those as quality or performance gaps. Three-B is to identify determinants of current practice. Is it usual care or treatment as usual? Three-C is to diagnose quality and performance gaps. Three-D is to identify barriers and facilitators to improvement.

There are a lot of different designs that you would want to consider for pre-implementation studies. There are observational studies, cross-sectional studies, and sometimes-longitudinal studies. There are a whole variety of methods you could consider and often-mixed methods are a good option for pre-implementation studies. Some methods, and this is not an exhaustive list, but these methods might include measurement of practice variation via chart review or looking at the administrative databases, modeling via simulation determinants of clinical practice. Qualitative methods are often a major component of pre-implementation studies. 

That might include semi-structured interviews, focus groups, and observations. There are some types of surveys at a moderate or minimal level. You can’t do a huge survey in the context of pre-implementation, but certainly a survey could be warranted. Policy and archival review and expert panels and Delphi consensus are all possibilities for methods that might help you in a pre-implementation study.

I’m now going to talk briefly through two examples on some work – both of these papers that I’m going to be talking about are in the special issue of implementation science from 2008 and the authors are on the slide. If you want to pursue these further, they actually describe their movements from step one of the QUERI framework through to step four and beyond. So, the purpose of the step three studies, which was done through the Diabetes-QUERI, was to understand the circumstances of preventable visible loss from diabetes. They have focused on the timing of laser eye surgery as a key issue in preventing visual loss. 

They did a few things in their step three study. They reviewed and examined medical records and they had pre-set criteria by which to review these records. They were looking particularly at optimal timing of photocoagulation. They identified patients whose visual loss was preventable and found that two-thirds of cases were associated with problems related to surveillance of those with identified disease. These were issues like inadequate follow-up, delays in treatment or rapid disease progression that was not predictable.

Their results in the step three studies identified a lack of close follow-up with known disease as a potentially important gap in quality of care, meaning that that is an actionable characteristic of this context. In sum, they had already identified a high-priority issue as step one they had the evidence to support a change in performance measure as step two. Via their step three work, they identified this gap in quality of care and in combination with other work led them to implementation work, which has gone on to step four, five and six. They have published subsequently a lot of that work.

The next example is from the HIV/Hepatitis QUERI. Their purpose of step three studies was to identify possible gaps in HIV testing in VA and to understand the source of gaps in care and discern facilitators that would improve current practice. They also undertook several different methods in their step three setting. They reviewed VA policies regarding HIV testing. They conducted a survey with providers to understand their practices and attitudes regarding HIV testing in two VA facilities. They also conducted a systematic review of sixty-two studies of HIV testing practices, attitudes and barriers. It’s important to note here that not all of your evidence that you’re collecting in step three has to come from primary data. In this case, they did look at prior published research to help them understand what some of those attitudes and barriers might be as identified in other studies.

Through these methods they found a few different things. Via policy, they found organizational barriers to testing. Through their survey, they found provider barriers to ordering HIV tests and through their systematic review, they found that higher acceptance rates to HIV testing were found with confidentiality protection and the provider’s belief that testing would be beneficial. They found actionable items within this context that they could use to propel their implementation research. 

So, again to reiterate this process, they had already identified a high-priority condition in step one: HIV. They had evidence-based guidelines for HIV testing in step two. And then the other work is step three: identifying the gap in quality care, which led to implementation in a number of areas, described in this paper first as step four. This group has gone on to produce an extensive body of implementation research.

Now I’m going to move on to hybrid designs, which takes a little bit more time to make our way through. What I’m going to do is give you a few different ways to understand hybrid designs. If you’ve read the paper for today, written by Geoff Curran and colleagues, you should be familiar with this so, hopefully this will just reiterate some of the things you already have read from the paper. We’re going to talk through an example of each type of hybrid design and raise some questions to think about.

So, although the medical care paper just came out recently, the concept of hybrid designs was introduced in that 2008 paper by Sheryl Settler in Implementation Science. It’s been developing and evolving since then to the point where it is a well spelled out framework for understanding hybrid design. From the 2008 paper, we learned that hybrid study designs, which I will define more thoroughly in a minute – hybrid studies look at both effectiveness and implementation. 

They are about a couple of different things. They should include the most realistically rapid timeline given the complexity. The idea is that putting effectiveness and implementation work together should give us results more quickly than moving slowly through one type of design and then another. Focus during the study should be on progress and identifying potential and actual influences on implementation and effectiveness. A key aspect of hybrid design is evaluation. I’ll get back to this point in a minute.

The formative evaluation or the evaluation as a process evaluation is designed to optimize several things, which I’m going to show you in the next slide. You are trying to optimize use of the implementation strategy even in preliminary states to at least assess its potential if not achieve its potential. You want to be aiming towards the goal of clinically meaningful evidence-based practice. This is not just practice that is statistically significant, but clinically meaningful in real clinical contexts. You are paying at least some attention to what we might think of as the black box of implementation, which might include a look at cost-benefit. You’re identifying some remaining research questions that may need to be answered in subsequent studies and you are ideally developing a replicable implementation program or at least identifying elements of something that might eventually become an implementation program.

Spatially and visually, hybrids go in the sort of transition that is between effectiveness and implementation research. As you’ll see, they fall a little bit more on the effectiveness side of things while another will fall on the implementation side of things and the other falls in the middle, balancing the two. 

I’m going to just start with a couple definitions to lay a foundation for the definitions for hybrid design. Clinical intervention is a clinical initiative, a manipulation or a change to be introduced to a healthcare venue. For example, a clinical intervention might be collaborative care for depression. This many include a health promotion intervention or a delivery system intervention. It could be quite complex. Implementation intervention in contrast is a single method or technique that is designed to facilitate the change that should occur via the intervention. These are the individual techniques – performance feedback, automated clinical reminder, opinion reader, et cetera. Some of the implementation or interventions you might already be familiar with.

Those might be bundled up or packaged up into what is called an implementation strategy. The strategy incorporates a set of implementations, interventions that you would utilize in order to optimize uptake of an intervention. So you are going to see these charts come back in the next few slides. I’m going to present about three slides now that show you the hybrid types in a few different ways. If you don’t get it from one way of looking at it, you might get it from the next way.

You can see along the top that it is isolated as an implementation piece of the pipeline. You’ve got three types of hybrid designs. So, on the left you have hybrid type one. The emphasis here is more on effectiveness. You are still testing the clinical intervention and observing or gathering information on implementation but not putting as much emphasis on that aspect of the study. A type two is a hybrid study that essentially balances the focus on effectiveness and implementation. You are still testing your clinical intervention but you are also studying your implementation intervention and if you have a set of interventions, your implementation strategy. Type three has more emphasis on implementation. Here you might be testing your implementation strategy while still gathering information on your clinical intervention but you are paying less attention to your effectiveness and more attention to your implementation.

Here’s another way to look at it. You see on the left clinical intervention. Is that your focus? Yes, and if implementation is also your focus, yes, then you probably have a hybrid type two. In contrast if your clinical intervention is not your main focus, but implementation is, that’s the hybrid type three. Finally, the hybrid type one has the main focus on clinical intervention and less on implementation, although it is not completely absent. That would be hybrid type one. If you are not looking at the intervention or the implementation, then you are obviously doing some sort of observational or efficacy trial.

Here is one more way of looking at it, by thinking about what questions are you asking. For some people just thinking about the research question helps to position the study to a relevant hybrid type. The primary question in a type one is does the clinical treatment or intervention work in this setting or maybe in this other set of patients, which haven’t been tested before. Your secondary question might be from an implementation side. What are the barriers or facilitators to treatments in this particular setting? Primary is your effectiveness, secondary is implementation. Type two has two primary questions going on at the same time. Will the treatment work in this setting and does the implementation method show promise if you are not going about trying to establish how it is working. In type three you get the flip side of hybrid type one. Your primary question is which method woks better at facilitating implementation and which components of that strategy are critical. Your secondary question is about effectiveness. In a hybrid type three you are still collecting some outcomes data but it is not your primary focus. Hopefully this will come to life in a minute.

There are a few – sorry – there are a couple of questions that you might want to think about when you are contemplating the idea of a hybrid design. You might want to look at barriers that emerge early on. You might want to think about what clinical intervention effects are sensitive to your implementation and what changes to the implementation strategy could be made to improve uptake? I’m going to raise some more questions in a few minutes.

Before I get into the examples, I just want to talk about a critical component of hybrid design. And that is evaluation. There are two types of evaluation that I’m going to talk about briefly here. The first is process evaluation. Process evaluation is designed to identify influences on process of implantation or the intervention prior to, during, and after the study. The key here is that the data is not being fed back to the intervention team during the course of the study. It is not effecting how the effectiveness trial is going on. This is typical of type one designs where you are not doing implementation but you are still gathering some data because you are trying an intervention in a new setting or with a new set of patients and you want to understand how it is working in this new context.

In contrast with type two and type three hybrid designs, you will be engaging in formative evaluation. You are still looking at influences on the process of implementation prior to, during and after the study, but with formative evaluation, you are using the data while the trial is going on. During the course of your hybrid study, you are using your evaluation data to optimize implementation or processes during the intervention during the study. There is a feedback loop from your evaluation to what is actually happening during the study. This is more typical of a type two or type three design. It certainly is a type two and sometimes typical of a type three design.

You are going to have sessions on evaluation later in the EIS series, so you are going to get a lot more information about what is involved in conducting the evaluation. I’m just going to talk about a hybrid type one design. Just to reiterate the main points of the type one, you are testing clinical intervention, and gathering information on implementation. That would be gathered via a processed evaluation. When would you use a type one? When you already have some effectiveness data available, also the clinical intervention would move more quickly toward implementation if implementation factors were identified.

So, a colleague conducted an example of a type one in the rewarding early abstinence in treatment studies. The clinical intervention was an incentive intervention in SUD treatment. Why was this study a type one? There have been no trials that have been done with a large sample of VA patients. There were effectiveness trials but not done with a particular population. They also needed to examine how the utilization of incentives would occur within the VA context. They needed to gather evidence of the feasibility of the intervention, which had its own complexity in the VA setting. The main aim of the study, to go back to that research question on a prior slide was to demonstrate effectiveness on a VA population. 

So, you might say well why not just does an effectiveness trial? Why do you have to make it a hybrid? One of the goals of the overall research agenda was to support broad implementation of this intervention in VA. They have the opportunity to conduct this process evaluation in order to inform implementation trials. If they had chosen just to go with an effectiveness trial, it would have taken a lot longer for this implementation to become part of the implementation VA. There are process evaluation measures – they kept a research team observation log. Those recorded interactions with staff that were particularly focused on how staff reacted to the implementation, barriers to implementation that they encountered and also recommendations for improving implementation in a VA setting. It was very specific to how does this happen in this setting not how does this happen under controlled conditions or under other settings that do not have the VA context.

Importantly here, as I mentioned before, the process evaluation data was not used to optimize implementation. They kept their evaluation data separate from their effectiveness trial, collected it and looked at it at the end. They did not inform the intervention with the implementation data.

Now I’m going to talk about type two. So we have the dual focus on clinical intervention on effectiveness and on implementation strategy, which is a set of implementation interventions through a bundle or package of implementation intervention. The trial, which has an effectiveness and implementation component has an evaluation that is going on throughout that identifies contextual influences on the clinical intervention and implementation. Ideally, you are collecting and analyzing data that will maximize uptake of the intervention throughout the study. What these means is that your evaluation data is enabling you to tailor material in order to optimize the intervention. You’re running as you go and using that information to help improve uptake.

When would you use a type two? When you have very strong clinical intervention data. You may have several randomized control trials supporting your intervention. You may have several effectiveness studies. The intervention itself is strong enough that you may want to collect effectiveness data but you do not need to emphasize effectiveness. You might want to consider type two when you already have barriers and facilitators. Maybe you’ve done some of the pre-implementation or step three works and that is helping to inform your hybrid type two design.

The example I am going to talk about now is the EQUIP study that I worked on led by Dr. Alex Young and Amy Cohen who is here with me today. In this study, which will be enhancing quality of care in psychosis, they designed the study on clinical intervention with the chronic care model for patients with various mental illnesses. Why did we choose a type two? It is important to know that we had already done a lot of pre-implementation work in our step three study EQUIP-1. We had a lot of our barriers and facilitators information, as you will see in a bullet point on the slide. In terms of our effectiveness data, there was already an established body of evidence-based practices with patients with serious mental illness. However, there’s been no multi-site study that has substantially improved quality of care for Schizophrenia within the context of usual care.

We needed an effectiveness component of this study that looked at implementation of the chronic care model in usual care or in usual specialty mental health clinics. Because we already had a strong evidence base for the intervention, we could balance out our investigation of effectiveness with a parallel investigation of implementation. We really needed to put more emphasis than we would have in a type one on our implementation approach in order to increase the uptake of these very established evidence-based practice. Our implementation of approach in this study was evidence-based quality improvement. 

We had our implementation strategy contained several different components that were linked to an evidence-based quality improvement approach that was in much of the work by Cohen. The design of our study was a clinic level controlled trial. We had eight clinics in four areas. We had an intervention and a control site at each VISN throughout the country. We enrolled both staff and patients. Our staff sample was comprised of two hundred and one clinicians and administrators across intervention and control and eight hundred and one patients with schizophrenia across intervention and control. 

Our formal evaluation measures were extensive. We studied organizational readiness routines and fast burnout before and after the study was conducted and we did that via an online survey using a variety of measures that are described in a few different places. We also conducted semi-structured interviews before, during and after implementation. The med-point interview was only conducted at intervention sites and that’s the type of data we used to inform implementation and optimize uptake of the evidence-based practices that we were promoting in the study.

We also had field notes being recorded regularly, logs of time spent on implementations and minutes from all of our meetings. In other words, there was a lot of documented and archival data that helps to support our understanding of implementation. This case, in the context of a hybrid type two study, we used our evaluation data to optimize implementation. 

Now I’m going to move into hybrid type three designs. An example of a hybrid type there – so here, again, you are flipping the emphasis from effectiveness to implementation. You are testing an implementation strategy, which again, is probably a bundle or a package of implementation and intervention. You are still dabbling and observing information on your clinical interventions and outcomes. The implementation trials would typically have formative evaluations throughout plus the evaluation of health outcomes. 

Again, you are still collecting the effectiveness data, but you are not as concerned about that end of the study as you are with implementation and testing your implementation strategy. When would you use a type three design? In the context of a robust clinical intervention, you might choose a type three design because you find that the effect of your clinical interventions is vulnerable during implementation. Another context in which you might use type three designs is when there’s a high level of need for developing an implementation strategy or clinical actions despite limited evidence base. For example, you might be familiar in VA with clinical interventions that are developed or that are put into place in patients in a medical home or primary care mental health integration, which is a good example, that are being put into place nationwide and could potentially be optimized via a hybrid type three study where you are looking at implementation strategies that might promote uptake of a national clinical initiative.

So, that was the case in the example that I’m going to talk about for a minute here. This is a study that was led by Joanne Kirchner and colleagues. It’s called Blended Facilitation to Enhance Primary Care Mental Health Program Implementation. This hybrid type three study related to a national clinical initiative to put primary care mental health implementation into place. So, what Dr. Kirchner and colleagues decided to do was a controlled trial of an implementation strategy. Here what they are testing is not the intervention or the clinical initiative of primary care mental health integration, but rather their effort to support adoption of this clinical initiative. Their implementation strategy was specifically focused on internal and external facilitation. 

For those of you who have learned a little bit about the parent model, you’ll recognize that term as a key component of parent model and they indeed use the parent model to drive this study. A future study in our EIS work is going to focus on theories and you will learn more about parent if you haven’t already heard about it. The example might come back to you during our workgroup session for those of you participating in the EIS program.

So, here the researchers were looking at how does this implementation strategy support adoption of different models of primary care health integration. Sixteen matched sites with comparison sites received standard dissemination support. Half of the sites received the implementation and half of the sites continued to receive the national support that was provided for primary care mental health integration. Sites that received the implementation were the sites that received the status layer of implementation, which was designed to optimize uptake or primary care mental health integration.

In their evaluations for this type three study, they used multiple uptake and fidelity measures across providers and sites. As you might imagine with that many sites and three different models occurring, there are a lot of moving parts and the evaluation was designed to pick up as many of those moving parts as possible and understand how facilitation was affording uptake of PCMHI. At the same time, while the focus was on the implementation strategy and the effectiveness in the context of the clinical national initiative, they also continued to collect patient level data on depression outcomes including both depression symptoms and hospitalization.

So, there are a few things to consider in the development of a hybrid design. One of the key questions that we often come across in training of this type is which hybrid type should I think about? What hybrid type is appropriate? What hybrid type is warranted of this juncture given evidence related to the intervention and potential for implementation and a complex implementation strategy, which might support uptake of the clinical intervention? Which implementation framework would work for the study that you want to do? Again, we are going to come back to implementation frameworks and theories in a subsequent session, but these hybrid designs are, I think in all cases that I’ve ever seen, driven by an implementation framework. Some types of conceptual framework that helps you understand what is it about your study that speaks to different aspects of change and helps you to understand implementation in order to move on, potentially, to scale up and spread. 

What type of design makes sense? Is it randomization or quasi-experimental? What’s going to be – where are you going to control and at what level? What is your sampling frame? Which population are you interested in? Are you interested in patients or are you interest in providers? Are you interested in facilities or clinical units? What are your units of analysis? Where are you outcomes primarily located? What areas of interest do you have and what are you intending to change? Are you still intending to change something at the patient level or the provider level or at the organization level, or more than on of the above? What measures will you need in order to gather the necessary information for the hybrid design that you select? 

As I hope has been conveyed through the examples, hybrid studies are typically complex because you are looking at more than one thing at a time and again the idea is that if you can look at more than one thing at a time, if it is warranted given the evidence, then you might be able to progress to implementation more quickly than if you did not engage in a hybrid design.

In addition, there are many study tasks.  You have to not only think about what those tasks are and again a process oriented framework can help you think about how to locate these tasks that different phases of your study have, but typically the main phases are pre-implementation, during implementation and post-implementation. Different frameworks label these phases in different ways. Certainly collecting data at these time points or across the life of a study becomes very important in a hybrid design. Also, how long do those phases last? What do you need to do in pre-implementation? How long is implementation going to last and what are you going to look at post-implementation and how long is your follow-up going to be?

There are a lot of challenges and a lot of rewards and a lot of challenges with conducting hybrid studies. One of the main challenges is that you have concurrent data collection and analyses occurring in a type two. In order to use your formative evaluation data in a type two, first you have to be collecting it and analyzing it in a timely manner so that you can actually make use of that data to inform implementation. This is not easy for those of you who have conducted studies. You know that it is often extremely busy and challenging just to collect the data much less analyze it while you are collecting data. These are not parallel. They are not happening in isolation from one another, especially in type two. You have to address both issues simultaneously and allow them to be mutually informative.

What this means is that you need a lot of different types of expertise on your team. Your team might need to be larger because you need to accommodate people with different expertise in order to accomplish the types of interpretation that are necessary in a hybrid design. If you are connecting a type two, you need team members that know how to collect the evaluation data and analyze it in order to feed it back to the intervention part of the team. 

Ideally in hybrid studies, you are publishing throughout. You are producing interim findings that are going to be of value to the field and push the clinical intervention forward in a more timely manner so one of the ways we do that as researchers is to get our work out there early and often. So, conducting the study and collecting the data, analyzing it and writing about it all at the same time poses many challenges, but in a relatively short period of time, you end up with a large volume of data that can be very informative with future steps in clinical intervention.

And, also, what is facing the field more and more is where to publish this type of work. Sometimes these designs are not well-recognized by different journals and by reviewers, so there may not be familiarity with the concept of a hybrid design and with the complexity of a hybrid design or the idea of an implementation strategy – of course journals know implementation science would know more with work like this than other journals that may not be familiar with this type of work. Fortunately with more and more work getting out there and the hybrid paper going out into medical care and other implementation papers being published in a variety of journals, awareness with these issues is growing. 

One of the key aspects of that growth in the field is the terminology. So, if you are looking to do this type of study, a hybrid design or even going back to pre-implementation studies, just carefully define whose concepts you are using and what your definitions are and how you operationalize these concepts so that when it does appear in the literature, people have a frame of reference and they know what an implementation strategy means and what a hybrid design means.

So, that leaves us with some time for questions and I am open to questions on the pre-implementation work and also the hybrid design aspects of the presentation. I have several other experts in the room with me who may chime in as well in our question and answer period. Molly, we are ready for questions.

Moderator:  Thank you very much, Doctor Hamilton. We have had several questions come in.

Dr. Alison Hamilton:  Okay, great. 

Moderator:  Okay. We are approaching the top of the hour. Please submit any questions or comments you have in writing using the Q&A portion on the right hand side of your screen on the go to webinar dashboard. Please write your question and press send. The first question we had come in – there was an indication that references would be provided. Where can we get those references?

Doctor Lipmann:  For those within the VA, those are on the SharePoint site for the EIS training program with Molly given that there are participants in the cyber seminar within the VA, we need to have those linked to your site as well .We will be back in touch but if you are in VA, they are in the SharePoint site for this program.

Moderator:  Thank you, Doctor Lipmann. We will receive those links and post them on the HSI webpage to make them available. The next question that came in – what is the DELPHI Consensus Method?

Dr. Alison Hamilton:  It is a factor that is designed originally by RAND, who looked at – it’s beyond the scope of today’s talk, but it is designed to bring together a panel of experts, broadly defined as patients or clients and consumers to establish priority and establish consensus around different areas of clinical care. So, it’s a consensus building process that has particular steps and it has been sort of variably used in different studies. There’s even to the point where there’s a modified DELPHI process and if you even Google DELPHI RAND, you’ll come across several different references and sources that can help explain it. We can possibly go into it in our workgroups if there is interest in learning more about that method.

Moderator:  Thank you for that response. This next question may also apply to the first one. Where do we get the quote unquote assigned manuscripts before the webinar?

Dr. Alison Hamilton:  Those are also on the SharePoint so if you have any trouble accessing the SharePoint, you can get in touch with Cyprus and we’ll be sure to get you those links and or sources directly.

Moderator:  Thank you. Next question, what are some key differences between the clinically meaningful and statistically significant evidence-based practices.

Dr. Alison Hamilton:  Statistically significant evidence-based practices are typically derived in randomized control trials. They are under highly controlled conditions that may or may not resemble and typically do not resemble actual clinical settings where care is being delivered. Under those tightly controlled conditions, statistically significant interventions might be developed and you would have an intervention group and a control group and if you find that the intervention group has statistically better outcomes under those controlled conditions, those would be statistically significant interventions. In contras and going along with studies that are done in controlled contexts would be effectiveness trials where you are more looking at can this intervention be put into place in real clinical conditions in a healthcare setting, a primary care clinic or a mental health clinic and still achieve outcomes that are favorable for patients. 

Those would be clinically meaningful interventions meaning that the interventions work in settings where care is actually being delivered rather than in settings where all of the noise as they call it of the complexity of the clinical setting is removed. Those controlled conditions try to minimize as much of the extraneous factors as possible and of course the clinical conditions have all of those factors built in to the context of clinical care. So when an intervention holds up under clinical care conditions in usual care settings, then it’s beginning to develop the effectiveness end of things where you say now if we test this in the setting or maybe with a different set of patients, maybe in a different geographic location or a different population demographic, those are where the effectiveness trials come in and you see how the intervention works or does not work under different types of conditions that are not controlled.

Unidentified:  I would also add that it sort of depends on your audience. Healthcare organizations would be very interested in clinical differences and may sometimes be statistically significant and sometimes not. For example, you may find that in your weight loss program, you get people to lose five pounds. You know from the literature that a five-pound or ten percent weight loss is very meaningful for someone’s health. But it may not, for a variety of reasons, be statistically significant. Your readership or your research readership may be very interested in statistical significance, but the clinician crowd and the people who are implementing this are very interested in clinical significance. I would just say that in general, you need to be very aware of both and you need to be talking about both when you publish.

Moderator:  Thank you both for those responses. The next question we have that came in – there are a few questions referencing this, so I’m just going to make a general announcement. There are supplemental materials and I understand that some of you have no received information about the papers that she was mentioning or the papers. We are going to compile all of those links and all of those references and we will send you an email specifically where you can access them on the SharePoint site and the HSR&D webpage. If you have any questions regarding any of the references or materials or any of the follow-up papers, please stand by. We will compile those and send you an email as soon as possible. Can I go ahead and skip that question, then?

Dr. Alison Hamilton:  Sure. Absolutely.

Moderator:  Thank you. ARE QUERI funding hybrid type one designs or is it best to go to HSR&D? 

Dr. Lipmann:The situation may be changing. I think in general it is best to go to HSR&D, but it’s relatively easy to get a quick response from QUERI when you contact them and describe the project and ask them whether it would be appropriate for either an RP or an SUP.

Moderator:  Thank you, Doctor Lipmann. The next question – is formative evaluation work in hybrid type two always used as feedback to that or are there other ways in which you can use formative evaluation in that in this design?

Dr. Alison Hamilton:  I’m going to assume the person means are there other ways you can use formative evaluations in a type two in addition to feeding back the data. 

Moderator:  That is my understanding.

Dr. Alison Hamilton:  In addition to using it to tailor implementation. I can speak for the EQUIP study that that is one of the ways in which we are using our formative evaluation during the course of implementation to tailor our implementation intervention during the course of the study. In addition, the evaluation data is helping us to contextualize a lot of our effectiveness findings so now the study is over and we’re analyzing mountains of data and our evaluation data in many different areas is allowing us to develop a better understanding of our outcomes and effectiveness outcomes as well as different aspects of our intervention. 

So, for example, we can look at quantitative outcomes data on utilization of one component of our clinical intervention and we may see statistical significance there, significant improvements in the intervention conditions. We also want to understand what’s behind that. We might want to be able to explain that this aspect of our clinical intervention worked, so to speak. It performed better in intervention than – our intervention group had better outcomes related to this. But we also want to understand – did providers like it? What did they feel about it? What did they feel about it overtime? 

Were there barriers to utilization of some components in our clinical intervention? And maybe that varied by site, so it’s extremely useful after the study to help put the outcome data into context and also to describe and basically tell the story of implementation so that we can potentially move on through the study that we conducted to a more wide scale implementation and potentially move in to a type three. In addition, moving backwards, not only did we use our evaluation data mid-stream to help optimize uptake of our clinical intervention, but we also used our pre-implementation data to help even in the setup and framing of our implementation. So, we used it throughout the life of the project and are continuing to use the evaluation data to this day, not only on its own but also to contextualize our effectiveness.

Unidentified:  I would add one thing to that. Just to be very literal, is the data fed back, meaning to the people at the site and I would say sometimes it is. So you package it and you might give them feedback about their quality of care or where they are in terms of implementing compared to other sites that are implementing it or to a benchmark, but sometimes what we mean when we say feedback is that we use the data to tailor what’s going on at the site. They may not know the exact results of the data, but we may be using it behind the scenes to specific training at one site to catch them up or do specific training at another site because they are weak in a certain area or using another site as an example site because they are doing something really well. I think there are, in terms of the literal words do we feed it back, there’s a difference in that.

Dr. Alison Hamilton:  That also brings up an important point which is that we have continually engaged in this study and I believe it is the case in other hybrid studies as well, that we have continually engaged with the key stakeholders at the site. They are aware on a regular basis of our results and are involved in presentation manuscripts and that kind of thing. There is that collaboration that occurs in our study as well. That collaboration is critical to implementation. Without collaboration, implementation is extremely difficult. It’s not easily possible.

Moderator:  Thank you both for those responses. We do have twelve more questions to get through. Is type two and type three hybrid designs considered pre-implementation or implementation studies? What is the difference and does it matter?

Dr. Alison Hamilton:  It definitely matters. They would not be considered pre-implementation. You might need to do pre-implementation research in order to set the stage for a type two or type three hybrid study, but actual implementation is occurring in a type three or type two and a type one for that matter. That implementation should be informed by pre-implementation work. You wouldn’t want to go about beginning an implementation study without a firm understanding of barriers, facilitators, gaps in care and evidence based guidelines that might exist or pre-existing evidence for the clinical intervention. All of the different things that you might investigate in pre-implementation work and QUERI step three types of study. You want to have very solid and thorough understanding of all the different aspects before you would embark in actual implementation research set up in a hybrid study of any type.

Moderator:  Thank you for that response, Doctor Hamilton. Are you available to continue answering questions? 

Dr. Alison Hamilton:  Sure.

Moderator:  Okay. We do have one request as a person who would like to consult with somebody regarding ARV issues in implementation studies. Can you go ahead and inform them that we will incorporate that in the follow-up information we are sending out?

Dr. Alison Hamilton:  Absolutely. They can contact Cyprus for that.

Unidentified:  It’s a very important issue and it is very complex in these types of studies. You usually have more than one sample that you are doing things with from an IRD perspective. To explain that and convey what’s happening with who when is very complex, so we’re sympathetic to the IRD concerns.

Dr. Alison Hamilton:  Are you still their Molly?

Moderator:  Could you please refresh your screen real quick? It’s okay. It should also be known that there are hybrids within hybrids in regard to implementation strategies. I have a lot of implementation efforts that are technically hybrids but also gather evidence to tailor the intervention to promote implementation success. This isn’t a question, just a comment.

Dr. Alison Hamilton:  We thank you for that comment. We appreciate the comment, thank you.

Moderator:  Thanks. Next question. Do the type two and type three designs pose threats to the internal validity of the conclusions?

Dr. Alison Hamilton:  No. I wouldn’t say that they do pose threats to the conclusion. I think what they help is to contextualize the conclusions and help build your confidence in your conclusions you are making because you are understanding not only the effects of the intervention but the actual way in which the intervention occurred and the black box of what made the intervention work and is facilitated by that close look and work in the implementation end of things.

Unidentified:  One addition comment is that in some ways the issue is whether effectiveness studies pose threats to internal validity to efficacy studies, especially when the study includes a much broader range in setting and a much larger list in variables that have to be taken into account. There is not a difference between a type two and type three hybrid in terms of the internal validity of the clinical effectiveness portion of the project.

Moderator:  Thank you both for those responses. Is use in a conventional framework essential in a pre-implementation study? Please provide an example.

Dr. Alison Hamilton:  I would say that our consensus on that would be yes, that even if you are not addressing all aspects of a conceptual model, that it is critical to develop an understanding of what you are attempting to change and how you are attempting to change it. That usually involves some combination of concepts and theory and again, you may only be addressing one part of a conceptual model, but having that as a foundation for the subsequent implementation work can help to guide and mold the direction that you take. So, having a firm understanding of the framework in which you are operating and the framework in which you are addressing effectiveness and implementation needs to start from that early work. 

Also in terms of grants, it helps to give the reviewer a picture of where your study fits in terms of the next study. That’s really helpful. Where does this end up leading? Usually the conceptual model can help you describe that efficiently.

Moderator:  Thank you both for those responses. The next question –if you are funding a project study using the hybrid framework for projects?

Dr. Alison Hamilton:  Yes. 

Moderator:  Excellent. When you talk about sampling frame and use it as an analysis, is it best to assess on multiple levels or focus on one? For example, we’re making a patient care change. Both patients and providers are affected.

Dr. Alison Hamilton:  Without sampling the providers, you might not know how they are affected. I think in hybrid design, the nature of the data that you collect from various levels might differ. So, you may not collect, for example, as much data from providers in a type one as you would in a type two, although there are no rules to that at all. But, really thinking about what’s your main emphasis. Clearly, wherever your main emphasis is, you want to be sure you are collecting the full collection of data that you would need to demonstrate that change. However, if you think for example, in a type on, you would certainly be focusing on the patients or if your type one is targeted at another level of change like the facility – I would say typically in most hybrid studies that I am familiar with, the unit of – there is more than one unit of analysis. You are looking at more than one aspect of what’s occurring during evaluation. You are looking at a system.

Moderator:  Thank you for those responses. I will keep reading until you tell me to do otherwise. Does implementation research always have a comparison group, for example if a previous study shows the effectiveness of an intervention and the subsequent study is looking at two year follow-up and stability across sites but no control group, would this be considered a type of implementation research?

Unidentified:  It would to the extent that it would define implementation research as research that is focused on documenting and understanding implementation processes. Obviously there are threats to internal validity, but it is not easily collected data on processes that falls under the broader umbrella of implementation research.

Moderator:  Thank you for that response. I am developing and pilot testing a nursing based intervention with a small and equally number of patients. I am interested in hearing about their experiences. Does this sound too early for a type one hybrid?

Dr. Alison Hamilton:  It depends on the intervention. We may have different ideas about this, even in the room, but to me, that question depends on how much evidence there is and evidence broadly speaking, that there is for the nurse led intervention. Has it ever been tested and in what context? Maybe some effectiveness research needs to be conducted on the intervention. It really depends. Maybe it is a clinical initiative that needs to be examined via a hybrid design and there is sort of a combination of evidence that supports its potential effectiveness but you also are under some type of urgency to look at implementation at the same time. But, the question for me is really what is the nature of the intervention and that guides its movement through the hybrid studies design.

Moderator:  Thank you for that response. I will go ahead and ask that person to contact for more information. Next question – do type one hybrid settings exclusively include implementation conceptual frameworks and implementation strategies?

Dr. Alison Hamilton:  I would say those that I am familiar with do include implementation frameworks. It is not necessarily an implementation strategy. The focus is on examining and studying implementation but not intentionally shaping implementation via an implementation strategy in order to affect uptake of the intervention. Brian has an additional comment.

Dr. Lipmann:  To make something explicit that I think is important and highlights a key link in the two paths that we talked about is that for the most part, a hybrid type one-implementation aspect is for the most part pre-implementation. They are QUERI step three types of activities. What are the key barriers and facilitators to eventual wide scale adoption of this practice that has been evaluated still. To the extent that attention to specific types of barriers and facilitators should be directed or guided by conceptual frameworks, it is critical but there is no active implementation effort in a hybrid type one. It is more of a matter of thinking head to the type of implementation strategy that might be needed once we have a better handle on the barriers and facilitators to implementation.

Dr. Alison Hamilton:  Molly, we have a workgroup here in LA gathering, so I don’t know how many more questions you have, but we should probably end by ten past the hour so that we can accomplish our task in the workgroup.

Moderator:  Okay. We are approaching ten past the hour and we have numerous questions left. I will send those to you offline and you can send me back written responses which I will then distribute to our registered members by email and also put them on the SharePoint site. At this time, I want to let you give any concluding comments. 

Dr. Alison Hamilton:  Concluding comments. I think we are fine. I think hopefully a lot of these issues will become more and more familiar as you continue to attend the EIS sessions. These issues of framework and theories, evaluations are all going to be covered in upcoming sessions. Just keep an eye on the calendar, especially those not enrolled in EIS. If you are interested in the various components of these designs and other implementation research issues, then attend more future sessions.

Moderator:  Thank you for that. I would like to shamelessly plug our next session, which is taking place on June 7th at 2:00 pm, EST. We will have Doctor Laura Dan Schroeder presenting a follow-up to this presentation. It will be the third session in a six series session and the title is the role and selection of theoretical frameworks in implementation research. You can go to the HSR&D webpage and click on cyber seminar catalogue and register there. 

Also, please do look for a follow-up email to this session, which will contain a hyper link directly to the archives. We will include additional information to where you can find references to SharePoint, et cetera. I want to pass along my many things to Dr. Hamilton, Mitman and Cohen for presenting today. Thank you to our audience members for joining us. This formally concludes today’s HSR&D cyber seminar.

**[ End of Recording ]**