**Enhancing Implementation Science**Veterans Administration

July 26, 2012- 16 -

**Veterans Administration**

**Enhancing Implementation Science**

**EIS- Intro Program Session 6: Enhancing Implementation Science Evaluation Details (Outcome Measures, Formative Evaluation)**

**Alexander S. Young**

**Hildi Hagedorn**

**July 26, 2012**

Moderator:  We are ready to begin.  And I would like to introduce our speakers.  First we have Dr.  Hagedorn speaking.  She is an implementation research coordinator for the Substance Use Disorders QUERI, a core investigator for the Center for Chronic Disease Outcomes Research, and staff psychologist with Minneapolis VA Medical Center. And second we have speaking Dr. Alex Young.  He is the director of the Health Services Unit of the Department of Veterans Affairs, Desert Pacific Mental Illness Research, Education and Clinical Center and a professor at UCLA department of psychiatry and the greater Los Angeles Healthcare system.  And at this time, I would like to turn it over to Dr. Hagedorn.  Are you ready to share your screen?

Hildi Hagedorn:  Yes, I am.

Moderator:  Thank you.  I’ll turn it over to you now.

Hildi Hagedorn:  All right.  Well we wanted to start with just—we wanted to start with just a couple of poll questions so we can see who exactly we are speaking to today.  So our first question is are you affiliated with the VA?

Moderator:  Okay.  We do have some responses coming in.  And—they are still streaming in so we’ll give everybody a few more seconds to respond.  Okay.  Looks like we’ve had about 83% vote.  I’m going to go ahead and close it out.  And share the results.  And Hildi you should be able to see the results now.  Would you like to speak through those real quick.

Hildi Hagedorn:  Yes.  It looks like about 78% of participants are associated with the VA and we have 22% that are not.  So about a quarter.  So welcome to everyone.

Moderator:  Thank you.  Go ahead and go into the next poll now.

Hildi Hagedorn:  Okay.  

Moderator:  And to our attendees go ahead and select the response that most closely associates your primary role.  We do understand this is not a comprehensive list, but we are limited to five answer choices.  And it looks like we’ve had about two thirds of our audience vote, so we’ll give people just a few more seconds.  Okay.  Pardon me.  The responses have stopped streaming in.  I’m going to go ahead and close this one and share the results.

Hildi Hagedorn:  It looks like the majority, 72%, clicked researcher and then we have under 10% that clicked for the other categories of clinician, manager, policymaker, student trainee or fellow or other—so primarily researchers.

Moderator:  Excellent.  Now we’ll go had and go to our final poll and I’m launching that now.  So what is your level of participation in implementation research?  Choice one, I have been a PI on an implementation study.  Two have been part of a study team for implementation study.  Three currently developing an implementation study proposal.  Or four no hands on experience, just getting started.  It looks like we’ve had about a third of our attendees respond.  We’ll give people a few more seconds.  Okay.  And we’ve had about 80% response rate of our 112 attendees.  I’m going to go ahead and close this one.  And I’ll share the results.

Hildi Hagedorn:  All right, so it looks like we’ve got a pretty even split.  We have about a third that have been part of a study team for an implementation study; about a third that are currently developing an implementation proposal, and about at third that say they are just getting started and then we have a small, about 10% that say that they have PI’d a study before.  So we really appreciate you doing the poll questions so we have a sense for who our audience is and I will move into my formal talk now.

Moderator:  Great.  Let me go ahead and give you control back and we’ll get started.  You’re going to see a popup and please press show my screen and I’ll let you know when we can see your slides.

Hildi Hagedorn:  Okay.

Moderator:  there you go.  Go ahead.  Thank you.

Hildi Hagedorn:  All right.  Let’s get past the questions here.  So the purpose of today’s presentation is to give you some examples of formative and process evaluation methods that Dr. Smith talked about in the previous cyber seminar.  I should have asked as a poll question how many people attended that, but hopefully most of you were able to attend that and so you have a little bit of a background for today.  I’m going to be talking today about the rewarding early abstinence and treatment participation study as an example.  The study objectives were to test the effectiveness of an incentive program with a large sample of veterans with alcohol or stimulant dependence and we were comparing their rates of negative alcohol and drug screens during the intervention, their rates of attendance during the intervention and their percent days abstinent out of the past thirty days at two, six, and twelve months follow-ups.  Our second objective was to assess the cost of the intervention and our final objective was to complete a process evaluation to inform future implementation efforts.  

So for today what we’re going to be focusing on is our second and third goal as those represent process and formative evaluative aspects of the study.  In relation to the QUERI pipeline, this study would be categorized as a mainstream HSR&D effectiveness study.  However, we decided to make it into a hybrid type one by including elements of a pre-implementation study.  So basically we had a standard effectiveness trial, but we added in some tools and evaluations to be able to identify barriers and facilitators to implementation that would help to inform future studies and future implementation studies in the pipeline.

So we recruited 330 veterans that were seeking treatment for alcohol or stimulant dependence at two VA substance use disorder clinics and they were randomly assigned to either usual care which was the standard care provided at the clinic, and also included breath and urine testing twice a week for eight weeks or to an incentive program which included the usual care and the breath and urine testing but then if they did test negative for alcohol and illicit substances they were given the opportunity to draw for incentives in the form of VA canteen vouchers.

As I said, we were going to cover the costs and also the process evaluation.  And this was—we did just a very simple evaluation of the cost of the intervention.  So basically we tracked the amount of vouchers that patients earned as they were going through the intervention and that was on average $103.  We had to have rapid urine test cups to use because with incentive programs you need to be able to provide immediate reinforcement.  So you can’t be sending the sample down to the lab and waiting for results.  So those were $5.25 a piece times 11.6 visits which was the mean number of visits that our patients attended.  So $61 for those supplies.  We also had to supply mouthpieces for the breathalyzer that was used.  The breathalyzer costs about $200.  But most substance use disorder clinics already have one on site so we didn’t include that as an additional expense that a clinic would need to account for if they started up this intervention.  The mouthpieces cost about a quarter, so times the 11.6 visits that was an additional $3.  Staffing costs:  The average visit length was fifteen minutes and we counted all sixteen appointments for staff time because if a patient did not attend one of their scheduled appointments, the staff would still be required to enter a no show note and reach out and make contact with the patient to determine why they no showed and to reschedule them.  If we add all those costs together, we had a mean of $269.  So that was good evidence that we had for future implementation that this was a low-cost intervention.

We did have our highest cost patient just for perspective was $462.  This was someone who tested negative, attended and tested negative at every one of his appointments and was very lucky with his drawings.

So moving on now to the process evaluation.  We used the RE-AIM framework to guide the development of our process evaluation.  RE-AIM stands—is an acronym which stands for reach, effectiveness, adoption, implementation and maintenance.  And each one of those single words is to –is meant to trigger you to think about certain questions that you want to ask about your intervention.

When you’re thinking about reach, what you want to know or what we wanted to know is of the patients that we approached to participate in the study, how many of them were interested.  Did the patients that agreed to participate differ from those who refused?  So if you put this intervention into a standard clinic, how many patients will it reach?  How many of them will be interested and want to be involved with this intervention?

Effectiveness has to do with basically the test of our main study hypothesis.  Under these conditions, does this evidence-based intervention—is it still effective in approving patient’s outcomes.  Adoption:  We did ask questions about what are going to be the greatest barriers to sites who are adopting this intervention.  And how can those be overcome?  Implementation means you ask questions about what kind of tools would programs need in order to deliver the intervention in a consistent manner and maintain fidelity to the evidence-based practice.

Maintenance, we ask questions about what type of resources would be required to maintain this practice in a clinic without the support of the research study.  And also what changes, if any, would have to be made to the intervention in order to sustain it beyond the research study support.

We also were guided in developing our process evaluation by the PARIHS theoretical framework.  PARIHS states that successful implementation is a function of strong evidence, of strong context, and facilitation.  And so again, those three elements lead you to think about specific questions that you might want to have answered regarding your intervention.  So evidence leads you to think about how do the staff perceive the evidence supporting the intervention that you want to implement?  And does the intervention fit with their current practice and with what they perceive to be the needs of their patients?  So we know or we consider this to be an evidence-based practice, but we don’t know if the staff agree with us on that and that is important to know.  Context leads you to ask questions about what are the characteristics of the culture and the leadership in the clinic and what resources are available that will either support or what characteristics will either support or create barriers to implementation?  And then facilitation lends us to ask questions about what type of resources, training and tools would staff feel would be most helpful to them to maintain the intervention?

If you look at these two frameworks that we use to develop our process evaluation, you’ll see that a lot of the questions that they lead you to do overlap.  But each one of these frameworks also provides some unique questions that are not covered by the other one.  So that’s why I felt that it was important that we kind of combined the two and covered additional bases.  So once we knew what types of questions we wanted to answer, then we needed to link our data collection to our frameworks to make sure that in the end we had the answers that we wanted or data to try to answer our questions.

For the RE-AIM constructs you can see the constructs listed on the left and the data source on the right.  To assess for reach we looked at the things like our recruitment rate and the demographic characteristics of our patients who agreed and who did not agree to participate.  For effectiveness that was the main study outcomes of rates of negative urine screens and rates of study retention.  For adoption, we observed—we planned to observe interactions or—I’m sorry—planned to observe the intervention going on in the clinic and also to—systematically collect the perceptions of staff and leadership about the intervention.

Similarly with implementation and maintenance we collected perceptions of staff for implementation.  What tools would they need to continue to provide this in a consistent manner?  And we collected perceptions of leadership regarding maintenance.  Did they have the resources to maintain it?  Were they planning to maintain it?  What additional resources would they need or how would they need to modify the intervention?

For the PARIHS framework, to make sure we covered evidence; we collected perceptions of staff or wanted to collect perceptions of staff and leadership regarding the evidence.  For context we wanted to use an organizational readiness measure that would be collected from staff and leadership and for facilitation, we decided that observations of the intervention occurring in the clinic and also staff and leadership perceptions would be valuable.  So when we knew where—what types of data we wanted, then we needed to move on to developing our tools.  So the tools that we use for our process evaluation included the organizational readiness to change assessment which we had staff complete at the beginning of the intervention—of the study.  And this is a readiness to change assessment that is based on the PARIHS model and it assesses staff knowledge of the evidence base.  It assesses their attitudes toward the intervention and also has organizational context questions related to leadership, culture, resources, and so on.

We also developed a research team observation log.  This was a log that was shared by the members of the research team and allowed them to record interactions that they had with staff, with particularly focusing on the reactions of staff to the intervention, on barriers that staff identified to implementation, and on recommendations that staff provided for ways to improve how the intervention functioned in the clinic.

We also developed a staff post- intervention interview and again, we wanted to know their reaction to having the intervention running in their clinic.  How they felt the intervention impacted the clinic, the patients, their workload, etc.  And what they found—what they felt would be the biggest barriers or facilitators to implementing this in other clinics and also whether they had recommendations for improvements.

We also did post intervention leadership with leadership interviews.  Those primarily focused on whether they planned to continue the intervention after the end of the study, why they made that decision.  And if they did decide to continue it whether there were any modifications that they were planning to make.

Finally we did patient post intervention interviews so we could find out what the patients liked or disliked about the intervention; how valuable they felt it was and whether they had recommendations for improvement.

So just to provide you with a few of the insights that we gained from going through this process, it—because this is a lot of work and I want to make sure that people do understand that there is some value that comes out in the end.  To give you a few examples, thinking about reach from the RE-AIM framework, we found that patients were very enthusiastic about entering the program and they did enjoy it.  Regarding evidence from the PARIHS framework we found that staff were not enthusiastic about the intervention at baseline.  So they knew the research evidence supporting the intervention, but they did not philosophically agree that this was an appropriate way to treat people for substance use disorders.  What we found very interesting was that their enthusiasm increased very dramatically during the intervention period.  So we had the intervention running in their clinic for about eighteen months and at the end of those eighteen months, staff were much more open and excited about this type of intervention, after they had seen how it affected the clinic and the patients.

Regarding maintenance, the staff suggested to us that a group intervention would be more feasible because it would cut down on staff time.  However, that was something that we asked patients about in the post intervention interview that we did with patients and they were not interested at all in having this intervention occur in a group because one of the aspects of the intervention that they reported as being most helpful and valuable to them was having individual one on one meetings twice a week for eight weeks with the same person who was very supportive of their efforts.  And so they did not want to give up that one on one aspect of the intervention.  So I think that just shows how important it is to collect perspectives from a wide range of stakeholders because then we can go back to the staff and say that’s a really great idea and actually I thought it was a great idea as well, but the patients do not like that and if we make that change, it’s going to undermine the intervention.

Finally for facilitation, I think the fact that we saw this change in staff attitudes really suggests an implementation strategy for the future.  I think that it suggests that one of the best ways to approach this would be to try to engage staff in implementing this intervention for a trial period.  So being able to say you know allow them to express their concerns, communicate that you understand those concerns, and say can we just try this for six months or nine months.  And then revisit this and see how you’re feeling about it after that.  I think that experience with having it ongoing in the clinic was so positive for the staff.

So finally I just want to talk a little bit about the variety of uses of process and formative evaluation.  We—this is an effectiveness trial, so we were collecting information to inform future implementation.  We did not use the information that we collected during this study to change how we were doing the intervention in the study.  We used it to inform how we might make changes after the study.  However, obviously, process and formative evaluation can be very, very important for implementation trials for a number of reasons.  It allows you to assess stakeholder’s perceptions of the feasibility of your intervention and the value of your strategy before you even get started so you’re able to identify some barriers in advance and hopefully address them before they become overwhelming.  It allows you to understand and adapt your implementation process in real time so as I said in this trial, we didn’t change our intervention but if you were doing an implementation trial and part way through identified a huge barrier and identified a strategy that you could use to adapt your implementation process, you could do that as part of a formative evaluation.

Finally, I think that this process can help you to assess the impact of theoretical constructs on your implementation outcome, so it allows us to make a contribution to implementation science to say how this theory is valuable.  Does this theory predict implementation the way that it claims that it should?  And that is also an important aspect of implementation trials. 

Those are my formal comments and I believe we are going to move directly to Dr. Young’s presentation and then do questions.  Is that correct?

Moderator:  Correct.  Thank you very much Dr. Hagedorn and yes Dr. Young are you ready to share your screen at this time?

Alexander Young:  I am.  Yes.

Moderator:  Excellent.  I’ll turn it over to you.  Please accept the share my screen icon.

Alex Young:  Okay.  

Moderator:  Great.  I’ll just pop it up in the slideshow mode and we’re good.  Thank you very much.

Alex Young:  Okay.  All right.  Thank you and thank you Hildi. I ‘m also going to be giving a similar talk to what Hildi presented and the idea here is to present an example of implementation research and the evaluation approaches that were used for particularly with regard to process and formative evaluation.  And this is a project that was done with a number of collaborators across networks.  This is the –I’m presenting results from the EQUIP project.  This was an effort in four VA networks to improve care for schizophrenia.  Schizophrenia is one of the more common conditions seen in specialty mental health.  It is a disorder that is eminently treatable but that if treatment is not appropriate and patients don’t get the treatment that they need can be quite disabling and costly to treat.  There have been problems with improving care for schizophrenia.  It’s been difficult to implement evidence to practice.  Even though we have national treatment guidelines and effective practices, in routine care they’re often not used.  This has been a challenge both for VA and other non-VA mental health providers.  The mental health QUERI working with us conducted this study which was the EQUIP study to implement evidence practice for schizophrenia.  We partnered with four networks. And you can see the networks on this slide.  It was VISN 3 Northport and Bronx, VISN 16 Houston/Shreveport, 17 Waco and Temple, and 22 Los Angeles and Long Beach.  Each of these networks had one implementation site.  They were to implement evidence-based practices and improve care and the control sites were to continue with treatment as usual.  This is a hybrid type two study meaning that we were working to evaluate both implementation and effectiveness to better understand the implementation of these practice and also the effectiveness of these practices in terms of improving patient outcomes.

These are the aims of the EQUIP project.  The specific aims.  Our first aim was to work with the network to implement evidence-based care for schizophrenia.  This was critical.  Without this partnership it’s really not possible to implement evidence-based practices to care.  We have to align our work as implementation researchers with the goals and priorities of the networks and the medical centers and the clinicians to understand how this can fit with the usual care arrangements and policy.  Second we wanted to evaluate the effect of implementing these practices on provider competencies.  These are the skills, knowledge, and beliefs of clinicians and they’re critical of course to getting evidence-based practices into care.  We also want to understand the effect of this implementation on treatment appropriateness and this means the extent to which patients receive treatments that are evidence-based and respond to their needs.  And they’re also interested in patient outcomes and the effects on utilization of services and costs.  I’m not presenting any cost utilization data today.  We’re currently analyzing that and then third, we wanted to evaluate the processes of and variation in care model implementation and effectiveness.  

This is the formative and process evaluation—Alison Hamilton is a health anthropologist who works with our team and led this component of the study is—we used mixed methods approaches both quantitative and qualitative methods to evaluate the processes of and variations and implementation of the care model and effectiveness.  There have been prior studies in specialty mental health implementing evidence-based practices, but the challenge has been that because they did not have robust qualitative components it is very hard to learn from them.  So as a rule implementation projects almost always produce less change than we would like.  We would like to produce large change in care and large change in outcomes.  That’s just not possible by and large and so what’s—even though as you’ll see in the study we did produce change in the care and outcome, there was less than we would have liked, so the process and formative evaluation is critical for understanding why that is and developing future work.  So for understanding what parts of the implementation were successful, what parts met with challenges or barriers.  And so in the future how they can be made more acceptable and stronger.  And so that future efforts can even increase the effectiveness and implementation of these interventions.

And so how we approach that is by using the qualitative method to understand the accessibility of the care model and various facilitators that occur throughout the implementation process which is the day to day process and this project was a one year implementation period and we want to understand how the strategies and tools of implementation affected implementation which ones were helpful and which ones were not and then be able to look back and understand the impact of individual care model components and treatment appropriateness to try and triangulate if we can get some information about what was working and what was not working.

At baseline we started with a strategic planning process for the networks to align the interests of the project with the network goals and presented them with a list of evidence-based practices that can be implemented into care and then work with them to understand which ones they wanted to work on and they all chose weight and work as outcomes of interest and then we started with the diagnostic evaluation at the sites looking at the structure of care for patients with schizophrenia at each of these sites which of course is quite variable.  These are sites which—we had sites such as Houston which were urban large complex systems and sites such as Waco, Texas which were quite rural and tremendous variation across VAs and how they get the work done depending on the local context and so we wanted to understand that and we also wanted to understand how care, current care, varied across the site. And in terms of implementation, we used an evidence-based quality improvement framework which is basically a—we used the usual quality improvement process of plan, do, study, act where we identify areas for change, and make organizational change and study the change that that produces, but at the same time the evidence-based approach is that we create a quality improvement team at each of these sites which were composed of project staff and also a line clinician and managers at each of the sites and so this allows us to work together to identify areas for re-organizing care and strategies that we know can produce change based upon prior research and our understanding of implementation.  And the goal of this is to focus the quality improvement process on the evidence.  On the evidence for organizational change and the evidence for process improvement.  In terms of the logistics of this, we implemented a kiosk at the site, a patient-facing kiosk, to elicit the information from patients on a routine basis and we had a quality manager at each site that was responsible for a panel of patients using the approach that’s used in chronic care models for managing chronic illness and we also had patient and provider education and we were fortunate to have ground support from leadership at these sites.  

We chose to use the Simpson transfer model as our conceptual model for evaluating implementation and organizational change.  This is a well established model that is used frequently in substance abuse treatment settings.  We made some adaptations of the instruments for mental health settings and for the VA and those are actually now available on the TCU website that you Google Simpson transfer models and you’ll see they have a very robust website with instruments and evaluation approaches and this is why one of the reasons why we chose it because it was well established and we didn’t have to reinvent the wheel in terms of evaluation approaches.  This involves four action steps and involves closure to the intervention, induction and training adoption, implementation and then practice movement into routine practice.  I’m not going to go into too much detail on this.  You can this slide will be—these slides are available for you if you have any further questions, you are welcome to ask me.  There’s also been some publication on this, Alison Hamilton led a publication which you can find in Implementation Science and which discusses this project amid so many details and then we’re also have some other publications on the way.  So the—basically we had a breach of the [inaudible] implementation and practice and we had specific strategies and schools that we were evaluating and then we also had data sources such as we kept keeping project field notes from the project and the staff.  Project documents, logs, tracking activities, we did provider and clinician management interviews at both the start of the project and partway through, halfway through and then at the end stage we used the organizational readiness for change instrument which is the standard approach that Hildi mentioned for evaluating readiness for change and we studied the Maslack instrument for provider burnout to measure confidence in established instruments and put all these together in the evaluation.

Moderator:  Dr. Young—

Alex Young:  Yes.

Moderator:  Dr. Young, I apologize for interrupting.  We do have a request for you to project your voice a little louder.  Thank you.

Alex Young:  Oh, okay.  I’m sorry.  So in terms of the formative evaluation, these are the steps, pre implementation, implementation and post implementation and you can see some of the data sources, as I mentioned, that we use for—including the interviews.  We had developed a survey instrument for clinician managers at the sites and then also did semi-structured interviews with those folks at the beginning and end of the study which are analyzed using qualitative methods. I’m going to present some of the evaluation—overall evaluation strategies.  So the summative evaluation meaning the evaluation in terms of affecting care quality.  We had 801 patients who participated, 201 providers, and again, these are the aims we’re evaluating—the effects on competency, appropriateness, outcome, and utilization.

These interviews are zero and twelve months and we also pulled in data from VISTA on treatment utilization.  In terms of our process evaluation, the surveys that I mentioned at zero, six and twelve month interviews, we also monitored use of the kiosk system and kept copious notes and logs which is really critical.  Because there’s a lot of activity that goes on—there’s a lot of work to do and it’s important to understand at the end exactly what those activities were and they would otherwise not necessarily be documented.  You can see here, these are the results—organizational readiness for change using the ORC scale or the TCU ORC scale.  And these are three larger domains, motivation for change, staff attributes, and organizational climate.  You can see the scores at each of the sties.  These are the four implementation sites, A, B, C, and D.  And you can see there was variation.  Also there are norms on these scales so we can also understand how these VA norms—how these VA numbers compare to other numbers nationally.  Now we did find substantial variation in readiness for change at the sites.  At baseline which we then used to guide our implementation efforts and our facilitation of the implementation and quality improvement process.  So for instance sites A and B were very ready for change and we did not have to share implementation strategies there.  Site C had a lower sense of the needs for change and we had to present information about increased awareness about the gaps in care and some knowledge about the potential for evidence-based practices that were not being used.

We needed to also—they also were low in terms of their mission and their autonomy.  So we had to work on the messaging with the sites and our presentation and discussion with the folks to based upon those differences.  Presenting some of the results, I—in terms of our organizational strategies we were fortunate to have strong support so each of the sites implementation sites were committed to the projects and work with us diligently in terms of the meetings and reorganizing care.  The collaboration between services was difficult. In VA there are service lines such as mental health and nutrition, primary care, and suggest these collaborations across service lines was challenging.

For this project we were able to make it happen in the areas that were necessary so for instance the wellness programs that were established for weight in specialty mental health settings we were able to make that happen in the project.

Now the clinician competencies did improve through the education that was provided and the practice in these new evidence based practices that the clinicians were using.  Also the managers made use of the data that we collected both at baseline and midpoints that we organized for care provision.  So for instance, for the weight implementation, scales were placed at the clinic for routine weighing of patients and clinical staff were trained to provide services. In the intervention to improve employment outcomes, there were actually staff hired for example to deliver the employment services when it was realized that there was a large gap and there was also retraining of some staff and clinicians supervising the necessary services.

In terms of changing care, these are the results for the weight intervention.  At baseline we found that 45% of patients at these specialty clinics were obese—that means they have a BMI of 30.  Body Mass Index and that indicates borderline for obesity so the mean weight is obese.  70% were on medications that can cause substantial weight gain and use of services was low across the sites.  As a result of the implementation effort, patients at intervention sites were more than two times likely to use services.  And that was a significant increase and the mean number of service sessions used for counseling with regard to diet and exercise was eleven, which was a substantial increase from baseline.  There were no changes seen at any of the control sites across the project.

At the end of the study, control site patients were thirteen pounds heavier which is a significant difference.  So there was a significant improvement in weight on the basis of providing improved services.

I’m going to wrap up.  In summary the message is I would say is that these qualitative formative and summative evaluations are—formative and process evaluations are critical and the goals in terms of strengthening the intervention throughout the study and thinking about what needs to be done differently.  These are—these studies are different in some ways than the more traditional controlled trials where there’s a structure and one definitive way of doing things which is carried throughout the project and it’s necessary in this implementation effort to really think creatively in terms of the organizational change that is being encouraged and produced and we need to have data to do that and this is so that data needs to happen throughout the project and be available to project teams so that there can be adjustments made and sites can reorganize their care.  It’s the same for the summative data that’s also necessary throughout the project to understand where the implementation is weak or not happening and finally we really need the data at the end to understand what works and what didn’t so we can inform future implementation efforts and implementation science is a relatively new field and we have some idea of what is helpful in terms of reorganizing and changing care and how to make it happen but for instance in this study it’s great that the patients at the intervention sites at the end were less—had less weight, fewer pounds but that data by itself, of course is of very little use, actually because it shows that there was some—there was a possibility of improving weight through implementation but it doesn’t tell policy makers or researchers how to do it in the future.  That’s where we need to draw on the data that was collected.

I’ll stop there and thank everyone for their attention and I think we have some time for questions—for both Hildi and myself.

Moderator:  We do.  And—yes.  I’d like to thank you both.  Do you want to advance down to your last slide for the references or your contact info just so we’re not—there we go.  Any one of those will do.  Thank you.

Okay for those of you that joined us at the top of the hour, to submit a question, just simply go to your Q&A function on your goto webinar dashboard and press submit and I will start reading the questions in the order that they were received.

These first ones are regarding Hildi’s presentation and—I apologize.  I’m skimming through them. So the first question that came in—it seems like you would need to test if a group intervention would work in a different clinic rather than concluding it won’t work just based on patient opinion.

Hildi Hagedorn:  Well, I think that’s absolutely true.  I mean, you can’t say we—the patients expressed their opinion and I may have stated when I was talking that the group intervention won’t work and that was a misstatement because we don’t know that it won’t work.  You would absolutely have to test it to see whether it would or not, but I think the feedback from patients would have the clinicians pause for a minute and say okay this study showed that the individual intervention does work and also showed that the patients don’t want it to be changed to a group format.  So if we’re going to continue on with this intervention with the information that we have now we would probably be best off sticking with the individual approach.  Certainly you know a study to test whether the group approach worked or not would be great and that information could be sent back to the clinics and if it did work they could change their intervention as well.

Moderator:  Thank you for that response.  The next question, how did you decide to add implementation measures to this particular study?  It seems like there would be many studies that would benefit from this.

Hildi Hagedorn:  I totally agree with that.  What happened was that I had originally proposed an implementation study because I had felt that with the review of the evidence base for contingency management interventions, I felt that we were at a place where we needed to move on to implementation.  The response I got to that proposal was that there was no evidence—no strong evidence for the effectiveness of contingency management interventions within the VA, with veterans, with VA patients.  So I modified my proposal to an effectiveness trial but I always had my eye on implementation and so I had an implementation expert that was a consultant on the project from the first submission and so with her help we decided to add in this process evaluation to inform future implementation.  And it is interesting because clearly it fits the hybrid one definition but there was no such thing as a hybrid one you know.  That terminology didn’t exist when we developed this study so it’s kind of interesting that now that terminology is out there, but I absolutely think that this type of information any effectiveness trial would benefit from this type of data collection, for informing future implementation.

Moderator:  Thank you for that response.  The next question: what is the difference between this type of implementation trial and a pilot study and this is also directed to you Dr. Hagedorn.

Hildi Hagedorn:  Okay.  Well, as I mentioned, this is a standard effectiveness trial, so this is a fairly large randomized controlled trial.  It’s—if I’m understanding the question correctly you’re asking about an implementation pilot so an implementation pilot might be the next step after this study.  So we have our effectiveness data.  We know that this works in the VA and we have all of our process information that we collected from this study so now we are really armed with a lot of information and knowledge to move forward with a pilot implementation trial.  So I would say the next step is—continuing on this line the next step would be a pilot implementation of perhaps three or four sites using all the knowledge that we gained from the effectiveness trial in process evaluation and trying to further develop our implementation strategy because this trial didn’t have an implementation strategy.  So that would need to be developed and piloted and hopefully that was what the question was.

Moderator:  Thank you for that response.  They have the option to further clarify if they want to write in. 

Hildi Hagedorn:  Okay.

Moderator:  These next questions.  We have about seven remaining and I believe they could be addressed to either of you.  They started right about the switch of the topics.  It seems like you would need to test if a group intervention would work in a different clinic rather than just concluding it won’t work—I’m sorry—I already read that one.  The next question—if the is implementation reach also considered “action research” and if so are the research steps processes the same?

Hildi Hagedorn:  Um.  I don’t know if Alex would have a ready answer for that one.  

Alex Young:  I guess I’m—I would need to—I’m not familiar—I’m not exactly familiar—I’m not sure what’s being meant by the term action research.

Hildi Hagedorn:  And I’m kind of in the same place.  I have a vague understanding but I don’t know the processes and steps that the person asking the question might be referring to, so it would be difficult for me to answer how they differ from each other.

Alex Young:  There is research—there are research guidelines on sort of pragmatic research.  I don’t now if that’s—exactly what’s meant by the term action research.  There are guidelines out there about effectiveness research so about studying the effectiveness of services in usual care arrangements so that the results are more externally valid and applicable.  So I’m not sure—perhaps the writer can say a little more about what they mean about action research or either now or we can answer it by e-mail later—this is certainly very action-oriented research where implementation is getting in the trenches. I’m just not familiar with that term.

Hildi Hagedorn:  And it’s also—I would say it’s very much participatory research as well because you’re involving your clinical experts as members of the team to develop the plan or refine the plan.

Alex Young:  That’s right.  I would strongly agree with that and it relates also to this large emerging field of community based participatory research and the ideas involving developing partnerships in the research with partners be they patients, clinicians, or policy makers early on and in meaningful ways throughout the project.  And those are related areas.  Implementation research can be either partnered or not and I think you can see from the way that our studies were presented that having that kind of partnership is probably really important to a successful outcome.

Moderator:  Thank you both for those responses.  The submitter did want to add, it also pertains to the study of interventions and will be used for my upcoming dissertation with the VA based study, so we thank you for that added information.

And the next question we have in the queue is what is the name of the organizational readiness for change instrument and is it possible to get a copy?

Hildi Hagedorn:  I did want to clarify that the measure that I was speaking about was the organizational readiness to change assessment or the ORCA which is different from the Organizational Readiness to Change the ORC that Alex was referring to.  As far as the ORCA goes, there is a publication in Implementation Science—the lead author is Christian Helfrich, and that would be the easiest way to get ahold of information on that measure.  It’s an open access journal.  All electronic, so anyone can pull up that article.  And maybe Alex can speak about the ORC.

Alex Young:  Right.  Different ORCs.  Our ORC was from the Simpson Transfer Model and if you go on and just put in Simpson Transfer Model into Google I suspect and look at the Texas Christian University Website, they have a large web presence there with discussion of the various kinds of instruments and approaches.  There are also a publication that we have on this and I’ll see if I can—you can see this publication by Alison Hamilton for General internal medicine in 2010 organizational readiness and specialty mental health care which I can provide much more information for you on this. I  would also, while we’re on the topic of reading, I’d like to point out that Anela Proctor led an edited book that was recently published on Implementation Research and in healthcare which is actually our—Brian Mittman has a chapter there and there’s substantial materials from the VA implementation science experience within that book.  That’s also—for folks who want a reader for this area; I think that’s a good one.

Moderator:  Thank you both for those responses.  We do have a large portion of our audience still with us, and we have four questions to go.  In general what is the difference between process evaluation and formative evaluations?

Hildi Hagedorn:  Well, I can tell you what my definition of the difference is, but I have heard other answers to that question.  The way I view it is that a process evaluation would be something like what I presented where you have a set intervention that you are testing and you are not going to modify that intervention during this particular trial, so you’re collecting information to inform your future activities whereas a formative evaluation is actually used during an implementation trial where the information that you collect at the time is fed back into the intervention so that the intervention can actually change over time during that particular study to account for what you learn through your formative evaluation.

Alex Young:  That’s my understanding of it too.  This is a problem in the field is that the nomenclature is variable and you have to really ask people what they mean by particular words, unfortunately.  They don’t have the dictionary yet for implementation science.  We tried to do both a formative and a process evaluation in the same study and so process being the look back evaluation.  Basically the evaluations where you get the data analyzed after the study is over and look back at the project and the formative evaluation is the work done during the project to try to strengthen the implementation efforts.

Hildi Hagedorn:  The other distinction that I have heard is that formative evaluation is what you do at the beginning when you collect all the information to design your intervention and then process evaluation is what you do during the implementation to track the process as it is unfolding.  I don’t use it that way.  I would refer to a formative evaluation that starts with a developmental evaluation and then moves to an implementation-focused evaluation.  Both being pieces of a formative evaluation.  So now that we have you all completely confused.

Alex Young:  Someone told me once that every profession needs it’s argyle schools, meaning it’s words and it’s tools so at least with implementation science we’re starting to develop some of these but you know again we don’t have the standardization so you make sure when people present things that you ask them exactly what their meaning.

Moderator:  thank you both for those responses.  I know I personally struggled with formative evaluation in my studies.  And one of our submitters would be glad to hear more about how you both gather, manage and organize all of the qualitative data involved.  The field notes, notes from multiple sources, the logs, minutes, etc.

Hildi Hagedorn:  Well, we manage it by entering it into one of the qualitative data programs such as Atlas TI or In Divo.  And so each piece whether it’s an interview or field notes is entered as a separate source document into that program.   Don’t know if Alex has more details.  I don’t have a lot of details because I’m not an expert in that particular—in with those particular software programs.

Alex Young:  I’m a clinician, so I’m going to get out of my depth even quicker than Hildi.  But I think that’s—Atlas TI what we use.  Alison Hamilton is our expert on that and does trainings in Atlas TI and management of those kind of data.  The data is basically interviews are transcribed.  We collect and key punch our quantitative data in the usual way.  We do it from web systems but for the qualitative data there’s the transcription information and really it is a great question because it’s an enormous volume of data and that’s what these packages are good at the Atlas TI and similar packages that are allowing you to manage that in some kind of rational way because without that you would be very quickly overwhelmed by the enormous amount of information that interviews can produce.

Moderator: Thank you both for those responses.  We do just have a few questions left.  Are you both available to stay on the line for a few moments and address these last few.

Alex Young:  Yes.

Hildi Hagedorn:  Sure

Moderator:  Thank you.  And Alex, before I move on to the next question, can you please back up to your contact slide.  There do seem to be some particular questions that would be better addressed to you personally through e-mail.

The person who just asked that question, they want to clarify—they weren’t asking about software in particular but rather about people tasks of getting various people in various roles to turn in their field notes or minutes to a central process where they’d get entered into the software.

Hildi Hagedorn:  Well in the particular study that I was talking about it was fairly straightforward because we only had two sites so we had two people that were doing the interviews and they would record the interviews and then transcribe the interviews and put them into Atlas TI.  The field notes we actually had like Excel spreadsheets with different headings so if you had a chat with a staff person in the hallway then you could go back to your computer and open up that spreadsheet and enter who you talked to and when and what the content was and those types of things and so we all shared that spreadsheet.  There was a lot of information entered into that log and so I felt like people were using it and I’m sure that there were things that were missed and I’m sure it gets exponentially more complicated when you’re talking about four sites or when you get up to eight or ten or something like that.

So I think it’s you know your standard good project management of keeping track of everyone.

Alex Young:  And that’s just—you know that’s why we use similar kind of processes to what Hildi was talking about. At a distance you have to train the people who are working on the project, to do the field notes and what you want.  You have to monitor it, but you also have to provide, make sure you have the resources.  So you know resources meaning in terms of funding for people’s time because you can’t really ask someone who’s fully committed in a job as a clinician or a manager to carve out time for doing field notes unless there is some sort of support for those activities locally and training.  So I just think that’s—the—it’s a very important area and it’s not an easy one to get people to engage in those functions, but it is absolutely critical so I think it’s something to be thought about from the beginning of the project.

Moderator:  thank you both for those responses.  We do have one more question.  Even though patients and Dr. Hagedorn this is directed at you—even though patients stated they preferred one on one counseling, would it not make sense to try the group approach suggested by the staff.  Maybe the patients would enjoy support of the group interactions if they tried it.  How do you decide to go with what you have versus adding another arm to the next study you do on this topic?

Hildi Hagedorn:  Well, I think there’s a distinction between adding to the research evidence and implementation.  So based on the study that I completed, if I was going to talk to a new clinic about how should you implement this, I would say we have strong evidence that an individual approach works and we also have patient feedback that they prefer not to do this in a group.  If I were going to do the next research study, then I think it would be great to compare group to individual and see if they work as well.  If they did, that would resolve some of the clinicians’ concerns about the time invested in doing the individual appointments.  Absolutely.  And you could work it into a hybrid trial.  You could, I suppose you could implement like you said have both types going and yes.  I suppose you can do that.  

Moderator:  Thank you for that response. And this is directed at Dr. Young.  One of our attendees wanted you to repeat the name of the book that you mentioned.  I believe it was when we were talking about perhaps formative evaluation.

Alex Young:  Yes.  It’s something like Implementation in Health Care, but the thing you can search for is Anela Proctor—P-r-o-c-t-o-r.   Enola with an E.  She’s one of the editors of—I think if you’re go to Amazon it should be—I think it’s under implementation science and healthcare, but any case you go to Amazon and look up Enola Proctor and you should be able to find it.  She was at St. Louis University in St. Louis if you need to look for her on the web.

Moderator:  Thank you.  We do have one last comment that has come in and then I will allow you both to make any concluding comments you’d like to.  This person wanted us to know for our information that there is a group-based CM system that reinforces group attendance.  And they state further that group CM that targets abstinence can be very problematic because of privacy issues regarding drug use.  

Hildi Hagedorn:  Yes.  There are a variety of different types of contingency management interventions that target different behaviors and have different behavioral goals that achieve an incentive and so we were specifically looking at abstinence goals, but absolutely treatment attendance can also be reinforced and that does—I have experience working with that in a group and that seems to function quite well in a group.  Yes.  Thank you.

Moderator:  Excellent.  Questions do continue—comments do continue to come in.  One is that dissemination and implementation research in health is published by Brownsen, Collitz and Proctor.  Part of Oxford Press.  Thank you for writing that in.  Also there is a request—can you repeat the author or article on the ORCA.

Hildi Hagedorn:  Sure.  The author is Christian Helfrich.  Published in Implementation Science.  

Moderator:  Thank you very much.  That does address all of our questions and comments and I cannot thank you both enough for sharing your expertise and to our audience members, at this time I’d like to give either of you the chance to make concluding comments and we’ll go ahead and start with you, Hildi.

Hildi Hagedorn:  Just I really appreciate everyone’s interest in this and thank you very much for joining us today and if you have any questions after the session, feel free to contact me.

Alex Young:  And I just say—

Moderator:  Thank you.  Dr. Young?

Alex Young:  I just echo that and you know I’m available.  You can reach me at the e-mail there and you know I thank everyone for showing up today and you know for their interest in the area.

**[End of Recording]**