**Transcript of Cyberseminar**

**Patient-Aligned Care Teams Demonstration Labs Series**

**Making Sense of the Ideal & the Real in PACT Implementation Using Qualitative Research Part 1: Methodology, Methods, and Research-Operations Partnership**

**Presenters: Alison Hamilton, PhD, MPH; Samantha Solimeo, PhD, MPH; Gala (Jennifer) True, PhD**

**September 11, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact**Samantha.Solimeo@va.gov**;**alisonh@ucla.edu**;**alison.hamilton@va.gov**; or**jennifer.true2@va.gov**.*

Moderator:And we’re going to go ahead and get started now; so I would like to introduce our speakers. Speaking first today we have Dr. Samantha Solimeo and she is a Medical Anthropologist with VISN 23 at the PACT demonstration lab center for comprehensive access and delivery research and evaluation that is located at Iowa City VA Health Care System. Following her will be Dr. Alison Hamilton a research health science specialist and lead of the qualitative methods group, at the HSR&D center of excellence of the study of health care provider behavior at the Greater Los Angeles Health Care System; she’s also affiliated with the women’s health services research consortium and the Desert Pacific Mental Illness Research Education and clinical center. She is also an associate research anthropologist in the department of psychiatry of behavioral sciences at UCLA. And finally we will have Dr. Gala True who is a core investigator at the center for health equity and research and promotion, known as CHERP. She is also qualitative core for VISN 4 center for evaluation of PACT and a research assistant professor in the division of geriatric medicine at the University of Pennsylvania at the school of medicine. 

So without further ado I would like to turn it over, give me just one second and we’ll get those slides up. Sorry about that, my computer is being a little tricky. Okay, so I’d like to turn it over to you now Samantha.

Dr. Samantha Solimeo:  Thank you, I’m just waiting for the slide to pop up so I can see what I’m -- what you all are seeing. 

Moderator:Excellent so once again everyone I would like you to be typing in your questions and answers into the Q & A box in the upper right hand corner if you have any questions for the presenters or myself; otherwise feel free to go up into full screen mode so that you can see a larger version of the slide. And as soon as they’re done loading we will be getting started. Thank you for your patience. 

Dr. Samantha Solimeo:  Okay great. So you should be on slide one. And I’ll start by mentioning a little bit of background. So in 2010 the Office of Patient Care Services funded five geographically disbursed demonstration labs as well as the coordinating centers.to build a research network for setting best practices in PACT implementations. And each lab sets its own research agenda and builds its own clinical research partnership to facilitate the labs individual goals and objectives. But at the same time also labs share two general purposes. One is to evaluate the PACT implementation and the second is to develop multi-level strategies for improving PACT implementation and related outcomes. 

The labs also share substantive investment and qualitative data. Really in part, because qualitative data can illustrate as we’ve peddled our PACT care, how the idea of PACT is implemented in real VA clinics. So one aspect of what we’ve observed across the labs is that as an organization VA culture really values the ideals and Veteran center care and ground level innovations. These ideals of tailored care and the ground level innovation are then held in balance with organizational needs for standardization across practice sites and also within realms of accountability. 

As an organization we have to be able to demonstrate better investment and human and capital resources improves patient outcomes. As an anthropologist the puzzle that I’ve been working on is essentially where is the sweet spot here? So how can we as an organization achieve the balance between standardization and innovation? And does the balance look different for different kinds of facilities or PACT teams?

One approach was taken here in Iowa City is we start by observing, listening to and working with the experts in PACT of limitations. Both staff were in the thick of making [Indiscernible] PACT in the context of their teams, their clinics and their professional lives. In other words, as we suggest we consider our front line employees and operational partners experts in the real. And as social scientists we then compare the ideal PACT model to what is actually happening on the ground. Qualitative data really reflects the experiences of people implementing PACT and they stay to help identify social and physical spaces for improving team performance. We draw our insights on the field.

What we’d like to do in this two part cyber seminar is to share some of this collective insight into the challenges of PACT of limitations as well as provide some models for how the VA might use qualitative methods to inform training, team function and [Indiscernible]. In today’s session I will first touch on why methodological diversity is important to health services research and explain why qualitative investigators are your best ally’s for developing meaningful and productive research [Indiscernible] partnerships. And Dr. Hamilton from VISN 22 will present their methodology for evaluation and Dr. True of the VISN 4 lab will discuss the role of the clinical research partnership. I’ll take any questions that you have at the end of Dr. True’s comments. And then on October 16 we’ll present part two of this seminar which will address what we’ve learned from PACT teams about implementation and team functions. Just a note to let you know we’re not going to dive into the nuts and bolts of methods in these sessions. I wanted to point to this resource on the cyber seminar website. The recent presentation [Indiscernible] really excellent and they give you a really nice overview of qualitative methods. 

First a comment about methodological diversity. On the side of research and methods and methodology associated with qualitative research. Those of you who then countered qualitative research may have come across any number of the terms I have listed here. This is really a positive reflection of the increasing integration of qualitative work within health services research. Despite this reception I recently observed a degree of orthodoxy regarding the best way to design a qualitative studies and there are a number of reasons for that; so I wanted to bring up three.

As far as -- is that some of this rigidity might simply be a product of what people are exposed to and the qualitative investigators that I know use this complexity clients so that’s how all qualitative studies should be designed. The second rationale or cause might be that despite growing a corporation of qualitative masses into health services there’s a lack of familiarity with the diversity of methodology. And this lack of similarity compels us to look for some level of standardization with which we can evaluate [Indiscernible] branch or a scholarly paper. And then the third I think has to do with some surface and how to use these terms in conversation, which is why I’ve listed both methodologies and methods on this slide. 

Moderator:I’m sorry to interrupt Samantha; can I ask you to speak up just a little bit?

Dr. Samantha Solimeo:  Yes, I got that a lot; sorry. I’ll try to speak a little louder.

Moderator:No problem.

Dr. Samantha Solimeo:  So we should be on slide eight and with this model as a guide. Just on the slide about the slippage between methodology and method. So methodology is tied to or served as a theoretical model under pinning the research design, which in turn dictates the methods that you’re going to use. Methodologies informed by epistemology or in other words what it is we believe constitutes evidence. Methodologies grow under which individual methods are selected to answer a particular question. Common methodologies in the VA include complexity science, informative evaluation, implementation science or even participatory research. The methodological framework you choose for your research must make sense for your individual project. You have to take into consideration your research teams expertise, your operational partners, your project time line and the anticipated products. Once you’ve identified the methodology, you should then reconsider your research question and ask yourself what evidence is required to answer this question? How will I collect the data in empirical fashion? And who are the experts on this topic and why? 

The answers to those questions then inform the selection of your specific methods such as interviews or participant observation. I think when you consider the relationships epistemology, methodology and method it’s easy to see that methodological diversity is a natural product of vying to scientific inquiry. As investigators to solve health services issues from a variety of perspective. This diversity of approach fosters creativity and moves us forward in the production of knowledge.

Methodological diversity also in part of product of the diverse data sources with which we can understand PACT. Across the five labs we’ve done from what you might call researcher created and systems created materials. So investigator research produced data sources such as sight visits or interviews allow us to build an understanding of implementation from the bottom up. So looking at the field experience from any stakeholder vantage point so that we’re equally equipped to understand a service line leaders constraints as we are clear goal associates. Whereas data drawn from ongoing PACT activities helps us to understand the different levels of organizational change and how they interact. But even though the five labs are dedicated to understanding the single PACT model our research questions differ, which means that our methodology amend the company methods also differ. These are positives; ultimately when we pull our collective insights and setting such as these we maximize the number of angles from which we can understand PACT.

Now onto the second point I wanted to make, as lead investigators we conduct our research to contribute to the greater scientific community, but at the same time we’re ultimately responsible for leading research which improves the care of our veterans. This accountability to our veterans is increasingly becoming more transparent as we’re encouraged to collaborate with operations. Those collaborations help ensure that our research is feasible, that it’s actionable, that it’s relevant. So qualitative investigators are bringing an emphasis on research operations collaboration. It seems very natural because it’s an integral feature of best practices and qualitative research. And I have three examples to illustrate this.

First it’s the process because the processes of embarking on qualitative research are collaborative. We establish colloquial relationships with the people from whom we wish to collect information. We spend a lot of time up front developing repoire and building trust to ensure that we have full ideals of confidentiality and informed consent. But also out of respect for people’s time and their willingness to give us some of that valuable time. We schedule our interviews at times that suit them, not us. And in some cases we provide individuals with copies of the data that we collect from them. 

The second is the methods. The methods are designed to reflect our views that respond to the authorities on our own experience. We enter into data collection with the relationship or an attitude of mutual respect, which carries over into instrument design and selection. The interview questions are framed with regard to time constraints, education attainment and confidentiality issues. Observations are conducted as unobtrusively as possible. When we ask people questions about what they’re doing we do it with transparent interest, curiosity and expressed assurance that our role is not to judge them, to tattle on them or to interfere; it’s really to record, to understand and communicate. And this way activities like interviews or sight visits become collaborations between investigator and the respondents. We create these data together.

The third point is about disseminations. We disseminate our findings in a variety of format to ensure that our participants benefit from what we learned. They publish scholarly articles and present work at professional meetings, but we also write briefs and reports and newsletters for the operations audience. These documents are typically produced in advance of scholarly materials so that we can disseminate findings back to operations as quickly as possible because we all know that scholarly dissemination can be a very lengthy process and operations typically need this information yesterday. 

Each of the PACT labs approaches research with this own methodology and associated methods, but we’re all working towards reaching a shared understanding of the same phenomenon, how do we achieve successful PACTing? Is there a single pass? Does the path differ for CBOC’s and the AMC’s? Does it differ for teams of residents compared to full time providers? Do rural clinics have a unique pathway? These questions and questions regarding how to develop mutually beneficial collaborations they’re going to be taken up from my co-presenters.

I’m going to turn it over to Alison Hamilton and VISN 22.

Dr. Alison Hamilton:  Thank you so much Samantha. I also want to say a special thank you to Samantha for the visual high quality of our slides. She really added a nice touch with all of our beautiful photos and everything; so thank you for that. 

So I’m going to talk briefly about using this qualitative method to evaluate implementation of PACT in VISN 22. Our demonstration laboratory is called the veteran’s assessment and improvement laboratory or VAIL. Then we have one driving research question, which is as follows. Will systematic technically supported implementation of PACT features through evidence based quality improvement methods and the research clinical partnerships they embody be more successful than routine PACT implementation? As the [Indiscernible] mentioned each lab is taking its own approach to figuring out strategies for optimizing PACT implementation. And the approach that we’re taking at VISN 22 is the approach of evidence based quality improvement or EDQI. 

You’ll see here that I have certain words in green and then certain words in orange. What this reflects is the mixed methods that we use to approach this main research question. All the words in green are issues that we’re explicitly exploring through our qualitative methods and more successful the words in orange are not the only area that are being addressed by qualitative methods. But what I wanted to highlight here is that the mixed strength that we have by having a mixed methods design. So when we’re looking at issues of measurement, for example looking at what constitutes more or less successful implementation we really need to bring our qualitative colleagues to the table so that we could benefit some of the measurement methods that they use and then use those to compliment the methods and the types of data that we’re able to collect via qualitative methods. 

In other words what we have in VISN 22 is sort of an overlay on PACT implementations, this overlay takes place in some sights and not others, but we have some sights where the EBQI intervention is occurring and I’ll get into those in a minute. And then we have other sites where PACT is so called naturally occurring. 

As Samantha mentioned each effort in each demonstration lab is driven by methodology which is sort of a combination of theoretical perspectives and frameworks. Our particular combined model as I mentioned is really based on a foundation of evidence based quality improvement, which is a multi-level of quality improvement approach in large part spearheaded by the PI of our demonstration labs, Dr. Lisa Rubenstein. The demonstration and sites where we’re actually doing EBQI are engaged to varying extents and planned to study Act cycles which are one of the pillars of quality improvement. We also have a strong systems redesigned contribution and we’ve built our entire approach on a point complex logic model that puts all the different parts together to hopefully achieve what we desire to end up with, which is a strong PACT implementation. Our theoretical basis is also informed by social marketing and leadership theory, in particular interdisciplinary leadership, which is the hallmark of EBQI and the types of partnerships that we strive to support and facilitate in our sites. 

Our design is a phased approach, a quasi-experimental evaluation design. Some of you might recognize this is what’s called a stepped wedge design. As I mentioned the EBQI is taking place at some sites and not others and we’re doing this in this phased approach; so you see back in June of 2010 we started with stage one with three facilities and we set up our infrastructure. So all the components that are part of our intervention were really initiated at that early phase which was pretty close to when PACT itself started. But there are aspects of the infrastructure like quality counsels which are interdisciplinary team based approaches to PACT implementation that to varying extents either reflect or add to what already existed in the clinics. But the idea is that the infrastructure that we set up in each of the practice sites would help to optimize implementation by bringing multiple disciplines to the table and then working on quality improvement endeavors. 

We moved into phase two with the addition of three more sites in about -- throughout fiscal year 12 and now we’re currently entering into phase three which involves spread of our intervention to the rest of VISN 22. Now this has many moving parts that are kind of underneath this design that you see here where you’ve got a longitudinal approach and we also have an emphasis on looking at the sort of additive as well as cumulative imPACT of our efforts over time. We’re looking at kind of like how to features of VAIL change over time, how does the infrastructure change over time both collectively across all of the practice or intervention sites as well as with any site. What types of creative approaches are they taking to using the intervention and perhaps highlighting some aspects of it and maybe not using other aspects of it and so it’s very important for us to identify the critical components of our intervention in a very implementation research orientation? 

As you might imagine because of the complexity of our intervention we have a very complex evaluation which is mixed methods design and involves multiple different efforts by a whole collection of teams. The two circles that you see here highlighted are the implementation evaluation and the team effectiveness evaluation both of which are involved in qualitative methods. And then the other circles that you see are more heavily utilizing qualitative methods. What is important to note here is that we -- even though we’re off doing our own things as teams, collecting our respective data we meet regularly, we meet once a month and we also any time we’re going into a new phase of data collection share our measures instruments questions, ideas with one another to try to ensure that there’s as much complimentarily across the types of data as possible. So we ultimately want to be able to look at what our survey told us and what our qualitative data told us for example in six months or a year. Of course that means there has to be some synergy of the types of questions that we’re asking and the domains that we’re covering in a different effort; so that takes a lot of communication. 

I’m just going to go into a few moments of detail about what we’re doing in the qualitative components of our evaluations and try to put them side by side so that you can see how we put these methods together to address that overall question that I brought up before. The objective of our implementation evaluation is to evaluate both VAIL and PACT implementation and imPACT. We’re only doing the implementation evaluation and for that matter the team effectiveness evaluation in the intervention sites at this point. Whereas the other data collection efforts such as the surveys also take place in non VAIL sites; so there is that sort of intervention control type of issue in our evaluation where we’re capturing so-called control sites and some aspects of the evaluation but not others. And for our team lead effectiveness evaluation we’re very interested in determinate of team effectiveness, changes over time and team composition and roles and also organizational facilitators and barriers to effective team work. Much of which we also get at via our survey but different ways asking different types of questions. The overall design of our evaluation is a longitudinal [Indiscernible] of mixed methods design where we’ve got different streams of data collection going on at the same time with multiple opportunities for interface across the different parts of the evaluation. For the qualitative data collection we have four ways of data collection. 

As Samantha mentioned earlier one of the key aspects of designing your methods is thinking about who can answer these questions, who are the people who are best suited to tell you what you need to know about implementation. For our implementation evaluation we’re obtaining the perspective of key stakeholders and most key stakeholders are comprised of VISN, VAIL, primary care and PACT leaders are some of whom are serving multiple roles so we may have individuals who are VAIL, primary care and PACT leaders all rolled into one person. We also have clinic leads, the quality counsel members that I mentioned before the interdisciplinary team and we’re also interviewing our researchers. So our VAIL leadership and other individuals who are critical to VAIL implementation are interviewed on a regular basis.

Our team effectiveness evaluation we are interviewing a random sample of team members across the intervention sites; so we’re interviewing every level of the team, primary care providers, the RN care managers, LVN, or clerk, MSA across all of the intervention sites. The methods that we’ve chosen to use to answer the key question that we’re asking, plus multiple ancillary questions are semi structured interviewed. Those are some of the core backbone of the qualitative evaluation components. As I mentioned these interviews take place -- of course the four data collection points or waves of six months in between each wave. In addition we have monthly briefed web based updates with a subset of our key stakeholders. To get at a little bit more of a detailed notion of what’s happening with implementation not -- you know not on a daily basis we can’t get that granular, but at the very least we wanted to get a monthly perspective from those key individuals about what was happening with PACT implementation and those are a little bit more structured to be able to accommodate a web based delivery which was preferred by the stakeholders who were willing to do that aspect of data collection with us. 

And finally we have continual document collection and cataloging so that downward arrow that Samantha had on a slide earlier about all the materials that are coming from within the facilities meeting minutes, etc are all being collected and documented and cataloged by us in order to augment our ability to tell the story of PACT implementation. 

Finally just to get a little bit more specific about what we’ve done thus far. We’ve accomplished two waves of data collection; you can see the sample sizes here and we’re currently in the midst of wave three. We did an alternating design so that we’re going back to our phase one site at this point and then four we’re going to go back to our phase two site, so there’s about a year in between interviews of the key stakeholders at each type of site and each phase of sites.

We are getting all of our interviews professionally transcribed and we’re using a little bit of a different approach for our preliminary analysis in order to accommodate the highly interactive nature of our evaluation. So one of the most common issues that people raise with qualitative research is “Well it takes a really long time” and so we’ve tried to identify some ways to make preliminary analysis more efficient, more timely and more productive on a continual basis. So what we do is summarize our interviews as we go along using a templated format; so our interview guides cover a set of defined domain although of course the questions are still open ended and then we prepare summaries across the team of people who are selecting data to just identify key points that were raised in each interview with the idea being that if I don’t conduct a particular interview I can look at a summary of that interview and understand the key points that that individual made in the interview. So it allows us to share a good deal of information about what we’ve learned without having to get deep into the formal coding process. We moved those transcript summaries into matrices so that we can look both with insights across the domain and across sites to start to formulate some preliminary ideas about our findings and observations and we’ve been able to use those to prepare presentations for our leadership on a regular basis. So after each wave of data collection we’re able to prepare these presentations based on our summaries and give the leadership an idea of what we learned in that particular wave of data collection. That’s also said very nicely into our more formal analytic process which is highly team based and we’re using qualitative software, we’re using Atlas and using a combination of deductive and inductive coding and that aspect of a formal analytic process is designed more to serve the manuscripts and so forth that we’re working on for other aspects of dissemination. But we did find that we really needed a more rapid approach to compliment the more formal approach in order to turn our interim findings around to leadership.

So at this point I’m going to turn things over to Dr. True and she is going to talk about what her lab is doing with clinical research partnerships and mixed methods. Thank you.

Dr. Gala True:  Great, thank you Dr. Hamilton. Today I’m going to build on what Dr. Solimeo and Hamilton have talked about by highlighting the intersection of mixed methods with clinical research partnerships for one particular project within the VISN for center for evaluation of patient and care teams or CEPACT. I’ll go on my discussion by describing how our demo lab collaborated with VISN four clinical apportions leadership to evaluate an innovative virtual learning collaborative that they had designed and decided to implement in order to support implementation and spread of PACT in VISN four. And in particular I’ll describe about how we went about designing our evaluation and why, some of the challenges we encountered and how we met those challenges and what we learned about the advantages of partnered approaches to research and implementation science. And given that this is truly a partnered project I’d like to acknowledge the significant contributions of my collaborators, so Anneliese Butler and Mary Pelak from CEPACT really implemented every aspect of the evaluation itself and Dr. David Macpherson and Jennifer Skoko of the VISN four offices conceived of, developed and implemented the virtual learning collaborative so it really belongs to them. 

So my focus today will be on how the virtual collaborative developed and how our partnership with VISN four clinical apportions leadership happened. I’m not going to talk about our findings from this evaluation but if you’re interested in that you can ask a question later or you can email me and I will direct you towards our write up of the findings from this work. So I want to set the context for you of how this partnership started. The CEPACT’s qualitative team was already deep into evaluating the national PACT spread strategies, which included the regional learning sessions and the learning centers that began pretty early on within the first 18 months or so of PACT implementation following those learning sessions. 

At the same time we have started to share our findings from that evaluation with our VISN clinical apparition partners. They were interested in what we were doing and we were interested in developing close relationships with them so we began to -- as Alison pointed out sort of try to write up quick and easy to read and digest reports of what was going on with the national PACT learning strategy so that our VISN leadership would be aware of where the problem sites were within our VISN and where the success stories were and they can think about what they wanted to do to interact with that learning strategy. Then around September of 2011 because of our close relationship with our VISN partners we learned that the chief medical officer Dr. Macpherson was planning a virtual collaborative within VISN four to support past implementation spread and the idea originally was that only the teams that had completed the learning center, center of excellence training for PACT would join this virtual collaborative and be able to continue to work on their plan to study acts and received additional training and learning and support around PACT implementation. And he wanted to have an evaluation component and we you know, said that we would like to partner with him and support that effort and he was open to that. Right around the same time the PACT learning center and the centers of excellence were suspended for about a year or so due to travel restrictions and other issues going on and so this sort of left a vacuum in several ways and I want to talk about how our partnership grew out of that. 

So the suspension of the learning centers really left CEPACT without a means to continue our evaluation of strategies for implementation and spread. We had been attending the learning sessions and then the learning centers and suddenly they stopped and we were sort of mid stream in data collection and thought, “Well how are we going to find out what’s going on now with the training PACT teams”? We also you know, because of travel issues and things like that it sort of left us without a means to have really close contact with the front line PACT team members to understand what was going on with them. We -- we were doing key contact interviews but that was not at the team level. At the same time it left a VISN 4 PACT leadership in an interesting position because they had originally conceived of this virtual collaborative as something that only teams who had already received some basic PACT training and exposure would join the virtual collaborative after receiving that training, but now they felt that they needed to have all PACT teams join the virtual collaborative right from the start because of the suspension of the learning centers. And at the same time they realized that national PACT leadership were looking upon this virtual collaborative as a potential model for web based PACT education and support meaning that any evaluation of the virtual collaborative needed to include a process evaluation to uncover implementation challenges and create a blueprint for spread to other VISN’s. 

So it was very fortuitive that we’d already started talking about an evaluation strategy but we needed to quickly kind of wrap up our efforts and think about how we could really partner on an evaluation. So objectives for the evaluation were to provide real time feedback to VISN leadership on implementation processes such as challenges and successes, especially the experiences of learners from front line teams who were attending the virtual collaborative and to identify and let them know about any variations across facilities and sites and differences between medical centers and CEPACTS and different facilities within the VISN. We also wanted to be able to understand the fidelity of implementation across sites and what effect fidelity implementation and finally we wanted to be able to contribute to the larger literature on web based and virtual learning which is a growing area and evaluating this virtual collaborative with a nice opportunity to contribute to that literature. 

So I just want to give you a quick sense of the complexity of the virtual collaborative and what we were trying to evaluate. The virtual collaborative covered the entire VISN, which is several states in the north east, mid north east. The 10 medical centers and over 40 CEPACTS, the 10 medical centers are facilities that are very different levels of complexity and there are over 350 primary care teams that are VISN serving hundreds of thousands of veteran patients. The evaluation had a number of stakeholders and constituencies as well including the VISN PACT leadership, the virtual collaborative steering committee, local coaches who identify to help with facilitation at the local level at each facility and others locally who are charged with guarding PACT implementation at their facility. In addition to obviously the front line primary care staff who were the targeted learners for this virtual collaborative. In addition the virtual collaborative itself had a lot of moving parts so there was this onsite coaching element that I referred to. There were team assignments so between virtual learning sessions online the teams were asked to complete different kinds of assignments and team tasks and also to meet between learning sessions. So they were supposed to have protected time of about one hour every two weeks per directive from the VISN chief medical officer. So there were a lot of -- there was basically an online component and an offline component and there were things coming from sent -- the central steering committee of PACT virtual collaborative but there was also local coaching going. So there were a lot of moving parts that we needed to be able to try to capture with this evaluation. 

So I want to talk a little bit about our mixed methods and partnered research approach, but again if you want more details about this you know, we’ve written this up and presented on this in other places. But basically we came up with a number of ways to capture what was going on to answer different research questions and also meet the needs of our clinical operations partners. So we developed a survey of learner experiences that was fielded twice during the first nine months of the collaborative and I think Samantha and Alison captured this really nicely that we really had to be responsive to both the concerns that VISN leadership and local leadership had about demands on primary care staff. So we made the evaluation very, very short; this survey was very short. Could be completed on survey monkey, so there was no extra sign in required in order to complete and it and we did make it an anonymous survey, which I’ll talk about in a minute presented some challenges to our findings. But it was in our discussions with our clinical operations partners that that was an important thing to do so that people would feel really open to be able to give you know, especially the open ended answers to the -- to the questions that we asked.

We also did key contact phone interviews with at least one contact of each facility; we did those at baseline about half way through the virtual collaborative and again at nine months and the point of those was to be able to understand what the general buzz was with positive and negative about the virtual collaborative and issues at the local sites with implementing the virtual collaborative, issues with technology, how the teams were meeting and try to sort of get a pulse of what was going on at each site. We also -- members of our team really participated and observed all of the planning calls and learning sessions that went on around the virtual collaborative, that we became members of the virtual collaborative steering committee where we would speak up and give immediate feedback if we had just completed the key contact interview and we knew something about an issue at a particular site. And as Samantha and Alison have talked about earlier we had to be very careful about that in terms of making sure that we didn’t say something that could be traced back to a particular person so they wouldn’t feel they had been sort of called out or their anonymity compromised. But at the same time we wanted to be able to give real time feedback to the steering committee so they could make changes to the virtual collaborative in response to any problems that were coming up. We also had a process diary where the administrative officer to the chief medical officer, Jennifer Skoko, she was really tapped with implementing aspects of this virtual collaborative including dealing with technological issues and questions from the sites. And so Anneliese Butler, part of our team, created a process diary and actually interviewed Jennifer Skoko once every other week to find out with the implementation challenges were in terms of putting the virtual collaborative into place. 

So I want to talk a little bit about how this collaboration grew and evolved over time and what [Indiscernible] came out I think for both CEPACT and for our clinical operations partners. So the collaboration happened because CEPACT, as I said had undertaken this evaluation as a national strategies for training in spite of PACT. And VISN leadership were already involved in CEPACT activities, they were on our advisory board so there were grounds for collaboration already. But a qualitative approach was really key to the making the collaboration happen and take root because members of the qualitative team had been sharing findings with the chief medical officer and been in regular contact with the chief medical officer and his administrative officer to share resources. That’s why we came -- became aware of the virtual collaborative planning and immediately jumped in to offer help and resources. So when the national training was suspended our team was able to response by expanding our evaluation of the virtual collaborative to meet both our own research needs and the needs of our clinical operations partners. The needs and resources of each partner were different and complimentary. So VISN clinical leadership had leverage with key stakeholders and access to front line primary care staff and then had the vision to put together the virtual collaborative, but they didn’t necessarily have the time or resources to take on deep and sustained evaluation because they were really dealing with creating the content and design of the virtual collaborative and dealing with technical issues and so they just really didn’t have the time to deal with a deep evaluation.

That was complimentary with the research team because we had a true interest in evaluation and time to explore some of the innovative methods that we ended up using such as the process diary and to link the literature on web based learning to what was going on with the virtual collaborative and to design data collection instruments and collect an analyze and write up the data. But we really lacked access to front line staff where an easy entrée in the workings of local efforts to implement and spread paths. So our work together was really very complimentary and grew over time as each of us were meeting each others needs. 

I’m just going to talk very briefly about some of the strengths and challenges of this approach but because I think Alison and Samantha have touched on this already. Some of the strengths are that we were able to be really nimble and reactive to changing conditions in the field. Our clinical operations partners were very open to negative findings that came from our interviews and surveys and they made changes in response to that feedback and I think that that created a loop where people who are responding -- front line staff who were responding to the surveys and key contacts who were responding to our requests for interviews were more willing to share their insights and their experiences because they saw that real change happened in response to earlier feedback they had given. And we the researchers learned to turn around our preliminary findings much more quickly and to identify actionable items for our clinical operations partners. 

Some of the challenges were that the survey we designed was -- it did facilitate trust and we got some incredibly open and really challenging comments from front line staff about their challenges of implementing paths and attending virtual collaborative learning sessions and completing assignments. But we had a limited ability to track change over time because we had made the survey anonymous so we couldn’t link responses over time to specific people or changes in attitudes. We also had some difficulty getting a really fine grain picture of what was going on at the clinic or team level even though we had these key contact interviews. It doesn’t mean that we really had a clear picture of what was going on at particular CEPACT or with a particular team. And our evaluation was not designed to link findings from our evaluation to PACT performance measures so that was one limitation or short coming of this. 

So I know I have about one more minute; I’m just going to sort of wrap up some findings. I think that really are shared across all three of our talks today about the lessons that we can learn from mixed method partnered research. One thing is that researchers must be prepared to go outside their comfort zone and it really; I think Samantha made this point but it’s really incumbent upon the researchers to go outside their comfort zone, not the clinical operations partners. So we really needed to learn how to identify actionable items even though we might be thinking about all the different nuances. Our clinical operations partners really needed to know what are things that we can change right now to support teams implementing PACT and we had to become more comfortable with being able to identify and stand behind those findings.

We found that you know, I think this is pretty clear across all three of our talks that having qualitative investigators on the team can really play a key role especially in projects such as something as diverse and spread out as PACT implementation, but those investigators need a measure of autonomy to be able to pursue unanticipated opportunities that might come up, you know part way through or one or two years into a four or five year project. We also really feel that it’s important to have close involvement of others with relevant areas of expertise at the start and I think Alison pointed that out really nicely that if you have team meetings and people are sharing what’s going on across different aspects of the demo lab then you have more of a chance of your findings being relevant across different projects and data that you collect helping inform other projects.

We learned that even though researchers and clinicians may be interested in different questions and goals all of those concerns and questions can be accommodated with careful planning and communication and that benefits to both parties really can end up extending beyond one project. So even though I highlighted one particular project here the collaboration between the VISN chief medical officer and his administrative officer and other members of the virtual collaborative steering committee have really -- we have continued to work with them in different ways and benefit from that collaboration with them in ways that go beyond this one project and I’d hope they feel that they benefited to the same extent from their partnership with us. 

So I’m actually going to finish things up by asking a poll question and -- and then I think we’ll have plenty of time for your questions about our presentations. The poll question would be what topics --

Moderator:I just wanted to let everybody know that if you’re in full screen mode, you’ll need to tap out of that to see the poll question, thank you.

Dr. Gala True:  Okay thanks. So the questions are which topics -- or the choices are which topics would you be interested in learning more about? The possible topics would be qualitative methodologies, practical, ethical and policy implications of community engaged research, human subject standards for qualitative methods, the application of medical anthropology theory to help services research and narrative methods in service research? So if you could just indicate which of those topics you’d be interested in I think that’s something that we can try to address in feature presentations.

Moderator:Thank you Dr. True, looks like most the answers have come in if you’d like to talk through those real quick, again you can see the broadcasted results if you just click out of full screen mode.

Dr. Gala True:  Okay. It looks like application of medical anthropology theory to help services research and practical, ethical and policy implications of community engaged research had quite a few votes and then also qualitative methodologies and narrative methods and health services research; so that’s very helpful for us to know. I know that Samantha is planning the talk October 16 and probably lots of future talks with great beautiful slides. So I think that would be very helpful to start thinking about what we can address going forward. So now I think we have plenty of time for questions, at least 10 minutes or so.

Moderator:Yes thank you so much. For those of you who joined us after the top of the hour you’ll notice in the upper right hand corner there is a box labeled Q & A, to submit a question or comment for any of the presenters please just type it into the box and press submit and as you do, please indicate which doctor it is for so we can get through the questions in a timely manner. The first one that came in is for Dr. Hamilton. “Did your team use a particular model for the interview summaries and matrices created in your preliminary analysis; recommended citation for these methods would be appreciated?”

Dr. Alison Hamilton:  Thank you Molly, we -- well we kind of designed our own model. We worked closely with the interview guide and with our collective knowledge and what we had heard in the interviews to develop a draft of the summary domain and then we worked with it to make it a user friendly as possible then make it fairly efficient. So we found for example that writing a summary of each interview took about 45 minutes to an hour which is pretty quick for those of you who know how long it takes to [Indiscernible] data so it wasn’t very directly lined up with the data that we selected. The overall approach I can provide some references for -- from courses that have taught other methods and working on some additional portions for it; so I can provide that to Molly after the cyber seminar for sharing with the participants.

Moderator:Thank you very much for that answer. And yes, for those of you who are wondering we do record these sessions and we will post them in any supplemental materials on our archived website. The next question is for Dr. True, what do you mean by nimble when talking about your [Indiscernible]?

Dr. Gala True:  Well that’s a good question. I think that -- I’m used to taking a long time to make changes to a research protocol or designing you know, having lots and lots of time to design a survey. And what I meant by nimble is that we had only just started to think about how to partner with our clinical operations partners to develop this evaluation of the virtual collaborative when the learning centers were suspended. And basically we had to scramble to develop you know, this survey that we thought was quite good and short and we gave it back to the virtual collaborative steering committee for feedback and they said “This is too long, this has to take five minutes or less for a primary care team members to complete”. And so then we quickly really had to really just distill it down to fewer questions and then quickly pilot it. Identify front line team members locally who we had good relationships with and ask them to pilot it for us and tell us if it made sense and if it took less than five minutes and it’s -- all of that took I think maybe seven or eight working days; that’s a really short time frame for the kind of work that I’m used to doing; so that’s just one example.

Moderator:Thank you for that reply. This question is for Dr. Solimeo, what kinds of things did you find helpful in overcoming the hesitation people often feel about reviewing things when they weren’t working well?

Dr. Samantha Solimeo:  That’s a tough question so let me make sure I’m understanding it. What are strategies to kind of overcome fear of reprisal or confidentiality breaches?

Moderator:Yeah it says if people don’t feel like they reveal things, if they’re not working well so I think it could be having to do with a little bit of reprisal but also just anonymity. But feel free to -- the submitter; feel free to write in for clarification.

Dr. Samantha Solimeo:  So I think tat one is that the biggest things we do is just to assure people of our role as communicators of information; so we take everything in aggregate when we report it and that this is actually usually working with qualitative investigators as an opportunity for people to communicate things in a way that is sort of blameless and be identified. They might not have a conduit otherwise; the other issue -- I don’t know that we touched on is that when people are participating in qualitative research it’s typically voluntary. So what that means is we often get people who are motivated to participate in an interview or a questionnaire because they are really excited in adopting this new model or because they’re very much disgruntled with it; so when we analyze our data we get very little -- middle of the road, “This model is fine” kind of response. So we get quite a bit of revealing information, sometimes it’s not usable because we can’t identify it in an appropriate way. But I haven’t really encountered that too much. 

Moderator:Thank you for that answer. This one is going back to Dr. True; if you use the same survey at different intervals or slide 26, couldn’t you track it over time?

Dr. Gala True:  Well but we didn’t track respondents so we can -- we’re able to say things. If you wanted to see our findings and we have a paper under review right now, we’re able to say if there were trends but we weren’t able to say whether a particular person changed in terms of their views. We couldn’t -- we had to treat them as separate samples basically because the responses were not linked to an individual so we weren’t tracking whether the same individuals. We could have -- we had to assume that we had two separate samples even though it’s likely that there was tons of overlap. I hope that answers the question.

Moderator:Thank you for that reply.

Dr. Gala True:  Our statistician who worked on the project can probably explain better than I did, but --

Moderator:The same person would like to know what about it being linked to clinics or teams?

Dr. Gala True:  So we -- all we did was we were able to link it to -- at the facility level but we decided not to link it at the CEPACT level because some CEPACTS have only one or two teams so we -- we know whether there were out of CEPACT and we know which medical center was their parent facility; so we were able to identify for example medical centers that had particular issues with implementation at their site. We were able to identify that and see that information, you know back to our clinical operations partners that could say “Okay this particular site is having a problem with leadership” or “This particular site is having a problem with getting protected meeting time even though it’s been mandated”. But that was the extent so basically it was at the facility level that we were able to identify patterns.

Moderator:Thank you for that reply. Also one of our attendees is wondering if they can get the findings from you.

Dr. Gala True:  Yes, so can I just have people email me directly or how do you want to handle that? We have a paper under review and Anneliese Butler gave us cyber seminar where she -- we talked about the findings so I can share either of those things. Is it easier for people to just email me directly if they want that or --

Moderator:I think that might be the best approach. If you do get flooded with a whole bunch of people then feel free to let me know and I can post it with the web site -- waiting for publication; it probably isn’t ready for public consumption.

Dr. Gala True:  Right, but the other -- we do have -- we’ve written a full report. We had -- we have multiple things that I could share, but you could email me at Jennifer.True2@VA.gov and I’d be happy to share with you some of the write up of findings.

Moderator:Wonderful and that same submitter did want to thank you for the answers to the last question, you did have useful information to provide to her and she appreciated it. Also I’d like all of you to know that several people have written in saying thank you for this webinar, it was very educational and very great. That -- that final pending question at this time. I just want to go through a couple more things. One is please do join us for part two of this presentation and that will be taking place on October 16, you will receive an email announcing that and all of our October sessions come this following morning; so we do hope to see you there, virtually see you there. And also I’d like to -- also I would like to allow our speakers to give any concluding comments so we can just go in order of operation, Samantha do you have anything you’d like to sign off with?

Dr. Samantha Solimeo:  I think I would say thank you for helping organize this and thanks to my co-presenters because I think even though the labs function independently of one another, we’ve done a very nice job of working together across the labs to try to share information. So I appreciate the continued collaboration with other labs. 

Moderator:Thank you and Alison would you like to sign off with anything?

Dr. Alison Hamilton:  Well we just [Indiscernible] comments thanking the team for helping us to get this organized and to my colleagues and the other demo labs, we’ve learned a lot from each other and I really enjoyed our collaboration and getting to share all things qualitative so it’s been great and it continues to be great and a special thanks to Samantha and Gala for always teaching me something new.

Moderator:Great, and Gala?

Dr. Gala True:  It’s hard to add to that but I’d like to thank Samantha and Alison and also I think I’m pretty amazed that people you know, at the great interest in the cyber seminar because I think for those of us who are immersed in this work we can really see the intersection between qualitative mixed method partner research and evaluating PACT implementation but I’m happy that other people can see that as well. I mean I think that the three of us would agree that we’ve all learned a lot by being involved in PACT implementation and evaluation. And that one of the things we’ve learned is how important qualitative and mixed methods are and how our work is really heightened by partnering with our clinical operations partners and so I think we were glad to get a chance to try to convey that to other people.

Moderator:Thank you. Well I also want to echo my thanks to our presenters for lending their expertise to the field and also to our attendees for joining us today and we here at cyber we’re happy to coordinate this session and we look forward to all the other PACT sessions and somebody did reference the poll question saying “Why can’t we have a cyber seminar on all those topics?”

I’m going to close the session here in just a minute and you’ll see a survey pop up on your screen so please do provide us your feedback, it helps us improve our sessions and also get those topics that you want to hear about into our catalog; so thank you again to everybody for joining us and this is Molly Kessler (ph) from Cyber saying have a wonderful day. Thanks.

Page 1 of 17