**Transcript of Cyberseminar**

**Mixed Methods Methodology**

**Incorporating Stakeholder Engagement into VA Research: What We Can Learn from the Patient-Centered Outcomes Research Institute (PCORI)?**

**Presenter: Susan Zickmund**

**December 12, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact Susan.Zickmund@va.gov.*

Moderator:It looks like we are just at the top of the hour here. I would like to introduce our presenter, Dr. Susan Zickmund serves as a health services researcher, mentor, and Co-Director of the Methods Core for the Center for Health Equity Research and Promotion. She also collaborates with her partner at Impact Oriented Research Program colleagues to improve patient experiences with VA healthcare and patient satisfaction. 

And, I would like to turn things over to Dr. Zickmund.

Dr. Zickmund:Okay, thanks very much, and thanks to everyone on the call. So, what I would like to do first off, make sure I know how to advance my screen. I know, Heidi, you have told me how to do that.

Moderator:Yeah, there are those two arrows in the lower left-hand corner. Just click on the one going to the right.

Dr. Zickmund:Excellent, thank you. Okay.

Moderator:You are welcome.

Dr. Zickmund:Thanks. So, the goal for this Cyber Seminar is that I would like to talk about the unique role of stakeholder engagement that is emerging in the PCORI applications. This is a new funding agency, new since 2012, that is part of the Affordable Care Act, and that has begun to distribute RFAs or PFAs, as they call it, focusing on comparative effectiveness research. So, I would like to share this concept of stakeholder engagement, as well as some lessons that we in VA HSR&D can learn from this initiative.

So, in terms of the audience—and the audience involves anybody who happens to be on the phone at the current moment—but it, you know, as I am writing this presentation, I am thinking about those who are interested in what I am describing as a paradigm shift in patient-centered research, which really is at the core of the PCORI mission. And, also how PCORI’s radical, or in my mind, radical redefinition of the patient’s role in research may actually alter our views on stakeholder engagement as we do our research within HSR&D.

Now, I always feel like I need to be a little bit careful with the term paradigm shift. I, I do come from a social science background and I think one should always make a little bit of fun of the notion of a paradigm shift, which is why I have a doctor saying to a patient, “I’m afraid you’ve had a paradigm shift.” Perhaps what I should really say is that I would like to share with you some suggestions from experiences as a qualitative methodologist working with about a dozen PCORI applications, most of them at the University of Pittsburgh, which is my university affiliation. However, I do think that there is something fairly profound in the definition of research that is occurring within PCORI, and that will be what I am going to focus on.

So, true to the idea embedded in this cartoon, where a speaker is saying, “First I want to give you an overview of what I will tell you over and over again during the entire presentation.” I want to make sure that you know that I will be working on describing how stakeholder engagement at PCORI constitutes a new approach, and discuss a model for stakeholder engagement that may be tailored to the kind of research that we are doing in the HSR&D.

So, I always enjoy an audience poll and I honestly do not know how familiar folks are with the PCORI funding mechanism. And, so I do have these three different options. Are you very familiar—when I say very, have you submitted one of the PI or Co-I—somewhat familiar, or not familiar at all? I think we are supposed to give that a moment there, and I can read the cartoon that says, “Okay, on my signal, we’re going to shift from pre-millennial to post-millennial thinking.” I think when you get done, you might see that PCORI is a little bit like that.

Okay, so there are, so most of you have not submitted a proposal, either as a PI or Co-I. And, it is sort of half and half between being somewhat familiar and not familiar at all. Okay, all right. That sounds, that sounds, that helps me to know a lot more about who is on the call.

One of the things that I will try to do is also talk about what PCORI is, and so I think there will be a process of making folks a little bit more familiar with it just by definition of talking about the stakeholder engagement. 

Okay. All right. So, let me start the first part, stakeholder engagement and patient-centered research and why this is the new model for health research. I think it can be helpful just to make sure we are all on the same page by providing definitions and so, let me start off with a definition of stakeholder engagement and I will say that this is the first time in any academically-oriented work that I actually wrote the word Wikipedia. I cannot say I am perfectly proud about that, and I hope that does not happen often. However, I do think it is a very good definition. So, the definition they provide of stakeholder engagement is “the process by which an organization involves people who may be affected by the decisions it makes or can influence the implementations of its decisions.” And, so really zeroing in on affected by the decision and influencing, I think is really kind of at the heart of stakeholder engagement.

I also want to make sure that I share as much of the language from PCORI itself, because I think that is helpful to investigators and anybody interested in PCORI, because it, it helps us to understand sort of how they define things. And, this is a definition of the mission that I found from one of their earlier, from a funding announcement from 2013, where they said “Our Mission: PCORI helps people make informed healthcare decisions...” Now, what I thought was interesting about this quote is that they also described who their stakeholders that they are thinking about, “...that comes from the research guided by patients, caregivers, and the broader healthcare community.” 

Also, I think it would be helpful to have a definition of patient-centered care. Reynolds defines patient-centered care as that which “focuses on the patient and the individual’s particular needs. The goal of patient-centered health care is to empower patients to become active participants in their care.” 

What I think is particularly fascinating is this notion of patient-centered research. Lee provides a definition of that which “focuses on the patient-centered approach, taking into account issues such as patient preferences and experiences.”

Now, how is this relevant to those of us who are on this call, many of whom, if not most of us being from VA HSR&D. We know, if we are from HSR&D, that the VA is the largest integrated healthcare system in the U.S. It focuses on improving the health and the well-being of a very specific population, that being the U.S. veteran. It embeds a core of health services researchers within it, and of course, it has the mission of improving healthcare system and enhancing patient care. What that means is that we are all part of a system, one system, if we are from the VA, and we have a unique link to a certain population of patients as well as a healthcare system. And, if you compare that reality to the reality of let us say investigators who are traditionally funded through let us say NIH, where you have perhaps a single funding mechanism, that the investigators are spread across many different healthcare systems. So, there is something that really connects us to certain patients and a healthcare system that I think may make us, make the PCORI work very effectively for the VA.

In addition, the VA, I think, and I think many of the, my other qualitative colleagues would say, really has been at the forefront of embracing qualitative, mixed methods research, really emphasizing the voice of the patient, the voice of the veteran, which emphasizes a patient-centered approach to research. Also, we know that the VA has, you know, embarked up centers of innovation and also creates, which really focus, which has as their focus partnering with VA operations. So, trying to sort of align research priorities with the healthcare system’s needs. And, so for this reason, it seems to me that, that the PCORI model may be particularly well-suited to our needs within the VA.

I was trying to figure out how to say that it seems that the interest in HSR&D is sort of moving in a direction of certainly patient-centered care, patient-centered research and stakeholder engagement. I was struggling like how do I know, you know, how do we sort of capture who we are as HSR&D? And, it occurred to me that there have been several really interesting pieces come out of the “Forum”, including the “Forum”, the most recent one that came out in October. In that “Forum,” there is a, you know, an interesting column in the Director’s Letter by our director, David Atkins, as well as two commentary pieces, Jeff Murawsky and Saul Weiner, talking about culture, culture change within organizations, patient-centered care, engagements. That again gives me a sense that this is, this is a direction that the VA HSR&D is moving. 

There is also, I think, a very interesting piece that came in the August “Forum” from Deputy Director of HSR&D, Sara Knight, talking about the emergence of the CREATEs and the COINs and the partnered research that, that is really part of the priority of those initiatives. And, there was an interested statement that she wrote, where she wrote, “However, the simultaneous engagement of patients, clinicians, and other stakeholders in research is potentially complex and costly.” And, it struck me that that not only is a challenge, but it is also an opportunity. And, I was reading that, it struck me that one potential solution that might come from that is to strengthen patient-centered research, using a PCORI model of stakeholder engagement.

So, what I think could be helpful now is to make a transition and really start to explain what, what I mean by exploring the new model of patient engagement within PCORI. So, I will go ahead and start with that. First of all, let me show that in my mind PCORI research is not research as usual. PCORI redefines research to best interface with the needs and values of patients and other stakeholders. 

So, PCORI is really grounded in these four master questions. They are questions that focus on how to make informed healthcare decisions, as well as answering patient-centered questions, focus on expectations, options, outcomes and decisions. And, these are such critical questions that really, I describe them as the master questions, that when I work with investigators, you know, I encourage them to try to 1) on their own make sure that their topic and their methods and their approach are really answering these questions, but even to spell that out on the proposals that they write.

Also, in terms of PCORI and stakeholder engagement, I would like to give voice to PCORI literature and just hear how it is that they describe things. And, they note that “A defining principle of PCOR is ensuring that the patient’s voice and perspective drive every step of the research process, including prioritizing the research questions, designing and conducting the research, and implementing the results in practice.”

And, I think what is critical, and I think there was some skepticism when PCORI first emerged, it is very clear that patients are the most critical stakeholders. I worked with some investigators early on, who were saying, “Well, our neurologist has diabetes,” and no, there is, they very much want to make sure that patients are part of the research process.

This is a figure that I found from PCORI materials and I thought it was very interesting in terms of viewing patients on a continuum, moving from the left as passive, how we traditional see participants in our studies as subjects, respondents, participants, all the way over to being engaged partners and researchers. And, I would like to try to explain their steps that they have taken to achieve that.

This is also an interesting figure that helps to explain the levels of engagement, and I will walk through this in my presentation as well, where this inverted triangle really talks about, shows us that they began with an engagement of stakeholders, including patient stakeholders by prioritizing the kinds of questions that they want to have answered. And, then issuing PFAs and then that funding announcements based on those topics, then asking researchers to partner with stakeholders in creating responsive proposals. And, then having a very inclusive review process.

Also, I have been reviewer, particular the first review cycle where it was still actually part of NIH, and I review locally, and I know that these are the kinds of questions that we are asked to review proposals, particularly in terms of stakeholder engagement. Does the proposal describe how patients and stakeholders were or will be identified and engaged in the research? What are the roles of patients and key stakeholders in formulating the study’s hypothesis and design, and the study’s conduct and dissemination of results? What roles do patients and stakeholders have in any planned dissemination or implementation plans? 

So, what I would like to do now is really step-by-step show how the patient engagement occurs across the research process. So, when PCORI emerged early on, I would say 2012, they had commissioned a meta-analysis where they tried to answer the question what is engagement across a research process really entail, and in this meta-analysis they really describe in terms of a preparatory stage, and execution stage and a translational stage. I will try to show a little bit more about what they may be like. 

So, first of all as I describe, they started soliciting patient input into the national agenda. And they have done this in various ways as they developed new funding priorities. So, PCORI reaches out to patient populations to solicit topics for funding. They hold public comment periods for the draft research agenda, particularly that was a big focus in the beginning. At that time, they then sent the themes to the PCORI Board of Governors that then issued the PFAs and then sort of started the priorities and the research paths in terms of having the proposals being written and then funded.

In addition, they really have been working to redefine the investigative team. There is two aspects, (B) and then I will move to (C) that I think are really at the heart of what is distinctly different about PCORI and really revolutionary in many ways. So, to be responsive to PCORI’s mission, a study should—and I really argue this—a study really needs to have a patient investigator. I think the acceptance of having a team without a patient, I think that period of time has kind of moved on and now one needs to have a patient. Also, involving patients in the research team is meant to profoundly change how a study is developed. And, I was just sort of reminded of this cartoon from *The New Yorker* where you have a very different type of person at the table for this board meeting, who then says, “May I offer a different scenario?” But, I think patients are designed to make us really think differently about the process of doing research. 

So, patient roles, they can be a PI. PCORI has a multiple PI mechanism among several proposals and grants, funded proposals where there are often one or two investigators who are traditional researchers, along with a patient or patient advocates. They can be a co-investigator. Certainly, one can have an advisory board with multiple patients, or you can have advisory board completely made up of patients. But, I do think it is important to make sure that they are part of a traditional team.

So, I am going to sharing snippets from a PCORI sample engagement plan, that if anyone is interested in submitting a PCORI, I think it would be really helpful to take a look at this. I am going to, I will not read all of the different sentences, but when they talk about the investigator from, this is from an actual grant proposal. Someone says that “stakeholders and patients will be compensated co-investigators. With their commitment and obvious dedication, the stakeholders are written into the grant as co-investigators will be compensated. They have written letters of support and submitted biosketches.” And, so they really are a member of the team. And, then, and just as an example of what a patient advisory panel would look like, and again, this website has all of that.

So, my personal experience as a methodologist on these various team meetings, I can share that having a patient on the team can really ground the discussion in real world concerns. And, I truly believe that having a patient helps to sort of break or at least allow us to reevaluate the research bubble that we all bring with us, as we sit down in a meeting where individuals will start talking about, you know, what it is that is important in terms of, you know, getting funded and what is a priority. And, then patients can, can say that this is something we really care about, and I have seen proposals change and really be improved by the process of having that voice of the patient as part of the team, not someone who is added on or someone who was there for the purposes of collecting data from. So, I think it is an interesting model and it is one that I feel personally, I have grown a lot from.

I do think if someone is interested in adding patients to a team, one should be aware of some of the logistics. If a patient is a co-PI, there is an NIH Commons ID that one, that is required as a co-investigator or as a PI. There is a Federalwide Assurance, or an FWA number that is required. NIH-style biosketch, PCORI has their own, but it is based on NIH. Letters of support. It is important to pay the patient, to have a budget, add them into a budget justification. And, some of the logistics, I think are important to keep in mind, or to be aware of before putting in a PCORI application.

Now, I mentioned that (C) is also one of those areas that I think really redefines the research endeavor and I am defining it as redefining the development of topics and specific aims. One would need to show that the research is responsive to patient and other stakeholder needs, and to make sure that it is not superimposed upon them. So, it needs to answer these questions. What roles have patients played in the selection of the research topic? What roles have they played in terms of the specific aims, and the study outcomes?

So, things that, that sort of emerge from that, it is very important to have preliminary qualitative work to show that the topic is important to patients. Also, to show that the specific aims were given to stakeholders and that they had a hand in revising them. Show that the patients chose those outcomes, and then to track all of that throughout the grant.

So, I will give you some examples, and then I will talk a little bit more about this. So, for example, this is from the PCORI sample engagement plans in terms of selection of the research topics. This segment, and I, it is very long. I am taking just little pieces of it here. “The stakeholders and patients have been involved from the conceptualization.” It states that “When the funding announcement was released” the PI was very interested about, in this topic, and then thought about the seniors in her clinic. She contacted a particular patient who receives care and that how it is that that relationship with that patient and experiences with patients really shaped the research topic. Again, it is fairly lengthy, so I am just giving you a flavor of it.

And, also patients and other stakeholder engagement in terms of the proposal. Again, reading from the same PCORI sample engagement plans, “Stakeholders and patients have been involved in the writing of the grant proposal. Each of the stakeholders above have written a part of the research plan. Numerous phone calls and” dah, dah, dah. The PI then performed editorial work to make the grant proposal cohesive and provide guidance. This kind of language should, should share that there is really a, you know, a central role that the patients are playing, again, not as individuals we collect data from, but being a, you know, part and parcel of the research team.

I wanted to share this table. This is from a funded PCORI proposal that I am the methodologist on. It has the investigators of Dr. Schuster and Reynolds and the patient advocate, Ms. Carney, as the three PIs. It comes from the UPMC Center for High Value Healthcare. Dr. Jane Cogan shared this table with me, which I appreciate. Basically, what it shows, what, the fact that they had done a series of preliminary studies and pilots related to the topic, and they have engaged patients over this long period of time. Then, the PFA was released and then what they did to be able to prepare for the LOI, and then they submitted the LOI and then how they, in detail, very specifically weeks-by-weeks, in terms of what they did in a preparation for actually putting that proposal together, all the while working with their stakeholders.

So, when I work with investigators, if they have time, I will give two scenarios, they have time, they do not have time. If they do have time, I really encourage investigators to have, say either if they have had research funding, which would be wonderful. If they do not, then to shoot for perhaps a small pilot where they can start to collect some engagement data from, or information from people who are likely to participate in their study, try to learn more from the participants, try to see if you can disseminate that in a published form before actually submitting a PCORI. So, having lain down a groundwork for demonstrating that you have been engaging with patients. From there, also start talking about what topics are important, how to refine the topics, refining which outcomes are important. In some process, bring what you anticipate to be a specific aims to that group, so you can develop almost using a participatory action research type of methodology, so that everybody is coming to the table in preparation for submitting a PCORI. 

Other individuals come to me and they say, “I need to submit this.” I run a methodology core here at CHERP in the VA, but I also do the same at the University of Pittsburgh. And, so what we have been doing is working with investigators and trying to help them put together focus groups. Now, these are not, not necessarily your typical research focus groups. It is more preparatory to research, where they bring their topics, their specific aims, and they start working with, with groups who are similar to the individuals that they might have as participants in their study, and really try to dialogue with them about the ideal design, the ideal specific aims, outcomes for their study. So, it is just that this is a bit of a different approach and it usually, my mind, it takes more time to prepare a PCORI, given the ground work that needs to be done.

And, I think I have shared most of those experiences already. I will say that it seems to me that I, that there is more qualitative and mixed methods data that is incorporated into these kinds of proposals given the nature of the stakeholder engagement as well as the patient centeredness of it.

Then we move on to the data collection, which is the heart of most studies. I guess we could call it the heart. I would say that the data collection for a PCORI is in line with what I have seen coming from an NIH or a VA style merit proposal. It seems that there is more, more of an emphasis on tailoring an intervention or decision aids to the unique needs of the study participants, often sort of closing the loop by having a series of interviews or focus groups with those participants at the end of the intervention. Perhaps, maybe tying up the observations with a Delphi panel, some type of panel that takes all the disparate thoughts and ideas and then starts to bring them down in a way of advocating for ways of disseminating the information or perhaps tailoring it for future, a future intervention or implementing that intervention for a future study. So, more qualitative approaches, but in general not nearly as revolutionary as the, the (B) and (C) headings.

I can share with you that, that some individuals that certainly for this PCORI sample engagement plan actually have patients involved in the data collection. So, this individual said “stakeholders and patients plan to be involved during all 3 years as is appropriate” and then mentions that “They will be instrumental in recruiting seniors and caregivers from their organizations.” So, I think it is important to make sure that it is not that we got funded and now the voice of the patient is gone from the proposal, that you continue that dialogue all the way throughout the proposal. 

And, then also in dissemination. It is important the patients need to be a part of the dissemination plan, often combined with a discussion of an organization that a patient is affiliated with. What I typically see, and you will see this in the example that I am about to read, it seems that what they are more likely to do is to say that this is our patient, and this patient is associated with a particular group, and together we are going to work with them in disseminating our materials as opposed to assuming that a single patient who is unaffiliated could be able to do what is obviously very difficult work. 

So, this example has “Stakeholders and patients will be active in the dissemination of this particular tool, completion of the RCT, the stakeholders will meet.” “...we will disseminate the tools to national seniors,” and I think the last sentence is probably the most applicable to this point, which is “We also plan to disseminate through national nursing associations of which this one patient is a current board member.” So, the more that the patient is affiliated, the more they can sort of combine the group that the patient belongs to as well as their own efforts in this dissemination.

So, the last element that is, it is also very unique, is placing patients on the review panels. So, PCORI incorporates patients into the review process. Patients and other stakeholders are assigned to each of the proposals and provide their own specific review. Now, when I was on the national panel, it was, like I said, it was the first year the PCORI was, the first wave of grant proposals. It was still being sort of orchestrated through NIH. We had maybe out of 30 largely health services researchers, we had maybe three patient or patient advocates. They did maybe three proposals each week. We had a lot of proposals that we were, that at least were on our docket before the triage process. My understanding from my colleagues who are reviewers now, that, is that you will have two scientific reviewers, you will have a patient reviewer and then potentially other stakeholder at the table reviewing all the proposals. The patients are not necessarily focused on the scoring of the scientific rigor of the methods as much as focusing on the appropriateness of the topics, stakeholder engagement, patient-centeredness. But, that their voices are really being heard throughout the sessions, and that is even an expansion of what I had experienced when it first began. 

So, what I wanted share here was a, sort of a visualization of everything that I just described, so that we can kind of wrap our minds around what this process looks like. So, reading from left to right, PCORI began and they always reinitiate this process of having a patient-centered research agenda, making sure that patient and other stakeholders have an opportunity to give their voice in terms of the funding announcements. From there, it goes to the individual team where you have the principle investigator or co-investigator who is a patient as well as advisory panel that is likely to involve patients or maybe completely patients. Then, moving on to what we described already as having patient engagement on the research topic, the specific aims and as well as on the outcomes. Then, we move on to the data collection, which again is more traditional. However, making sure the patients are, that whatever the intervention is or the device, etc., that there is really a tailoring it to the unique needs as well as some kind of patient or subject, participants of, feedback at the end. And, then potentially having patients even involved in data collection. I mean, that, I think is still work in progress in terms of whether or not that it will occur for all proposals or not. And, then finally having patients involved in the dissemination of the research. And, all of this occurring within the context of having a grant review process that involves patient stakeholders.

So, this is sort of the description of the PCORI process in terms of stakeholder engagement. What I would very much like to do now is to sort of turn and start contemplating, you know, how applicable is the PCORI model to the VA HSR&D. And, what I am going to do is sort of add questions for speculations. I do not necessarily have the answers myself, and certainly not making a call for certain kinds of changes, but I think the kinds of questions that I think we need to ask, I would like to bring out. And, then I would like to leave time for us to try to have as much of a robust discussion as we can.

So, it strikes me that there are larger questions, such as should the VA HSR&D embrace part or all of the PCORI model? And, if so, what parts should we focus on, should be focused on first? I mean, I am struck by a, you know, a really interesting portion of the column written by Sarah Knight in the August “Forum” when she was talking about the CREATEs and the COINs and our relationships to our research partners and engaging with them, and noted that “The development of these and other principles of stakeholder engagement and of new conceptual frameworks for partnered research are essential as diverse stakeholders participate in research processes.” And, so, you know, it seems that we need to move somewhere, and it may be toward a PCORI model. Dr. Knight did talk about all of this occurring within the context of, you know, PCORI emerging on the horizon in terms of medical research, not that it was directly driving decisions within the VA. So, it seems that, that there are central tenets, sort of paths that are moving in a parallel direction.

So, in terms of VA engagement, you know, engagement can potentially begin by, you know, expanding stakeholder engagement. Oh, and one thought that I have is, you know, potentially we could start with making sure that our operation partners, much like patients within PCORI, we start making certain that they have a, you know, have their voice incorporated into all the different levels of the research process. I mean, this is not as fully embracing the PCORI model; this is much more limited. But, there is a new study that is looking at alcohol, care for veterans who are alcohol dependent, that really uses sort of a stakeholder model by really focusing on interviews at the beginning and then at the end, with key administrators, clinical and operations stakeholders across the length of the study, particularly as it tries to understand and sort of map out changes that might be emerging in the context of the Affordable Care Act.

So, you know, that starts to move us in the direction of really focusing on our operation partners, but perhaps the goal is to move beyond investigators to operations and patients. If so, would that be an immediate switch to a PCORI model? Should it be a gradual model of expanding inclusion of stakeholders and HSR&D research? If so, you know, should that be a tiered approach? Would we want to start off making sure that we have our operation partners, and then also patients in terms of determining research funding priorities, data collection, you know, are basically there already. Dissemination teams, typically operation partners are going to be part of the dissemination for, for query projects certainly, also for HSR&D merit review awards. What about topics and specific aims, and outcomes? Do we really want to try to make sure we are dialoging with the operation and partners as well? And, what about the review process? Is that somewhere down the line? Should we consider having an operations partner sitting at the table for the, the traditional SMRB or other review boards for VA projects, adding patients into that project, or into that process? So, just sort of describing where this, where this process of embracing a PCORI model could take us, not that necessarily it should, but it is opening the door for discussion.

And, if it did, then what we would find is that, or eventually down the road, if we fully embraced the PCORI model, what we might be looking at is a VA-based approach to stakeholder engagement that looks very similar to the PCORI model. Except that it, what it would do would sort of combine the patient, operations, as well as investigator-centered approaches to all of these aspects, the team, advisory panels, more input from outside, research topics, specific aims, outcomes, data collection, and certainly also for dissemination. 

So, based on that, and again, I am trying to make sure that I leave sufficient time for us to use our whiteboard and then also have a conversation about this, I can conclude by saying PCORI applications place patients and stakeholders at the center of the research process in a way that is new to many investigators. The VA may consider this an approach to best increase patient and operations-centered research with VA HSR&D. And, I learned that there is the whiteboard approach and I have a question for, for this. It would be very interesting hearing should HSR&D use the PCORI model for engagement? And, if yes, what should they incorporate first? My first time using the whiteboard, so I am, you know, give me a minute, that hopefully this as well as opening it up to the questions or however, however Heidi does it, may enable us to have more of a conversation about this. 

Moderator:So, I just opened up the whiteboard in here and you should all at the top of your screen, at the top left there is a capital ‘T’. Just click on that ‘T’ and click down in the whiteboard and you will be able to type in free text. We will not be able to see what you typed until you hit that enter button, so it will take a few seconds before we see anything here. But, that is how we get it out there.

Dr. Zickmund:Okay, and then, Heidi, as I see things should I comment on them, is that how the speakers have been using the whiteboard in the past?

Moderator:A lot of times they have been using it that way, yes.

Dr. Zickmund:Okay, great, great. And, does everyone see the same whiteboard that I do?

Moderator:Yeah, everyone sees it.

Dr. Zickmund:Okay, all right. So, we see a yes, operations and local staff, okay. Ooh.

Moderator:Oh, that is okay, I move them around.

Dr. Zickmund:Okay, excellent. Do we need more evidence of the research produced by the approach is better than the standard research? I think that is an excellent question. I think part of it may also have to do with a philosophy that if you think, yeah, it may be difficult to do a head-to-head comparison, one PCORI is really focused on comparative effectiveness research. And, we incorporate that, so I answer yes, but I also wonder how. There would be challenges in trying to make that decision based upon that. Encourage patient involvement and research, on research teams, okay. Inclusion on research team as co-investigator staff and patients. Yes, technical assistance to incorporating patients as co-investigators. I agree, that how to do this within the VA, the structures for payment, for example, I am not, I have never done this myself, and I do not know how to do it, and I think it could be very interesting to have a sense of, you know, the feasibility. What are the mechanisms for being able to do this.

If, okay, yes, funding priorities and aims development, outcomes that matter to patients, okay. So, it seems to be some receptivity to this. Let us see, anything else here? If patients choose the outcomes of the research, is a certain outcome implied? That is an interesting question. I think it probably would depend on the individual, it would be the individual study. I mean, I have been on projects before where the reviewer, an excellent wonderful projects, beautifully written grants, where the reviewers came back and said, “How do we know that patients want these additional services?” And, it, these were the kinds of questions that would have been really helpful to be able to have an answer for. So, it is, I think that the research scientist on the projects need to ask these kinds of questions. I think engagement means you are at the table, you do not necessarily automatically take results without discussing it amongst the larger team.

Okay, choose the outcomes, before moving to a PCORI do, okay, sorry, there, are they in a certain order, Heidi?

Moderator:I wish they were, but I am just trying to keep them not have on top of each other.

Dr. Zickmund:That is fine. What about conflicts of interest, so the patients with their on board the research team. You know, I had, on the various, the various projects that I have been on, I have not seen that happen. Now, we also have to recognize that there are different kinds of patients. There are patients who are parts of, you know, almost advocacy groups or they are working for certain services and so have a lot of experience in working with research. So, I think part of it, I also think it is, would be really important to choose a patient who is verbal, because I have worked on, with a few teams where the patients tended to defer and I think you want someone and others where the patients are very vocal and have said a great deal. And, so I think you want to have patients who, you know, are willing to challenge investigators.

Should, sorry I am trying to think, outcomes that matter, encourage, start with the advisory boards, because they are used flexibly to inform projects in a variety of ways. That makes a lot of sense. I mean, one of the things unique about PCORI versus VA HSR&D is it started this way. And, we of course, have existed for quite a long time, and so to, it is more difficult to throw a switch and suddenly start doing things very different, and when there is a historical pattern. So, it is sometimes easier to be, to be a little bit more bold when you are starting out.

Yes, but I wonder if there is enough recognition, the time it takes in terms of effort, in terms of duration to really put the patient in the center. The CREAT projects provide a first toe in the water to gain experience, and from this we are finding out little we know about how to do this. Approaches for doing this are needed, more examples. Again, I agree. Okay, all right. Oh, interesting. I know that one of my mentees has developed a post-deployment interest group of veterans. And, it takes a tremendous amount of time. It is very worthwhile and it is something that she is incredibly excited about. But, we have to recognize that this kind of research takes time, it takes energy to be able to do. 

I see something written here, are there HIPAA concerns with having patients on the review boards, of patient-centered research. Hmm. I, I do not have an answer to that. Maybe when I take questions, if anybody has an answer and wants to send a question to Heidi, maybe we can get an answer to that. I mean, are there HIPAA concerns, are there not HIPAA concerns for researchers when we are on the boards, because we have all been credentialed? Can patients be credentialed in the same ways that we have in terms of research training, which my sense is that if you are going to have a patient as a member of the team, they need to do the full range of IRB and ethical trainings in order to do this. So, I do not know, I honestly do not know an answer to that.

Here is a new one. As good as this is, it is revolutionary to most reviewers, so going slow will be necessary. I, again, I understand that, and truly in the last part of the presentation, and I do not mean to be hedging, I honestly do not know. I think it is a fascinating model, and I, and having been on as many teams as I have been on, as many conference calls, I see a real value to it. But, how to superimpose that, or how to, how to broaden that model or extend that model to the VA, honestly I am not quite as sure. 

All right, every time I see a little squiggle, I am going to presume it is something I should be reading. Okay, all right. I think I read that. Heidi, are there more questions that are coming?

Moderator:We have actually a lot of questions in the Q&A screen, but we are not getting a lot. I think the whiteboard right now is so full that we cannot take much more on there. And, that is why that is slowing down.

Dr. Zickmund:That is fine, that is just fine. So, should we transition to questions?

Moderator:That would probably be a good idea at this point.

Dr. Zickmund:Okay, all right, excellent.

Moderator:Now, I just want to save the whiteboard here quick, before we get started, so I have that information. Okay. The first question we have here, how do we empower patients to develop hypothesis? It seems like a real challenge.

Dr. Zickmund:I, I agree. And, I think the hardest thing that I have seen so far is, is involving patients in specific aims, because, you know, when we learn to read and write, we do not start with specific aims. There is something unique to the research world with specific aims. Topics we understand, outcomes, I think as long as it is articulated in a way that makes sense, is something that—I can go the next one in case people need to see my email address—I think that is something that, that, you know, you can sit down and have a focus group discussion. I think, what I have seen work successfully, 1) it has worked fairly successfully when we have had the PIs come in and either we have been running discussions, or they run the discussion, but they are in the room, and describe what their interest is and genuinely making sure that they are asking the right kind of questions. It has not been a situation where, so far in my experience, where they walk into the room with like a blank sheet of paper and indicated that they are interested in walking out with specific aims. And, they have not been asking specifically what the hypothesis is. They have been asking more, you know, are we asking the right kind of questions, and they walked in with, you know, specific aims written in such a way that they are, they can be understood by a lay population. Like, these, are these the right kind of questions, you know, we sense that it might be going in this direction. And, then having a conversation about that. We have been typically recording and we might transcribe them, and then sit down with the investigator and make sure that the feedback that has come through this process is most effectively captured so that they can incorporate that into the specific aims.

So, that is what I have seen as most effective. And, then when they come back to the team with a patient or, you know, potentially multiple patients on the extended team, they can have a conversation with folks who know the topic more, because, you know, they have been huddling for the last four or five months on that topic. So, that is what I have seen work most effectively.

Moderator:Great. Thank you. The next question here, should the patient investigator be a full-time member of the team?

Dr. Zickmund:A full-time member. So, let me use language of PI, again because of the dual, the multiple PI mechanism, which is different of course in the VA, or the Co-I. I mean, they really are on for 10 or, think percent effort, 10, 20%. They are compensated at that level. So, it is not that they are like a full-time research assistant where their entire job is dedicated to the study, because really at that point they stop being a patient, right. They become, their world becomes research much like it is for all of us. So, I would say 10% of their time, which I believe is 20% is one day, so it is half a day a week, and then usually they are coming to the main meetings. So, and I, you know, an hour a week, certainly more during certain periods of time such as when the grant is being written, or if they are being asked to, you know, help with interview scripts, or whatever task they have been, their input is being requested, they may be doing more. So, I do not see it as full-time.

Moderator:Okay. We have an additional question on the patient investigators. Are they expected to be researchers who happen to have the medical condition of interest, or a patient who has no research experience or training?

Dr. Zickmund:So, I have worked, it is the latter. I mean, you can, one can do anything, obviously, but when I have worked with groups on PCORI in the beginning there was more of the well, this is our patient representative, because he is a neurologist who happens to have diabetes. What I have seen is a movement toward embracing patients, and I like, will I also add, patients are, also patient advocates, often they are both, someone who is, you know, deeply committed to improving the life of individuals with serious mental illness, may in fact have had experiences or family members who have had that experience, and then they have, you know, they have chosen a career as being a patient advocate. So, I think it is important that in choosing, you are choosing someone because of their experiences and what they can bring to the table as a patient, rather than almost it could like you are trying to skirt around the, it is not the rules, but the mission of, you know, investing patients with, sort of empowering patients if you just take someone who would look like they are part of the research team anyway, and calling them the patient representative. That is my thought, my suggestion. 

Moderator:Great. Thank you. The next question here, in a proposal that would be done in a dialysis clinic to promote physical activity, I want to include a patient coach, who was trained and motivated to help encourage other patients. There is always a concern about HIPAA issues in having a patient interact with other patients in a formal way. They obviously do interact informally all the time. Do you have any suggestions or insights?

Dr. Zickmund:I mean, obviously, that is the second question on HIPAA, and I do not know the answer to that. Also, if you have somebody whose function is collecting, collecting data, I mean, you could potentially embrace the PCORI model and not have patients collect the data. So, I do not know, I do not know if is it the problem that they might be in the room, as in adding them into the review process versus having them collect data. It would seem like they, if you did trainings—again, I am not IRB expert, unfortunately. If they did trainings, I would wonder why they could not be allowed to be a part of the IRB, to have the same kind of waivers or, you know, accreditation that all of us would have. So, I do not necessarily have an answer to that. I do not see why they could not certainly be in the room as reviewers, particularly given the fact that it is happening with PCORI. I mean, I have, I sat next to patients, patient representatives, so unless there is some unique VA restriction, it would seem like that is a model that could potentially work. There might be other issues with it, but in terms of confidentiality in HIPAA, that seems like it can be worked around. But, it sounds like a frustrating situation to be having to battle, and I am sorry that person is having to deal with it.

Moderator:They sent in a follow-up question that you touched on a little bit, but they are wondering does all of this need an IRB approval?

Dr. Zickmund:I am sure that you would need to add—I think you definitely need, from what I have seen from the PCORI materials, you need to make sure that a patient on the team has done the IRB trainings, everything that you would need to be able to go through, you know, ethical conduct, you know, ethical conduct of research studies that we all have to do to be a part of a grant. I am sure they would have to do that, but, you know, otherwise it seems to me that should be able to, should be able to do that.

Moderator:Okay, great. Thank you. The next question that we have here, do you or anyone else on the call have any experience engaging home care patients in research? I expect it may be more challenging, given patients are geographical dispersed and may be harder to reach. How do you find patients to actively participate in research?

Dr. Zickmund:I do not have experience. I mean this is a more traditional research question, which is fine. Ways of reaching out, going through providers who might suggest patients. I know within the VA there is a mechanism for, you know, once you have the approval to be able to approach a patient, you can certainly send out letters. The VA has a mechanism for an oral informed consent. If one wanted, if the mobility was an issue to collect data over the telephone. Certainly, one could always drive and go to homes. But, a letter to a patient would seem to be something that IRB would allow, or at least going through the provider and then be allowed in conjunction with that provider to send a letter, which would allow you to be able to reach out to patients and then collect the data that seemed most feasible, as well as appropriate for their lifestyle needs, seems like something that could work.

Moderator:Great. Thank you. The next question here, what is SMI?

Dr. Zickmund:SMI. Serious mental illness, I assume, I am not sure, that was on the, it might have been on one of the slides, but probably serious mental illness is what SMI was most likely referring to.

Moderator:Okay, great. Thank you.

Dr. Zickmund:Sure.

Moderator:Next question, can advisory board members be rotated out if they wish to do so with a plan in place from the beginning of the project?

Dr. Zickmund:I, I would assume yes. I think all of that would need to be organized and described in the grant process. However, yes, I think that should be something that is possible.

Moderator:Yep. They sent in a follow-up here. We are working with a higher risk population that may not want to stay on the project the full three years.

Dr. Zickmund:Yeah, I understand. I would presume that that is feasible.

Moderator:Okay, great. The next question, how is patient defined in PCORI research?

Dr. Zickmund:Oh, my goodness. It is an excellent question. I believe that it would be, I do not want to Google as I am online here. I can operationally describe how I am seeing patient defined in the various projects that I am on, and it has run the gamut from somebody who is a healthcare, you know, either healthcare provider or a researcher, often both, who happens to also be a patient, to a patient advocate. Certainly, my one experience of being on the PCORI review panel involved having individuals who were, you know, probably having more of an advocacy role all the way to having, I mean, clearly individuals who are patients. And, someone who the provider, who was, you know, perhaps the PI or somebody who is on the main team said, you know, this is a really gifted, high energy, motivated patient. They really want to care about this issue, and they have invited them. Oftentimes, there was a pregnancy proposal that I was on, and we wanted to make sure coming under the disparities portfolio, and we wanted to make sure that we had our representative, who, you know, was similar to the people we would be collecting data from, very important, who was interested in the subject and was going to tell us, you know, was not going to back down when we were going off in our little research bubble and talking about, you know, making sure the intervention arms were balanced, who was actually also going to provide input that we needed to hear. So, I, from operational perspective of my experiences, I think it is a pretty wide definition.

Moderator:Okay, great. Thank you. We are almost at the top of the hour here. I have five or six pending questions here. Susan, are you able to stay on a little bit longer?

Dr. Zickmund:Absolutely, absolutely.

Moderator:Okay, perfect. For the audience, I know a lot of you are only able to stay for an hour. We are recording this call and we will be sending the link to the archive out so you will, if you want to check on these questions later, we will have that available for you out here. The next question that we have is the translational phase included in the actual funded study period?

Dr. Zickmund:My sense is that the last portion of the—the PCORI’s are three years, much like the typical merit award. And, so as a result, you know, there is that period of time when you are supposed to be finishing up the data and making sure that there is a dissemination plan in place. So, it should be occurring during the course of that proposal.

Moderator:Okay, great. Thank you. The next question, we will be working with a population that has low health literacy. Is this an issue?

Dr. Zickmund:I think if you are working with a population with unique needs, and I think it is particularly an issue, I think it is particularly important to make sure that you have a patient as a representative on the team, to make sure that you are not making assumptions about language, about approaches. It is amazing how many assumptions we in research make once we, you know, finish those terminal degrees and get through a career development award. You know, we are just, we become different kinds of people, and that is okay. I mean, our skillsets are needed, but having the reality check of someone at the table, I think it is really important. So, I think it is especially important under those circumstances.

Moderator:Great. Thank you. The next question, do you patient PIs appreciate their responsibility for projects as a whole? They must respond to adverse event or non-compliance issues, data collection errors by patients. What protections are in place for patient PIs?

Dr. Zickmund:You know, my sense at the current moment is that the day-to-day running of the study falls more typically on the research team and those investigators, and that the patients are coming in and giving their viewpoints at the, you know, all the meetings and at everything that would really require their unique perspectives. Those kinds of more nitty gritty day-to-day operations, I have not seen patients having to make those kinds of decisions.

Moderator:Okay, great. Thank you. The next question, how much do you pay your co-investigator patients?

Dr. Zickmund:So, that is an excellent question, and if I had thought to look at some of the budgets, I would, I would see that. None of these PCORIs are my projects and so I have not looked at that. But, you know, it may be that if you are looking at someone’s, maybe income, trying to figure out what 10, 20% of their income might be. I honestly do not know, but I know it is under revision, so I am not able at the current moment to share. But, in our university, there is a comparative effectiveness research center and they, they have addressed these kinds of issues. So, it may also be something that is listed under a FAQ under PCORI. So, I think there are places to find that information, I just, I do not know myself.

Moderator:Okay, thank you. The next question, do you seek ethics approval for your non-formal focus groups that you mentioned?

Dr. Zickmund:So, we always talk to our IRB. No, these are all the university side, and the view of the university side is that when it is preparatory to research, where we are not collecting names of individuals, we are not releasing this for publication, they have not considered this to be something that we need to do anything, because the other investigators were the methodologists. They will mention it as part of their overarching, overarching proposal. They will say that this is going to occur, but they have not been asked to do much more specific information than that. Our IRB on the University of Pittsburgh side actually considers qualitative studies in general to be exempt. So, they do not have a lot of requests in terms of very detailed information. Another university would certainly be different, and it would be different in the VA. We would need, I do not believe that this would be able to be functioning as preparatory to research, although, I think that would be an individual discussion with each IRB chair at each site.

Moderator:Okay, great. Thank you. The next question, are there good references for how to negotiate outside/insider perspective in research where patient and non-patient researchers are co-PIs?

Dr. Zickmund:I, there may be. I have not run across anything like that. You know, I was working to put together some literature on patient stakeholder engagement for this talk. Those terms are also very widely, you know, it is hard to zero in on exactly what literature you are looking for. There is not nearly as much as I would like out there at the current moment, and I certainly have not seen on that. But, that does not mean that with a more fine-toothed comb one could not find it.

Moderator:Great, thank you. And, this is our last question. In the focus groups used for preliminary research, is that something that needs IRB approval, and do you have any contacts who might be able to provide guidance on the HIPAA IRB issues?

Dr. Zickmund:You know, again, I, certainly in the university it is not an issue of HIPAA, and because all of the qualitative studies are exempt. I am sure that you would need something on the VA side, we always do, whether it could be preparatory to research I am unclear. If you had investigators doing, collecting the data and the PI in the room, I guess I struggle to figure out why HIPAA would be more involved than any normal focus group where you are focusing on making sure that you have the informed consent signed and the, if the, in the VA the document to make sure that you have, you know, permission to record voice and voice and photographs. But, I do not know why HIPAA would be any more of a unique concern than it would normally. So, I am not, I would not be unduly worried about that.

Moderator:Okay, fantastic. And, that does conclude the questions we have here. Susan, did you want to make any final remarks before we close things out here?

Dr. Zickmund:No, I very much liked the whiteboard. It is very interesting to see the thoughts that folks bring to the table, particularly for something that is speculative. I have never done as speculative a talk as this one, and it is an interesting conversation. These are, you know, really, you know, what do we do with PCORI model? I am not quite sure, and so it was enjoyable to have a chance to discuss this.

Moderator:Fantastic. We love using the whiteboard for that. It is kind of like having a conversation without having to open up the phone lines, so.

Dr. Zickmund:Absolutely.

Moderator:It works nicely. For the audience, we want to thank everyone for joining us today. If you can hold on for just a minute or two, I am going to put up my feedback form. If you could take a moment to fill that out, we would really appreciate. Susan, thank you so much for taking the time to prepare and present for today’s Cyber Seminar, we really appreciate that. Thank you everyone for joining us for today’s HSR&D Cyber Seminar and we hope to see you at a future session. Thank you.

[End of audio]

Page 1 of 17