VIReC Clinical Informatics Seminar- 13 -Department of Veteran Affairs

March 20, 2012

**Department of Veteran Affairs**

**VIReC Clinical Informatics Seminar**

**A Novel Use of Electronic Health Record Data To Inform Patient and Clinician Treatment Choices**

**March 20, 2012**

Moderator:  [Inaudible].  With these technical issues, usually we have a great technical performance that whatever is going on today.  John Crilly is our speaker today.  John Crilly, PhD, MPH, MSW.   He’s a health sciences researcher in the VISN 16 MIRECC, New Orleans research service.  He’s been with the VA since 2007, and he was a recipient of one of the first VA innovation awards under the Greenfield program, and he’s presenting on that today.  Dr. Crilly is also on the faculty of Tulane University in the department of Psychiatry.  So, John, take it away.  I hope maybe the audio improves—magically, randomly, constantly.  Thank you.  

Dr. Crilly:  Alright.  Margaret and Heidi, thanks very much and hello everyone.  Welcome to this MIRECC session.  I’m going to be talking, as the intro said, about this project that we did over the past few years.  It actually just ended a few months back—maybe it was a little more than that, and talking about our subsequent work to kind of move it forward.  I came up with this title back in September or October, and this is really sort of the title of our paper that I was writing for this.  But, I’ve gotten some feedback since that might be better to talk about this as patient centered outcomes more so than patient-clinician treatment choices, but you’ll see as I go through how these things tie together.  

I’ve got this slide in twice because this is how important I feel my team is for helping to make this all happen.  Diane Neimann is up in VISN 2.  She’s the CIO up there, and her team was just instrumental in making this happen.  You can’t do anything without your CIO, especially anything with data and business.  So, without them, this would not have happened.  

Jim McCain was my innovations team contact; he was there all the time.  He’s real knowledgeable, real supportive, and very helpful.  Ashley Byrd and his team at ICF were the ones that did the heavy lifting on the programming.   They’re just wonderful and you’ll see their work later on.  Joe Linda, the lead vendor for the programming area, and when I got this award is was in the VISN 2 center for suicide prevention, and really the focus of this was to look at trying to identify treatment and better treatments to then improve outcomes around suicide-type indications.  Down here in VISN 16 MIRECC in New Orleans, great place to be, very supportive.  Forward thinking MIRECC is a great place.  

I’m going to jump right to the objectives that we published around this time, and really what I want to do is describe not necessarily start-to-finish but kind of give you the guts of how the program developed, what we were looking to do, and what we came up with, and what were going to do in the future.  So I’ll kind of tell you a bit about what it was like to be one of the early innovation projects at the VA, and any great idea or any idea can simply be an idea you think of on your way to work, but for this one we’ve worked on this for a while and you have to have a theoretical, underlying model here.  So, one of the ones we used was comparative effectiveness research, and I’ll talk a little bit about that model and how it can help bring that to the frontlines of this project.  I’ll talk about the data and the relative realtime data we tried to use and what we came up with.  And then finally, what we’re thinking about and how to expand this tool for other uses later on.  

The VA started their first program in 2008.  There was plenty of planning before that.  The first stage was this Greenfield Incubation stage.  What they were looking to do was to encourage innovation from where people kind of knew what they were talking about—people weren’t on the frontlines.  I’ve worked as a field chief for about twenty years, I worked in state hospital systems in outpatient departments and so on before I moved into research, so I kind of felt I have been on the frontlines and that I could really come forward with a fairly well informed innovation.  This first project asked for grant like submissions.  So, it’s just like submitting a grant to a peer review, we had to have backgrounds and significant budget, we had to know the types of programming software we’d be using, what we wanted to create, what we were going to do with it, all those kinds of things.  It was quite formal.  Out of a couple hundred who submitted, we were one of the ones who were chosen for funding.  

Now we begin with submitting brief ideas, and those get voted on, and then those with the most votes are asked to submit a formal submission.  That’s going on right now, I’m sure many people here know the innovation project.  We found that after the award this was a crash course in government contracting.   So, it wasn’t that we could go ahead and begin, what we needed to do was to find vendors who could do all of our programming.  We couldn’t really do it in the VA system and none of us were really expert programmers, so typically we get partners who are going to do what heavy lifting.  Government contracting is not something we do typically in research; there are big offices that do that.  It was really up to us as program recipients that we would do the lifting around the government contracting.  So we really get into the scope of language—sorry, scope of work language.  When you write a grant you’re pretty specific, but for the scope of work you really need to identify what it is you want to do and make sure that the vendors, you know, it’s clear that they’re going to be able to do it.  We got a lot of guidance:  Jason Carley who’s the head of it, Bill Cerniuk and Jim McCain who I mentioned earlier.  So, it took us about a year to do this preliminary work, work out the scope of work, put out the RFP, and finally choose a contractor, and begin the project.  So, we’ll go through a lot of that formulation stuff.  

So what we wanted to do—we had found from our earlier work that there’s a gap in what kind of information patients have that help them think about what kind of treatments they want to do.  There’s plenty of information out there—no shortage of it.  But, the worst mechanism for me as a patient is to find out who other people did on a certain medication that I’m looking to try.  And not just other people- how do people like me, around where I live, do on this kind of medication?  That’s real important for somebody who’s wanting to get better but doesn’t want to go through a lot of the heartaches and headaches of going through lists of medications, trying them, and not succeeding.  So, what was available at the time is sort of what’s available now.  We have a lot of results form small sample studies and clinical trials.  And those samples and the way they’re chosen, of course, are sort of representative of the population.  That allows the findings to be generalized, but it doesn’t really get to me as a patient living in a certain area of the country with a certain type of disorder as closely as it can.  Yet, our evidence base is built on these kinds of studies.  So we found that there really wasn’t any direct model that we could use a foundation, and that there were two approaches that were pretty useful to us in formulating where we wanted to go.  The first one is this patient self-report.  There are online sites, and one of the most popular is patientslikeme.com.  You can look it up here as you’re listening.  

What these sites do is they allow patients to write in and describe their own personal experiences on a certain treatment.  When a clinical trial is published, they typically funded by pharmaceutical companies and pharma companies want to sell their products.  That’s why they’re in business, it makes perfect sense.  They will try to accentuate the positive and minimize the negatives, and were not really sure all the time what the more negative features are of certain medications.  This type of site allows individual patients to report how they felt, or how they experience these meds.  These sites, though, don’t have great search capability.  I could find out somebody who’s a male and who’s my age and has my diagnosis, but that’s about as far as it goes.  I’m not able to look at other very important things, like comorbidity, or smoking, or weight factors, family history—all these other things that can really impact how I do or who others do on certain medications.  The one that is critical here is the geographical region.  

So, not knowing a patient report is from Alaska when I’m say out in Boston is important.  Boston has a very different healthcare system then a town in Alaska may have, so my outcomes would be different.  It’s important to know those things when we’re comparing treatments.  We also like the site because it has the social media aspect to it.  Now, one of the many reason why Facebook is popular is because we can know what our friends like and what our friends don’t like, and what they recommend, or not.  If we have people like us who are suffering from different diagnoses that we have recommending or not recommend certain treatments, we may give it a lot more weight than doctor or provider who says I recommend this medication based on evidence out there in the field.  It adds a different perspective that can be real valuable.  

The second part is this comparative effectiveness research, as I mentioned.  So we have patient self report, how they did on these treatments.  But, it would also be great to know how the treatments do, one compared to the other.  Which is more effective than another?  In clinical trials, as we know, a target medication compared to either an older medication or a placebo, or there may be a second current medication compared as well.  But, rarely do any of those trials compare medications head to head.  CER does that kind of comparison, and CER is one of the big models that the healthcare reform act is relying on to show which treatments work best.  Now, there are problems with this model.  It’s quite big, so in order to do CER study, you need a lot of money and a lot of time to be able to do it.  So the efficiency is difficult to bring down to the frontline level.  It’s relevance, at the point of publication, real relevant.  But, there’s a depreciation in the value of that information as time goes on.  

New medications come out, there’s find new findings about some of the things that original study found.  So, it’s difficult to maintain that relevance of findings over time, and that leads to trying to replicate these studies.  It’s difficult to replicate a fifty million dollar study, and there are no provisions to update information as time goes on.  These are really policy-based studies that don’t make it to the frontline of care.  They can affect the healthcare system, but not really the frontline of care.  So, we feel the modal was good but if we could bring that approach down to the frontline, we’d be doing something really useful.  We felt the solution was in the electronic health records.  There is a beauty in EHRs in a number of ways.  One of them is that they exist in health care systems that have finite geographical boundaries.  What this does is allow local health care systems to get the geographical impact on their outcomes that can’t be done through larger studies.  So it has that regional impact.  It also provides a number of fields and variables, which can help us, group different groupings of patients that aren’t available on site, like patientslikeme.   And, you know, this is an arguable point.  But, there are common methods of data collection.  EHRs get more sophisticated, the data collection or data entry processed are much more contained so that errors are minimized and missing data is caught, and so on.  So what we chose for this study, of course, since we were part of the VHA, was the vistA system.  The vistA, of course, is a national EHR.  It’s national in the sense that everyone across nation can use it, but each region has its own version of vistA.  So when we were putting it together at VISN 2, we used VISN 2’s version of vistA.  We were able to get quite a local flavor using this national EHR.  

We had our formulation background and our theoretical framework, so now we had to talk about how we were going to formulate this project.   So, we all know that EHRs have their own problems.  If you know vistA, you know it’s not problem free; there’s plenty of great fields in this stuff.  I’m looking at my bookshelf that has these very large volumes of CPT codes in them that identify the different services; there are plenty of CPT codes that have great titles that you think, “Hey, that would be great to use”.  But many of them are outdated, some are completely empty, some have had changes in definition over time and some have spotty data.  So, we really have to determine what the best fields are to use.  And then there are the treatments, which determine outcomes.  Lots of treatments are a little fuzzy to some people, so we needed to choose treatments that were pretty easily definable.   So we chose, of course, medications as that.  They’re easily quantifiable, have great data fields, and they’re real friendly to informatics type projects.  So we called out project outcomes-based prescribing.  We wanted to use the outcomes of patients, their experience with the medication, to kind of help in prescribing treatment for individual patients.  So, using the outcomes of others who are like me to prescribe new medications for me.  

So, in our formulation we had two steps.  The first step, which was the critical one, is which of these data fields are any good.  So, we started by looking at these huge volumes of CPT codes to pick out areas that would help us define treatment, help us define patients, their characteristics, and also to define outcomes.  There aren’t just outcomes variables, you have to create proxy variables from a number of other fields.  So, we could choose the ones we wanted and then we had to go through them and find the ones that had data we could sue, so we used a lot of fileman queries to do that, and then went to down to analyze those data fields with SAS consistency, and then came up with this usable list.  

And the second step was to put our project in the scheme of the patient-provider interaction around to new medications.  If we look at numbers1, 2, and 5, this is pretty much the extent of discussion around new medication.  There are other factors that come in, of course, but just in a nutshell.  We have a new diagnosis, the patient goes and reviews information and comes back to the provider.  The provider has his or her own input, and then they jointly reach a treatment decision.  What we wanted to do was add in data on how other similar patients have fared on the different treatment options under consideration, and then also get additional information that could help us make a choice on the different medication treatment.  Information on costs, whether or not it’s available generically, and so on; FDA warnings, and those kinds of things.  So we needed to kind of fit that into that scenario.  Then we needed to develop this general software road map—where were we going to go with this?  If you think of making software programs that can help somebody on a computer, once you get started it just blossoms into all kinds of directions you want to go.  We really had to contain what we wanted to do.  It helped that we had a budget we couldn’t go over to make this workable.  We boiled down to 3 main areas.  

The first one- we needed to have a way to retrieve data and define it.  Then we needed a way so that data would be going out and getting my info to stage a search to look for other people who describe this later, too.  Then we need to use that information to collate the outcomes that we were looking for, and then a function to analyze all that information and report it in a way that’s useful to both the provider and the patient.  So we created all of background, we were ready to go, and we originally wanted to have an actionable program that we could deploy when it was done into our healthcare system up in VISN 2.  We were told in no uncertain terms that this is not something that really can be done, takes a lot of process to do that.  You need to first develop a program, get a lot of feedback around that, deploy in a test setting, have feedback to adjust program, and so on.  We didn’t have money or time to do that so what we decided was to develop a working prototype and this would fit in our budget.  

So, as we were developing all of those things I just mentioned, we had finished our scope of work and put out the RFT for vendors to send in their proposals for.  We got a number of them, we rated them on potential and these groups were chosen:  CWI, which is a better known company for project oversight, and they partnered with ICF International, which is a software development company with Ashley Byrd and his team.  These were just great people and we were very fortunate to work with them.  Meanwhile, we worked pretty heavily with Diane Neimann’s team to create to construct a working data set from vistA which I’ll talk about a couple slides hence, which really formed the backbone of what we were trying to do.  Just a brief overview of system architecture—this is what a real compressed version of what ICF was working on.  We have these two layers:  the presentation layer, and the persistence layer, and how data would interact with what’s on the screen and in database.  Suffice it to say, this is actually a very sophisticated coding project that they did.  It was all open-source, there are hundreds of pages of documentation, and it is really quite robust.  But, the output is real simple, you’ll see as we get into what the program looks like.

The data source itself was real cause for concern from the beginning on, and anytime you’re working with vistA data, there’s plenty of cautions and hoops to jump through, which is only to be expected.  Having somebody like Diane as part of the team really made this happen.  We started off by wanting to get real-time data.  We wanted live data that was actually happening that we could pull from that a clinician could use to this program It’s possible in a sense but what we found was that if we did that, it would cause such a drag in vista system that it would basically crash it.  Enough people were using this at the same time, drawing on vista to create these subgroups; it would handle what the system could do.  So, we developed these call routines using data fields that we had decided on and created this mirror database that seemed huge, actually there are quite a lot of records in it.  But it was much more manageable than using the entire vistA system.  We had this mirror data set we would then ruin the OBP program off of.  Once the main build was there, the main data set, it could be update on a regular basis pretty easily, all that stuff would be run on off hours and wouldn’t crash the system during the day.  We did have that data set and, just a note, we used live data for determining the data fields because we needed to know really what was in there, but then to do the OBP programming we use patient test data.  So we tested building this data set, which we did, and tested that system on a subset of that system for performance.  So, all of these steps worked, and eventually we will be able to bring these all together.  

Looking at time, and I’m keeping an eye on it.  I just want you to know were moving ahead on schedule.  

This is my data map, and I have it for a couple of reasons.  One, to show you some of the connections between variables but also the different types of variables we decided to use.  You could use huge amounts; we kind of kept it minimized so we weren’t way out of scope.  So you see towards the top on the right—we had worked with Katie Rice out of Hawaii, because at the time we were building this she was in the midst of building MHA, the Mental Health Assessment Library, and we wanted to be able to use some of those tools and pull them in if we needed to for this project.   The Mental Health Assessment Library at the time had a number of different tools that any clinician could use, but certain ones that were mandated.  So, those were the ones we tried to pull in from there.  You can see these other areas; patient identification information, and then down on the bottom is prescription data, and then a whole list of medication variables.  If you look a third of the way down, the data patent and data MDA approval- all of those things are pulled in from an outside data set.  We wanted to be able to present to the provider information they really wouldn’t have access to.   So knowing when a potential treatment or medication is coming off of their exclusivity, or their patents expiring, can indicate whether or not a generic medication might be available.  So if it’s a popular medication, chances are that a generic will follow very soon after patent expires.  It’s an assumption on our part, but it’s information that could be helpful.  So, this type of map helps us set up how the program would run form one screen to the next, and I’m going to hop to that and show you what the final product was.  

Now, I could run this live, but I’ve seen too many power points where something gets messed up in the running of it, so I’m going to give you screenshots.  There are only six screens of this program even though there’s a lot behind it, so I think this will work pretty well to show you where we are with this.  

It starts out with this login screen.  The way we set it up, it would be run by the provider, but the screen would be looked at by both the provider and patient.  So, the username is the provider, enter a password, and they would log right in.  It’s not this pop-up that comes up to the screen, it’s a program they would actually go and initiate themselves.  And then a welcome screen, and this would be kind of homepage.  We only have on here the patient lookup by medical record number, that is we have this screen to expand it if need be but probably MRN is the best way to do it.  That way, you’re sitting there with a patient, MRN is a known connector to where it is and would call it up.  There might be a more efficient way to do it, but this is what we landed on for now.  You can tell the layout is real simple—we intended it to be that way, we didn’t want a whole lot of clutter to it.   We were thinking about adding more to it.  We still want to keep clutter down, but have it just as useful as possible.  

The third screen is patient lookup, so this is where we enter in my medical record number so that it can go out and search for me out in the data set, and call up my information, and use that as a stage to then go out and search for this cohort of people like me that we can then look at their outcomes.  So the patient lookup—we enter the MRN, hit submit, and then the next screen is a confirmation that this is exactly the person who we intended to search for.  It’s kind of a check that we got the right person.  So, this is not me, it’s not my age, or a retired vet or anything like that.   It is just test data.  But in this screen also was where we can then chose the information that we wanted to look at outcomes for.  We have classes of medication and then the specific names of medication.  So in this prototype, we have just two classes:  antipsychotics, and antidepressants.  So we can click on antipsychotics, and the medication field would have a drop down menu where they’re all listed.  We would choose one with the box, and we would hit submit, and then we would jump to the next screen.  

So we would then receive this records group.   In this one we have a sample of 2000 people who look like me, who are on the medication that were looking for, and are in our system.  What we did behind this is we built parameters around the different outcomes.  We built parameters around the different reference groups so that we’re not just getting specific people.  The mean age range was from one or two years before or after the target year.  It’s something like the MOM.  I’m not sure what is, not sure if it was monitored or anything, some acronym, but it was meaningful and we wanted to show you what the output was.  So then from there we have noted mean at is the index patient, we know the reference group is accurate and what we wanted, and then we jump to the outcomes.  So, in this screen, here we would have the physical outcome text where this can be expanded to show as much or as little as we wanted.  So with the physical, we’d have things such as blood pressure changes, weight changes over time, any other kind of issues that occurred after the start date of the medications, so there were time intervals in there we were able to parse out and bring in the analysis.  

Mental Health Outcomes—the main focus when I was up at VISN 2 was looking at suicide indicators.  Was there an increase in suicide symptoms with this medication?  Somebody who may already have suicide indications may not be the best med to give at this time; Maybe at another time, but not now.   Satisfaction outcome—of course, there’s no satisfaction variable and the data were proxy variables that were pulled together, and of course there’s utilizations.  And we had this analyze in a few different ways.  A lot of times, we think is positive that patients come in for services, and are consistent and use more services, but it could also meant they’re getting worse on this medication.  They needed to come in more often, and not coming may mean they either fell away from system, or they may have gotten better, so we came up with different ways to analyze those two.  This is everything all together, so you kind of see them and how they would flow.  We could probably combine some of these and make it fewer screens, but this is what we came up with.  

I have in the title the word “novel”.  Is it really novel?  You know, something we keep asking.  The definition is something new or unusual in an interesting way.  We feel it is, and for these reasons here.  EHRs don’t have decision assistance tools for patients.  There’s plenty for providers, but OBP can be used by both patients and clinicians.  EHRs use input from outside sources, but its mainly literature based which is great.  But, it is missing this whole other area here.  OBP uses clinical data from the EHR was well as outside sources.  EHRs don’t clinically apply patient experience.  Well, OBP allows that direct application.   There are few applications that can be directly portable to personal health records.  OBP can be, and it might be something that’s less useful on provider’s desk than it is in a personal health record, but my health [inaudible].  But one of the key things is that this uses that social media framework that I mentioned earlier in helping some patients to see what works for others.  It’s not quite the Facebook model but it is along those lines, which really can be very powerful pieces of information for patients.  

A couple observations that we had from the project—Some EHR systems like epic, they have tools that can build this kind of stuff.  So when epic is ruled out into a hospital system, it comes with a pre-built set that we can then change once it arrives and kind of formulate that to our own facility.  But, it takes some doing to program it to do this kind of a thing, so it’s no small leap, even though they have the tools, to be able to create the kind of a process.  But, the vast majority of EHRs have data that can drive this type of system.  It can be really useful and pretty flexible in not just being a one-system module; it can really be very useful.  We feel that OBP is simple, non intrusive, and portable to different environments.

Now, we feel that OBP has the potential to be the standalone module integrated into other EHRs and then to be available to PHRs as well- personal health records.  Is this applicable to the VA?  We feel, absolutely.  Is it something that can go to a class one level rather quickly?  We feel that given the so many priorities that the VA faces, especially my helping vet cases from doing their own development and then a thousand people like me with ideas that might be really great and might help these vets, that may not be the road to follow right now.  It may be that we kind of make this more of something available to other outside type systems, and perhaps make this something we can market.  You know, a product that can go a bit further.  We’re currently in writing to HRQ, we want to expand the prototype.  We want to assess how we can use it in non-vistA EHRs, and we think that’s going to go over well.  

Of course, we want to develop versions for other platforms, like mobile versions.  But, we really want to add the ability for patient commentary regarding treatment experience to this.  One of the main things we want to do is really expand the algorithm capability.  We definitely want to use additional variables, life expectancy, side effects, and so on.  We’re tossing around the option of building this prescription type of approach where we can look at not just what people did better on but using that information to help come up with an optimized prescription recommendation.  Now, that’s an electronically generated one that a human provider than has to decide yes or no, so it’s something that will take the place of a provider decision, but it pulls together information that a busy provider may not be able to pull together as well.  We feel it’d be a useful approach.  It can allow users to adjust the weighting of the objective functions, so what we wouldn’t want the outcome to be.  And it allows for the weighting of the terms, so we may want to put more weight on cost rather than the disparity of side effects that may not be the case in real life, but just as an example.  And then, finally, this may not be great for everybody, so we want to determine optimal users.  And then, how do we target people who we want for the secondary clinical effects.  If we really want to make this very useful for people who are at risk for suicide, we would have it in such a way that it would be useful and there’d be a different version for people who are at that risk at the time of assessment.  

So, were really excited about moving forward with this, and as I mentioned, we haven’t a deployed it within the VA because of the backlog of the priorities.  But, we feel there are many other areas that we can grow this and use it.  So, I do want to acknowledge, for the second time, these people involved because literally these kinds of things are so difficult to do that you need key people involved, and this is the group, again.  Dianne, who was critical.  Jim, Ashley, Joe.  VISN 2 for being originally where I was when I had this, and VISN 16 is just so supportive of these kinds of approaches.  So, I think we’re at the forty-five minute mark, and I”s just like to say thank you for listening and I’d really love your comments and questions and criticisms about this approach, and I’ll just be quiet and let Margaret or Heidi take over from here.  

Moderator:  John, that was excellent.  Thank you so much, and you’ve given us plenty of time for questions.  And, there are questions.  Let me just say to the audience, I hope I’m not too broken up.  There is an evaluation when you sign out of go to webinar, and we really would appreciate if you would complete that before you closed it.  

Ok, question #1.  John, can you hear me well enough?

Dr. Crilly:  Yeah, I can hear you great

Moderator:  Ok.  Did you also look at drug interactions at potentially inappropriate meds, like the HEDIS...whatever that survey is.  Anyway, did you look at drug interactions?

Dr. Crilly:  No, they HEDIS to the survey is that it’s a great one.  We wanted to look at drug interactions, but we only had a little over one hundred thousand to do this and that would’ve taken us beyond.   I know it’s not the state of reality that someone’s only on one medication, so that’s going to be some of our next steps, to build in multiple drugs and definitely the drug interactions as our outcomes.  Great question, but it really was budget constraints that stopped us.  

Moderator:  How was the comparison group defined?  Is it different by the type of medications you were looking at?  What patient characteristics are important?  I’m sorry if I go in and out, I’m looking at my screen and coming down to my phone.

Dr. Crilly:  You sound real consistent.  The comparison group was based first on my own characteristics.  The changes in the comp group were looking at a different treatment would be a change in medication.  So if I wanted to look at somebody on Zyprexa, because I was going to be going on Zyprexa, it would only pick out that group.  If I wanted to look at someone on Resperidol, it would recalculate a new group for people on that medication.  It’s simplistic, but we’re looking to develop the proof of concept and it would--you know, I say this with real caution.  It’d be a simple thing to add in different components that can make that a more robust sample size.  We also have to be careful and we didn’t really see the scalability we had but as we build in more search terms will restrict the number of search terms in these groups.  But, the number of records we have and the original database we pulled from vistA was over eight hundred thousand, so there was a good amount of people.  This was just up at VISN 2; it was a pretty robust group we could draw from.  I hope that answers the question.  

Moderator:  Ok.  What are the text fields?  Does the provider enter text?  Is there any structured language?  

Dr. Crilly:  Good question.  I think the way we labeled those for the presentation may not have been as clear.  So, what would show up in there would be the outcomes, which would come back from the analysis in a text form.  They wouldn’t be actionable fields that we would then click on and then expand, it would really only be text.  That’s what we meant by text in that area.  What we could do is click on that box and expand it so that it takes up more of the page and you could read it better.  But that’s where we ended up with that last outcome screen.  

Moderator:  Ok, I just want to give you a comment in capital letters, “THIS IS SO WONDERFUL”

Dr. Crilly:  Thank you very much.  We appreciate that.

Moderator:  Next question.  Did I miss what results are looked for?  How...I don’t quite understand this john, you can interpret it.  How crowd sourced outcomes could then into the patient?  How are crowd sourced outcomes presented to the patient?  

Dr. Crilly:  Yes.  So crowd sourcing is sort of that social media area that we mentioned, and that is trying to get input from the crowd and from the population in a way that’s fairly unstructured, but that you get a lot of great information back.  This was sort of a not strictly crowd source and not strictly social media.  This was us going out and getting information from others that we proxied other’s outcomes as their experiences and brought it in, bring it to the next crowd sourcing level later.  Thanks for raising that, it’s an excellent term I should’ve used instead.  Crowd sourcing is great.  We developed and I think you might be asking how we put together the outcomes.  What we did is we only had a limited number of variables, as I mentioned we could’ve had a lot more but needed to constrict it a bit, so we had to developed for every outcome was a proxy for a number of different variables together so that we had to then for each of those variables create parameters that would allow us to say, “ok, this is an outcome that falls into a warning zone.” For example, that a patient may want to know about and then how do those all come together for that single outcome variable that were looking at.  And then, for blood pressure, change the blood pressure over time.  There’s no proxy for that, we just kind of report that as fluctuation over time in blood pressure.  I’m hoping I answered that sufficiently, too.  

Moderator:  John, you can tell I’m not a social media user, I’m stumbling over that word.  Ok, next question.  I think you have anticipated it, but I’m going to read it to you anyway because it’s long and there might be something else in it.  In slide 23, you showed the 4 domains—physical/mental satisfaction, mental satisfaction, service utilization, I believe information in these fields are a summary of the reference group.  Is this correct?  Could you speak more about how the data for the reference group is pared down to concise summaries to fill these fields in the application?  

Dr. Crilly:  Ok, I sort of touched on that in the last response.  This was pretty painstaking.  When we went through and decided what the parameters were, we actually based that on the literature.  We really didn’t have anything else to base it on.  Ideally, we’d base it on our own population in VISN 2 and the data that was drawn from.  But, we felt it better to do it through the literature.  So, for each of the variables, we did that boundary setting.  And then, within the algorithms, we had the decision processes, which would then decide what is the best thing.  Do we just add warnings up there?  The warnings of course would be based on calculations we get on parameters of the individual pieces.  But it could also be valuable if people know that there were no problems with blood pressure and so on.  If I was a patient with blood pressure problems, I’d want to know that.  I’m going to tell you, its loose, and you can hear it.  It’s loose as I’m describing it.  But, this we feel really sets a great foundation for the next steps to make it more tight.  But it also points to the question from the person is right on, as this is a complicated thing that we don’t want to have a lot of our own biases in when we do it so this part of it is going to need help from focus groups and from other experts who can give good critiques about what we’ve come up with and we can help it move forward in a better way.

Moderator:  Great, thank you.  Next question.  What if any privacy issues have you had to deal with so far developing this prototype?  

Dr. Crilly:  We were able to look at the individual data fields and we used, like I said, live patient data for that.  The reason as that we need to know what’s in there.   We’re not interested in what’s in there, but we need to know there are things in there.  Then, we could analyze them and the aggregate to figure out whether it’s consistent and usable.  That was fine.  Pulling the database, that large data set from vistA to serve as the backbone for the whole project, was fine.  It stayed with them in a secure environment; it was done securely by VA programmers from VISN 2.  From the OINT it wasn’t any, from the critical side.  But, when we wanted to use live patient data in the actual prototype, which was where we ran into problems.  It wasn’t that “oh, man, you’re using my data,” it was that we were contracting with people outside of the VA, and this would need to go on a number of VA laptops that were given for the project, and they were kind of around the country for the people putting this together.  

But, it would still be in a place that would be less secure than what the VHA was comfortable with.  So, the OI&T people had plenty of patient test data around that they could use, and that’s what they ended up using for that.  So, once it’s in and deployed it in a VHA environment, which would not be the issue.  And, the only PHI that was used was my own.  So, me as a patient sitting there with provider giving consent that,  “Yeah, you can type my MRN into your computer, and I want to find people like me,” no names, no anything come up except for their outcomes.  That’s only thing that we were interested in.  So, that part was also clean as far as the information security people.  This wasn’t just “Oh, ok, that’s fine”, there was a lot of discussion around this.  I met with some very well informed security people up in VISN 2, and we had lots of discussions, and decided different ways to do it.  But, those were the main issues that we had to struggle with.  We feel we came up with a good product with all of those oversights, which I feel are important.  We need to make sure that those are being looked at.  

Moderator:  Ok thank you.  Another question.  Would there be a way to analyze OBD data for off label prescriptions?

Dr. Crilly:  Yeah, that’s a great question.  We didn’t think of that one.  The off label prescriptions could really be done I think not easily, but using the criteria for prescribing a certain medication.  If the indication falls outside of that, then we could have that as an off label prescription, and then look at the effects of that.  I think it’s a great idea.  Yeah, there are certain ways to do that.  If you don’t mind, I’m going to make note of that and probably add it into our blueprint.  

Moderator:  Good, good.  Question, ok?  Next question.  I have found there is variability in private positions in terms of how open they are to patient input into their treatment plans and to the patient’s treatment plans.  How do you anticipate your work will be received in the VA culture?  

Dr. Crilly:  Yeah, that’s a great question, too.  What we had in our original proposal was we had this team of reviewers of this, and it included the head psychiatrists up in VISN 2, head of psychiatry, nurses on frontlines, prescribing physicians, and some administrators, too.  And that was with the intent of deploying this thing as an actual working product.   Then when we couldn’t fit any of that stuff into our budget, we really couldn’t do it.  From what I heard when we put together that team, people were positive.  We actually tried to find people who were negative about this who were interested in giving us feedback but again, that never really came out.  But just keeping my eye on the time, just a little caveat, recently in one of the healthcare blogs, there came out an article about 5 most intrusive electronic tools within healthcare.  You would think it had to do with those darn pop-ups, “Boy, that’s intrusive”.  

But, they were these tools that are actually intrusive to the healthcare system because they force change.  They force the health care system to do something that they’re not used to doing.  So, it wasn’t that it was done in a bad way but these are things that kind of force physicians to think differently.  We think this might be kind of a long route, so I’m thinking what well get is some positive responses, we’ll definitely get a lot of negative responses, but we also want this to be a provider choice that they can use, and there’s no way to stop a patient from looking at the internet and coming in with their own printout of what they found.   This is something they are using in their own healthcare system, we all involved with our own treatment.  It’s less varied, but I think it the patient still really might have input.  It’s something that’s definitely not going to go away, and we were sensitive to that as we built it.  I think it’s a great point.  

Moderator:  John, it’s 12 o’clock.  I personally have about 10 minutes more.  There are more questions.  Do you have more time?  

Dr. Crilly:  Yes, absolutely.

Moderator:  And, Heidi, do you have more time?

Heidi:  Yep, I can stand up longer.

Moderator:  So, maybe we’ll say another 5 minutes of questions?  In terms of outcomes, is there an indicator for what is considered to be a successful treatment outcome?

Dr. Crilly:  Successful treatment outcome would be the person getting better, and lack of trouble to indicate the facts, and no other outside issues.  We try to capture that in this, we wanted to give patients a list of things that went wrong, and I don’t think we focused enough on things that went right.  So, I guess that it gives more in successful treatment outcome, gives patients more info about them for them to decide what would be a successful outcome.  I hope that’s not dodging the question.

Moderator:  Great.  Asking for your VA email?

Dr. Crilly:  John.Crilly@VA.gov.  I have a unique enough name that I don’t have ones, twos, or threes after mine.  Just John.Crilly.  

Moderator:  Actually, this is the last question.  Does this have potential for integration to other vet accessible applications, e.g., My Healthy Vet?  I think you talked about that a bit, but you might say something else.  

Dr. Crilly:  Yeah, we really felt that this was, and actually there were a couple people that mentioned how well this might fit as terrifically as a patient tool, not just something that a patient-provider would use.  That they wouldn’t be posted, personal health records could go out to the EHR and gather this data that has no PHI in it, its all aggregate and not looking at something specific, and bring that back and allow patients to bring it back and do it on their own with a HR.  We feel that the PHR is kind of the key place for this thing to be.  

Moderator:  Ok, end of the questions.  Thank you so much, that was excellent.  I want to say to the audience that our next seminar is may 15th, Dr Carl [inaudible] from the San Diego VA Medical Center speaking on the topic of wireless monitoring of sleep apnea.  Dr. Crilly, thank you so much.  That was just wonderful.  

Dr. Crilly:  Thanks.

Moderator:  Thank you everybody.  Bye!