**Transcript of Cyberseminar**

**VIReC Database and Methods Seminar**

**Using CAPRI and VistAWeb**

**Presenter: Linda S. Williams, MD**

**March 3, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact virec@va.gov*

Arika: Good morning and good afternoon and welcome to VIReC Database and Methods Cyber Seminar. Thank you to CIDER for providing technical and promotional support for this series. Today's speaker is Dr. Linda Williams. Dr. William is a core investigator the VA HSR&D Center for Health Information and Communication and professor of neurology at Indiana University School of Medicine. Questions will be monitored during the talk and will be presented to Dr. Williams at the end of the session. A brief evaluation questionnaire will pop up when we close the session. If possible please stay until the very end and take a few moments to complete it. I am pleased to welcome today's speaker Dr. Linda Williams. 

Dr. Williams: Thanks Arika and thanks everybody for joining us today. I will just let you know what we'll be talking about. I mean first I will tell you a little bit about the project in which the Vista Web chart reviews, INSPIRE QUERI service directed project, and then try to help answer the question why did we choose to do central chart review for this project? I'll then give you some examples of using Vista Web and CAPRI for this type of research and finish up with some lessons that we learned in the course of this project and hopefully leave plenty of time at the end for questions. Our first question was an audience poll which is going to be set up here, I think, by Erica just to make sure everybody's awake and not freezing hopefully today. But I was curious to know this year in 2014 one of the coldest temperatures you've experienced at your VA so far this year. It looks like we have a lot of people experiencing the polar vortex on this call. I'm very happy to report that in Indianapolis it was only minus six-wind chill today so I didn't have to go with the full-face gear like the person in the picture. 

Alright, the INSPIRE service directed project was a project that we just completed this past year in the Stroke QUERI and I wanted to just to give you a little background on that project. I think that helps understand and communicate why we chose to do the chart reviews the way we did for this project. So the Stroke QUERI did several things about five years ago that kind of led up to this project. We started working with, what was then called, the office of quality and performance inpatient VA stroke care. At the time there were no inpatient quality indicators that were routinely collected related to processes of care around inpatient stroke management except for access to rehabilitation. So we worked with the office of quality and performance to do a national chart review study at that time. That, however, was conducted via the EPRP chart abstractor. So the chart abstractors that are affiliated with each facility, doing chart review mainly using the local CPRS electronic health record. So at the same time this was going on our VISN that the Indianapolis VA is in VISN-11 was focused on improving stroke care is one element for VISN-wide focus that was in fiscal '08. And so those things kind of came together with the interest in the Stroke QUERI and improving inpatient stroke care to result in the INSPIRE project. 

When we do the OQP stroke special study for this first national assessment of inpatient national stroke care within the VA I just summarized the results here by phase of hospitalization. This was a chart review on about five thousand stroke admissions in the VA. And so you can see that especially early in the stroke care process is where we have the greatest opportunity to improve with indicators. The two indicators I want to highlight are the two that became that focus of our INSPIRE project and that is dysphagia screening before oral intake, which at the time of the OQP chart reviews fiscal year '07 data was happening in 23.4 percent of Veterans admitted to a VA facility for ischemic stroke and also venus thromboembolism prophylaxis which was present in 78 percent of eligible Veterans admitted for ischemic stroke. We chose those two indicators because they both had opportunities for improvement and they were also present or many patients were eligible for them. So a large proportion of Veterans with stroke are eligible for dysphagia screening and for venus thromboembolism prophylaxis. 

So the INSPIRE project came sort of in response to this observation of our initial quality assessment. We had two qualitative aims that I will not be talking about today. Dr. Teresa Damush, I believe has given a cyber seminar related to some of this work. But we were interested in the effect of this OQP study on the VA stroke care in general and then we were interested in studying the VAs we were going to be improving in terms of their organization and context of stroke care. But the quality improvement portion of this project was to test the systems redesign or operational system engineering based intervention versus performance data feedback alone to improve those two stroke indicators. 

We had 11 sites that got randomized in this project, intervention versus control. They had to have at least 50 ischemic stroke admissions annually. And at the intervention sites we had an in-person collaborative. We worked with our VA center for applied systems engineering, our VISN-11 work, to do systems engineering, systems redesign training for those intervention sites. We followed that with six months of coaching or external facilitation through their tests of change, and then we gave them money quality indicator feedback. The control sites; however, did not get the quality improvement training, the in-person collaborative, they only got quality indicator feedback. At both intervention and control sites though this, again, was something new. There was no other way of getting these data unless the sites just collected them themselves. And then we measured these quality indicators via central chart review for one year prior to the intervention and also 12 months post intervention. We were interested in improvement and of two quality indicators and also looking secondarily at the temporal pace of change and performance and how sustainable it was over time. 

So venous thromboembolism and dysphagia screening were the two primary indicators of interest. We did collect eight other joint commission based quality indicators and one stroke quality indicator that was importance to the VA as well. And so I wanted you to be aware that we didn't just collect information on these two things but actually all elements of care that were defined by the joint commission at that time as being part of primary stroke center certification. So that ended up being quite a large amount of data that needed to be collected via chart review. 

So why did we decide to do this by chart review? Well, the fundamental issue for us is that, except for access to rehabilitation care, none of the inpatient stroke quality indicators were currently being collected in any way as part of routine VA care either in paper form, by some operational or clinical entity or in electronic form. Currently we do have an IPEC stroke module that’s available for self-reporting three quality indicators. This first became open for self-reporting in July of 2012 and the three quality indicators currently being collected on a month basis include TPA for eligible patients, dysphagia screening before oral intake and completion of the NIH stroke scale. In physical year 2013 we had about one-third of VA facilities self-reporting within this database so the database isn't yet complete. This is being sort of taken up individually by different facilities at different time points but we're hoping to increase the proportion of facilities contributing to that self-report database in this fiscal year. 

So why did we decide to use Vista Web? Well, the data that are required to construct the quality indicators for stroke care by in large are not part of the VA electronic health record data. There are many key elements in these indicators, some of which I'll go on to show you in a little bit, that are not present in the electronic health record. for example, knowing whether someone has received dysphagia screening is not something that's recorded in a standardized way at all facilities. It's often not recorded with a note title that says something about dysphagia screening. So it's not very easy to search for a note title. It certainly doesn't have a standardized health factor that would allow you to extract information about dysphagia screening from the electronic record. It would be possible for dysphagia to extract information about speech language pathology consults but that formal sort of dysphagia consultant and evaluation takes place much further down the clinical pathway. We were looking for dysphagia screening that happens before admission or on admission that helps determine whether the patient can have mediations and food ordered and delivered to them. So simply looking for something like a consultation by speech language pathology would not be adequate for that particular measure. 

Another one of particular interest to this study was venous thromboembolism prophylaxis with mechanical devices. Those have non-standard ways of being ordered at different facilities. So again, it's very difficult to use a health factor or some variable in the electronic health record to robustly identify whether patients are receiving these kinds of devices at the bedside. Another thing that plays into a number of the quality indicators for inpatient stroke care is documentation of comfort measures, which again doesn't have a standard health factor. Not every VA documents it the same way. Many use just text narrative orders that don't have a health factor associated with them. So again, knowing that a comfort measures only order was taking placing and when it would take place is not something that's currently very robustly or accurately identifiable via the electronic health record data. 

This slide just shows you an example of our venous thromboembolism prophylaxis flowchart just to give you an idea of the kinds of things we were looking for and what made us decide to use Vista Web and then centralized chart review for this kind of process. The things that I've highlight here with the circles show you things that are not very feasible to collect simply with electronic data. So the first question in this flow chart asks whether the patient was hospitalized for at least two days. That's fairly easy to assess with the electronic health record data. But the next part of the algorithm then asks if they were ambulatory by hospital day two. And as you can imagine there's not a very good indicator in the electronic health record of whether a patient is ambulatory or not. Next, we come to the question about whether comfort measures only were documented by hospital day two. If they are, that makes the patient in this algorithm ineligible for the indicator. But as I just mentioned, that's also not routinely available in the electronic health record. Then we come down to what medications or mechanical prophylaxis the patient receives. Medications are pretty easy to access through the EHR but these mechanical devices again are the part that makes it very difficult to know for the stroke patient whether they truly got an approved treatment or venous thromboembolism prophylaxis. Then the last part of the algorithm asked whether the provider documented any contraindications to either medications or mechanical prophylaxis. And as you can imagine there's really not a way to capture that in the electronic health record. So even in what--by some considerations would be a fairly straightforward quality indicator there are multiple parts of that indicator that are not readily accessible with the EHR. 

So why did we decide to use Vista Web? Well, at the time the study began the central data warehouse and the Vinci portal to access the central data warehouse was not operational. So that was not an option for us. At the current time that might be something that someone would contemplate. I believe that the TIU notes are available now through Vinci in the CDW. So that may be a route that someone could go in the future without going through either Vista Web or Capri like I'm going to talk about today. I think there are some theoretical disadvantages depending on how you're trying to use those notes however. My understanding, which might not be up-to-date, so maybe we have someone on the call that can correct me if I'm wrong at the end, but my current understanding is that notes would not be packaged in a chronological way in the same way that they are in Capri or CPRS or Vista Web so that if you were trying to reconstruct an actual multi-day episode of care that might be challenging to do within Vinci environment. Although, again, access to individual notes and note title I believe is now possible. 

The other option, of course, would be local chart review. So in a multi-site study perhaps you could train a local chart reviewer to do chart reviews at their site. That probably has some advantages. A person at a single site is going to be more in-tune with how things work at that site and how the current clinical practices are going and where they do things that might be somewhat easier in that regard for them to do it. But also in a quality improvement study it does help engage the local team in a way that central chart review does not. However, it's very expensive to place multiple research assistants at different hospitals. You have to train them which is challenging, again, from a distance to train and sort of maintain their training and to ensure that there's very high quality and that all sites are consistently abstracting data in the same way. So for us that was part of the decision about not trying to place local chart reviewers within each facility where we were intervening in this study. 

When we thought about this just from a project-planning standpoint we realized we would need about three FTE of chart review working over at least a 12-month period. Our chart review form had around a 120 variables that were abstracted to construct these 11 quality indicators. Some of the variables had multiple responses, for example, listing the medications that patients were on at admission, listing the medications they were on at discharge. So it was a fairly complicated sort of chart review system to work. We centrally were able to very thorough weekly abstractor meetings where we would review questions, make clarifications, update our manual, bring examples both in training and then once the study started. And while it's certainly possible to do that via the telephone or via a live meeting, I think it does help to have people in the same room especially the abstractors able to work with each other closely every day to make sure that they're doing things in the same manner. Our initial estimates in this project is that we would have to actually open about 24 hundred, 23 hundred, charts and about 16 hundred for full review. Some of the charts that we opened, of course, turned out to not actually be patients with ischemic stroke. So we knew that we would have to do a full review on just a proportion maybe 80 or 85 percent of those that we actually got from the ICD-9 code list. 

Within our 11 site study we reviewed two and a half years of stroke admissions going across all of the facilities and about 24 hundred charts actually got fully reviewed by our abstractors here which is certainly a lot and our abstractors were working very hard to get that all completed. If you think about that at the local site level, however, you can also kind of see one of the calculations about not using a local chart review. The site level load with that volume of stroke admissions would be about 75 stroke cases per year. So in a given week there are really just a small handful of charts that would need to be reviewed. Since we were looking both backward in time and forward in time after the sites got trained it wasn't as simple as just budgeting in a retrospective study for how long it would take a given person to get X number of charts done. We had to do this prospective monitoring and feeding back of data as well. But an individual person at each site would not be especially busy since there might only be three or four stroke patients per week that they would be extracting data for. So it seemed like it would be a difficult task to find sites that would be able to hire or find some small percent of a research assist or a clinical nurse to do some of that chart abstracting. And we really found that when we budgeted it out that it wouldn't save us any money compared to doing the central chart review. And we had concerns, as I mentioned earlier, about training site personnel, maintaining site personnel, retaining them is also a concern if someone leaves and you have to retrain then that's quite daunting to your timeline quite substantially in a chart review project like this. So for those reasons we decided to move forward with central chart review. 

One thing that I'm extremely pleased with is the very high quality of data that our chart reviewers collected. Out of the 118 variables that we collected a 113 of them ended up having greater than a 0.8 interclass correlation coefficient or a CAPA score, none of those variables, none of the five that were left in that actually were part of our quality indicator algorithm. So we were pleased about that. We continued to monitor that throughout the study doing a 10 percent random resampling where a different chart reviewer would resample and redo the chart review in 10 percent of our cases. We looked at another way looking at just the quality indicator result on those resampled cases. That was a patient level analysis where we defined each patient for each indicator as either being ineligible for that indicator, eligible in passing or eligible in failing the indicator, so a three level variable for that quality indicator. And when we looked at it that way we had excellent quality indicator agreement with the CAPA's ranging from 0.84 to 0.96 on the 11 quality indicators that we did track as part of the study. So I do think that one of our concerns which was data quality we were able to meet that by doing central chart review with the research assistants here in Indianapolis with our team. 

So kind of a summary for some of these advantages I think before we get into how we actually did it, I think it's nice to be able to access multiple sites from a central location. It facilitates your training, it facilitates review of what's actually happen as chart reviews are ongoing and quality control especially interrater reliability and agreement testing. It allowed us to collect text-based data elements with high intersite variability. Some people might ask, well, what about using text mining or natural language processing? And I think for the kinds of things that we were trying to ascertain on the one hand there were many of them which makes it challenging to use text mining or natural language processing and they were also quite complex and variable. And so different staff, different methods of doing things, different names, different locations where things might be located within notes in a given facility all sort of made that text mining or natural language processing route quite complex. There were also a number of data elements that reflected clinician judgment, for example, documenting reasons that a patient might not be anti-coagulated. That also is quite hard to abstract electronically or with text mining. And I do think it also facilitated data feedback and discussion with the site teams. We were able to see the record in pretty much the same way that the site team could see it. So if we were feeding back quality data to one of our stroke improvement teams we could tell them where we found documentation of dysphagia screening or where we might have a question about how something should be interpreted. Did the way the clinician talked about this really indicate that they had done dysphagia screening or not? It allowed us to have some dialogue with the clinical teams at each site that I think was very useful in this kind of quality improvement project. 

Okay, so let's go to our second question before we get into some examples of how we used it. I'm interested to know for the folks on the call what experience do you have using central chart review for VA research? We'll give it a minute or so for those answers to come up. Okay. it looks like it's good that we have a lot of people on the call who haven't tried to do this before. Hopefully that means what we're talking about today will be useful to you. And several people have used Vista Web, not so many using CAPRI, some have used both, some of those may be some chart reviewers from Indianapolis who I'm sure will correct me if I say anything wrong in the section to come because they really are the real experts about how to use these tools. Okay, so let's talk about the options, maybe some similarities and differences, and I'll show you some examples with some screen shots of how we've been able to use these tools. 

So the two main options for central electronic health record review in the VA are Vista Web and CAPRI. At the time that we were initiating the INSPIRE study I have to say I did not know about the CAPRI program. I'm not sure if it was exactly available at the time we started. It may have been. It was not something that I was familiar with. So we did not look into that at the time that we launched the INSPIRE project about four and a half years ago. But Vista--they both provide read-only EHR access for a given patient. They both require approval to access for research purposes. Some of the differences I've highlighted here, and then I'll show you a table on the next slide that compares them more specifically, but key thing that's different is that Vista Web provides access for a given patient across all sites of care for that patient. The CAPRI tool provides access for a single site at a time. It does, however, have a Vista Web tab that can shift the reviewer into Vista Web if Vista Web has also been approved for that particular reviewer. 

This slide shows some very specific comparisons of CAPRI versus Vista Web. CAPRI requires installation of special software. Vista Web is generally available on our VA computers and as part of CPRS and doesn't require special software installation. Because it's special software CAPRI also requires a unique access and verified code pair so it doesn’t use your current local Vista access verified code pairing. CAPRI provides review of the EHR for a given patient one site at a time. You can switch back and forth between sites but you only see one site at time whereas Vista Web allows you to view the electronic health record of all the sites visited by that patient at any time. It's organized chronologically so, for example, if you look at--if you're interested in a given date range that a patient's been admitted somewhere you can request in the view, you can select that data range and see only the clinical information that's available for that admission. But you can also look over time to see if perhaps that Veteran was in an emergency room at another VA hospital maybe a month before. That might be especially important, that sort of transitions in care might be important then that would be something that would a little easier and more straightforward perhaps to do in Vista Web and CAPRI. 

One thing that CAPRI does that Vista Web does not currently have the ability to do is to provide text searching within progress notes. I'll show you an example of that in a moment. Vista Web has a search function or a find function that works in some other sorts of data like lab data and ordering, order menus but it doesn't provide that within progress notes. And then finally CAPRI provides this transition between EHR applications. So if you get approval to access both CAPRI and Vista Web you can access both from the CAPRI platform. So sort of the bottom line, my personal recommendation, would be that you get access to both, that allows you to have maximum flexibility in your study. One thing that we've learned through the INSPIRE study and then a current study that we're doing in Indianapolis, Dr. Dawn Bravada is the PI on a TIA, transient ischemic attack management quality study, is that there are just--there are different ways that you can use the two tools differently. Depending on the exact piece of information you're trying to find you might it quicker to go in through CAPRI or you might find it quicker, more efficient for you to go in through Vista Web. So getting access to both I think really gives your chart reviewers maximum flexibility. 

So let's spend some time talking about some examples. Here are some possible considerations when you use Vista Web. Now the first thing I mentioned here is something that really seems to have been improved in the past year. At the time that we were doing the INSPIRE study Vista Web was not accessed as a tab in CAPRI and so there was quite a variable time to load pages. Sometimes loading a page would take seconds. Sometimes it would take 10 or even 15 minutes to load notes from a selected date range if you were trying to hone in just on clinical information that pertains to a specific date range. For us we were interested in the dates of the admission that we knew about. It didn’t seem that this really varied by anything that we could really put our hats on. We called the help desk a lot and talked to a lot of people and just couldn’t really tell if it happened to be internet traffic or what the problem was. However, as I mentioned for the current TIA study that we're doing, I've talked recently with the chart reviewers on that project and this seems to not be something that they are experiencing when they access Vista Web through the CAPRI tab. The Vista Web, as I mentioned, allows you to do some text searching, a fine feature to look for things but not within notes. So you can look at orders, you can look at medications, you can look at--for a particular lab value if you're interested in a glucose you can go into the labs and find that and that helps and makes it faster for you. But notes themselves, which is often what we're looking for in this study can't be searched. 

It's also in Vista Web easy to make abstraction errors if separate locations share a single facility identifier. And I'll give you an example of that in just a moment. So if there are--if there's more than one facility with the same facility identifier it can be a little bit tricky to make sure that you're clear on where the patient was receiving care for a particular documented note or episode of care. So if tracking these transfers is important Vista Web can make that more obvious. Again, it's showing you views from multiple facilities at the same time whereas CAPRI is showing you only notes, labs, things pertaining one facility but you do have to be careful in doing notes from these stations with multiple facilities. And I'll show you an example of that. One of the sites in our project was the Nashville VA. Nashville is actually part of the Tennessee Valley Healthcare System. and so there are actually two hospitals with inpatient care within Tennessee Valley Healthcare System. our study was only taking place at Nashville. But Murphysboro also sometimes admitted the stroke patients. Sometimes they transferred to Nashville. And so this slide just shows that we were looking here at the expanded ADT which is a place you can look in the records for the admission and discharge diagnosis. It tells you where the patient was admitted, what service admitted them the date, the time, and so really you had to be sure that you were paying attention. You can see the Nashville admission is identified by an NA and the Murphysboro admission is identified by the letters MU. So the abstractors just had to pay a lot of attention to make sure they were really tracking the admission that happened at Nashville, not one that might have happened some days earlier or later at the Murphysboro VA. And again, that's a unique problem when you have multiple facilities that are capable of delivering the same kind of care that do share a single facility identifier. 

Here's just another place that sort of makes that clear. If you're looking at notes you can see the location of the note, one of the history and physical, one's a neurology note. Again, it uses the MU for Murphysboro, the NA for Nashville, but if you're in a note, you're abstracting lots of things you just have to be careful that you're in the correct note from the facility that you're really wanting to pay attention to. So Vista Web, again, it has that advantage of showing multiple sites but depending, again, on what exactly you're wanting to do in your project, you're chart abstractors have to be aware that it's a little bit easy to get confused if you're not careful. 

CAPRI as a system has its own login. This just shows you here. It looks like sort of the standard VA access code and verify login. And then once you get into it you will select a location and then a patient ID that you want to look for. And so you'll use the social security number. You have to have real social security numbers to operate in these systems. And so you'll select the patient that you want to look at, and then you'll see a view that looks like something like this. So you can see across the top--I'll try to show here--there are different tabs running across the top of CAPRI. The one I've got circled, the one that’s active right now is called clinical documents. Here's the Vista Web tab if you wanted to go into that view. But there are other things, health summaries, different reports, but then within clinical documents if you look down at the bottom there are another series of tabs. So within clinical documents what we're currently showing is the notes tab. You would go to a different tab for discharge summaries, for consults, vital signs, medications, labs, so it's similar to CPRS if you're familiar with looking at that. But it's just a little bit different, imaging, diet orders, you can see those things along the bottom. So it's constructed in a little bit different but fairly straightforward way, fairly comparable to CPRS. So here you can see the notes are listed along the left hand side and you would click on one to see it displayed here in the main pane. 

Now this shows you an example of the search function which can be just extremely helpful. So here you can see in this patient's example there are a lot of notes. There are all these notes along the left hand side. You can see down here in the right hand corner, this find little button. So if we wanted to look for something very specific like in our example maybe in stroke care we would be interested in knowing something about carotid artery disease or carotid stenosis or carotid endarterectomy surgery we could look for that word carotid. We see--we put that in here and that filters out all the notes that don’t have anything to do with carotid. It only leaves us with the ones that do have that word included in the note. And then that word is actually highlighted. In this example it's shown as one of actual in the patient's diagnosis list. But it will also highlight any times the word carotid might be used in an assessment, in a plan, in anything that the provider is writing. So it allows you to have some nice tools for searching if you're really looking for very specific kinds of symptoms that are mentioned, unique word that might relate to a procedure or a diagnosis. That can be a really helpful way so that your chart reviewers aren't slogging through thousands of notes trying to find that one kernel of something that they're really interested in. So that is one of the key differences between CAPRI and Vista Web. 

Here's--I just wanted to show you how some of that is available within Vista Web. Here's--we're now in the Vista Web view. You can see it looks a little bit different. Here's the Vista Web sign over here. The different options to click on are all shown along the left hand side maybe a little bit small to read but there are things like discharge summaries shown here, laboratories, or within the orders tab here and we're looking here at an order summary for this patient. But there are progress notes, radiology reports, all kinds of things. So the find command you can use within Vista Web but not within the notes. So here we are within the order summaries. Perhaps we're looking for a particular medication. So I just put in aspirin. If I wanted to know if this patient got aspirin it would look through and actually highlight for me all the different times that the patient got aspirin. Again, if you are wanting to know something about whether a medication was given and the first time it was given this can be a way that really does speedup and makes the burden on your chart reviewers a lot less to be able to use something like this when they're looking for a very specific medication. So that is available in Vista Web as well as in CAPRI. But again, the searching within the notes is something that is not available within Vista Web. 

So just to kind of summarize some of the lessons that we learned using CAPRI and Vista Web for research maybe here in the last five or ten minutes and then we'll have 15 minutes or so for questions I think that we--we're very pleased with the way this worked out for our project. We felt that is was the optimal way to collect clinical chart recorded data that were fairly complex and fairly broad I would say in the terms of the kinds of data that we were trying to collect. Is very important when you're thinking about planning a study like this to consider how complex is it going to be? How much judgment is involved versus simply finding a piece, a factual piece, of information like the first time that aspirin was given. That can really help you decide what's the best route to go in terms of the way to collect this data for your study. I think any time you do a study like this it's very important to develop standardized chart review manuals. I can't over emphasize how important this is and how important it is to continue to update it as either things change on a local level. You might have a sight that develops a new way of documenting dysphasia screening and so now there's a different place that you have to look for it. It's very important that you update your manual with that. So all chart reviewers are looking in the same place. And sometimes the data systems change and there's a different way of accessing it or a different feature or sometimes the project team changes their mind, makes a new rule about how to find something and that has to be updated. So that is absolutely critical to a project like this to having very high quality data. 

When you're using things like the find feature that I showed you I think it's very important to standardize those features. One abstractor might use them a lot and use them in lots of different ways and that would lead to then finding things in different places than another reviewer. So that's part of the training process. That's part of getting people familiar and figuring out what is the best way to search for things. But that's something to keep in mind when you're using a tool like this that has one of these search features that are part of it. I do think as a clinician who uses CPRS I do think it's important that you need to increase the estimated time for chart review a little bit compared to when you're reviewing something locally in CPRS. Perhaps this is just reflecting the fact that I use CPRS a lot in my clinical practice but I think it takes a little bit longer just because you're switching more you might have some page loading issues doing things in a web-based format but may 10 or 15 percent longer than it might take a chart reviewer to access something in the local CPRS data system. However, as I pointed out some of these additional search features actually might end up saving you some time. So it probably really depends on what you're looking for. I think the important thing to realize is that it doesn't take a tremendously larger amount of time to do an electronic chart review via CAPRI or Vista Web than it does if you're using just your local CPRS system. That would be my assessment having now completed the INSPIRE study and seeing what's going on within our TIA study as well. 

I think it's very important to organize the process for obtaining access. So at the time that the INSPIRE study started the access process did not go through the standard VA DART process. It does now, which again I think has improved it substantially. It's much more straightforward but it's still really important that you designate one person to sort of own that. You stay in communication with them. There are a series of approvals that have to obtained and, of course, any time personnel change on the studies then approval has to be granted to them. That can add achieving that approval for all of your research assistants or personnel that might be involved in this in the study can be something that takes on the order of one to two months at the beginning of your study to get all of those approvals in place. 

And then finally, I think just as someone who's working with chart reviewers doing a lot of reviews I think it's really important to keep your chart reviewers happy. I don't know how many of you have actually tried to look at the screen and read clinical notes all day long with the terrible spelling and abbreviations that you can't understand but it can be a kind of a mind numbing and sometimes frustrating experience. So if you're planning a project that involves a lot of this kind of review I think it's very important that you have you’re abstractors take breaks for other kinds of work even something that seems also kind of mind numbing like data entry or verifying data that someone else has entered can be a nice break from looking at the chart. In that way it's helpful if people can sort of share different tasks across positions. I think it was very important for our team to have regular team meetings to discuss questions, learn how different people were accessing and finding different things, resolve any differences in the way someone might be evaluating or scoring a certain item in our chart review, so that was very important for us. 

One of the things that we did that was kind of fun in our chart review meetings is we ended up giving a prize for whoever had the best story of the week. So that was just something kind of fun that allowed people to share something interesting that they read in the chart. And since our Veterans are awesome doing interesting and kind of brave things, you know, it didn't--I don’t think any weeks went by where we didn't have any interesting stories about maybe a stroke patient who went to the ER and then left and decided to take a plane to go see his girlfriend and ended up in another VA ER in some other city where he actually then got admitted for stroke. So that's just kind of a fun way to kind of keep the team going and develop some camaraderie when you're lots of these chart reviews which can be really quite cognitively draining to do and to do well over a long period of time. 

Some resources and helps that you can get here. So accessing the DART forms for CAPRI and Vista Web access go through the VHA data portal. That is a VA intranet site only. So if you're within the VA you can access that easily. If you're outside the VA you can contact VIReC who can help you with that link. There's also a Vista documentation library that just tells you about the Vista system and some tips about using it and how it's set up and how it's structured. There's also a VIReC tutorial which is called using Vista Web for the first time for research. That is also on the VA intranet. So again if you have interest in accessing that tutorial you can contact VIReC. But I think that those resources are all very helpful if you're just looking at the first time, you don’t have anyone else at your facility who's used a tool like this for research that can be really useful to go to any of these resources and review them because I do think this kind of central chart review is something that is becoming more common within health services research especially within the VA. 

I don’t want to finish without giving lots of kudos to the very amazing group of folks that I work with here in Indianapolis. Most of the people involved in the INSPIRE chart reviews are shown here on this slide and they did just a fantastic job as I mentioned, and I do want to thank the whole INSPIRE study team, not only are HRS&D investigators and bio statisticians but also the engineers under the direction Heather Woodward-Hagg, our VA Center for Applied Systems Engineering, our VA work director here in Indianapolis. Really a project like this would not be possible without such a team of dedicated folks. So I do want to acknowledge them and the work that they did to make our project possible but is the last little bit that I wanted to say. So I think at this time now we're ready to take some questions. And at the end if there's a technical question that I can't answer, perhaps there is someone else from VIReC that might know the answer. 

Arika: Thank you so much. The first question starts off like this. I am not a user of CPRS except to document enrollment in a study. But nursing notes routinely document whether or not the patient was out of bed. Is this not retrievable because it is not a health factor? 

Dr. Williams: Yes. So a nurse note would contain texts that might say something about the patient out of bed. And so that would not be retrievable as a health factor. Actually admitting orders where you have daily activity orders are also not done in a standardized way. So the way that we record activity orders in Indianapolis although they might have a local health factor there would not necessarily be the same health factor at another VA facility. So, for example, this issue I think you're probably thinking of documenting whether the patient was ambulatory is just really, really challenging to do only with those health factors that are available. If you were doing a single site study I think you might be able to figure out the way that site works and the way they document things. And maybe you could get a fairly close approximation with the electronic health factor data but at a multisite study that just would be a very complex thing to do. 

Arika: Okay, here's another question. How do you access Vista Web if not through CAPRI? Did you experience the long waits using Vista Web though CAPRI? 

Dr. Williams:  So Vista Web has its own website that you can go in for when you're looking for Vista Web access to multiple facilities. If you have ever used CPRS at your local facility you probably already know you can--if you're looking at an individual patient's record you can go up to the upper right hand corner and click the Vista Web link and then you see that patient's clinical care, not only at your facility, but at any other facilities they visited. Essentially once you log onto the Vista Web screen and you select--you request approval for whatever facilities you have IRB approval to be doing your research and then that list of facilities shows up, you select from those. So if we were going to be abstracting records from Nashville we would go to the Vista Web website. We would select--put in your regular VA login information, select Nashville as the site you wanted to go to and then put in the patient information. So doing it that way through that original Vista Web website was, again, several years when we were experiencing it, that was when we would have these longer load times. So a TIA study that's ongoing right now currently and has both Vista Web and CAPRI access here in Indianapolis, those folks are accessing Vista Web through that tab in CAPRI and they are not reporting that they're having difficulty with page loading times. So perhaps that's something with this sort of new accessed way of going through CAPRI that has made it simpler or less likely to have problems with that. I also--maybe there's someone else on the call from VIReC or elsewhere that might know that. But we have not been experiencing trouble with long load times since we've been going to Vista Web through CAPRI. 

Arika: Great! Thank you for that. How--where did you do your case finding? 

Dr. Williams:  That's a great question. So the original way that we identified the patients was with ICD-9 codes. We used the traditional stroke, ischemic stroke, ICD-9 codes in primary position for discharge diagnosis in the VA. And then what we did is we had each facility, we wanted the facilities to be somewhat engaged with the project and not just be totally being spoon-fed data from us. So we had the facilities run actually a CPRS reminder report that would generate a monthly list of those patients who had in the prior months, who had a diagnosis code of ischemic stroke. So that formed the initial list of patients that we would review in terms of identifying cases. When we actually--the chart abstractors opened the chart, looked at the record, looked at the discharge summary, there were around 15 to 20 percent that were not deemed to be stroke admissions. Those were excluded and so then we were left with the stroke cases that we went onto do a full chart review in this project. But we did start with that list of ICD-9 codes. 

Arika: Great! And this attendee just wants to ask if you are aware of IMMERSED? 

Dr. Williams:  I am not aware of what that acronym means. I don't know if anyone else on the call is that might be able to type in. But that's not something that I've heard of, no. 

Arika:  Okay. Can you review the process of bringing up the find menu to search within Vista Web? 

Dr. Williams:  Sure. That's one of those little keyboard shortcuts and there may be to do it but simply hitting control F pulls up that find menu and you can just type in then--I think in the example I showed we were looking for in an order summary trying to find when the patient had receive aspirin. Then we just type that in the box. So control F pulls up the find menu within Vista Web. I did--when I was in clinic the other day--just check and make sure that that didn't work when I was looking within progress notes. And as of a couple days ago last week that was something that I was still not able to do within progress notes. But that's definitely feasible to search I know within laboratory results, within orders, within medications. 

Arika:  And I see a comment that says IMMERSE equals Vista Web plus sophisticated text searching. 

Dr. Williams: Oh that sounds like an interesting tool. So maybe we should ask that person to get on the next cyber seminar. I would love to learn more about that to have the ability to do additional text searching within Vista Web. That would be great. 

Arika:  What software installation is required for use of CAPRI? 

Dr. Williams: I don't know if there's one of our CAPRI experts on but really I believe it's just the CAPRI software that ends up getting installed on your computer. I think it's just called the CAPRI software tool. 

Joanne Stevens: Hi Linda. This Joanne Stevens. I'll just add to that that there is a ZIP file of the CAPRI software located on the VHA data portal page and someone can download the software, of course, after they have the appropriate approvals. They might have to get local IT to help support but it's available there. 

Dr. Williams:  Great! Thank you Joanne. 

Arika:  Is the other search tool Vinci, similar to CAPRI and Vista Web? 

Dr. Williams:  No, it isn't and I am certainly not the expert here although I'm using Vinci on another project. But Vinci is really a portal to access VA central data warehouse data. So data that are in the central data warehouse are accessible through Vinci. And so there are certain kinds of things in the central data warehouse and certain things that are not. So not all of the local data end up getting uploaded and cleaned and merged and put into the central data warehouse. So, for example, even a couple years ago the text notes were not part of the central data warehouse. So you would not have been able to look at any kind of notes. That now, my understanding is, that those notes are available within Vinci so you could see a given patient's notes within the central data warehouse using the Vinci system for accessing the central data warehouse. However, I believe that to identify the notes you would look for certain kinds of note titles. I don't believe that you were shown in the same way that I kind of showed you with some of the screen shots where you have all kinds of aggregated notes chronically for an individual patient. That's not the kind of text searching and note searching that you would be able to do very easily through the central data warehouse system. Again, I have not tried to do note review through the CDW using Vinci. So again perhaps someone else that might be able to comment on that. But the notes are available now but not very well put together as part of an actual clinical record for the patient in the same way that you have chronological grouped notes altogether when we see them either in CPRS or CAPRI or Vista Web. 

Arika: Great! Here's another question. It seems like the search function in CAPRI would be comparable to program searches using natural language processing. What do you think about using NLP for this sort of project? 

Dr. Williams:  Well, some of the search functions in CAPRI, I think if you're looking for just an isolated word or a text string would be feasible. The thing that natural language processing can do for you is doing things like negating text. So if something says no carotid stenosis, natural language process would allow you to exclude that from an algorithm where you might be looking for patients with carotid stenosis. Whereas the simple kind of text searching like I showed you with CAPRI is going to identify wherever it says carotid no matter what comes before it. So I think natural language processing, you know, if you're trying to exact specific data out of perhaps a report, for example, an imaging report. Maybe you're trying to extract some very specific data and then deploy that algorithm on a large collection of patient imaging reports. That's where natural language processing, the work that it would take to do to set up that NLP tool might be worth it. In our project we were looking for so many different kinds of things that were only represented in text, it would not have been feasible. We would have needed many, many kinds of NLP engines to find information and it was simply just a lot faster to have people do it. 

Arika:  Another question--I noticed Report Builder button on the CAPRI screen. Have you tried this as a method to save the search criteria to be able to apply the same find string on each patient? 

Dr. Williams: That's a great question and I have not tried to do that. it may be that some of our chart reviewers have done that. But I have not done. If that, in fact, is feasible that would be a great way to kind of standardize some of those searching features that you might be wanting different people on a research project to do. 

Joanne Stevens: Linda, this is Joanne. Can I add a little bit there? 

Dr. Williams: Sure. 

Joanne Stevens: Okay, so when I had experience doing a chart review about 18 months ago we actually used that function and the function is basically for a date function, a date range function as well as the number of records. 

Dr. Williams: Oh okay. 

Joanne Stevens: So it does not save the search term. 

Dr. Williams: Okay, that would actually--like in our example though where we were looking at a given stroke admission and we already knew the admission and discharge dates so that may be how people might be using it then. You could--I guess it sounds like extract just the things from the dates that you were looking for and that might speed things up a little bit for you. 

Joanne Stevens: Right, that's correct. 

Arika: Great! Another question, what about searching within CAPRI? 

Dr. Williams:  Well, within CAPRI you can use that find feature that I illustrated with the carotid example. So again, I showed it with notes but I think other things are searchable in the same way, that find box exists for medications, for orders, for any of the given screens that you might look at. So I think you can use that in the same way even if you're looking for something outside of notes like a medication or a lab result. 

Arika: Okay. Just a couple more just to make sure we have time for attendees to complete the feedback survey. Thanks for the insightful presentation. I have a question regarding the time taken for every chart. It was mentioned earlier an equivalent of three FPEs for three years were needed to review 24 hundred charts. How much time did every chart reviewer take to access a typical chart? 

Dr. Williams:  That's a great question. You know, it's always a little bit hard to say. You have some of those records that are really, really long and some that are really short. I would estimate for our particular project with about a 120 data elements chart reviewers were probably taking somewhere like 30-40 minutes on average. Some would be shorter if the stay was very short. Some would be longer. But I think that’s an important thing that you do when you're first developing your chart review tool to know, in fact, if it is going to be feasible for you to get the number of charts that you think you need. 

Arika: Last question. Does CAPRI allow you to view scanned documents? 

Dr. Williams:  That is a great question and I'm hoping Joanne can answer that. Vista Web does allow you to view scanned documents. I assume that CAPRI does but I don't actually know that. Maybe Joanne can help us. 

Joanne Stevens: Actually CAPRI does not. If you do have access to Vista Web through CAPRI then you can access it that way. 

Dr. Williams: Oh, that's a great question. So that would be another argument for having access to both because you're right. Sometimes, for example, in our project looking at what happened in the emergency department sometimes required us to look at scanned documents because, again, even as with--up to a few years ago not all VA emergency departments we documenting everything in the electronic record. So we did need to look at those scanned documents and it's good to know that's something then that would be via Vista Web, not via CAPRI so another argument for kind of getting access to both when you're starting your study. 

Joanne Stevens: Right. This is Joanne. Just a little caveat to that is that if someone is going to look--want to look at images there is specific software for that. it's called advanced web image viewer or AWIV and that information is available on the VHA data portal as well. 

Arika: And one person is asking Dr. Williams if you can provide your email address? 

Dr. Williams: Oh sure. I am Linda.Williams6@VA.gov. 

Arika: Great! For questions, additional questions that have not been answered please contact the VIReC help desk at virec@VA.gov. Thank you to Dr. Williams for developing and presenting today's lecture. Our next session is scheduled for Monday April 7th from 1:00 to 2:00 pm Eastern and it's entitled measuring laboratory use and results using the VA DSS National Labs data. It will be presented by Anne O'Hare. We hope that you can join us. I hope that everyone has a great afternoon and thank you again to Dr. Williams. 

Dr. Williams: Thank you Arika. Take care everyone. Bye. 

Page 1 of 16