**Transcript of Cyberseminar**

**VIReC Clinical Informatics Cyber Seminar Series**

**Knowing and Doing: Automating Performance Measures and Clinical Decision Support**

**Presenters: Mary K. Goldstein, MD; Tammy S. Hwang, BA ; Kaeli W. Yuen, BS**

**July 16, 2013**

**Knowing and Doing: Automating Performance Measures and Clinical Decision Support**

**July 16, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact: virec@va.gov.*

Moderator:Good afternoon and welcome everyone, this session is part of the VA Information Resource Center’s ongoing Clinical Informatics Cyber Seminar Series. The series aims are to provide information about research and quality improvement applications in clinical informatics, and inform about approaches for evaluating clinical informatics applications. Thank you to CIDER for providing technical and promotional support for this series. 

Questions will be monitored during the talk and the Q and A portion of Go To Webinar, and will be presented to the speakers at the end of this session. A brief evaluation questionnaire will appear when you close Go To Webinar. Please take a few moments to complete it. Let us know if there is a specific topic area, or suggested speaker, that you would like us to consider for a future session. 

At this time, I would like to introduce our speakers for today; Dr. Mary Goldstein, Miss Tammy Hwang, and Miss Kaeli Yuen. Dr. Goldstein is a health services researcher with an emphasis on informatics and a geriatrician who serves as director of the Palo Alto Geriatrics Research Education and Clinical Center, in the VA Palo Alto Healthcare System. She is also professor of medicine at the Center for Primary Care and Outcomes Research at Stanford University School of Medicine. 

Miss Hwang is a Health Science Specialist and Miss Yuen is a Clinical Research Associate in the VA Palo Alto Health System. 

Without further ado, may I present our speakers for today? 

Dr. Goldstein:Hi, well, thank you very much for the introduction, welcome to everybody, glad to be here, good afternoon or good morning, depending upon what time zone you’re in, and we will get started. I want to first check with Erica; can you see my title slide. 

Moderator:Yes, I can. 

Dr. Goldstein:Okay great, and the speakers have already been introduced, and I will just mention that any views we express are those of the speakers, not necessarily those of the VA. 

The second slide shows our investigator team for the heart failure performance measure automation. I will not read the entire list, but I wanted to include it here as a slide for future reference, and to note that we have wonderful collaborators who have contributed enormously to the many different projects we have. We also have further acknowledgement slides at the end. 

The goals for this session in our seminar today, we are hoping that by the end of this seminar, the participants will be able to describe the differences between automated performance measurements and automated clinical decision support. Describe the challenges and opportunities in automating performance measurements, explain the steps of encoding existing performance measures as well as new quality measures for automation into computable formats; and to compare and contrast different approaches including database query approach versus knowledge representation approaches to automating performance measurement. 

The next slide is a poll question. Erica I turn it to you, and so the poll is open and we are asking that you help us understand our audience. So please select as many of these as apply. A. I am interested in potential applications of software. B. I am interested in the underlying technology for the software. C. I do clinical work as a licensed health professional at VA. D. Quality assessments or measurement is part of my work; and E. Research is a substantial part of my work. So, take a moment to fill those in and Erica will let us know when we have the responses. 

Moderator: We are at about seventy percent right now, so I am going to give it just a couple more seconds to get a few more answers here, and then I will show the results. 

Dr. Goldstein: Okay, so it sounds like about sixty-eight percent indicated they are interested in the application of the software, and fifty-four percent in the technology. Twenty-five percent work as licensed health professional at VA, a full seventy-four percent indicate the quality assessment or measurement is part of their work, and about forty-two percent, research is a substantial part of the work. Thanks for helping us understand who is here, and we will try to adjust details of what we say to go in accordance with that. 

We should be back now to my screen and showing the outline. Erica will let me know if that is not the case. In order to achieve the goals we set out for this session, this is the outline we plan to follow. Give some general background, and then talk about encoding existing performance measures. Then describe system development for performance measurement versus clinical decision support; and finally address approaches to automating performance measures with database queries versus the knowledge-based approach. 

Starting on the background, moving from clinical decision support to performance measurement, many of us are very aware of the modern problem of too much information. James Gleick, who wrote the wonderful book, *The Information*, talks about the devil of information overload and his busy, impish underling, in which he includes the PowerPoint presentation, so we need to keep some sense of humor about our own presentation. 

In thinking about the too much information in medicine, it includes the enormous volume of medical literature to stay on top of, in addition to some of the other items listed on that slide. But addressing... now on the next slide... addressing the issue of the volume of medical literature, there is... one way to filter information is to synthesize this vast clinical literature into clinical practice guidelines written in ways that are actionable. 

So you go from evidence to patient care by providing decision support with actionable guidelines. The emphasis here is on actionable. One way to do this is to encode evidence based clinical knowledge into computable formats and then link it with patient data to present to the clinician. 

Why do we want to make the knowledge, the clinical knowledge, actionable? Well, one way of thinking about that is knowing and doing. The famous quote from Goethe of “knowing is not enough, we must apply. Willing is not enough, we must do.” So we are looking for ways to help ourselves and others with doing as well as knowing. 

The next slide shows an image of a simple diagram concept of performance measurements and clinical decision support. As with all models, it is over simplified, but for purposes of discussion, we can think of both clinical decision support and performance measurement as being based on clinical evidence often summarized into clinical practice guidelines. It synthesizes a great deal of evidence. 

The emphasis on clinical decision support is to improve care through timely information and advisories. The emphasis in performance measurement is to monitor the quality of care to improve care. And as is often said in quality management; if you can’t measure it, you can’t improve it. So performance measurement is important. 

It is also the case that performance measurement can be fed back to improve care, and potentially can even be incorporated into clinical decision support fore close to real time feedback. Not something we have done yet, but a direction we see for the future. Before we can run, we need to walk. Before we walk, we need to crawl. So we are going to focus first on the performance measurement apart from clinical decision support by being informed by what we have learned from clinical decision support. 

Slide eleven shows what information might a performance measure display, probably familiar information to people who already work with performance measures. An example from heart failure, is automated computation of two heart failure performance measures for appropriate patients and these would include the use of ACE inhibitors or ARB’s, heavy use of Beta Receptor Antagonist, Beta Blockers, for appropriate patients; clinical settings for each of these, and out patient to in patient. We have been working to develop automated measures for these doing computation on patient data from the electronic health record, which has been stored in VA’s Corporate Data Warehouse, known as CDW, and computed and displayed on DaVinci secure server. 

The next slide shows an example of the types of data that can be displayed. This is slide twelve, which gives a summary, and not in something designed for great user interface, but just something for a quick first pass at pulling out information. These were results computed by the system we developed operating on Vinci, using CDW data for many of the data inputs, but using simulated ejection fractions. I will mention more about that later. These are not actually performance measures of any actual patients; but this shows the types of information. We will go through this is more detail later, but it shows which are the measures, which NQF measures, and the summary outcomes that shows what were the exclusion criteria applied, and sorts patients excluded by each of those separately for out patient and in patient exclusion. 

The next slide, slide thirteen, we are going to now tell you about how we generate the information on the previous slide. We use a knowledge based, knowledge representation approach. So, we built this based on our previous clinical decision support work, which was based on development by Mark Musen and Samson Tu of Stanford Biomedical Informatics Research, originally done for the ATHENA-Hypertension Clinical Decision Support System in which we encode clinical knowledge into a computer interpretable knowledge base. For those who would like to read more about this, there are several references on the slide for early development of the system. 

As a quick introduction to this system known as ATHENA Clinical Decision Support System, we started with hypertension as a highly prevalent condition for which there were excellent guidelines. But some evidence people were not following the guidelines, and built the system that was intended as a prototype and early proof of concept, with a plan to extend to other areas if successful. So, clinical knowledge represented in computable formats, can be used for multiple purposes. It can be used for clinical decision support as we have already done, and you can include quite extensive nuances and complexities. Our hypertension system has hundreds and hundreds of grains of knowledge, so you can get into quite a lot of detail about specific clinical situations. But you can also use this to do quality measurement. The quality measurement then has potential to take account the complexities that go far beyond simple performance measures. We will deal with that a little more later. 

Slide seventeen shows some of the sites that we used for our ATHENA Multi-site hypertension studies. In the early 2000s, we did a three-site study at the sites shown in black, San Francisco, Palo Alto, Durham; and later in the 2000s in VISN1, which is New England Healthcare Network, and these involved in the groups randomized to receive the system, more than fifty primary care providers in each, and receiving advisories about thousands of patients. 

The overall architecture for that system is shown in slide eighteen. There is a patient database, which in our case is from the VA VISTA data, pulled into SQL Server. If you start with corporate data warehouse or regional data warehouse, you already have it in SQL Server format. There is a separate knowledge base that encodes the clinical knowledge, as I mentioned before. These are pulled together to a guideline interpreter execution engine that processes these to develop conclusions about the state of the patient, recommendations for next steps and therapy, which can then be sent back various places. One of them can be to send back for display within CPRS about the patient who is being seen. 

The next slide shows an example of CPRS cover sheet, and the way the ATHENA hypertension advisory appears on top of it. I do not think this is even for the same patient; it is just to show approximately what it looks like. 

The next slide shows a newer version of the user-interface that was developed with group primary care providers around the VA and in designed company. 

Slide twenty-one talks about moving from clinical decision support to performance measurement. In the process of going through the determination of the state of the patient in the execution engine applying the guidelines to the patient data, we make a lot of conclusions about what is this patient’s current management and current achievement of targets with respect to the evidence-based clinical practice guidelines that apply? We are looking at what the clinical scenario the patient is in, which can be an ascertainment of quality of care. We were interested for a very long time in how this could potentially be used as a way to do performance measurement, or quality measurement. 

Mentioned on the next slide are some of the limitations of simple performance measures. For example, a simple measure is for patients with hypertension who are not diabetic; there is a blood pressure target. Less than one-forty over ninety is tolerated, this is extremely useful in healthcare systems in which people have not been intensifying therapy or adequately working with patients on achieving their regimen, and they have a large performance gap. But as control gets better and better, then the proportion of patients left for whom it may not apply, becomes a larger proportion of those who are left, and there are a lot of places where it does get complicated. For example, patients who have a very low diastolic blood pressure and ischemia and potential risk from further attempts to lower the systolic. Patients with risks of falls, patients already taking four or more anti-hypertensives, which may place them outside of the evidence realm, and other situations in which it gets quite complicated. There is a need for more complex performance measures that promote optimal care requirements using detailed clinical data and complex measures. 

The overly complicit performance measures either include in the denominators the patient cases for whom it doesn’t really apply, and which can lead to skepticism among the clinicians who say gee, this just doesn’t include my patient. Or, alternatively, they may exclude so many patients from the denominator that they work well for the patients who are left in the denominator, but they don’t apply to a large portion of patients with the target disease and have nothing to contribute to quality of care for those patients. 

Getting started on using modern systems for advancing automation, we note that the VA is a national leader in quality of care, and the VA is well positioned to advance performance measurements because of its highly sophisticated systems. It has been well studied and referenced by many people.

This takes us to the next step of encoding existing performance measures. To get started on this work, we were fortunate to have funding from a query rapid response project to try to develop it for heart failure. The National Quality Forum is a non-profit organization that seeks to improve the quality of healthcare, reviews, endorses and recommends standardized performance measures; and we introduce who they are because we will be referring to some NQF measures. 

I am now going to turn this over to Tammy for the next segment. 

Tammy Hwang:Hi, this is Tammy and I will be talking about a project that we recently completed. We completed our one-year project, funded by Queri Heart Failure, titled Guidelines to Performance Measures Automating Quality Review for Heart Failure. The primary aim of the project was to develop a prototype to automate the computation of heart failure performance measures.  Specifically we attempted to automate performance measures that were considered high priority by the VA performance measurement staff. These included NQF 81 which addresses the use of ACE Inhibitor and ARB therapy in heart failure patients; and NQF 83 which addresses the use of Beta Blocker therapy in heart failure patients. 

These two performance measures are stewarded by The American College of Cardiology Foundation, The American Heart Association and The Physicians Consortium for Performance Improvement. We automated an in patient and out patient version of both of these measures and the performance measures were computed using data from CDW with output displayed in tables on Vinci using SQL Server which is a database management system. 

Next we have a poll question for our audience. Please tell us which of these you are familiar with, if any. Select all that apply. The first one is I have some familiarity with NQF measures. I know where to find items in clinical charts; and I know how to map concepts to standard codes. For example, ICD and Link codes. 

Moderator:Responses are coming in, but we will give it a few more seconds to get some more answers in here. Okay, and there you go. 

Tammy Hwang:Thank you. It looks like a little more than half have familiarity with the NQF measures; sixty-one percent know where to find items in clinical charts and twenty-six percent are familiar with how to map concepts to standard codes. So, I’ll be talking a little bit more about the last one in my presentation. 

Can you see my screen okay now? 

Moderator:Yes, we can. 

Tammy Hwang:Okay great. Now, I am going to talk a little more in detail about the process of encoding performance measures. Here is an excerpt from a performance measure, which addresses beta-blocker therapy. The measure is generally described as the percentage of patients with a diagnosis of heart failure with a current, or prior left ventricular ejection faction less than forty percent, who are prescribed beta-blocker therapy either within a twelve-month period when seen in the out patient setting, or at hospital discharge. This was pulled from the ACC performance measure packet. 

On initial look, the performance measure may appear to be fully specified, however whether operationalizing these performance measures through manual chart extraction or by automation, there are some complex questions that arise. For instance, what data domain should we be using to compute these measures? What are the time constraints that should be applied to the data? How do we define the denominator factions? What are the ranges that should be applied to lab values and vitals? And what should happen when pertinent data is missing?

To delve a little bit further into the encoding process, it will be useful to look at the components of a typical performance measure. For NQS 83, which was the one that I showed earlier, the numerator is described as patients prescribed beta-blocker therapy either within a twelve-month period when seen in the out patient setting, or at hospital discharge. The numerator describes the set of patients whose medical regimen during the measurement period adhered to the guideline recommendation. 

The denominator is described as all patients aged eighteen years and older with a diagnosis of heart failure with a current or prior left ventricular ejection faction of less than forty percent. The denominator defines the set of patients eligible for a given recommendation during a performance measures measurement period. The denominator excludes patients who have specific medical system or preferential reason for not adhering to the standard therapy. These are typically referred to as the denominator exceptions. 

Denominator exceptions rely heavily on clinical judgement. The numerator over the denominator produces the percentage of patients who met the goal for a given performance measure. 

Encoding performance measures was a process of breaking down the measure into individual concepts. For instance, taking apart the beta-blockers performance measure, the denominator has several criteria to consider. For the outpatient beta-blockers measure, we pulled all patients who had an ACPT code for an outpatient visit within a defined performance measurement period. Those patients were further filtered to the ones who had an ICD9 code for heart failure and were at least of a certain age. We also filtered the patients to those whose most recent EF measurement was less than forty within a defined period. For the numerator we considered the patient’s prescription history and used drug IN codes in the pharmacy data domain to determine whether the patient was on beta-blocker therapy. 

The denominator exceptions, also known as the exclusion criteria, were the most complex to apply because there were so many to consider. We applied eighteen exclusion criteria for the beta-blocker out patient performance measure, with input for heart failure subject matter experts. NQF 83, which was the performance measure that we were primarily attempting to encode, had a broad description of denominator exceptions as follows documentation of medical, patient or system reasons for not prescribing beta-blocker therapy. In order to better specify exclusion critera, we consulted other heart failure performance measures such as NQS 615, which had more specific exclusion criteria such as aortic stenosis, hypertension, and metastatic disease. 

For the ACE inhibitors in patient performance measure, we applied thirty exclusion criteria adopted from several NQS measures, 81, 162 and 610. These included exclusions such as heart transplant, pregnancy and hyperkalemia. Each of these criteria have to be mapped to these specifically coded patient data. to give an example, some of the performance measures excluded patients with chronic kidney disease stage three or four, so we operationalize this by excluding patients who met either of the two criteria shown. We excluded patients who had an in-patient admission diagnosis, IC9 code for chronic kidney disease stage three or four any time prior to the end of the performance measurement period. We also excluded patients who had an eGFR value between fifteen and sixty. For this particular exclusion we used two different data domains, the in-patient encounters as well as the Lab Chem and Data Domain. 

For each concept, we were defining DCDW data source, the time period search for the data, rules for missing data, and the data range where applicable. 

Here is a flow chart, which gives an overall picture of the automated performance measurement system, the VA patient population is filtered to those patients who remain after denominator and exclusion criteria are applied. This produces the set of eligible heart failure patients to be considered for the performance measure also known as the denominator. Then a system searches for the numerator criteria, and this produces the final measure, which is the percentage of eligible patients who met the performance measurable. 

In our prototype system, the summary results are displayed in a SQL Server display on Vinci, which shows summary numbers and we may work on elaborating on this summary display in future projects. Now I am going to turn it over to Mary, who will talk about encoding the equality measures based on clinical practice guidelines. 

Dr. Goldstein:Some of the high priority heart failure practice guidelines have not yet been officially endorsed as performance measures, but are regarded by the subject matter experts for heart failure in the VA as being very important quality measures, so we began work on specifying these. The next slide shows an example of those, the hydralazine and long acting nitrates, aldosterone blockers, cardiac re synchronization therapy and the implantable cardio defibrillators and the source of those guideline recommendations. We have done the specifications for these, but not yet done the encoding of them in the knowledge base. 

The next step is that we are going to have Kaeli talk about the system development performance measurements versus clinical decision support. 

Kaeli Yuen:Hi, I’m Kaeli and for the next segment I will be talking about the differences between the ways that CDS and performance measurement systems are designed. 

To review, both CDS and performance measures are derived from clinical practice guidelines, but the two types of systems have different approaches to improving care. While performance measures are designed to improve care through feedback from monitoring quality of care, CDS is designed to improve care through timely information and advisories. This leads to a number of differences in the ways that the practice guidelines are operationalized for the two types of systems. 

First, CDS and performance measures use different methods for selecting the cohorts of patients to whom the system will be applied. For CDS the system must be available with timely information at point of care. That is, when the patient is being seen and the health professional needs the information for immediate clinical action. The system attempts to offer useful recommendations for as many patients as possible with the target condition and it uses simple eligibility criteria to filter the patients. For example, ATHENA heart failure CDS uses two criteria, left ventricular ejection fraction less than or equal to forty percent, and  IC9 code for heart failure as the two criteria for including patients in the population who are eligible to be considered for recommendations. 

In contrast, performance measures systems can process data retrospectively for patients with encounters during a defined performance measure period. For our heart failure RRPs that Tammy just talked about, we defined the performance measure period to be the twelve month period from June 2010 to June 2011. Performance measures systems attempt to exclude patients with the target condition for whom a particular narrow recommendation may not apply by using complex inclusion and exclusion criteria to filter a potentially large population. 

Once the initial potentially large population of patients has been identified, it is filtered down to one denominator population, and one numerator population for each measure. The numerator and denominator populations are used to compute the percentage of patients meeting the goal. CDS, on the other hand, makes granular distinctions among patients creating fine-grain sub populations and giving different recommendations for each. 

This slide shows a sample from a visual representation of a hypertension guideline that we have created in Protégé and demonstrates that there are many decision making notes that produce these fine grain sub populations. The result is essentially a collection of performance measures that deal with patients who have a common diagnosis. 

As I mentioned before, performance measures use much more complex inclusion and exclusion criteria than does CDS. This is because of performance measures. We want to give our providers the benefit of the doubt, so patients are excluded liberally from the measures. On the other hand, with CDS, we want to provide recommendations for as many patients as possible who may benefit, so the exclusion criteria are less strict. Instead, there is an attempt to modify the recommendations to what would be correct management for patients in special circumstances. 

This visual here is to give you an idea of the contrast between exclusion criteria for CDS and for performance measures. Using the exclusion criteria for beta-blockers as an example, patients with any of the conditions listed here on this slide would be excluded from the beta-blocker’s performance measure, but only patients with an allergy or adverse reaction, as shown in the blue circle, to beta blockers would be absolutely excluded from the beta-blockers recommendation of CDS. 

It is important to note that while the conditions on the right do not exclude patients from receiving a CDS recommendation for beta-blockers, many of them are considered by CDS to be relative contraindications to beta-blockers. This means that he CDS will display the beta-blocker recommendations along with special recommendations for the prescriber to consider. 

Another way in which CDS and PM, or performance measures, differs, is their use of historical data. CDS is mainly interested in current labs, problems, and medications, with occasional use of historical data as eligibility or exclusion criteria. On the other hand, performance measures are mainly interested in data from the performance measure period with occasional use of data from a window of time prior to that period. Performance measures do not work with any data generated after the end of the performance measure period. 

In terms of modeling, CDS and performance measures look different as well. CDS is structured in accordance with the elements of clinical practice guidelines. These elements are eligibility criteria, patient characterization such as the risk group of the patient, goals from the guidelines, clinical algorithms and recommendations and messages. performance measures are structured more simply being calculated using just the nominator, numerator, inclusion and exclusion criteria as we saw with our example that Tammy showed. 

Lastly, the output of the two systems is very different. As you saw briefly, earlier in the presentation, the output of ATHENA CDS includes goals, messages, recommended actions such as ordering labs, procedures or referrals, and recommended changes to drug prescriptions with collateral messages. These collateral messages are the ones I mentioned earlier, that could contain warnings about relative contraindications to certain therapies. 

This slide, which you have seen before is an example of ATHENA hypertension outputs. Here you can see in the red... here is a display of the goal, and under that are recommendations and under that therapeutic possibilities for changing prescriptions or adding certain prescriptions. 

The output of performance measures software is different from that of CDS. The tool developed in our heart failure RRT, outputs the adherence of each eligible patient’s treatment to the guideline recommendations, however, we do not have an example of that to show here because it contains patient data. The system also outputs summary data on the percentage of patients whom those performance measure goals during the performance measure period and on the number of patients excluded for each exclusion criteria. 

This slide, which you have also seen before, shows two outpatient and two inpatient performance measures as well at the criteria for exclusion and how many patients met each exclusion criteria here in this area... number of outpatients excluded. That is my last slide, so I will hand it back over to Mary to talk about approaches to automated performance measurements. 

Dr. Goldstein:Thank you, Kaeli. Our last few minutes here before we move to questions, we are going to discuss some of the pros and cons of the alternate ways to automate the performance measures. 

Slide fifty-eight shows the SQL Server approach. This uses a collection of SQL queries to evaluate the sets of patients. One of the advantages or pros of this approach are that you can define all the criteria using one well known formalism of SQL type queries, and you can evaluate the queries on large sets of patients at one time. 

Some of the disadvantages are it’s not as easy to manage a large collection of SQL queries and there’s a need to write multiple queries for each criteria to keep track of which patients satisfy the criterion and have missing data. 

The next slide shows the knowledge based approach, which is the one that we have used, and has been mentioned before, we are drawing data from SQL and we end up outputting our results to SQL and then use SQL queries to display the results; but in between, there’s processing by the knowledge base and the execution engine.  In this approach of a knowledge based, plus execution engine such as in our clinical decisions port system; but revised specifically for performance measurement. The advantages are that the criteria are managed within Protégé projects, which makes them very easy to view and work with and manage, and also that the execution engine computes and records the desired information for each criterion, which is much easier than having to do this separately for each separate step in SQL. 

The disadvantages are that you do have to have the execution engine custom programed for the requirements and performance measurement, and you need to compute the performance measures one patient at a time, and then output that to create the summary data. Which approach to choose depends on the needs of the specific project, and may also depend to some extent on the skill sets available. 

In future directions, we now have a prototype system that computes the ACE-R and beta-blocker measures as we described, but we want to move forward to develop it into a fully functional system for automating performance measures for heart failure. One very important piece of it is for us to get the ejection fractions which are not available at structure data elements currently, but which can be converted to structure data elements by information extraction procedures such as natural language processing on echocardiography reports and other reports. An example of the system for doing that is the commander EF system, which has been developed and evaluated in a project led by Jennifer Garvin and was published in JAMIA in 2012. there is further work going on now with her new system called CHIEF to further automate that and we want to link those up so that the ejection fractions coming out of those systems can be used as input to our performance measurement system. 

Another future direction is to move from the specifications we have done and code those additional quality measures to actually encoding them for processing... for the automated processing. We would also like to work with stakeholders who would be using the outputs of this to improve the user interface design. We are also working on a current project with the VSN21 PBM who have developed a clinical dashboard that’s very widely used in VSN 21, and triggers for groups of patients or individual patients who are not needing performance measures. We are developing clinical decision support to add into their system, and then we are also developing these more complex performance measures that can take account of the patient’s clinical nuance. 

I would like to acknowledge our funding sources from VA HSR and D, as listed on this slide. There have been different grants for different projects. And to mention some of our collaborators including Paul Heidenreich, who is the PI for Queri Heart Failure. Samson Tu, Susana Martins, Amy Furman, PharmD, with VSN 21 Pharmacy Benefits Management, Brian Hoffman who has been an enormous source of input, especially for hypertension; Dan Wang, Elaine Furmaga, Barry Massie, and many others who we are listing in many of our different papers and other places. I would also like to thank Vinci for providing the secure server workspace for this work. We will move now to just thank you for your attention and see if there are questions. 

Moderator:Thank you, we do have a few questions, here is the first. Is Protégé available in the Da Vinci environment? 

Dr. Goldstein:Yes, we have Protégé operating. I do not know the exact details. The person who wanted to use Protégé on Vinci would need to talk with Da Vinci staff about getting it installed, but we do have this system running on Vinci and if someone had questions about that, they could contact us separately afterwards and we will be happy to talk with them about it. 

Moderator:Did you have a group of clinicians that helped you to define exceptions, parameters, et cetera? 

Dr. Goldstein:Yes, we did in the sense that we were working for heart failure with the Queri Heart Failure group, and we referred questions to them, but I do not hold them responsible for everything we have yet, because we consider what we have so far. It is still a work in progress. Another part of the future work we would like to do is to have all of that vetted in more detail. We have developed a technical manual that runs about fifty pages or so, that has the explicit lists of everything that was a decision. It includes in it a number of places where we say, this or that needs further checking and so, if this is to go forward, which we hope it is, it will, then we hope there will be a broader group of quality managers and clinicians who will help with that. We did have a lot of help from Paul Heidenreich on some of the key decisions in the early development. 

Moderator:Are the mapping algorithms for your inclusion/exclusion criteria publicly available? Specifically, it would be useful to know which data domains you are extracting data from, and the code sets for defining conditions. 

Dr. Goldstein:Where we would like to go with this is we would ultimately, eventually like to make the codes sets available to people who could access them through Vinci. We currently have all of it in the Vinci work space and it is up to the people who manage the project on Vinci to... there are some issues about... things are set up according to a particular project and you need your IRB and your approval, whatever, to work with things on other projects. But the goal is to make it all available, and to make the code available. The code does not actually have patient data in it, but to make the code available. So as I said, we have this specifications manual which still needs a lot of input from more people and so we are not posting it on a web site because it’s a work in progress, but we are very happy to share it with other investigators or quality management people who would like to have a look at it. We also hope that the code sets can be shared through Vinci. 

Moderator:I should think that one of the biggest challenges is gaining consensus on definitions and terms clinically. For example, we are struggling because people confuse CHS with left ventricular systolic dysfunction, and the consequences of confusing just those two terms are immense. Can you comment on this? 

Dr. Goldstein:I would say I agree. This is... we have found this consistently through our clinical decision support work too. The hard part is not the encoding, the logic, and the technical side, although there has been a lot of advances and interesting work in that. The most time consuming thing is pinning down clinical agreement on exactly what that person commented on. Now, the heart failure includes both diastolic dysfunction and systolic dysfunction and there are a lot of people that have a diagnosis of heart failure, but do not have systolic dysfunction, so that’s one area. There is a host of other things about exactly what criteria will be used to define. Where they are available, we like to use published specifications that have been vetted. For things that are done by EPRP, we use that manual. For things in NQF as Tammy was describing, we will pull from there, but we find that there is no one source that has it all completely spelled out and so we do often need to turn to the clinical owners of that clinical domain to say do you agree with using this or that; or what would you recommend. 

We have seen that there is a very nice move toward guidelines authors putting in ICD9 codes, and to the extent that we can all encourage both performance measure writers and guidelines writers to put in the standardized terms that will link to the patient data. That is great so the people in the appropriate authority to make that decision are making the decision. That is great, but we are not all the way there yet, so we do need to keep working with clinical groups on it. 

Moderator:Do you think giving providers information on the specificity, or PPV for the CDS recommendation would help them evaluate the applicability of the CDS recommendation? 

Dr. Goldstein:Oh, I think that is a great comment and a very thoughtful comment. It is an area that requires I think a lot of future research. There is probably people working on it now, and I hope that gets incorporated. I think that it is very helpful in clinical decision making, to understand what its potential benefit, and also the potential risks and harms from following any particular recommendation. This would come from analysis of the evidence base, and then there’s also a step of having to extrapolate from the patients who are the ones who are in the clinical trials that form the evidence base, to the patient for whom you’re actually providing care. The patient in front of you who might be, in some ways, quite different from the patients who were in those clinical trials; but I think the more that we can pull out that kind of information and make it available in clinical decision support, the more useful the CDS will be, so I like that question. 

Moderator:Why did you rely solely on SQL instead of using SAS as your primary language in encoding measures? 

Dr. Goldstein:We actually did not encode most of the measures in SQL. The approach we took was encoding the knowledge to compute the measures as clinical knowledge of performance measures in Protégé. What we used SQL for was to get the... get the data, because the CDW are in SQL formats already. Then because we had a SQL environment, and the data were in SQL, we then just used SQL to do the output to SQL and used SQL to do database queries to display the results. It was just very simple and straightforward to do it that way. If you... I guess what is being suggested is in the alternative approach of writing the whole thing without encoding in Protégé, or another knowledge-based system, but just by using another tool, in addition to using SQL, perhaps that could be done in SAS. That is just not something we have explored. It was just very straightforward to go from SQL to SQL. 

Moderator:Is the data scheme shown in the slide representative of how you would define a cohort in Protégé? Where can I learn more about Protégé and are your algorithms for defining conditions described by Protégé ontology? 

Dr. Goldstein:I may need to ask you to take some of those questions piece by piece. I am trying to go back through the slides if people still see them, there is a slide entitled “Starting with Knowledge Representation” that has some general information about Protégé, which is an open source Java tool. It is very widely used internationally, has a huge user group, and there is a link on Protégé, it is at Stanford that has a free download for Protégé. It also has some introductory online information that can be used for initial education about Protégé, and then for people who want to get into Protégé in more detail, Stanford conducts Protégé short courses. I think they are about a week long, from time to time for people who want to get into more detail with it. What were the other pieces of that question? 

Moderator:This is regarding the filtering, the population in Protégé slide... is the data scheme shown in the slide representative of how you would define a cohort in Protégé? 

Dr. Goldstein:I am not sure I understand the question. I am not sure which slide it’s referring to; we had the flow chart of filtering the population, and it may be that that is what that is about. I think this might be a question that we need to have people contact us separately for us to be sure we are answering the question that is being asked. 

Moderator:Okay, and the last one related to that was just are your algorithms for defining conditions described by Protégé ontologists? 

Dr. Goldstein:We do use Protégé ontologies and then there’s a guideline management system within Protégé, so I think that yes, these are really good technical questions that would probably be best for us to take off line and if someone is truly interested in this, we will bring in Samson to help with answering those questions. 

Moderator:Looks like that is it for our questions unless the attendees have any more that they would like to type in. 

Dr. Goldstein:On the last slide, contact information for Tammy and Kaeli, who are good folks to get in touch with for follow up on any of these questions. 

Moderator:Great, thank you so much and thank you to all of our speakers for taking the time to develop and present this talk. Please forward any remaining questions to VIReC Help Desk at virec@va.gov and we will forward to the speakers. 

Our next session is scheduled for September 17, at 12 p.m. Eastern, and it is entitled Using Patient Facing Kiosks to Support Quality Improvement at Mental Health Clinics. It will be presented by Dr. Amy Cohen. We hope that you can join us. 

Dr. Goldstein:Okay, thank you. Bye. 

Moderator:Thank you and for our audience once again, as you are leaving today’s session, you will be prompted with a feedback form, if you can take a few moments to fill that out, we would very much appreciate it. Thank you and we hope to see you at a future HSRD cyberseminar. 

Page 1 of 14

Page 14 of 14