12/20/2011  Relationship of Usability and Patient Safety with Health Information Technology VIReC Clinical Informatics Seminar Presented by: Patterson, Emily

This series aims are to provide information about research and quality improvement applications in clinical informatics and also information about approaches for evaluating clinical informatics applications. Thank you to CIDER for providing technical and promotional support for this series. Questions will be monitored during the talk in the Q&A portion of GoToWebinar and VIReC will present them to the speaker at the end of her talk. A brief evaluation questionnaire wil lappear when you close your GoToWebinar today, we would appreciate it if you would take a few minutes to complete. Please let us know if there is a specific topic area or suggested speaker that you would like us to consider for future sessions. At this time I would like to introduce our speaker, Emily 

Patterson, PhD. She's an assistant professor in college of medicine school of allied medical professions at Ohio State University. Her work is familiar to many since she was formally at the Cincinnati VA which she left in 2008.

I want to make sure that I acknowledge my contributers on this presentation. It is basically a summary of multiple papers that are listed in the references at the end. There are many people who contributed to this work and I apologize if I've forgotten anybody. Some of this work even dates back to work done at the VA in 2000. 

The first thing I wanted to do was start off with a poll about who is in the audience. I want to get a feel for who is here today. If you could please answer the question as to which best describes you.

It looks like they have put up the poll and we have 48% researchers, 13% programmers, 11% usability expert, 7% patient safety experts, 22% administration/policymaker. 

My objectives today are to cover a perspective on health information technology from macro cognition. From a view of complex work and how transformative health information technology plays into transforming that complex work. I wanted to define usability. I take the international standards definition and primarily the focus of the talk is what is the relationship between usability and patient safety with respect to health information technology and in particular electronic health records. I would like to discuss a little bit about the usability testing methodology. What are the various options that are open to people. I want to discuss some of 

the challenges in creating really complex scenarios that ensure complexity. 

The era of health information technology and complexity, so there is plenty of work for myself and anyone else interested in the area of how technology can make work complex. I just had a fascinating discussion with the radiation oncologist about all the innovative technologies and how complex work is getting in that area. Certainly everywhere in healthcare there is transformative nature of electronic health records and barcoding for medication administration. This picture shows a tele-ICU, that's another interesting application, all these devices that are doing all this advanced monitoring for patients. There are plenty of areas where technology is increasing our choices and benefits of technology and then we have this responsibility to think about proactively, how all these technologies might create threats to safety and redesign them before accidents occur. This is not a new idea. The IOM report put this out years ago. Probably the biggest example on the horizon right now is electronic health records. At the VA people can take it easy as others go through this, but obviously there are is the other transformative HIT at the VA going on.

So what is it that people complain about with transformative information technology? The general complaints fall into categories like workflows that do not match, so there can be sometimes relatively rigid workflow design embedded within health information technology that can create inefficiencies when they don't match the clinical processes. People complain about the screen design possibly slowing down the user and possibly endangering patients. Many people, particularly with electronic health records have complained about large numbers of files, especially if they are across different platforms and different systems and how do we navigate, sometimes we cannot even search over them. How do I indentify trends over them. Error messages have been an area where many users complain, particularly for personal health records we have to worry a lot about conflicting error messages given to that population. Elisa Roth and Jason Sween have been doing a lot of fascinating work around alerts, when do users ignor them, are there too many, do they potentially have critically important information but then are lost in an overload. And obviously there's many opportunities to reduce down the number of clicks to do things. Particularly during 3 point task, we can make those more efficient. So what is your knowledge of usability testing before I define it.

These are much better numbers than one might expect. It looks like most people in the audience have quite a bit of knowledge on what it is or have done one. And some people are hoping to learn some things from this talk. That is wonderful, that is great.

The international standards organization defines usability in a way that I think is fairly uncontroversial. The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in specified context of use. This definition has effectiveness first. It is not simply about how it looks and how quick it is, it also gets into whether or not it helps someone in their job. The question that was posed to a group of people funded by NIFT which I was part of that team, is what is the relationship between usability and patient safety for health information technology and specifically for electronic health records. We defined patient safety as a working definition as a system attribute that influences the risk of patient harm due to errors. 

Let's talk about the framework that we came up with and then the references you can see the NIFt report that this is from. This report is still in draft form, but has released for public comment. There are four main components of this framework. This framework is a summary of research findings and critical use areas of all electronic health record systems that may or may not be applicable more broadly to health information technology. The first is use error root causes. This is aspects of the user interface design that may effect use errors. There are risk parameters that moderate the potential for patient harm due to these use errors. Many people are used to seeing severity, frequency and detectability, these are the standard failure modes in effects analysis moderators. We also added complexity because we kept hearing over and over again that for certain patient populations there are special considerations from a usability perspective. For example, neonates in the ICU are particularly fragile. And Medication errors in that setting have a different risk for patient harm than say for adults patients in an outpaitent setting. Some more people brought up transfusion patients or patients that have compromised immune systems. What that complexity is getting at is essentially what are the special populations that we need to worry about with relation to some of these use errors and how might that influence the risk. 

The third category is adverse events. We didn't try and recreate the wheel here, there are plenty of taxonomies out there. These are just the basic ways to categorized adverse events that can lead to patient harm. Nothing particularly new here: substandard care, morbility/ mortaility. What we did add was evaluative indicators. What we mean by that is if you are hearing repeating complaints or repeating themes from the field, that you can use these to help search for what might be causing these errors. So workarounds, redundancies, burnout in general, are people leaving after the introduction of a new health information technology, and another one is a low task completion 

rate. This is analogous to when you are shopping on the web and you start buying something, but you do not complete it. There are analogies in healthcare to when you start a tadk but don't finish it, that usually indicates some usability issue with the interface. There might be other ones we are not claiming to be comprehensive there. 

I wanted to go ahead and go through these use errors with some examples. These are not taken from any one place. They are all over the place, a variety of organizations, HITs and Health records. The first one is dealing with having two copies of an electronic health record open on the same computer. If you have two different patients that are open, an example of use error that could occur with relation to a design choice and then health information technology is if you have the second patient record open and you open say VISTA imaging or another imaging package from there it will have the data from patient A with the name from patient B showing. That is an example where you could print the image and have the wrong name or give a diagnosis based on the wrong name if you are not aware that this is embedded in the design. We have two separate software packages that we have discovered this problem with. 

Another problem would be wrong modes. For example there was a patient- a 100 kg patient- who received 100 times over dose of a vasso-active drug because weight dosing was selected instead of direct dosing, this was due to a lack of feedback that an unusual mode choice. There was no 

warning about it being an unusually high dose. When the user looked down at the display, you actually could not tell which mode was on which line for which button. It is an example of how you could have the wrong mode for action and that could lead to patient harm. Another example was a new software functionality that was being tested at a hospital where inadvertently the person who was testing the software was required to login as a test login in order to test. and instead logged in with their regular log in and because the test mode and production mode looks the same on the interface, and he ordered a medicine that almost reached the patient while just testing out the system. So that's an example where a design choices that were with the relation of showing the difference between test and production mode could have been distinguished more clearly to reduce the risk.

Another example is new electronic health records that are being pushed out quickly. where they don't display the dose of a medication, so they'll have a pick list with many medications, but they will have a cut off in terms of how many characters they will show. You might have three different choices of lidocaine hydrochloride, but it will not show on the pick list which dose is showing until you select it. So that's an example of how innaccurate data can be displayed in the screen. Another one is incomplete data. This was a case where the intended dose of the medication was actually 20 mg, but on the electronic health record interface in list for what to give it said 80mg. The reason for that is because there was a tapper dose and so in the comments section it said give 60 for two days, 80 for two days, 20 for two days unless you read the comments, you would not know to modify the dose. That's an example of incomp-lete data being displayed.

There was a case of nonstandard measurement system. Particularly for pediatric application. A patient received a much larger dose of a medication than was intended because nearly all of the displays used the English system for dose, but the pediatric dose calculation feature uses the metric system. And so the physician was required to remember to switch between those two, which obviously makes it harder. 

For recall, for example if you have a different way of ordering medications. So one-time dose is ordered differently the scheduled dose and so the one time dose you have to remember manually to type in the dose whereas the others are defaulted for you. So that's an example of where you might have recall errors. 

Inadequate feedback about automation. This is an example where if you transfer a patient from the inpatient to the outpatient setting, and you do a group transfer, say 10 medications at a time, in the inpatient setting it was acceptable to have a quarter of a tablet of a dose, but in the 

outpatient setting the thought was well we should not have our patient splitting pills as that is dangerous. So there was a feature that was automatically written in to change any partial dose tablets to full dose tablets. So if you take a medication four times a day and it changes from a quarter tablet to a full tablet then you have a patient who's getting eight times [sic] the dose in an outpatient setting. As in this case the software did not give any indication that there was an automated change of the tablets. There many different kinds of automation examples that we've collected but this is a basic category of where automation is done that is difficult to observe for the physician. 

For corrupted data storage for example you can have an interface where if you have a clinical reminder, and you click one button, 'finish' then what is saved is the information about ordering a new vaccine that's being administered and that goes into the note, whereas if you click 'next' then that infomration is not saved in the note. It looks like it is, but it is not. There are other examples where it will have character limits where you can only type in a number that has 4 characters in a row and if someone types in 5 it seems like the system is saving 10,000 when really it only saves 1,000. 

I am curious as to whether anyone here has had them or a family member experience any of these cases personally? I am going to go ahead and open that poll. 

No one has experienced wrong patient. Almost nobody has experienced the wrong treatment, but there were wrong medication, delayed treatment, and more than one.That's interesting. So we have the acknowledge that there are a lot of people that got none of the above. 

Again, here is pulling it all back into the framework. Overall, we are not claiming that we captured every possible use error, but we do feel like we have captured the most common ones that we're aware of across many packages many hospitals. These seem to be reasonably distinct from each other. Although, when we tried to code cases, again so often one example could have two of these root causes. But it does give something that we feel is a little bit more specific about what risks to look for than just in general make sure the screen is usable or that it's focused carefully and focuses more directly on what could lead to patient harm as opposed to what might just be annoying or frustrating for the user. Again there's nothing particularly new about our categories for adverse events. Wrong patient, wrong treatment, wrong medication, delay of treatment, unintended or improper treatment. We started breaking them down by acts of commission and ommission but basically everyone else does that too. We didn't feel like we made a huge contribution to the body of literature in this area. 

Then we have, what do we do about it? There are some different thoughts out there about- particularly what can we do to make electronic health records safer for the American public ? The most obvious one is here no evil, see no evil, see no evil, say no evil. We can say that there is no problem and so the solution is to take no action because there is no problem. Some variants of this are: electronic health records are so beneficial to patient safety that whatever risks are incurred are so minor that the last thing we want to do is slow the adoption of electronic health records. I'm not making any stand on that claim, I'm just saying if there's a transformative technology, regardless of how good it is or beneficial it is, we want to be proactive and address whatever risks proactively might be. Solution one is there's no problem so let's not do anything. 

Solution number two is the Code of Hammurabi, 229: If a builder builds a house for someone, does not construct it properly, and the house which he built falls in and kills its owner, then that builder shall be put to death. So maybe the solution is to sue all the vendors. Whoever is involved in building electronic health records we would give them great negative problems if something happens with the software.

So on this slides it is not 'United' it is 'un tied'. So what this is it's not sponsored by United airline but someone felt frustrated with United airlines service and they created this complaint website. So if someone feels like their issue is not being addressed then they can report issues publicly on this website anonymously. Someone suggested that we have vendors X or all vendors - a name and shame website where people can put up issues to get conversations flowing about what these problems are because one of the issues is it's hard to talk about them. We like our solution better but there are issues with this as well. 

Solution number four would be to do some form of checking that an interface that is planned for sale in the marketplace for electronic health records, has gone through some form of testing before it hits real patients. So in the NIFT there is layed out a three step process of vendors doing critical use risk analysis then having expert reviews looking at screen shots. It's not a full usability test it's just reviewing screens followed by a summative usability test before it hits the market. 

What is your preferred primary approach to making HIT safer for patient?

This does assume that making health information technology safer is a priority. Maybe you think there are no issues to address at this time. Everybody likes usability testing. Only one person wanted to sue the vendor. Looks like some people like 'something else'.

I do have contact information on my last slide and I would be very interested in what the 'something else' is that people are thinking about. 70% of people view usability testing as at least part of the solution.

Let's go ahead and go over some things that are a little bit farther along than what I knew about when I thought writing usability testing for my prior experience. This slide is from Robert Schumacher at user centric and he lays out a bunch of methods that are useful, methods that do not involve users and methods that involve users. There are many options out there for making software safer and they have different costs in terms of how expensive they are and what different benefits they give in terms of identifying issues. I want to point your attention to some of the usability testing and formative usability testing. These are two of the methods that involve users. I want to discuss these for a moment.

When I first started the projet, I assumed that we were talking about formative usability testing, which is taking 4-6 users during design cycles and giving them whatever is available at the time in terms of where we are heading with design and giving them use cases and tasks to do having them talk out loud and having them identify confusing issuesor surprising issues and having multiple ones of those so by the time the software rolls out the door quite a few users have had there input, but they tend to be in blocks of 4 or 6. It tends to be not very careful about comparing the outputs of each user against each other, we were just being creative and seeing where issues were and then dealing with them. The primary outcome was designed modification and we suggested it to the designer directly.

Instead what NIFT is talking about is summative usability testing which is a very different thing. It is many more users, high on the quantitative aspect, it involves behavior rather than opinions so it's not asking if you like the software, it is very specifically observing whether or not people click on the right thing or avoid clicking on the wrong thing. The process is similar to formative usability testing in that there is various steps, kickoff, preparation, data collection, analysis, reporting. But it is much more formalized and extensive and have a different role in the process.

What we did is identify a checklist. We put all of the use error contributors that we know of and made a checklist out of it. Not that everysingle one of these would be tested with every single system, but that these summative usability tests would focus on specifically what would cause patient harm and the ones that are most relevant to a particular application would be gone through methodically. For example when a second patient record is open, is the first patient record automatically closed. This is a technique to avoid wring patient errors. If a second user opens a patient chart, is the first user automatically logged out? So if a nurse opens an EHR is the physicians EHR automatically logged out to reduce confusion about who is working where.

The idea is that there is a standardized reporting form that says what is the potential use error that has been identified by this test and what is the mitigation plan. The idea is that the software does not necessarily have to be retested except in exceptional cases, if the mitigation plan works but that there is a specific plan for high-priority items to address issues that have been raised. It is not enough to provide training, there has to be a design element to it. Use errors can be tracked to see whether or not they are resolved over time. This is as much a post-implementation strategy as well as a pre-implementation strategy but either way you have specific issues that have specific numbers, specific people assigned to them and you are tracking whether or not they have been addressed, which is similar to how the VA deals with patient safety issues.  

Then the elements of the summative usability report include very specific things and this document literally has a template you fill in that discusses general usability things as well as specific potential use errors as a prominent place in the discussion.

I wanted to go ahead and spend some time on the scenario design. We did include three scenarios in this document. We did not make any claims that these ones should be used, but we wanted to give guidance to the field about the type of things that we think is needed in order to have enough complexity in a lab setting to really challenge the system. We did get feedback that you cannot really ever get in a lab setting the type of complexity that you would get from an implemented system. You can get much farther along if you have scenarios that capture some of the complexity. One of the specific suggestions we have is not to use scenarios that are used for demo purposes for sales of software because those are on purpose very simplified in order to communicate the features of the software and do not purposely test areas where things might be more complicated or there might be aspects that are failing. 

I am now entering the portion of the talk that's primarily from Emily Roth in a book chapter we wrote, Emily Roth, PhD in human factors, has done a lot of work for how to design scenarios 

for places like nuclear power generation and the military where they do what they call 'human in the loop' simulations before they decide whether or not to implement these big systems. She and I worked together and pulled together what is it we are looking for, what are our target characteristics for scenario designs. There are primarily five. The one that most people already know about is that you want to have a face valid scenario that engages professionals. The last thing you want is to have a usability test where a nurse or a physician come in and say that's not my work environment. You want to capture a face valid description of the kinds of things that the professionals do, so that service validity is very important. In addition for what we are looking at, there are other elements of the scenario that we want to embed as well. We view it almost as a jigsaw puzzle. Usually these systems have claims on what cognition is. We want to have automation that will allow people to be faster or we want to use automation that will allow people to check that there doses is right before they dose the patient. Usually there is some impact on cognition, reducing errors, reducing workload, etc. and what we want to do is take known capability gaps in those claims and embed them in the scenarios. Where we know that dosing doesn't work if you put a zero in for the dose; we want to include those in our scenarios. 

Another is the justification for implementations. Usually there are two main perspectives. What are the users going to have as their justification for why they would want to use it and there is also the organization's perspective. There might be a desire to have a roll-up of data to do quick and easy quality performance measures. Each of those perspectives have claims that are usually 

used to justify the implementation of the system. Go ahead and test those embedded in the scenarios.

Very specifically, we want to have normal cases, anominal cases, particularly to get started during the usability test to get people comfortable with running them. You want to have standard, everyday, but usually at the end you should have one challenge case. Something that really pushes the software capability so you can predict what might happen when it goes out into the real world. 

We always have externally observable actions and other instances so we will design the scenarios as such so that we can tell if people fell for the trap we set for them or not. 

We have many complexity factors in our chapter. I wanted to pull out the top seven. These are 

the main independent complexity factors. These are things that can be embedded in demo scenarios and can be embedded in scenarios to make them more complex. Data overload, attention bottlenecks, take out information, make it unreliable, put them in novel situations where there is no procedure, make it so they can't do everything they are asked, and make the entire task under 

time pressure.

How does this play out for the electronic health record case? 

In the scenarios I'm going to present in just a second, these are the embedded complexity factors that are starting to approach real-world complexity. We asked people to increase the dose of an existing medication, we had a false alarm, we wanted to see how tapered doses are handled. What about if there's a first dose now and subsequent doses tomorrow- how does the systme handle that case? Verbal orders, changing the form of medication, handoffs in the middles, interruptions, how do you follow-up documentation of prior work and how you batch transfer more than one medication at a time. Ambulatory care, mid-level provider (PA, NP), and a diabetic patient with chronic issues. What are the complexity factors there? Do you have to remove a patch, deal with care and medications, a taper dose escalation dose to deal with and substance abuse with particularly sensitive documentation aspect to deal with. How do you deal with it in a elecrtonic health record world? 

The second one we had a physician provider dealing with a cardiac patient in an inpatient setting and so we had the verbal order, workflow interruptions, handoff and patch transfer of medications at the end of the inpatient scenario.

For the third one we took a nurse provider in critical care with a cardiac patient and had them remove a DNR. One thing we've heard is removing an allergy and removing a DNR is harder than removing one in the first place. Documenting inputs, outputs and medication administration. 

Overall, I think what we are suggesting is putting the software at risk of failure in a safe place. You have a lab setting and you're going to identify issues that might slow down your production time and might appear to be too costly to be done. For example, there's a software company that said that if you want patient safety, you have to pay for it, this is going to slow down our time to implementation to the market place. My response to that is that if doing something the first time is too expensive then doing it a second time is going to be even more expensive. I don't think we want to be in a place where we have electronic health records implemented as an infrastructure item arocss the US that is not the safest that it can possibly be before it goes in. As the complexity of care goes up, things will be more difficult to change afterward. It is much easier to do them earlier rather than later. I don't think that I am the first person to say this. It is an interesting point that we all need to take a deep breath and realize that we are not talking about very expensive techniques or techniques that take a ridiculously long time to do in order to have a fairly high payback quickly for reducing risk to patients. 

I have my references here on this slide. The framework that I described is in the first reference. The second one is the book chapter on our scenario design. There are also patient safety references on this slide. 

Thank you for your attention and I look forward to hearing your questions. I 

want to have more dialogue, here is my contact information. 

Thank you, Emily, there are lots of questions and compliments. I am just going to read the questions to you: clinical data capture and documentation has been well documented in the literature as a problem area in usability. How is the work you are doing going to be reflected in the IEHR initiative that the combined VA and DOD initiative? 

I think what I can say is that I completely agree that documentation issues are a significant burden in usability. One of the surprising areas of agreement that came up when we had a NIFT usability workshop was that often many documentation issues do not stem from the EHR design, it is more from the complexity of the requirements for documentation from legal drivers and Medicare driver's. It is outside the scope of our work to talk about how we could reduce the complexity of billing and documentation, but I think it is real that basically much of the usability issues that are driving major complaints by Doctors in particular about the time to document out in the field is really driven by reimbursement policy and how much documentation has to be in there and how it has to be in there. I would love to see someone deal with that on a national level. I would love to learn more about what the VA and DOD are doing.

One of our writers wrote: When are you coming back to work for the VA ? 

When I took this job they actually wrote into my offer letter that I am not allowed to work for the VA at all. I wanted to leave the potential for having a 25% appointment for the VA. They specifically said I was not allowed to do that.

The next question comes in pieces. It is addressing the corporate data warehouse. What could have motivated some user to enter "so many things yet undone" into what became the component column of the corporate data warehouse table named "component"? 

I don't know anything about this. It sounds like an interesting design. 

I think we will move on and this person can get in touch with you directly because it is not clear what the question is. Next question: How long does it take vendors to repair the usability problems you have identified? You can give the answer anyway that makes sense (avg. type of problem, type of vendor, etc.).

This is the question, isn't it? With our formative usability testing experiences interanlly working with VA vendors who were very receptive, it was usually on the order of two weeks or three weeks. For a summative usability test our question is how quickly are summative usability testers getting back to designers, because that would impact turn around. I know that epic put an estimate in their response to NIFT of 12 weeks which I thought was longer than it would take. But given that nobody has done this, I think the answer is that we have no data.

This next question: does summative ability testing involve testing before implementation ? 

We wrote a standard that is meant to be like a thermometer and so the idea is when you design a thermometer is, you do not say when or where it is used in the process. It is our working assumption that it would be done before implementation because that is what the FDA does with devices. When we designed it, it is agnostic as to when.

Once it gets to the testing stage, a lot of things are already locked in and people are committed  emotionally to their designs. It is better to get the end-user involved from the very beginning of the design process. 

Absolutely. We had long discussions about this. Because NIFT put out a standard that only deals with the end of the process that does not mean that formative techniques, early techniques involving the user techniques are not important. One of the main criticisms of a standard at all is that it would stifle innovation and so we specifically stuck to the most formal, most documentation intensive aspects and said we do not want to stifle creativity. Whatever innovation people have to get early and quick input from users, that is wonderful. We gave no guidance on those first two steps about how do vendors identify what the critical areas are and early on. It's not that we don't think it's important, it's more that should be driven like vendors, populations, systems etc. that innovation in that space is really fast.

Seems to me like usability testing is a necessary, but not sufficient method. Don't other software vendors do this before they send out their software on the market? It seems like this is a minimal standard to expect of HIT packages.

We are basically saying what is the minimum standard and were hoping that leaders in the field are doing much more than this already. One of the special issues with electronic health records is that because they are incentivised by the ARRA act, there is a strong financial incentive to go to market quickly and I think there are over 700 vendors, electronic health record vendors. One of the things that ONC is doing now is requiring every electronic health record vendor to get on a list that ONC has so they know who is out there. One of our biggest concerns are the two and three people shops that are just trying to get anything quick to their business model is to sell and go out of business quickly.

What of clinicians who are so used to hard to use software that they hardly notice software anymore and/or mainly focus on workarounds? How can we get them to recognize the need for usability testing? 

Workarounds are huge indicators and that is why we put them in the framework as a valuative indicator of the first one. Obviously workarounds are an indicator that something is not fitting with the workflow. Having some kind of observational methods or whatever methods allow you to get at are work arounds being used, how extensive are they? How risky are they? That is outside of the scope of what I presented and outside the scope of what we were looking at. We hope that people will go into that space and worry about that 

space.

If you are the consumer in this case, the hospital, how do you work with software vendors to get what you need out of the software and get a product that has been thoroughly tested and that fits your particular clinic's work flow? 

That is a great question. I actually wrote a grant proposal in this area. I do think that hospitals are in a unique position where they are responsible for whatever software is implemented and yet they have very little authority and negotiating power with software vendors. They do have more power than they know about. One thing that I noticed with the VA's work with EDS, We gave this long presentation on usability testing was and they said oh we have usability testing people, the VA just never asked for it. We have actually started talking with leapfrog, which is a patient safety organization, they have a CPOE testbed. They are already getting about 25 to 75 hospitals feelling safe sharing what issues are with these systems. I think that is where we need to get to. Hospitals feeling safe and sharing with other hospitals like the VA does. What are these issues so that we can start to negotiate with vendors. If vendors say that is just specific to your hospital, and you don't know any different, you have to just accept that. I think a frequent comment from a vendor is well that is your hospital's specific issue so we're not going to deal with it. It's a frustrating answer.

I'm a researcher working in the emergency department. I think a lot of staff here have a good idea about how to improve the usability of EDIS, which is the VA emergency software. Do you know how to make these ideas reach you or the people who are responsible for it? 

Certainly they can contact me. We actually did do some research on EDIS and we did work with the vendors to improve the software.  

Provider interviews indicate, and quantitative analysis supports, adversity of EHR usage patterns depending on patient in terms of the volume by mouse clicks and a range of sequential process measures. How do you define task completion rates and efficiency of EHR given patients' clinical and treatment management heterogeneity? 

So I think the answer there is that we did not think that hard about it. We knew that the FDA tracks low task completion rates on divices as one of their indicators. That whole valuative indicator section is a little bit of a straw man at this point and it's really not covered in the document. It is a good question and that would be important. The complexity moderator on risk parameters also captures that notion that not every patient is the same and that is one of the major challenges HIT.

There are just two more questions. 

You're saying that places with integrated medical records have never done usability testing? 

Correct. 

I think we are at the end, Emily.

Thank you. 

[event concluded]