Measuring usability in healthcare IT: it's a practice, not a competition

usability.each case involves the development of metrics that are unique in their level of detail to the specific context of use, user characteristics, potential risks, and tasks of the system against which they are tested” [2, p. 29]. The authors of the EUP fully understand that comparative usability metrics are not possible (or even necessarily meaningful) given the complexities and unique environments of EHRs. But this was never the point. Measuring usability is a practice. The only standard of comparison is original intent. The expert evaluation procedures in the EUP are intended to drive a process whereby qualified experts focus on the user interface and then make positive changes based on the findings. The critics of the EUP should look beyond the premise of comparison or competition, and cease the unproductive conversations about why it is impossible to create a level playing field. Instead, they should focus on the critical task of developing effective methodologies to evaluate whether a user interface puts patient safety at unnecessary risk. Let’s reflect on how we got here. The EUP was developed primarily to define a standard protocol for evaluTo abstract the arguments in interactions by Swanson and Lind, the authors claim there are too many moving parts to conduct valid and reliable validation tests on the usability of EHRs [1]. They argue that an EHR could be configured completely differently in different medical environments where any number of unique users, user tasks, patient data, and connecting data sources are possible. We agree that, mathematically, the combinations are endless, making systematic user testing in every scenario seem impractical. EHRs undoubtedly represent a complex domain, but where Swanson and Lind err is in assuming it’s too hard to conduct validation (aka summative) testing if the end goal is to use the results “as a measure of comparative usability. ” The authors’ critique is built upon a fallacious warrant that the EUP is mainly about comparability of results. As a result, much of the criticism is at cross-purposes to the premise of the EUP. Comparison of EHR performance in summative validation testing is the anti-objective, and it says so in the body of the EUP NISTIR 7804: “The EUP is not intended to result in a relative or comparable score of Recently, NIST released guidance NISTIR 7804, “Technical Evaluation, Testing and Validation of the Usability of Electronic Health Records,” also known as the EHR Usability Protocol (EUP). The document is part of the NIST Health IT Usability initiative, which is focused on establishing a framework to define and assess the usability of health information technology (HIT). The reason for the initiative is the belief that usability represents a critical yet often overlooked factor affecting the acceptance, adoption, and use of electronic health record (EHR) systems. Simply put, measuring usability of HIT is an essential step toward improving the usability and safety of these applications. The response to this guidance, however, has been focused on the assumption that usability is difficult, if not impossible, to measure, and the resulting fear that asymmetrical measurement will create unfair standards of comparison. This concerns us, because the fact is, good usability research has the opposite effect on software development: It spurs people-focused innovation, which begets more effective, efficient, and satisfied users, which begets more satisfied customers. in te ra c ti o n s J u ly + A u g u s t 2 0 1 2