Model-Based Usability Analysis of Safety-Critical Systems
Authors:   , Abbate Andrew J. , Throckmorton Amy , Bass Ellen J. 
Published: July 2021

Complex, safety-critical systems are designed with a broad range of automated and configurable components, and usability problems often emerge for the end user during setup, operation, and troubleshooting procedures. Usability evaluations should consider the entire human-device interface including displays, controls, hardware configurations, and user documentation/procedures. To support the analyst, human factors researchers have developed a set of methods and measures for evaluating human-system interface usability, while formal methods researchers have developed a set of model-based technologies that enable mathematical verification of desired system behaviors. At the intersection of these disciplines, an evolving set of model-based frameworks enable highly automated verification of usability early in the design cycle. Models can be abstracted to enable broad coverage of possible problems, while measures can be formally verified to "prove" that the system is usable. Currently, frameworks cover a subset of the target system and user behaviors that must be modeled to ensure usability: procedures, visual displays, user controls, automation, and possible interactions among them. Similarly, verification methodologies focus on a subset of potential usability problems with respect to modeled interactions. This work provides an integrated formal methods framework enabling the holistic modeling and verification of safety-critical system usability. Building toward the framework, a set of five, novel approaches extend the capabilities of extant frameworks in different ways. Each approach is demonstrated in a medical device case study to show how the methods can be employed to identify potential usability problems in existing systems. A formal approach to documentation navigation models an end user navigating through a printed or electronic document and verifies page reachability. A formal approach to procedures in documentation models an end user executing steps as written and aids in identifying problems involving what device components are identified in task descriptions, what system configurations are addressed, and what temporal orderings of procedural steps could be improved. A formal approach to hardware configurability models end-user motor capabilities, relationships among the user and device components in the spatial environment, and opportunities for the user to physically manipulate components. An encoding tool facilitates the modeling process, while a verification methodology aids in ensuring that configurable hardware supports correct end- user actions and prevents incorrect ones. A formal approach to interface understandability models what information is provided to the end user through visual, audible, and haptic sensory channels, including explanations provided in accompanying documentation. An encoding tools facilitates the development of models and specifications, while the verification methodology aids in ensuring that what is displayed on the device is consistent; and, if needed, an explanation of what is displayed is provided in documentation. A formal approach to controlled actuators leverages an existing modeling technique and data collected from other engineering activities to model actuator dynamics mapping to referent data. An encoding tool facilitates model development, and a verification methodology aids in validating the model with respect to source data. Finally, new methodologies are combined within the integrated framework. A model architecture supports the analyst in representing a broad range of interactions among constituent framework models, and a set of ten specifications is developed to enable holistic usability verification. An implementation of the framework is demonstrated within a case study based on a medical device under development. This application shows how the framework could be utilized early in the design of a safety-critical system, without the need for a fully implemented device or a team of human evaluators.