Usability testing for practical use

A high usability has a positive influence on the usefulness and safety of systems (Vieritz, Yazdi, Schilberg, Ghöner, & Jeschke, 2011; Harty 2011). To ensure a systems usability, it has to be tested (Gould, Boied & Lewis, 1991). In practical use and real world settings testing usability is a question of the cost/benefit ratio and it is necessary to find the optimal method and required sample size for testing usability with the highest benefit and the lowest costs (Nielsen & Landauer, 1993; Pape, 2012; Schmettow, Vos & Schraagen, 2013). Sullivan (2013) gives lead to the idea that testing usability via a communication tool could be a cheaper but equally good approach as testing face to face. Schmettow, Vos and Schraagen (2013) described an approach called zero-truncated logit-normal binomial model (LNBzt) that is able to predict the required sample size and to monitor the process of data collection making it able to estimate a completion rate and a number of undiscovered problems. This method suffers from impacts of false positives, making it necessary to identify them. Schmettow, Vos and Schraagen (2013) used an approach to identify false positives after data collection was finished, but an approach being able to identify false positives parallel to the collection of data would be more suitable for the late control strategy. To investigate whether testing face to face works differently for testing usability than testing via a communication tool a sample of 29 participants was divided in two groups, testing one face to face and the other one via a communication tool. Additional data was gathered using semi-structured interviews and questionnaires. The process of data collection was monitored using the late control strategy while a new protocol called False-positive Identification Protocol (FIP), developed in this study, was used to identify false positives during the process. The results showed that FIP performed well for identifying false positives parallel to the data collection, improving the results of the late control strategy. The late control strategy performed more effective than older approaches to estimate required sample sizes like the magic number approach or the early control strategy. While performing good at identifying false positives, FIP failed at the point of additional data collection and an additional expert assessment was suggested to be a good addition for FIP. Furthermore, both methods, testing usability face to face and via a communication tool resulted in similar numbers of problems, yet being different problems. The question whether one method performs better for testing usability could not be answered finally because they perform differently. It is concluded that both methods, testing face to face and via a communication tool can be of benefit to the results, but more research is needed. Furthermore, in combination with the new developed protocol FIP for identifying false positives the late control strategy provides more precise results than without using FIP, estimating the required sample size and monitoring the process of data collection. Running head: USABILITY TESTING FOR PRACTICAL USE 3