Usability Problem Description and the Evaluator Effect in Usability Testing
(2006)

Previous usability evaluation method (UEM) comparison studies have noted an evaluator effect on problem detection in heuristic evaluation, with evaluators differing in problems found and problem severity judgments. There have been few studies of the evaluator effect in usability testing (UT), task-based testing with end-use rs. UEM comparison studies focus on counting usability problems detected, but we also need to assess the content of usability problem descriptions (UPDs) to more fully mea sure evaluation effectiveness. The goals of this research were to develop UPD gui delines, explore the evaluator effect in UT, and evaluate the usefulness of the guidelines for grading UPD content. Ten guidelines for writing UPDs were developed by consulting usability practitioners through two questionnaires and a card sort. These guidelines are ( briefly): be clear and avoid jargon, describe problem severity, provide backing data, describe problem causes, describe user actions, provide a solution, consider politics and diplomacy, be professional and scientific, describe your methodology, and help the re ader sympathize with the user. A fourth study compared usability reports collect ed from 44 evaluators, both practitioners and graduate students, watching the same 10-minute UT session recording. Three judges measured problem detection for each evaluator a nd graded the reports for following 6 of the UPD guidelines. There was support for existence of an evaluator effect, even when watching prerecorded sessions, with low to moderate individual thoroughness of problem detection across all/severe problems (22%/34%), reliability of problem detection (37%/50%) and reliability of severity judgments (57% for severe ratings). Practiti oners received higher grades averaged across the 6 guidelines than students did, suggesting that the guide lin s may be useful for grading reports. The grades for the guidelines were not cor related with thoroughness, suggesting that the guideline grades complement measures of problem detection. A simulation of evaluators working in groups found a 34% increase in severe problems found by adding a second evaluator. The simulation also found that thoroughness of individual evaluators would have been overestimated if the study had included a small number of evaluators. The final recommendations are to use multipl e evaluators in UT, and to assess both problem detection and description when measuring evaluation effectiveness. iii ACKNOWLEDGEMENTS I would like to thank my advisory committee, Tonya Smith-Jackson, John Burton, Rex Hartson, Brian Kleiner, and Maury Nussbaum. In particular, Tonya Smith -Jackson provided suggestions for experimental design and statistical analysis, and Rex Hartson and Tonya Smith-Jackson provided suggestions for the judging procedure used to analyze the usability problems collected from evaluators. I also owe a special thanks to Laurian Hobby, John Howarth, and Pardha Pyla, each of whom generously donated over 50 hours of their time to read and evaluate hundreds of usability problems. The final study would not have been possible without them. Thanks also to my thesis advisor, Bob Williges, who guided me when I began studying usability methods. This dissertation would not have been possible without the support of my husband, Rob Capra, who shared the journey with me as we both completed our dissertations within a month of each other. I might not have discovered Human Factors or pursued a PhD without him. He is my partner in research, and many important ideas gre w out of discussions of our research at school, at home, and over meals. He is my partner in life, and I love him with all my heart. Many other people contributed to this research. Terence Andre shared the usability movies and reports from studies run by him and his students. Rob Capra assi sted with data coding in the first and fourth studies. Suzanne Aref suggested using factor analysis to cluster the items in the card sort. Joe Dumas shared his experience s analyzing the CUE-4 reports, provided criteria for identifying descriptions that discuss t he same usability problem, and helped recruit practitioners. Rolf Molich gave permiss ion to use the report template and severity rating scales from the CUE studies. The co mparison of usability diagnosis to medical diagnosis was refined through discussions with Rex Hartson and Steve Belz. Thanks to the dozens of usability practitioners and graduate students who volunteered to participate in my studies or provided feedback as pilot participants, especially the practitioners that took extra time above and beyond the study requirem nts to discuss their usability practices and reporting habits. Thanks to my famil and friends for years of encouragement and patience, and fellow students for support and commiseration along the way. Thanks to my former colleagues at what is now AT &T Labs for introducing me to Human Factors and sparking my interest in the field. Thanks to IMDb, Inc. for permission to include screen shots of their website in this document. Information courtesy of: The Internet Movie Database (http://www.imdb.com/ ). Used with permission. Portions of this research were conducted while I was supported by the Alexander E. Walter Fellowship from the Grado Department of Industrial and Systems Eng ineering (2001-2004). iv TABLE OF CONTENTS ABSTRACT . ................................................................................................................. ii ACKNOWLEDGEMENTS............................................................................................. iii TABLE OF CONTENTS................................................................................................. iv LIST OF TABLES .........................................................................................................viii LIST OF FIGURES .......................................................................................................... x LIST OF EQUATIONS ..................................................................................................xii CHAPTER