Experiences with usability testing: Effects of thinking aloud and moderator presence

Aalto University, P.O. Box 11000, FI-00076 Aalto www.aalto. fi Author Sirpa Riihiaho Name of the doctoral dissertation Experiences with usability testing: Effects of thinking aloud and moderator presence Publisher School of Science Unit Department of Computer Science Series Aalto University publication series DOCTORAL DISSERTATIONS 75/2015 Field of research Computer Science and Engineering Manuscript submitted 15th April 2014 Date of the defence 26th June 2015 Permission to publish granted (date) 8th April 2015 Language English Monograph Article dissertation (summary + original articles) Abstract Usability testing has become a standard method when evaluating the usability of various systems with real users. Despite this, the factors of usability testing have been given little attention in the academic forums. For example, the use of the thinking aloud method has resulted in quite conflicting effects on the users' performance in the context of usability testing, and the effects of the moderator presence have been studied very seldom in usability research. This thesis studies methods of usability testing and contextual factors in the test settings that may affect the results of a test, focusing on the effects of relaxed thinking aloud and the presence of a test moderator. It combines an extensive literature review with experiences on usability testing from 22 years covering 143 usability studies. The challenges of usability testing reported in the top academic HCI forums focus on sampling, context of use, use over time, and assessment of utility and value. The methods presented in this thesis provide some solutions to problems related to the context of use and to the assessment of utility and value. The most significant contribution of this thesis is in the experiment with thinking aloud and moderator presence as its independent variables. The results of relaxed thinking aloud show that it has no significant effect compared to silent performance on the number of usability problems the users face or in their subjective ratings, but it does slow down their performance. A significant effect of the moderator presence is found in the users' subjective rating, as users with a moderator next to them rate the system preferences significantly higher than participants performing alone. Given the benefits of having a moderator next to the user who is able to ask clarifying questions when the experiences are fresh in user's mind, the thesis still recommends this approach in formative usability testing when it is important to come up with practical redesign proposals for the development team. Thinking aloud in this experiment does not enhance the evaluators' confidence in the detected usability problems and their causes, and the test users mostly report it as an unnatural extra effort. Even so, thinking aloud gives more information about the problems to customers observing the tests, and thereby may motivate designers to make the required changes to the system. Therefore, if performance measurements are required, silent performance or classic concurrent thinking aloud with minimal interventions should be used, but in formative testing, the more explanatory relaxed thinking aloud can be used, as long as its potential effects on users' performance are kept in mind.Usability testing has become a standard method when evaluating the usability of various systems with real users. Despite this, the factors of usability testing have been given little attention in the academic forums. For example, the use of the thinking aloud method has resulted in quite conflicting effects on the users' performance in the context of usability testing, and the effects of the moderator presence have been studied very seldom in usability research. This thesis studies methods of usability testing and contextual factors in the test settings that may affect the results of a test, focusing on the effects of relaxed thinking aloud and the presence of a test moderator. It combines an extensive literature review with experiences on usability testing from 22 years covering 143 usability studies. The challenges of usability testing reported in the top academic HCI forums focus on sampling, context of use, use over time, and assessment of utility and value. The methods presented in this thesis provide some solutions to problems related to the context of use and to the assessment of utility and value. The most significant contribution of this thesis is in the experiment with thinking aloud and moderator presence as its independent variables. The results of relaxed thinking aloud show that it has no significant effect compared to silent performance on the number of usability problems the users face or in their subjective ratings, but it does slow down their performance. A significant effect of the moderator presence is found in the users' subjective rating, as users with a moderator next to them rate the system preferences significantly higher than participants performing alone. Given the benefits of having a moderator next to the user who is able to ask clarifying questions when the experiences are fresh in user's mind, the thesis still recommends this approach in formative usability testing when it is important to come up with practical redesign proposals for the development team. Thinking aloud in this experiment does not enhance the evaluators' confidence in the detected usability problems and their causes, and the test users mostly report it as an unnatural extra effort. Even so, thinking aloud gives more information about the problems to customers observing the tests, and thereby may motivate designers to make the required changes to the system. Therefore, if performance measurements are required, silent performance or classic concurrent thinking aloud with minimal interventions should be used, but in formative testing, the more explanatory relaxed thinking aloud can be used, as long as its potential effects on users' performance are kept in mind.