A Comparison of two concurrent think-aloud protocols: Categories and relevancy of utterances

This paper discusses the results of an exploratory study that compared type and quality of participant verbalizations experienced from two concurrent think-aloud methods. The speech communication and traditional think-aloud methods were compared in terms of the number of participant utterances spoken and relevancy of those utterances in terms of further usability analysis. Though the speech communication method produced fewer utterances, it produced more relevant utterances than the traditional method. Participants preferred hearing the moderator’s acknowledgment tokens in the speech communication condition to the moderator’s silence in the traditional think-aloud method. There was a significant difference with how natural participants felt while experiencing the various protocols. These findings suggest that the moderation style has a potential impact on the type of verbalizations produced during usability sessions and on how participants feel about thinking aloud. A COMPARISON OF TWO CONCURRENT THINK-ALOUD METHODS 6 Introduction It is common practice for usability practitioners to use the think-aloud protocol during studies to gain access to what participants are thinking while interacting with a particular system (Ramey, Boren, Cuddihy, Dumas, Guan, van den Haak, & de Jong, 2006). Participant verbalizations are important in think-aloud studies as they offer valuable feedback concerning the product being tested. Evaluators quickly gain information about participants’ concerns, frustrations, or surprises that arise during the testing process; these verbalizations assist in identifying usability issues and areas of the product or interface to improve (Rubin & Chisnell, 2008; van den Haak, & de Jong, 2003). Using the think-aloud protocol helps to better understand users’ intentions that sometimes cannot be understood through observation alone (Cook, 2010). One of the most commonly used think-aloud protocols in the usability field is concurrent thinkaloud (Bowers & Snyder, 1990; Olmsted-Hawala, Murphy, Hawala, & Ashenfelter, 2010a; Ramey et al., 2006). Following this method, participants are encouraged to provide a continuous commentary of their thoughts while working on a task (Rubin & Chisnell, 2008). Despite its common use in the field, the concurrent think-aloud procedures vary widely among usability professionals (Boren & Ramey, 2000). More specifically, the types and frequency of test administrators’ probes and interventions differ among researchers and professionals who use the concurrent think-aloud protocol (Boren & Ramey, 2000; Ramey et al., 2006). This variety in protocol may cause validity and reliability issues in verbal data analysis, making it difficult to compare research results or replicate studies (Boren & Ramey, 2000). To date, there have been few empirical studies on these variations of concurrent think-aloud protocols (Olmsted et al., 2010a). Our study intends to provide further understanding of two variants of the concurrent think-aloud protocol: the traditional think-aloud protocol by Ericsson and Simon (1993) and the speech communication protocol proposed by Boren and Ramey (2000). The categories of participants’ verbalizations were examined to determine whether one of these concurrent think-aloud methods collected more useful utterances for usability evaluation than the other. Participants’ perceptions about the two methods were additionally explored. A COMPARISON OF TWO CONCURRENT THINK-ALOUD METHODS 7 Literature Review Traditional Think-Aloud Often cited in usability textbooks and research publications is the think-aloud framework proposed by Ericsson and Simon (1993). Based on their research in cognitive psychology, Ericsson and Simon (1993) revealed three levels of verbalizations that occur during think-aloud sessions. The first two levels of verbalizations are considered reliable verbal data as the information is provided by the participant’s short-term memory (STM). If the researcher prompts or questions the participant, the participant’s subsequent verbalizations are considered Level 3 data because the flow of content in STM has changed during the task (Boren & Ramey, 2000). Ericsson and Simon (1993) recommend not relying on Level 3 data since the participant’s verbalizations have been affected by interventions, and the utterances are therefore considered unreliable data. Examples of evaluator probes that would be categorized as Level 1, 2, or 3 verbalizations (Olmsted et al., 2010a) are as follows: • Level 1 and Level 2 Probes such as Keep talking or Uh-huh? do not divert attention away from participants’ focus. Participants continue working on the task without distraction. • Level 3 Questions such as Why did you click on the Home tab? need additional cognitive processing in order to be answered. Participants would need to access their long-term memory to respond. Verbalizations that follow Level 3 interventions have a higher risk of being unreliable (Ericsson & Simon, 1993; Nisbett & Wilson, 1977). According to Ericsson and Simon’s (1993) model, the experimenter should not interfere during the think-aloud process. Rather, the communication between experimenter and participant should be single-directional with the participant continuously verbalizing his thoughts and the experimenter only listening (Krahmer & Ummelen, 2004). If a participant falls silent for a predetermined amount of time (i.e., 15-60 seconds), the evaluator provides reminders to talk aloud by saying, Keep talking. A COMPARISON OF TWO CONCURRENT THINK-ALOUD METHODS 8 Think-Aloud in Practice The unnatural environment of the think-aloud procedure may cause participants to feel uncomfortable and prevent them from speaking effortlessly about their thoughts (Rubin & Chisnell, 2008). In an attempt to ease the anxiety level and remove silence, test administrators, who often cite Ericsson and Simon’s method, may probe the participants with abstract and leading questions to gain more relevant utterances for usability evaluation (Nørgaard & Hornbæk, 2006). Examples of these variations of the traditional think-aloud procedure include the coaching, relaxed, and active intervention protocols (Olmsted-Hawala et al., 2010a). Despite Ericsson and Simon’s (1993) suggestion to exclude Level 3 data because of its threat to reliability, usability professionals continue to use these probing protocols to collect information they believe might not be achievable with the traditional protocol (Boren & Ramey, 2000). Ericsson and Simon (1993) believe that evaluators’ interventions and questions affect participants’ future verbalizations and task performance while impacting the validity of data. Carter (2007) agrees and suggests some hypothetical user responses that might occur as a result of Level 3 interventions such as What are you thinking? or What are you experiencing? He states that the varying interventions and probes interfere with participants’ cognitive processes and shifts their attention. Exploratory studies such as Hertzum, Hansen, and Andersen (2009), Olmsted et al. (2010a), and Krahmer and Ummelen (2004), discovered that these ‘probing’ think-aloud protocols compared to the traditional think-aloud protocol affect participants’ behavior and mental workload, accuracy and user satisfaction, and task success and lostness, respectively. The studies suggest that test administrators’ interventions cause validity issues and affect participants’ performances. The benefits and concerns of probing questions or comments have received little research attention, thus the debate continues as to how often and in what manner to intervene in usability studies (Anderson, 2004; Tamler, 1998). A COMPARISON OF TWO CONCURRENT THINK-ALOUD METHODS 9 An Alternate Theory: Speech Communication Protocol The variety of concurrent think-aloud protocols and lack of adherence to Ericsson and Simon’s framework has caused professionals to question if another think-aloud protocol is more effective in usability research (Olmsted et al., 2010b). Boren and Ramey (2000) suggest that a think-aloud protocol based on speech communication theory may be more effective in usability research and provide a better framework for collecting reliable verbalizations. Boren and Ramey (2000) attempt to reconcile theory and practice by creating a more natural interaction environment between facilitator and participant, based on how humans normally communicate. According to speech communication theory, each time words are spoken intentionally for another’s benefit (Boren & Ramey, 2000), the active roles of a speaker-listener relationship exist. The speaker’s role (participant) is to predominantly talk and send information while the listener’s role (test administrator) is to respond as much or as little as necessary (Drummond & Hopper, 1993). This twostep information exchange establishes an interaction between speaker and listener (Clark & Schaefer, 1989; Goodwin, 1986). Speech communication theory states that listeners use various back channels in the form of facial expressions, body language, and verbal cues to notify the speaker that they are actively listening and acknowledging the conversation (Olmsted-Hawala et al., 2010a). As previously mentioned, usability professionals use neutral probes such as What are you thinking? or How close was that to what you expected? to gather additional information from participants about their task performances and preferences about the interface (e.g., Chisnell & Rubin, 2008; Nielsen, 1993; Stone, Jarrett, Woodroffe, & Minocha, 2005). These types of questioning probes are not to be used with Ericsson and Simon’s model because they disrupt the task flow (Boren & Ramey, 2000). Boren and Ramey (2000) recommend the use of acknowledgement tokens such as ok, mm hmm, or yeah as they can provide the expected response from an active listener and still remain nondirective. The acknowledgement token of mm hmm is suggested to be the best utterance to no