Facial Features for Affective State Detection in Learning Environments

Facial Features for Affective State Detection in Learning Environments Bethany McDaniel (btmcdanl@memphis.edu) Department of Psychology, University of Memphis Sidney D’Mello (sdmello@memphis.edu) Department of Computer Science, University of Memphis Brandon King (bgking@memphis.edu) Department of Psychology, University of Memphis Patrick Chipman (pchipman@memphis.edu) Department of Psychology, University of Memphis Kristy Tapp (kmsnyder@memphis.edu) Department of Psychology, University of Memphis Art Graesser (a-graesser@memphis.edu) Department of Psychology, University of Memphis System. This system specified how specific facial behaviors, based on the muscles that produce them, could identify “basic emotions”. Each movement in the face is referred to as an action unit (or AU). There are approximately 58 action units. These facial patterns were used to identify the emotions of happiness, sadness, surprise, disgust, anger, and fear (Ekman & Friesen, 1978; Elfenbein & Ambady, 2002). Doubts have been raised, however, that these six emotions are frequent and functionally significant in the learning process (D’Mello et al., 2006; Kapoor, Mota, & Picard, 2001). Some have challenged the adequacy of basing a theory of emotions on these “basic” emotions (Rozin & Cohen, 2003). Moreover, Ekman’s coding system was tested primarily on static pictures rather than on changing expressions over time. There is some evidence for a different set of emotions that influence learning and cognition, specifically boredom (Csikszentmihalyi, 1990; Miserandino, 1996), confusion (Graesser & Olde, 2003; Kort, Reilly, & Picard, 2001), flow (i.e. engagement, Csikszentmihalyi, 1990), and frustration (Kort, Reilly, & Picard, 2001; Patrick et al., 1993). Curiosity and eureka (i.e. the “a-ha” experience) are also believed to accompany learning. A study was recently conducted to investigate the occurrence of these emotions, as well as Ekman’s basic emotions. The study used an emote-aloud procedure (D’Mello et al., 2006), a variant of the think-aloud procedure (Ericsson & Simon, 1993), as an online measure of the learners’ affective states during learning. College students were asked to express the affective states they were feeling while working on a task, in this case being tutored in computer literacy with AutoTutor. Using the emote-aloud method allowed for the on-line identification of emotions while working on the learning task. A sample of 215 emote-aloud observations were Abstract This study investigated facial features to detect the affective states (or emotions) that accompany deep-level learning of conceptual material. Videos of the participants’ faces were captured while they interacted with AutoTutor on computer literacy topics. After the tutoring session, the affective states (boredom, confusion, delight, flow, frustration, and surprise) of the student were identified by the learner, a peer, and two trained judges. Participants’ facial expressions were coded by two independent judges using Ekman’s Facial Action Coding System. Correlational analyses indicated that specific facial features could segregate confusion, delight, and frustration form the baseline state of neutral, but boredom was indistinguishable from neutral. We discuss the prospects of automatically detecting these emotions on the basis of facial features that are highly diagnostic. Keywords: Facial features; action units, affective states; emotions; learning; AutoTutor; classifying affect Introduction It is widely acknowledged that cognition, motivation, and emotions are three fundamental components of learning (Snow, Corno, & Jackson, 1996). Emotion has been viewed as source of motivational energy (Harter, 1981; Miserandino, 1996; Stipek, 1998), but it can also be viewed as a more complex independent factor that plays an explanatory role in both learning and motivation (Ford, 1992; Meyer & Turner, 2002). The link between emotions and learning has received more attention during the last decade in the fields of psychology, education, and computer science (Craig, Graesser, Sullins, & Gholson, 2004; Graesser, Jackson, & McDaniel, 2007; Kort, Reilly, & Picard, 2001; Picard 1997; Meyer & Turner, 2002). Ekman and Friesen (1978) highlighted the expressive aspects of emotions with their Facial Action Coding