Detection of Emotions during Learning with AutoTutor

Detection of Emotions during Learning with AutoTutor Art Graesser (a-graesser@memphis.edu) Amy Witherspoon (awthrspn@mail.psyc.memphis.edu) Department of Psychology, 202 Psychology Building Memphis. TN 38152 USA Department of Psychology, 202 Psychology Building Memphis. TN 38152 USA Bethany McDaniel (btmcdanl@memphis.edu) Department of Psychology, 202 Psychology Building Memphis. TN 38152 USA Sidney D’Mello (sdmello@memphis.edu) Institute for Intelligent Systems, 365 Innovation Drive Memphis, TN 38152 USA Patrick Chipman (pchipman@memphis.edu) Department of Psychology, 202 Psychology Building Memphis. TN 38152 USA Barry Gholson (b.gholson@memphis.edu) Department of Psychology, 202 Psychology Building Memphis. TN 38152 USA between cognition and emotions, they do not directly explain and predict the sort of emotions that occur during complex learning, such as attempts to master physics, biology, or computer literacy. Researchers in many different fields are familiar with Ekman’s work on the detection of emotions from facial expressions (Ekman & Friesen, 1978). However, the emotions that Ekman intensely investigated (e.g., sadness, happiness, anger, fear, disgust, surprise) have minimal relevance to learning per se (Kort et al., 2001). The pervasive affective states during complex learning include confusion, frustration, boredom, flow/engagement, interest, and being stuck (Craig, Graesser, Sullins, & Gholson, 2004; Csikszentmihalyi, 1990). There are a number of ways in which tutors (and other types of learning environments) might adaptively respond to the learner’s affective states in the course of enhancing learning (D’Mello, Craig, Sullins, & Graesser, in press; Graesser et al., 2005; Lepper & Woolverton, 2002). If the learner is frustrated, for example, the tutor can give hints to advance the learner in constructing knowledge or can make supportive empathetic comments to enhance motivation. If the learner is bored, the tutor needs to present more engaging or challenging problems for the learner to work on. The tutor would probably want to lay low and stay out the learner’s way when the learner is in a state of flow (Csikszentmihaly, 1990), i.e., when the learner is so deeply engaged in learning the material that time and fatigue disappear. The flow experience is believed to occur when the learning rate is high and the learner has achieved a high level of mastery at the region of proximal learning (Metcalfe & Kornell, 2005). The affective state of confusion is particularly interesting because it is believed to play an important role in learning (Graesser et al., 2005; Guhe et al., 2004) and has a large correlation with learning gains (Craig et al., 2004). Confusion is diagnostic of cognitive disequilibrium, a state that occurs when learners face obstacles to goals, contradictions, incongruities, anomalies, uncertainty, and salient contrasts (Festinger, 1957; Graesser, Lu, Olde, Pye- Cooper, & Whitten, 2005; Graesser & Olde, 2003; Piaget, Abstract The relationship between emotions and learning was investigated by tracking the affective states that college students experienced while interacting with AutoTutor, an intelligent tutoring system with conversational dialogue. An emotionally responsive tutor would presumably facilitate learning, but this would only occur if learner emotions can be accurately identified. After a learning session with AutoTutor, the affective states of the learner were classified by the learner, a peer, and judges trained on Ekman’s Facial Action Coding system. The classification of the trained judges was more reliable and matched the learners much better than the low scores of untrained peers. This result suggests that peer tutors may be limited in detecting the affective states of peer learners. Classification accuracy was poor at constant intervals of polling (every 20 seconds) but much higher when individuals declared that an affect state had been experienced. Keywords: Emotion; Instruction and teaching; Human computer interaction; AutoTutor; Affective states Introduction Connections between complex learning and emotions have received increasing attention in the fields of psychology (Carver, 2004; Deci & Ryan, 2002; Dweck, 2002), education (Lepper & Henderlong, 2000; Linnenbrink & Pintrich, 2004; Meyer & Turner, 2002), neuroscience (Damasio, 2003), and computer science (Kort, Reilly, & Picard, 2001; Picard, 1997). A deep understanding of such affect-learning connections is needed in order to design engaging educational artifacts that range from responsive intelligent tutoring systems on technical material (DeVicente & Pain, 2002; Graesser, Person, Lu, Jeon, & McDaniel, 2005; Guhe, Gray, Schoelles, & Ji, 2004; Litman & Silliman, 2004) to entertaining media and games (Conati, 2002; Gee, 2003; Vorderer, 2003). There have been several theories that link cognition and affect very generally (Bower, 1981; Mandler, 1984; Ortony, Clore, & Collins, 1988; Russell, 2003; Stein & Levine, 1991). While these theories convey general links