Affect-aware tutors: recognising and responding to student affect

Theories and technologies are needed to understand and integrate knowledge of student affect (e.g., frustration, motivation, and selfconfidence) into models of learning. One goal of this research is to integrate tools that model student affect into intelligent tutors and enable tutors to elicit, sense, communicate, measure and respond to student affect. This article presents multiple solutions towards this goal and discusses systems that begin to redress the cognitive vs. affective imbalance in teaching. We describe our broad approach, how we will evaluate the impact of affect intervention on student learning and three main development objectives: i) affect recognition, ii) interventions in response to student affect, and iii) emotionally animated agents. 1 Vision and challenge Intelligent tutors provide individualized teaching in multiple domains and demonstrate learning gains similar to or greater than those provided by human tutors (Fletcher, 1996; Koedinger et al., 1997; Shute and Psotka, 1995). However, much previous research has tended to privilege the cognitive over the affective in which theories of learning view thinking and learning as information processing, marginalizing or ignoring affect (Picard et al., 2004). If computers are to interact naturally with humans, they must recognize affect and express social competencies. However, the role of affect in instruction is at best in its infancy (Picard et al., 2004). One obvious next frontier in computational instruction is to systematically examine the relationship(s) between student affective state and learning outcomes (Shute, 2008). This research looks at the role of student affect in learning and the role that new technology plays in recognizing and responding to affect. We describe research to measure, model, study, and support the affective dimension of learning in ways that were not previously possible. Our systems are used in the classroom with the goal of improving learning and perseverance. Affective interventions encourage learning, lessen student humiliation and provide support and motivation that outweighs or distracts from the unpleasant aspects of failure. This research is based on efforts at the University of Massachusetts, Arizona State University and the MIT Media Lab. The next section describes theories of affect, learning and human emotion. We look at the constellation of student behaviors that we label as emotion and provide a brief overview of how computers can recognize and respond to student affect. The next three sections describe three approaches to affect recognition (human observation, hardware sensors and machine learning Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig. 2 techniques), several interventions that respond to a student’s cognitive-affective state and two sets of emotional embodied pedagogical agents. The final two sections provide a discussion and a view of future work. 1.1 Theories of affect, learning and human emotion Modeling student emotion has become increasingly important for computational teaching systems and emotion has been named as one of the twelve major challenges for cognitive science (Norman, 1981). Human emotion is often defined as an intuitive feeling derived from one’s circumstance, mood or relation with others. Teachers have long recognized the central role of emotion in learning and the extent to which emotional upsets can interfere with mental life. Student interest and active participation are important factors in the learning process (e.g. Bransford et al., 2000). Students learn less well if they are anxious, angry, or depressed; students who are caught in these states do not take in information efficiently or deal with it well (Burleson and Picard, 2004; Picard et al., 2004; Goleman, 1995). Teachers often devote as much time to the achievement of students’ motivational goals as to their cognitive and informational goals in one-to-one human tutoring situations (Lepper and Hodell, 1989). Emotions can paralyze a student’s ability to retain information about a task (Baddeley, 1986). Several studies have addressed emotions involved in learning (e.g. Lepper and Chabay, 1988; Mandler, 1984; Kort et al, 2001). Human emotion is completely intertwined with cognition in guiding rational behavior, including memory and decision-making. The human brain is a system in which emotion and cognitive functions are inextricably integrated (Cytowic, 1989). Emotional skills have been shown to be more influential than cognitive abilities for personal, career and scholastic success (Goleman, 1996). For instance, in the comparison of impulsivity and verbal IQ as predictors of future delinquent behavior, impulsivity was twice as powerful a predictor (Block, 1995). Recent findings suggest that when basic mechanisms of emotion are missing, intelligent functioning is hindered. Nearly a hundred definitions of emotion have been categorized as of 1981(Kleinginna and Kleinginna, 1981). Yet no comprehensive, validated, theory of emotion exists that addresses learning, explains which emotions are most important in learning, or identifies how emotion influences learning (Picard et al., 2004). Most studies of emotion do not include the phenomena observed in natural learning situations, such as interest, boredom, or surprise. Rather, emotion definitions emphasize cognitive and information processing aspects and encode them into machinebased rules used in learning interaction, e.g., OCC model of emotion (Ortony et al., 1988). Acceptance of ideas about emotion in learning is based largely on intuition and generalized references to constructivist theorists (Piaget and Inhelder 1969; Vygotsky, 1962, 1978). These theories discuss how to motivate, engage, and assist students in a general way. Yet, they do not provide descriptions at the level of individual human-to-human interactions and clearly do not provide methods suitable for implementation in intelligent tutors. Motivation is one emotion strongly linked to learning and has been defined as a person’s direction, intensity and persistence in an activity. Students with high intrinsic motivation often outperform students with low intrinsic motivation. A slight positive approach by a student is often accompanied by a tendency toward greater creativity and flexibility in problem solving, as well as more efficiency and thoroughness in decision-making (Isen, 2000). If student motivation is sustained throughout periods of disengagement, students might persevere to a greater extent through frustration (Burleson and Picard, 2004). Studies of motivation in learning consider the role of intrinsic versus extrinsic influences, selfefficacy, students’ beliefs about their efficacy, the influence of pleasurable past learning experiences, feelings of contributing to something that matters and the importance of having an Submitted to special issue on Modeling and Scaffolding Affective Experiences To Impact Learning, International Journal of Learning Technology edited by Rana el Kaliouby and Scotty Craig. 3 audience that cares, among other factors (Vroom, 1964; Keller, 1983; 1987; Ames, 1992; Vail, 1994; Bandura, 1977; Pajares, 1996; Schunk, 1989; Zimmerman, 2000). Theories of motivation are often built around affective and cognitive components of goal directed behavior (e.g. Dweck, 1986, 1999; Dweck and Leggett, 1988). Flow, or optimal experience is often defined as a feeling of being in control, concentrated and highly focused, enjoying an activity for its own sake, or a match between the challenge at hand and one's skills (Csikszentmihalyi, 1990). In direct contrast Stuck, or a state of non-optimal experience, is characterized by elements of negative affect and includes a feeling of being out of control, a lack of concentration, inability to maintain focused attention, mental fatigue and distress (Burleson and Picard, 2004). The phenomenon of “negative asymmetry” or the staying power of negative affect, which tends to outweigh the more transient experience of positive affect, is also an important component of learning and motivation (Giuseppe & Brass, 2003). The concept of affect is often distinguished from that of emotion. Affect refers to an observed emotional state or biological response to an external stimuli. Incorporating affect within humanhuman interactions is very powerful. In their research on “thin slices,” Ambady and Rosenthal demonstrated that when participants were shown a short segment of video, as little as six seconds of a teacher’s first interactions with a student, they could predict that teacher’s effectiveness and student end-of-term grades (Ambady and Rosenthal, 1992). Wentzel has shown that caring bonds between middle school children and their teachers are predictive of learners’ performance (1997). Some researchers have raised the concern that one cannot begin to measure or respond to emotion until a clear theory of emotion is articulated. However, even without a fully-fledged theory of emotion, computers can be given some ability to recognize and respond to affect (Picard et al., 2004). In fact, research shows that efforts to build models of a less understood phenomenon will aid in improving the understanding of that very phenomenon (Picard et al., 2004). Thus we simultaneously engage in both the practice and the theory directly related to developing affect-aware tutors in an attempt to advance both. Ekman’s Categorization Cognitive-Affective Term Emotion Scale High pleasure Joy Low pleasure “I am enjoying this.” . . . “This is not fun.” Frustration Anger Low-frustration “I am very frustrated.” . . “I am not frustrated at all.” Novelty Surprise Boredom “I am very hooked.” . . . “I am bored.” Anxiety Fear Confidence “I feel anxious” .. . . “I feel very confident” Table 1. Cognitive-affective terms based on human face studies (Ekman et