Emotion-Driven Reinforcement Learning

Emotion-Driven Reinforcement Learning Robert P. Marinier III, John E. Laird ({rmarinie,laird}@umich.edu) Electrical Engineering and Computer Science Department, 2260 Hayward Ann Arbor, MI 48109 USA augmented Soar with a new module, our emotion system, described below. Abstract Existing computational models of emotion are primarily concerned with creating more realistic agents, with recent efforts looking into matching human data, including qualitative emotional responses and dynamics. In this paper, our work focuses on the functional benefits of emotion in a cognitive system where emotional feedback helps drive reinforcement learning. Our system is an integration of our emotion theory with Soar, an independently-motivated cognitive architecture. Integrating Appraisal Theories and Cognition Our work is grounded in appraisal theories (Roseman & Smith, 2001, for an overview) and Newell’s (1990) PEACTIDM (pronounced PEE-ACK-TEH-DIM). Appraisal theories hypothesize that an emotional reaction to a stimulus is the result of an evaluation of that stimulus along a number of dimensions, most of which relate it to current goals. The particular appraisals that our system uses is a subset of the appraisals described by Scherer (2001) (see Table 1). The subset our system uses can be split into two main groups: appraisals that help the agent decide which stimulus attend to (Suddenness, Unpredictability, Intrinsic Pleasantness, Relevance) and those appraisals that help the agent decide what do in response to an attended stimulus (causal agent and motive, outcome probability, discrepancy from expectation, conduciveness, control, power). Appraisal theories generally do not give much detail about how appraisals are generated or why. That is, the details of the process are left unspecified. Thus, computational models must fill in those details, but with little or no direction from appraisal theory, the details are often arbitrary. PEACTIDM, on the other hand, describes necessary and sufficient processes for immediate behavior (see Table 2). The PEACTIDM hypothesis is that stimuli are Perceived and Encoded so cognition can work with them. Then Attend focuses cognition on one stimulus to process, which is then Comprehended. Tasking is managing tasks and goals (e.g., in response to a change in the situation), whereas Intend is determining what actions to take. Decode and Motor are translating cognitive choices into physical actions. PEACTIDM, however, does not describe the data upon which these processes operate. The basis of our theory is that appraisals are the information upon which the PEACTIDM theory operates (Marinier & Laird, 2006) (see Table 3). For example, the attend process determines what to process next based on appraisal information generated by the perceive and encode processes (i.e., suddenness, unpredictability, intrinsic pleasantness, and relevance). Thus, appraisals not only determine the information that PEACTIDM processes, PEACTIDM also imposes dependencies between the appraisals (e.g., the appraisals for Comprehend cannot occur until after appraisals for Attend have been generated). Keywords: Emotion, reinforcement learning, intrinsic reward, cognitive architecture, appraisal theories. Introduction Folk psychology often casts emotions in a negative light. For example, in Star Trek, Vulcans are portrayed as superior to humans because they are not distracted by emotions, and thus can make purely logical decisions. As far back as Phineas Gage, however, it has been clear that emotions play a critical role in proper functioning in humans, and over the last several decades psychological research has explored how emotions influence behavior. We are interested in exploring how some of the functional capabilities of emotion can be utilized in computational agents; that is, we want to bring the functionality of emotions to artificial intelligence. This is in contrast to most existing computational models of emotion, which focused primarily on creating believable agents (Gratch & Marsella, 2004; Hudlicka, 2004), modeling human data (Marsella & Gratch 2006; Gratch, Marsella, and Mao, 2006), or entertainment (Loyall, Neal Reilly, Bates, and Weyhrauch, 2004). In this paper, we present work in which reinforcement learning is driven by emotion. Intuitively, feelings serve as a reward signal. The agent learns to behave in a way that makes it feel good while avoiding feeling bad. Coupled with a task that the agent wants to complete, the agent learns that completing the task makes it feel good. This work contributes not only to research on emotion in providing a functional computational grounding for feelings, but it also contributes to research in reinforcement learning by providing a detailed theory of the origin and basis of intrinsically-motivated reward. Background Our system in implemented in the Soar cognitive architecture (Newell, 1990). Soar is a complete agent framework composed of interacting, task-independent memory and processing modules that include short- and long-term memory, decision making, learning, and perception. We have