# Should I Stop Thinking About It: A Computational Exploration of Reappraisal Based Emotion Regulation

Authors: Mei Si
Journal: Advances in Human-Computer Interaction
 (2015)

---

## Abstract

Agent-based simulation of people’s behaviors and minds has become increasingly popular in recent years. It provides a research platform to simulate and compare alternative psychological and social theories, as well as to create virtual characters that can interact with people or among each other to provide pedagogical or entertainment effects. In this paper, we investigate computationally modeling people’s coping behaviors and in particular in relation to depression, in decision-theoretic agents. Recent studies have suggested that depression can result from failed emotion regulation under limited cognitive resources. In this work, we demonstrate how reappraisal can fail under high levels of stress and limited cognitive resources using an agent-based simulation. Further, we explored the effectiveness of reappraisal under different conditions. Our experiments suggest that for people who are more likely to recall positive memories, it is more beneficial to think about the recalled events from multiple perspectives. However, for people who are more likely to recall negative memories, the better strategy is to not evaluate the recalled events against multiple goals.


---

## Body

## 1. Introduction

Agent-based simulation of people’s behaviors and minds has become increasingly popular in recent years. In particular, computational modeling of people’s cognitive and emotional processes provides a research platform to simulate and compare alternative psychological and cognitive theories [1, 2], as well as to create virtual characters that can interact with people or among each other to provide pedagogical or entertainment effects [3–7].In this work, we developed computational approaches for simulating coping strategies within a decision-theoretic agent framework for interactive narratives, Thespian [2, 4, 8]. Emotion and mood play an important role in forming people’s experiences. They are not simply reactions to what happens in the environment. People actively strive to manage their emotional experiences. When negative feelings are triggered, in addition to looking for ways to improve the situation, people also use emotion-focused coping strategies, such as wishful thinking and distancing from the stressful thought to regulate their emotional experience. Emotion regulation is regarded vital for a successful social life [9], and people do so constantly either consciously or unconsciously. In fact, at any moment of time, our experience is a combination of the direct emotional reaction to the environment and the results from applying various coping strategies. In this paper, we investigate computationally modeling people’s emotion-focused coping behaviors and in particular in relation to depression.Depression has become a major health issue. Millions of people suffer from depression world-wide. Depression is often triggered by stressful life events. The experience of depression is usually associated with negative views about oneself and the environment. Beck defines depression as negative schemas which can be summarized by a cognitive triad [10]:(i)
The self: “I am worthless,” “I am defective or inadequate.”(ii)
The world or environment: “The world is unfair,” “all my experiences result in failures.”(iii)
The future: “The future is hopeless.”Beck pointed out that there is “an astonishing contrast between the depressed person’s image of himself and the objective facts,” and further, people with depression “often perform acts that seem to enhance their sufferings” [10].Depression is a complex psychological and physiological phenomenon. Despite its biological basis, depression is also associated with one’s cognitive processes and, in particular, the way one appraises his/her environment and copes with stressful events. For example, it has been found that people with depression are more likely to have biases such as learned helplessness [11], self-focused cognition [12], and negative self-schema [10, 13]. In fact, maladaptive appraisals and coping strategies have been shown as a risky factor for various psychological disorders, including posttraumatic stress disorder (PTSD) [14] and depression [15, 16]. On the other hand, the appraisal and coping strategies used by people with depression are not unique. They are part of the pool of appraisal and coping strategies that everyone may use.The purpose of this work is twofold. First, we want to develop computational models for emotion-focused coping for digital characters. These models are developed within an existing decision-theoretic agent framework, Thespian for simulating humanlike conversational agents and interactive narratives. Through our simulations, we demonstrate a generic set of algorithms can be used for simulating both successful and unsuccessful copings. The result of this work will improve the capacity of Thespian agent for creating rich and believable characters. It has been repeatedly argued that social agents should be designed with a “mind” that gives them characteristics and reveals their intentions and emotional experiences [17]. The second purpose of this work is to create a simulation platform for exploring adaptive coping strategies. In particular, in the work reported here we simulate cognitive reappraisal which is a commonly suggested emotion regulation strategy for depression and demonstrate its likelihood of success under various circumstances. Our simulation results demonstrate a trend consistent with what have been observed in clinical studies of depression: reappraisal requires more cognitive resources to succeed as one needs to cope with a more stressful situation; at the same time, the likelihood of experiencing negative emotions during reappraisal increases. Further, we want to explore how to use the reappraisal strategy more effectively in different situations. We often hear suggestions such as “let go of things that do not serve you.” On the other hand, we are also often told that there is always a positive side to every situation. If a negative memory is triggered during reappraisal, should the person let it go or try to find the positive side of it? In this work, we try to throw some light on this question through agent-based computational simulation.


## 2. Related Work

Various theories have been proposed for explaining emotion and mood, such as Schachter and Singer’s two-factor theory of emotion [18], the opponent-process theory of emotion [19], the coherence theory of emotion [20], and the appraisal theories of emotion. The appraisal theories of emotion are most relevant for the work we are conducting here. Appraisal theory argues that a person’s subjective assessment of his/her relationship to the environment, the person-environment relation, drives the person’s emotional responses [21–26]. In this section we briefly review cognitive appraisal theories, the computational models of appraisals and coping strategies, and the relationship between depression and emotion regulation.


### 2.1. Cognitive Appraisal Theories

Appraisal theory argues that a person’s emotional experience is determined by his/her subjective assessment and interpretation of internal or external events. Following these theories, one’s emotional experience triggered by the same event can be different from time to time based on how the person interprets the event. This also explains why people’s attitudes and feeling towards an event can change over time.Formally, the assessment occurs along several dimensions, called appraisal variables or checks, such as motivational congruence, accountability, novelty, and control. Emotion is decided by the combination of results from these checks. For example, an event that leads to a bad outcome for a person (motivationally incongruent) and is caused by others (accountability) is likely to elicit an anger response; but if the event is caused by the person himself/herself, that person is more likely to feel guilt or regret [21].Various theories of cognitive appraisals exist. They are different in terms of the appraisal dimensions being considered, how emotions are predicted from the appraisal dimensions, and the process of appraisal, for example, whether it is a fast and shallow process that leverages on the reasoning for decision-making and whether it is a sequential process.One theory that is particularly relevant to this work is Smith and Lazarus’s theory of appraisal [23, 24]. They specify two major types of appraisal methods: (1) primary appraisal, directed at the establishment of the significance or meaning of the event to the organism containing similar appraisal dimensions as in other cognitive appraisal theories, and (2) secondary appraisal, directed at the assessment of the ability of the individual to cope with the consequences of the event. Secondary appraisal is invoked when the event is appraised as negative. It evaluates the person’s potential for coping. The result of the evaluation will be taken into account by the person’s following primary appraisal, and thus form an appraisal-coping-reappraisal loop in people’s cognitive/emotion generation process. Various coping strategies can be applied. They can be roughly divided into two categories: problem-focused coping and emotion-focused coping. The first category contains strategies that seek to change the situation, such as seeking information, action planning, and seeking instrumental support. The second category contains strategies that change one’s beliefs and goals, such as wishful thinking, positive reappraisal, and resignation. People may use any of these strategies to cope with their stressful situations.


### 2.2. Computational Models of Appraisal and Coping

Various computational models for appraisals have been proposed. For applications that aim at simulating emotions for fictional characters, the modeling of emotions is often embedded in the modeling of the interactive story or the character’s decision-making process and uses rule based models. For example, in FLAME, El Nasr et al. use domain-independent fuzzy logic rules to simulate appraisal [27]. Cathexis uses a threshold model to simulate basic variables, which are called “sensors,” related to emotion [28]. Elliott’s Affective Reasoner uses a set of domain-specific rules to appraise events based on the OCC model of emotion [29].Most often in systems for simulating virtual humans or artificial lives, decision making and emotions are modeled separately. In a few agent-based systems, models of emotion leverage the agent’s decision-making representations to model the person-environment relation. For example, EMA [30] defines appraisal processes as operations over a uniform plan-based representation of the agent’s goals, and how events impact those goals. Both EM [31] and ParleE [5] deployed the OCC model of emotion over plan-based agents. In our previous work, Thespian [2] models appraisal as a light weighted process utilizing representations generated by POMDP-based agents’ decision making and belief revision processes. In these systems the agents’ decision making and emotion simulation modules are tightly integrated. We argue that such agents have better potential for approximating people’s emotion generation processes because for these agents appraisal is an integral part of the cognitive processes they have to perform for interacting with others in a social context.Coping, and in particular emotion-focused coping, is often not included in models of emotion. EMA is a notable exception. EMA is cable of reasoning about coping strategies using the agent’s underlying planning based decision-making process. In this work, we are going to extend Thespian’s existing models of appraisal and coping and add domain-independent models of emotion-focused coping strategies.


### 2.3. Ironic Processes Theory of Depression

Though depression is often associated with low self-esteem, low self-esteem does not necessarily precede the onset of a depression cycle [32]. Rather, sometimes people can maintain a relatively constant self-esteem, and then a rapid drop appears right before depression starts. Recent studies have suggested that depression can result from failed emotion regulation under limited cognitive resources.Cognitive reappraisal, reevaluating the situation in a positive way, has often been used to reduce the unpleasant experience evoked by a stressful event and has been demonstrated to be an effective approach in many cases [33, 34]. On the other hand, it has the risk of invoking negative emotions because the person may overthink the situation [10, 15, 16]. When this happens, these schemas fuel a continuous cycle of depression and can make a situation seem more hopeless to an individual. Moreover, reappraisal requires cognitive resources because the person needs to search through his/her beliefs and past memories for finding the positive side of the current situation. When there are not enough cognitive resources, the person may end up with an even worse mood.The Ironic Processes Theory has suggested that people’s mental control involves two mechanisms: an intentional operating process that seeks to promote the preferred state and an ironic monitoring system that searches for thoughts that indicates the desired state is not yet achieved [15]. The intentional operating process requires effort and the monitoring process is usually unconscious and requires less mental effort. When there are sufficient cognitive resources available, the two processes work together to ensure one can maintain a positive mood. However, when cognitive resources are limited, the effortful operating process becomes less effective, and the monitor process will increasingly find the presence of negative thoughts. Moreover, by highlighting these negative thoughts, the monitoring process also makes the unwanted thoughts more accessible [15, 16]. The Ironic Processes Theory of mental control suggests that the success of emotion regulation thus depends not only on using effective strategies, but also on having sufficient cognitive resources to maintain the effortful, intentional operating process. In this work, we explored an agent-based simulation for suggesting reappraisal strategies under stressful situations.


## 2.1. Cognitive Appraisal Theories

Appraisal theory argues that a person’s emotional experience is determined by his/her subjective assessment and interpretation of internal or external events. Following these theories, one’s emotional experience triggered by the same event can be different from time to time based on how the person interprets the event. This also explains why people’s attitudes and feeling towards an event can change over time.Formally, the assessment occurs along several dimensions, called appraisal variables or checks, such as motivational congruence, accountability, novelty, and control. Emotion is decided by the combination of results from these checks. For example, an event that leads to a bad outcome for a person (motivationally incongruent) and is caused by others (accountability) is likely to elicit an anger response; but if the event is caused by the person himself/herself, that person is more likely to feel guilt or regret [21].Various theories of cognitive appraisals exist. They are different in terms of the appraisal dimensions being considered, how emotions are predicted from the appraisal dimensions, and the process of appraisal, for example, whether it is a fast and shallow process that leverages on the reasoning for decision-making and whether it is a sequential process.One theory that is particularly relevant to this work is Smith and Lazarus’s theory of appraisal [23, 24]. They specify two major types of appraisal methods: (1) primary appraisal, directed at the establishment of the significance or meaning of the event to the organism containing similar appraisal dimensions as in other cognitive appraisal theories, and (2) secondary appraisal, directed at the assessment of the ability of the individual to cope with the consequences of the event. Secondary appraisal is invoked when the event is appraised as negative. It evaluates the person’s potential for coping. The result of the evaluation will be taken into account by the person’s following primary appraisal, and thus form an appraisal-coping-reappraisal loop in people’s cognitive/emotion generation process. Various coping strategies can be applied. They can be roughly divided into two categories: problem-focused coping and emotion-focused coping. The first category contains strategies that seek to change the situation, such as seeking information, action planning, and seeking instrumental support. The second category contains strategies that change one’s beliefs and goals, such as wishful thinking, positive reappraisal, and resignation. People may use any of these strategies to cope with their stressful situations.


## 2.2. Computational Models of Appraisal and Coping

Various computational models for appraisals have been proposed. For applications that aim at simulating emotions for fictional characters, the modeling of emotions is often embedded in the modeling of the interactive story or the character’s decision-making process and uses rule based models. For example, in FLAME, El Nasr et al. use domain-independent fuzzy logic rules to simulate appraisal [27]. Cathexis uses a threshold model to simulate basic variables, which are called “sensors,” related to emotion [28]. Elliott’s Affective Reasoner uses a set of domain-specific rules to appraise events based on the OCC model of emotion [29].Most often in systems for simulating virtual humans or artificial lives, decision making and emotions are modeled separately. In a few agent-based systems, models of emotion leverage the agent’s decision-making representations to model the person-environment relation. For example, EMA [30] defines appraisal processes as operations over a uniform plan-based representation of the agent’s goals, and how events impact those goals. Both EM [31] and ParleE [5] deployed the OCC model of emotion over plan-based agents. In our previous work, Thespian [2] models appraisal as a light weighted process utilizing representations generated by POMDP-based agents’ decision making and belief revision processes. In these systems the agents’ decision making and emotion simulation modules are tightly integrated. We argue that such agents have better potential for approximating people’s emotion generation processes because for these agents appraisal is an integral part of the cognitive processes they have to perform for interacting with others in a social context.Coping, and in particular emotion-focused coping, is often not included in models of emotion. EMA is a notable exception. EMA is cable of reasoning about coping strategies using the agent’s underlying planning based decision-making process. In this work, we are going to extend Thespian’s existing models of appraisal and coping and add domain-independent models of emotion-focused coping strategies.


## 2.3. Ironic Processes Theory of Depression

Though depression is often associated with low self-esteem, low self-esteem does not necessarily precede the onset of a depression cycle [32]. Rather, sometimes people can maintain a relatively constant self-esteem, and then a rapid drop appears right before depression starts. Recent studies have suggested that depression can result from failed emotion regulation under limited cognitive resources.Cognitive reappraisal, reevaluating the situation in a positive way, has often been used to reduce the unpleasant experience evoked by a stressful event and has been demonstrated to be an effective approach in many cases [33, 34]. On the other hand, it has the risk of invoking negative emotions because the person may overthink the situation [10, 15, 16]. When this happens, these schemas fuel a continuous cycle of depression and can make a situation seem more hopeless to an individual. Moreover, reappraisal requires cognitive resources because the person needs to search through his/her beliefs and past memories for finding the positive side of the current situation. When there are not enough cognitive resources, the person may end up with an even worse mood.The Ironic Processes Theory has suggested that people’s mental control involves two mechanisms: an intentional operating process that seeks to promote the preferred state and an ironic monitoring system that searches for thoughts that indicates the desired state is not yet achieved [15]. The intentional operating process requires effort and the monitoring process is usually unconscious and requires less mental effort. When there are sufficient cognitive resources available, the two processes work together to ensure one can maintain a positive mood. However, when cognitive resources are limited, the effortful operating process becomes less effective, and the monitor process will increasingly find the presence of negative thoughts. Moreover, by highlighting these negative thoughts, the monitoring process also makes the unwanted thoughts more accessible [15, 16]. The Ironic Processes Theory of mental control suggests that the success of emotion regulation thus depends not only on using effective strategies, but also on having sufficient cognitive resources to maintain the effortful, intentional operating process. In this work, we explored an agent-based simulation for suggesting reappraisal strategies under stressful situations.


## 3. Model Appraisal and Coping Strategies in Thespian Agent

Thespian contains a model of emotions with five appraisal variables: motivational relevance, motivational congruence, accountability, control, and novelty [2]. Its model of control is roughly equivalent to Smith and Lazarus’s definition of problem focused coping potential [23]. It is a general evaluation of the overall changeability of the situation, but not individual agent’s power to make a change. This work extends the existing model and concentrates on simulating emotion-focused coping strategies. We will first briefly describe the agents in Thespian system.


### 3.1. Thespian Agents

Thespian is a multiagent framework for authoring and simulating virtual characters and interactive. Thespian is built upon PsychSim [35], a multiagent framework for social simulation based on Partially Observable Markov Decision Process (POMDP).Each character in Thespian is modeled as a POMDP-based agent that consists ofstate, actions, goals, andbeliefs. State contains information about an agent’s current status in the world and is defined as a set of state features, such as the name and age of the character, and the social distances between the character and others. Both physical actions and conversational acts can change the values of the state features. The effects of actions on state are probabilistic though sometimes they can be set to certain. For example, walking typically changes the agent’s location in a fixed manner. When an agent tries to gain trust from another agent, the effect of the action is often probabilistic.A character’s motivation and personality profile are modeled as a set of goals and their relative importance (weights). Goals are expressed as a reward function over the various state features an agent seeks to maximize or minimize. The state features can belong to both the agent’s own state and other agents’ states (including the user’s). For example, a character can have goals of both maximizing its happiness and another character’s happiness, with the first goal being ten times more important than the second one. These goals will drive the agent to act accordingly and make tradeoffs if necessary.Thespian agents have a Theory of Mind that allows them to form mental models about other agents. The agent’s subjective view of the world includes its beliefs about itself and other agents and their subjective views of the world, a form of recursive agent modeling. For each character in the simulation, an agent can have multiple mental models about it with different probabilities. For example, Little Red Riding Hood can have two mental models about the wolf: one with the goals and actions of eating people and one without. As the story proceeds, the agent can adjust the probabilities of the alternative mental models.For making decisions, Thespian agents by default use a lookahead policy which projects a number of steps into the future for evaluating alternative actions. The agent chooses the action with the highest expected utility. The expected utility for taking an action is simply calculated asbeliefs∗goals,wherebeliefsdenote the character’s beliefs about the states of itself and others if the action is taken. In Thespian, an agent’s beliefs and goals are both represented as vectors. Using linear algebra, the expected utility can be calculated by multiplying these two vectors. When an agent projects multiple steps into the future, its mental models about others are used for predicting future actions of others, and the agent will pick the action with the highest expected utilities accumulated over the multiple steps.


### 3.2. Existing Models of Appraisal Variables

In our previous work, we created domain-independent models of appraisal variables in Thespian. These appraisal variables are evaluated using intermediate results, states, beliefs, and utilities, obtained during the agent’s lookahead decision making processes [2]. Five appraisal variables have been modeled including motivational relevance, motivational congruence, accountability, control, and novelty. The evaluations of motivational relevance, motivational congruence, and accountability are relevant to the new models proposed in this work and will be briefly described.Motivational Relevance and Motivational Congruence. Motivational relevance evaluates the extent to which an encounter touches upon personal goals, and motivational congruence or incongruence measures the extent to which the encounter thwarts or facilitates the personal goals. They are modeled as a product of the agent’s utility calculations. Thespian agents are goal based decision-theoretic agents. A character’s goals including its social goals, such as following social norms, are encoded as the agent’s goals with different weights. We use the ratio of relative utility change and the direction of utility change to model motivational relevance and motivational congruence. For instance, if earning $500 increases a person’s utility by 10, it will be 10 times more relevant and motivationally congruent if the person’s original utility is 1 compared to the original utility of 10. Thus, as a person’s wealth increases, earning the same amount of money would not bring the same amount of satisfaction. When an event reduces the agent’s utility, the event is motivationally incongruent to the agent, to the extent ofabs (motivational congruence). In this work, when estimating how bad a person feels, we make a simplified assumption that the more an event reduces an agent’s utility, the worse the agent feels.Accountability. Accountability characterizes which person deserves credit or blame for a given event. The algorithm for accountability checks whether the agent who directly causes the harm/benefit to another agent is fully responsible for the event. If the agent is coerced by other agents to take the action, then the agent only takes partial responsibility. Whether an agent is coerced by others is computationally modeled as whether the agent has other action options that will not result in lowering its utility. If such option does not exist, the agent is coerced. For example, when a commander orders a marksman to shoot a prisoner, the marksman will be punished if he does anything else other than following the order.


### 3.3. Model Emotion-Focused Coping

Using Thespian’s agent architecture, it is straightforward to map many of the emotion-focused coping strategies to domain independent operations. In Table1, we mapped two belief related coping strategies, shift responsibility and wishful thinking, and two desired related coping strategies, distance and positive reinterpretation, to operations we can perform in Thespian for simulating the strategies. Note that, for simulating “shift responsibility,” the evaluations of whether the agent is coerced by someone and who coerced the agent are based on existing functionality for evaluating accountability in [2].Table 1
Operations for emotion-focused coping.
Coping strategy
Operations in Thespian
Shift responsibility
If I amcoerced, increase the responsibility of the agent who coerced me
Wishful thinking
Change probabilities of alternative mental models
Distance/mental disengagement
Reduce the importance of the goals that are negatively impacted
Positive reinterpretation/silver lining
Increase the importance of the goals that are positively impacted
Reappraisal of the current situation
Evaluate the impact of one or more previously ignored eventsWhen a person is facing a stressful situation, it is hard to predict which coping strategy the person is going to use. The person may try multiple strategies until finding the one that works best for the situation. For example, if the marksman feels uncomfortable that the prisoner is going to die because of him, he may do any of the following for coping: find out a way to change the commander’s decision (problem-focused coping), decide that he is not responsible (shift responsibility), hope that the command will be changed (wishful thinking), try to not think about the prisoner any more (distance/mental disengagement), feel good that at least the prisoner will not be able to harm other people or the society any more (positive reinterpretation/silver lining), or feel good that at least he has a job (reappraisal of the current situation). Therefore, we make no assumption for the priorities of these strategies when simulating a character. On the other hand, it has been suggested that some patterns of appraisal and coping strategies are maladaptive under certain situations. For example, the Ironic Processes Theory of depression points out that reappraisal is not an effective strategy when cognitive resources are limited. This is because the event/thought that comes to one’s mind during reappraisal may or may not be positive. When cognitive resources are limited, the person may fail to find enough positive events/thoughts to make him/her feel better about the situation. The person may in fact feel worse because of the negative thoughts/events raised during reappraisal. Because reappraisal strategy has been widely used for dealing with depression and its effectiveness has been studied by many researchers, we next simulate the Ironic Processes Theory of depression using reappraisal.


## 3.1. Thespian Agents

Thespian is a multiagent framework for authoring and simulating virtual characters and interactive. Thespian is built upon PsychSim [35], a multiagent framework for social simulation based on Partially Observable Markov Decision Process (POMDP).Each character in Thespian is modeled as a POMDP-based agent that consists ofstate, actions, goals, andbeliefs. State contains information about an agent’s current status in the world and is defined as a set of state features, such as the name and age of the character, and the social distances between the character and others. Both physical actions and conversational acts can change the values of the state features. The effects of actions on state are probabilistic though sometimes they can be set to certain. For example, walking typically changes the agent’s location in a fixed manner. When an agent tries to gain trust from another agent, the effect of the action is often probabilistic.A character’s motivation and personality profile are modeled as a set of goals and their relative importance (weights). Goals are expressed as a reward function over the various state features an agent seeks to maximize or minimize. The state features can belong to both the agent’s own state and other agents’ states (including the user’s). For example, a character can have goals of both maximizing its happiness and another character’s happiness, with the first goal being ten times more important than the second one. These goals will drive the agent to act accordingly and make tradeoffs if necessary.Thespian agents have a Theory of Mind that allows them to form mental models about other agents. The agent’s subjective view of the world includes its beliefs about itself and other agents and their subjective views of the world, a form of recursive agent modeling. For each character in the simulation, an agent can have multiple mental models about it with different probabilities. For example, Little Red Riding Hood can have two mental models about the wolf: one with the goals and actions of eating people and one without. As the story proceeds, the agent can adjust the probabilities of the alternative mental models.For making decisions, Thespian agents by default use a lookahead policy which projects a number of steps into the future for evaluating alternative actions. The agent chooses the action with the highest expected utility. The expected utility for taking an action is simply calculated asbeliefs∗goals,wherebeliefsdenote the character’s beliefs about the states of itself and others if the action is taken. In Thespian, an agent’s beliefs and goals are both represented as vectors. Using linear algebra, the expected utility can be calculated by multiplying these two vectors. When an agent projects multiple steps into the future, its mental models about others are used for predicting future actions of others, and the agent will pick the action with the highest expected utilities accumulated over the multiple steps.


## 3.2. Existing Models of Appraisal Variables

In our previous work, we created domain-independent models of appraisal variables in Thespian. These appraisal variables are evaluated using intermediate results, states, beliefs, and utilities, obtained during the agent’s lookahead decision making processes [2]. Five appraisal variables have been modeled including motivational relevance, motivational congruence, accountability, control, and novelty. The evaluations of motivational relevance, motivational congruence, and accountability are relevant to the new models proposed in this work and will be briefly described.Motivational Relevance and Motivational Congruence. Motivational relevance evaluates the extent to which an encounter touches upon personal goals, and motivational congruence or incongruence measures the extent to which the encounter thwarts or facilitates the personal goals. They are modeled as a product of the agent’s utility calculations. Thespian agents are goal based decision-theoretic agents. A character’s goals including its social goals, such as following social norms, are encoded as the agent’s goals with different weights. We use the ratio of relative utility change and the direction of utility change to model motivational relevance and motivational congruence. For instance, if earning $500 increases a person’s utility by 10, it will be 10 times more relevant and motivationally congruent if the person’s original utility is 1 compared to the original utility of 10. Thus, as a person’s wealth increases, earning the same amount of money would not bring the same amount of satisfaction. When an event reduces the agent’s utility, the event is motivationally incongruent to the agent, to the extent ofabs (motivational congruence). In this work, when estimating how bad a person feels, we make a simplified assumption that the more an event reduces an agent’s utility, the worse the agent feels.Accountability. Accountability characterizes which person deserves credit or blame for a given event. The algorithm for accountability checks whether the agent who directly causes the harm/benefit to another agent is fully responsible for the event. If the agent is coerced by other agents to take the action, then the agent only takes partial responsibility. Whether an agent is coerced by others is computationally modeled as whether the agent has other action options that will not result in lowering its utility. If such option does not exist, the agent is coerced. For example, when a commander orders a marksman to shoot a prisoner, the marksman will be punished if he does anything else other than following the order.


## 3.3. Model Emotion-Focused Coping

Using Thespian’s agent architecture, it is straightforward to map many of the emotion-focused coping strategies to domain independent operations. In Table1, we mapped two belief related coping strategies, shift responsibility and wishful thinking, and two desired related coping strategies, distance and positive reinterpretation, to operations we can perform in Thespian for simulating the strategies. Note that, for simulating “shift responsibility,” the evaluations of whether the agent is coerced by someone and who coerced the agent are based on existing functionality for evaluating accountability in [2].Table 1
Operations for emotion-focused coping.
Coping strategy
Operations in Thespian
Shift responsibility
If I amcoerced, increase the responsibility of the agent who coerced me
Wishful thinking
Change probabilities of alternative mental models
Distance/mental disengagement
Reduce the importance of the goals that are negatively impacted
Positive reinterpretation/silver lining
Increase the importance of the goals that are positively impacted
Reappraisal of the current situation
Evaluate the impact of one or more previously ignored eventsWhen a person is facing a stressful situation, it is hard to predict which coping strategy the person is going to use. The person may try multiple strategies until finding the one that works best for the situation. For example, if the marksman feels uncomfortable that the prisoner is going to die because of him, he may do any of the following for coping: find out a way to change the commander’s decision (problem-focused coping), decide that he is not responsible (shift responsibility), hope that the command will be changed (wishful thinking), try to not think about the prisoner any more (distance/mental disengagement), feel good that at least the prisoner will not be able to harm other people or the society any more (positive reinterpretation/silver lining), or feel good that at least he has a job (reappraisal of the current situation). Therefore, we make no assumption for the priorities of these strategies when simulating a character. On the other hand, it has been suggested that some patterns of appraisal and coping strategies are maladaptive under certain situations. For example, the Ironic Processes Theory of depression points out that reappraisal is not an effective strategy when cognitive resources are limited. This is because the event/thought that comes to one’s mind during reappraisal may or may not be positive. When cognitive resources are limited, the person may fail to find enough positive events/thoughts to make him/her feel better about the situation. The person may in fact feel worse because of the negative thoughts/events raised during reappraisal. Because reappraisal strategy has been widely used for dealing with depression and its effectiveness has been studied by many researchers, we next simulate the Ironic Processes Theory of depression using reappraisal.


## 4. An Agent-Based Experiment of Reappraisal

In this work, we simulate reappraisal and conduct an explorative experiment using autonomous agents simulated by the Thespian framework. Reappraisal is a technique that helps people change their negative mood by asking them to think of something/some event that is positive. By reinforcing the positive side of the situation, people will have a more positive attitude. As mentioned before, this technique will not always work because the side effect that is brought to attention may or may not be positive. If a negative memory is triggered during reappraisal, should the person let it go or try to find the positive side of it? In this work, we try to throw some light into this question through agent-based computational simulation. In this experiment, we want to examine the effect of reappraisal under different stress levels. Moreover, we want to explore what is a better strategy for reappraisal through computational simulations. In particular, when a thought or past event comes to our mind, shall we try to look at it from multiple perspectives, evaluate it against multiple goals, or just the first perspective that comes to mind?


### 4.1. Simulate the Ironic Processes

Following the Ironic Processes Theory of depression, we want to simulate both the intentional operating process and the ironic monitoring process for this experiment. Algorithm1 contains the pseudo code for our experiment.Algorithm 1:Reappraisal_Evl.
Reappraisal_Evl (U0, U1, breadth)
(1) #U0: utility before the stressful event happened
(2) #U1: utility after the stressful event and before reappraisal
(3) #breadth: number of goals to be considered when evaluating an event
(4)
(5) #recovertime: the number of attempts needed before utility recovered
(6) #worse: the number of times the agent feels worse than before the stressful event
(7)recovertimes = [ ]; #an array of results over multiple tests
(8)worses = [ ]; #an array of results over multiple tests
(9)for  t  in range (100):  #test for 100 times
(10) found = false;
(11) worse = 0;
(12)
(13) #reinitialize the agent’s state to right after the stressful event
(14) reinit ();
(15) for  i  in range  (100): #allow the agent to maximally reevaluate the event 100 times
(16)if  found:
(17)recovertime = recovertime  +  [i];
(18)break;
(19)else:
(20)Uti = reappraisal (breadth);
(21)     #in case reappraisal recovered original utility
(22)if  Uti≥U0:
(23)found = true;
(24)     #reappraisal results in even lower utility
(25)elif  Uti < U1:
(26)worse = worse  +  1;
(27) worses = worses  +  [worse];
(28)return  recovertimes, worses;
(29)
(30)reappraisal (breadth): returns the agent’s utility after evaluating the impact of an event/thought that randomly affects
breadthnumber of goals, and at a random level for each goalFor simulating a person having multiple goals with various relative importance, we set up a Thespian agent which has 50 goals with ascending importance; that is, the first goal is weighted at 1, the second goal is weighted at 2. The weights of these 50 goals are normalized. Without losing generality, initially the value of each of the goals is set to 0.1. To put the agent in a situation with different levels of stress, we simulate an event that negatively impacts the agent’s most important goal at three different levels: 0.9 (high stress), 0.5 (moderate stress), and 0.1 (light stress). Next the agent will cope with the situation with reappraisal.The intentional operating process is the process that the agent actively looks for the positive side of the current situation. For simulating the unforeseeable effect of reevaluating one’s current situation, we simulate the scenario that a random event/thought (the agent has no control of whether it is positive or negative) comes to the agent’s mind. This event/thought may be relevant to a number of the agent’s goals and affects each goal at a different level (line 20). If this event/thought brings the agent’s utility to the level of or higher than the level (U0) before the stressful event happened (line 22), then reappraisal is considered successful and we record the number of steps it takes the agent to complete reappraisal (line 17). Otherwise, the agent will evaluate another event/thought. The impact of the latter event/thought is combined with the impacts of all previous events/thoughts the agent has evaluated for reappraising the situation. In other words, if the previous attempts have lowered the agent’s utility, leading the agent into an even worse mood, the agent needs to be extra “lucky” for finding an event that will bring its utility back. This process consumes cognitive resources. The larger the number of steps the agent takes to recover its mood, the more cognitive resources are needed. In this experiment, we set the maximum number of steps to 100.During the intentional operating process, the automated monitoring process is also happening. We simulate this process by checking whether at any moment of the time the agent’s utility is dropped even below its utility before doing reappraisal (U1), which suggests the agent may feel even worse than before it started using reappraisal for coping with its stress. We count the number of times this happens before the agent is able to recover it utility/mood (line 26).


### 4.2. Experiment Setup

To answer the question of when a thought or past event is recalled, whether we should try to look at it from multiple perspectives or just the first perspective that comes to mind, we simulate an agent which evaluates the event/thought with thebreadthof 1, 3, and 5 goals prospectively. In line 20, while the agent performs reappraisal, we let the event/thought affect 1, 3, and 5 of the agent’s goals. For each stress level (high/moderate/light), we simulated the agent trying to cope with the stressful situation using reappraisal 100 times.In this experiment, we set up Thespian agents for simulating four types of populations. The first one is the depressed population. Schwartz suggested for depressed people the percentage of positive thoughts ranges from 0.13 to 0.28 and for healthy population it ranges from of 0.72 to 0.88 [36]. When the depressed agent is performing reappraisal, we set it to having a 20% chance invoking a positive memory and a 80% chance invoking a negative memory. The second one is the healthy population, which has a 80% chance invoking positive memories and a 20% chance invoking negative memories. We also set up two populations in between. One will invoke negative memories 60% of the time, and positive memories 40% of the time. The other has a 50% chance invoking either positive or negative memories.


### 4.3. Results and Discussion

We will first look at the agent that has a 50-50 chance of either invoking a positive or a negative memory. Figure1 shows the histograms of the numbers of attempts the agent made for successfully coping with the situation under each stress level. It can be seen that when the stress level is low, the number of goals that the agent evaluates the event/thought against does not matter. With all three settings, the agent is able to recover within about 10 attempts. However, as the situation becomes more stressful, a more diversified thinking style is more helpful. Following this style, the agent will not give up and look for a new thought/event if the current one does not quite help its mood or even makes its mood worse. It will try to interpret the event from multiple perspectives.Figure 1
Number of attempts for successful reappraisal for 50-50 agents.
(a)
(b)
(c)As shown in Figure1, when coping with a highly stressful situation, if for each event/thought the agent evaluates it against 5 of its goals, there is roughly a 60% of chance the agent will be able to recover its mood within 10 attempts of reappraisal. In contrast, if the agent only evaluates the event/thought against a single goal, there is less than 10% of chance. Therefore, if cognitive resources are limited and the agent does not have enough resources/time to search for positive events/thoughts, the agent is more likely to fail when only looking at events/thoughts from a single perspective.Similarly, a more diversified style of thinking can help prevent the agent from experiencing negative moods resulting from reappraisal. Figure2 shows the histograms of the number of times the agent’s utility falls even below the level when it just experienced the stressful event during its coping process. As we can see, when the situation is highly stressful, evaluating the events/thoughts using only one of the agent’s goals makes the agents much more vulnerable to negative experiences.Figure 2
Number of failures during reappraisal for 50-50 agents.
(a)
(b)
(c)This trend becomes different when the agent is depressed. Unlike the 50-50 agent, the depressed agent (80% of negative memories and 20% of positive memories) cannot successfully cope with the simulation in many cases. Because the agent fails most of the time, we used a table rather than a histogram to report these results. Table2 lists the results from this simulation. The first number is the number of times the agent succeeded out of the 100 simulations we sampled. The second number is the average number of attempts the agent made for coping with the situation when it did succeed. The third number is the agent’s average number of failures—feeling even worse than before reappraisal started—during reappraisal. For example, when coping with moderate level of stress and evaluating each thought/event against 5 different goals, the depressed agent only succeeds 4 out of 100 times. Among these 4 times, the average number of reappraisal needed is 2. Across the 100 simulations, each of which is up to 100 steps, the agent on average feels even worse than without doing reappraisal at all 95% of the time. A trend can be observed here. It is relatively more effective when the agent only evaluates the event/thought against one of its goals instead of multiple goals.Table 2
Results for the depressed agent.
Reappraisal Width
Stress
High
Moderate
Low
5
1 (2) (98)
4 (2) (95)
14 (1) (86)
3
6 (3) (93)
10 (1) (89)
18 (2) (82)
1
15 (4) (84)
18 (4) (80)
18 (2) (80)The results from simulating the agent that has a 60% chance evoking negative memories and a 40% chance evoking positive memories are listed in Table3. Similar to the depressed agent, the agent performs better when only evaluating the event/thought against one of its goals. Here we can also see a similar trend as in the 50-50 agent. The average number of reappraisals needed when the agent did succeed is larger if the event/thought is evaluated against only one goal compared to against multiple goals.Table 3
Results for the agent which has 60% chance evoking negative memories.
Reappraisal Width
Stress
High
Moderate
Low
5
39 (4) (61)
42 (5) (59)
45 (4) (56)
3
50 (6) (50)
54 (5) (46)
62 (7) (41)
1
67 (13) (42)
60 (11) (47)
62 (4) (39)Finally, we simulated the healthy population with 80% chance invoking positive memories and 20% chance invoking negative memories during reappraisal. Similar to the 50-50 agent, this agent successfully coped with the situation all the time. Across all of the conditions, the average number of reappraisal needed in less than 5. There are not many variations in the data, so the details are not reported here. In summary, the stresses we simulated in this study can be very easily dealt with by a normal person, while imposing a great challenge to a depressed person.


## 4.1. Simulate the Ironic Processes

Following the Ironic Processes Theory of depression, we want to simulate both the intentional operating process and the ironic monitoring process for this experiment. Algorithm1 contains the pseudo code for our experiment.Algorithm 1:Reappraisal_Evl.
Reappraisal_Evl (U0, U1, breadth)
(1) #U0: utility before the stressful event happened
(2) #U1: utility after the stressful event and before reappraisal
(3) #breadth: number of goals to be considered when evaluating an event
(4)
(5) #recovertime: the number of attempts needed before utility recovered
(6) #worse: the number of times the agent feels worse than before the stressful event
(7)recovertimes = [ ]; #an array of results over multiple tests
(8)worses = [ ]; #an array of results over multiple tests
(9)for  t  in range (100):  #test for 100 times
(10) found = false;
(11) worse = 0;
(12)
(13) #reinitialize the agent’s state to right after the stressful event
(14) reinit ();
(15) for  i  in range  (100): #allow the agent to maximally reevaluate the event 100 times
(16)if  found:
(17)recovertime = recovertime  +  [i];
(18)break;
(19)else:
(20)Uti = reappraisal (breadth);
(21)     #in case reappraisal recovered original utility
(22)if  Uti≥U0:
(23)found = true;
(24)     #reappraisal results in even lower utility
(25)elif  Uti < U1:
(26)worse = worse  +  1;
(27) worses = worses  +  [worse];
(28)return  recovertimes, worses;
(29)
(30)reappraisal (breadth): returns the agent’s utility after evaluating the impact of an event/thought that randomly affects
breadthnumber of goals, and at a random level for each goalFor simulating a person having multiple goals with various relative importance, we set up a Thespian agent which has 50 goals with ascending importance; that is, the first goal is weighted at 1, the second goal is weighted at 2. The weights of these 50 goals are normalized. Without losing generality, initially the value of each of the goals is set to 0.1. To put the agent in a situation with different levels of stress, we simulate an event that negatively impacts the agent’s most important goal at three different levels: 0.9 (high stress), 0.5 (moderate stress), and 0.1 (light stress). Next the agent will cope with the situation with reappraisal.The intentional operating process is the process that the agent actively looks for the positive side of the current situation. For simulating the unforeseeable effect of reevaluating one’s current situation, we simulate the scenario that a random event/thought (the agent has no control of whether it is positive or negative) comes to the agent’s mind. This event/thought may be relevant to a number of the agent’s goals and affects each goal at a different level (line 20). If this event/thought brings the agent’s utility to the level of or higher than the level (U0) before the stressful event happened (line 22), then reappraisal is considered successful and we record the number of steps it takes the agent to complete reappraisal (line 17). Otherwise, the agent will evaluate another event/thought. The impact of the latter event/thought is combined with the impacts of all previous events/thoughts the agent has evaluated for reappraising the situation. In other words, if the previous attempts have lowered the agent’s utility, leading the agent into an even worse mood, the agent needs to be extra “lucky” for finding an event that will bring its utility back. This process consumes cognitive resources. The larger the number of steps the agent takes to recover its mood, the more cognitive resources are needed. In this experiment, we set the maximum number of steps to 100.During the intentional operating process, the automated monitoring process is also happening. We simulate this process by checking whether at any moment of the time the agent’s utility is dropped even below its utility before doing reappraisal (U1), which suggests the agent may feel even worse than before it started using reappraisal for coping with its stress. We count the number of times this happens before the agent is able to recover it utility/mood (line 26).


## 4.2. Experiment Setup

To answer the question of when a thought or past event is recalled, whether we should try to look at it from multiple perspectives or just the first perspective that comes to mind, we simulate an agent which evaluates the event/thought with thebreadthof 1, 3, and 5 goals prospectively. In line 20, while the agent performs reappraisal, we let the event/thought affect 1, 3, and 5 of the agent’s goals. For each stress level (high/moderate/light), we simulated the agent trying to cope with the stressful situation using reappraisal 100 times.In this experiment, we set up Thespian agents for simulating four types of populations. The first one is the depressed population. Schwartz suggested for depressed people the percentage of positive thoughts ranges from 0.13 to 0.28 and for healthy population it ranges from of 0.72 to 0.88 [36]. When the depressed agent is performing reappraisal, we set it to having a 20% chance invoking a positive memory and a 80% chance invoking a negative memory. The second one is the healthy population, which has a 80% chance invoking positive memories and a 20% chance invoking negative memories. We also set up two populations in between. One will invoke negative memories 60% of the time, and positive memories 40% of the time. The other has a 50% chance invoking either positive or negative memories.


## 4.3. Results and Discussion

We will first look at the agent that has a 50-50 chance of either invoking a positive or a negative memory. Figure1 shows the histograms of the numbers of attempts the agent made for successfully coping with the situation under each stress level. It can be seen that when the stress level is low, the number of goals that the agent evaluates the event/thought against does not matter. With all three settings, the agent is able to recover within about 10 attempts. However, as the situation becomes more stressful, a more diversified thinking style is more helpful. Following this style, the agent will not give up and look for a new thought/event if the current one does not quite help its mood or even makes its mood worse. It will try to interpret the event from multiple perspectives.Figure 1
Number of attempts for successful reappraisal for 50-50 agents.
(a)
(b)
(c)As shown in Figure1, when coping with a highly stressful situation, if for each event/thought the agent evaluates it against 5 of its goals, there is roughly a 60% of chance the agent will be able to recover its mood within 10 attempts of reappraisal. In contrast, if the agent only evaluates the event/thought against a single goal, there is less than 10% of chance. Therefore, if cognitive resources are limited and the agent does not have enough resources/time to search for positive events/thoughts, the agent is more likely to fail when only looking at events/thoughts from a single perspective.Similarly, a more diversified style of thinking can help prevent the agent from experiencing negative moods resulting from reappraisal. Figure2 shows the histograms of the number of times the agent’s utility falls even below the level when it just experienced the stressful event during its coping process. As we can see, when the situation is highly stressful, evaluating the events/thoughts using only one of the agent’s goals makes the agents much more vulnerable to negative experiences.Figure 2
Number of failures during reappraisal for 50-50 agents.
(a)
(b)
(c)This trend becomes different when the agent is depressed. Unlike the 50-50 agent, the depressed agent (80% of negative memories and 20% of positive memories) cannot successfully cope with the simulation in many cases. Because the agent fails most of the time, we used a table rather than a histogram to report these results. Table2 lists the results from this simulation. The first number is the number of times the agent succeeded out of the 100 simulations we sampled. The second number is the average number of attempts the agent made for coping with the situation when it did succeed. The third number is the agent’s average number of failures—feeling even worse than before reappraisal started—during reappraisal. For example, when coping with moderate level of stress and evaluating each thought/event against 5 different goals, the depressed agent only succeeds 4 out of 100 times. Among these 4 times, the average number of reappraisal needed is 2. Across the 100 simulations, each of which is up to 100 steps, the agent on average feels even worse than without doing reappraisal at all 95% of the time. A trend can be observed here. It is relatively more effective when the agent only evaluates the event/thought against one of its goals instead of multiple goals.Table 2
Results for the depressed agent.
Reappraisal Width
Stress
High
Moderate
Low
5
1 (2) (98)
4 (2) (95)
14 (1) (86)
3
6 (3) (93)
10 (1) (89)
18 (2) (82)
1
15 (4) (84)
18 (4) (80)
18 (2) (80)The results from simulating the agent that has a 60% chance evoking negative memories and a 40% chance evoking positive memories are listed in Table3. Similar to the depressed agent, the agent performs better when only evaluating the event/thought against one of its goals. Here we can also see a similar trend as in the 50-50 agent. The average number of reappraisals needed when the agent did succeed is larger if the event/thought is evaluated against only one goal compared to against multiple goals.Table 3
Results for the agent which has 60% chance evoking negative memories.
Reappraisal Width
Stress
High
Moderate
Low
5
39 (4) (61)
42 (5) (59)
45 (4) (56)
3
50 (6) (50)
54 (5) (46)
62 (7) (41)
1
67 (13) (42)
60 (11) (47)
62 (4) (39)Finally, we simulated the healthy population with 80% chance invoking positive memories and 20% chance invoking negative memories during reappraisal. Similar to the 50-50 agent, this agent successfully coped with the situation all the time. Across all of the conditions, the average number of reappraisal needed in less than 5. There are not many variations in the data, so the details are not reported here. In summary, the stresses we simulated in this study can be very easily dealt with by a normal person, while imposing a great challenge to a depressed person.


## 5. Future Work

The goal in this work is to create domain independent algorithms for modeling emotion-focused coping behaviors in Thespian agent. We hope to be able to use these algorithms to model coping behaviors in normal people, people with depression, and people with other psychological disorders, such as anxiety and posttraumatic stress disorder.Our future work is planned in several directions. The immediate future work is to enrich the types of coping strategies we model, such as attention based coping strategies. Based on a collection of coping strategies, for the next step we want to study what makes people pick one strategy over another and make transitions among the strategies. We will explore both theory-based and data-driven approaches for creating future computational models.The ultimate goal is to understand at the individual level why one feels in certain way and be able to predict the future psychological experiences and behavior patterns for individuals as well as creating believable interactive characters. Emotion and mood play an important role in forming people’s experiences. They are not simply subjective feelings. Emotion can help one to prepare to react to events happening in the environment and coordinate people’s social behaviors. The expression and feeling of social emotions, such as shame not only affect social interaction, but also help glue a social group together [37, 38]; that is, people do not want to feel guilty and therefore they will not perform actions that will harm others or the society. Emotion and mood can also bias how people perceive the environment, interpret events, make decisions, and retrieve past memories. For example, when people are in a happy and pleasant mood they may perform shallower information processing than when they are in a bad mood [39]. We are interested in extending Thespian agent for modeling these phenomena.


## 6. Conclusion

In this work, we proposed domain independent models for emotion-focused coping in decision-theoretic goal based agents. We simulated emotion regulation by reappraisal and demonstrated how reappraisal can lead to negative experiences under high levels of stress and restricted cognitive resources. We performed a sequence of agent-based simulations for exploring the relationship between reappraisal, cognitive resources, and depression. We are able to replicate the scenario predicted by the Ironic Processes Theory of depression that reappraisal is more likely to fail when cognitive resources are limited. Moreover, our simulation suggests that, for the healthy population, it is more beneficial to evaluate an event against multiple goals. In contrast, for the depressed population it is more beneficial to only evaluate against one of the person’s goals. In the future, we hope to collaborate with psychiatrists for evaluating this result empirically.



---

*Source: 856726-2015-08-12.xml*