Cognitive Emotions in Reinforcement Learning Agents
By: Joel Huang
Tags: Emotions, Cognitive Psychology, Cognitive Science, Artificial Intelligence, Reinforcement Learning

Cognitive Emotions in Reinforcement Learning Agents

In this article, we suggest methods of implementing appraisals in reinforcement learning agents, who use them to alter their behavior. By computing the cognitive appraisals of motivational relevance, novelty, and accountability, reinforcement learning agents can use to appraise fully or partially-observable state representations from the environment. A neural network based agent architecture learns internal models of these appraisal variables, to a slight improvement of performance over agents without appraisals.

Pac-man, a grid-world agent, displaying emotions. Closely related, but this article isn’t about Pac-man. Image collated from the game Pac-man Party (2010).

What are appraisals?

Human emotions have been studied as discrete emotions (happy, angry, sad, etc.), dimensional models (valence/arousal), cognitive and rational processes, and social mechanisms. Appraisal theory is part of the cognitive view, and modelling emotion as appraisals allows us to express emotional state natively in terms of parameters and variables in the environment.

The component view as a loop: The environmental state is appraised, affect is derived, emotion is felt, and behavior is changed, feeding back into the environment. Figure from Marsella et. al, 2010.

This figure shows the component view of computational emotion models, in which a closed loop is formed between the environmental context, appraisal variables, emotional state, and resultant behavior. There is the agent-environment relationship, which defines the space of possible interactions; the appraisal-derivation model, which transforms the interpretation of the environment and situations into appraisal variables; and appraisal variables, like motivational relevance, uncertainty, and recency.

In the full component view the appraisal variables are then used to determine affective state and physiological emotion, but we hypothesize that appraisal variables already contain enough information about emotional state to inform choices of actions directly (but if you were designing an embodied agent, you would include physiological aspects of emotion).

Why can’t we just predict ‘happiness’ or ‘anger’?

There are lots of studies on using AI to interpret discrete emotions like ‘happiness’ or ‘sadness’, but we consider them simply lexical symbols imposed on patterns of behavior and physiological states which can differ from human to human. Modelling emotion as appraisal closely parallels intrinsic motivation in agents, and several studies suggest that agents perform better at tasks when they possess models of emotion. Many of these come from the fields of decision theory and game theory, where these states are often model-free, and are implicitly written into global reward signals or loss functions. For instance, counterfactual regret minimization algorithms model regret as the difference between the utility of an action taken and the utility of a possible action, directly minimizing it as an optimization objective. If we wish to decode these signals into discrete categories, we are still able to (see classification tables such as in Roseman et. al, 1996).

What can we model instead?

The appraisal variables must also be modelled with the specific environment in mind, since many of them are environment-dependent. While external reward is not strictly necessary for agents to learn from their environment, such as when external motivators, such as pleasure, or homeostasis — such as hunger and thirst — function as a primary signal or can be collapsed with intrinsic reward into a single or multi-dimensional signal, implementing the environment with external rewards can speed up convergence and allow you to easily evaluate its performance.

We previously mentioned that appraisal variables are native to the environment. This means that the intrinsic, or internal, nature of appraisal variables — such as consistency motives, probability of events, agency, motivational state, and degree of control over situations — allows them to be modelled in terms of environmental situations and parameters, as opposed to discrete and dimensional predictions, which have no direct relationship with the environment.

Let’s look at an example of three frequently investigated appraisal variables: motivational relevance, novelty, and accountability. Motivational relevance is basically asking: “How relevant is this situation to my goals?”, novelty is asking: “How surprising is this situation?”, and accountability is asking: “How much credit or blame do I assign myself and others for this situation?”. Novelty is generally applicable to environments that are partially observable. We can thus formulate motivational relevance and novelty as appraisal signals tied highly to the environment, and accountability as a signal coupled with the interaction with other agents.

Reinforcement Learning

Reinforcement learning has been widely used for planning and decision-making in games and simulated environments. The environment, together with the presence of other agents, imposes behavioral and learning constraints on each agent. This is particularly important as our appraisal process requires the estimation of signals in the environment in order to model emotion. Let’s look at a very simple grid world environment to see how we can integrate appraisal into the learning process.

The dynamic obstacles environment we are going to be using. The world state is a grid tensor. Figure from gym-minigrid, open source.

In this experiment we make use of proximal policy optimization (PPO). Inspired by trust region policy optimization, PPO forces the new policy parameters to stay within a small range of the old ones at every iteration, intended to improve training stability. The policy is usually modeled with a parameterized function (e.g., a deep neural network). The value of the reward function depends on this policy, which can be optimized with a variety of algorithms, especially gradient-based optimization.

The agent as a deep neural network

At every time step, all the agent is supposed to do is take in the world state, and output a value for that state, and the policy (the distribution over actions to take in that state).

While the gradient is backpropagated on the entire model’s parameters via a single loss function, there are four functional components of the model: the eyes (a convolutional neural network, or CNN), the episodic memory (a long short-term memory module, or LSTM), and two fully-connected neural networks that produce the value and policy.

The model architecture. The red layer is the episodic memory LSTM. The blue layers take in an optional text string describing the environment or the task. Figure from rl-starter-files, open source.

The agent takes in the world state through a small, three-layer convolutional neural network (CNN), to capture the spatial information in the input tensor in the partially-observed view dimensions (e.g. if the world state is a 10x10 grid, then we might only observe 7x7). The resulting state embedding is a flattened vector, which is used by the subsequent episodic memory layer.

The agent retains information throughout each episode of training via recurrent memory, implemented using a LSTM cell that passes on a hidden state and a cell state to the LSTM cell at the next timestep. The starting hidden and cell states are set to zero. At each timestep, the episodic memory takes as input the flattened embedding. From these inputs it creates the next hidden state, which still represents the state feature while containing additional temporal information. The hidden state is then used as the output embedding passed on to the actor and critic layers to compute the final output.

The actor and critic layers are simple two-layer fully connected (FC) networks that independently receive the memory embedding from the episodic memory layer, and produce the action logits and value estimate respectively. The number of actions is 3, so the actor layer has 3 output units. The critic layer has 1 output unit. The outputs are used to calculate the loss terms, and the gradient based on the total loss is then backpropagated all the way to the first convolutional layer of the model.

How do we add appraisals?

So everything described above is a traditional PPO agent trained on a three-term loss objective. But how do we add emotion to this agent? A cool idea is to maintain this emotion through the episodic memory.

In an episode, the agent maintains an episodic memory through the hidden and cell states of its LSTM layer. We can augment the cell state of the LSTM to include the three appraisal variables, so that the appraisal variables always influence the memory embedding at each timestep. This requires increasing the hidden state of the LSTM by the number of appraisal variables, resulting in both the hidden and cell states being three units larger than before. Let’s look at possible formulations of motivational relevance, novelty, and accountability:

Motivational relevance

When a situation is highly relevant to one’s well-being, it elicits a more intense emotional response. Motivational relevance is the measure of this situational relevance, and is usually highly dependent on the environment — in path or graph traversal problems, they have been formulated based on distances to goal states. We argue that a grid world is similar — since the agent’s state input is constrained by its view, it may not always encounter state representations that are relevant. we can consider a state input highly relevant if it contains the goal (the goal is in view of the agent). As the agent moves closer to the goal, the relevance of the state should increase. We therefore implement motivational relevance based on the agent’s current Manhattan (L1) distance to the goal.

The Manhattan, or L1, distance to the goal.

The motivational relevance is a linear function of the L1 distance, intended to ensure that MR is nonzero when the goal is in sight, and increases linearly as the distance to the goal gets shorter up till MR=1, when the agent is right next to the goal (L1 distance=1).

Novelty

Similarly tied to the environment and especially useful in a partially observable setting, novelty is the amount of surprise (or uncertainty) present in a state. Note that the policy learnt by the agent specifies good actions to take given states that it has previously seen (i.e., not novel). When a novel state is encountered, the policy network may not be able to select an action with confidence or certainty. We thus quantify this uncertainty as the negative reverse KL-Divergence between the uniform distribution (which represents perfect uncertainty) and the probability distribution over the logits predicted by the actor network at the previous timestep. We use the softmax function to obtain a distribution P from the logits, and Q is the uniform distribution. Since a high divergence indicates certainty in a particular action, we take the negative of the divergence to quantify the uncertainty.

P is the softmax over the logits, and novelty is the negative reverse KL-Divergence between the uniform and P.

Accountability

Accountability is a time dependent appraisal variable that represents the assignment of credit (for a positive outcome) or blame (for a negative one) to the self or the other (an overloaded term encompassing other agents as well as environmental factors). At every step, the agent selects an action given a state. However, before this action is taken, other agents are allowed to move into any empty tiles (tiles that do not contain the goal, other agents, or the primary agent). If another agent were to move into the tile that the agent intends to move forward to, then we receive a reward of -1 and the episode ends. We consider the agent less accountable for this outcome than if the agent deliberately collided with another agent that did not change its position, since other agents can either move with probability p, or stay in place with probability (1-p). We can set the accountability as:

Accountability as a binary variable dependent on the cause of action that causes a negative reward.

As a result, accountability is a sparse signal present when a nonzero reward is obtained. In our environment, this is approximately once per episode, since the episode terminates when the reward is collected (in certain policies, we might hit the maximum number of frames instead, since the agent can be stuck in a training minima such that it does not achieve the goal, but manages not to collide).

The result of appraisals in agents

The environment and agent setup with graphical indications of the appraisal values at each timestep. Image by author.

In all our experiments, the inclusion of the appraisal variables in the cell state of the agent seems to allow it to converge faster. We provide experimental results from the average of three runs. The agent was trained for 500k iterations, under PPO with 4 epochs, 16 parallel processes, discount of 0.99, a learning rate of 5e-3, a generalized advantage estimate coefficient of 0.95, loss coefficients of 0.5, a clipping range of 0.2, and a recurrence of 4. Recurrence refers to the number of times that the gradient is backpropagated through time (BPTT).

Mean reward offers an indication of performance of the agents. This figure shows the comparison between the agent without appraisal (orange) and the agent with appraisal (blue). With appraisals, the agent reaches a positive mean faster by about 50k iterations than without appraisals. The average reward after 500k iterations is also higher for the agent with appraisals. Image by author.

Mean number of frames between the agent without appraisal (orange) and the agent with appraisal (blue). The mean number of frames tells us how many frames, on average, the agent takes to complete the episode. A high number of frames indicates the agent is “surviving” by avoiding obstacles, but not reaching the goal. The appraisal agent does not enter a survival mode as extreme as the regular agent, which can “survive” for up to 400 frames (the maximum number of frames allowed per episode). Image by author.

The value loss at each iteration for the agent without appraisal (orange) and the agent with appraisal (blue). The value loss is the squared error between the value estimate and the value target. In the early stages of training, the appraisal agent incurs a higher value loss, while the regular agent has a lower value loss. However, towards the end of training, the appraisal agent manages to maintain a stable, and in some cases, lower, value loss than the regular one. Note that the lowest value loss incurred by the regular agent (the orange dip at the beginning) is due to the survival phase (see the spike in number of frames in the previous figure). Image by author.

Behaviorally, the trained agent exhibits observable difference in its actions. During learning, the appraisal agent exhibited more exploratory behavior. It does not get stuck in a serious minima where its survival phase restricts it from approaching the goal state (see orange spike at 40k iterations in the second plot). This could be due to the novelty or accountability signals providing more state information that is relevant to the agent’s training curriculum.

Improvements

Recall that we formulate motivational relevance as a linear function of the L1 distance. However, motivational relevance can not only be positive, it can be negative. We can look back some timesteps from a collision or a goal move and consider those states highly relevant. We also formulated novelty as a function of the policy logits, similar to entropy. Further connection between these two variables should be examined, since one of them is used in a memory state while the other is directly maximized in the objective function. Currently, accountability is modelled as a binary variable, but alternative formulations can be considered, such as a moving average over accountability values, especially since this signal is sparse.

While we did not use the appraisals as a training objective, it is possible to include them in the loss function. For example, the agent might be trained to maximize motivational relevance, or keep motivational relevance above a certain threshold, which could assist in steering its policy towards the vicinity of the goal area. On the other hand, since both novelty and the PPO entropy term are derived from the policy logits, either maximizing or minimizing novelty might interfere with the entropy loss and it is advisable to drop one or the other.

Applications

Better human-computer interaction

Humans have been used in the deep reinforcement learning loop to train and correct agents, and having appraisal models can allow humans to verify formulations and correct them on the fly during the training process, creating a more truly social curriculum with humans rather than fellow agents. We suggest results of being able to visualize some form of affect in seemingly black-box agents (such as the environment plot above) helps us be more accepting and understanding of algorithmic choices.

Appraisal-driven conversational agents

Existing conversational agents are able to understand the semantics of a conversation, yet, fail to uphold a complete, realistic conversation. This is due to the lack of pragmatic and emotional understanding of the dialogue. Appraisal-driven agents can bridge this chasm by providing a contextual and affective knowledge to the agent. In the field of natural language processing, chatbots have been trained with reinforcement learning, and training visual and environmental question answering agents has gained traction in the last three years. Appraisal-driven language models that are learnt from the environment might be an interesting line of research to follow, since it seems to be supported from our learning experiences as well as psychology literature (e.g. Piaget’s theory of development).

Well, that’s it!

There are many other appraisals that we can implement, and some of them are already an established part of RL literature (such as curiosity, regret, etc.). This experiment was particularly enjoyable because we get a clearer picture of the previously black-box decision-making processes that agents seemed to take.