Acquiring emotion mappings through the interaction
between a user and a life-like agent
YAMADA Seiji
CISS, IGSSE
Tokyo Institute of Technology
yamada@ymd.dis.titech.ac.jp

YAMAGUCHI Tomohiro
Department of Information Science
Nara National College of Technology
yamaguch@info.nara-k.ac.jp
Abstract

This paper describes a human-agent interaction framework in which a user and a life-like agent
mutually acquire their emotion mappings through a mutual mind reading game. In these several years,
a lot of studies have been done on a life-like agent like a Micro Soft agent, an interface agent, and so on.
Through the development of various life-like agents, emotion has been recognized to play an importantrole in making them believable to a user. For making eﬀective and natural communication between alife-like agent and a human user, they need to identify the other’s emotion state from expressions andwe call mappings from expressions to emotions emotion mappings. If an agent and a user don’t obtainthese emotion mappings, they can not utilize behaviors which signiﬁcantly depend on the other’s emotion
states. We try to formalize the emotion mapping and a human-agent interaction framework in which auser and a life-like agent mutually acquire emotion mappings each other. In our framework, a user plays
a mutual mind reading game with a life-like agent and they gradually learn emotion mappings each otherthrough the game.

1

Introduction

In these several years, a lot of studies have been done on a life-like agent like a Micro Soft agent[1], aninterface agent[4][5], and so on. A typical life-like agent appears on a Web shopping page and support auser in inputting his/her order and data into the page by speech recognition and synthesis. Through thedevelopment of various life-like agents, emotion has been recognized to play a very important role in making
them believable to a user[2]. Emotional expressions are also eﬀective to computer-mediated communication
between humans[8]. These facts are supported by many psychologists’ reports that the emotion signiﬁcantly
inﬂuences even rational behaviors as well as instinctive behaviors of a human[7]. Thus some researchers aretrying to implement an emotion model on a life-like agent for developing a believable one to a user[2][11].
However there is a signiﬁcant problem that emotion identiﬁcation is diﬃcult to both of an user and an agent.
For making the eﬀective communication between a life-like agent and a human user, they need to beable to identify the other’s emotion state through the other’s expression and we call this task emotion
identiﬁcation. If the emotion identiﬁcation is impossible, they are not able to act human-like behaviors
which signiﬁcantly depend on the other’s emotion states. For example, we consider a life-like agent shouldkindly and carefully behave to a depressed user, and intuitively communicate its computational state to auser through a facial expression. Though emotion identiﬁcation is always done among human, the emotion
identiﬁcation between a life-like agent and a user becomes far more diﬃcult than between human. Because
design of life-like agent’s expressions signiﬁcantly depends on personal preference, culture, and not all theusers can understand the emotion by seeing an expression. Consequently a life-like agent and a user needto acquire relation between an expression and an emotion state when they actually encounter. We call such
relation emotion mapping.
In this paper, we propose a human-agent interaction framework in which a user and a life-like agent
mutually acquire emotion mappings each other. In our framework, a user plays a mutual mind reading gamewith a life-like agent and they gradually learn emotion mappings each other through the game.

First, for describing emotional interactions between a life-like agent and a human, we deﬁne an emotion
state, primitive emotions, expressions, emotion transition rules and emotion mapping. Next we develop
procedures of an agent to learn a user’s emotion mappings. Finally, to acquire emotion mappings each other,
we develop a mutual mind reading game in which a user and a life-like agent try to recognize the other’semotion state through the other’s (facial) expression. They mutually answer their results on the other’semotion, and one having a correct answer gets a score. This game is designed so that a user may enjoy it,
and as results, the user’s cognitive load is reduced.
Velásquez proposed a emotion model which is based on Minsky’s society of mind[6]. His model[11][10] isfor generating human-like emotions using a multi-agent system architecture in which each agent corresponds
to a primitive emotion like fear, joy and so on. Interactions among agents are deﬁned and emotions are
emerged as a result of the interactions. However the purpose of his research is to generate various emotions
and moods like a human, and a framework for interaction between an agent and a user like an emotion
mapping.

2

Emotion mapping

In this section , we deﬁne our framework to deal with emotional interactions between a agent and a humanuser. First the following primitives are introduced for describing the framework.
• Emotion state S1 , · · · , S l : A variable standing for a state of an agent i’s emotion. The next primitive
emotion is substituted for this variable.
• Primitive emotion E i = {ei1 , · · · , e im }: Ei is a set of elements of agent i’s emotion. We can utilize
typical elements like fear, joy and so on.
• Primitive expression Xi = {xi1 , · · ·, x in }: Xi is a set of primitive expressions. We deal with a single
primitive expression, not a combination of plural primitive expressions.
i= {x1 → es , · · · , x n → et }: This means an agent(or a user) j’s one-to-one• Emotion mapping Mj:x→emapping from a primitive expression to a primitive emotion which was learned by an agent(or a user) i.
For simplicity, we assume that a single primitive expression corresponds to a single primitive emotion.
i= {e1rightarrowx s , · · · , e m → xt }: An agent(or a user) i’s one-to-one• Expression mapping Mi:e→x
mapping from a primitive emotion to a primitive expression.

• Emotion transition rule T a = {eat × ēbt → eat+1 }: An agent(or a user) a’s rule determining a transitionof an emotion state from a primitive emotion e at to a primitive emotion e at+1 when the a estimates theother agent(or a user) b’s emotion ebt from its expression. The ē is an estimated primitive emotion ofe.
Using the above notations, we describe a framework in which a life-like agent a and a user h mutuallyinteract through expressions as shown in Fig.1.

3

Mutual learning of emotion mappings

3.1

What should be learned?

With the framework described in Fig.1, we can deﬁne learning of emotion mappings and mutual learning of
emotion mappings in the following.
a
Learning of emotion mappings: An agent(or a human user) acquires emotion mappings Mh:x→e(or
h
Ma:x→e ) of a partner.

Mutual learning of emotion mappings: An agent and a human user mutually acquire the other’s emoahand Ma:x→e.
tion mappings, Mh:x→eHuman hhhAgent ahxh = M (e i )h: e → aeiaM (x h) = emh

h: x → ehe na = M (x a )a: x → ehej

Figure 1

xh

xa

Th

a

em
Ta

ena
x a= M (e na )aa: e → a

A framework for emotional interactions between a agent and a user.

Since a partner of a user in our framework is a life-like agent which simulates human behaviors, we canassume the following assumptions to restrict our framework for simplifying mutual learning of emotional
mappings.
(A1) Common primitive emotions: Primitive emotions of a user and an agent are identical, thus E = E a =E h . Also
(A2) Partially common emotion transition rules: Emotion transition rules of a user and an agent are partially
identical. It is just partially because the emotion transition rules are not so stable depending on
situations and contexts.

3.2

Learning in an agent

Because a human user is able to autonomously learn agent’s emotion mappings in our framework, we giveno restriction to a user within his/her learning. Thus we develop only the learning procedures of an agent.
From an assumption (A1), an agent does not need to acquire user’s primitive emotions. Furthermore theuser’s expressions are observed by an agent with a computer vision system using a CCD camera. Hence if auser’s primitive emotion eh is estimated by a sort of reasoning when a user’s expression xh is observed, anagent acquires an instance of an emotion mapping, x h → eh . After an agent stores suﬃcient such instances
through interactions with a user, it becomes able to estimate a user’s primitive emotion from his/her observed
primitive expression by a N N (nearest neighbor) method[3].
Thus we need to estimate a user’s primitive emotion without his/her observed primitive expression.
Under an assumption (A2), we utilize some rules to reason the primitive emotion like the following.
• IF much failure continues THEN a user gets distressed.
• IF a user successively wins THEN he/she feels joy.

4

Mutual mind reading game

This section describes a mutual mind reading game in which a user and a life-like agent try to recognize thepartner’s emotion state through the other’s (facial) expression as the learning task of emotion mappings.

4.1

The Objective of the Mutual Mind Reading GameThe objective of the mutual mind reading game is to collect a set of instances for a N N method both
eﬃciently and broadly for the learning of emotion mapping. An instance is a pair of a estimated primitive
emotion and a observed facial expression. In this paper, a game in which a player estimates the partner’semotion state through the facial expression to compete for the accuracy is called a mind reading game.
“Mutual” means that both the game in which a life-like agent estimates the user’s emotion state throughhis/her facial expression and the game in which the user estimates the agent’s emotion state through its facial
expression are performed in parallel. The data sampling process for an identiﬁcation of the user modelingis called enrollment. The main problem of an enrollment is that the user’s cognitive load becomes high.
To solve this, this game is designed so that a user may enjoy it to play a part in collecting training data
actively, and as results, the user’s cognitive load becomes low. In general, interactions among multi-agent
are classiﬁed into two types, competitive or cooperative[9]. A competitive mind reading game is to competewith each other for the accuracy in reading the emotion of an opponent between two players in the game.
On the other hand, a cooperative mind reading game is to compete with other pairs for the time to ﬁnish
reading the emotion of a partner accurately between two players in the game. Note that in both games, aneﬀort to win the game must not give bad inﬂuence for sampling the training examples of natural pairs ofemotion and expression.

4.2

The Overview of a Cooperative Mind Reading Game

We apply a cooperative mind reading game in which a user and an agent play cooperatively to the learningof emotion mapping.
Assumptions of the gameThe primary objective of the game is the emotion identiﬁcation between a life-like agent and a user. Therefore, both a life-like agent and a user play the game with ﬁxed emotion mapping each other. Besides, users
play the game with not to be stretched but to be natural facial expressions as if they behave usual expressions
in their daily life.
The rules of the game
1. When a player is able to guess the partner’s emotion state through the partner’s facial expression, theplayer says the supposed primitive emotion to the partner.
2. When a player is said a primitive emotion, the player must reply whether “Yes” (the guess is right),“No” (the guess is wrong) or “Near-Miss” (the guess is wrong, but it is close to the answer) as the
judgment against the partner’s guess.
3. These processes are played until a ﬁnish condition of the game is satisﬁed.
A finish condition of the game
When each two players guess the partner’s diﬀerent primitive emotions for n times of continuation, the gameis ﬁnished. The frequency of the correct guess of n can be changed according to a degree of diﬃculty of thegame. Note that ﬁnally, it is equal to the number of primitive emotions treated in the emotion model.

4.3

Discussions

First, we discuss the assumption of our mind reading game. There are two reasons why ﬁxed emotion mapping
is assumed during the game. First reason is that ﬁxed emotion mapping which is equal to that of daily use
generates natural facial expressions. Second reason is that ﬁxed emotion mapping makes the learning of
opponent mapping easier. Next, we argue about the intention that we used a game for emotion mapping
identiﬁcation. It is not easy for a human to hold various natural emotions in a short time while collecting
training examples of facial expressions. For this, the learning of emotion mapping based on the mutual mind
reading game is continued until the end condition of the game is satisﬁed. During the game, various kindsof user’s emotions that arise from the result of the guess cause his/her natural facial expression. Finally, we
describe convergence conditions of the mutual learning of emotion mapping. The necessary conditions of themutual learning are the following;1. Each learning of the opponent emotion mapping is converged.
hawhich a user learned becomes equal to an inverse mapping of Ma:e→xof an agent
2. Ma:x→eahwhich the agent learned becomes equal to an inverse mapping of Mh:e→xof the user.
3. Mh:x→e
h

M =

a:x→e

5



a

M

a:e→x

−1

a

M

h:x→e


=

h

−1

M

h:e→x

Conclusion

This paper describes a human-agent interaction framework in which a user and a life-like agent mutuallyacquire their emotion mappings through a mutual mind reading game. For describing emotional interactions
between a life-like agent and a human user, we deﬁned emotion states, primitive emotions, expressions,emotion transition rules and emotion/expression mappings. Then, to acquire the emotion mapping eachother, we developed a mutual mind reading game in which a user and a life-like agent try to recognize theother’s emotion state through the other’s expression.
Unfortunately the descriptions are conceptual. Thus our system should be fully implemented and weneed to verify the feasibility of our framework through various experiments with subjects. We are currently
developing a whole framework in which a life-like agent and a human user naturally interact each other.

References[1] MS agent web page. http://msdn.microsoft.com/msagent/.
[2] J. Bates. The role of emotion in believable agents. Communications of the ACM, 37(7):122–125, 1994.
[3] B. V. Dasarathy. Nearest Neighbor (NN) Norms: NN Pattern Classiﬁcation Techniques. IEEE Computer
Society Press, 1991.
[4] P. Maes. Agents that reduce work and information overload. Communications of the ACM, 37(7):30–40,
July 1994.
[5] P. Maes and A. Wexelblat. Interface agents. In Proceedings of the CHI ’96 conference companion on
Human factors in computing systems: common ground, page 369, 1996.
[6] M. Minsky. The Society of Mind. Simon & Schuster, 1986.
[7] R. W. Picard. Aﬀective computing. The MIT Press, 1997.
[8] K. Rivera, N. J. Cooke, and J. A. Bauhs. The eﬀects of emotional icons on remote communication. In
CHI’96 companion, pages 99–100, 1996.
[9] S. Sen and G. Weiss. Learning in multiagent systems. In G. Weiss, editor, Multiagent Systems - A
Modern Approach to Distributed Artiﬁcial Intelligence, chapter 6, pages 259–298. MIT Press, 1999.
[10] J. D. Velásquez. Modeling emotions and other motivations in synthetic agents. In Proceedings of the
Fourteenth National Conference on Artiﬁcial Intelligence, pages 10–15, 1997.
[11] J. D. Velásquez and P. Maes. Cathexis: A computational model of emotions. In Proceedings of the FirstInternational Conference on Autonomous Agent, pages 518–519, 1997.