MUTUAL ADAPTAION IN COMMUNICATIONBETWEEN A HUMAN AND A LIFE-LIKE AGENTBY SHARING CONTEXT DEPENDENT FACIAL EXPRESSIONS*Tomohiro Yamaguchi, *Yuta Ochi, **Seiji Yamada*Dept. of Information Science, Nara National College of Technology**National Institute of Informatics
ABSTRACTThis paper describes a human-agent interaction in which auser and a life-like agent mutually acquire the other's mind
mapping through a mutual mind reading game. In theseseveral years, a lot of studies have been done on a life-likeagent. Through development of various life-like agents, amind like emotion, processing load has been recognized toplay an important role in making them believable to a user.
For establishing effective and natural communication
between an agent and a user, they need to read the other's
mind from expressions and we call the mapping fromexpressions to mind states mind mapping. We propose amutual adaptive interaction between a human and a lifelike agent. In our framework, a user plays a mutual mind
reading game with an agent and they gradually learn toread the other's mind through the game. Eventually wemake experiments to investigate its effectiveness.

significantly depend on the other's mind states. For
example, a life-like agent should kindly and carefully
behave to a depressed or busy user, and intuitively
communicate its processing load to a user through a facial
expression. Though mind reading is always done among
human, it between a life-like agent and a user becomes far
more difficult. The reason is that design of agent's
expressions significantly depends on personal preference,or social culture. For example, Fig.1 shows various
expressions and corresponding minds of Microsoft agents.
We can easily identify minds from some expressions(Surprised, Congratulate for authors), however minds fromsome expressions (Confused, Decline, Process for authors)
may be hard to be identified. Consequently a life-likeagent and a user need to acquire relation between an
expression and a mind when they actually encounter. We
call such a mapping from an expression to a mind a mind
mapping. Major difference between a mind and an
expression is whether directly observable or not.

1. INTRODUCTION
Surprised Congratulations Pleased Confused Think Uncertain Decline

In these several years, a lot of studies have been done on alife-like agent like a Microsoft agent [1], an interface
agent [2]. A typical life-like agent appears on a Web
shopping page and supports a user in inputting his/herorder. Through the development of various life-like agents,
an agent's mind (i.e., computational internal states of anagent) like emotion, processing load has been recognized
to play a very important role in making them believable toa user [3]. Besides, expressions of minds are also effective
to computer-mediated communication. However, even if amind mechanism is fully implemented on a life-like agent,
there is a significant problem that to recognizing apartner’s mind state between a user and an agent isdifficult.
For establishing effective communication
between a life-like agent and a user, they need to be able
to identify the other's mind from an expression and we callthis task mind reading. Note that mind is a set of internal
states those are non-observable directly. If mind reading is
impossible, they can’t act human-like behaviors thoseFig.1 Various expressions of Microsoft agents.
To deal with the mind mapping task, we haveproposed a human-agent interaction framework in which auser and a life-like agent mutually acquire mind mappings
each other [4,5]. They play a mutual mind reading game
together and gradually learn mind mappings each other.
Instance-based learning is applied to agent's learning.
However, there is a major problem of mutual learning thatone’s learning affects the convergence of the partner’s
learning. This also occurs in human agent interaction when
they try to learn the partner’s mind mappings.
To solve this, this paper presents a mutual
adaptive interaction with context sharing between a humanand a life-like agent that considers learning ability of ahuman user. The main idea is context sharing through aninteraction between an agent and a user. When they noticeand share the common rules as the context of theinteraction, we call this context sharing that makes themutual learning easier. Major mutual learning problem isthat each learning behavior or result may affects orinterferes with other's learning since they interact eachother to learn [6]. The context is a constraint to reduce or
to avoid the mutual learning problem in the mutual
adaptive interaction. The shared context guides eachlearning properly as the constraint in the mutual learning.
In this paper, we show the experimental results
on a mutual mind reading game with the fixed context ofan agent and with a random context of an agent. In that,when a user adapts the agent that expresses the emotionthat depends on a game context, the learning task
performance of the user is improved. Then the consistencyof the user’s expression is also improved, this makes agent
learning easier. At last, their communication will go better.
2. MUTUAL ADAPTATIVE INTERACTION
2.1 Overview of the mutual adaptive interaction
The mutual adaptive interaction is an interaction that not
only deals with an agent’s adaptation for a user but also
considers the user’s adaptability to the agent. Major
difference from adaptive interaction is that whether
considering a human user’s learning behavior or not. For
the mutual adaptation, it is important that both an agent
and a user share the same context through their interaction.
The reason is that if a behavior of the agent in theinteraction depends on the context, a user can adapt easily
by recognizing the way of the context dependence. This
will make the user’s cognitive load lower. Therefore, forthe agent’s adaptation with adaptability, it is important thatthe agent follows a simple context that a user can follow
easily. In this paper, context means the tendency of mind
state transitions in n times through a mind mapping game.
2.2 Conditions those occur mutual adaptation
We assume the conditions to occur the mutual adaptation
between an agent and a user as follows:
1. An agent should behave regularly that is easy to befollowed by a user.
2. A user does not need to do a useless teach that affects
the agent’s learning.
The key points are those the context following behaviors
may reduce the user’s cognitive load, and they make thepartner’s behavior learning easier each other.
2.3 Task: the mutual mind reading game
This section describes a mutual mind reading game inwhich a user and a life-like agent try to recognize theother's mind state through the other's (facial) expression as
the learning task of a mind mapping. They mutuallyanswer their results on the other's mind, and one having acorrect answer gets a score. This game is designed toacquire mind mappings each other so that a user may
enjoy it, and as results, the user's cognitive load is reduced.
A game in which a player estimates the other's
mind state through the facial expression to compete for theaccuracy is called a mutual mind reading game. ”Mutual”
means that both a game in which an agent estimates auser's mind from his/her expression and a game in which auser estimates an agent's mind from its expression are
performed in turn.
A primary objective of a game is to learn mind
reading between an agent and a user. Therefore, both anagent and a user play a game with fixed mind mappings
each other. For a user, a main objective of the game is tolearn mind mappings of an agent, and a sub objective is tosee to follow the facial expression strategy of the agent asa context of their interaction. A facial expression strategy
is the behavior patterns to decide which emotion to be
expressed according to each situation. For an agent, aprimal objective of a mutual mind reading game is tocollect instances for instance-based learning of an agent
both efficiently and broadly. An instance is a pair of an
estimated primitive mind (i.e. an emotion symbol) and aobserved facial expression. In this paper, we assume thatthe agent follow the fixed facial expression strategy
according to the game context that is natural for a humanuser. An example of expression strategy is that an agent
expresses pleased emotion when it’s guess is correct.
Procedures of a mutual mind reading game are
given in the following. Note that an agent tells its correct
mind to a user when his/her answer is incorrect, but a user
does not do so to reduce his/her cognitive load.
1. An expression of an agent is displayed to a user throughGUI.
2. A user guesses agent's mind from seeing the expression,and tells the mind to an agent by clicking a button.
3. An agent replies “Yes” (the guess is correct) or “No”
(the guess is incorrect) with the correct mind as judgment
against the other's guess.
4. An agent sees an expression of a user by a CCD camera.
5. An agent guesses user's mind from the captured
expression, and shows the mind to a user through GUI.
6. A user replies “Yes” (the guess is correct) or “No” (theguess is incorrect) as judgment against the other's guess.
7. The above procedures are repeated until a finish
condition of mutual learning is satisfied.
Then we describe a finish condition of the game.
When each two players guess the partner's each primitivemind for n times of continuation, the game is finished. The
frequency of the correct guess of n can be changedaccording to a degree of difficulty of the game. In thispaper, the parameter n is set to 2.
2.4 User Interface
Our system is implemented on VineLinux2.1, by C/C++
programming language and GTK+ for building GUI. AlsoVideo4Linux API was employed for image capture
programming. An experimental environment is shown in
Fig.2.

seeing the progress bars. A game finishes when both of
two progress bars reaches to the right edges.
Fig.4 shows interface where an agent guesses
user's mind. When a user clicks a “Start agent's
recognition” button, an agent begins to capture user's
images. After a stable expression is captured, the four
images are shown in the window. Also stored instances areindicated with labels and the distance d between them anda captured image. Using the most similar instance, anagent guesses a user's mind and tells it to a user like Fig.4.
A user answers to it by clicking `”Yes” or “No” buttons.

Fig.2 Environment of human-agent interaction.
Fig.4 An agent guesses a user's mind.
Fig.3 shows a snapshot of GUI when a user
guesses agent's mind. When a user clicks the “Start”
button, an agent shows its expression. Then a user guesses
agent's mind, and clicks one of “Primitive Mind” buttons.
If a user clicks the button, an agent tells the judgment withthe correct mind like a message in Fig.3. Also two
progress bars are shown for indicating average successrates on the finish condition of a user and an agent. A user
can understand the degrees of learning progresses by

3. EXPERIMENTS
To examine the conditions to occur the mutual adaptation
between an agent and a user those are described in section
2.2, we made the comparative experiments. Playing themutual mind reading game between a user and the agent,
we examine two kinds of facial expression strategies of theagent, a random expression strategy (i.e., no context
dependent strategy) and the game context dependent
strategy. We compare the finish time of the game in thatboth a user and the agent acquire correct their mind
mappings.
3.1 The Hypotheses
The hypotheses those we verify are following;
Comparing the agent that expresses its emotion by thegame context dependent strategy with the agent by arandom strategy,
1. A user’s learning of its mind mapping is faster.
2. A user is easier to see or follow some regularity on theagent expression strategy.
3. A user can guess more correct emotions of the agent.

Fig.3 Human guesses agent's mind.

Examining these hypotheses will support that it improves auser’s learning that the agent follows the game context
dependent strategy.
3.2 Experimental Methods
Through all the experiments, we employed sixteen
subjects of the students majoring Information Science at
Nara National College of Technology. We divide thesesubjects into two groups as following;Group1: Each user plays with the agent of the random
expression strategy.
Group2: Each user plays with the agent of the gamecontext dependent strategy.
We used four primitive minds and primitive
expressions for an agent as shown in Fig.5 and threeprimitive minds “Ordinary”, ”Thinking”, “Decline” for auser. Fig.6 shows the game context dependent strategy of
the agent. This is one of the typical reactions in a processof the mind reading game. After a user guesses the agent’s
emotion, the agent expresses its emotion according to thecontext of the result of the user’s guess. If the user fails toguess the agent’s emotion, the agent expresses “Confused”.
If the user’s guess is correct, the agent expresses “Pleased”,
and when the user succeeds to guess it again, the agent
expresses “Surprised”. Besides, if the user succeeds toguess it again and again, the agent expresses “Thinking”.
Note that each parameter in Fig. 6 is tuned so that eachpossibility to express an emotion becomes to be the same.
On the other hand, on the random strategy, the agent
expresses one of its four emotions randomly that is
independent of the result of a user’s guess.
After the experiments, each subject is requested
to answer the questionnaire as follows;
1. Can you follow the expression strategy of the agent?
2. Do you feel that the game takes long time?
3. Do you think that the agent’s guess is correct?
4. Free comments.
Confused

Pleased

Surprised

Think

Fig.5 Four expressions used for experiments.

Fig.6 The game context dependent strategy of the agent.

3.3 Experimental Results
The section shows the experimental results and the resultsof the questionnaires. First, the averaged number of user’s
learning times to finish the game is compared. To compare
the game context dependent strategy group (Group 2) withthan the random strategy group (Group 1), Group1 takes
26 times and the Group2 takes only 19 times. Then theGroup 2 plays the game shorter than Group1. Though thestatistical test, the hypothesis 1 of section 3.1 is verified by
p=0.1 of significant level.
Second, the hypothesis 2 of section 3.1 is verified
by p=0.1 of significant level from the result ofquestionnaire. In Group 2, six out of eight subjects find theregularity of the agent’s strategy. Note that in Group 1,
two of eight subjects answered that they found theregularity of the agent’s though there is no regularity.
Third, the number of failures of a user’s guess iscompared. In Group 1, six out of eight subjects failed toguess the agent emotion at least once. In Group 2, no one
failed to guess the agent’s emotion. It suggests that the
hypothesis 3 of section 3.1 is verified.
Therefore, the game context dependent strategy
group play the mind reading game better than the random
strategy group. Note that there is no significant difference
from the result of the agent learning times.
4. CONCLUSIONSWe presented a mutual adaptive interaction with context
sharing between a human and a life-like agent thatconsiders learning ability of a human user. With themutual mind reading game, we made comparative
experiments. Then we showed that the game context
dependent strategy group plays the mind reading game
better than the random strategy group. Our future work is
to implement the agent that identifies a user’s expression
strategy to learn as its expression strategy.
5. REFERENCES[1] Ms agent Web page. http://msdn.Microsoft.com/
msagent/[2] P.Maes, “Agents that reduce work and information
overload”, Communications of the ACM 37(7), pp.393396, 1994
[3] J.Bates, “The role of emotion in believable agents”,
Communications of the ACM 37(7), pp.122-125, 1994
[4] S. Yamada, T. Yamaguchi, “Acquiring emotion
mappings through the interaction between a user and alife-like agent, 9th International Conference on
Human-Computer Interaction (HCI2001), M. Smith. etal (Eds), Usability Evaluation and Interface Design,
Vol.1, pp.993-997, 2001

[5] S. Yamada, T. Yamaguchi, “Mutual Learning of Mind
Reading between a Human and a Life-like Agent”, The
5th Pacific Rim International Workshop on MultiAgents (PRIMA-2002), to appear, Tokyo, Japan,
August 18 - 19, 2002
[6] S.Sen, G.Weiss, "Learning in Multiagent Systems", inMultiagent Systems - A Modern Approach toDistributed Artificial Intelligence, Chapter 6, Ed. by
G.Weiss, pp.259-298, 1999