CHI 2007
2006 •· Work-in-Progress

April 28-May 3, 2007 • San Jose, CA, USA

The Facial Expression Effect of an
Animated Agent on the Decisions
Taken in the Negotiation Game
Masahide Yuasa

Abstract

Tokyo Denki University

This paper investigates the manner in which decisionmaking is influenced by the impressions given by lifelike agents in negotiation situations. These impressions
comprise an agent’s facial expressions such as happyand sad, and the history of relationship with the agent.
In this paper, we introduce a negotiation game as oneof the basic interactions based on the soft game theory.
The experimental results reveal that expressions andhistory significantly influence the receiver’s impressions
and decision-making. The findings of this study can bebeneficial for designing the nonverbal expressions of ananimated agent who can negotiate and make a deal
with users.

2-1200 Muzai Gakuendai
Inzai, Chiba, JAPANyuasa@sie.dendai.ac.jp
Naoki Mukawa
Tokyo Denki University
2-1200 Muzai Gakuendai
Inzai, Chiba, JAPANmukawa@sie.dendai.ac.jp

Keywords
Life-like character, animated agent, game theory, facial
expression, appearance, negotiation

ACM Classification Keywords
Copyright is held by the author/owner(s).

H5.1. Information interfaces and presentation (e.g.,HCI): Multimedia Information Systems.

CHI 2007, April 28 – May 3, 2007, San Jose, California, USA
ACM 978-1-59593-642-4/07/0004.

2795

CHI 2007
2006 •· Work-in-Progress

April 28-May 3, 2007 • San Jose, CA, USA

Introduction
Recent years have witnessed the development ofanimated agents (occasionally known as life-likecharacters, avatars, interface agents, Embodied
Conversational Agents (ECAs), etc.) [12,13,14]. These
animated agents have been used in a wide range ofapplication areas, including entertainment, e-commerce,and online negotiations. Several researchers havedeveloped models that automatically represent theemotions and facial expressions of animated agentsand robots [9,11,16]. However, these models have notbeen used in practical applications and hence themanner in which they influence human behaviors is notclear. In order to evaluate the effect of emotions andfacial expressions, we introduce a negotiation game, atypical game involved in game theory. Some of theresearch conducted on game theory involved the use offaces [7,10]. Eckel et al. focused on the abstract facial
icons and photo images in the ultimatum game [2]. On
the other hand, Howard proposed soft game theory[5,6], in which two players take decisions byconsidering the positive or negative impressions given
to each other in a situation that is similar to the one in
Prisoner’s Dilemma. While previous research on gametheory focused solely on the face maintained in a game,the soft game theory considered only positive andnegative emotions.
Therefore, this study takes into account a variety of
facial expressions, such as happy and sad, and thehistory of agent behavior. In this paper, we introduce avariation in facial expressions and the impressions
received from opponent agents. Further, we investigate
the impressions and decisions of users who areinfluenced by facial expressions in an online negotiation
game. We presently aim to develop animated agents

who can substitute humans in an online negotiation
game. Negotiation is one of the basic interactions inwhich both verbal and nonverbal information isexchanged among humans on a daily basis [15].
Additionally, this paper investigates the manner inwhich an agent’s nonverbal information influences
users in the negotiation game based on the soft gametheory. We define a new decision relevance model inwhich impression factors such as friendliness,dominance, and trustworthiness influence decisionmaking, as opposed to only positive or negative
emotions, in the soft game theory. We believe that anagent’s appearance, history, and expression influence
the impression factors. The agent’s appearance, history,
and nonverbal expression that are best suited to gain
cooperation from partners in each case can be decidedby agent designers and developers.

Decision Relevance Model
Using the concept of soft game theory, we propose a
new decision relevance model for human behavior. As
mentioned previously, this paper focuses on an agent’s
appearance, history, and expression. Here, appearance
implies the facial features of a human being or the
opponent’s face [4]. According to several researchers,
each face of agents or robots conveys different
impressions [1,8]. History implies whether or not apartner is an acquaintance and whether he or she will
influence the subsequent negotiation [18]. Finally,expression implies nonverbal expression, including
facial expression, posture, gaze behavior, and prosody[3,17]. We believe that appearance, history, andexpression help to create impressions about theopponent; these impressions influence players’

2796

CHI 2007
2006 •· Work-in-Progress

April 28-May 3, 2007 • San Jose, CA, USAdecision-making. We use the following definitions andequations to describe these relationships.

(1) Impression vector: Ap × Hp × Ep → I–p = (i0, i1, i2,…)t(2) Reasoning function: Mp × I–p → S–p(3) Decision function:
M–p × Ip × S–p → Spand dominance). For example, when player 1 gives theimpressions of A1, H1, and E1 to player 2, he/she
receives these impressions and they influence I2.
Subsequently, he/she takes a decision S2 that is
influenced by not only M1 but also I2. Furthermore, forexample, in Prisoner’s Dilemma (table 1), when bothplayer 1 and player 2 give good impressions to eachother, and if player 1 believes that player 2 will selectthe strategy S2 = c2 and player 2, in turn, believes that
player 1 will select the strategy S1 = c1, then “c1, c2”
becomes the final scenario. These inferences are in
keeping with the concept of soft game theory in thatthey describe how the players will achieve cooperation[5,6].

table 1. Prisoner’s dilemma. The first and second numbers inExperimentMessage
Mk = {m0, m1, m2, …}
Appearance Ak = {a0, a1, a2, …}
History
Hk = {h0, h1, h2, …}Expression
Ek = {e0, e1, e2, …}
Strategy
Sk = {s0, s1, s2, …}={ck, dk}(In Prisoner’s Dilemma, Sk = {ck, dk}, i.e., cooperation
or defect.)

each cell represent the benefits of player 1 and player 2,respectively.

player 2

player 1

c2
Cooperated2
Defectc1 Cooperate

3,3

1,4

d1 Defect

4,1

2,2

The variation in parameters is indicated by a subscript(0, 1, 2, …). The subscript “k” represents player 1 orplayer 2; “p,” a player; and “-p,” the other player. In
soft game theory, players are allowed to use limited
messages (“Mk”) (for example, “I will select cooperate,
so please select cooperate”) [5]. Impression (“Ik”) is avector, and “t” denotes a transposed vector. The
impression vector is represented by the factors ofpersonality (for example, friendliness, trustworthiness,

This experiment aims to confirm whether history andexpression will influence the impressions and decisionsof players. In this experiment, we assume that it issufficient to only confirm Equations (1) and (2) because
the user who selects “cooperate” or “defect” (3) is
responsible for inferring his/her partner’s selection. We
conducted the experiment using “gas stations” in thesituation chosen for a typical negotiation game. Theplayers own stations A and B, respectively. Each ownercontinues to sell gas at the rate of 100 yen per liter,and each of them notches up sales of 20 million yen.
When both the owners sell gas at the rate of 90 yen perliter, each of them notches up sales of 10 million yen.
When one owner sells gas at 90 yen per liter, he/sheearns up to 30 million yen. Consequently, the otherowner suffers a loss of 10 million yen. The subjects ofthis study played the game with animated agentsthrough a computer network. We informed the subjects
that the animated agents were controlled by thesubjects’ opponents. However, the animated agents

2797

CHI 2007
2006 •· Work-in-Progress

April 28-May 3, 2007 • San Jose, CA, USA

were controlled by our program, which randomly
generated the facial expressions. The subjects were notallowed to communicate with the agents but could use
limited expressions and messages. The following is a
description of the procedure used in our experiment:(1) An agent appears with the message that he/she ismeeting with the subject for the first or the secondtime.
(2) The agent declares his/her promise by changing
facial expressions (Figure 1).
(3) The subject is required to evaluate his/herimpression of the agent.
(4) The subject is required to decide whether the priceof the gas should be 100 yen or 90 yen per liter.
(5) The agent selects his/her decision.
(6) The result (cooperation or defect) is displayed.
The following is the relationship between our model andthe experimentAppearance Several agents were designed bydeforming many photos that were obtained frommagazines and websites. We selected an agent whoappeared to be less dominant and less friendly because
we wanted the agent to have minimal influence on thesubject’s decision-making.
History The subjects were required to answer theagents who met them for the first or the second timeafter the agent selected “cooperate” or “defect.” In thisexperiment, the subjects believed that the agents whoappeared were the same but the players were different
in some cases. We believe that after an agent selects“cooperate,” he/she will give the impression of being
trustworthy and cooperation will occur at high rates.

Expression The agent can express himself or herself
by using BOW, AVERT, HAPPY, ANGRY, and COOL
(Figure 2). The HAPPY may induce a higher friendliness
impression and the ANGRY may induce a lower
friendliness impression and a higher dominanceimpression [17]. The BOW and AVERT may induce alower dominance impression than any other facial
expressions [3].

figure 1. The interface of the experiment.

figure 2. Facial expression and gaze direction (BOW, AVERT,HAPPY, ANGRY, and COOL).

ResultThis study involved twelve students from theDepartment of Computer Science. The experiment
involved a total of 456 cases. We analyzed the subjects’impressions. They were required to select the mostappropriate response from 20 adjective pairs, using aseven-point semantic differential scale. Factor analysis
was used to determine the structure of the data. The2798

CHI 2007
2006 •· Work-in-Progress

April 28-May 3, 2007 • San Jose, CA, USA

first dimension assessed the general measurement oftrustworthiness; the second dimension, friendliness;and the third dimension, dominance.
We verified the correlation between impression andsubjects’ selection. The factor score of trustworthinessafter an agent selected “cooperate” (AC) was higher
than the first time (F) and after he /she selected“defect” (AD). Next, the subjects selected “cooperate”
at higher rates in the case of AC (average = 53.9%) ascompared with AD (27.4%) and F (36.8%). In
particular, the cooperation occurred at the highest rate
(63.1%) in the case of HAPPY at AC. Further, we foundthat the factor scores of trustworthiness correlated
weakly with the subjects’ selections of “cooperate” (r =0.4715). Moreover, the average value of the threefactors correlated weakly with the subjects’ selections(r = 0.3967).
We verified the relationship between facial expressions
and factor scores and found that HAPPY induced ahigher friendliness impression, but ANGRY induced alower one than any other expressions. A comparison ofthe factor score of friendliness in the AC between
HAPPY and COOL and between ANGRY and COOL
revealed that these differences are statisticallysignificant at the .05 level (p-value, t-test). Moreover,
we found that ANGRY induce a higher dominanceimpression but BOW and AVERT induce a lower
dominance impression than any other expressions. The
comparison of the factor score of dominance in the AC
between these expressions and COOL, are statisticallysignificant at the .05 level.

Discussion
The results revealed that facial expressions and history
have a significant effect on the impressions related tofriendliness, trustworthiness, and dominance. We also
discovered that the impressions influenced the subjects’
decisions to select “cooperate” or “defect.” Further, theresults revealed that in the negotiation game, humansbehave partly like that suggested in our model. Thismodel can be used to plan the agent’s behaviors tomake a user cooperate during negotiation or toencourage factors such as friendliness andtrustworthiness. It is important to identify theparameters that will be efficient and significant in
helping humans design agents and robots.
In this experiment, we used a simple negotiation game.
In the future, we will use price negotiation that requires
users to exchange messages and facial expressions
with each other [18]. Although we used only one
history in our experiment, price negotiation will require
more histories and also the incorporation of the mannerin which the previously proposed moves of a player
influenced the other player. Furthermore, according toThompson et al. [15], not only positive emotional
expression but also the expression of extreme irrational
behavior can be highly effective in negotiation. Our
next aim is to develop a sophisticated model thatconsiders both rational parameters like the evaluation
value in game theory and irrational (in fact, rational)
behavior like emotional expression in negotiation.
Moreover, we will improve the model gradually,observing human behavior by a negotiation game.

Conclusion
In this paper, we proposed a new decision relevancemodel for the facial expressions of an animated agent

2799

CHI 2007
2006 •· Work-in-Progress

April 28-May 3, 2007 • San Jose, CA, USA

based on soft game theory and applied it to anegotiation game. This model can be used to design theagent’s and robot’s behaviors. In the future, in order toimprove our model, we will attempt to investigate othertypes of games.

Information and Systems Engineering 1, 2 (1995), 131148.

Acknowledgments[8] Koda, T. and Maes, P. Agents with faces: The effectof personification. In Proc. Roman ‘96, (1996).

This research was partially supported by the Ministry ofEducation, Culture, Sports, Science and Technology(Grant-in-Aid for Scientific Research (B) 16300032,2006), by the Research Center for Advanced
Technologies, Tokyo Denki University, and by ResearchInstitute for Science and Technology of Tokyo Denki
University Q06J-14/ Japan.

References[1] Baylor, A.L., Rosenberg-Kima, R.B and Plant, E.A.
Interface agents as social models: The impact ofappearance on females’ attitude toward engineering. InProc. CHI '06, (2006), 526-531.
[2] Eckel, C. and Wilson, R. Why fairness? Facial
expressions, evolutionary psychology and theemergence of fairness in simple bargaining games.
Workshop on the Workshop II conference, (1999).

[7] Kiesler, S., Sproull, L. and Waters, K. A Prisoner’sDilemma experiment on cooperation with people andhuman-like computers. Journal of Personality andSocial Psychology 70, 1 (1996), 47-65.

[9] Ortony, A., Clore, G.L. and Collins, A. The Cognitive
Structure of Emotions. Cambridge Univ. Press, 1998.
[10] Parise, S., Kiesler, S., Sproull, L. and Waters, K. My
partner is a real dog: Cooperation with social agents. In
Proc. CSCW '96, (1996), 399-408.
[11] Picard, R.W. Affective Computing. The MIT Press,1995.
[12] Poggi, I., Pelachaud, C. and Caldognetto, E.M.
Gestural mind markers in ECAs. In Proc AAMAS 2003,(2003), 1098-1099.
[13] Prendinger, H. and Ishizuka, M. Life-Like
Characters: Tools, Affective Functions, and Applications.
Springer, 2003.
[14] Rist, T., Andre, E. and Baldes, S. A flexible platform
for building applications with life-like characters. In
Proc Intelligent User Interfaces 2003, (2003), 158-165.

[3] Fukayama, A., Ohno, T., Mukawa, N., Sawaki, M.
and Hagita, N. Messages embedded in gaze of interface
agents - impression management with agent’s gaze. In
Proc CHI2002, (2002), 41-49.

[15] Thompson, L.L., Medvec, V.H., Seiden, V. andKopelman, S. Poker face, smiley face, and rant’n’ rave:
Myths and realities about emotion in negotiation.
Blackwell Handbook in Social Psychology 3 (2000).

[4] Goetz, J., Kiesler, S. and Powers, A. Matching robotappearance and behavior to tasks to improve humanrobot cooperation. In Proc. ROMAN 2003, (2003), 5560.

[16] Velasquez, J. Modeling emotions and othermotivations in synthetic agents. In Proc. AAAI 97,(1997), 10-15.

[5] Howard, N. ‘Soft’ Game Theory. Information andDecision Technologies 16, 3 (1990), 215-227.
[6] Inohara, T. and Nakano, T. Properties of ‘soft’games with mutual exchange of inducement tactics.

[17] Yuasa, M. Strategies using facial expressions andgaze behaviors for animated agents. In Proc. AMIRE
2005, (2005), 337-343.
[18] Yuasa, M., Yasumura, Y. and Nitta, K. Negotiation
support tool using emotional factors. In Proc. IFSANAFIPS2001, (2001), 2096-2911.

2800