Strategies using Facial Expressions and Gaze
Behaviors for Animated Agents
Masahide Yuasa
Tokyo Denki University
2-1200 Muzai Gakuendai, Inzai, Chiba, 270-1382, Japanyuasa@sie.dendai.ac.jp

Abstract. This paper presents two strategies for selecting volitional facial
expressions and gaze behaviors of animated agents, in the case of Online Negotiation and Soft Game Theory. It is difficult to make general models selectvolitional facial expressions for their use in many kinds of applications. In thispaper, in order to develop these strategies, I investigated the effects of facial
expressions and gaze behaviors through experiments based on Online Negotiation and Game Theory. I describe strategies by using estimations of reserve
prices based on facial expressions in the case of Online Negotiation. Additionally, I deal with Soft Game Theory, which considers players’ emotions, anddescribe strategies for using facial expressions. We can apply these strategies
to animated agents and robots as a mechanism to select facial expressions andgaze behaviors.

1 Introduction
In recent years, animated agents have been used in a wide range of application
areas, including entertainment, virtual environments, and e-commerce. If we
do not wish to reveal our personal identity, we can use an animated agent
who represents us to an opponent on a computer network. By using a toolthat controls the animated agent, we can not only use verbal language but
also select facial expressions, body posture, gaze behaviors, and so on. These
nonverbal signals enrich communication among participants. Additionally, atool that automatically selects the facial expression reduces the participants’burden of selection. Some researchers have developed models that can beused for producing the facial expressions of an animated agent automatically,such as the HMM model [1], the OCC model [2], and the Cathexis model[3]. However, these models have not been adapted to select volitional facial

2

Masahide Yuasa

Fig. 1. Online Negotiation Processexpressions. A model should be able to distinguish between emotional andvolitional facial expressions because both are used by humans [4].
However, it is difficult to make general models select volitional facial expressions for their use in many kinds of applications because humans use
situation-specific strategies and intentions when selecting volitional facial expressions. If a user is provided with information on an opponent for eachsituation, the former could select a suitable volitional facial expression thatgives him/her an advantage over the opponent.
In this paper, I propose strategies for selecting volitional facial expressions
and gaze behaviors of animated agents through a computer network, in thecase of Online Negotiation and Soft Game Theory. Online Negotiation is akind of e-commerce in which negotiations are made with an opponent througha computer network. I describe strategies for using facial expressions in thecase of Online Negotiation. Additionally, I deal with Soft Game Theory, which
considers players’ feelings, and describe strategies for using facial expressions.
In the future, we can apply these strategies to animated agents and robots asa mechanism to select facial expressions and gaze behaviors.
In order to develop these strategies, I investigated the effects of facial
expressions and gaze behaviors through experiments based on Online Negotiation and Game Theory.
In Section 2, I explain the method of selecting facial expressions based onOnline Negotiation. In Section 3, I present the method of selection based on
Soft Game Theory. This is followed by some concluding remarks in Section 4.

2 Method of Selecting a Volitional Facial Expression inPrice Negotiation
2.1 Online Negotiation
Figure 1 shows an example of the Online Negotiation process. This example
includes two participants – a buyer and a seller. They take turns sendingStrategies using Facial Expressions and Gaze Behaviors for Agents

3Fig. 2. A Panel of the Online Negotiation Tool Using Facial Expressions

proposals to each other. X0, X2, X4... represent the proposals made by theseller, and X1, X3, X5... denote the proposals made by the buyer. They reachan agreement when they make the same proposal. The seller and the buyer
have reserve prices in negotiation. They can reach an agreement in the area
between the seller’s and the buyer’s reserve prices. It is important for each toestimate the opponent’s reserve price.
2.2 Effect of Changing an Estimation of the Reserve Price
The effect of changing estimation is that the receiver may change his/her
estimation of an opponent’s reserve price based on the opponent’s facial expression. For example, when the buyer’s facial expression is ANGRY, the seller
may guess that the buyer does not want to compromise further because theseller’s proposal is close to the reserve price; therefore, the buyer expresses
himself/herself with the use of an ANGRY facial expression. The seller thenestimates that the buyer’s reserve price is raised and makes the next proposal
based on this estimation.
I conducted an experiment to observe the relationship between estimationand facial expression. For this purpose, I developed a user interface to negotiate with an opponent through a computer network (Figure 2). Throughthis interface, a participant can send not only a proposal but also a facialexpression to an opponent. The facial expressions used by the participant are
displayed on the opponent’s screen.
On receiving a proposal and facial expression from the opponent, the participant can send a new proposal and facial expression. The participant canselect any one of the facial expressions of the animated agent (COOL, ANGRY, SAD, HAPPY, or SURPRISED) [5]. I used MFACE [6] to produce thefacial expressions.
I obtained probabilities of participants’ estimations by observing their decisions during price negotiation. I assume that they estimate the reserve price on
the basis of the last four proposals as follows: D = |Xn −Xn−2 |−|Xn−1 −Xn−3 |(n indicates the number of proposals made while negotiating).
Table 1 shows the results of this experiment. Eight students from thedepartment of computer science in the Tokyo Institute of Technology were4

Masahide Yuasa
Table 1. An Estimation of the Reserve Price (In the Case of Buyers)
Buyer’s Estimation ofFacial
D Seller’s Reserve Price
Expression
Up 0Down
HAPPY D ≥ 0 0% 30%
70%
D < 0 8% 46%
46%
ANGRY D ≥ 0 21% 50%
29%
D < 0 62% 13%25%
SAD
D ≥ 0 15% 33%
52%
D < 0 8% 77%
15%
....
....
... ...
...

the subjects of this study. In the case of a HAPPY expression, when thecompromise made by the seller is higher than the reserve price (D ≥ 0), theestimation is lowered (70%). In the case of an ANGRY facial expression, when
the compromise made by the seller is lower than the reserve price (D < 0),
the estimation is raised (62%).
Based on the effects of facial expressions, I describe a method of selecting
a suitable facial expression in order to change the opponent’s decision. Inthe case of using the effect of changing the estimation, the following is themain method to select a suitable facial expression. If the seller wishes tohave an advantage, the seller should use an ANGRY expression. The buyer
would then increase the estimation, the point of agreement would be raised,
the buyer would compromise by a large margin, and the seller could have anadvantage. If the seller uses a HAPPY expression, the buyer would decrease
the estimation, the point of agreement would be lowered, and the buyer would
compromise by a small margin, and the seller would have a disadvantage.
In this manner, the participants can estimate the effects of facial expressions and accordingly make an appropriate selection of their facial expression.
If we consider many cases and tendencies of probabilities during negotiation indetail and for each individual, we could estimate the reserve price and the opponent’s next move. I have attempted to develop strategies for more complex
negotiations in which the effect of facial expressions may be changed.

3 Method of Selecting a Volitional Facial Expression inGame Theory
3.1 Soft Game Theory
Soft Game Theory deals with the decisions of two players who exhibit feelings
toward each other in a situation similar to Prisoners’ Dilemma [7]. Traditionalmodels based on Game Theory do not consider these feelings among players.

Strategies using Facial Expressions and Gaze Behaviors for Agents

5However, practically, players may have certain feelings toward their opponents; for example, player A likes player B or player B does not like player C.
Additionally, the relationships between players may influence their decisions.
For example, if a player likes an opponent, he/she may select a better option
for the opponent. Soft Game Theory proposes that the player may estimate
the opponent’s choice by using the feelings exhibited by the opponent.
The models of Soft Game Theory assume that players have just one opportunity to exchange messages with each other. The player must estimate
the opponent’s choice and select his/her own option solely on the basis of theexchanged messages. For example, in Prisoners’ Dilemma (Table 2), prisoner
A can send the following message to prisoner B: “I will select Cooperate, soplease select Cooperate.” After sending the message to their opponents, theplayers must select that option. Traditional models based on Game Theory
do not consider the exchange of messages between players.
Nevertheless, after sending the message, the player may be tempted todeceive the opponent. For example, the player could select “Defect” despite
informing the opponent that he/she would select “Cooperate.” Soft Game
Theory assumes that this temptation is resolved by the players’ feeling toward
the opponent. Although there is a temptation to deceive the opponent, if
the player has a positive feeling toward the opponent, the player could trust
him/her.
3.2 The Effect of Changing a Choice in a Dilemma
Although the Soft Game Theory deals well with the feelings of players, it does
not describe the generation of these feelings in the players. Instead, it assumes
that players already have feelings. It is not unnatural for the player to haveunmotivated feelings.
Therefore, in this paper, I assume that the player’s feelings toward theopponent are caused by the opponent’s nonverbal behavior (for example, facialexpressions and gaze behaviors) with the messages that are exchanged in theSoft Game Theory. Additionally, I assume that the player’s trust is affected by
the nonverbal behavior. For example, Prisoner A says the following to Prisoner
B along with a HAPPY expression: “I will select Cooperate, so please selectCooperate.” The HAPPY expression makes Prisoner B trust the message sentby Prison A and select “Cooperate,” although the former may notice that theTable 2. A Game of Prisoners’ Dilemma. The first number in each cell represents
Prisoner A’s benefit and the second number represents that of Prisoner B
Prisoner B
Cooperate DefectPrisoner A Cooperate
3,3
1,4Defect
4,12,2

6

Masahide Yuasa

Fig. 3. CG animations: BOW, LOOK AWAY, HAPPY, ANGRY, SAD, COOL
(This is only a sample. I used other animations in this experiment)
Table 3. Probabilities of Selecting Cooperate and Defect
BOW LOOK ANGRY HAPPYAWAY
Cooperate 36.1% 29.0% 30.0% 30.0%Defect 63.9% 71.0% 70.0% 70.0%

latter is tempted to deceive him/her by selecting “Defect.” On the other hand,
if Prisoner A sends the same message to Prisoner B but avoids making eye
contact, the latter will not trust the message and he/she will select “Defect.”
I conducted an experiment to observe the relationship between selection
and CG animation. I developed a new tool to send facial expressions andgaze behaviors through a computer network by using a tool for animatedagent (TAA) [5]. In this tool, players must select “Cooperate” or “Defect”
after they receive the message with CG animations as nonverbal information.
There are six patterns of CG animations (Figure 3).
Eight students from the department of computer science in the Tokyo
Denki University were the subjects of this study. As their opponent, I used aprogram to automatically and randomly select “Cooperate” or “Defect” andsend messages to them. The subjects did not know who their opponent was.
After receiving these messages with CG animations, they were required toselect “Cooperate” or “Defect.” Table 3 shows the results of this experiment.
This table indicates the probabilities of the messages – sent with animation– from the opponent to the subject stating “I will select Cooperate, so please
select Cooperate.” I received 204 messages from the eight subjects during thisexperiment. The cases involving the use of the SAD and COOL facial expressions are not shown in this table due to insufficient usage. Before performing
this experiment, I observed that four subjects selected “Cooperate” and theremaining four selected “Defect” without CG animation through a computernetwork.
This table shows that the subjects selected “Cooperate” a greater numberof times in response to the use of the BOW expression (36.1%) than in thecase of other CG animations. I infer that the subjects trusted the message to agreater extent because of the use of the BOW, and thus, selected “Cooperate.”
On the other hand, the use of the HAPPY expression may not have resulted
in the same effect, and some subjects agreed to this possibility. Additionally,

Strategies using Facial Expressions and Gaze Behaviors for Agents

7

it appeared that the subjects were unable to understand the opponent’s intention when the ANGRY and LOOK AWAY expressions were used. I willattempt to perform the experiment with other subjects and obtain the same
results by using another game in Game Theory.

4 Conclusion
In this paper, I proposed two strategies for selecting volitional facial expressions and gaze behaviors of an animated agent based on Online Negotiationand Soft Game Theory. Although these strategies have limited application,i.e., negotiating and playing the game, they will be used for producing facial
expressions and gaze behaviors of animated agents and robots in the future.

Acknowledgments. I would like to thank Katsumi Nitta, Yoshiaki Yasumura, Daisuke Katagami, and other members of Nitta laboratory in theTokyo Institute of Technology, for advising me with this work.

References
1. Picard RW (1995) Affective Computing. The MIT Press
2. Ortony A, Clore GL, Collins A (1988) The Cognitive Structure of Emotions.
Cambridge University Press
3. Velasquez J (1997) Modeling Emotions and Other Motivations in Synthetic
Agents. Proceedings of AAAI 97:10–15
4. Ekman P, Friesen WV (1975) Unmasking the Face. Prentice Hall Trade
5. Yuasa M, Yasumura Y, Nitta K (2003) A Tool for Animated Agents in NetworkBased Negotiation. Proceedings of RO-MAN 2003 Conference:259–264
6. Hasegawa O, Sakaue K (1997) CG Tool for Constructing Anthropomorphic Interface Agents. Proceedings of IJCAI-97 WS (W5), ANIMATED INTERFACEAGENTS:23–26
7. Howard N (1990) ‘Soft’ Game Theory. Information and Decision Technologies
16(3)