Journal of Robotics, Networking and Artificial Life, Vol. 3, No. 2 (September 2016), 74-78

Affective Human Computer Interaction
Kaoru Sumi

Future University Hakodate,Hakodate, Hokkaido, Japan
E-mail: kaoru.sumi@acm.org

Abstract
This paper introduces a study of spoken dialogue agent systems using emotional expressions as affective human
computer interaction. The paper describes an experiment investigating the effect that the expression and words ofthe agent have on people, introduces a spoken agent for customer services using expressive facial expressions and aspoken agent for mental care using expressive facial expressions and positive psychology as application systems for
affective human computer interaction, and presents a discussion and a conclusion.
Keywords: Affective computing, facial expression, human computer interaction

1. Introduction
According to Media Equation [1], people treat
computers, television, and new media as real people andplaces, thereby making the users uncomfortable if anagent behaves in a disagreeable manner. In the field ofpersuasion technology research [2] it is said that if auser recognizes the presence of something in acomputer, he or she will respond to it according to thenormal social rules. However, there are still many things
that we do not know about how an agent’s response
affects a user during their interaction.
In the development of intelligent systems, it is important
to consider how best a feeling of affinity with thesystem and show the presence of the system that has
human-like
intelligent
functionssuchasrecommendation or persuasion. Therefore, evaluating
the interpersonal impressions conveyed by agents isvery important.

Our research group has developed intelligent dialoguesystem that proactively interacts with a user accordingto the user’s circumstances. As a first step, we examined
how to react with the user under several emotionalsituations through the experiment.
We performed an experiment to evaluate how the facial
expressions of an agent and the words used by the agent
affected users during agent-user interaction.
In the past, ELIZA [3] was developed to imitate aRogerian psychotherapist, and was an early example ofprimitive natural language processing. SimCoach [4] isa spoken dialogue agent system used in the mental careof returned soldiers. It displays a speaking counseloragent and analyses the psychology of the user using aquestion and answer technique.
The paper describes an experiment investigating theeffect that the expression and words of the agent haveon people, introduces a spoken agent for customer
services using expressive facial expressions and a

Published by Atlantis Press
Copyright: the authors
74

Kaoru Sumi

spoken agent for mental care using expressive facial
expressions and positive psychology[5][6] asapplication systems for affective human computerinteractions, presents a discussion and a conclusion.
2. Experiment investigating impressions andbehavior change caused by replies from theagent
We conducted an experiment to examine how theimpression that the user gets from the agent’s answer isaffected by the combination of facial and word
expressions. It was intended to clarify the impressionthat the agent gave the user by answering wheninteracting with the user in an emotion-arousing
scenario.
We chose six kinds of feelings. From the total of 216
combinations, covering multiple feelings that the user
felt (6 patterns) and the facial expressions for theagent’s interaction with the user (6 patterns) and word
expressions used by the agent (6 patterns), we selected
96 patterns in this experiment. These covered 16
patterns in each feeling: empathetic words andconsistent facial expressions, nonempathetic words andconsistent facial expressions, word consistent and facialinconsistent, and word inconsistent and facial consistent.
This is because the conditions of nonempathetic andboth inconsistent word and facial expressions arenonsensical in normal communication. This is thecondition where the word and facial expressions areinconsistent, which is the condition for double bindcommunication, but it can be considered as either wordor facial being empathetic to the user. The case ofnonempathetic condition and inconsistent word andfacial expressions can be considered as pathological.
A total of 1236 people, 568 male and 668 female (AV.
38.0, SD 11.5), were assigned 96 contents. More thanten users were assigned to each content [7].
As a result, people are easy to be persuaded when theagent was favorable impression. This result is very
natural; however, the combination of favorable wasdifferent from our prediction. First, we predicted thatwords and facial expressions reflected on theemotions aroused by the scenario would lead to themost favorable impression, so we set these data as thecontrol group. In fact, there were more favorable

impressions than those obtained for the control group.
For example, the words and facial expressions were“joy” when the user’s emotion was “joy” for the control
group. It is very interesting that when the user’s emotion
was “joy”, the agent’s words for “joy” with facial
expressions of “surprise”, “sadness”, or “fright” weremost favorable. On the other hand, when the user’semotion was “fright”, the agent’s words for “fright”
with facial expressions of “disgust” or “sadness” werethe most favorable.
These facial expressions were recognized as theemotion conveyed by the words and were more
empathetic and somewhat meaningful emotions. For
example, when the user’s emotion was “joy”, theagent’s words of “joy” with facial expressions of“surprise” or “fright” might have been recognized as theagent being exaggeratedly surprised at the “joy”
scenario. When the user’s emotion was “joy”, theagent’s words of “joy ” with facial expressions of“sadness” might have been recognized as the agent
being highly pleased from the heart at the “joy” scenario.
When the user’s emotion was “fright”, the agent’s
words of “fright” with facial expressions of “sadness”
might have been recognized as the agent grieving
deeply at the user’s “fright” scenario. When the user’s
emotion was “fright”, the agent’s words of “fright” withfacial expressions of “disgust” might have beenrecognized as the agent feeling deep hate at the user’s“fright” scenario.
Through these observations, we concluded that there isa rule for facial expressions: in a certain scenario,
synchronizing foreseen emotion of the user caused by
the situation will make a favorable impression. For
example, when the user has the emotion of “ joy”,
he/she wants someone to be surprised or highly pleased.
Then, showing surprised or highly pleased face
expression make the user feels favorable impression.
When the user has the emotion of “fright”, he/she wants
someone to grieve deeply or disgust. Then, showing
grieved or disgust face expression make the user feels
favorable impression.
Users want the agent to ooze synchronized theirforeseen emotion by hearing the news instead of simply
showing synchronized reaction according to emotion atpresent time.

Published by Atlantis Press
Copyright: the authors75

Affective Human Computer Interaction

3. A Spoken Agent
Customer Services

SystemforLearning

As an application system for affective human computerinteraction, this first spoken agent system consists of a
3D spoken agent display system, a speech recognition
system, a speech synthesis system, a dialogue controlsystem and a facial expression recognition system. Thissystem is for providing educational training in
hospitality through dialogue with a spoken agent.
This system focuses on the Japanese style of servicemindedness, which is typified by paying attention toindividual customers [8][9].
The 3D spoken agent display system displays a 3D
spoken agent, which can show the facial expressions of“"smile", "laugh", "anger", "sadness", "disgust", "fright",
and "surprise”. The system superimposes the mouth
shapes of the vowels onto each facial expression, lipsynching with the sounds. Fluid movement of the facial
expressions and lip-synching is made possible using
Microsoft XNA morphing technology. Our system uses
Google speech API as the speech recognition system,
and AITalk as the Japanese speech synthesis system. As
a dialogue control system, we revised Artificial
Intelligence Markup Language (AIML [10]) which isbased on Extensible Markup Language (XML) to use
the Japanese language. Using templates, we can express
a dialogue freely.
<pattern>* tell me your name *</pattern><template>My name is Ayaka</template>
For example, if there is a question “Tell me your name?”from a user, then the system answers, “My name isAyaka,” in this case. The important pattern here is “*tellme your name*”, so the system can answer the sentence“Please tell me your name” or “Could you tell me yourname?” because these sentences include the phrase “tellme your name.” When we analyze Japanese language,
we have to add spaces between words because there areno spaces in Japanese sentences. We use MeCab, which
is a fast and customizable Japanese morphologicalanalyzer, to add spaces between words. As the facial
expression recognition system, this system uses thefacial recognition application of brain wave measuring
equipment called “Emotive EPOC”. It recognizes theintensity of facial expressions digitally.

Using our system, a user can talk to the spoken agent,and the spoken agent teaches the user how to interact
with customers. The spoken agent is displayed on thescreen and sometimes the system displays lines in ascene which the user should practice, along with anappropriate facial expression. The system judges
whether the user has spoken the lines appropriately bycomparing them with the speech recognition systemusing AIML templates. The system judges whether theuser’s facial expression is appropriate or not bycomparing it with the facial expression recognition
system.
4. A Spoken Agent System for Mental Care using
Expressive Facial Expressions and PositivePsychology
We developed a spoken agent system for mental care of
hikikomori (a Japanese term to refer to the phenomenonof reclusive adolescents or young adults who withdraw
from social life) persons improving previous system(Figure 1). An increasing number of people are in needof mental care, because associated social issues include
problems of bullying in schools, mental disability andhikikomori or social withdrawal of young people,
suicide of young people, people being pressured to quit
work in middle age, and PTSD (posttraumatic stressdisorder) of the victim. According to the result of theexperiment of first spoken agent system, it was revealed
we must improve interface of the system, especially for

Figure 1. A spoken agent system for mental care

Published by Atlantis Press
Copyright: the authors
76

Kaoru Sumi

the attachment of Emotiv. We improved a spoken agentsystem. The system consists of a 3D spoken agent
display system, a dialogue control system, Japaneseword spacing system, a speech recognition system, aspeech synthesis system, and a facial/gesture expression
recognition system. The improved points are as follows.
1) KINECT sensor recognizes a user’s facial
expressions and gestures, 2) An agent displays gestures ,3) The facial expression/gestures can be controlled inintensity digitally, 4) An agent can be controlled its size,
direction and position, 5) A background of the system ischangeable , and 6) An agent blinks its eyes.
Interactive talks using gestures are enabled byrecognizing a user’s motion from the skeleton using aKINECT sensor. In addition, a simple facial expression
is enabled by recognition of a user’s face using
KINECT sensor without an Emotiv. The system can beswitched between a KINECT and/or an Emotiv. An
agent displays gestures of “nodding”, “nodding no”,
“waving her hand”, “bowing her head” and “bowing herhead deeply”. An agent can be controlled its size (fullbody/ upper body/ face or parentage of full body ) ,direction and position. The background of the system
can be changed using the dialogue editor. The agent
blinks its eyes for making the agent more human.
Because, according to the experiment, there are many
comments that the agent should blink its eyes.
In addition, this system offers the function of answering
positively when a user speaks negatively, for creating apositive mood. This system answers with positive words
to a user’s negative words using Negapo Dictionary [11],which is a dictionary for translate negative words topositive words. Such counseling is provided in a field ofpositive psychology. For example, such as follows istalked.
The agent: How was your day? The user: I went to myuniversity.
The agent: What kind of thing happened?
The user: I get frustrated because only a friend might begood.
The agent: I felt sorry for you. (Facial expression is“Sad”) I think you should be able to
make the feeling of frustrated with a motivation.
(Positive words from Negapo Dic.)

5. Discussion
Even if a partner is not a human being, we arepersuaded by an agent with superficial information suchas facial expressions or words. Media Equation [1] andPersuasive Technology [2] researches reveal the same.
From the result of the first experiment in section 2,
facial expressions and words are very important because
people are easy to be persuaded when the agent wasfavorable impression. It is thought that an agent canmake in-depth contact with the user by foreseeing theuser’s emotion and empathizing with the user.
Using the results, we have developed spoken agent
systems for learning customer services and mental careusing expressive facial expressions. These are working
well as the system for an agent as an instructor or a
guidance counselor using affective human interfaces.
According to the interviews, users felt the brain wave
measuring equipment and the speech recognition andsynthesis system were novelties, and the system’s
specialty was that its use was facilitated throughdialogues and it was highly motivating. The brain wave
measuring equipment and the speech recognition andsynthesis system used in our system were highly
motivating; however the subjects were divided overtheir use, and some felt them to be awkward to use.
Some of them made comments on the speech
recognition system such as “It was difficult to recognizewords”, “It was difficult”, etc. Consideration should begiven to improving the system’s ease of use by
developing or using a speech recognition system ofgreater accuracy. Also, regarding the brain wave
measuring equipment, some made comments on it suchas “It was hard to wear” and “It was painful”, so it could
be considered that it was not easy to accept.
Consideration should be given to improving the humaninterface, including using image processing and theheadset concurrently.
According to the interviews we must improve theinterface of the system, especially for the attachment of
the brain wave measuring equipment. Then wedeveloped next version of the system for mental care of

Published by Atlantis Press
Copyright: the authors77

Affective Human Computer Interaction

hikikomori social withdrawal persons or using
KINECT,
which
recognizes
thefacialexpression/gestures of the user.
I think this system is helpful for mental care of
hikikomori social withdrawal persons if it can be usedeasily in house. Because even hikikomori or social
withdrawal persons are at home and stay in front of acomputer most of the time. I think that supporting thesepeople by a personal computer is a key to solution.
Study of a spoken dialogue agent system needs to beinvestigated further including emotions, technology anddesign domains. Our research group is going to advanceour own studies of a spoken agent system using
superficial information in the future.
6. Conclusion
This paper introduces an experiment and the spoken
agent system developed according to the results. We
evaluated the first version of the system and revised.
Even if a partner is not a human being, we are perceived
by an agent with superficial information such as facial
expressions or words.
Acknowledgements
This work was supported in part by JSPS KAKENHI
Grant-in-Aid for Scientific Research on Innovative
Areas Numbers 22118503.

References

1. Reeves, Byron and Clifford Nass The Media Equation:
How People Treat Computers, Television, and New
Media Like Real People and Places, Cambridge
University Press, 1996.
2. B.J.Fogg: Persuasive Technology –Using Computers toChange What We Think and Do-, Elsevier, 2003.
3. Weizenbaum, Joseph: "ELIZA—A Computer Program
For the Study of Natural Language Communication
Between Man And Machine", Communications of theACM 9 (1): 36–45, (January 1966).
4. A. Rizzo, K. Sagae , E. Forbell, J. Kim, B. Lange, J. G.
Buckwalter, J. Williams, T. D. Parsons, P. Kenny, DavidTraum, J. Difede, B. O. Rothbaum: SimCoach: an
intelligent virtual human system for providing
healthcare
informationandsupport,
The
Interservice/Industry Training, Simulation & Education
Conference (I/ITSEC), 2011.
5. Christopher Peterson : A Primer in Positive Psychology,
Oxford University Press;
6. Martin E. P. Seligman: Using the New Positive
Psychology to Realize Your Potential for Lasting
Fulfillment, Atria Books (2004)
7. Kaoru Sumi and Mizue Nagata: Evaluating a Virtual
Agent as Persuasive Technology, Psychology ofPersuasion, Janos Csapó and Andor Magyar eds.,
Nova Science Publishers.2010.
8. Kaoru Sumi, Ryuji Ebata: A Character Agent System for
Promoting Service-Minded Communication, Intelligent
Virtual Agents, Lecture Notes in Computer Science,
LNAI8108, pp.438, Springer (2013.8).
9. Kaoru Sumi, Ryuji Ebata: Human Agent Interaction for
Learning Service-Minded Communication, iHAI2013,
1st international conference on Human-AgentInteraction, (2013.8).
10. AIML: http://www.alicebot.org/documentation/
11. Negapo Dictionary, Syufuno-Tomo-Sya (2012). InJapanese.

Published by Atlantis Press
Copyright: the authors
78