CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

Emoticons Convey Emotions without
Cognition of Faces: An fMRI Study
Masahide Yuasa

Abstract

Tokyo Denki University

In this paper, we describe the brain activities that areassociated with emoticons by using functional MRI (fMRI).
In communication over a computer network, we use
abstract faces such as computer graphics (CG) avatars
and emoticons. These faces convey users’ emotions andenrich their communications. In particular, when we seesome abstract faces, we feel that they are more vivid andlively than photorealistic faces. However, the manner inwhich these faces influence the mental process is as yet
unknown. In this research, we conducted an experimentby using fMRI for the most abstract faces―emoticons.
The experimental results show that emoticons convey
emotions without the cognition of faces. This result isvery important in order to promote an understanding of
how abstract faces affect our behaviors.

2-1200 Muzai Gakuendai
Inzai, Chiba 270-1382, Japanyuasa@sie.dendai.ac.jp
Keiichi Saito
Research Center for AdvancedTechnologies
Tokyo Denki University
2-1200 Muzai Gakuendai
Inzai, Chiba 270-1382, Japansaito@rcat.dendai.ac.jp
Naoki Mukawa
Tokyo Denki University
2-1200 Muzai Gakuendai
Inzai, Chiba 270-1382, JapanKeywords

mukawa@sie.dendai.ac.jp

Functional MRI, facial expression, brain activity, emoticon,
face mark, nonverbal communication, life-like character,personified agent

ACM Classification Keywords
H.1.2 User/Machine Systems: Human factors.
Copyright is held by the author/owner(s).

Introduction

CHI 2006, April 22–27, 2006, Montréal, Québec, Canada.

People who exchange messages via e-mail and chat often
use symbols for faces, such as the smiley :-). Emoticons

ACM 1-59593-298-4/06/0004.

1565

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

convey users’ emotions and have an important role inemphasizing the emotions conveyed in a sentence.
We also use animated agents and avatars in virtual
reality; they convey not only verbal messages but also
nonverbal information such as facial expressions andbody gestures. Animated agents are applied to online
sales negotiation via networks [12]. It was found that theuse of facial expressions has important effects on theother party’s decision-making. For example, in online
negotiation, a user tends to agree with a proposal when
the opponent’s face is happy and discontinues thenegotiation when the opponent’s face is angry.

users. However, the mechanism by which the abstract
faces are treated in the mental process and how theyaffect the behaviors and the decision-making incommunication over a computer network is unknown.
In this paper, we investigated the brain activities that areassociated with emoticons―the most abstract faces― byusing functional MRI (fMRI) and described the results ofthe experiment with appropriate remarks.

We use many types of faces from abstract tophotorealistic in computer network applications. Abstract
faces include composite sketches, computer graphics(CG) avatars, emoticons, etc. Emoticons, in particular,
are highly abstract (Figure 1 [2, 6]).
Figure 1. Faces used in communication over computer networks.

Many researchers in psychology and neurophysiology
investigate the manner in which human faces affect otherpeople. These findings are applicable to contemporary
network communication if photorealistic face images areused.
However, photorealistic images cannot be used in e-mails
and chats, and more importantly, they convey
impressions of the paradoxical “the Uncanny Valley” [7,10] or look like images of zombies. Fortunately, when wesee abstract faces such as in cartoons, our brain fills in
the gaps of reality and we see abstract faces that arelively and convincing [5, 6]. The human brain may
perceive the abstract face in an entirely different manner,depending on its level of reality. An avatar composed ofan abstract face may convey a better impression to theJapanese Emoticons
This research uses Japanese emoticons for the following
two reasons. First, they represent emotions more
expressively than Western emoticons because the former
are composed of a double-byte character set. Figure 2
shows the Japanese and Western emoticons.
The second reason is that Japanese emoticons arevertically oriented and are more characteristic of real
faces; Western emoticons, on the other hand, arehorizontally oriented (Figure 3). The former are therefore
more representative of human faces and are somewhat
in between characters and real faces. Thus, they aresuitable as stimuli for analyzing the responses of thebrain.

1566

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

reported the activation of this region during thediscrimination of prosody and the assessment of emotion
based on prosodic cues in voices. Additionally, throughan emotional valence decision task, Nakamura [8] foundthat faces in photographs activated most of the fusiform
gyrus, the right inferior frontal gyrus, the right inferiorparietal lobule, and the right middle frontal gyrus.
Therefore, it is said that the right inferior frontal gyrus isrelated to the processing of emotional communicativesignals.

MethodFigure 2. Japanese and Western emoticons.

Subjects
Seven Japanese students (age range: 20–22; five males
and two females) who frequently used emoticons in theire-mails and chats participated in the experiment.
Functional MRI and Block Design

Figure 3. Composition of Japanese (left) and Western emoticons(right).

Previous Studies
According to previous studies, the right fusiform gyrus isstrongly associated with the perception of faces [3]. Tonget al. stated that this region was more active during theviewing of photographs of human faces than of schematic
faces [11].
Previous studies have reported that the right inferior
frontal gyrus processes emotional communicative signals,
which could be visual or auditory. Kawashima [4]Data were acquired on a 1.5-T HITACHI STRATIS MRI
scanner equipped with a quadrature RF head coil. We
conducted three types of experiments (1–3) in this study.
The order of the experiments was randomly selected foreach subject. Data were collected during three 61-scan(i.e., 305 s) runs. Each run lasted for 5 min andpresented 6 alternating epochs of a condition of selecting
the target or a baseline condition consisting of viewing
visual stimuli. The targets and stimuli were presented
using a projector. In Exp.1 and Exp.2, the subjects
underwent block-designed scans while evaluating theemotional valence of targets (happy or sad) andresponded by pushing a button.
Exp.1 Face Image and Non-Face Image
In this experiment, we used average face images thatwere prepared in the Harashima Laboratory of Tokyo
University [1] (Figure 4). We used two average images:a male image from students in Tokyo University and a1567

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

female one from researchers. These faces were modified
to express happiness and sadness. The reason why weused average faces is that they are less individualistic
than the face of a single person. Scrambled images ofthe average faces were used as control stimuli.
Exp.2 Emoticon and Non-emoticonThis study employed emoticons (adopted from Takehara[9]) and non-emoticons, which consisted of randomcharacters used in emoticons (Figure 5).
Exp.3 Sentences with and without Emoticons
Figure 6 shows examples of sentences with emoticons(T1 and T2) and those without them (R1 and R2). T1states, “I enjoyed singing the song :-)”; T2 states, “I lostmy precious clock :-(.” We also prepared sentences that
included emoticons expressing emotions contrary to themeaning of the sentence, for example, “I enjoyed singing
the song :-(” and “I lost my precious clock :-).” Thesubjects were required to judge the use of the emoticons
in the sentences provided and respond by pushing abutton.

Figure 5. Examples of Japanese emoticons and non-emoticons.

Figure 6. Examples of sentences with emoticons (T1 and T2)
and without emoticons (R1 and R2).

Results

Figure 4. Examples of average faces and scrambled images.

For the statistical parametric mapping analysis (SPM), a
canonical homodynamic response function within SPM99
was employed as a basis function. The resulting SPM {t}map was transformed to the unit normal distribution SPM{Z} and the threshold at a significance level of p < 0.05
corrected for multiple comparisons in SPM. Figure 7
shows a portion of the results obtained during theexperiment. The activity areas are colored black. Figure 7
shows the (1) right fusiform gyrus, (2) right inferior
frontal gyrus, (3) right middle frontal gyrus, and (4) rightinferior parietal lobule. All of these areas were related tothe emotional valence decision task in previous studies
on faces.

1568

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, CanadaFigure 7. Brain activities in experiments 1–3. The shaded areas indicate the activity areas: (1) right fusiform gyrus (2) right inferior
frontal gyrus (3) right middle frontal gyrus, and (4) right inferior parietal lobule.

Discussion
The right fusiform gyrus was activated in response to thephotographs presented in Exp.1. However, this region
was not activated in response to the emoticons presented
in Exp.2. The right fusiform gyrus is known to be
associated with the perception of human faces [3, 11]. It
is possible that the emoticons were not sufficient toconvince the subjects that the target was a face and toaffect the right fusiform gyrus. On the other hand,surprisingly, the right inferior frontal gyrus, which is
known to be activated by the emotional valence decisiontask [8], was activated in both Exp.1 and Exp.2.
Based on the results of these two cases, we infer thatwhen humans see the emoticon, they can detect theemotional valence of their opponent even though they do
not perceive the emoticons as a face. Since theemoticons were created to reflect the real human facial
expressions as accurately as possible and to express
many kinds of emotions, we believed that they would
activate the fusiform gyrus. However, this region was notfound to be activated during the experiment. Meanwhile,

the activation of the right inferior frontal gyrus indicatesthat emoticons certainly play a role in enriching users’emotions, just as prosody enriches vocal expressions.
Remarkably, emoticons convey emotions without
cognition of faces. This is a very important finding, and it
will be useful in understanding how abstract faces affectour behaviors and decision-making in communication
over a computer network.
The right middle frontal gyrus and the right inferiorparietal lobule were activated in Exp.1 and Exp.2. These
areas were activated by the differences in attention given
to determine whether or not the target was a face(whether or not it should be discriminated) in the blockdesign, as seen in Nakamura’s study [8].
The right inferior frontal gyrus and the right middle
frontal gyrus were activated in response to the sentenceswith emoticons that were presented in Exp.3. These
areas were also activated in response to the photographs
and emoticons presented in Exp.1 and Exp.2,
respectively. However, there was a subtle difference inthe location of the activity in the case of Exp.3. Based on1569

CHI 2006 · Work-in-Progress

April 22-27, 2006 · Montréal, Québec, Canada

this, we believe that these areas play the role of theworking-memory. In our opinion, when a person
encounters a sentence with an emoticon, he/shememorizes the content of the sentence and then looks at
the emoticon in order to recognize both the sentence andthe emoticon. The use of the working-memory in verbal
and nonverbal (sentence and emoticon) communicationis an invaluable finding. The results will contribute tofuture studies investigating how abstract faces affect ourbehaviors.

[2] Hasegawa, O. and Sakaue, K. A CG tool for
constructing anthropomorphic interface agents. In Proc.
IJCAI-97, (1997), 23-26.

Conclusion[5] Koda, T. Agents with Faces: A Study on the Effects
of Personification of Software Agents.
http://alumni.media.mit.edu/~tomoko/home-page.html.

In this paper, we conducted experiments using fMRI toinvestigate the brain activities that are associated withemoticons. Based on the experimental results, we believethat emoticons convey emotions without the cognition offaces and that these results are important implications
for investigating how abstract faces affect our behaviors.
In the future, based on these results, we will attempt to
conduct experiments on abstract faces and investigate
how such faces are treated in the mental process andhow they affect the behaviors in communication overnetworks.
Acknowledgments We would like to thank Takuma
Takehara at Hokusei Gakuen University for providing uswith advice. This research was partially supported by theMinistry of Education, Culture, Sports, Science andTechnology (Grant-in-Aid for Scientific Research (B)16300032, 2005) and by the Research Center for
Advanced Technologies, Tokyo Denki University.

References[1] Harashima-Naemura Lab., University of Tokyo.
http://www.hc.ic.i.u-tokyo.ac.jp/index.php.

[3] Kanwisher, N., McDermott, J. and Chun, M.M. The
fusiform face area: A module in human extrastriate
cortex specialized for face perception. Journal of
Neuroscience 17, 11 (1997), 4302-4311.
[4] Kawashima, R., Itoh, M., Hatazawa, J., Miyazawa, H.,Yamada, K., Matsuzawa, T. and Fukuda, H. Changes ofregional cerebral blood flow during listening to an
unfamiliar spoken language. Neuroscience Letters 161(1993), 69-72.

[6] McCloud, S. Understanding Comics. HarperPerennial,(1993).
[7] Monsters of Photorealism.
http://www.wired.com/news/culture/0,1284,69739,00.ht
ml.
[8] Nakamura, K. Activation of the right inferior frontal
cortex during assessment of facial emotion. Advances in
Neurological Sciences 43, 4 (1999), 519-527.
[9] Takehara, T., Kuribayashi, Y., Mukawa, N., Mizuoka,
I. and Takinami, E. The effect of emotionally inconsistentface marks embedded in emotional messages. Journal ofJapanese Academy of Facial Studies 5, 1 (2005), 75-89.
[10] The Uncanny Valley.
http://www.arclight.net/~pdb/nonfiction/uncannyvalley.html.
[11] Tong, F., Nakayama, K., Moscovitch, M., Weinrib, O.
and Kanwisher, N. Response properties of the humanfusiform face area. Cognitive Neuropsychology 17(1/2/3), (2000), 257-279.
[12] Yuasa, M., Yasumura, Y. and Nitta, K. A tool foranimated agents in network-based negotiation. In Proc.
RO-MAN'2003. 4B3, (2003), 259-264.

1570