ICROS-SICE International Joint Conference 2009
August 18-21, 2009, Fukuoka International Congress Center, Japan

Emerging System for Affectively Charged Interpersonal Communication
Alena Neviarouskaya1, Dzmitry Tsetserukou2, Helmut Prendinger3, Naoki Kawakami2,
Susumu Tachi4, Mitsuru Ishizuka1
1

Department of Information and Communication Engineering, University of Tokyo, Tokyo, Japan(Tel : +81-3-5841-6774; E-mail: lena@mi.ci.i.u-tokyo.ac.jp, ishizuka@i.u-tokyo.ac.jp)
2
Department of Information Physics and Computing, University of Tokyo, Tokyo, Japan(Tel : +81-3-5841-6917; E-mail: dima_teterukov@ipc.i.u-tokyo.ac.jp, kawakami@star.t.u-tokyo.ac.jp)
3
National Institute of Informatics, Tokyo, Japan (E-mail: helmut@nii.ac.jp)
4
Graduate School of Media Design, Keio University, Tokyo, Japan (E-mail: tachi@tachilab.org)
Abstract: The paper focuses on a novel system iFeel_IM! that integrates 3D virtual world Second Life, intelligent
component for automatic emotion recognition from text messages, and innovative affective haptic interfaces providing
additional nonverbal communication channels through simulation of emotional feedback and social touch (physicalco-presence). The motivation behind our work is to enrich social interaction and emotional involvement of online
interpersonal communication. iFeel_IM! users can not only exchange messages but also emotionally and physically feelthe presence of the communication partner (e.g., family member, friend, or beloved person).
Keywords: Affective haptics, affective user interface, wearable devices, communication in virtual world.

1. INTRODUCTION
The popularity and appeal of 3D virtual worlds withembedded chat or instant messenger (e.g., Second Life,OpenSim) have made their ways around the globe. Suchsystems encourage people to establish or strengtheninterpersonal relations, to share ideas, to gain newexperiences, and to feel genuine emotions
accompanying all adventures of virtual reality.
Conventional mediated systems usually (1) support only
simple textual cues like emoticons; (2) lack visualemotional signals such as facial expressions andgestures; (3) support only manual control ofexpressiveness of graphical representations of users(avatars); and (4) completely ignore such importantchannel of social communication as sense of touch.
Besides emotions conveyed through text, researchersdeveloped an additional modality for communicating
emotions in Instant Messenger (IM) through tactileinterfaces with vibration patterns [1, 2]. However, in the
proposed methods users have to memorize the vibrationor pin matrix patterns and cognitively interpret thecommunicated emotional state. Demodulation of
haptically coded emotion is not natural forhuman-human communication, and direct evocation ofemotion cannot be achieved in such kind of systems.
Driven by the motivation to enhance social
interactivity and emotionally immersive experience ofreal-time messaging, we pioneered in the idea ofreinforcing (intensifying) own feelings and reproducing(simulating) the emotions felt by the partner throughspecially designed system, iFeel_IM!. The philosophy
behind the iFeel_IM! (intelligent system for Feelingenhancement powered by affect sensitive InstantMessenger) is “I feel [therefore] I am!”. The emotion
elicited by physical stimulation might imbue ourcommunication with passion and increase the emotional
intimacy, ability to be close, loving, and vulnerable. The- 3376 -interpersonal relationships and the ability to express
empathy grow strongly when people become
emotionally closer through disclosing thoughts, feelings,
and emotions for the sake of understanding.

2. ARCHITECTURE OF THE iFeel_IM!
SYSTEM
In the iFeel_IM! system, great importance is placedon the automatic sensing of emotions conveyed throughtextual messages in 3D virtual world Second Life, thevisualization of the detected emotions by avatars in
virtual environment, enhancement of user’s affective
state, and reproduction of feeling of social touch (e.g.,hug) by means of haptic stimulation in a real world. The
architecture of the iFeel_IM! is presented in Fig. 1.
In order to communicate through iFeel_IM! system,
users have to wear innovative affective haptic devices(HaptiHeart, HaptiHug, HaptiButterfly, HaptiTickler,
HaptiTemper, and HaptiShiver) developed by us. As amedia for communication, we employ Second Life,
which allows users to flexibly create their online
identities (avatars) and to play various animations (e.g.,
facial expressions and gestures) of avatars by typing
special abbreviations in a chat window.
The control of the conversation is implemented
through the Second Life object called EmoHeart(invisible in case of ‘neutral’ state) attached to theavatar’s chest. In addition to communication with thesystem for textual affect sensing (Affect Analysis
Model), EmoHeart is responsible for sensing symboliccues or keywords of ‘hug’ communicative function
conveyed by text, and for visualization (triggering
related animation) of ‘hugging’ in Second Life. The
results from the Affect Analysis Model (dominantemotion and intensity) and EmoHeart (‘hug’
communicative function) are stored along with chatmessages in a file on local computer of each user.

PR0002/09/0000-3376 ¥400 © 2009 SICEchat text

AffectAnalysisModelemotion: intensity

3D world Second Life
PC
HapticDevices
Controller

Chat
log
file
D/A

HaptiTemper
HaptiHeartand

HaptiHug

HaptiShiver
HaptiTickler

HaptiBatterfly

Fig. 1 Architecture of iFeel_IM! system.
Haptic Devices Controller analyses these data in areal time and generates control signals for
Digital/Analog converter (D/A), which then feeds
Driver Box for haptic devices with control cues. Basedon the transmitted signal, the corresponding haptic
device(HaptiHeart,
HaptiHug,
HaptiButterfly,
HaptiTickler, HaptiTemper, and HaptiShiver) worn byuser is activated.

domination, neutralization, and intensification, atvarious grammatical levels. Analyzing each sentence insequential stages, this method is capable of processing
sentences of different complexity, including simple,
compound, complex (with complement and relative
clauses), and complex-compound sentences. The
working flow of the algorithm is presented in Fig. 2.
To measure the accuracy of the proposed emotion
recognition algorithm, we extracted 700 sentences froma collection of diary-like blog posts provided by
BuzzMetrics (http://www.nielsenbuzzmetrics.com). We
focused on online diary or personal blog entries, which
are typically written in a free style and are rich inemotional colourations. Three independent annotatorslabelled the sentences with one of nine emotions (orneutral) and a corresponding intensity value.
We developed two versions of the Affect Analysis
Model (AAM) differing in syntactic parsers employedduring the second stage of affect recognition algorithm:(1) AAM with commercial parser Connexor Machinese
Syntax (http://www.connexor.eu) (AAM-CMS); (2)AAM with GNU GPL licensed Stanford Parser(http://nlp.stanford.edu/software/lex-parser.shtml)(AAM-SP). The performance of the AAM-CMS and
AAM-SP was evaluated against two sets of sentences

3. RECOGNITION OF AFFECT FROM TEXTThe Affect Analysis Model [3] sense nine emotions
conveyed through text (‘anger’, ‘disgust’, ‘fear’, ‘guilt’,‘interest’, ‘joy’, ‘sadness’, ‘shame’, and ‘surprise’). The
affect recognition algorithm, which takes into accountspecific style and evolving language of online
conversation, consists of five main stages: (1) symboliccue analysis; (2) syntactical structure analysis; (3)word-level analysis; (4) phrase-level analysis; and (5)sentence-level analysis. Our Affect Analysis Model was
designed based on the compositionality principle,
according to which we determine the emotional
meaning of a sentence by composing the pieces that
correspond to lexical units or other linguistic constituent
types governed by the rules of aggregation, propagation,- 3377 -sentence
Symbolic cue analysis module
Test for emoticons, abbreviations,acronyms, interjections, “?” and“!” marks, repeated punctuationand capital letters

AffectDatabase
Syntacticalstructure
analysis module

Emoticon or em. abbr.
“yes”
“no”

Syntactical Parser

Phrase-level
analysis module

Parser output
processing

Sentence-level
analysis module

Sentence preprocessing forparserEstimation of
resultingemotion stateemotion category: intensity

Word-level
analysis module

emotion category: intensity

Sentence annotated byemotion stateFig. 2 Working flow of the Affect Analysis Model.
related to ‘gold standards’: 656 sentences, on which twoor three human raters completely agreed; (2) 249
sentences, on which all three human raters completely
agreed. Averaged accuracy, precision, and recall are
shown in Table 1 for each emotion category. An
empirical evaluation of the AAM algorithm showedpromising results regarding its capability to accurately(AAM-CMS achieves accuracy in 81.5 %) classify
affective information in text from an existing corpus of
informal online communication.

for visualization was to represent the communicatedemotions in a vivid and expressive way.
During a two month period (December 2008 –January 2009), 89 Second Life users became owners of
EmoHeart, and 74 of them actually communicated using
it. Text messages along with the results from AAM werestored in an EmoHeart log database. From all sentences,
20 % were categorized as emotional by the AAM and
80 % as neutral (Fig. 3). We observed that thepercentage of sentences annotated by positive emotions(‘joy’, ‘interest’, ‘surprise’) essentially prevailed(84.6 %) over sentences annotated by negative emotions(‘anger’, ‘disgust’, ‘fear’, ‘guilt’, ‘sadness’, ‘shame’).
We believe that this dominance of positivity expressed
through text is due to the nature and purpose of online
communication media.

4. EmoHeart
Once attached to the avatar in Second Life, EmoHeartobject (1) listens to each message of its owner, (2) sends
it to the web-based interface of the AAM, (3) receives
the result (dominant emotion and intensity), and visuallyreflects the sensed affective state through the animationof avatar’s facial expression, EmoHeart texture(indicating the type of emotion), and size of the texture(indicating the strength of emotion, namely, ‘low’,‘middle’, or ‘high’). If no emotion is detected in the text,
the EmoHeart remains invisible and the avatar facial
expression remains neutral. The motivation behindusing the heart-shaped object as an additional channel

percent of all sentences, %Neutral
0

20

40

60

80

100

Emotional

percent of em otional sentences, %Negative
0

20

40

60

80

100

Positive

Fig. 3 Percentage distribution of sentences.

Table 1 The results of experiment with Affect Analysis Model employing different parsers.
Gold
standard
2-3
annotators
agreed3
annotators
agreedVersion of
AAM
AAM-CMSAAM-SP

AAM-CMSAAM-SP

MeasureAccuracy
Precision
RecallAccuracy
Precision
RecallAccuracy
Precision
RecallAccuracy
Precision
Recallneutral anger

disgust

0.46
0.55

0.83
0.41

0.63
0.73

0.30
0.55

0.77
0.34

0.64
0.70

0.26
0.75

0.92
0.65

0.83
0.56

0.15
0.75

0.92
0.65

0.83
0.56

Fine-grained emotion categories
fear
guilt interest joy
sadness shame surprise
0.726
0.76
0.75
0.56
0.87
0.78
0.57
0.85
0.84
0.68
0.88
0.83
0.72
0.89
0.77
0.649
0.74
0.71
0.61
0.83
0.74
0.50
0.76
0.80
0.55
0.81
0.71
0.64
0.67
0.72
0.815
0.91
0.83
0.44
0.95
0.88
0.67
0.86
0.88
0.83
0.88
0.88
0.79
0.67
0.82
0.751
0.87
0.80
0.50
0.96
0.88
0.50
0.82
0.83
0.67
0.75
0.78
0.74
0.33
0.82

- 3378 -We analysed the distribution of emotional sentencesfrom EmoHeart log data according to the fine-grained
emotion labels from our Affect Analysis Model (Fig. 4).
anger

8.8
0.6
9.0

68.8

1.1
1.0
6.9

1.8
2.1

disgustfear
guilt
interestjoysadnessshame
surpriseFig. 5 Affective haptic devices worn on a human body.
4.1 HaptiHug: realistic hugging over distanceAmong many forms of physical contact, hug is themost emotionally charged one. It conveys warmth, love,
and affiliation. DiSalvo et al. [6] introduced “The Hug”
interface. When person desires to communicate hug,
he/she can squeeze the pillow, so that such action resultsin the vibration and temperature changes in the partner’s
device. However, this interface suffers from inability toresemble natural hug sensation and, hence, to elicit
strong affective experience (only slight pressure is
generated by vibration actuators); and lacks the visual
representation of the partner. A simple haptic displaythat uses dual motors and belt for presentation ofphysical contact by pressing is described in [7].
The key feature of the developed HaptiHug is that it
physically reproduces the hug pattern similar to that ofhuman-human interaction. The hands for a HaptiHug
are sketched from a real human and made from soft
material. The couple of oppositely rotating motors areincorporated into the holder placed on the user chest
area. The Soft Hands, which are aligned horizontally,
contact back of the user. Once ‘hug’ command isreceived, couple of motors tense the belt, pressing thus
Soft Hands and chest part of the HaptiHug in the
direction of human body (Fig. 6).

Fig. 4 Percentage distribution of sentences with
fine-grained emotion annotations.
We found that the most frequent emotion conveyed
through text messages is ‘joy’ (68.8 % of all emotional
sentences), followed by ‘surprise’, ‘sadness’ and‘interest’ (9.0 %, 8.8 %, and 6.9 %, respectively).

4. AFFECTIVE HAPTIC DEVICES
According to James-Lange theory [4], the conscious
experience of emotion occurs after the cortex receives
signals about changes in physiological state.
Researchers argued that feelings are preceded by certain
physiological changes. Recent empirical studies support
non-cognitive theories of nature of emotions. It wasproven that we can easily evoke our emotions bysomething as simple as changing facial expression (e.g.,smile brings on a feeling of happiness) [5].
In order to support the affective communication, we
implemented several novel haptic gadgets embedded iniFeel_IM!. They make up three groups. First group isintended for emotion elicitation implicitly (HaptiHeart,
HaptiButterfly, HaptiTemper, and HaptiShiver), second
type evokes affect in a direct way (HaptiTickler), andthird one uses sense of social touch (HaptiHug) for
influencing on the mood and providing some sense ofphysical co-presence. All these devices produce
different senses of touch including kinesthetic andcoetaneous channels. Kinesthetic stimulations, which
are produced by forces exerted on the body, are sensed
by mechanoreceptors in the tendons and muscles. This
channel is highly involved in sensing stimulus produced
by HaptiHug device. On the other hand,mechanoreceptors in the skin layers are responsible forthe perception of cutaneous stimulation. Different typesof tactile corpuscles allow us sensing thermal propertyof the object (HaptiTemper), pressure (HaptiHeart,
HaptiHug), vibration frequency (HaptiButterfly,
HaptiTickler, and HaptiShiver), and stimuli location(localization of stimulating device enables association
with particular physical contact).
The affective haptic devices worn on a human bodyand their 3D models are presented on Fig. 5.

Direction ofbelt tension

Soft Hand
Pressure
Soft Hand

Human bodyMotorholder

- 3379 -Pressure
Couple of motors
PressureDirection of belt tension

Fig. 6 Structure of wearable HaptiHug device.

The duration and intensity of the hug are controlled
by the software in accordance with the emoticon or a
keyword, detected from text.
4.2 HaptiHeart enhancing user emotions
We selected four distinct emotions having strong
physical features: ‘anger’, ‘fear’, ‘sadness’, and ‘joy’.
The precision of AAM in recognition of these emotions
is considerably higher (‘anger’ – 92 %, ‘fear’ – 91 %,‘joy’ – 95 %, ‘sadness’ – 88 %) than of other emotions.
Of the bodily organs, the heart plays a particularly
important role in our emotional experience. The abilityof false heart rate feedback to change our emotional
state was reported in [8]. There are two major sounds
that are heard in the normal heart and are often
described as a lub and a dub (“lubb-dub” sound occurs
in sequence with each heartbeat) [9].
We developed heart imitator HaptiHeart to produce
special heartbeat patterns according to emotion to beconveyed or elicited (sadness is associated with slightly
intense heartbeat, anger with quick and violent heartbeat,fear with intense heart rate). We take advantage of thefact that our heart naturally synchronizes with the heartof a person we hold or hug. Thus, the heart rate of a user
is influenced by haptic perception of the beat rate of theHaptiHeart. Furthermore, false heart beat feedback canbe directly interpreted as a real heart beat, so it canchange the emotional perception. The 3D model ofHaptiHeart is presented in Fig. 7.

Plastic frame
Revolute joint
Vibration
motor

HolderFig. 8 Structure of HaptiButterfly.
The temperature symptoms are great indicators ofdifferences between emotions. The empirical studies[10] showed that (1) fear and (in a lesser degree)sadness, are characterized as ‘cold’ emotions, (2) joy is
the only emotion experienced as being ‘warm’, while(3) anger is ‘hot’ emotion.
In order to boost fear emotion physically, we
designed HaptiShiver interface that sends “Shivers
down/up human body’s spine” by means of a row ofvibration motors (HaptiShiver), and “Chills down/up
human body’s spine” through both cold airflow from DC
fan and cold side of Peltier element (HaptiTemper). The
structure of HaptiShiver/HaptiTemper device is shownin Fig. 9.
DC FanFront view

Peltierelement

Vibration
motor

Aluminum
plate

Back viewFig. 9 Structure of HaptiShiver/HaptiTemper.
Flat Speaker

Heart-shaped
Speaker Case

HaptiTemper is also intended for simulation of
warmth on the human skin to evoke either pleasantfeeling or aggression (it was proved that uncomfortably
hot temperatures arouse anger feelings [11]).

Fig. 7 HaptiHeart layout.
The pre-recorded sound signal with low frequency
generates the pressure on the human chest throughvibration of the speaker surface.
4.3 Butterflies in the stomach (HaptiButterfly) andshivers on body’s spine (HaptiShiver/HaptiTemper)
HaptiButterfly was developed with the aim to evocate
joy emotion (see Fig. 8). The idea behind this device isto reproduce effect of “Butterflies in the stomach”(fluttery or tickling feeling in the stomach felt by people
experiencing love) by means of the arrays of vibration
motors attached to the abdomen area of a person.
We conducted the experiment aimed at investigationof the patterns of vibration motor activation that
produce most pleasurable and natural sensations on theabdomen area. Based on the results, we employ‘circular’ and ‘spiral’ vibration patterns.

- 3380 -4.4 HaptiTickler: device for positive emotions
Two different types of tickling are recognized. The
first type is knismesis referring to feather-like (light)
type of tickling. It is elicited by a light touch or by alight electrical current at almost any part of the body[12]. It should be emphasized that this type of tickling
does not evoke laugh and is generally accompanied byan itching sensation that creates the desire to rub the
tickled part of the body. The second type of tickle called
gargalesis is evoked by a heavier touch to particularareas of the body such as armpits or ribs. Such kind of
stimuli usually results in laugher and squirming. In
contract to knismesis, one cannot produce gargalesis inoneself. Two explanations were suggested to explain thereasons of inability to self-tickling. The scientists
supporting interpersonal explanations argue that tickling

is fundamentally interpersonal experience and thus
requires another person as the source of the touch [13].
On the other side of the debate is a reflex view,suggesting that tickle requires the element of
unpredictability or uncontrollability. The experimental
results from [14] support the later view and reveal thatticklish laugher evidently does not require that
stimulation be attributed to another person. However,the social and emotional factors in ticklishness affect the
tickle response greatly.
We developed HaptiTickler with the purpose to evoke
positive affect (joy emotion) in a direct way by tickling
the ribs of the user. The device includes four vibration
motors reproducing stimuli that are similar to humanfinger movements during rib tickling.
Large vibration
motor (index finger)
Small vibration
motors (middle andring finger)
Large vibration
motor (little finger)

Fig. 10 HaptiTickler device.
The uniqueness of our approach is in (1) combinationof the unpredictability and uncontrollability of thetickling sensation through random activation of stimuli,(2) high involvement of the social and emotional factors
in the process of tickling (positively charged on-line
conversation potentiates the tickle response).

5. CONCLUSIONS
While developing the iFeel_IM! system, we
attempted to bridge the gap between mediated andface-to-face communications by enabling and enriching
the spectrum of senses such as vision and touch alongwith cognition and inner personal state. In the paper we
described the architecture of the iFeel_IM! and thedevelopment of novel haptic devices, such as
HaptiHeart, HaptiHug, HaptiTickler, HaptiButterfly,
HaptiShiver, and HaptiTemper. These devices were
designed with particular emphasis on natural andrealistic representation of the physical stimuli, modular
expandability, and ergonomic human-friendly design.
The significance of our idea to realistically reproduce
hugging is in integration of active-haptic device
HaptiHug and pseudo-haptic touch simulated byhugging animation. Thus, high immersion into thephysical contact of partners while hugging is achieved.
iFeel_IM! system has great potential to impact on thecommunication in ‘sociomental’ (rather than ‘virtual’)
online environments, that facilitate contact with others
and affect the nature of social life in terms of bothinterpersonal relationships and the character ofcommunity. Our primary goal for the future research is
to conduct extensive user study on iFeel_IM! system.

- 3381 -REFERENCES[1] A.F. Rovers and H.A. Van Essen, “HIM: A
Framework for Haptic Instant Messaging”, in Proc.
SIGCHI Conf. on Human Factors in Computing Systems,
2004, pp. 1313-1316.
[2] H. Shin, J. Lee, J. Park, Y. Kim, H. Oh, T. Lee, “A
Tactile Emotional Interface for Instant Messenger
Chart”, in Proc. Int. Conf. on Human-Computer
Interaction, 2007, pp. 166-175.
[3] A. Neviarouskaya, H. Prendinger, M. Ishizuka,“Compositionality Principle in Recognition ofFine-Grained Emotions from Text”, in Proc. Int. Conf.
on Weblogs and Social Media, 2009.
[4] W. James, “What is an Emotion?”, Mind, Vol. 9, pp.
188-205, 1884.
[5] R.B. Zajonc, S.T. Murphy, M. Inglehart, “Feelingand Facial Efference: Implication of the Vascular
Theory of Emotion”, Psychological review, Vol. 96, pp.
395-416, 1989.
[6] C. DiSalvo, F. Gemperle, J. Forlizzi, E. Montgomery,“The Hug: an Exploration of Robotic Form for Intimate
Communication”, in Proc. IEEE Int. Workshop on
Robot and Human Interactive Communication, 2003, pp.
403-408.
[7] K. Minamizawa, H. Kajimoto, N. Kawakami, S.
Tachi, “A Wearable Haptic Display to Present theGravity Sensation – Preliminary Observation andDevice Design”, in Proc. 2007 IEEE World Haptics
Conf., 2007, pp. 133-138.
[8] M.D. Decaria, S. Proctor, T.E. Malloy, “The Effectof False Heart Rate Feedback on Self Reports ofAnxiety and on Actual Heart Rate”, Behavior research& Therapy. Vol. 12, pp. 251-253, 1974.
[9] L.S. Bickley. Bates’ Guide to Physical Examinationand History Taking. Philadelphia: Lippincott Williams& Wilkins, 2008.
[10] H.G. Wallbott and K.R. Scherer, “How Universal
and Specific is Emotional Experience? Evidence from27 Countries on Five Continents”, in K.R. Scherer(Eds.), Facets of Emotion: Recent Research. Hillsdale(N.J.): Lawrence Erlbaum Inc., pp. 31-56, 1988.
[11] C.A. Anderson, “Temperature and Aggression:
Ubiquitous Effects of Heat on Occurrence of Human
Violence”, Psychological Bulletin, Vol. 106, No. 1, pp.
74-96, 1989.
[12] Y. Sato, K. Sato, M. Sato, S. Fukushima, Y. Okano,
K. Matsuo, S. Ooshima, Y. Kojima, R. Matsue, S.
Nakata, Y. Hashimoto, H. Kajimoto, “Ants in the Pants– Ticklish Tactile Display using Rotating Brushes”, in
Proc. Int. Conf. on Instrumentation, Control andInformation Technology, 2008, pp. 461-466.
[13] H.C. Foot and A.J. Chapman, “The Social
Responsiveness of Young Children in Humorous
Situation”, in A.J. Chapman & H.C. Foot (Eds.),
Humour and Laugher: Theory, research, andapplications. London: Wiley, pp. 187-214, 1976.
[14] C.R. Harris and N. Christenfeld, “Can a Machine
Tickle?”, Psychonomic Bulletin & Review, Vol. 6, No. 3,
pp. 504-510, 1999.