ASIAGRAPH 2009 PROCEEDINGS

iFeel_IM!: オンラインコミュニケーションにおける感情増幅のための着用型触感提示システム
iFeel_IM!: Affective Haptic Garment for Emotion Enhancement during
Online Communication
ティエティエルコブ

ズミトリー/東京大学 情報理工学系研究科システム情報学専攻, ネビアロスカヤ学系研究科電子情報学専攻, プレンディンガー
工学系研究科, 石塚

アレナ/東京大学 情報理工

ヘルムト/国立情報学研究所 コンテンツ科学研究系, 川上満/東京大学 情報理工学系研究科電子情報学専攻, 舘直樹/東京大学 情報理

暲/慶應義塾大 メデイアデザイン研究科

Dzmitry Tsetserukou1/Department of Information Physics and Computing, University of Tokyo, Alena Neviarouskaya2/Department of Information
and Communication Engineering, University of Tokyo, Helmut Prendinger3/ Digital Content and Media Sciences Research Division, National
Institute of Informatics, Naoki Kawakami4/ Dept. of Information Physics and Computing, University of Tokyo, Mitsuru Ishizuka5/ Department ofInformation and Communication Engineering, University of Tokyo, Susumu Tachi6/Graduate School of Media Design, Keio University*1dima_teterukov@ipc.i.u-tokyo.ac.jp, *2lena@mi.ci.i.u-tokyo.ac.jp, *3helmut@nii.ac.jp, *4kawakami@star.t.u-tokyo.ac.jp,*5ishizuka@i.u-tokyo.ac.jp, *6tachi@tachilab.org

Abstract:

The paper focuses on a novel system iFeel_IM! that integrates 3D virtual world Second Life, intelligent component for
automatic emotion recognition from text messages, and innovative affective haptic interfaces providing additional nonverbal
communication channels through simulation of emotional feedback and social touch (physical co-presence). The motivation
behind our work is to enrich social interaction and emotional involvement of online interpersonal communication. iFeel_IM!
users can not only exchange messages but also emotionally and physically feel the presence of the communication partner(e.g., family member, friend, or beloved person).

Keywords:

Affective haptics, affective user interface, wearable devices, virtual reality, communication in virtual world.

1. Introduction
Interpersonal communication in 3D virtual world (e.g.,Second Life, OpenSim) through instant messenger and chatis very popular nowadays. Recently mobile communicationcompanies launched Instant Messenger service on cellularphones (e.g. AIM on iPhone). Conventional mediatedsystems usually (1) support only simple textual cues likeemoticons; (2) lack visual emotional signals such as facial
expressions and gestures; (3) support only manual control ofexpressiveness of graphical representations of users(avatars); and (4) completely ignore such important channelof social communication as sense of touch.
Besides emotions conveyed through text, researchersdeveloped an additional modality for communicating
emotions in Instant Messenger (IM) through tactileinterfaces with vibration patterns [Rovers04]. However, inthe proposed methods users have to memorize the vibrationor pin matrix patterns and cognitively interpret thecommunicated emotional state.
Driven by the motivation to enhance social interactivity
and emotionally immersive experience of real-time
messaging, we pioneered in the idea of reinforcing(intensifying) own feelings and reproducing (simulating) theemotions felt by the partner through specially designed
system, iFeel_IM! (Fig. 1). The philosophy behind theiFeel_IM! (intelligent system for Feeling enhancement
powered by affect sensitive Instant Messenger) is “I feel[therefore] I am!”. The emotion elicited by physicalstimulation might imbue our communication with passion
and increase the emotional intimacy. The interpersonal
relationships and the ability to express empathy growstrongly when people become emotionally closer.

Figure 1: User communicating through iFeel_IM!

2. Architecture of the iFeel_IM! SystemIn the iFeel_IM! system, great importance is placed on theautomatic sensing of emotions conveyed through textual
messages in 3D virtual world Second Life, the visualizationof the detected emotions by avatars in virtual environment,enhancement of user’s affective state, and reproduction offeeling of social touch (e.g., hug) by means of haptic
stimulation in a real world. The architecture of theiFeel_IM! is presented in Fig. 2.
In order to communicate through iFeel_IM! system,
users have to wear innovative affective haptic devices(HaptiHeart, HaptiHug, HaptiButterfly, HaptiTickler,
HaptiTemper, and HaptiShiver) developed by us. As a120

AffectAnalysis
Modelchat text
emotion: intensity

3D world Second LifePC
HapticDevices
Controller

Chatlog
file
D/ADriver Box

HaptiHeart

HaptiTemperand HaptiShiver

HaptiHug

HaptiTickler

HaptiBatterfly

Figure 2: Architecture of iFeel_IM! system.

Analysis Model and EmoHeart are stored along with chatmessages in a file on local computer of each user. Haptic

media for communication, we employ Second Life, whichallows users to flexibly create their online identities(avatars) and to play various animations (e.g., facialexpressions and gestures) of avatars by typing special
abbreviations in a chat window.
The control of the conversation is implemented throughthe Second Life object called EmoHeart attached to theavatar’s chest. Once attached to the avatar, EmoHeart object(1) listens to each message of its owner, (2) sends it to theweb-based interface of the Affect Analysis Model (AAM)[Neviarouskaya09] located on the server, (3) receives theresult (dominant emotion and intensity), and visuallyreflects the sensed affective state through the animation ofavatar’s facial expression, EmoHeart texture (indicating thetype of emotion), and size of the texture (indicating thestrength of emotion, namely, ‘low’, ‘middle’, or ‘high’). The
motivation behind using the heart-shaped object as anadditional channel for visualization was to represent thecommunicated emotions in a vivid and expressive way. The
examples of avatar facial expressions and EmoHearttextures are shown in Fig. 3.
In addition to communication with Affect Analysis
Model, EmoHeart is responsible for sensing symbolic cues
or keywords of ‘hug’ communicative function conveyed bytext, and for visualization (triggering related animation) of‘hugging’ in Second Life. The results from the AffectJoySadnessAngerFear

Figure 3: Avatar facial expressions and EmoHeart textures.

121

Devices Controller analyses these data in a real time andgenerates control signals for Digital/Analog converter(D/A), which then feeds Driver Box for haptic devices withcontrol cues. Based on the transmitted signal, thecorresponding haptic device (HaptiHeart, HaptiHug,
HaptiButterfly,
HaptiTickler,
HaptiTemper,
andHaptiShiver) worn by user is activated.

3. Affective Haptic Devices
In order to support the affective communication, we
implemented several novel haptic gadgets embedded iniFeel_IM!. They make up three groups. First group isintended for emotion elicitation implicitly (HaptiHeart,HaptiButterfly, HaptiTemper, and HaptiShiver), second typeevokes affect in a direct way (HaptiTickler), and third one
uses sense of social touch (HaptiHug) for influencing on themood and providing some sense of physical co-presence.
The affective haptic devices worn on a human body arepresented in Fig. 4.

3.1. HaptiHug:
Distance

RealisticHugging

Figure 4: Affective haptic devices worn on a human body.

The duration and intensity of the hug are controlled by
the software in accordance with the emoticon or a keyword,
detected from text. For the presentation of a plain hug level(e.g., ‘(>^_^)>’, ‘{}’, ‘<h>’), a big hug level (e.g., ‘>:D<’,‘{{}}’), and a great big hug level (e.g., ‘gbh’, ‘{{{}}}’), thepressure of 200 N/m2 with duration of 2 sec, the pressureof 300 N/m2 with duration of 3 sec, and the pressure of 450
N/m2 with duration of 4 sec, was applied on the user’s backand chest, respectively.

OverOn-line interactions rely on senses of vision and hearing,and there is a substantial need in mediated social touch[Haans06]. Among many forms of physical contact, hug isthe most emotionally charged one. It conveys warmth, love,and affiliation.
When people are hugging, they generate pressure on thechest area and on the back of each other by the hands,simultaneously. The key feature of the developed HaptiHugis that it physically reproduces the hug pattern similar to thatof human-human interaction.
The hands for a HaptiHug are sketched from a realhuman and made from soft material so that hugging partners
can realistically feel social presence of each other. The
couple of oppositely rotating motors are incorporated into
the holder placed on the user chest area. The Soft Hands,
which are aligned horizontally, contact back of the user.
Once ‘hug’ command is received, couple of motors tense thebelt, pressing thus Soft Hands and chest part of theHaptiHug in the direction of human body (Fig. 5).
Direction ofbelt tension

Figure 6: Hugging animation in Second Life.

The significance of our idea to realistically reproduce
hugging is in integration of active-haptic device HaptiHugand pseudo-haptic touch simulated by hugging animation(Fig. 6). Thus, high immersion into the physical contact ofpartners while hugging is achieved.

Soft Hand
Pressure

3.2. HaptiHeart Enhancing Our Emotions

Soft Hand

Each emotion is characterized by a specific pattern ofphysiological changes. We selected four distinct emotions
having strong physical features [Wallbott88]: ‘anger’, ‘fear’,‘sadness’, and ‘joy’.
Of the bodily organs, the heart plays a particularly
important role in our emotional experience. The ability offalse heart rate feedback to change our emotional state wasreported in [Decaria74]. We developed heart imitator
HaptiHeart to produce special heartbeat patterns according
to emotion to be conveyed or elicited (sadness is associatedwith slightly intense heartbeat, anger with quick and violent
heartbeat, fear with intense heart rate). False heart beat

Human bodyMotorholder
PressureCouple of motors
PressureDirection of belt tension

Figure 5: Structure of wearable HaptiHug device.

122

feedback can be directly interpreted as a real heart beat, so it
can change the emotional perception.
The HaptiHeart consists of two modules: flat speakerFPS 0304 and speaker holder. The flat speaker sizes (66.5 x107 x 8 mm) and rated input power of 10 W allowed us todesign powerful and relatively compact HaptiHeart device.
It is able to produce realistic heartbeating sensation withhigh fidelity. The 3D model is presented in Fig. 7.

Flat Speaker

positive affect (joy emotion) in a direct way by tickling theribs of the user. The uniqueness of our approach is in (1)combination of the unpredictability and uncontrollability ofthe tickling sensation through random activation of vibration
motors, (2) high involvement of the social and emotional
factors in the process of tickling (positively charged on-line
conversation potentiates the tickle response).

3.4. Emotional Haptic Design
Aesthetically pleasing objects appear to the user to be more
effective by virtue of their sensual appeal [Norman04]. Our
core idea is to make user to feel affinity for the device bymeans of (1) appealing shapes evoking the desire to touchand haptically explore them, (2) usage of material
pleasurable to touch, and (3) the pleasure anticipatedthrough wearing. The designed devices are pleasurable tolook at and to touch and have personalized features (e.g. theSoft Hands in HaptiHug). The heart-shaped HaptiHeart wasdesigned with primary objective to emotionally connect theuser with the device, as heart is mainly associated with loveand emotional experience. The HaptiButterfly, its shape, andactivated vibration motors induce the association with a realbutterfly lightly touching the human.

Heart-shaped
Speaker Case

Figure 7: HaptiHeart layout.

The pre-recorded sound signal with low frequency
generates the pressure on the human chest through vibrationof the speaker surface.

4. Conclusions

3.3. Butterflies in the Stomach and Shivers on
the Body’s Spine

While developing the iFeel_IM! system, we attempted tobridge the gap between mediated and face-to-facecommunications by enabling and enriching the spectrum ofsenses such as vision and touch along with cognition andinner personal state. In the paper we described thearchitecture of the iFeel_IM! and the development of novel
haptic devices, such as HaptiHeart, HaptiHug, HaptiTickler,
HaptiButterfly, HaptiShiver, and HaptiTemper. User ofiFeel_IM! can perceive the intensive emotions during online
communication, use desirable type of stimuli, comfortablywear and easily detach devices from torso.

HaptiButterfly is responsible for the evocation of joy
emotion (Fig. 8). The idea behind this device is to reproduce
effect of “Butterflies in the stomach” (fluttery or tickling
feeling in the stomach felt by people experiencing love) bymeans of the arrays of vibration motors attached to theabdomen area of a person.
Plastic frame
Revolute joint
Vibration
motor

References
Decaria, M.D., Proctor, S., Malloy, T.E., 1974. The effect of false
heart rate feedback on self reports of anxiety and on actual heartrate. Behavior research & Therapy, 12:251–253.
Haans, A., Ijsselsteijn, W. I., 2006. Mediated social touch: a reviewof current research and future directions. Virtual Reality, 9:149–159.
Neviarouskaya, A., Prendinger, H., Ishizuka, M., 2009.
Compositionality principle in recognition of fine-grained
emotions from text. In Proc. 3rd International Conference onWeblogs and Social Media, pp. 278–281.
Norman, D.A., 2004. Emotional design. Why we love (or hate)everyday things, Basic Book. New-York.
Rovers, A.F., Van Essen, H.A., 2004. HIM: A framework forhaptic Instant Messaging. In Proc. SIGCHI Conference on
Human Factors in Computing Systems, pp. 1313–1316.
Wallbott, H.G., Scherer, K.R., 1988. How universal and specific isemotional experience? Evidence from 27 countries on fivecontinents. In K.R. Scherer (Eds.), Facets of Emotion: RecentResearch, Lawrence Erlbaum Inc. Hillsdale (N.J.), pp. 31–56.

HolderFigure 8: Structure of HaptiButterfly.

We conducted the experiment aimed at investigation of
the patterns of vibration motor activation that produce mostpleasurable and natural sensations on the abdomen area.
In order to boost fear emotion physically, we designed
HaptiShiver interface that sends “Shivers down/up humanbody’s spine” by means of a row of vibration motors(HaptiShiver), and “Chills down/up human body’s spine”
through both cold airflow from DC fan and cold side of
Peltier element (HaptiTemper). HaptiTemper is also
intended for simulation of warmth on the human skin toevoke either pleasant feeling or aggression.
We developed HaptiTickler with the purpose to evoke
123