Enhancing Mediated Interpersonal Communication
through Affective Haptics
Dzmitry Tsetserukou1, Alena Neviarouskaya2, Helmut Prendinger3,
Naoki Kawakami1, Mitsuru Ishizuka2, and Susumu Tachi11Department of Information Physics and Computing, University of Tokyo, Japan
dima_teterukov@ipc.i.u-tokyo.ac.jp,{kawakami,tachi}@star.t.u-tokyo.ac.jp
2
Department of Information and Communication Engineering, University of Tokyo, Japanlena@mi.ci.i.u-tokyo.ac.jp, ishizuka@i.u-tokyo.ac.jp
3
National Institute of Informatics, Japan
helmut@nii.ac.jp

Abstract. Driven by the motivation to enhance emotionally immersive
experience of real-time messaging in 3D virtual world Second Life, we areproposing a conceptually novel approach to reinforcing (intensifying) own
feelings and reproducing (simulating) the emotions felt by the partner throughspecially designed system, iFeel_IM!. In the paper we are describing thedevelopment of novel haptic devices (HaptiHeart, HaptiHug, HaptiTickler,HaptiCooler, and HaptiWarmer) integrated into iFeel_IM! system, whicharchitecture is presented in detail.
Keywords: Affective haptics, affective user interface, wearable devices.

1 IntroductionIn a real world, whenever one person interacts with another, both observe, perceive
and interpret each other’s emotional expressions communicated through a variety ofsignals. Valuable information is also transferred by non-verbal communication (e.g.
social touch). Nowadays, communication in 3D virtual world through instantmessenger and chat is very popular. However, during on-line communication people
are concentrating on textual information and are only slightly affected emotionally.
Conventional mediated systems usually (1) support only simple textual cues likeemoticons; (2) lack visual emotional signals such as facial expressions and gestures;(3) support only manual control of expressiveness of graphical representations ofusers (avatars); and (4) completely ignore such important channel of social
communication as sense of touch.
To deliver emotions through instant messenger, Shin et al. [1] developed a tactileinterface. However, in the proposed methods users have to memorize the vibration or
pin matrix patterns and cognitively interpret the communicated emotional state.
Moreover, such device cannot evoke user’s emotion in a direct way. To our knowledge,
A. Nijholt, D. Reidsma, and H. Hondorp (Eds.): INTETAIN 2009, LNICST 9, pp. 246–251, 2009.
© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2009

Enhancing Mediated Interpersonal Communication through Affective Haptics

247

the only work concentrated on the physical emotion elicitation through physical means
is described in [2].
Driven by the motivation to enhance social interactivity and emotionally
immersive experience of real-time messaging, we are proposing a conceptually novel
approach to reinforcing (intensifying) own feelings and reproducing (simulating) theemotions felt by the partner through specially designed system, iFeel_IM!. The
philosophy behind the iFeel_IM! (intelligent system for Feeling enhancement
powered by affect sensitive Instant Messenger) is “I feel [therefore] I am!”. The
emotion evoked by physical stimulation might imbue our communication withpassion and increase the emotional intimacy, ability to be close, loving, andvulnerable. We argue that interpersonal relationship and the ability to express
empathy grow strongly when people become emotionally closer through disclosing
thoughts, feelings, and emotions for the sake of understanding.
In this work, we focus on implementation of novel devices for generation ofphysical stimulation aimed to convey (and influence on) the emotion experienced
during online conversations.

2 iFeel_IM! Architecture
In the iFeel_IM! system, great importance is placed on the automatic sensing ofemotions conveyed through textual messages in 3D virtual world Second Life, thevisualization of the detected emotions by avatars in virtual environment, andenhancement of user’s affective state and reproduction of feeling of social touch (e.g.,hug) by means of haptic stimulation in a real world. The architecture of the iFeel_IM!
system is presented in Fig. 1.
In order to communicate through iFeel_IM! system, users have to wear innovative
haptic devices (HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, and HaptiWarmer)
developed by us. As a media for communication, we employ Second Life allowing
users to flexibly create their online identities (avatars) and to play various animations(e.g., facial expressions and gestures) of avatars by typing special abbreviations (e.g.,‘/laugh’ for laughing) in a chat window.
To automate emotional behaviour of avatar and to avoid thus manual control byuser, we developed EmoHeart object [3] (invisible in case of ‘neutral’ state) that
listens to each message of avatar, sends it to the Affect Analysis Model [3] located on
the server, gets the result (dominant emotion and intensity for each sentence), andvisually reflects the sensed affective state through avatar facial expression andEmoHeart texture and size. The motivation behind using the heart-shaped object as anadditional channel for visualization was to represent the communicated emotions in avivid and intense way.
In addition to communication with Affect Analysis Model, EmoHeart is responsible for sensing symbolic cues or keywords of ‘hug’ communicative function conveyed by text, and for visualization (triggering related animation) of ‘hugging’ inSecond Life. The results from the Affect Analysis Model and EmoHeart are stored
along with chat messages in a file on local computer of each user. Haptic Devices
Controller analyses these data in a real time and generates control signals for
Digital/Analog converter (D/A), which then feeds Driver Box for haptic devices with248

D. Tsetserukou et al.

control cues. Based on the transmitted signal, the corresponding haptic device(HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, or HaptiWarmer) worn by user isactivated.
AffectAnalysis
Modelchat text
emotion: intensity

3D world Second LifePC
HapticDevicesController

Chatlog
file
Driver Box
D/A

HaptiCooler

HaptiHeart

HaptiHug

HaptiWarmer

HaptiTickler

Fig. 1. Architecture of iFeel_IM! system

3 Affective Haptic Devices
In order to support the affective communication, we implemented several novel haptic
gadgets embedded in iFeel_IM!. They make up three groups. First one is intended foremotion elicitation implicitly (HaptiHeart, HaptiCooler, HaptiWarmer), second typeevokes affect in a direct way (HaptiTickler), and third one uses sense of social touch(HaptiHug) for mood influence. All these devices produce a sense of touch includingEnhancing Mediated Interpersonal Communication through Affective Haptics

249

kinesthetic and coetaneous channels. Kinesthetic stimulations, produced by forces
exerted on body, are sensed by mechanoreceptors in the tendons and muscles. This
channel is highly involved in sensing stimulus produced by HaptiHug device. On theother hand, mechanoreceptors in the skin layers are responsible for cutaneous
stimulation perception. Different types of tactile corpuscles allow us sensing thermal
property of the object (HaptiCooler, HaptiWarmer), pressure (HaptiHeart,HaptiTickler), vibration frequency (HaptiTickler), and stimuli location (localizationof stimulating device position enables association with particular physical contact).
3.1 HaptiHug: The Way of Realistic Hugging over DistanceWhen people are hugging, they generate pressure on the back of each other by thehands, and on the chest area simultaneously. The key feature of the developed
HaptiHug is that it physically reproduces the hug pattern similar to that of humanhuman interaction. The hands for a HaptiHug are sketched from a real human andmade from soft material. The important point is that the hands were designed in sucha way that the user feels as if the friend’s hands actually contact him. Couple ofoppositely rotating motors are incorporated into the holder placed on the user chest
area. Soft Hands, aligned horizontally, contact back of the user. Shoulder strips,
supporting the motor holder and Soft Hands, allow aligning the vertical position ofdevice. Once ‘hug’ command is received, couple of motors tense the belt, pressing
thus Soft Hands and chest part of HaptiHug in the direction of human body (Fig. 2).
The duration and intensity
Soft Hands
Pressureof the hug is controlled by theDirection of belt
software according to thetension
Pressure
detected emoticon or keyword.
The Driver Box regulates thecurrent magnitude in DCmotor from 0 mA to 500 mA.
Human body
For presentation of plain hug
Motorholder
level (e.g., ‘(>^_^)>’, ‘{}’,‘<h>’), big hug level (e.g.,‘>:D<’, ‘{{}}’), and great bigPressure
hug level (e.g.,‘gbh’,Soft Hands
Couple of motors‘{{{}}}’), the pressure of 200Pressure
Direction of belt tension
N/m2 with duration of 2 sec,
the pressure of 300 N/m2 withFig. 2. Structure of wearable HaptiHug device
duration of 3 sec, and thepressure of 450 N/m2 with duration of 4 sec, was applied on the user’s back and chest,respectively.
The technical specification of the device is as follows. DC motors RE 10 1.5 W in
combination with planetary gearhead GP 10 A (gear ratio of 64:1) generate stall
torque of 192.6 mNm. The pressure produced on the human chest by tensed belt
equals 450 N/m2 at the most. Motor holder was manufactured on 3D printerDimension DS 768 from ABS plastic material. Special slots were provided to makethe pass for pressing belt.

250D. Tsetserukou et al.

Preliminary, Soft Hands
were produced on 3D printerBelt slots
from ABS plastic material
with thickness of 1 mm.
However, plate flexibility was
not enough to achieve touch
Soft Hand
sensation similar to human
Belt
Soft Hand
hand. Therefore, we decided
to make hands from compliant
Fig. 3. Left: Soft Hand dimensions. Right: sandwichedrubber-sponge material. The
structure of Soft Hands.
contour profile of Soft Hand
is sketched from the malehuman and has front-face area of 155.6 cm2. Two identical pieces of Soft Hand of 5mm thickness were sandwiched by narrow belt slots and connected by plastic screws.
Such structure provides enough flexibility to tightly fit to the human back surface
while being pressed by belt. Moreover, belt can loosely move inside the Soft Hands
during tension. The dimensions and structure of Soft Hands are presented in Fig. 3.
HaptiHug device has lightweight tri-glide bucklers and side release fastenerintegrated into the motor holder to facilitate easy adjustment of the belt size to her/his
body sizes and detaching it in a natural and rapid manner.
Cover fabric3.2 HaptiHeart, HaptiCooler, HaptiWarmer, HaptiTickler. Or How We Can
Enhance and Influence on Our Emotions by Haptics
We selected four distinct emotions with strong and unique physiological patterns:‘anger’, ‘fear’, ‘sadness’, and ‘joy’ [4].
There is no doubt that feelings are intuitively connected with the heart, and ourlexicon confirms this. The research on interplay between heart rate and emotions
revealed that different emotions are associated with distinctive patterns of heart ratevariations. We developed heart imitator HaptiHeart to produce special heartbeat
patterns according to emotion to be conveyed or elicited (sad is associated with slightly
intense heartbeat, anger with quick and violent heartbeat, fear with intense heart rate).
We take advantage of the fact that ourheart naturally synchronizes with theheart of a person we hold or hug. Thus,
the heart rate of a user is influenced byhaptic perception of the beat rate of theHaptiHeart.
The HaptiHeart consists of twomodules: flat speaker FPS 0304 andspeaker holder. The flat speaker sizes(66.5 x 107 x 8 mm) and rated input
power of 10 W allowed us to design
Speaker Case
powerful and relatively compact
Flat Speaker
HaptiHeart device producing realistic
heartbeating sensation with high fidelity.
Fig. 4. HaptiHeart layout
The 3D model is presented in Fig. 4.

Enhancing Mediated Interpersonal Communication through Affective Haptics

251

We recorded the heartbeat patterns for the cases of experiencing fear, anger, andsadness. The pre-recorded sound signal with low frequency generates the pressure on
the human chest through vibration of the speaker surface.
HaptiTickler is responsible for joy emotion evocation. The idea behind this device
is to reproduce effect of “Butterflies in the stomach” (fluttery or tickling feeling in thestomach felt by people experiencing love) by means of circular arrays of vibration
motors attached to user abdomen area. HaptiWarmer is intended for the rise of skintemperature to evoke either aggression or pleasant feeling (hot or warm, respectively).
In order to boost fear emotion physically, we designed HaptiCooler interface thatsends “Shivers down/up human body’s spine” by means of a row of vibration motors,
and “Chills down/up human body’s spine” through both cold airflow from DC fan andcold side of Peltier element.

4 Conclusion
In a nutshell, while developing the iFeel_IM! system, we attempted to bridge the gap
between mediated and face-to-face communications by enabling and enriching thespectrum of senses such as vision and touch along with cognition and inner personalstate.
In the paper we described the architecture of iFeel_IM and development of novel
haptic devices, such as HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, andHaptiWarmer. All devices were designed with particular emphasis on ergonomic
design. The user can comfortably wear them and easily detach from torso. We believethat iFeel_IM! will encourage users to get in “touch” with their emotions and to makesocial contact with other people in on-line communication.
Acknowledgments. The research is supported in part by a Japan Society for thePromotion of Science (JSPS) Postdoctoral Fellowship for Foreign Scholars.

References
1. Shin, H., Lee, J., Park, J., Kim, Y., Oh, H., Lee, T.: A Tactile Emotional Interface for
Instant Messenger Chart. In: Proceedings of Symposium on Human Interface (HCIInternational 2007), pp. 166–175 (2007)
2. Kok, R., Broekens, J.: Physical Emotion Induction and Its Use in Entertainment: Lessons
Learned. In: Proceedings of First IFIP Entertainment Computing Symposium, pp. 33–48(2008)
3. Neviarouskaya, A., Prendinger, H., Ishizuka, M.: EmoHeart: Automation of Expressive
Communication of Emotions in Second Life. In: Proceedings of International Conferenceon Human-Computer Interaction (2009)
4. Wallbott, H.G., Scherer, K.R.: How Universal and Specific is Emotional Experience? In:Scherer, K.R. (ed.) Evidence from 27 Countries on Five Continents. Facets of Emotion:Recent Research, pp. 31–56. Lawrence Erlbaum Inc., Hillsdale (1988)