Affective Haptics in Emotional Communication
Dzmitry
Tsetserukou
University ofTokyo, Japan

Alena
Neviarouskaya
University ofTokyo, Japan

dima_teterukov@ipc.i.u-tokyo.ac.jp

lena@mi.ci.i.u-tokyo.ac.jp

HelmutPrendinger
National Instituteof Informatics,
Japan
helmut@nii.ac.jp

Abstract
In the paper we are proposing a conceptually novel
approach to reinforcing (intensifying) own feelings andreproducing (simulating) the emotions felt by thepartner during online communication through speciallydesigned system, iFeel_IM!. The core component, AffectAnalysis Model, automatically recognizes nine emotions
from text. The detected emotion is stimulated byinnovative haptic devices integrated into iFeel_IM!. The
implemented system can considerably enhanceemotionally immersive experience of real-time
messaging.

1. Introduction“And now here is my secret, a very simple secret; it is
only with the heart that one can see rightly, what isessential is invisible to the eye.”
Antoine de Saint-Exupery
Everything we know about the world entered our
minds through the senses of sight, hearing, test, touch,and smell. Our feelings are rich and continuous flow ofchanging percepts. All our senses play significant role
in recognition of emotional states of communication
partner. Human emotions can be easily evoked bydifferent cues, and sense of touch is one of the mostemotionally charged channels.
Affective Haptics is the emerging area of researchwhich focuses on the design of devices and systems thatcan elicit, enhance, or influence on emotional state of ahuman by means of sense of touch. We distinguish fourbasic haptic (tactile) channels governing our emotions:(1) physiological changes (e.g., heart beat rate, bodytemperature, etc.), (2) physical stimulation (e.g.,tickling), (3) social touch (e.g., hug), (4) emotional
haptic design (e.g., shape of device, material, texture).
Driven by the motivation to enhance social
interactivity and emotionally immersive experience ofreal-time messaging, we pioneered in the idea ofreinforcing (intensifying) own feelings and reproducing(simulating) the emotions felt by the partner throughspecially designed system, iFeel_IM!. The philosophy

978-1-4244-4799-2/09/$25.00 ©2009 IEEE

Naoki
Kawakami
University ofTokyo, Japan

Susumu
Tachi
Keio University,Japankawakami@star.t.u-tokyo.ac.jp

http://tachilab.org/behind the iFeel_IM! (intelligent system for Feelingenhancement powered by affect sensitive InstantMessenger) is “I feel [therefore] I am!”. The emotion
elicited by physical stimulation might imbue ourcommunication with passion and increase the emotional
intimacy, ability to be close, loving, and vulnerable. Theinterpersonal relationships and the ability to express
empathy grow strongly when people become
emotionally closer through disclosing thoughts, feelings,
and emotions for the sake of understanding.
In this work, we focus on the implementation of an
innovative system, which includes haptic devices forgeneration of physical stimulations aimed at conveying
the emotions experienced during online conversations.
We attempt to influence on human emotions byphysiological changes, physical stimulation, social
touch, and emotional design.

2. Architecture of the iFeel_IM! system
In the iFeel_IM! system, great importance is placedon the automatic sensing of emotions conveyed throughtextual messages in 3D virtual world Second Life, thevisualization of the detected emotions by avatars in
virtual environment, enhancement of user’s affective
state, and reproduction of feeling of social touch (e.g.,hug) by means of haptic stimulation in a real world. The
architecture of the iFeel_IM! is presented in Fig. 1.
In order to communicate through iFeel_IM! system,
users have to wear innovative affective haptic devices(HaptiHeart, HaptiHug, HaptiButterfly, HaptiTickler,
HaptiTemper, and HaptiShiver) developed by us. As amedia for communication, we employ Second Life,
which allows users to flexibly create their online
identities (avatars) and to play various animations (e.g.,
facial expressions and gestures) of avatars by typing
special abbreviations (e.g., ‘/laugh’ for laughing) in achat window.
The control of the conversation is implemented
through the Second Life object called EmoHeart(invisible in case of ‘neutral’ state) attached to theavatar’s chest. Once attached to the avatar, EmoHeartobject (1) listens to each message of its owner, (2) sends
it to the web-based interface of the Affect Analysis
Model (AAM) [1] located on the server, (3) receives theresult (dominant emotion and intensity), and visuallyAffect
AnalysisModelchat text
emotion: intensity

3D world Second Life
PC
HapticDevices
Controller

Chat
log
file
D/A

Driver Box

HaptiHeart

HaptiTemper

HaptiHugandHaptiShiver

HaptiTickler

HaptiBatterfly

Figure 1: Architecture of iFeel_IM! system.

reflects the sensed affective state through the animationof avatar’s facial expression, EmoHeart texture(indicating the type of emotion), and size of the texture(indicating the strength of emotion, namely, ‘low’,‘middle’, or ‘high’). If no emotion is detected in thetext, the EmoHeart remains invisible and the avatar
facial expression remains neutral. The motivation
behind using the heart-shaped object as an additional
channel for visualization was to represent thecommunicated emotions in a vivid and expressive way.
In addition to communication with Affect Analysis
Model, EmoHeart is responsible for sensing symboliccues or keywords of ‘hug’ communicative function
conveyed by text, and for visualization (triggering
related animation) of ‘hugging’ in Second Life. The
results from the Affect Analysis Model and EmoHeartare stored along with chat messages in a file on local
computer of each user. Haptic Devices Controller
analyses these data in a real time and generates control
signals for Digital/Analog converter (D/A), which thenfeeds Driver Box for haptic devices with control cues.
Based on the transmitted signal, the corresponding
haptic device (HaptiHeart, HaptiHug, HaptiButterfly,
HaptiTickler, HaptiTemper, and HaptiShiver) worn byuser is activated.

3. Recognition of affect from text messages
The Affect Analysis Model senses nine emotions
conveyed through text (‘anger’, ‘disgust’, ‘fear’, ‘guilt’,‘interest’, ‘joy’, ‘sadness’, ‘shame’, and ‘surprise’). The
affect recognition algorithm, which takes into accountspecific style and evolving language of online
conversation, consists of five main stages: (1) symboliccue analysis; (2) syntactical structure analysis; (3) wordlevel analysis; (4) phrase-level analysis; and (5)sentence-level analysis. Analyzing each sentence insequential stages, this method is capable of processing
sentences of different complexity, including simple,
compound, complex (with complement and relative
clauses), and complex-compound sentences. The
working flow of the algorithm is presented in Fig. 2.
An empirical evaluation of the AAM algorithm
showed promising results regarding its capability toaccurately classify affective information in text from anexisting corpus of informal online communication.
AAM with Connexor Machinese Syntax achieves
accuracy in 72.6% on sentences, on which two or threesentence
Symbolic cue analysis module
Test for emoticons, abbreviations,acronyms, interjections, “?” and“!” marks, repeated punctuationand capital letters

AffectDatabase
Syntactical structure
analysis module
Connexor Parser

Phrase-level
analysis module

Parser output
processing

Sentence-level
analysis module

Sentence annotated byemotion stateemotion category: intensity

Emoticon or em. abbr.
“yes”
“no”
Estimation of
resultingemotion state

Sentence preprocessing forparseremotion category: intensity

Word-level
analysis module

Figure 2: Working flow of the Affect Analysis Model.

human raters completely agreed, and 81.5% on
sentences, on which all three human raters completely
agreed.

4. Affective haptic devices
According to James-Lange theory [2], the conscious
experience of emotion occurs after the cortex receives
signals about changes in physiological state.
Researchers argued that feelings are preceded by certain
physiological changes. Thus, when we see a venomous
snake, we feel fear, because our cortex has received
signals about our racing heart, knocking knees, etc.
Damasio [3] distinguishes primary and secondary
emotions. Both involve changes in bodily states, but thesecondary emotions are evoked by thoughts. Recent
empirical studies support non-cognitive theories ofnature of emotions. It was proven that we can easily
evoke our emotions by something as simple as changing
facial expression (e.g., smile brings on a feeling of
happiness) [4]. Moreover, it is believed that some of ouremotion responses are mediated by direct pathways
from perceptual centers in temporal cortex and thethalamus to the amygdala [5].
In order to support the affective communication, we
implemented several novel haptic gadgets embedded iniFeel_IM!. They make up three groups. First group isintended for emotion elicitation implicitly (HaptiHeart,
HaptiButterfly, HaptiTemper, and HaptiShiver), second
type evokes affect in a direct way (HaptiTickler), andthird one uses sense of social touch (HaptiHug) for
influencing on the mood and providing some sense ofphysical co-presence. All these devices produce
different senses of touch including kinesthetic andcoetaneous channels. Kinesthetic stimulations, which
are produced by forces exerted on the body, are sensed
by mechanoreceptors in the tendons and muscles. This
channel is highly involved in sensing stimulus produced
by HaptiHug device. On the other hand,mechanoreceptors in the skin layers are responsible forthe perception of cutaneous stimulation. Different typesof tactile corpuscles allow us sensing thermal propertyof the object (HaptiTemper), pressure (HaptiHeart,
HaptiHug), vibration frequency (HaptiButterfly,
HaptiTickler, and HaptiShiver), and stimuli location(localization of stimulating device enables association
with particular physical contact).
Front side

Back side

HaptiHeart
HaptiHug

HaptiShiver

HaptiButterfly
HaptiTickler

HaptiTemper

Figure 3: Affective haptic devices worn on a human body.

The affective haptic devices worn on a human bodyare presented in Fig. 3.

4.1. HaptiHug: realistic hugging over distanceOn-line interactions rely on senses of vision andhearing, and there is a substantial need in mediatedsocial touch [6]. Among many forms of physicalcontact, hug is the most emotionally charged one. It
conveys warmth, love, and affiliation. DiSalvo et al. [7]introduced “The Hug” interface. When person desires tocommunicate hug, he/she can squeeze the pillow, so thatsuch action results in the vibration and temperaturechanges in the partner’s device. However, this interface

suffers from inability to resemble natural hug sensation
and, hence, to elicit strong affective experience (onlyslight pressure is generated by vibration actuators); andlacks the visual representation of the partner, whichadds ambiguity (hugging in a real life involves bothvisual and physical experience).
When people are hugging, they generate pressure on
the chest area and on the back of each other by thehands, simultaneously. The key feature of the developed
HaptiHug is that it physically reproduces the hug patternsimilar to that of human-human interaction. The hands
for a HaptiHug are sketched from a real human andmade from soft material so that hugging partners canrealistically feel social presence of each other.
The couple of oppositely rotating motors areincorporated into the holder placed on the user chest
area. The Soft Hands, which are aligned horizontally,
contact back of the user. Once ‘hug’ command isreceived, couple of motors tense the belt, pressing thus
Soft Hands and chest part of the HaptiHug in the
direction of human body (Fig. 4).
Direction ofbelt tension

Cover fabricBelt slots

Soft Hand
Soft Hand

Belt

Figure 5: Left: Soft Hand dimensions.
Right: sandwiched structure of Soft Hands.

The significance of our idea to realistically reproduce
hugging is in integration of active-haptic device
HaptiHug and pseudo-haptic touch simulated byhugging animation (see still image of hugging animationin Fig. 1). Thus, high immersion into the physicalcontact of partners while hugging is achieved.

4.2. HaptiHeart enhancing our emotions

Soft Hand
Pressure
Soft Hand

Human bodyMotorholderPressure
Couple of motors
Pressureloosely move inside the Soft Hands during tension. The
dimensions and structure of Soft Hands are presented inFig. 5.

Direction of belt tension

Figure 4: Structure of wearable HaptiHug device.

The duration and intensity of the hug are controlled
by the software in accordance with the emoticon or a
keyword, detected from text. For the presentation of aplain hug level (e.g., ‘(>^_^)>’, ‘{}’, ‘<h>’), a big hug
level (e.g., ‘>:D<’, ‘{{}}’), and a great big hug level(e.g., ‘gbh’, ‘{{{}}}’), the pressure of 200 N/m2 withduration of 2 sec, the pressure of 300 N/m2 withduration of 3 sec, and the pressure of 450 N/m2 withduration of 4 sec, was applied on the user’s back andchest, respectively.
The Soft Hands are made from the compliant rubbersponge material. The contour profile of a Soft Hand is
sketched from the male human and has front-face areaof 155.6 cm2. Two identical pieces of Soft Hand of 5mm thickness were sandwiched by narrow belt slots andconnected by plastic screws. Such structure provides
enough flexibility to tightly fit to the human backsurface, while being pressed by belt. Moreover, belt can

Each emotion is characterized by a specific pattern ofphysiological changes. We selected four distinct
emotions having strong physical features [8]: ‘anger’,‘fear’, ‘sadness’, and ‘joy’. The precision of AAM inrecognition of these emotions is considerably higher(‘anger’ – 92 %, ‘fear’ – 91 %, ‘joy’ – 95 %, ‘sadness’ –88 %) than of other emotions.
Of the bodily organs, the heart plays a particularly
important role in our emotional experience. The abilityof false heart rate feedback to change our emotional
state was reported in [9]. The research on interplay
between heart rate and emotions revealed that different
emotions are associated with distinct patterns of heart
rate variations [10].
We developed heart imitator HaptiHeart to produce
special heartbeat patterns according to emotion to beconveyed or elicited (sadness is associated with slightly
intense heartbeat, anger with quick and violent
heartbeat, fear with intense heart rate). We takeadvantage of the fact that our heart naturally
synchronizes with the heart of a person we hold or hug.
Thus, the heart rate of a user is influenced by haptic
perception of the beat rate of the HaptiHeart.
Furthermore, false heart beat feedback can be directlyinterpreted as a real heart beat, so it can change theemotional perception. The HaptiHeart consists of two
modules: flat speaker FPS 0304 and speaker holder. The
flat speaker sizes (66.5 x 107 x 8 mm) and rated input
power of 10 W allowed us to design powerful andrelatively compact HaptiHeart device. It is able toproduce realistic heartbeating sensation with highfidelity. The 3D model is presented in Fig. 6.
The pre-recorded sound signal with low frequency
generates the pressure on the human chest throughvibration of the speaker surface.

Peltier
element

DC FanVibration
motor

Back view

Front view

Flat Speaker

Heart-shaped
Speaker Case

Figure 6: HaptiHeart layout.

4.3. Butterflies in the stomach and shivers on
the body’s spine
HaptiButterfly is responsible for the evocation of joy
emotion (Fig. 7). The idea behind this device is toreproduce effect of “Butterflies in the stomach” (fluttery
or tickling feeling in the stomach felt by people
experiencing love) by means of the arrays of vibration
motors attached to the abdomen area of a person.
Plastic frame
Revolute joint
Vibration
motor

HolderFigure 7: Structure of HaptiButterfly.

We conducted the experiment aimed at investigationof the patterns of vibration motor activation that
produce most pleasurable and natural sensations on theabdomen area. Based on the results, we employ‘circular’ and ‘spiral’ vibration patterns in
HaptiButterfly.
The temperature symptoms are great indicators ofdifferences between emotions. The empirical studies [8]showed that (1) fear and, to a lesser degree, sadness, arecharacterized as ‘cold’ emotions, (2) joy is the only
emotion experienced as being ‘warm’, while (3) anger is‘hot’ emotion.
In order to boost fear emotion physically, we
designed HaptiShiver interface that sends “Shivers
down/up human body’s spine” by means of a row ofvibration motors (HaptiShiver), and “Chills down/up
human body’s spine” through both cold airflow from
DC fan and cold side of Peltier element (HaptiTemper).
The structure of HaptiShiver/HaptiTemper device isshown in Fig. 8.

Aluminum
plate

Figure 8: Structure of HaptiShiver/HaptiTemper.

HaptiTemper is also intended for simulation of
warmth on the human skin to evoke either pleasantfeeling or aggression (it was proved that uncomfortably
hot temperatures arouse anger feelings [11]).

4.4. HaptiTickler: device for positive emotions
Two different types of tickling are recognized. The
first type is knismesis referring to feather-like (light)
type of tickling. It is elicited by a light touch and isgenerally accompanied by an itching sensation. The
second type of tickle called gargalesis is evoked by aheavier touch to particular areas of the body such asarmpits or ribs. Such kind of stimuli usually results inlaugher and squirming. In contract to knismesis, one
cannot produce gargalesis in oneself. Two explanations
were suggested to explain the reasons of inability to
self-tickling. The scientists supporting interpersonal
explanations argue that tickling is fundamentallyinterpersonal experience and thus requires anotherperson as the source of the touch [12]. On the other sideof the debate is a reflex view, suggesting that tickle
requires the element of unpredictability or
uncontrollability. The experimental results from [13]support the later view and reveal that ticklish laugher
evidently does not require that stimulation be attributed
to another person. However, the social and emotional
factors in ticklishness affect the tickle response greatly.
We developed HaptiTickler with the purpose to
evoke positive affect (joy emotion) in a direct way bytickling the ribs of the user. The device includes four
vibration motors reproducing stimuli that are similar tohuman finger movements during rib tickling (Fig. 9).
Large vibration
motor (index finger)
Small vibration
motors (middle andring finger)
Large vibration
motor (little finger)

Figure 9: HaptiTickler device.

The uniqueness of our approach is in (1) combination

of the unpredictability and uncontrollability of thetickling sensation through random activation of stimuli,(2) high involvement of the social and emotional factors
in the process of tickling (positively charged on-line
conversation potentiates the tickle response).

4.5. Emotional haptic design
Aesthetically pleasing objects appear to the user to be
more effective by virtue of their sensual appeal [14].
The affinity the user feels for an object is resulted fromthe formation of an emotional connection with theobject. Recent findings show that attractive things makepeople feel good, which in turn makes them think morecreative. We propose the concept of Emotional Haptic
Design. The core idea is to make user to feel affinity forthe device by means of (1) appealing shapes evoking thedesire to touch and haptically explore them, (2) usage ofmaterial pleasurable to touch, and (3) the pleasure
anticipated through wearing.
The designed devices are pleasurable to look at and totouch (colorful velvet material was used to decorate thedevices) and have personalized features (in particular,
the Soft Hands in HaptiHug can be sketched from thehands of the real communication partner). The essenceof emotional, moral, and spiritual aspects of a humanbeing has long been depicted using heart-shaped
symbol. The heart-shaped HaptiHeart was designed withprimary objective to emotionally connect the user withthe device, as heart is mainly associated with love andemotional experience. The HaptiButterfly, its shape, andactivated vibration motors induce the association with areal butterfly lightly touching the human body by
spreading its wings.
We placed great attention on the comfortable wearingof garment. Such devices as HaptiButterfly,
HaptiTickler, and HaptiShiver have the inner sides
made from foam-rubber. While contacting the human
body, surface shape is self-adjusted to fit particular
contour of the human body, and any uncomfortablepressure is therefore avoided. All of the designed
devices have flexible and intuitive in use system ofbucklers and fasteners to enable user to easily adjust thedevices to the body shape.

5. Conclusions
While developing the iFeel_IM! system, we
attempted to bridge the gap between mediated and faceto-face communications by enabling and enriching thespectrum of senses such as vision and touch along withcognition and inner personal state. In the paper we
described the architecture of the iFeel_IM! and thedevelopment of novel haptic devices, such as
HaptiHeart, HaptiHug, HaptiTickler, HaptiButterfly,
HaptiShiver, and HaptiTemper. These devices were
designed with particular emphasis on natural andrealistic representation of the physical stimuli, modular
expandability, and ergonomic human-friendly design.

User can perceive the intensive emotions during online
communication, use desirable type of stimuli,comfortably wear and easily detach devices from torso.
The preliminary user study has revealed thatdeveloped devices are capable of influencing on ouremotional state intensively. Users were captivated bychatting and simultaneously experiencing emotionalarousal caused by affective haptic devices. Our primary
goal for the future research is to conduct extensive user
study on iFeel_IM! system.
We believe that iFeel_IM! will encourage users to getin “touch” with their emotions and to make social
contact with other people in on-line communicationricher and more enjoyable.

6. References[1] A. Neviarouskaya, H. Prendinger, M. Ishizuka.
Compositionality principle in recognition of fine-grained
emotions from text. Proceedings of the 3rd International
Conference on Weblogs and Social Media, pp. 278-281,
2009.
[2] W. James. What is an emotion? Mind. 9, 188–205, 1884.
[3] A. Damasio. The feeling of what happens: body, emotion
and the making of consciousness. London: Vintage, 2000.
[4] R.B. Zajonc, S.T. Murphy, M. Inglehart. Feeling andfacial efference: implication of the vascular theory of
emotion. Psychological review, 96:395–416, 1989.
[5] J.E. LeDoux. The emotional brain. New York: Symon &Shuster, 1996.
[6] A. Haans, W. I. Ijsselsteijn. Mediated social touch: areview of current research and future directions. Virtual
Reality, 9:149–159, 2006.
[7] C. DiSalvo, F. Gemperle, J. Forlizzi, E. Montgomery.
The hug: an exploration of robotic form for intimate
communication. Proceedings of the 12th IEEE
International Workshop on Robot and Human Interactive
Communication, pp. 403–408, 2003.
[8] H.G. Wallbott, K.R. Scherer. How universal and specificis emotional experience? Evidence from 27 countries onfive continents. In K.R. Scherer (Eds.), Facets ofEmotion: Recent Research. Hillsdale (N.J.): Lawrence
Erlbaum Inc., pp. 31–56, 1988.
[9] M.D. Decaria, S. Proctor, T.E. Malloy. The effect of false
heart rate feedback on self reports of anxiety and on
actual heart rate. Behavior research & Therapy, 12:251–
253, 1974.
[10] J. Anttonen, V. Surakka. Emotions and heart rate while
sitting on a chair. Proceedings of the 27th SIGCHI
Conference on Human Factors in Computing Systems,
pp. 491–499, 2005.
[11] C.A. Anderson. Temperature and aggression: ubiquitouseffects of heat on occurrence of human violence.
Psychological Bulletin, 106 (1):74–96, 1989.
[12] H.C. Foot, A.J. Chapman. The social responsiveness ofyoung children in humorous situation. In A.J. Chapman& H.C. Foot (Eds.), Humour and Laugher: Theory,
research, and applications. London: Wiley, pp. 187–214,
1976.
[13] C.R. Harris, N. Christenfeld. Can a machine tickle?
Psychonomic Bulletin & Review, 6(3):504–510, 1999.
[14] D.A. Norman. Emotional design. Why we love (or hate)everyday things. New-York: Basic Book, 2004.