iFeel_IM! Emotion Enhancing Garment forCommunication in Affect Sensitive Instant Messenger
Dzmitry Tsetserukou1, Alena Neviarouskaya2, Helmut Prendinger3,
Naoki Kawakami1, Mitsuru Ishizuka2, and Susumu Tachi11Department of Information Physics and Computing, University of Tokyo, Japan
dima_teterukov@ipc.i.u-tokyo.ac.jp,{kawakami,tachi}@star.t.u-tokyo.ac.jp
2
Department of Information and Communication Engineering, University of Tokyo, Japanlena@mi.ci.i.u-tokyo.ac.jp, ishizuka@i.u-tokyo.ac.jp
3
National Institute of Informatics, Japan
helmut@nii.ac.jp

Abstract. In the paper we are proposing a conceptually novel approach to reinforcing (intensifying) own feelings and reproducing (simulating) the emotions
felt by the partner during online communication through specially designed system, iFeel_IM!. The core component, Affect Analysis Model, automaticallysense nine emotions from text. The detected emotion is stimulated by innovative haptic devices (HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, andHaptiWarmer) integrated into iFeel_IM! The implemented system can considerably enhance emotionally immersive experience of real-time messaging.
Keywords: Affective haptics, affective user interface, wearable devices.

1 Introduction“All emotions use the body as their theater…”
Antonio Damasio [1]Pleasure, euphoria, sadness, fear, anger – these and other emotions color our lives,enrich our experiences, and imbue our actions with passion and character. Researchon emotion in both psychology and neuroscience indicates that emotions also play apivotal role in social interaction, decision-making, perception, attention, memory,learning, etc. Affective Computing [2], which aims to realize emotionally intelligent
human-computer interaction, is a well-established research area that deals with investigation of the role of emotions in technology and poses a question: “How can we
make computer to be aware of emotions?” While conducting our research in an
emerging frontier of Affective Haptics, we are attempting to answer another question:“How to elicit (or enhance) the user’s affective state in human-human mediatedcommunication by haptic technologies?”
Besides emotions conveyed through text, researchers developed an additional
modality for communicating emotions in Instant Messenger (IM) through tactileinterfaces with vibration patterns [3,4,5,6]. However, in the proposed methods users
M.J. Smith and G. Salvendy (Eds.): Human Interface, Part I, HCII 2009, LNCS 5617, pp. 628–637, 2009.
© Springer-Verlag Berlin Heidelberg 2009

iFeel_IM! Emotion Enhancing Garment for Communication

629

have to memorize the vibration or pin matrix patterns and cognitively interpret thecommunicated emotional state. Demodulation of haptically coded emotion is notnatural for human-human communication, and direct evocation of emotion cannot be
achieved in such system.
Driven by the motivation to enhance social interactivity and emotionally immersive experience of real-time messaging, we are proposing a conceptually novel approach to reinforcing (intensifying) own feelings and reproducing (simulating) theemotions felt by the partner through specially designed system, iFeel_IM!. The philosophy behind the iFeel_IM! (intelligent system for Feeling enhancement poweredby affect sensitive Instant Messenger) is “I feel [therefore] I am!”. The emotion
evoked by physical stimulation might imbue our communication with passion andincrease the emotional intimacy, ability to be close, loving, and vulnerable. We argue
that interpersonal relationship and the ability to express empathy grow strongly whenpeople become emotionally closer through disclosing thoughts, feelings and emotions
for the sake of understanding.
In this work, we focus on implementation of innovative system and devices forgeneration of physical stimulation aimed to convey the emotion experienced during
online conversations. We attempt to influence on human emotions by physiologicalchanges, physical stimulation, and social touch.
1.1 Physiological Changes Evoke Human Emotions
Our emotional states include two components: physical sensations (heart rate, respiration, etc.) and conscious feelings (joy, anger, etc.). Emotional states are mediated by afamily of peripheral, autonomic, endocrine, and skeletomotor responses. When frightened, we not only feel afraid but also experience increased heart rate and respiration,
dryness of the mouth, tense muscles, and sweaty palms [7].
According to James-Lange theory [8], the conscious experience of emotion occurs
after the cortex receives signals about changes in physiological state. Researchers
argued that feelings are preceded by certain physiological changes. Thus, when we
see a venomous snake, we feel afraid because our cortex has received signals about
our racing heart, knocking knees, etc. In [9], Prinz revived the James-Lange theory,
supported it with reference to recent empirical research, and thoroughly criticized
cognitive theories of emotions for downplaying the role of the body.
Among a variety of physiological changes accompanying the experience ofemotion (heart rate, temperature, respiration, pulse, blood pressure, muscle action
potentials), for our research we selected those (heart rate and temperature) that can beaffected externally in a safe manner. The ability of false heart rate feedback to change
our emotional state was reported in [10]. Briese [11] found a tight relationship between emotions and body temperature.
To our knowledge, the only work concentrating on the emotion induction throughphysical means is described in [12]. The interactive chair changes emotional state of
the subject watching the movie with rich emotional content. However, user study did
not show any significant effect of the chair on intensity of experienced emotion. The
reported reasons are: (1) weakness of the technical solutions (in many cases users did
not perceive any physical influence, and complained on interruption effect on themovie watching); (2) user-study procedure was not well elaborated.

630

D. Tsetserukou et al.

1.2 Mediated Social Touch and Haptic Communication of Emotions
In a real world, whenever one person interacts with another, both observe, perceive
and interpret each other’s emotional expressions communicated through a variety ofsignals. Valuable information is also transferred by non-verbal communication (e.g.
social touch). It is well known that touching is one of the most powerful means forestablishing and maintaining social contact. The fact that two people are willing totouch implies an element of trust [13]. Expressive potential of touch is the ability toconvey and elicit strong emotions. Although current on-line interaction mainly relies
on senses of vision and hearing, there is a substantial need in mediated social touch.
Comprehensive review of research on social touch is presented in [14].
Among many forms of physical contact, the most emotionally charged one is hug.
It conveys warmth, love, and affiliation. The Hug Shirt allows people who are missing each other to send physical sensation of the hug over distance [15]. User can wear
this shirt, embedded with actuators and sensors, in everyday life. The communicationis performed through cell phone facilitated with Bluetooth and Java software. DiSalvo
et al. [16] devised “The Hug” interface. When person desires to communicate hug,
he/she squeezes the pillow that results in vibration and temperature changes in thepartner’s device.
Even though such devices can potentially increase interactivity, they have suchdrawbacks as: (1) the functionality is highly limited and is not extendable (only hug
pattern can be transmitted); (2) inconvenient design (users found them difficult to use
in real life); (3) inability to resemble natural hug sensation and, hence, elicit strong
affective experience (only slight pressure is generated by vibration actuators); (4) lackof visual representation of the partner.
We believe that it is possible to highly increase the emotional immersion throughthe iFeel_IM! system featuring automatic affect recognition from text, visualization,and physical stimulation feedback.

2 iFeel_IM! Architecture
Nowadays, communication through IM and chat is very popular. However, during online communication people are concentrating on textual information and are only
slightly affected emotionally. Conventional mediated systems usually (1) support only
simple textual cues like emoticons; (2) lack visual emotional signals such as facial
expressions and gestures; (3) support only manual control of expressiveness ofgraphical representations of users (avatars); (4) completely ignore such importantchannel of social communication as sense of touch.
In the iFeel_IM! system, great importance is placed on the automatic sensing of
emotions conveyed through textual messages, visual reflection of the detected emotions and communicative behaviour through a 2D cartoon-like avatars, enhancementof user’s affective state, and reproduction of feeling of social touch (e.g., hug) bymeans of haptic stimulation in a real world. The architecture of the iFeel_IM! system
is presented in Fig. 1.

iFeel_IM! Emotion Enhancing Garment for Communication

631

Fig. 1. Architecture of iFeel_IM! system

In order to communicate through iFeel_IM! system, users have to wear innovative
haptic devices (HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, and HaptiWarmer)
developed by us.
As a media for communication, we developed AffectIM [17] instant messenger.
AffectIM receives the typed utterance, sends it to the Affect Analysis Model [18]responsible for recognition of emotions conveyed through text and located on theserver, gets the result (dominant emotion and intensity for each sentence), and visually reflects the sensed affective state through avatar facial expression and gestures.
The results from the Affect Analysis Model are stored along with chat messages ina file on local computer of each user. Haptic Device Controller analyses these data ina real time and generates control signals for Digital/Analog converter (D/A), whichthen feeds Driver Box for haptic devices with control cues. Based on the transmitted
signal, the corresponding haptic device (HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, or HaptiWarmer) worn by user is activated.

3 AffectIM: Intelligent Affect Sensitive Instant Messenger
AffectIM, supported by the Affect Analysis Model, was developed as a web-based
application running in the Internet browser.

632

D. Tsetserukou et al.

For affect categorization, we have decided to use the subset of emotional states defined by Izard [19]: ‘anger’, ‘disgust’, ‘fear’, ‘guilt’, ‘interest’, ‘joy’, ‘sadness’,‘shame’, and ‘surprise’. While constructing our Affect Analysis Model we took into
account crucial aspects of informal online conversation such as its specific style andevolving language. The affect sensing algorithm consists of five main stages: (1)symbolic cue analysis; (2) syntactical structure analysis; (3) word-level analysis; (4)phrase-level analysis; and (5) sentence-level analysis. Analyzing each sentence insequential stages, our method is capable of processing sentences of different complexity, including simple, compound, complex (with complement and relative clauses),and complex-compound sentences (details are given in [18]). The working flow of theAffect Analysis Model is presented in Fig. 2.

Fig. 2. Working flow of the Affect Analysis ModelAn empirical evaluation of the Affect Analysis Model algorithm showed promising
results regarding its capability to accurately classify affective information in text froman existing corpus of informal online communication. In a study based on blog entries, the system result agreed with at least two out of three human annotators in 70%of the cases.
The example of conversation in AffectIM (with emotion annotations from AffectAnalysis Model) is given below:
A: Hi, good to see you there! [JOY]
B: Hi, it is great that AffectIM eliminates spacial distance and connects us. [SURPRISE]
A: Right {} [HUG]
B: Oh, I really feel the warmth of your hug!!
A: Yes, it's so exciting, as if we met in a real world. [INTEREST]
B: How is your study at the University of Tokyo going on?
A: I have successfully graduated last month. [JOY]
B: Congratulations, I am very proud of you! [JOY]
A: My mind is now filled with pleasant memories about student life, but at the same time I
regret this period, full of fun, flew away so fast. [GUILT]

iFeel_IM! Emotion Enhancing Garment for Communication

633

B: You certainly deserve rest.
A: Former students of our Lab have invited me to the trip to Ski resort.
B: I remember the first time I tried to balance on a snowboard was definitely a very thrilling
experience! Have you ever been descending a steep slope on a snowboard? [FEAR]
A: I always feel angry at myself for not having enough courage for snowboarding. Shamefully, I prefer safe skiing. [ANGER, SHAME]
B: Sure, extreme sport needs thorough training. My first attempt resulted in a broken leg. But
now my experience and protective gear allow me to enjoy snowboarding. [SADNESS, JOY]
A: For me, risky snowboarding might be safe only in a virtual world. :) [JOY]

The twenty-user study conducted on AffectIM [17] showed that the IM systemwith automatic emotion recognition function was successful at conveying users’ emotional states, thus enriching expressivity and social interactivity of online communications.

4 Affective Haptic Devices
In order to support the affective communication, we implemented several novel
haptic gadgets embedded in iFeel_IM!. They make up three groups. First one isintended for emotion elicitation implicitly (HaptiHeart, HaptiCooler, HaptiWarmer), second type evokes affect in a direct way (HaptiTickler), and third one
uses sense of social touch (HaptiHug) for mood influence. All these devices produce a sense of touch including kinesthetic and coetaneous channels. Kinesthetic
stimulations, produced by forces exerted on body, are sensed by mechanoreceptors
in the tendons and muscles [7]. This channel is highly involved in sensing thestimulus produced by HaptiHug device. On the other hand, mechanoreceptors in theskin layers are responsible for cutaneous stimulation perception. Different types oftactile corpuscles allow us sensing thermal property of the object (HaptiCooler,
HaptiWarmer), pressure (HaptiHeart, HaptiTickler), vibration frequency (HaptiTickler), and stimuli location (localization of stimulating device position enables
association with particular physical contact).
4.1 HaptiHug: The Way of Realistic Hugging over Distance
When people are hugging, they generate pressure on the back of each other by hands,
and on the chest area simultaneously. The key feature of the developed HaptiHug isthat it physically reproduces the hug pattern similar to that of human-human interaction. The hands for a HaptiHug are sketched from a real human and made from soft
material. The important point is that the hands were designed in such a way that theuser feels as if the friend’s hands actually contact him. Couple of oppositely rotating
motors are incorporated into the holder placed on the user’s chest area. Soft Hands,aligned horizontally, contact back of the human. Shoulder strips, supporting motor
holder and Soft Hands, allow aligning vertical position of the device on human torso.
Once ‘hug’ command is received, couple of motors tense the belt, pressing thus hands
and breast part of HaptiHug in direction of human body (Fig. 3).

634

D. Tsetserukou et al.

Fig. 3. Structure of wearable HaptiHug device

The duration and intensity of the hug is controlled by the software according to thedetected emoticon or keyword. The Driver Box regulates the current magnitude in DC
motor from 0 mA to 500 mA. For presentation of plain hug level (e.g., ‘(>^_^)>’,‘{}’, ‘<h>’), big hug level (e.g., ‘>:D<’, ‘{{}}’), and great big hug level (e.g., ‘gbh’,‘{{{}}}’), the pressure of 200 N/m with duration of 2 sec, the pressure of 300 N/mwith duration of 3 sec, and the pressure of 450 N/m with duration of 4 sec, was applied on the user’s back and chest, respectively.
The technical specification of the device is as follows. DC motors RE 10 1.5 W in
combination with planetary gearhead GP 10 A (gear ratio of 64:1) generate stall
torque of 192.6 mNm. The pressure produced on the human chest by tensed belt
equals 450 N/m at the most. Motor holder was manufactured on 3D printer Dimension DS 768 from ABS plastic material. Special slots were provided to make the passfor pressing belt.
Preliminary, Soft Hands were printed on 3D printer from ABS with thickness of 1
mm. However, plate flexibility was not enough to achieve touch sensation similar tohuman hand. Therefore, we decided to make hands from compliant rubber-spongematerial. The contour profile of Soft Hand is sketched from the male human and has
front-face area of 155.6 cm2. Two identical pieces of Soft Hand of 5 mm thickness
were sandwiched by narrow belt slots and connected by plastic screws. Such structure
provides enough flexibility to tightly fit to the human back surface while beingpressed by belt. Moreover, belt can loosely move inside the Soft Hands during tension
period. The structure of HaptiHug is presented in Fig. 4.

Fig. 4. Structure of Soft Hands

iFeel_IM! Emotion Enhancing Garment for Communication

635

HaptiHug device has lightweight tri-glide bucklers and side release fastener integrated into the motor holder to facilitate easy adjustment of the belt size to her/his
body sizes and detaching it in a natural and rapid manner.
4.2 HaptiHeart, HaptiCooler, HaptiWarmer, HaptiTickler. Or How We Can
Enhance and Influence on Our Emotions by Haptics
Each emotion is characterized by a specific pattern of physiological changes. Fearfeatures cold, sweaty hands, tensed muscles, increased heart rate, joy – warm hands,
increased heart rate. We selected four distinct emotions with strong physical features:‘anger’, ‘fear’, ‘sadness’, and ‘joy’.
There is no doubt that feelings are intuitively connected with the heart, and ourlexicon confirms this. The research on interplay between heart rate and emotions
revealed that different emotions are associated with distinct patterns of heart ratevariations [20]. We developed heart imitator HaptiHeart to produce special heartbeat
patterns according to emotion to be conveyed or elicited (sad associated with slightly
intense heartbeat, anger with quick and violent heartbeat, fear with intense heart rate).
We take advantage of the fact that our heart naturally synchronizes with the heartof a person we hold or hug. Thus, the heart rate of a user is influenced by haptic perception of the beat rate of the HaptiHeart. Furthermore, false heart beat feedback candirectly be interpreted as real heart beat, so it can change the emotional perception.
The HaptiHeart consists of two modules: flat speaker FPS 0304 and speaker holder.
The flat speaker sizes (66.5 x 107 x 8 mm) and rated input power of 10 W allowed us todesign powerful and relatively compact HaptiHeart device producing realistic heartbeating sensation with high fidelity. The 3D model is presented in Fig. 5.

Fig. 5. HaptiHeart layout

We recorded the heartbeat patterns for the cases of experiencing fear, anger, andsadness. The pre-recorded sound signal with low frequency generates the pressure on
the human breast through vibration of the speaker surface.
HaptiTickler is responsible for joy emotion evocation. The idea behind this device
is to reproduce effect of “Butterflies in the stomach” (fluttery or tickling feeling inthe stomach felt by people experiencing love) by means of circular arrays of vibration
motors attached to user abdomen area.

636

D. Tsetserukou et al.

HaptiWarmer is intended for simulation of warmth on the human skin to evoke either aggression or pleasant feeling. It was proved that uncomfortably hot temperatures
arouse anger feelings [21]. On the other hand, temperatures within the comfortableregion can elicit positive emotions. Based on this evidence, we change the temperature of Peltier element inside HaptiWarmer to be in the limits of 25-36 °C in the caseof joy and temperature range of 40-50 °C for influencing on anger state. With motivation to boost the fear, we designed HaptiCooler that produces “Coldness on back”
effect simulated by the cold airflow along with cold side of Peltier element.

5 Conclusion and Future Research
In a nutshell, while developing the iFeel_IM! system, we attempted to bridge the gap
between mediated and face-to-face communications by enabling and enriching thespectrum of senses such as vision and touch along with cognition and inner personalstate. In the paper we described the architecture of iFeel_IM! and development ofnovel haptic devices, such as HaptiHeart, HaptiHug, HaptiTickler, HaptiCooler, andHaptiWarmer. These devices were designed with particular emphasis on natural andrealistic representation of the physical stimuli, modular expandability, and ergonomic
human-friendly design. User can perceive the intensive emotions during online communication, use desirable type of stimuli, comfortably wear and easily detach devices
from torso. We are planning to conduct the user study experiments to estimate theperformance of the developed system.
On-line communicating users are usually invisible for each other (non-verbal
channel is deactivated), and can dissemble actual emotions. This can lead to confusion during text messaging. The iFeel_IM! system can make communication emotionally truthful, so that users will tend to express emotions they actually perceive. It
should be noted, that our system can not only enhance the emotional state of the user
but also generate emotional feedback (empathy) in communicating person. In future
research, we will investigate the topic “how can we make the partner to feel our own
emotions”. We believe that iFeel_IM! will encourage users to get in “touch” with
their emotions and to make social contact with other people in on-line communicationricher and more enjoyable.
Computer gaming is very interesting application area for affective haptic devices.
The success of Wii has demonstrated that better graphics alone (PS3, Xbox360) is not
the key to the game success. Its popularity is mainly due to high involvement of theuser into the game through interactivity. By using kinaesthetic and tactile senses withWii controller, players experience greater physiological arousal. We are planning todesign two types of interactive affective game with emotional feedback to create
fantastic and emotionally charged playing experiences: (1) player goal is to changeemotional state of the virtual characters by performing social touch actions (hugging,
patting on the back, handshaking, etc.); (2) during on-line communication players
attempt to convey own emotions to partner.
Acknowledgments. The research is supported in part by a Japan Society for the Promotion of Science (JSPS) Postdoctoral Fellowship for Foreign Scholars.

iFeel_IM! Emotion Enhancing Garment for Communication

637

References
1. Damasio, A.: The Feeling of What Happens: Body, Emotion and the Making of Consciousness. Vintage, London (2000)
2. Picard, R.W.: Affective Computing. MIT Press, Cambridge (2000)
3. MacLean, K., Enriquez, M.: Perceptual Design of Haptic Icons. In: EuroHaptics, pp. 351–
363 (2003)
4. Shin, H., Lee, J., Park, J., Kim, Y., Oh, H., Lee, T.: A Tactile Emotional Interface for Instant Messenger Chart. In: 12th International Conference on Human-Computer Interaction,
pp. 166–175. Springer, Heidelberg (2007)
5. Rovers, A.F., Van Essen, H.A.: HIM: A Framework for Haptic Instant Messaging. In: 26th
SIGCHI Conference on Human Factors in Computing Systems, pp. 1313–1316. ACM,
New York (2004)
6. Mathew, D.: vSmileys: Imagine Emotions through Vibration Patterns. In: Alternative Access: Feeling and Games (2005)
7. Kandel, E.R., Schwartz, J.H., Jessel, T.M.: Principles of Neural Science. McGraw-Hill,
New-York (2000)
8. James, W.: What is an Emotion? Mind 9, 188–205 (1884)
9. Prinz, J.J.: Gut Reactions: A Perceptual Theory of Emotion. Oxford University Press, Oxford (2004)
10. Decaria, M.D., Proctor, S., Malloy, T.E.: The Effect of False Heart Rate Feedback on Self
Reports of Anxiety and on Actual Heart Rate. Behavior research & Therapy 12, 251–253(1974)
11. Briese, E.: Emotional Hyperthermia and Performance in Humans. Physiology & Behavior
58(3), 615–618 (1995)
12. Kok, R., Broekens, J.: Physical Emotion Induction and Its Use in Entertainment: Lessons
Learned. In: 1st IFIP Entertainment Computing Symposium, pp. 33–48. Springer, Heidelberg (2008)
13. Collier, G.: Emotional Expression. Lawrence Erlbaum Associates Inc., New Jersey (1985)
14. Haans, A., Jsselsteijn, W.I.: Mediated Social Touch: A Review of Current Research andFuture Directions. Virtual Reality 9, 149–159 (2006)
15. Hug Shirt. CuteCircuit Company, http://www.cutecircuit.com
16. DiSalvo, C., Gemperle, F., Forlizzi, J., Montgomery, E.: The Hug: An Exploration of Robotic Form for Intimate Communication. In: 12th IEEE International Workshop on Robotand Human Interactive Communication, pp. 403–408. IEEE Press, New-York (2003)
17. Neviarouskaya, A., Prendinger, H., Ishizuka, M.: User Study of AffectIM, an EmotionallyIntelligent Instant Messaging System. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.)
IVA 2008. LNCS, vol. 5208, pp. 29–36. Springer, Heidelberg (2008)
18. Neviarouskaya, A., Prendinger, H., Ishizuka, M.: Textual Affect Sensing for Sociable andExpressive Online Communication. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII
2007. LNCS, vol. 4738, pp. 220–231. Springer, Heidelberg (2007)
19. Izard, C.E.: Human Emotions. Plenum Press, New York (1977)
20. Anttonen, J., Surakka, V.: Emotions and Heart Rate while Sitting on a Chair. In: 27th
SIGCHI Conference on Human Factors in Computing Systems, pp. 491–499. ACM, NewYork (2005)
21. Anderson, C.A.: Temperature and Aggression: Ubiquitous Effects of Heat on Occurrenceof Human Violence. Psychological Bulletin 106(1), 74–96 (1989)