UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

EMS Icons: Conveying Information by
Analogy to Enhance Communication
Through Electrical Muscle Stimulation
Tilman Dingler
Osaka Prefecture University
Osaka, Japantilman@cs.osakafu-u.ac.jp

Benjamin Tag
Keio University Graduate Schoolof Media Design
Yokohama, Japantagbenja@kmd.keio.ac.jp

Takashi Goto
Kai Kunze
Keio University Graduate School Keio University Graduate Schoolof Media Designof Media Design
Yokohama, Japan
Yokohama, Japan
takashi.goto@kmd.keio.ac.jp
kai@kmd.keio.ac.jp

Abstract
Electrical Muscle Stimulation (EMS) has recently received
an increased amount of attention from the HCI community.
It has been used to remote control users for navigation andinstrument playing, but also as a method to convey haptic
feedback in VR, for example. As EMS devices become commercially available and application research continues, we
explore EMS as a modality to convey information throughactuation and as a means to induce and communicate emotions and moods. In this position paper, we present theresults from two focus groups on using EMS for interpersonal communication as a way to send and receive emoticons through electrical stimulation. We argue that so-called"EMS Icons" have the potential to become part of multimedia experiences and more broadly of User Interfaces as a
haptic variant in analogy to visual and auditory icons.

Author Keywords
EMS Icon; EMS; Emoji; Emoticon; Remote Communication;
Immersion
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the Owner/Author.
UbiComp/ISWC’17 Adjunct , September 11-15, 2017, Maui, HI, USA© 2017 Copyright is held by the owner/author(s).
ACM ISBN 978-1-4503-5190-4/17/09.

https://doi.org/10.1145/3123024.3129275

732

ACM Classification Keywords
H.5.2 [Information interfaces and presentation (e.g., HCI)]:
User Interfaces

UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

Introduction
First found on Japanese mobile phones, the late 1990s
gave birth to a new tool to communicate emotions, moods,
and sentiments: emotional icons, so called "emoticons",
combined mostly alphanumeric characters and punctuation
to create simple face-like expressions depicting feelings,
moods, and sensations. Even though emoticons opened
up various new ways of conveying moods, their simplicity
limited the expressive range and breadth of usage. In contrast, the pictographic characteristics of the "emoji" (i.e., "e"from Japanese "Picture", and "moji" from Japanese "character" or "letter") promised much wider reaching explanatory
qualities. Not only could facial expressions be transmitted,
but depictions of hand gestures, images of food, flags ofcountries, activities etc. became cornerstones of a new
ubiquitous language.

in which we explored combinations of facial expressions
and body postures to express, among others, the six basic
human emotions (anger, happiness, surprise, disgust, fear,sadness) [2]. In this paper, we explore a first set of EMS
Icons that represent respective emotional expressions tobe applied to remote communications through emotions,moods, and sentiments. The application scenarios presented push the boundaries from sole emoji usage for emotion transmission and communication to multimodal entertainment experiences (concerts, cinemas, theaters) throughmood induction and use in computing systems more generally. Based on the discussions in our focus groups we
conclude this paper by presenting a broader range of EMS
Icons and where they can be implemented in future User
Interfaces (UIs) and applications.

Related Work

Figure 1: EMS Icons enable a new
output modality for communicating
information, conveying emotions,triggering moods, and supporting
immersive experiences.

While emoticons are universally intelligible with the ability
to stress and support verbal expressions, conveying emotional sentiments is mostly restricted to conventional output
modalities, such as visuals and sometimes audio. Therefore, we propose "EMS Icons" as a new output modality thatutilizes Electrical Muscle Stimulation (EMS) to communicate
information, convey emotions, trigger moods, and support
immersive experiences. Simple icons, such as emoji or
emoticons can be used to trigger the transmission of a setof electrical stimulation patterns that cause the creation ofa smile, frown, or specified posture on the receiving person. Emoji typically show simplified facial details that depict
expressions of emotions, such as happiness, sadness, oranger. These could be used as templates for applying EMS
technology to key regions in the recipient’s face, limbs or
upper body.
To investigate the notion of sending emoticons and actuating recipients’ body parts, we conducted two focus groups,

733

EMS uses electric impulses to mimic the action potential that comes from the central nervous system, therefore
causing a muscle to contract. Commonly, EMS is administered using electrode pads placed on the muscle group
to be contracted. Researchers apply EMS mostly to influence physical movements of users: from triggering hand
movements to play an instrument [11] to altering the user’s
walking path, a "cruise control" for pedestrians [9]. Lopes
et al. investigated the use of EMS as an output modality to
guide the user’s hand while drawing or giving haptic feedback in VR environments [5, 6]. EMS has also been used
to encode notifications in an embodied manner [10]. Yen
et al. [13] looked into stimulating a smile using EMS: theyevaluated electrode locations and described a setup to
test whether a stimulated smile could affect people’s mood.
The positive impact of electronic smile stimulation on theemotional and mental well-being of people with major depressive disorders was presented as a possible alternative
to medication by Zariffa et al. [14] in 2014. Other indications

SESSION: WAHM

for the tight connection between physiology and psychology can be found in a study concerning the embodiment of
power through non-verbal displays conducted by Carney etal. [1].

Figure 2: Focus group participant
trying out EMS on her forearm fordemonstration purposes.

The results showed that different poses (low-power or highpower poses) tend to induce psychological, physiological,
and even behavioral changes accordingly. One method tocommunicate the emotions encoded in the embodiments
and physiological expressions is presented by Hassib etal. [4] in form of a sender-receiver system, whereas Electroencephalography (EEG) is used on the sender’s side
to read characteristics of the current emotional state and
EMS is used to receive the information and actuate the recipient’s body. An explicit way to communicate mood andpersonalize communication is the use of so-called emoji.
Emoji have been studied by Lu et al. [7] showing theirwide use to convey moods, emotions, and information about
emotion. And even though they can be understood as aubiquitous language, their study shows that different cultural
backgrounds lead to differently preferred emoji. Moreover,
Lu et al. found that culture plays an even more importantrole than language when it comes to the preferences forcertain emoji. But instead of modifying the sentiment in notifications, emoji are mostly used to amplify the content andsentiment expressed in words [12]). Furthermore, the application used to send and receive emoji seems to have an
influence on the type of pictogram used, e.g. Twitter, Facebook, WhatsApp, where private messages are dominatedby faces, but public posts by decorative emoji.
Icons are representative symbols that are widely usedin UIs as a means to communicate system or application
information in a metaphorical way. Visual icons were firstused on a Desktop PC for the Xerox Star 8010. Invented

734

by Davis Smith and designed by Norm Cox, these little
application icons used metaphors from everyday life to convey their functionality to users in a familiar way. Gaver [3]then adopted this concept and introduced auditory icons,
the audio equivalent to visual icons in computing systems.
Nowadays, such icons can be found in almost every UI—on
tablets, mobile phones, watches etc.—due to their intuitive
nature, resulting learnability, memorability, and therefore
their ability to augment the user experience. While the useof haptic icons has also been explored for UIs [8], we focused our investigation on the use of EMS to provide sensations stimulated through electric currents, so called "EMS
Icons".

Focus GroupWe conducted two focus groups with the goal of identifying
and mapping emoticons and emoji to electrical stimulationsof different body parts. Therefore, we recruited 14 participants (3 female) through university mailing lists and invitedthem to our lab in two groups of 9 and 5 respectively. Participants had a mean age of 27 years (SD = 4.2) and were
mostly graduate students with backgrounds in design andengineering. We discussed seven central questions about
EMS including application use cases in general, identifying appropriate body parts for stimulation, and its potential
to be used for communicating information with a focus ontransferring emotions and moods.
Procedure
After signing a consent form explaining the participation andallowing us to videotape the session, we started off introducing the purpose of the focus group. We presented theconcept of EMS by showing a live-demo, during which weplaced four electrode pads on the forearm of a volunteer toform a closed hand through electrical stimulation (Figure 2).

UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

Then, the moderator guided the group discussion by subsequently posing questions with regard to four aspects of
EMS:
1. General application areas for EMS and more specifically using EMS for communication purposes.
2. Listing different body parts feasible for stimulation.
3. The use of EMS to communicate/invoke different
emotions and moods, especially the six basic humanemotions as defined by Ekman [2] (see Table 1).
Figure 3: Body parts as identified
by our focus groups participants
through which emotions andmoods could conveyed through.

4. Replacing emoticons and emoji commonly used intext communication with their EMS equivalent.
For each question, participants were given a few minutes
to write down ideas on sticky notes, which were subsequently discussed and grouped by the entire group. Eachfocus group lasted about 90 minutes, for which participantsreceived candy as compensation.
Results
The first round of the discussion focused on defining general use cases for EMS and identifying appropriate bodyparts, where emotions and moods could be conveyed on.
The most prominent application themes included learning scenarios (skill transfer, N=12), use in therapy or rehabilitation programs (N=8), remote control and performances (N=9), communication (N=7), immersive experiences (N=4)), and transfer of emotions (N=5). Further
applications included haptics as an output modality, a system for enforcing safety postures in emergency situations,and the creation of phantom limbs. The most prominentbody parts to be stimulated for communication purposes
were facial muscles, neck, shoulders, diaphragm, and limbs,
such as wrists, fingers, knees, and toes. Figure 4 depicts

735

Figure 4: Participants developed general use cases for EMS andgrouped them by themes in discussions.

application use cases grouped by themes, while Figure 3
shows the identified body parts, which muscle stimulation
could be performed on.
Next, we discussed in particular how emotions could becommunicated through EMS. Participants debated the stimulation of muscle groups in order to move entire limbs aswell as how to induce facial muscle motion through micromuscle stimulation. Micro whole-body muscle stimulation
was suggested to be used to induce tingling sensations,shivering, and goosebumps, which could represent sensations and emotions, such as fear, and excitement. At thispoint, discussions moved from how to convey basic emotions such as joy, sadness, fear, and anger to more complex
and eclectic states, such as nervousness and contemptthrough, for example, finger and leg tapping as well as triggered eye blinks. Moreover, stimulating respiratory muscles
was suggested to be used to train and induce breathing
rhythms appropriate for specific situations, e.g. inducing acalm rhythm panic attacks are impending, whereas certain
tingling sensations in the abdominal region could induce
sensations of affection or joy.

SESSION: WAHM

Figure 5: Most used emoji asstated by participants, for which we
asked them to design equivalents
as EMS Icons.

Figure 6: Second set of emoji andtheir translation to muscle
stimulations.

Subsequently, we posed the question on how the specific
set of Ekman’s six basic human emotions [2] could be communicated through EMS. On that note, we also discussed
which body parts would be most expressive for conveying
these emotions. In the following, we summarize the resultsof the discussion around using EMS in relation to emoji andinducing emotions and moods.

EmotionBody Part

Stimulation

HappinessFace, upper body,arms, hands.

Forming a smile using mouth, eye squinting, increasing heart-rate,straightening upper body posture, throwing arms and hands up.

SadnessFace, neck, shoulders. Activating lacrimal gland, hunching shoulders, head bending down.

Fear

Head, upper body.

Body shiver along neck, quick breathing contractions of upper body,quick diaphragm contractions, increasing heart-rate, creating
goosebumps.

Anger

Head, shoulders, arm,hands, lower legs.

Jaw clenching, frowning brows, creating tension in shoulders,closing hands to fists, clenching teeth and jaw, deep and quick
breathing, tension in calfs.

Surprise

Face, upper body,hands.

Eyes widening, mouth opening, single sudden whole upper bodycontraction, tingling in stomach, sudden breath, throwing hands up.

EMS IconsTable 1 lists different options of body parts for each basic
human emotion as well as stimulation types that could represent these emotions. Most emotions were assigned toparts of the upper body, which shows a rather direct thanmetaphorical allocation. Obviously, aspects of technicalfeasibility and social acceptability were neglected for this
discussion, so we find stimulations that represent effects
as they appear as a direct result of an emotion felt. Thus,a triggered smile was suggested by most participants as a
happy face, but for a more comprehensive emotional sensation, body parts such as the torso, arms, and hands were to
be involved, too. Shoulders, for example, can be used to express sadness through hunching, so can anger by applying
greater tension. Participants also agreed on the importanceof subjacent muscles, such as can be found in the abdominal region, which could complementary be stimulated totrigger disgust. Stimulation of the hand and those muscles
responsible for breathing could be done to express anger
through forming a fist and adjusting the breathing frequency(shallow: fear, deep: anger).

DisgustFace, upper body.

Stomach contractions (vomiting sensation), leaning back, mouth
warping, closing mouth with tongue, stretching nose.

We further explored an EMS set of emojis by asking participants for voicing their most used emoji (see Figure 5 and 6)
and finding an EMS equivalent for each of them. One of
the most popular was the wink emoji. Participants agreed
on its equivalent EMS Icon as a one-sided triggered blink.
A similar agreement could be found for gestures of apolo-

736

Table 1: Participants were asked to select appropriate body partsfor stimulation and conveyance of six basic human emotions as
defined by Elkman [2].

getic nature, such as the bow. The respective EMS Icon
was clearly defined as a change in posture of the uprightback into a bow. Another expression both groups agreed
on was that of happiness or smiles. Participants thoughtthat triggering the cheek muscles to slightly move up andthe eyes to stretch are the appropriate way to express joy.
The biggest difference between both groups could be foundin the selection of preferred emoji. Whereas the first focus
group came up mostly with facial expressions as their dominantly used emoji, the second group presented several
choices for non-facial expressions of emotions, such as thethumbs-up gesture, or a sudden random touch as an EMS
Icon for fear.
Following this part of the discussion, we lastly analyzed
1
possible scenarios, for which theuse of communication enhanced through EMS would be feasible. The predominant
application scenario was emotion conveyance. In particular,
the potential to send exact feelings and their physical expressions to people close to the user, such as close friends,
family, or the partner. The use of EMS was therefore seen
as rather intimate. Participants further proposed exceptions,UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

however, such as bridging differences in body language thatarise from different cultural backgrounds or creating empathy with people in different situations to improve mutual
understanding. The groups also discussed the potential fordeveloping communication interfaces with autistic patientsand people with anxiety issues. Not only human-humaninteraction could be augmented with EMS Icons, but also
human-computer interaction. Especially in the gaming domain, EMS Icons would allow for new immersive experiences and communication modalities.

Discussion
In the focus groups presented, we explored a possible
transfer of emoticons and emoji to representative EMS
expressions. A notable discussion evolved around the observation that sending an emoji and stimulating the recipient’s body through EMS may change the communication
itself as the recipient rather feels or acts out the sender’semotion. This seems inherently different from simply receiving a visual emoji as confirmation of an emotion felt on thesender’s side. While currently available EMS Devices do
not necessarily provide the sufficient finesse to stimulate
complex muscle groups, there are obvious limitations as
to where electric pads can be applied to (from a technical,
social acceptance, but also safety point of view). Also, a
proliferation of EMS technologies bears a range of risks:erroneous application of the actuators could lead to uncontrolled stimulation of muscles, specifically critical when it
comes to vitally essential muscle systems such as the respiratory tracts or heart muscles. Moreover, there should
be limits to frequent stimulations of certain muscle groups,
such as facial muscles. Would frequent stimulation leadto muscle growth and deformation and therefore cause
aesthetic issues? Further, how can we prevent abuse of remotely triggering affect? Could there even arise bigger picture implications, such as emotion control and consolidation

737

by third parties? Our focus group participants pointed out
the intimate nature of communication that EMS presents,
but also identified scenarios that would involve strangers,
e.g. for communication with autistic people and bridging
cultural differences. Besides being used for interpersonalcommunication, EMS Icons further show potential to be
applied in Human-Computer interactions considering thebroader application of icons. In the following, we outline
three application scenarios, which partly emerged in our
focus group discussions and which we see subsequently
come into fruition with increasing adoption of EMS technology.
Augmenting Media Experiences through EMS
In cinema, the combination of film music and moving images create an emotional experience for the viewer, which
could not be fully conveyed by visuals only. The use of
EMS Icons in a cinematic context would allow film makers
to include an additional dimension to induce and emphasize film moods and evoke emotions. Inducing shivers
or muscle tension, for example, could be used to emphasize sensations of fear in a horror movie or theater piece.
Nervousness could further be underlined by shaking theviewer’s leg. Hence, actuators built into cinema chairs, forexample, could stimulate viewers and provide a more immersive film experience. Similarly, live music events andmusic recordings could be enhanced by triggering songspecific emotional responses, e.g. making a ballad evenmore heart-wrenching by causing the listener’s shoulder
to hunch down. Media experiences can thus be shared bytransferring emotional states across viewers and locations.
EMS Icons in Software UIs
Besides providing entertainment value, EMS Icons could
also be utilized by application programmers in Software UIs.
Just like auditory icons are commonly used to communicate

SESSION: WAHM

system status messages or trigger warnings, EMS Icons
present an additional modality to inform users. Instead ofhearing an alarm sound, the user’s hand could twitch whenattempting to delete a file or close a program. In games,
EMS Icons could be used to support immersion, while web
sites could induce a tingling feeling when hovering over a
hyperlink. Therefore, EMS Icons could provide designers
and application builders with an additional output channel.
Home desktops, but also workplaces could be augmented
by integrating EMS actuators in furniture, such as chairs
and desktops. They could be used to avoid slouching during long work sessions and initiate a healthy posture fromtime to time. Moreover, augmented workspaces could help
to keep tensions high in meetings and during tiring tasks.
In order to avoid an unfriendly and unproductive working
climate, sentiments of joy and happiness could occasionallybe triggered.
EMS in Smart Garments
While augmenting media experiences poses the obvious
next step for incorporating EMS in more consumer experiences, EMS Icons in UIs requires consumer products to be
introduced to home and office environments. Continuing thethought of EMS becoming more ubiquitous, the final stepwould be an integration into our everyday environments. We
have already mentioned furniture augmented with actuators,
but for EMS Icons to be completely mobile and pervasive,
we would need to integrate them into the very fabric of ourclothing itself. The currently developing field of smart garments could, therefore, be extended by EMS devices thatallow their wearer to receive EMS Icons anytime and anywhere. Such EMS-actuated garments can turn our dailyclothing, specific functional wear, or even seat covers—e.g.
in cinemas, therapy chairs, and sport stadiums—into receivers for EMS Icons and wirelessly communicate with738

whatever application we are currently using, be it on thecinema screen, on our laptop or mobile phone in our hand.
Multimodal sensations can, for example, be induced whenmessages are received and read so that visual emoji canbe felt on the user’s body as the message is being read.
EMS-actuated garments also have the potential to be used
to influence breathing rates and heart rates on-the-fly in order to create immersion on the one hand or calm a person
down on the other hand during emerging stages of fright,
panic or anxiety attacks.

Conclusion
In this position paper, we explore EMS as a modality toconvey information through actuation and as a means to
induce and communicate emotions and moods. Therefore,
we propose EMS Icons as a new output modality that canbe applied to augment multimedia experiences and more
broadly in User Interfaces as a haptic variant in analogy tovisual and auditory icons. We present results from two focus groups, in which we explore the extension of the emoji
concept through EMS away from a purely visual form ofcommunicating mood and sentiment towards a highly immersive physical experience. Insights from the discussions
on application use cases, body parts feasible for EMS stimulation, and ways of communicating moods and sentiments,
eventually lead us to the formulation of a broader application space for EMS Icons including multimedia experiences,
Software UIs, and smart garments. While research in the
coming years will need to focus on technicalities and aspects of social acceptance of EMS, this paper presents
a long-term vision of the concept of EMS Icons and theirpervasive use in future computing interfaces.

Acknowledgments
This work is supported by JST (CREST, Presto), Grant No:
JP- MJPR16D4.

UBICOMP/ISWC ’17 ADJUNCT, SEPTEMBER 11-15, 2017, MAUI, HAWAII, USA

References

York, NY, USA, 770–780. DOI:http://dx.doi.org/10.1145/[1] Dana R. Carney, Amy J.C. Cuddy, and Andy J. Yap.
2010. Power Posing. Psychological Science 21,
10 (2010), 1363–1368. DOI:http://dx.doi.org/10.1177/
0956797610383437[2] Paul Ekman. 1992. An argument for basic emotions.
Cognition & emotion 6, 3-4 (1992), 169–200.
[3] William W. Gaver. 1986. Auditory Icons: Using Soundin Computer Interfaces. Hum.-Comput. Interact. 2, 2
(June 1986), 167–177. DOI:http://dx.doi.org/10.1207/
s15327051hci0202_3[4] Mariam Hassib, Max Pfeiffer, Stefan Schneegass,
Michael Rohs, and Florian Alt. 2017. Emotion Actuator: Embodied Emotional Feedback Through Electroencephalography and Electrical Muscle Stimulation. In Proceedings of the 2017 CHI Conferenceon Human Factors in Computing Systems (CHI’17). ACM, New York, NY, USA, 6133–6146. DOI:
http://dx.doi.org/10.1145/3025453.3025953[5] Pedro Lopes, Sijing You, Lung-Pan Cheng, Sebastian Marwecki, and Patrick Baudisch. 2017. Providing
Haptics to Walls & Heavy Objects in Virtual Reality by
Means of Electrical Muscle Stimulation. In Proceedings of the 2017 CHI. ACM, 1471–1482.
[6] Pedro Lopes, Doăa Yüksel, François Guimbretiere,and Patrick Baudisch. 2016. Muscle-plotter: An interactive system based on electrical muscle stimulation that
produces spatial output. In Proceedings of the 29th
UIST. ACM, 207–217.
[7] Xuan Lu, Wei Ai, Xuanzhe Liu, Qian Li, Ning Wang,
Gang Huang, and Qiaozhu Mei. 2016. Learning fromthe Ubiquitous Language: An Empirical Analysis of
Emoji Usage of Smartphone Users. In Proceedings ofthe 2016 ACM UbiComp (UbiComp ’16). ACM, New739

2971648.2971724[8] Karon MacLean and Mario Enriquez. 2003. Perceptual
design of haptic icons. In Proc. of EuroHaptics. 351–
363.
[9] Max Pfeiffer, Tim Dünte, Stefan Schneegass, FlorianAlt, and Michael Rohs. 2015. Cruise control for pedestrians: Controlling walking direction using electricalmuscle stimulation. In Proceedings of the 33rd Annual
CHI. ACM, 2505–2514.
[10] Stefan Schneegass and Rufat Rzayev. 2016. Embodied
notifications: Implicit notifications through electricalmuscle stimulation. In Proceedings of the 18th HCI.
ACM, 954–959.
[11] Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2011.
PossessedHand: techniques for controlling humanhands using electrical muscles stimuli. In Proceedingsof the SIGCHI. ACM, 543–552.
[12] Channary Tauch and Eiman Kanjo. 2016. The Roles of
Emojis in Mobile Phone Notifications. In Proceedingsof the 2016 ACM UbiComp: Adjunct (UbiComp ’16).
ACM, New York, NY, USA, 1560–1565. DOI:http://dx.doi.org/10.1145/2968219.2968549[13] Lai Yen-Chin, YuanLing Feng, Kai Kunze, Junich
Shimizu, and Takuro Nakao. 2017. Eyewear to Make
Me Smile: Can Electric Muscle Stimulation increase
Happiness?. In Proceedings of the TEI. ACM, 579–
582.
[14] José Zariffa, Sander L. Hitzig, and Milos R. Popovic.
2014. Neuromodulation of emotion using functional electrical stimulation applied to facial muscles. Neuromodulation 17, 1 (2014), 85–92. DOI:
http://dx.doi.org/10.1111/ner.12056