Towards Enhancing Emotional Responses to Media using
Auto-Calibrating Electric Muscle Stimulation (EMS)
Takashi Goto

Benjamin Tag

Keio University Graduate School of Media Design
takashi.goto@kmd.keio.ac.jp

Keio University Graduate School of Media Design
tagbenja@kmd.keio.ac.jp

Kai Kunze

Tilman Dingler

Keio University Graduate School of Media Design
kai@kmd.keio.ac.jp

ABSTRACTWe evaluate the use of Electric Muscle Stimulation (EMS) as amethod of amplifying emotional responses to multimedia content.
This paper presents an auto-calibration method to stimulate two
facial expressions using EMS. We focus on two expressions: frownand smile. We attempted control of facial muscles with facial feedback for automatically calibrating these facial expressions: ourcomputer vision system detects the facial expression and autocalibrates the EMS parameters (intensity and duration) based on
the user’s current facial expression. We present results from a pilot
study with four participants evaluating the auto-calibration systemand collecting initial feedback on the use of EMS to augment, forexample, media experiences: while watching movies we can enhance the emotional response of the users during happy and sadscenes by stimulating corresponding face muscles.

CCS CONCEPTS•Human-centered computing → Human computer interaction (HCI); Interactive systems and tools;

KEYWORDS
Affective computing, Electrical Muscle Stimulation (EMS), facial
feedback, emotion
ACM Reference format:
Takashi Goto, Benjamin Tag, Kai Kunze, and Tilman Dingler. 2018. Towards
Enhancing Emotional Responses to Media using Auto-Calibrating ElectricMuscle Stimulation (EMS). In Proceedings of The 9th Augmented Human International Conference, Seoul, Republic of Korea, February 7–9, 2018 (AH2018),
2 pages.
DOI: 10.1145/3174910.3174939

1BACKGROUND AND MOTIVATION

When being agitated, people often begin to tremble while theirheart starts racing. According to the James-Lange theory [1], emotions are accompanying symptoms of physiological reactions toan event. In other words, peoples’ movements or gestures related
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
AH2018, Seoul, Republic of Korea© 2018 Copyright held by the owner/author(s). 978-1-4503-5415-8/18/02. . . $15.00
DOI: 10.1145/3174910.3174939

Osaka Prefecture University
tilman@cs.osakafu-u.ac.jp
to physiological reactions initiate the experience of an emotional
state. A similar theory is described by Tomkins [6], who found that
facial movements can also influence emotional experiences: themore people smile the happier they feel, while when they frown,
they can be made feel less happy. According to these theories,
we can postulate that emotions are, therefore, influenced by ourphysiological reactions to stimuli.
Current multimedia experiences comprise high-quality images
and high-fidelity audio. The different modalities add to the viewer’s
experience in different ways, e.g., by enhancing mood, intensifying
emotions, and adding to the immersion felt. In our work, we explore a new way to add another dimension, namely by augmenting
viewers’ physiological experience with EMS devices. We trigger
certain facial expressions with the goal of inducing the mood associated with these expressions. By actuating facial muscle groups
through a set of electrode arrays, we are able to externally trigger
smiles and frowns on the user’s face.
To facilitate the calibration of the EMS intensity, we created a system that uses visual imaging recognition for self-calibration. Basedon OpenCV, the system detects smiles as well as frowns and adjusts
its own intensity for triggering the respective facial expressions.
According to the facial feedback hypothesis, facial movements caninfluence emotional experiences. The HappinessCounter proposedby Tsujita et al. [7] combines visual smile recognition and user
feedback. Hassib et al. [3] communicated emotional states throughemotional gestures by means of EMS between two persons. Bothof these studies intend to trigger emotional changes through physiological activity.
In our work, we investigate the feasibility of augmenting media
experiences by inducing emotional states through EMS applied tothe facial region. Results of our pilot study indicate that EMS could
be a suitable modality to enhance the emotional response to media,
while most study participants welcomed stimulation enhancing
sadness while watching a movie.

2EXPERIMENTTo assess the feasibility of our approach, we conducted a pilot
study, in which we tested the system’s self-calibration and collectedsubjective feedback on enhancing media experiences through EMS
as proposed by Dingler et al. [2].
Apparatus. In this study, we used the EMS device presented byPfeiffer et al. [4]: it uses four electrodes and thereby actuates twomuscles. In addition, we used OpenCV to detect facial expressions.

AH2018, February 7–9, 2018, Seoul, Republic of Korea

Goto et al.

3DISCUSSION

Participants reported feeling stronger emotions when EMS was applied while watching videos. P1 stated that EMS helped to ”enhancenegative emotion”. EMS as an external stimulus circumvents theuser’s initiative to actuate muscles. A similar phenomenon, however, is common when we think of our hands starting to tremble
when we are nervous. The timing of applying EMS was critical:
during the experiment, EMS applied at random times tended toconfuse participants rather than add to the media experience. After this initial study exploration, we are planning to expose moreparticipants to different types of emotional intensities in order todetermine whether EMS stimulation can actually enhance the subjective emotional response. We will, thereby, evaluate electrodeplacement, which seems to be subject to individual face topologies.
Figure 1: Our system uses image processing to detect facial
expressions and adjusts the duration and intensity of stimulation accordingly to trigger smiles and frowns via EMS stimulation on the user’s face.
The facial recognition system is based on Enrique Correa’s work andclassifies emotions, such as happiness and sadness1 . For calibrating
facial expression, the intensity is initially set to 0 and changes by+-1 every second until the expression algorithm detects a happy or
sad face.
Procedure. Four participants took part in this study (2 female,mean age M=27, SD=6). Prior to the experiment, we explained thepurpose of the study and asked to sign a consent form and fill out
a demographics questionnaire. The experiment consisted of twosteps: 1) the self-calibration and 2) the exposure to two short videos
containing happy and sad scenes, which were enhanced by EMS.
First, we evaluated the self-calibration feature: each participant wasequipped with electrodes placed on their faces (see Figure 1). To
make participants smile or frown, the system proceeded as follows:
first, the participant’s face was actuated by applying EMS for one
second. Next, depending on the image processing and detectionof the facial expression, the intensity of the EMS stimulation wasadjusted until a smile or frown could be detected reliably. Measures included emotion detection success, applied intensity of thestimulation. Finally, we applied a survey to collect participants’feedback on their perception of face actuation. In the second part ofthe study, we exposed participants to short video clips, 2-3 minutes
in duration. We selected four videos from the ”FilmStim” movie
clips database [5] which were ranked among the top ten videos on
evoking positive vs. negative emotions. Besides two positive andtwo negative videos, we used a neutral one to elicit a neutral state
in between video exposure. We alternated between positive andnegative videos, their sequence was randomized.
Results. We recorded participant’s facial expression between experiments. During the calibration phase, participants often claimed
that they perceived facial expressions different from the ones intended. They could feel the changing intensity of the stimulation,
especially during increases. EMS provided during the videos waswelcomed as an interesting modality that could potentially enhancetheir emotion during video watching.
1 https://github.com/isseu/emotion-recognition-neural-networks

4CONCLUSION

We present a self-calibrating EMS system for triggering facial expressions during media consumption. In a pilot study, we assessed
the feasibility of using facial expression detection to calibrate theintensity of EMS stimuli. The system triggers smiles and frowns
during video clips with the goal of amplifying happy or sad emotions. While results and feedback varied greatly between users,
triggering frowns was welcomed the most. As emotions can beenhanced by performing appropriate movements [6], our investigation is a first step towards using EMS as an augmentation of users’emotional experiences during, for example, cinematic experiences.
While the exact stimulation still needs to be tweaked, we see great
potential for using EMS to enhance emotions, trigger moods, andadd immersion to media experiences.

ACKNOWLEDGMENTS
We thank the participants of our study. We further acknowledge thefinancial support by JST (CREST, Presto), Grant No: JP- MJPR16D4.

REFERENCES[1] Walter B Cannon. 1927. The James-Lange theory of emotions: A critical examination and an alternative theory. The American journal of psychology 39, 1/4 (1927),106–124.
[2] Tilman Dingler, Takashi Goto, Benjamin Tag, and Kai Kunze. 2017. EMS Icons:
Conveying Information by Analogy to Enhance Communication Through Electrical Muscle Stimulation. In Proc. of the 2017 ACM International Joint Conference on
Pervasive and Ubiquitous Computing (UbiComp ’17). ACM, New York, NY, USA,732–739. https://doi.org/10.1145/3123024.3129275[3] Mariam Hassib, Max Pfeiffer, Stefan Schneegass, Michael Rohs, and Florian Alt.
2017. Emotion Actuator: Embodied Emotional Feedback Through Electroencephalography and Electrical Muscle Stimulation. In Proc. of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17). ACM, New York, NY,
USA, 6133–6146. https://doi.org/10.1145/3025453.3025953[4] Max Pfeiffer, Tim Duente, and Michael Rohs. 2016. Let Your Body Move: A Prototyping Toolkit for Wearable Force Feedback with Electrical Muscle Stimulation.
In Proc. of the 18th International Conference on Human-Computer Interaction withMobile Devices and Services (MobileHCI ’16). ACM, New York, NY, USA, 418–427.
https://doi.org/10.1145/2935334.2935348[5] Alexandre Schaefer, Frédéric Nils, Xavier Sanchez, and Pierre Philippot. 2010.
Assessing the effectiveness of a large database of emotion-eliciting films: A new
tool for emotion researchers. Cognition and Emotion 24, 7 (2010), 1153–1172.
[6] Silvan Tomkins. 1962. Affect imagery consciousness: Volume I: The positive affects.
Springer publishing company.
[7] Hitomi Tsujita and Jun Rekimoto. 2011. Smiling Makes Us Happier: Enhancing
Positive Mood and Communication with Smile-encouraging Digital Appliances.
In Proc. of the 13th International Conference on Ubiquitous Computing (UbiComp’11). ACM, New York, NY, USA, 1–10. https://doi.org/10.1145/2030112.2030114