Make-a-Face: A Hands-free, Non-Intrusive Device for
Tongue/Mouth/Cheek Input Using EMG
Takuro Nakao

Keio University Graduate School ofMedia Design,
a8111140@keio.jp

Yun Suen Pai

Keio University Graduate School ofMedia Design,
yspai1412@gmail.com

Hideaki Kimata

NTT Media Intelligence LaboratoriesJapanhideaki.kimata.yu@hco.ntt.co.jp

Megumi Isogai

NTT Media Intelligence Laboratories
Japan
megumi.isogai.ks@hco.ntt.co.jp

Kai Kunze

Keio University Graduate School ofMedia Design,
kai.kunze@gmail.com

Figure 1: (a) Make-a-Face as a wearable and integrated with VR, (b) sensed raw electromygraphy data, and (c) Make-a-Faceinterface and gesture classification

ABSTRACT
Current devices aim to be more hands-free by providing users withthe means to interact with them using other forms of input, such as
voice which can be intrusive. We propose Make-a-Face; a wearable
device that allows the user to use tongue, mouth, or cheek gestures
via a mask-shaped device that senses muscle movement on thelower half of the face. The significance of this approach is threefold:
1) It allows a more non-intrusive approach to interaction, 2) wedesigned both the hardware and software from the ground-up toaccommodate the sensor electrodes and 3) we proposed several
use-case scenarios ranging from smartphones to interactions withvirtual reality (VR) content.

Input Using EMG. In Proceedings of SIGGRAPH ’18 Posters, Vancouver, BC,Canada, August 12-16, 2018, 2 pages.
https://doi.org/10.1145/3230744.3230784

1INTRODUCTION

We use more and more computing in everyday situations from IoT
in our environment to wearable glasses like HoloLens or Intel’sVaunt. Interacting with them is usually not hands-free or if mostinteraction modes (speech etc.) can be considered disruptive orsocially unacceptable especially when socializing. The hands andespecially fingers are perfect tools for interactions as they containsome of the densest areas of nerve endings in the human body. TheCCS CONCEPTS
only other part that has similar areas is the human face, particularly• Computing methodologies → Virtual reality; • Hardware
the tongue [Coren 2003]. There is a lot of related work regarding→ Sensor devices and platforms;
tongue-based interactions. However, most solutions use a camera
to capture input, require an item in the mouth, or are radar-based
KEYWORDS
more suitable for fast gestures [Goel et al. 2015; Li et al. 2015; Niu
Hands-free, non-intrusive, electromyography, face input
et al. 2014; Sasaki et al. 2011]. There are also a couple of electromyoACM Reference format:
graphy based solutions, yet they mostly have obtrusive setups with
Takuro Nakao, Yun Suen Pai, Megumi Isogai, Hideaki Kimata, and Kai Kunze.
more electrodes (as they are aimed at the disabled or paralyzed) or2018. Make-a-Face: A Hands-free, Non-Intrusive Device for Tongue/Mouth/Cheek they just aim at detecting tongue movements [Sasaki et al. 2016].
Therefore, we propose Make-a-Face, a hands-free wearable soluPermission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributed
tion that detects the tongue, mouth, and cheek position based on
for profit or commercial advantage and that copies bear this notice and the full citationelectromyography (EMG) sensors. The use of EMG to detect tongue
on the first page. Copyrights for third-party components of this work must be honored.
position has been explored before [Zhang et al. 2014], as well as
For all other uses, contact the owner/author(s).
SIGGRAPH ’18 Posters, August 12-16, 2018, Vancouver, BC, Canada
using other means like face-mounted pressure sensors [Cheng et al.
© 2018 Copyright held by the owner/author(s).
2014]. However, these approaches do not include any hardware deACM ISBN 978-1-4503-5817-0/18/08.
sign and focused on tongue sensing only, making them a nonviable
https://doi.org/10.1145/3230744.3230784

SIGGRAPH ’18 Posters, August 12-16, 2018, Vancouver, BC, Canada

Nakao, et al.

device for public use. Make-a-Face comes with its own customdesigned mounting device that attaches to the user’s ears and canbe combined with a face mask for increased subtlety and privacy.
Since Make-a-Face focuses on detection on the lower half of the face,it does not obstruct the use of head-mounted display(HMD), and we
also developed a VR-based solution where it is mounted to the HMDfor ease of use. In addition to the pre-trained tongue/mouth/cheekgestures for each application scenario, the users also can train theircustom gestures using Make-A-Face getting a confidence level as
feedback on how well the custom gesture can be detected.

2SYSTEM DESCRIPTION

Make-a-Face uses the OpenBCI Cyton board with four activatedchannels placed on the user’s face, specifically the orbicularis orisof the face, two above and two below the mouth. One referenceelectrode touches the ear lobe to reduce noise and the bias electrode
touches another. The signals are processed with a notch filter of 60
Hz and bandpass filter of 7 to 13 Hz for further noise removal. Thewires of each electrode are carefully lined through the 3D-printed
mount that touches the electrodes to all the desired contact points.
The Cyton board itself is powered by four 1.5V AA batteries andcommunicates wirelessly with our Windows or Mac computer via
a universal serial bus (USB) dongle. The mount itself was designed
using Fusion 3D to accommodate each of the mounted electrodesand fits onto most ear, with its design taking cues from Bluetoothpowered microphones or earphones. We also have another custom
mount for VR headset, with our first prototype being for the HTC
Vive. The mount hooks onto the cavity of the headset under thefoam guard.
We then streamed the EMG signals via the user datagram protocol (UDP) to a Python program that further processes each of
the signals as features for our machine learning algorithm. Fromthese signals, we added four additional features; mean, standard
deviation, maximum, minimum, and root mean square values toincrease the accuracy. We then use the Logistic Regression algorithm to classify these features into five classes; neutral, top left,top right, bottom left, and bottom right, depending on the current
tongue position and mouth deformation. The program interface
displays the confidence level of each classification and a facial heat
map for each gesture. From there, we further stream UDP data of
the classified signals to corresponding paired devices, ranging fromsmart phones to VR displays.

3

USER EXPERIENCE

Based on our user experience, we found that Make-a-Face can beadapted to a wide variety of scenarios. We used Make-a-Face oneveryday devices while performing activities like standing in atransit train and jogging to control our smart phone activity. To
interact with a smart phone while being in transit, our device was
especially useful for larger smart phones that occasionally require
both hands to use, which can potentially be dangerous as one handshould be occupied on a handle at all times. For running, it wasparticularly useful to control our smart phone without the need
to use our hands. We also developed a VR scenario that utilizes
Make-a-Face for input and interaction, freeing our hands for othertracking and gesture activities. We believe it provides an especially

Figure 2: Make-a-Face used in various hands-free scenarios
like (a) standing in a transit train with a mask, (b) running,(c) driving, and (d) gaming in VR
interesting input modality for mobile VR and AR headsedts. Oneof the added benefits of Make-a-Face is due the mounting of theelectrodes on the face instead of the interior of the mouth which
can be unhygienic.

4CONCLUSIONS

Make-a-Face provides the user with a hands-free input device thateliminates any intrusive or embarrassing properties of other available devices. In the future, we wish to perform an in-depth studyof the accuracy of detection and the use of deep learning to allow
calibration-free use cases.

REFERENCES
Jingyuan Cheng, Ayano Okoso, Kai Kunze, Niels Henze, Albrecht Schmidt, Paul Lukowicz, and Koichi Kise. 2014. On the tip of my tongue: a non-invasive pressure-based
tongue interface. In Proceedings of the 5th Augmented Human International Conference. ACM, 12.
Stanley Coren. 2003. Sensation and perception. Wiley Online Library.
Mayank Goel, Chen Zhao, Ruth Vinisha, and Shwetak N Patel. 2015. Tongue-inCheek: Using Wireless Signals to Enable Non-Intrusive and Flexible Facial Gestures
Detection. In Proceedings of the 33rd Annual ACM Conference on Human Factors in
Computing Systems. ACM, 255–258.
Zheng Li, Ryan Robucci, Nilanjan Banerjee, and Chintan Patel. 2015. Tongue-n-cheek:
non-contact tongue gesture recognition. In Proceedings of the 14th International
Conference on Information Processing in Sensor Networks. ACM, 95–105.
Shuo Niu, Li Liu, and D Scott McCrickard. 2014. Tongue-able Interfaces: Evaluating
Techniques for a Camera Based Tongue Gesture Input System. In Proceedings of the16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS’14). ACM, New York, NY, USA, 277–278. https://doi.org/10.1145/2661334.2661395
Makoto Sasaki, Takayuki Arakawa, Atsushi Nakayama, Goro Obinata, and Masaki
Yamaguchi. 2011. Estimation of tongue movement based on suprahyoid muscle
activity. In Micro-NanoMechatronics and Human Science (MHS), 2011 International
Symposium on. IEEE, 433–438.
Makoto Sasaki, Kohei Onishi, Dimitar Stefanov, Katsuhiro Kamata, Atsushi Nakayama,Masahiro Yoshikawa, and Goro Obinata. 2016. Tongue interface based on surface
EMG signals of suprahyoid muscles. ROBOMECH Journal 3, 1 (2016), 9.
Qiao Zhang, Shyamnath Gollakota, Ben Taskar, and Raj P N Rao. 2014. Non-intrusive
tongue machine interface. In Proceedings of the 32nd annual ACM conference on
Human factors in computing systems. ACM, 2555–2558.